diff options
| author | Thomas Guillem <thomas@gllm.fr> | 2018-07-04 09:05:22 +0200 |
|---|---|---|
| committer | Aman Gupta <aman@tmm1.net> | 2018-07-19 09:26:30 -0700 |
| commit | db923b3fbd3af266f8484ac55cca149797c6e18e (patch) | |
| tree | 40eb8f2033fe8e2ce54bd9af8daccd4beae22517 | |
| parent | 0a155c57bd8eb92ccaf7f5857dc6ab276d235846 (diff) | |
| download | ffmpeg-db923b3fbd3af266f8484ac55cca149797c6e18e.tar.gz | |
avcodec/videotoolboxenc: fix undefined behavior with rc_max_rate=0
On macOS, a zero rc_max_rate cause an error from
VTSessionSetProperty(kVTCompressionPropertyKey_DataRateLimits).
on iOS (depending on device/version), a zero rc_max_rate cause invalid
arguments from the vtenc_output_callback after few frames and then a crash
within the VideoToolbox library.
Signed-off-by: Aman Gupta <aman@tmm1.net>
(cherry picked from commit 93e157f40f415119ea0f94b35596965e9870f863)
| -rw-r--r-- | libavcodec/videotoolboxenc.c | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c index ac847358ab..f311149e40 100644 --- a/libavcodec/videotoolboxenc.c +++ b/libavcodec/videotoolboxenc.c @@ -1017,7 +1017,7 @@ static int vtenc_create_encoder(AVCodecContext *avctx, return AVERROR_EXTERNAL; } - if (vtctx->codec_id == AV_CODEC_ID_H264) { + if (vtctx->codec_id == AV_CODEC_ID_H264 && max_rate > 0) { // kVTCompressionPropertyKey_DataRateLimits is not available for HEVC bytes_per_second_value = max_rate >> 3; bytes_per_second = CFNumberCreate(kCFAllocatorDefault, @@ -1058,7 +1058,10 @@ static int vtenc_create_encoder(AVCodecContext *avctx, av_log(avctx, AV_LOG_ERROR, "Error setting max bitrate property: %d\n", status); return AVERROR_EXTERNAL; } + } + if (vtctx->codec_id == AV_CODEC_ID_H264) { + // kVTCompressionPropertyKey_ProfileLevel is not available for HEVC if (profile_level) { status = VTSessionSetProperty(vtctx->session, kVTCompressionPropertyKey_ProfileLevel, |