diff options
| author | Matthew Einhorn <moiein2000@gmail.com> | 2011-08-24 20:14:03 -0400 |
|---|---|---|
| committer | Michael Niedermayer <michaelni@gmx.at> | 2011-10-19 23:28:04 +0200 |
| commit | e662b263d9c500270a8f1dc7e1b81b51d5bdfd4e (patch) | |
| tree | 2c35f9fd139c6da27151019d5e0d69057d696f6c | |
| parent | e31c5ebe1146d98d17a5121312c5444432c81904 (diff) | |
| download | ffmpeg-e662b263d9c500270a8f1dc7e1b81b51d5bdfd4e.tar.gz | |
Fixes avpicture_layout to not write past buffer end.
avpicture_get_size() returns the size of buffer required for avpicture_layout.
For pseudo-paletted formats (gray8...) this size does not include the palette.
However, avpicture_layout doesn't know this and still writes the palette. Consequently,
avpicture_layout writes passed the length of the buffer. This fixes it
by fixing avpicture_layout so that it doesn't write the palette for these formats.
Signed-off-by: Matthew Einhorn <moiein2000@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
| -rw-r--r-- | libavcodec/imgconvert.c | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/libavcodec/imgconvert.c b/libavcodec/imgconvert.c index 5f5a50ea19..64b0838857 100644 --- a/libavcodec/imgconvert.c +++ b/libavcodec/imgconvert.c @@ -337,6 +337,16 @@ int avpicture_layout(const AVPicture* src, enum PixelFormat pix_fmt, int width, } } + switch (pix_fmt) { + case PIX_FMT_RGB8: + case PIX_FMT_BGR8: + case PIX_FMT_RGB4_BYTE: + case PIX_FMT_BGR4_BYTE: + case PIX_FMT_GRAY8: + // do not include palette for these pseudo-paletted formats + return size; + } + if (desc->flags & PIX_FMT_PAL) memcpy((unsigned char *)(((size_t)dest + 3) & ~3), src->data[1], 256 * 4); |