diff options
| author | Michael Niedermayer <michael@niedermayer.cc> | 2017-05-08 12:04:09 +0200 |
|---|---|---|
| committer | Michael Niedermayer <michael@niedermayer.cc> | 2017-05-14 00:21:49 +0200 |
| commit | 0d17ecffa56c5cb1b99bc5c10a3b7c49e48db917 (patch) | |
| tree | 884a0a27720299b6825dd4eff1719702362e86dc /libavcodec | |
| parent | 88893627a1c646461f1e35b87c1d6b804bc63b91 (diff) | |
| download | ffmpeg-0d17ecffa56c5cb1b99bc5c10a3b7c49e48db917.tar.gz | |
avcodec/hqxdsp: Fix multiple runtime error: signed integer overflow: 248220 * 21407 cannot be represented in type 'int' in idct_col()
Fixes: 1405/clusterfuzz-testcase-minimized-5011491835084800
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 5d5118f81bd51b9c33500616b3c637123e8e4691)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Diffstat (limited to 'libavcodec')
| -rw-r--r-- | libavcodec/hqxdsp.c | 12 |
1 files changed, 6 insertions, 6 deletions
diff --git a/libavcodec/hqxdsp.c b/libavcodec/hqxdsp.c index feff9c0b68..04a65e7767 100644 --- a/libavcodec/hqxdsp.c +++ b/libavcodec/hqxdsp.c @@ -39,18 +39,18 @@ static inline void idct_col(int16_t *blk, const uint8_t *quant) s6 = (int) blk[6 * 8] * quant[6 * 8]; s7 = (int) blk[7 * 8] * quant[7 * 8]; - t0 = (s3 * 19266 + s5 * 12873) >> 15; - t1 = (s5 * 19266 - s3 * 12873) >> 15; - t2 = ((s7 * 4520 + s1 * 22725) >> 15) - t0; - t3 = ((s1 * 4520 - s7 * 22725) >> 15) - t1; + t0 = (int)(s3 * 19266U + s5 * 12873U) >> 15; + t1 = (int)(s5 * 19266U - s3 * 12873U) >> 15; + t2 = ((int)(s7 * 4520U + s1 * 22725U) >> 15) - t0; + t3 = ((int)(s1 * 4520U - s7 * 22725U) >> 15) - t1; t4 = t0 * 2 + t2; t5 = t1 * 2 + t3; t6 = t2 - t3; t7 = t3 * 2 + t6; t8 = (t6 * 11585) >> 14; t9 = (t7 * 11585) >> 14; - tA = (s2 * 8867 - s6 * 21407) >> 14; - tB = (s6 * 8867 + s2 * 21407) >> 14; + tA = (int)(s2 * 8867U - s6 * 21407U) >> 14; + tB = (int)(s6 * 8867U + s2 * 21407U) >> 14; tC = (s0 >> 1) - (s4 >> 1); tD = (s4 >> 1) * 2 + tC; tE = tC - (tA >> 1); |