aboutsummaryrefslogtreecommitdiffstats
path: root/doc/ffmpeg-formats.texi
diff options
context:
space:
mode:
authorMartin Storsjö <martin@martin.st>2019-12-04 13:04:41 +0200
committerMartin Storsjö <martin@martin.st>2019-12-18 15:15:29 +0200
commitaad0e26f9312d380e613e312a3b307609296fe58 (patch)
treef19909bd36c30a86c13c44202900d76afff53b0d /doc/ffmpeg-formats.texi
parente10654de2b0db0f3b4828a3cc066d6fa21d02516 (diff)
downloadffmpeg-aad0e26f9312d380e613e312a3b307609296fe58.tar.gz
checkasm: aacpsdsp: Tolerate extra intermediate precision in stereo_interpolate
The stereo_interpolate functions add h_step to the values h BUF_SIZE times. Within the stereo_interpolate C functions, the values h (h0-h3, h00-h13) are declared as local float variables, but the compiler is free to keep them in a register with extra precision. If the accumulation is rounded to 32 bit float precision after each step, the less significant bits of h_step end up ignored and the sum can deviate, affecting the end result more than the currently set EPS. By clearing the log2(BUF_SIZE) lower bits of h_step, we make sure that the accumulation shouldn't differ significantly, regardless of any extra precision in the accmulating register/variable. This fixes the aacpsdsp checkasm test when built with clang for mingw/x86_32. Signed-off-by: Martin Storsjö <martin@martin.st>
Diffstat (limited to 'doc/ffmpeg-formats.texi')
0 files changed, 0 insertions, 0 deletions