aboutsummaryrefslogtreecommitdiffstats
path: root/doc/optimization.txt
diff options
context:
space:
mode:
authorMartin Storsjö <martin@martin.st>2020-04-23 09:19:25 +0300
committerMartin Storsjö <martin@martin.st>2020-04-24 14:41:06 +0300
commitf4d8fad802b59d3099eb453dcafb08219ecfa22c (patch)
tree16acb30f62c4783fab537ae3abcc92a6d76b9e5a /doc/optimization.txt
parenta501947a840debe772458764d2b4595e403678d2 (diff)
downloadffmpeg-f4d8fad802b59d3099eb453dcafb08219ecfa22c.tar.gz
dnn-layer-mathbinary-test: Fix tests for cases with extra intermediate precision
This fixes tests on 32 bit x86 mingw with clang, which uses x87 fpu by default. In this setup, while the get_expected function is declared to return float, the compiler is (especially given the optimization flags set) free to keep the intermediate values (in this case, the return value from the inlined function) in higher precision. This results in the situation where 7.28 (which actually, as a float, ends up as 7.2800002098), multiplied by 100, is 728.000000 when really forced into a 32 bit float, but 728.000021 when kept with higher intermediate precision. For the multiplication case, a more suitable epsilon would e.g. be 2*FLT_EPSILON*fabs(expected_output), but just increase the current hardcoded threshold for now. Signed-off-by: Martin Storsjö <martin@martin.st>
Diffstat (limited to 'doc/optimization.txt')
0 files changed, 0 insertions, 0 deletions