aacencdsp: Improve consistency with assembly, for x87 mathHEADmaster

Currently, the aacencdsp checkasm tests fails for many seeds,
if the C code has been built with x87 math. This happens because
the excess precision of x87 math can make it end up rounding
to a different integer, and the checkasm tests checks that the
output integers match exactly between C and assembly.

One such failing case is "tests/checkasm/checkasm --test=aacencdsp
41" when compiled with GCC. When compiled with Clang, the test
seed 21 produces a failure.

To avoid the issue, we need to limit the precision of intermediates
to their nominal float range, matching the assembly implementations.

This can be achieved when compiling with GCC, by just adding a single
cast.

To observe the effect of this cast, compile the following
snippet,

    int cast(float a, float b) {
        return (int)
    #ifdef CAST
            (float)
    #endif
            (a + b);
    }

with "gcc -m32 -std=c17 -O2", with/without -DCAST. For x86_64
cases (without the "-m32"), the cast doesn't make any difference
on the generated code.

This cast would seem to not have any effect, as a binary expression
with float inputs also would have the type float.

However, if compiling with GCC with -fexcess-precision=standard,
the cast forces limiting the precision according to the language
standard here - according to the GCC docs [1]:

> When compiling C or C++, if -fexcess-precision=standard is
> specified then excess precision follows the rules specified in
> ISO C99 or C++; in particular, both casts and assignments cause
> values to be rounded to their semantic types (whereas -ffloat-store
> only affects assignments). This option is enabled by default for
> C or C++ if a strict conformance option such as -std=c99 or
> -std=c++17 is used.

Ffmpeg's configure scripts enables -std=c17 by default.

This only helps with GCC though - the cast doesn't make any
difference for Clang. (Although, upstream Clang seems to default
to SSE math, while Ubuntu provided Clang defaults to x87 math.)
Limiting the precision with Clang would require casting to volatile
float for both intermediates here - and that does have a code
generation effect on all architectures.

[1] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

0 files changed, 0 insertions, 0 deletions