avcodec/qtrleenc: Fix negative linesizes, don't use NULL + offset

Before commit f1e17eb446577180ee9976730aacb46563766518, the qtrle
encoder had undefined pointer arithmetic: Outside of a loop, two
pointers were set to point to the ith element (with index i-1) of
a line of a frame. At the end of each loop iteration, these pointers
were decremented, so that they pointed to the -1th element of the line
after the loop. Furthermore, one of these pointers can be NULL (in which
case all pointer arithmetic is automatically undefined behaviour).

Commit f1e17eb44 added a check in order to ensure that the elements
never point to the -1th element of the array: The pointers are only
decremented if they are bigger than the frame's base pointer
(i.e. AVFrame.data[0]). Yet this check does not work at all in case of
negative linesizes; furthermore in case the pointer that can be NULL is
NULL initializing it still involves undefined pointer arithmetic.

This commit fixes both of these issues: First, non-NULL pointers are
initialized to point to the element after the ith element and
decrementing is moved to the beginning of the loop. Second, if a pointer
is NULL, it is just made to point to the other pointer, as this allows
to avoid checks before decrementing it.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

0 files changed, 0 insertions, 0 deletions