avfilter/formats: Fix double frees and memleaks on error

The formats API deals with lists of channel layouts, sample rates,
pixel formats and sample formats. These lists are refcounted in a way in
which the list structure itself contains pointers to all of its owners.
Furthermore, it is possible for a list to be not owned by anyone yet;
this status is temporary until the list has been attached to an owner.
Adding an owner to a list involves reallocating the list's list of
owners and can therefore fail.

In order to reduce the amount of checks and cleanup code for the users
of this API, the API is supposed to be lenient when faced with input
lists that are NULL and it is supposed to clean up if adding an owner
to a list fails, so that a simple use case like

list = ff_make_format_list(foo_fmts);
if ((ret = ff_formats_ref(list, &ctx->inputs[0]->out_formats)) < 0)
    return ret;

needn't check whether list could be successfully allocated
(ff_formats_ref() return AVERROR(ENOMEM) if it couldn't) and it also
needn't free list if ff_formats_ref() couldn't add an owner for it.

But the cleaning up after itself was broken. The root cause was that
the refcount was decremented during unreferencing whether or not the
element to be unreferenced was actually an owner of the list or not.
This means that if the above sample code is continued by

if ((ret = ff_formats_ref(list, &ctx->inputs[1]->out_formats)) < 0)
    return ret;

and that if an error happens at the second ff_formats_ref() call, the
automatic cleaning of list will decrement the refcount from 1 (the sole
owner of list at this moment is ctx->input[0]->out_formats) to 0 and so
the list will be freed; yet ctx->input[0]->out_formats still points to
the list and this will lead to a double free/use-after-free when
ctx->input[0] is freed later.

Presumably in order to work around such an issue, commit
93afb338a405eac0f9e7b092bc26603378bfcca6 restricted unreferencing to
lists with owners. This does not solve the root cause (the above example
is not fixed by this) at all, but it solves some crashs.

This commit fixes the API: The list's refcount is only decremented if
an owner is removed from the list of owners and not if the
unref-function is called with a pointer that is not among the owners of
the list. Furtermore, the requirement for the list to have owners is
dropped.

This implies that if the first call to ff_formats_ref() in the above
example fails, the refcount which is initially zero during unreferencing
is not modified, so that the list will be freed automatically in said
call to ff_formats_ref() as every list whose refcount reaches zero is.

If on the other hand, the second call to ff_formats_ref() is the first
to fail, the refcount would stay at one during the automatic
unreferencing in ff_formats_ref(). The list would later be freed when
its last (and in this case sole) owner (namely
ctx->inputs[0]->out_formats) gets unreferenced.

The issues described here for ff_formats_ref() also affected the other
functions of this API. E.g. ff_add_format() failed to clean up after
itself if adding an entry to an already existing list failed (the case
of a freshly allocated list was handled specially and this commit also
removes said code). E.g. ff_all_formats() inherited the flaw.

Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

0 files changed, 0 insertions, 0 deletions