rtsp: Fix infinite loop in listen mode with UDP transport

In listen mode with UDP transport, once the sender has sent
the TEARDOWN and closed the connection, poll will indicate that
one can read from the connection (indicating that the socket has
reached EOF and should be closed by the receiver as well). In this
case, parse_rtsp_message won't try to parse the command (because
it's no longer in state STREAMING), but previously just returned
zero.

Prior to f6161fccf8c5720ceac1ed1df8ba60ff8fed69f5, this caused
udp_read_packet to return zero, which is treated as EOF by
read_packet. But after that commit, udp_read_packet would continue
if parse_rtsp_message didn't return an explicit error code.

To keep the original behaviour from before that commit, more
explicitly return an error in parse_rtsp_message when in the wrong
state.

Fixes: #8840
Signed-off-by: Martin Storsjö <martin@martin.st>

1 files changed, 1 insertions, 1 deletions

diff --git a/libavformat/rtsp.c b/libavformat/rtsp.c
index 5d8491b74b..ad12f2ae98 100644
--- a/libavformat/rtsp.c
+++ b/libavformat/rtsp.c
@@ -1970,7 +1970,7 @@ static int parse_rtsp_message(AVFormatContext *s)
                 av_log(s, AV_LOG_WARNING,
                        "Unable to answer to TEARDOWN\n");
         } else
-            return 0;
+            return AVERROR_EOF;
     } else {
         RTSPMessageHeader reply;
         ret = ff_rtsp_read_reply(s, &reply, NULL, 0, NULL);