The problem with UDP for video is that, on a lossy network, you either end up sending a lot of key frames or reinventing TCP.
There is a trick for achieving low latency video with TCP: set SO_SNDBUF to the lowest possible value and do your own buffering. If your buffer grows too large, lower the bitrate and/or drop frames.
I'd argue that on a lossy network you will never have decent video (with decent latency) so for interactive sessions you don't really have to care about lossy network, it isn't worth the hassle anyway.
That is true, the network needs to be stable in any case, but the keyframes have "bursty" traffic, therefore increasing the latency compared to b-frames (and p-frames)
Ever since h264 there is a feature called intra-refresh, which allows you spread key frames over time, reducing the burstiness of the transmission (and therefore improving latency in most scenarios)
There is a trick for achieving low latency video with TCP: set SO_SNDBUF to the lowest possible value and do your own buffering. If your buffer grows too large, lower the bitrate and/or drop frames.