I hit upon a similar idea back in about 2000... the key at the time was video encoding hardware that was designed to incorporate MPEG compression. Pulling motion vectors from the encoded stream could have given realtime optical flow way back then. Unfortunately I was nowhere good enough as a programmer to take advantage of it for real projects. A team at MIT had a paper about this that I was able to find.
Had a page about it on robots.net, but it seems to be down now
In our case it's a hardware h264 encoder in an embedded product, so our method wouldn't be any of any use for you in a more general case. We sussed out an undocumented debug api to get the chip to dma the motion vectors to a debug buffer, and then we can extract and use them. I think those apis are originally intended to be used if a customer is having issues with the product so that the vendor engineers can help troubleshoot.
We were originally looking at analyzing the h264 bitstream in software and getting the vectors out that way. Honestly I don't think that would be extraordinarily difficult, there's already code in ffmpeg to draw the motion vectors as arrows on the decoded frame, so that would be a good place to look if there aren't any preexisting apis or tools to extract them.
Had a page about it on robots.net, but it seems to be down now