For me I have that with VLC and MPV is the stable one (in Archlinux). In my experience, trying different sound output device or module settings sometimes fixes such issues (and usually pulseaudio is the laggy one)
These two do seem connected, at least in my case, maybe the video is waiting for the audio codec. Of course your issue could be something entirely different than what I'm typically encountering!
I have like 3 things trying to be audio device on my linux (of which the graphics card is the most unwanted one and sometimes it randomly manages to take over despite me not changing any settings) and the list of audio modules that show up is simply enormous (pulseaudio, pipewire, alsa, ...), some of those make VLC stutter, others don't. It's a mess but at least pipewire works better than puseaudio in my case...
