Are there no widely used flak/AoE weapons that heavily punish tight formations?

There are plenty. The problem is that it's created a game dynamic where battles turn in an instant. You either dodge the splash and your army melts theirs or you eat the splash and your army evaporates in a flash. The game feels really swingy and punishing when whoever gets the right angle auto-wins.

In contrast to Brood War, where battles tend to unfold as an aggregate of smaller engagements happening simultaneously all around the map, Starcraft 2 hinges a lot on positional play with two giant armies dancing around and probing each other trying to engage at an advantageous position. And then once they get drawn into an engagement it's just BOOM! ZAP! POW! and it's over. If you blink you could miss it.

In Starcraft: Broodwar... Muta-harass isn't about winning, its about forcing your opponent to make more clicks than you.

In the ZvT metagame, Zerg can deploy Mutalisks many minutes before the Terrain have access to Valkyries (AoE anti-air). Terrain's only response at that stage in the game is Marines.

Marines (who shoot one-vs-one) need to group up tightly to have a chance vs Mutalisks. However, a Marine only has 40 HP, and can be KO'd by a group of 11 Mutalisks taking one shot simultaneously.

This forces the enemy Terrain player to group up their Marines before approaching Mutalisks: the Mutalisks are flying however, so they just run away when they see the opponent grouping up. By forcing the opponent to group up, respond, and deploy, you win the "APM / Clicks" war. You're just trying to force the opponent to "waste their clicks" and eat up their mental capacity.

If the opponent fails to respond, you just run into their economy and destroy their workers. If you break the opponent's economy, you pretty much instantly win. Because you have flying units, you always have access to the opponent's backline / economy. (IE: most Terrain players "wall off" their economy so that ground units must destroy a 500HP defensive structure before reaching the backline. But you can get around those walls by using flying units, like Mutalisks)

Otherwise, you take advantage of map features: high-ground makes your creatures invisible (even air creatures are invisible on the high ground in Starcraft: Brood War). So you swing in from the cliffs / retreat into the cliffs repeatedly.

A missile turret has 200 HP, and is a bit harder to deal with. However, a Muta-stack with of 11 can KO the missile turret before it even takes a 2nd shot. It takes much practice: you need to learn the timing of the Mutalisk, carefully watch their animation and positioning (they only take a "instant shot" if they're flying in the same direction that the shot will take place), and Mutalisks can instantly turn-around (you need to move-click 180-degrees offset from the direction of their current movement).

With accurate timing, you can move-click, attack the missile turret, move-click away (causing an instant turnaround, leaving the missile turret range, preventing the 2nd shot), then move-click attack the missile turret a 2nd time and KO the defensive structure. This is the "muscle memory" practice that made Starcraft: Brood War famous. Only those who dedicate many hours of practice to memorizing this strict timing will even reach the barebone basics of the game.

-------

Eventually, Terrain can go Valkyries and with a large enough group, can AoE kill the entire Mutalisk stack. However, that's later in the game. By that point, the Zerg player has moved onto their main strategy (probably mass ground-forces, like Hydralisks). Valkyries are useless against that, and its well known that Mutalisks are otherwise an inefficient unit in combat. So its unlikely for the Zerg player to heavily invest into Mutalisks.