It should be pretty easy: keep track of the ancestry of each object and kill off any family tree that's expanding too quickly if the server gets overloaded.
You'd think so, but when all of those objects are (potentially) things purchased by players, it gets a bit more complex than kill it at the root. Also, I believe that (at least at one point), objects could attach scripts to other objects. So the family tree wouldn't work in all cases.
Internally Second Life calls their countermeasure the "gray goo fence" but I can't find many good descriptions of how it actually works online, except some mentions that the restrictions on how much something can "rez" increase exponentially. This link was the best description of it I could find:
I believe that, since the attack mentioned therein, they've done other things, such as restricting certain scripting functions to trusted people. But folks have been known to find exploits anyhow.
Of course, feel free to correct me if you actually play Second Life. I've only been following news of it from the outside. I've never actually played it.
I've never understood why people are that worried about it with nanomachines, though. We already have gray goo (though much of it, like algae, is green). I mean, most single-celled organisms are capable of that kind of mass replication. And they can be found pretty much everywhere already. I'd be more worried about accidentally engineering a super-virus or whatever than gray goo per se.
