The simple solution here would have been something like a bunch of netapps with snapmirrors to a secondary backup site.

Or ZFS or DRBD or whatever homegrown or equivalent non-proprietart alternative is available these days and you prefer.