Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I’m working on it. Unfortunately ‘high reliability’ also means ‘tons of testing’, and ‘finding all the performance holes before your customers’.

It’s kind of a shame, because Github keeps giving me fantastic ways to toot my own horn, but I can hardly do that if I can’t improve on their numbers.



Architecturally, how are you storing Git data? If it isn't fundamentally different from bare-repo-on-disk with libgit2 or wrapping the git binary, you're not going to scale any better.


Blobs are unique. Whether you have 700 forks or not, data doesn’t increase. I’m unpacking the git stuff, keeping all the blobs on disk as alternates. Then syncing to object storage. This proved really slow so the blobs are still repacked to sync to object storage. It was fun finding out that some of the repos I test with have 500k blobs, I didn’t know either, but half a million individual put calls made it excessively obvious that wasn’t going to work.

Between the git hosts things move around based on available space, IO pressure and whether it’s been recently accessed or not (spinning disks are far cheaper these days). Theoretically there will be a point where the routing becomes a problem, but probably not until there’s several hundred git hosts (I’ll probably have bandwidth problems before that happens).

I’m not sure what the largest git repo is that github has, but I can theoretically keep repos up to 1TB around (plus a sheer unlimited number of forks for the same). I imagine the API’s get problematic before git does at that point (e.g. shuffling data around is much harder than storing it).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: