It looks really neat. Isn't 660 GB not that much really? I grant that the slides say they've used an optimized binary for the storage, but how does this compare to pandas?
660GB was just a small benchmark. The real thing uses more than a petabyte of raw data.
Pandas uses NumPy internally. You could use Deliroll as a replacement for NumPy in Pandas to get a nice interactive environment for amounts of data that can't be easily handled with plain NumPy.
This is interesting. My current project is a fraud detection system. We currently leverage Cascading/Hadoop. But I wanted to make sure the system is not Hadoop-centric. So I made a point of having the system be language agnostic. It looks like there might be a fit for this tool.
I passed the slides along to my team to see what they think. If they just impress upon the team that we need to store something other than just 0x0A delimited text files, I'll consider it a win.
