But what I'm thinking of is the tokenization of a large corpus of text into bigrams and trigrams for example.
Hmm, that reminds me, I have an interesting toy problem which is sequential I/O limited...
Note that there's a code contest for Common Crawl that recently was announced: http://commoncrawl.org/first-ever-code-contest/
But what I'm thinking of is the tokenization of a large corpus of text into bigrams and trigrams for example.