> Will SpaCy's fate be different, 10 years down the road?
Yes --- because I consolidate my algorithms and delete dead code. I've probably written five or six times as much code as currently lives in spaCy.
I hope by then spaCy will be smaller, not bigger, as we reach a more concise understanding of how to actually solve the problem. For instance, it's reasonable to expect the boundary between the POS tagger, parser and entity recogniser to disappear, in the same way that spaCy doesn't feature a separate chunker or sentence boundary detector. I read these annotations off the parse tree.
A library that is constantly changing and removing "dead" code is a library nobody can rely on for production applications.
At first the complaint about NLTK was that it was too academic and not appropriate for real-world code, but no real-world code is going to rely on an unreliable library that keeps changing how it works.
You can maintain the API while overhauling the models underneath. spaCy so far has had almost no API breakages.
For instance, you get sentences as follows:
doc = nlp(u'Hello world. This is a document.')
for sent in doc.sents:
for word in sent:
...
It doesn't matter to users whether behind the scenes, the sentence boundaries are being calculated from character heuristics, or from the syntactic parse. It was the former, now it's the latter. Similarly, part-of-speech tags are currently predicted in their own processing step. In future they may be predicted jointly with the parsing. The API won't change.
Other libraries ask users to choose between a variety of different statistical models, e.g. they ask you to specify that you want the "neural network dependency parser", or the "probabilistic context-free grammar parser", or whatever. By doing this they tie the API to those models.
spaCy just picks the best one and gives it to you. The benefit is that you don't need to be informed when a new model is implemented, even if the change is quite drastic. The modelling is a transient implementation detail, not exposed in the API.
Yes --- because I consolidate my algorithms and delete dead code. I've probably written five or six times as much code as currently lives in spaCy.
I hope by then spaCy will be smaller, not bigger, as we reach a more concise understanding of how to actually solve the problem. For instance, it's reasonable to expect the boundary between the POS tagger, parser and entity recogniser to disappear, in the same way that spaCy doesn't feature a separate chunker or sentence boundary detector. I read these annotations off the parse tree.