There are a lot of good ideas and sentiments in this article. And then at the end I find out the trail-blazing product he left academia for is "a newsfeed based on your interests."
How many startups like this are there now? 300? That is probably a low estimate. I'm sure that everyone living in the Valley personally knows at least one founder working on the exact same product. And none of them are better than my Hacker News/ Facebook/ Reddit/ RSS feed combo.
I've heard some of the many Prismatic competitors describe themselves as "a Pandora for news" which is an apt description, since rdio, soundcloud, and Spotify are better than Pandora. Self, social and community sourcing for digital entertainment tends to be better than algorithms. While I believe the "problem" domain is severely overworked, if you're going to stick with it then my bet is that social algorithms like collaborative filtering a la Netflix does a better job than topic modeling.
On the optimistic side, a good company is more than a single product. Maybe the author's considerable expertise and experience from building Prismatic will lead to something cool down the road.
As a prismatic user, I think the product he's built is pretty far ahead of any competitors I've seen.
I don't think the fact that others have tried with limited success is an indictment of prismatic at all. If anything, that strengthens that argument that this project required a lot of NLP skill.
When I think about what I want the future to look like and what's missing now, a smarter news aggregator isn't on the top 10 list. I have no problem wasting infinite amounts of time with existing entertainment technologies. At the same time, thousands of engineers clearly disagree with me as they are risking their livelihoods and investing their their talents in that field.
I do find the whole field boring. Making a news aggregator that's X% better than existing ones doesn't improve the world a whole lot, or even offer a compelling consumer value proposition.
I don't want to sound like I'm against all consumer entertainment tech. It's clearly something that people enjoy, and it's exciting for some people to make. But this particular application is one whose value I am very skeptical of.
Completely agree that a news aggregator isn't one of the world's 10 most pressing needs. It isn't in the top 100 either.
But Prismatic has told me about articles that make me better at what I do. Depending how widespread that experience is, they may be having a larger impact on our 10 most pressing needs than most teams that attack those needs directly.
Thank you for the kind words! Indeed, the NLP and ML behind Prismatic are pretty intricate. Not just because they use something mathematically sophisticated, but you have to make a lot of good decisions about what can be tackled with a simple approach and when to spend a month thinking about a harder problem.
My feelings exactly. It's made worse by the fact that, out of all the academic departments out there, AI and NLP research has perhaps the greatest potential to transform society in so many ways.
He leaves the field trying to create intelligent machines in order to write a Google News clone? I truly don't get it.
To me, it feels like this person's talent is wasted on this project. With such a strong background in natural language processing, why isn't this person involved in a Siri competitor, in Web real time chat, VOIP compression, or AI speech? Is this the limited scope at which a typical CS Ph.D operates?
Reach for the stars. Let people of lesser ability make news apps, even if it's a struggle for them.
A really good news aggregation/management site would help me out a lot more than a marginally better Siri. I waste a lot of time on the news and miss a lot of stuff.
Agree with pseut. I would need a Siri smart enough to aggregate and manage news for me. We need someone to build that great aggregator, and then we'll talk about Siri integrating that.
Sure, but your original point seemed to be that any small degree of progress on a Siri variant would be more important than any large degree of progress on a news aggregator. My only point is that the magnitude of the progress matters, and a big change in the "smaller" problem of news aggregation would free up a lot more of my time than a small change in the "bigger" problem. As much as I'd like to be able to dictate research papers and code up a bunch of data analysis over my phone via Siri++ while I'm on a long run on the beach, at some point you have to trust that people choose to work where they think they can make the most progress.
I appreciate your passion but before you build a tower to space, you should probably make sure you're not building on shifting sands first. It's a telling arrogance you display when you say "let those other people struggle for their small gains, I have bigger fish." As if they would never be a customer down the line or that maybe you'll "get out ahead of them" and own the space... I'm not sure what your motivation is for this perspective, anyway.
The author already addresses your view though since he shared it. We already have been using PHDs like this for some time now. I think you just don't understand that communication hubs like reddit and HN are not "a waste of talent" especially when you consider portals like these get sold to Conde Naste for millions x 10^x of dollars.
Contrary, I think "reaching for the stars" here is taking ownership of the application instead and understanding the real burden of operations and the impact to your clients and what the real net benefit is (if any) in your application, not the number of 0s on your check. It's not sexy, but nothing complicated is. You're not going to do anything groundbreaking implementing what you were taught in school in the tiny scope of your contract, you're just creating tiny bubbles of technology that may or may not ever offer a benefit to anyone outside that space; contracting yourself out to another Fortune 500 to design a protocol or format to be used in some rented-out walled garden. It might be a great model for retiring, but not so much for a society, it it certainly isn't "reaching for the stars" unless you want to be alone when you get out there.
We can waste an incredible amount of time making sure the foundation is secure. Or we can plan for the shifting sands, like Google building commodity-hardware-failure into their business plans. Larry and Sergey didn't try to make computing hardware more reliable; they planned for it to fail.
To the contrary, many complex things are sexy. The iPhone is greatly complex, both in software and hardware, but it's still sexy. By the way, what if Steve Jobs and Steve Wozniak were happy just running a small electronics store in Palo Alto?
Interesting to see your opinion of "reaching for the stars". Also interesting to see you write 2x as much about what "isn't reaching for the stars".
Ok, another perspective: why would a PHD waste his time on trivial conveniences alike web chat and voice recognition just because they are hard to solve, when there are unmet needs in a much larger market called media?
Or how about this: the key to solving problems correctly is to solve them as close to the root cause as possible. Why write a speech recognition when you can get the same functionality much more effectively by eliminating the microphone altogether and interfacing directly with the user (just a hypothetical scenario)? Why write it for iOS when the only reason the person is carrying a phone is so they can be connected on the go, and the only reason they are on the go is to learn something they could have read on Prismatic this morning? You'll never tackle any of these problems if you think that the lower layers are already solved problems cemented in concrete, only worthy to be maintained by tech monkeys.
Why do we make software and hardware? Why do people farm? Why do we do anything?
It's about advancing our society. Making more efficient use of our human resources to enable us to discover new ways of making more efficient use of our human resources. To raise quality of life. To allow more humans to live, which increases our pool of labor and of brainpower, both of which we can use to multiply the other further.
Why do we do this? At this point we're descending into existential ennui. When you look into the abyss, remember the staring game -- make it blink.
I'm all for exploration and self discovery or whatnot, but we're talking about here is effective use of resources. My perspective of your opinion is you feel that PhDs should keep fitting into the spots where they are requested by big corporations, to develop their products that require high technology and expertise to be done quickly and cheaply, because everything else is just a solved problem that an expert would be wasting his time on.
My counter is that these things that we take for granted are not "solved problems" and innovation in this space is the most fundamental and disrupting innovation one can engage in, and increasingly the press would turn the public's eye away from this fact since obviously those who control the aggregation of news can control what a large number of people think about different topics. But perhaps you already knew that and find change at this level and its consequences too scary to contemplate.
Prismatic has a much larger vision of bringing together elements from my AI and machine learning research background along with modern interaction design to build smart everyday consumer products.
The current app is just the first step in a longer roadmap of building smart products. Currently, we're great at discovering new articles, but in the future will include discovering relevant apps, music, movies, and local events. Don’t be too surprised if we’re on your TV soon.
Keep in mind that if it happened that you had an AI that could quickly scope people's desires out, the application you would want to be selling would be:
"The 'it gives you what you want' thing"
Rather than any kind of fancy description. It wouldn't be the description that sold customers, it would the fact that it really, actually gave them what they want effortlessly that would sell them. And course, news is gives you a lot information to easily parse. So from the "I have a program that's unique in its language comprehension" perspective, this makes perfect sense. If the product was more specialized, it quite possibly would not have as much scope to demonstrate that it could choose for you.
Of course, whether AI could possibly work for this is another question but I can understand why he'd want to try this route.
To be fair, PG is the one advocating that engineers satisfy their immediate, and often extraordinarily boring, needs.
There are 300 startups that are funded doing newsfeeds, even while those startups might be a tiny minority of exciting ideas.
On the other hand, the role of business is not to realize and finance science fiction. Become a writer (or perhaps a Researcher) rather than a programmer if you genuinely have a great imagination.
How many startups like this are there now? 300? That is probably a low estimate. I'm sure that everyone living in the Valley personally knows at least one founder working on the exact same product. And none of them are better than my Hacker News/ Facebook/ Reddit/ RSS feed combo.
I've heard some of the many Prismatic competitors describe themselves as "a Pandora for news" which is an apt description, since rdio, soundcloud, and Spotify are better than Pandora. Self, social and community sourcing for digital entertainment tends to be better than algorithms. While I believe the "problem" domain is severely overworked, if you're going to stick with it then my bet is that social algorithms like collaborative filtering a la Netflix does a better job than topic modeling.
On the optimistic side, a good company is more than a single product. Maybe the author's considerable expertise and experience from building Prismatic will lead to something cool down the road.