This post is spot on. It's encouraging to see one's experience reflected back from someone else. I agree with the author that the psychology of this is critical and that it doesn't get discussed much.
One point I'd like to add is that history gives a very misleading picture. History is divided into people who worked through hard problems and triumphed (these are called "geniuses") and people who either gave up or died trying (these are called "failures"). Those seem like opposites, but it's only hindsight that makes them so. Psychologically, when you're in the middle of it, which is when it really matters, there's no way to know. All you know for sure is that you've been walking down a dark tunnel for a really long time.
An interesting question, more interesting than it appears, is: why would someone do that? That is, at some point, having walked down the tunnel for quite some time, the rational thing would appear to be to turn back. Why would someone not turn back?
A lot of people like to repeat the idea that "failure is a learning experience" but it is only recently that the essence of this idea in practical terms became concrete for me (thanks to "177 Mental Toughness Secrets of the World Class"): you need to be prepared to fail, which - among other things - means you need to be aware of resisting failure.
Resisting failure can manifest itself very subtly, but the most common manifestation for me is resisting something because it involves new learning ("I just want to solve the core problem, this is never going to end!"), or doing something that I "know" I won't be as good as others at e.g. the user interface.
It's obviously better to get something working than not, and knowing this keeps me committed to working in principle. So I don't abandon the project outright. What actually happens is more insidious: I do something else to avoid the pain of dealing with the problem at hand. As time passes, I (a) lose momentum on the project and (b) the likelihood of a bright, new shiny idea occurring to me, with challenges I "know" I can solve, increases.
When (b) happens, I move on, and it feels good because I'm making progress whereas with the other project I "couldn't" make progress. I never specifically or intentionally abandon the original project, but in effect I do, with the rationalisation that I have a higher priority item to focus on i.e. something I believe I can 'make progress' on. At least, as it turns out, according to my dysfunctional definition of progress, which is rooted in perfectionism.
What I've learned is to monitor my objective progress: did I take steps to solve the problem, and can I solve it for the bulk of the use cases that are really important, and if not can I find somebody else to solve it? e.g. do I have a user interface yet or not, and do I need to find somebody to do it for me? Not my subjective progress: e.g. "I really want this to look great and professional like all those other sites out there, and I'll never get there because I don't know how and I don't want to blow money on this because I'm not even sure how I want it to look, maybe I should research more and find something similar, this is boring it's not what I want to do, I'm making no progress, etc."
Attempting to learn music late (~20)taught me this.
Start with a strong theory that you pursue until you know it's 99% wrong, sob ,get a new hunch (if you're really masoc.. passionated), repeat.
After many cycles likes these two things happens :
- the tunnel ain't that dark anymore, you can see shades, sometimes even shiny spots*. You get a carnal feeling that those little details you dismissed earlier have a lot more value.
- you went from 100% blind to 99.9% blind.
Tackling something vast and unknown by yourself should be discussed more often, I feel it's a large human nature myth in `modern` cultures. We rely too much on society to accelerate problem solving and get stuck into a little spot of dependencies.
ps: oh and by the way, fear/anger triggered by failure was adressed by stoicists somehow. ( at least according to wikipedia ~_~ )
No, that's a reason to turn back. We're talking about the opposite - why someone would not only try, but keep going even after countless data points indicate failure.
"That is, at some point, having walked down the tunnel for quite some time, the rational thing would appear to be to turn back. Why would someone not turn back?"
I would phrase category A differently: when the work is valuable for its own sake. That's not necessarily fun. It's more likely grueling. Even when it's fun, the fun parts don't add up to nearly as much fun as the painful parts do pain. Yet one can continue because one senses (I almost wrote "knows" - can one know?) that the work is valuable, necessary, important. That's deeper than fun.
What I find interesting is that this comes from within. The other two motivations you list are external. I wonder if the external ones burn out sooner.
When your motivation is intrinsic (that is, you have a felt sense of the work and you know what it is and you think it's important) you have a basis for deciding when to listen or not listen to other people's opinions. But when it's extrinsic, you're more likely to do what other people say and perhaps give up if it doesn't work.
Edit: another thing about intrinsic motivation is that because other people don't have the same sense that you do, what you're doing may seem absurd and they will tell you you're doing it wrong. So the challenge of this path is not just the work is hard and that you fail a lot; it's also that it will probably force you beyond the pale of social proof, something humans are not hard-wired to enjoy.
One point I'd like to add is that history gives a very misleading picture. History is divided into people who worked through hard problems and triumphed (these are called "geniuses") and people who either gave up or died trying (these are called "failures"). Those seem like opposites, but it's only hindsight that makes them so. Psychologically, when you're in the middle of it, which is when it really matters, there's no way to know. All you know for sure is that you've been walking down a dark tunnel for a really long time.
An interesting question, more interesting than it appears, is: why would someone do that? That is, at some point, having walked down the tunnel for quite some time, the rational thing would appear to be to turn back. Why would someone not turn back?