> Concretely, even if you don't know what the answer is going to be at the end of your research, you must think about, or have an idea about, the format of what that amazing answer is going to be. Write the outline of your thesis and "ghost out" what the major charts will be. Write the intro sentences of each chapter -- what are they? (and I don't just mean the boring review of the field part, but your findings part)
> You should know what major type of finding, plot, or table your research is going to output. What are the columns and rows of that table, or axes of that plot? How many data points are required? How many of them can already be guessed? Where is the surprise going to be? What is the conclusion going to be?
You are absolutely right and it's depressing.
The only practical, predictable and steady way to a PhD is to come up with the storyline in advance, do the experiments, bend the interpretation and analysis to support the story you already wanted in advance, then pretend the finding is surprising and novel (when actually you knew you could safely make it work in advance).
There is a tension here, on the one hand PhDs have to pretend play that they discovered major novel stuff and innovated a lot (which is only possible through immense risk), on the other hand in reality they must finish in a given number of years and must be very risk averse.
Only a major cultural shift towards handing out PhD titles for meticulous, systematic, risky but unfruitful research may help. Forcing people to find flashy new stuff will lead to tons of overinflated findings that are more about storytelling than actual surprising novel things that bring a field forwards. It's all a huge pretend play.
----
Ideally in an alternate universe of different incentive structures that support real science, a person would work in research, use existing approaches in different scenarios, push and prod them in different use cases then perhaps once every year you'd stumble upon some interesting effect you don't understand. Then you'd dive in and prod this specific effect with well designed experiments. Once it's clearer what is going on, you come up with the best way to convey this newly earned knowledge. It is at this point (once you actually know what you want to alert the research community to) that you decide how to best present the knowledge, what tables, sections and paragraphs will succinctly and ergonomically convey the information.
If you knew all the story and layout beforehand and you know you will be able to just plug in the numbers and make the paper stand, then you are not doing real science, you are playing the academic game, a cargo cult of science, a game stealing and consuming the prestige of science.
Yes, as I said this only works in the alternate universe. Here in this universe we must churn out papers and must overinflate our findings. Of course, we need to eat, we need jobs, we need titles etc.
It has driven so many PhD students I know to bitterness and cynicism. Several have given up and went to industry. Others, after the disillusionment, have cynically pushed through with whatever minimal things they knew would be required (plus the right story and presentation) then bailed to consulting. You also need to bullshit in consulting but are at least compensated for it well enough.
In the rarest cases do people come out of it as a positive passionate person. Even the successful ones become a hedging, scheming, lukewarm, slimy academic personality. Everything encourages this. The way you get professor positions, the way you get grant money etc. It optimizes and chisels people into becoming the antithesis of curious, open scientific inquiry.
>In the rarest cases do people come out of it as a positive passionate person. Even the successful ones become a hedging, scheming, lukewarm, slimy academic personality. Everything encourages this. The way you get professor positions, the way you get grant money etc. It optimizes and chisels people into becoming the antithesis of curious, open scientific inquiry.
Got a point there, all kinds of academic pressure other than the push for scientific progress.
There should be more than one way to outperform what you could do in such an environment, or in the industrial counterparts following that pattern.
>Several have given up and went to industry.
I think worked for these bozos before, not fun but milestones are milestones.
>Ideally in an alternate universe of different incentive structures that support real science, a person would work in research, use existing approaches in different scenarios, push and prod them in different use cases then perhaps once every year you'd stumble upon some interesting effect you don't understand. Then you'd dive in and prod this specific effect with well designed experiments. Once it's clearer what is going on, you come up with the best way to convey this newly earned knowledge. It is at this point (once you actually know what you want to alert the research community to) that you decide how to best present the knowledge, what tables, sections and paragraphs will succinctly and ergonomically convey the information.
Well, I think one of the shifts that happened is that research changed from "I'm charting my own course in an undiscovered field and accept all liability" to "a university factory promises me a degree and experience".
But here there is an incomplete jump -- in some programs it looks like a well-charted degree program, but has all the liability of research that may not work out. That's why I think (as pointed out above) some practical engineering fields do much better than purely theoretical fields in this regard.
It has also become much more expensive to give it a try and be wrong.
People who don't understand these factors and go in with the wrong expectations are highly likely to fall through the cracks, and not recognize when their program is going sideways or their advisor/topic relationship is not working out.
The illusion that we are supposed to uphold is that brilliance and serendipity can be planned for, can be scheduled and made into an academic program. Actually research and scientific discovery should never be the job of someone. It should be a side product of general activity in a field, something that is already useful even without the discovery (if you're poor). Or just lesure hobby activity that is fine if nothing comes out of it (if you're rich).
It's also part of the whole inflation and treadmill thing going on in education and titles. Generations ago, going to high school was something significant. For the next generation, high school was the default and college became a special thing. Now the majority (in the US) goes to college. Within the family it feels like satisfying progress. Parents look on their kids and feel proud that they are making it further than they themselves did. But now a Master's degree can be a bit too generic. Having a PhD can be nice in industry nowadays (though of course not required).
So what used to be a special thing for a limited circle of select few with the background and fallback that allowed failure, is now becoming a factory process. PhDs are produced by the thousands and thousands, and we pretend they all pushed science forward. There is a flood of papers accompanying this. Tiny steps forward or mixing and matching existing things, but selling a big story around it, flag planting, trying to claim a huge area when only a tiny aspect of it is actually demonstrated, drawing out huge conclusions etc.
There are many reasons for this. The general societal inflation is just one thing. The other is the overall quantification, standardization and uniformization in our zeitgeist. There must be a standard process for everything. The politicians and bureaucrats demand to have a standard process for innovation. You must know in advance exactly "how many pieces of innovation" will you produce per year, what will be the impact of it etc. Then when you tell them about your plans, all they hear is the number of buzzwords mentioned from the latest fads and tune out otherwise. Ah and they look at your publication list of course. If you churned out many papers in the past, you will probably do that as well in this round of funding, and those papers can all be attached to the funding agency reports, so that will make the agency look good towards the ministry or whatever.
> If you knew all the story and layout beforehand and you know you will be able to just plug in the numbers and make the paper stand, then you are not doing real science, you are playing the academic game, a cargo cult of science, a game stealing and consuming the prestige of science.
I actually agree with your immediate statement here, but that is not at all what I understood the OP to be saying. I read their "You should know what major type of finding..." argument as being one of starting with a concrete and well-formed research question, and carrying out carefully designed experiments, and thought that it was excellent advice.
In my little corner of computer science, I very frequently see people (at all stages in their scientific careers) start working on some new bit of research by a) getting a bunch of data, which they then b) feed into some nifty model du jour, and then c) spend a ton of time overcoming all manner of technical trials and tribulations, then finally d) get a number out the other side. They then e) find themselves totally stuck when it comes to actually interpreting their result, because before they ran their "experiment" they hadn't actually bothered to formulate a concrete hypothesis, and so it's not clear what they are supposed to _do_ with their shiny new number, or where to go next.
That's what people often don't get about science, whether it's wet or dry. The mechanical process of actually performing the experiment itself is usually the easy part, relatively speaking. The hard part is thinking carefully about the thing you're trying to study, formulating a theory, coming up with testable hypotheses, and designing experiments to perform those tests.
Part of that last phase involves planning ahead very carefully to what you're going to measure, what your control and intervention criteria will be, what specific statistical analysis you'll perform on the resulting data, and what your various interpretations will be. The more concrete and explicit you can make this, the better: "We're going to measure 'X' under conditions 'A' and 'B', because we think that 'X' will be a valid/useful measure of $PHENOMENON_WE_CARE_ABOUT, for reasons ____, ____, and ____. If X_A ends up being bigger than X_B, our interpretation will be ______, and if X_B is bigger than X_A, we will instead conclude ______'; if they are the same, that will suggest ______." [1]
Obviously you don't yet _know_ which of those conclusions you'll be drawing (if you did, it wouldn't be an experiment), but it is absolutely essential that you've gamed out the various possibilities to at least this level of detail _before_ you do the experiment. This is doubly true for exploratory analyses where you don't really have an intuition about what the outcome will be, as it helps keep the analysis from turning into an endless fishing expedition ("Well, what if I normalize this variable _this_ way? OK, what about _that_ way? ...").
In my experience, one of the best techniques for doing this is, yes, to actually write out blank versions of the tables that you think you'll need to tell the story of your experiment ahead of time, and to make dummy sketches of the various figures you'll need to help interpret the data. Not only will this help you clarify your thinking about what you are hoping to learn from doing the experiment, it has the added benefit of making sure that whatever code you write actually logs/outputs all of the needed data elements! More than once I've had to re-do an experiment because there was an important piece of data that I hadn't realized I would need until it was time to do the analysis. With just a bit more prior preparation, that poor performance would have been prevented.
To return to the OP's argument, they weren't saying that you should pre-specify your conclusions (which would be a terrible idea, for the reasons that you clearly spell out in your post). They were saying that you should have a plan about what specific experiments you're going to run and _how_ you're going to describe the motivation and results of those experiments.
And, if I may editorialize for a moment here, having a more structured approach to doing and writing about research can go a long way to helping to reduce the angst that comes with doing a PhD. I do very much think that many CS PhD programs are dropping the ball in terms of teaching experimental design and evaluation- but that is a rant for another time, as my TED talk today is already running long enough. :-D
--------------------------------
1: Very, very, very often, the process of formulating things this way takes several iterations, because usually once one is forced to write it out this explicitly, all sorts of little questions pop up- "Wait, is that actually what it will mean if X_A > X_B? What if means ____ instead? Hmmm... maybe I should be measuring X', instead? Oh, I'll need different data, in that case, because..."
> You should know what major type of finding, plot, or table your research is going to output. What are the columns and rows of that table, or axes of that plot? How many data points are required? How many of them can already be guessed? Where is the surprise going to be? What is the conclusion going to be?
You are absolutely right and it's depressing.
The only practical, predictable and steady way to a PhD is to come up with the storyline in advance, do the experiments, bend the interpretation and analysis to support the story you already wanted in advance, then pretend the finding is surprising and novel (when actually you knew you could safely make it work in advance).
There is a tension here, on the one hand PhDs have to pretend play that they discovered major novel stuff and innovated a lot (which is only possible through immense risk), on the other hand in reality they must finish in a given number of years and must be very risk averse.
Only a major cultural shift towards handing out PhD titles for meticulous, systematic, risky but unfruitful research may help. Forcing people to find flashy new stuff will lead to tons of overinflated findings that are more about storytelling than actual surprising novel things that bring a field forwards. It's all a huge pretend play.
----
Ideally in an alternate universe of different incentive structures that support real science, a person would work in research, use existing approaches in different scenarios, push and prod them in different use cases then perhaps once every year you'd stumble upon some interesting effect you don't understand. Then you'd dive in and prod this specific effect with well designed experiments. Once it's clearer what is going on, you come up with the best way to convey this newly earned knowledge. It is at this point (once you actually know what you want to alert the research community to) that you decide how to best present the knowledge, what tables, sections and paragraphs will succinctly and ergonomically convey the information.
If you knew all the story and layout beforehand and you know you will be able to just plug in the numbers and make the paper stand, then you are not doing real science, you are playing the academic game, a cargo cult of science, a game stealing and consuming the prestige of science.
Yes, as I said this only works in the alternate universe. Here in this universe we must churn out papers and must overinflate our findings. Of course, we need to eat, we need jobs, we need titles etc.
It has driven so many PhD students I know to bitterness and cynicism. Several have given up and went to industry. Others, after the disillusionment, have cynically pushed through with whatever minimal things they knew would be required (plus the right story and presentation) then bailed to consulting. You also need to bullshit in consulting but are at least compensated for it well enough.
In the rarest cases do people come out of it as a positive passionate person. Even the successful ones become a hedging, scheming, lukewarm, slimy academic personality. Everything encourages this. The way you get professor positions, the way you get grant money etc. It optimizes and chisels people into becoming the antithesis of curious, open scientific inquiry.