Sorry, but how is this not just anti-intellectualism?
> 30 emails, wow!
It can take up to half a day to reply to an email containing even one fairly technical question. Obviously the time invested depends on the content of those 30 emails, but it's not at all difficult to imagine this eating a week worth of a PI's time.
> 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even?
The typical attitude is that this is a justification for not making them spend their time managing these sorts of tasks. You don't get to say "we get to pay you crap because your job is exciting and rewarding" and then load that same person down with a bunch of grunt work (on top of all the normal grunt work). That's how you bleed students and kill the lab's productivity.
Also, grad students are not well paid but they are typically no cheaper than post docs; PIs pay tuition.
> The demands on their time seem so low, yet the complaints are so high
So let's imagine a world where people calling for massive improvements to reproducability get their way. Let's say it is a month worth of time for the lab. For each paper. Multiple papers a year. That's a pretty massive time investment. If you're a top lab, that could become a full-time position. And believe me, you're not going to be able to fill that position with a grad student. That person will have to be well-paid, because their job is going to suck.
So it's reasonable that scientists are peeved when they invest all this time and don't perceive their collaborators as acting in good faith, or feel like their collaborators are trying to cut corners to pinch pennies.
If an engineer builds the worlds greatest new engine but says "unfortunately it'll only run in my lab, no one else is competent enough to run it or build a copy" then what good is it to society?
If the researchers in a lab are such geniuses that they are doing experiments almost nobody else can duplicate and it is therefore impossible to determine the veracity of their claims, how is that helping society and why should society fund them?
Isn't the onus on the researchers to focus experiments that are also reproducible by non-supergeniuses?
Science is not engineering. In engineering, we understand the problem domain enough to make everything reproducible, the governing principles are well enough understood that reasoning from first principles can usually come up with answers.
In contrast, in science, we are trying to discover those first principles. So if something doesn't replicate in another lab, we have a responsibility to check it out, but that doesn't mean that the scientist has to discover all the first principles before making a publication! The non-replication could be due to new, solid, first principles, and the assumption that it should be "reproducible" exactly from the English words on a sheet of paper are a faulty assumption.
If you want to be able to pick up the scientific literature and read it like a science text book, you're doing it wrong. That is not the purpose of publishing papers.
> If an engineer builds the worlds greatest new engine
There is more to science than advanced product development. Conflating the two is wrong-headed.
> but says "unfortunately it'll only run in my lab, no one else is competent enough to run it or build a copy" then what good is it to society?
It's fantastically good to society. A company interested in monetizing the research could provide that researcher with a multi-year sabbatical to come to their company and turn is Research into a Product.
Incidentally, that happens. It's also not unheard of for phd students to carry an adviser's idea forward toward application in the context of a permanent position at a relevant company.
> almost nobody else can duplicate
There's a difference between not being able to duplicate, and duplication being expensive.
> and it is therefore impossible to determine the veracity of their claims
Again, reproducibility should focus on the veracity of the claims, not the economics of reproducing them. Nothing is wrong with calling for better reproducibility. The problem is in expecting to get it for free, and assuming that it's always appropriate at every stage of research.
Investment in reproducibility should be in proportion to the degree of trust the scientific community puts in the claim, and it is absolutely reasonable for that investment to grow over time. But, don't kid yourself, it's an investment. And society would have to be stupid to invest enormous amounts of money into ensuring every single scientific paper ever published is held to an extremely strong standard for reproducibility.
> Isn't the onus on the researchers to focus experiments that are also reproducible by non-supergeniuses?
If by "super genius" you mean "someone else who does research in the same or a closely related field", then Hell. No. The onus on the scientists is to focus on experiments that push science forward in service of mankind.
Sometimes this means helping mega-corps figure out how to reliably reproduce your research without expert help and thereby increase profit by decreasing required investment. Sometimes this means focusing on discovery.
"A paper that Young, a biologist at the Massachusetts Institute of Technology in Cambridge, had published inCell in 2012 on how a protein called c-Myc spurs tumor growth was among 50 high-impact papers chosen for scrutiny by the Reproducibility Project: Cancer Biology."
That hardly sounds like "every single scientific paper ever published".
You are correct; I was being hyperbolic. But still, not sure Nature/Science/Cell is a high enough standard for "anyone should be able to replicate with little effort" -- lots of "non-late-game" results that aren't necessarily ready for industry applications get published in those venues (which, I guess I've been arguing, is a good thing.)
Well, the economics for the LHC are difficult, precisely for this reason.
Yes, it is possible the money spent on LHC could have been spent better on other "smaller scale" projects with a better cost/benefit ratio, and better aligned incentives (possibly even outside of the field of high energy physics)
I would have to study the economics of the LHC better to have a strong opinion on this though.
Alternatively, any regulatory approval should be limited to versions of the resulting drug at that particular lab, since only they know how to produce it ... although output may be limited. :-p
When I was an academic, such questions did not take half a day to answer. That's because my papers were detailed and answered the questions - I would simply say "go read section 4.3".
Further, I also put source code on my webpage (this was in the pre-github era). "To replicate and check methodology, `darcs clone ...`".
If replying to technical emails takes too much time, it is because your publications are incomplete. Full stop.
If you can even come close to explaining how to replicate your results in 12 or 20 pages, you're not really doing an experiment that is non-trivial to replicate. Constraining science to protocols that can be explained in 12-20 pages would be stupid and wrong; almost any protocol used in a modern lab will take longer to explain than that.
With all due respect, the assertion that a typical protocol should be able to fit in the length of a typical CS publication is pure hubris. (And when you need a 200 page TR to explain things, my arguments about unjustified cost start resonating because you're demanding a dissertation or two per published paper.)
> such questions did not take half a day to answer.
Just because the science you did was trivial to replicate does not mean all science is trivial to replicate.
Yeah, when replication involves nothing more or less than compiling code on commodity hardware, I can imagine replication protocols are probably pretty easy to explain... it's a small wonder anyone had to email you about this at all.
> "To replicate and check methodology, `darcs clone ...`".
However, the assertion that replicating any experiment is basically no more difficult that downloading some source code and compiling/running it is... arrogant in the extreme.
It's nice that some Computer Scientists can "replicate" their results by saying "oh yeah compile and run this code on the same dataset".
But asserting that all science is like this is underestimating the difficulty of replication even in CS. What happens when your dataset is terabytes in size and requires a specially configured cluster of GPUs or FPGAs to process in the next year? What if you have a half million dollar robot that has to be built to spec? Give me the 12 page IKEA manual for building a half-million dollar robot that has only ever been built once before. Or even the 200 page manual. Not all Computer Science can be done on a laptop, and the assertion that it can is, again, arrogant.
Even more to the point, most laboratory experiments are not run entirely by autonomous robots that you can load code into and walk away; in fact, many require steps that require skills in the lab that can only come with lots of practice. Hell, limiting ourselves to protocols that are easy to use and fit in 12-20 pages would ban the contents of upper-level undergraduate lab assignments.
There is nothing wrong with an experiment requiring skill to replicate. Banning protocols that are difficult for non-experts to follow for intrinsic reasons would basically ban all modern experimental science.
My papers were mostly a lot longer than 12-20 pages (one was 10x this), or at least referred to such a paper where details were explained. I have no objection to complex protocols, I simply assert that the protocol needs to be explained at most once.
> 30 emails, wow!
It can take up to half a day to reply to an email containing even one fairly technical question. Obviously the time invested depends on the content of those 30 emails, but it's not at all difficult to imagine this eating a week worth of a PI's time.
> 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even?
The typical attitude is that this is a justification for not making them spend their time managing these sorts of tasks. You don't get to say "we get to pay you crap because your job is exciting and rewarding" and then load that same person down with a bunch of grunt work (on top of all the normal grunt work). That's how you bleed students and kill the lab's productivity.
Also, grad students are not well paid but they are typically no cheaper than post docs; PIs pay tuition.
> The demands on their time seem so low, yet the complaints are so high
So let's imagine a world where people calling for massive improvements to reproducability get their way. Let's say it is a month worth of time for the lab. For each paper. Multiple papers a year. That's a pretty massive time investment. If you're a top lab, that could become a full-time position. And believe me, you're not going to be able to fill that position with a grad student. That person will have to be well-paid, because their job is going to suck.
So it's reasonable that scientists are peeved when they invest all this time and don't perceive their collaborators as acting in good faith, or feel like their collaborators are trying to cut corners to pinch pennies.