> You are misinterpreting him; he's not saying that there are no more replications needed, he's saying that this method is not necessary.
I am not misinterpreting him at all. Look at the disjunctive argument he offers; both forks imply a lack of appreciation for sampling error and heterogeneity. (If the experiment is replicated, then far from being useless, the specifics will add substantial precision in the effect size and/or existence and also help estimating heterogeneity across labs attempting the procedure; and if not, then it 'may' be due to incompetence, but may also not be, in which case one has learned even more...)
> if they do get it going, fine, and if they don't replicate, then it's not evidence of anything.
I disagree, it is evidence. If the lab has an 80% chance of replicating correctly yadda yadda bayes theorem yadda yadda. Unless the contract lab is so incompetent that all things it does are purely random, their success or failure is evidence.
> Isn't it better to use that same time and money to build a new experiment that has a higher upside rather than one that has a very limited upside?
No. I don't know why you would think all experiments are of equal value and that replication is worthless. The more important an experiment is, the more important it is that it be replicated. The less important an experiment, the less important that anyone spend the resources replicating it. It can very easily be much better to replicate an experiment to gain additional confidence and/or precision. (See Bayesian search & optimization, decision theory, value of information, etc.) Imagine a medical clinical trial for a cancer treatment, would you really argue 'well, the first experiment turned in p<0.05, let's roll this sucker out to all the patients in the world! We could run more trials to make sure it works in more than one place and is worth the money and side-effects, but after all, isn't it better to use that same time and money to study new drugs, which have higher upsides rather than one with a very limited upside?'
I am not misinterpreting him at all. Look at the disjunctive argument he offers; both forks imply a lack of appreciation for sampling error and heterogeneity. (If the experiment is replicated, then far from being useless, the specifics will add substantial precision in the effect size and/or existence and also help estimating heterogeneity across labs attempting the procedure; and if not, then it 'may' be due to incompetence, but may also not be, in which case one has learned even more...)
> if they do get it going, fine, and if they don't replicate, then it's not evidence of anything.
I disagree, it is evidence. If the lab has an 80% chance of replicating correctly yadda yadda bayes theorem yadda yadda. Unless the contract lab is so incompetent that all things it does are purely random, their success or failure is evidence.
> Isn't it better to use that same time and money to build a new experiment that has a higher upside rather than one that has a very limited upside?
No. I don't know why you would think all experiments are of equal value and that replication is worthless. The more important an experiment is, the more important it is that it be replicated. The less important an experiment, the less important that anyone spend the resources replicating it. It can very easily be much better to replicate an experiment to gain additional confidence and/or precision. (See Bayesian search & optimization, decision theory, value of information, etc.) Imagine a medical clinical trial for a cancer treatment, would you really argue 'well, the first experiment turned in p<0.05, let's roll this sucker out to all the patients in the world! We could run more trials to make sure it works in more than one place and is worth the money and side-effects, but after all, isn't it better to use that same time and money to study new drugs, which have higher upsides rather than one with a very limited upside?'