This is a pretty rosy interpretation of how the science sausage is made.
1. The scientific method might be reproducible by design, but there can be (and often is) a very large gap between "Science" and what gets published as a paper.
2. Peer review is a grab bag. Sometimes it's obvious the reviewers barely read or understood what was written. Other times they provide feedback that is asinine. Or simply they provide commends and you tell the editor you're ignoring those comments and the paper still gets published.
3. Only a fraction of reviewers (depending on field) will check that the paper includes enough information to be reproduced. This is easy to verify, just go ask different academics how many papers they read that don't include enough information to reproduce. And only a fraction of a fraction make any attempt at reproducing the paper to any degree before signing off on it. (and if they say that they couldn't reproduce it, then see the end of point 2 above)
> If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile.
This is a false dichotomy. It misses some very real dynamics, none of which are conspiratorial.
1. It's hard to publish negative results, especially if the paper written by someone influential in the field. Which means that there's a great disincentive to even try.
2. Competing theories are difficult to study not just because of peer review issues but also because of financial and repetitional issues. Getting funding is significantly harder if your work doesn't use the methods that the funding agencies expect. Same idea with reputation, hard to get your career started if you want to work on theories that go against the prevailing theory or if you have work that contradicts it.
This is the whole idea behind the concept of paradigm shifts.
> Quantum Mechanics and General Relativity may not agree with each other but they both have destroyed mountains of attempts to disprove them.
I agree. Unfortunately for all of us the rest of published literature is nowhere near as battle test as these two theories, nor would they survive such testing.
> Not saying you are saying that but some posters are.
I didn’t mean to imply that you were. But the it doesn’t sound like the personally you replied to was on the side of “all science is bad” either.
I am closer to the “much ‘science’ is junk”. There is much good work a being done, but the garbage is very much there, more so in some fields than others.
The null hypothesis at this point for most new papers one reads is certain fields (looking at you nutrition & health) is “this won’t replicate”. It’s certainly also true for much work primarily involving modelling; the null hypothesis as a read should be “if this doesn’t have code, I won’t be able to reproduce/replicate it”.
We should not throw the baby out with the bath water, but we should be frank about the current state of things.
No one (in this thread) has suggested science is bad or we should throw it away, that is your own straw man.
I have suggested we throw away a part that is causing problems. You seem to be rushing to defend scientific failures by saying “it doesn’t matter that a lot of studies don’t replicate, we should follow the status quo regardless.” We can throw out the status quo and keep (the good parts of) science intact.
1. The scientific method might be reproducible by design, but there can be (and often is) a very large gap between "Science" and what gets published as a paper.
2. Peer review is a grab bag. Sometimes it's obvious the reviewers barely read or understood what was written. Other times they provide feedback that is asinine. Or simply they provide commends and you tell the editor you're ignoring those comments and the paper still gets published.
3. Only a fraction of reviewers (depending on field) will check that the paper includes enough information to be reproduced. This is easy to verify, just go ask different academics how many papers they read that don't include enough information to reproduce. And only a fraction of a fraction make any attempt at reproducing the paper to any degree before signing off on it. (and if they say that they couldn't reproduce it, then see the end of point 2 above)
> If the majority of author papers align that means either there is a conspiracy or some fraction of things was reproducibile.
This is a false dichotomy. It misses some very real dynamics, none of which are conspiratorial.
1. It's hard to publish negative results, especially if the paper written by someone influential in the field. Which means that there's a great disincentive to even try.
2. Competing theories are difficult to study not just because of peer review issues but also because of financial and repetitional issues. Getting funding is significantly harder if your work doesn't use the methods that the funding agencies expect. Same idea with reputation, hard to get your career started if you want to work on theories that go against the prevailing theory or if you have work that contradicts it.
This is the whole idea behind the concept of paradigm shifts.