But who could do the spot-checking? In many fields, there is only one team or a few authors capable of understanding, reproducing, or verifying certain research. In a two-party situation, if you pay the opposing team, they have incentives to exaggerate doubts. Often research papers are written in such a confusing way that it is impossible or very expensive to reproduce or verify the results (e.g. repeat a year-long study).
I think it would be better if there were incentives that rewarded quality over quantity. At the moment, my university always says that quality is of the utmost importance, but then threatens to terminate my job if I cannot publish x number of papers in a given year.
>, there is only one team or a few authors capable of understanding, reproducing, or verifying certain research
If the research isn't documented well enough to reproduce/verify, then the paper shouldn't pass review in the first place. The NIH could make it a condition of funding that papers are detailed enough to be reproducible.
My sense is that this would prevent 98% of papers in the social sciences from ever being published. How do you decide whether to renew a researcher's position when the average time to get a single paper through peer review is 5+ years (my average is 2-3 contract renewals per year)? This is not compatible with today's pace.
I am not saying that all this is a desirable situation. It is very unfortunate, and I wish there was an easy solution. My first research paper took 5 major revisions and 6 years to get through peer review. All the reviewers criticized was the wording and my unwillingness to conform to the accepted views in that particular community; I almost lost my job over this, but once the paper was accepted, it won several awards. All of this leads me to believe that peer review is very subjective and prone to error, and I don't have a solution for that.
The goal of science isn't to publish papers, it's to investigate hypotheses. If 98% of social science papers cannot be replicated then that's an indictment of social science, not the requirement to replicate.
I suspect a lot of "hard" science papers would be caught as well so it's a necessary quality control method
Replication is one of the central issues in any empirical science. To confirm results or hypotheses by a repetition procedure is at the basis of any scientific conception. A replication experiment to demonstrate that the same findings can be obtained in any other place by any other researcher is conceived as an operationalization of objectivity. It is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained.
Public research that nobody else can do should be considered utterly worthless. Publication should be predicated on independent replication. Papers are supposed to be instructions for replication! If it's impossible to replicate from the paper, the paper fails its primary purpose IMO.
> Often research papers are written in such a confusing way that it is impossible or very expensive to reproduce or verify the results (e.g. repeat a year-long study)
From an economic perspective, is this a very desirable situation?
I think it would be better if there were incentives that rewarded quality over quantity. At the moment, my university always says that quality is of the utmost importance, but then threatens to terminate my job if I cannot publish x number of papers in a given year.