”One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters....I'd exercise some humility.”
You shouldn’t be cringing. You should be educating people about how science actually works, and how it simply doesn’t matter very much whether any particular paper is reproducible. It’s a straw-man argument, because most papers aren’t worth reproducing. I don’t know many good scientists who take what they read (in any journal) at face value. If you do, you’ve been mislead somewhere during your training (fwiw, I also have a PhD). At best, papers are sources of ideas. Interesting ideas get tested again. Most get ignored. Even if a few hokum theories become popular for a while, eventually they’re revealed for what they are.
The tiny percentage of subjects that rise to the level of public policy discussion end up being so extensively investigated that reproduction of results is essentially guaranteed. And yeah, you hear lots of silly noise from university PR departments, but that stuff is a flash in the pan.
For example, nobody legitimate is in doubt of the broader facts of global climate change or evolution or vaccination, even if 95% of (say) social science reaults turn out to be complete bunk. Yet climate deniers, anti-vaxxers and “intelligent design” trolls absolutely love it when this distinction is ignored, because it allows them to confuse the public on the legitimacy of science as a process.
It's true that science doesn't value every published paper equally, but it's also true that publish or perish is creating ever-growing mountains of worthless papers. This is a real problem, and drags the quality of everything down.
Besides, the fact that there isn't one reputable journal in most fields that remains untarnished by the replication crisis is both a practical problem and a problem of public trust. A lot of this BS science is paid for directly by the public's tax dollars, or else by their student loans. I wouldn't expect the public to be so forgiving if 95% of it is bunk.
The public is oblivious at the moment because science still has credibility . It s going to take a major catastrophic misstep of science (which will inevitably happen the way things are going) for the public to lose trust
“it's also true that publish or perish is creating ever-growing mountains of worthless papers.”
Is this true? Prove your claim.
”This is a real problem, and drags the quality of everything down.“
Let’s assume your first assertion is true. Is it automatically true that your second claim follows? Why?
I see no evidence that the individual productivity of scientists has changed much in the last 30 years, nor do I notice much of a change in the aggregate quality of science. Crappy science existed hundreds of years ago, and it continues to exist today. The main difference, as far as I can tell, is that we have a lot more scientists now.
In any case, these are just assertions, not arguments.
You would think crappy science would go down with progress? I think not.
You should look for papers in psychology/sociology, AI (recommendation engines, accuracy), economics, nutrition and medicine. Marketing papers are also interesting, I guess.
As an anecdote, I dug into sexuality, gender papers recently and was baffled at the amount of shit I came across. I couldn't believe someone published it.
> Crappy science existed hundreds of years ago, and it continues to exist today. The main difference, as far as I can tell, is that we have a lot more scientists now
And ability to influence a lot more people faster than ever before with increasing level of distrust.
Not to mention, papers from US and EU affects other nations perhaps more. Many people blindly piggyback due to not enough funding to replicate or perform our own analysis. Funding being scarce promotes hype and flock people towards whatever media popularizes.
It results in a very weird disconnection on topics that are dependent on population, history, culture, and other location sensitive data. The base is contaminated, anything built on it is not going to suddenly turn into truth.
There have always been weaker scientists, but there hasn't always been the economic incentive to publish in order to maintain a teaching position. This is a relatively recent (few decades) thing and is due to structural factors in academia and society. If you require proof, I'm not really sure what to say to you, as evidence is not hard to find, but if you don't already see it, I'm unlikely to change your mind.
If you want to claim that “publish or perish” (which, btw, has been a part of academic life essentially forever) is somehow recently affecting the volume of papers being produced, you should be able to provide evidence of that in a straightforward manner. One obvious test: is the per-capita rate of publication increasing? (my experience says “no”, but I’m open to contrary evidence.)
You have a hypothesis of what’s going on, but you’ve provided no evidence for that hypothesis, and when challenged to provide some, you tell other people it’s their job to do it for you.
It’s not my job to prove your extraordinary claims.
The number of publications per year per academic seems to me to have increased over the last 50 years. I don't have a citation.
Regardless, my original claim was that the absolute number of papers is growing, and most of them are trash. I think the sheer volume of trash has consequences that were not so serious 100 years ago, even if the percentage of trash was the same. I strongly suspect the percentage of trash has been going up, as well.
My argument is that "publish or perish" makes less and less sense the more active scientists and researchers there are, even if the average quality and the rate of publication per academic were constant, because the appetite and rate at which research can be assimilated by society is limited, and does not scale with population, while the number of scientists does.
I don't think these claims are extraordinary, and if you do, I'm not going to go looking for extraordinary evidence to try to convince you. I don't think I'm the only that sees these effects, however.
> One obvious test: is the per-capita rate of publication increasing?
It definitely is, in biology at least. My graduate mentor was really interested in publication metrics (as in, he published studies on them). The main driver is not necessarily crappy journals though, it is the increasing number of authors per paper.
I have no idea how you would evaluate something like "the average quality of papers is decreasing". I actually agree with GP that it is, but that's just, like, my opinion, man.
Author lists may (again:
evidence required) be growing, but that’s not at all the same thing as per-capita publication rate, and doesn’t support the GP claim.
A priori, long author lists indicate collaboration, which is generally a good thing.
The number of citations/references has ballooned as well. So, if papers stand on more shoulders , most of which are crappy shoulders, they ll make crappy science
Reminds me of few antivaxers and trolls who grab those research papers to recruit people. If you see their arguments without contrary points, it's not impossible to fall for it.
> it simply doesn’t matter very much whether any particular paper is reproducible
If we are talking about highly abstract types of science, I agree entirely. The problem is that there are strong incentives for groupthink, even where public politics aren't involved.
For example, I'm involved in the aging field. One of the popular aging hypotheses was oxidative stress. Because of the number of scientists with careers invested in that hypothesis, research kept on for well over a decade after it was debunked. In fact, I work with a person who cowrote the paper that authoritatively debunked it over a decade ago, and that person still studies it and writes as if it were still true!
How much more so if the subject is politicized beyond a narrow community of scientists, then. I do not want to get into a political debate here, but evolution and vaccinations have over a century of scrutiny, whereas other fields do not.
Another example relatively close to my area is nutrition. Scientists have been totally convinced that fat is bad, sugar is bad, both are bad, all things are good in moderation... Even the public considers nutrition to be a joke for this reason. It is not enough for a community of scientists to agree on something, a fact needs time to "settle".
I agree totally with the "process vs individual paper" distinction, I would just propose the heuristic that "the more politicized the subject, the longer the process takes".
> I don’t know many good scientists who take what they read (in any journal) at face value
Sure, in journal club people tear apart papers. And maybe in private conversation. But then, on the other hand, look at the statistics in this survey. Or look at the way this same papers that they might privately pooh-pooh will be uncritically cited in a grant application or paper if it supports their hypothesis.
"The tiny percentage of subjects that rise to the level of public policy discussion end up being so extensively investigated that reproduction of results is essentially guaranteed."
I really don't think this is true.
Ideas are war in politics, and so the truth generally is the first casualty.
Golly I wish bureaucrats would pay more attention to the nuances of science.
> Golly I wish bureaucrats would pay more attention to the nuances of science.
Instead we get a PR campaign to put an self-described unstable 16yo who can “literally see invisible CO2” on the world stage frowning and yelling about stolen childhood.
I have no hopes for political reforms to actually look at nuance while this nonsense seems to work.
This rings true. But I think basic science is poor on interesting ideas. Thats due to the pursuit of ‘minimal publishable ideas’ and consequent lack of conviction/long term perseverance. It’s easier to follow the next trendy thing rather than exhausting a search space
You shouldn’t be cringing. You should be educating people about how science actually works, and how it simply doesn’t matter very much whether any particular paper is reproducible. It’s a straw-man argument, because most papers aren’t worth reproducing. I don’t know many good scientists who take what they read (in any journal) at face value. If you do, you’ve been mislead somewhere during your training (fwiw, I also have a PhD). At best, papers are sources of ideas. Interesting ideas get tested again. Most get ignored. Even if a few hokum theories become popular for a while, eventually they’re revealed for what they are.
The tiny percentage of subjects that rise to the level of public policy discussion end up being so extensively investigated that reproduction of results is essentially guaranteed. And yeah, you hear lots of silly noise from university PR departments, but that stuff is a flash in the pan.
For example, nobody legitimate is in doubt of the broader facts of global climate change or evolution or vaccination, even if 95% of (say) social science reaults turn out to be complete bunk. Yet climate deniers, anti-vaxxers and “intelligent design” trolls absolutely love it when this distinction is ignored, because it allows them to confuse the public on the legitimacy of science as a process.