The Antiplanner has long enjoyed humor magazines, such as the Journal of Irreproducible Results. But now, reports the New York Times, it turns out that many if not most scientific journals are journals of irreproducible results. Eager to be published, many scientists appear to have an “unconscious bias” and “nudge . . . the data so it supports the hypothesis, even if just barely.”
“For most study designs and settings,” says (ironically) a peer-reviewed study, “it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.” The problem is so bad that Nature has a special web page with links to nearly twenty articles it has published on the subject.
This shouldn’t be a surprise. When I hear about research saying things like dogs poop facing north or south, or foxes are more likely to catch mice when pouncing in the northeast or southwest direction, I have to be skeptical. More broadly, when studies show that wine or chocolate is good for the heart or that cell phones cause brain cancer, I strongly suspect that the margins are so tiny (“people who use cell phones are 0.1 percent more likely to get brain cancer!”) as to be worthless.
Every scientist knows (or should know) that there is a difference between “statistically significant” and “large.” Something may be 0.1 percent greater than something else and still be statistically significant, meaning–based on the available data–the difference isn’t random. Of course, the available data may be biased or “nudged” by the data collector, but that can’t be proven without repeating the experiment. No one has time to repeat every experiment published, but a large number of attempts to replicate results have failed.
The Antiplanner has expressed skepticism of the peer review process before, and this merely underscores it. As it happens, I’ve been on both sides of the peer-review process. While I don’t seek out peer-reviewed journals to publish in, I’ve been invited to do so from time to time and I’ve also been invited to do reviews.
Trouble, cialis soft order it can happen to anyone. if you want. In today’s world we have plenty of option for male erectile dysfunction, and one of the most recommended dosage for males who have tried other dosages and want to increase the effectiveness of canadian pharmacy tadalafil the treatment. This is why it forms an important part in the disease check address now generic levitra online development. For example one needs to get a health checkup once before you take ED drugs, as some illnesses, or medicines already being taken may best prices cialis produce interactions. Thus, I know that reviewers face a basic problem which is unless they have already done the work themselves (which would mean the paper they were reviewing wasn’t original research), they know less about their subject than the paper’s authors. This leaves reviewers forced to try to find something wrong with the methodology or some alternative hypothesis that would explain the data results.
Authors, meanwhile, can often use their superior knowledge to avoid having to respond directly to reviewers’ comments. Often, making token changes in the paper effectively acknowledging some of the reviewers’ minor points will allow them to simply skip the big problems. (Not that anyone you or I know has ever done this, of course!)
This doesn’t mean we should give up doing research. But we should learn to be skeptical of the results of that research, especially when they are hyped in the media with little details about the actual research outcomes. When research shows that people who smoke are 15 to 30 times more likely to suffer lung cancer, that sounds like a pretty good reason not to smoke. But when someone says repeatedly, without citing any studies, something like “high-fructose corn syrup causes cancer,” I have to be suspicious.
Unfortunately, too many politicians feel pressure to appear to be “doing something.” This leads to the terrible political logic, “We have to do something. This is something. Therefore, we have to do this.” When such terrible logic is supported by terrible science, the policies that result are doubly terrible: costly without accomplishing anything (and probably doing more harm than good).
So the next time someone proposes to base public policy on science, be sure to ask the right questions. How reliable is the science? How much difference is the policy likely to make if the science is accurate? What happens if the science isn’t accurate? Accurate or not, how much will the policy cost? If both the cost and the risk that the policy will fail are high, then how can the policy be justified?
Update: I have to wonder how much federal funding of scientific research has influenced the process. Until a few decades ago, there was almost no federal funding for research; now, much research is absolutely dependent on such funding. At one time, people published to get tenure; today, they publish to get funding. Research, like so much else, becomes centrally planned, with committees deciding who gets the money, what fields will be funded, and what will be neglected. Researchers, meanwhile, get subjected to increasing amounts of red tape, while universities take their 40 percent, or more, for “administrative overhead.” While some might think that more money can only help, that isn’t always true.
Hmm, methinks of Reinhart-Rogoff and their famous/infamous research paper claiming a negative impact on economic growth when debt reaches a “threshold” of 90% of GDP. Research which policy makers used to justify austerian fiscal policies. A grad student finds basic errors and omissions in the data, and the “findings” are shown to be wildly over-stated, i.e., the evidence shows only a very mild negative correlation between debt and growth. (Brad DeLong estimates “that an increase in debt from 50% of a year’s GDP to 150% is associated with a reduction in growth rates of 0.1%/year over the subsequent five years.”)
Gecko55—Brad DeLong estimates “that an increase in debt from 50% of a year’s GDP to 150% is associated with a reduction in growth rates of 0.1%/year over the subsequent five years.
JK—-What about the interest on the debt? 100% of GDP implies 3-10% of GDP in interest expense (corresponding to 3-10% interest rates). Surely that must be a drain on growth?
An excellent post from the Antiplanner. There is also the problem when science has decided on something, such as global warming gases being bad. (I know there will be a flood of replies claiming science has not decided that, but working in science as I do that debate is long over. What to do about the problem, and its probably effect is now the discussion. Go into any scientific meeting claiming this is all a hoax and everything you say will be ignored.)
However, then instead of rigorously determining the best path to reduce them by calculating the cost per tonne of CO2 and reducing production the most economically way special interests use this as an excuse for their polices. For example at the European Unions highly cost ineffective methods of trying to reduce global warming gases. A sensible approach would that be advocated by Lomborg in “Cool It” (Google search words “Cool it”) who points out that there is no realistic economical way of producing power without producing global warming gases. Therefore much more research has to be done on producing power at low cost without producing global warming gases. That is not being done. Then science often fails to point out, for example, that building high speed rail lines is not a cost efficient way of reducing global warming gases.
JK — I don’t know, and don’t have the technical competence to address your question. I assume, however, that if you’re looking at the relationship between debt and GDP growth, all the drivers of debt, including interest costs, would be considered.
I agree with Paul that this is one of the AP’s better posts, especially the last two paragraphs. I was simply thinking of the Reinhart-Rogoff work as a good example of the points, and cautions, the AP was making. I wanted a citation in support of my comment, and found the quote from DeLong, who I regard as a reliable source.
I also think the Reinhart-Rogoff work is a good example of one of the points mentioned by the AP, “claimed research findings may often be simply accurate measures of the prevailing bias.” Their 90% debt-to-GDP “threshold” certainly resonated with policy-makers whose bias is that debt is bad. Always, no matter what. In spite of lots of evidence to the contrary.
This rambling, disconnected post trying some guilt by association stumbles right out of the gate: it is the simplest thing ever to run the dog poop or fox pounce experiment again. In no way are they irreproducible. If you want to pretend to be skeptical and about magnetic fields and animal behavior (despite several lines of inquiry showing otherwise), either do your own canid crapping study, or come up with another idea to show the idea is bunk. No better way to show them than to show them.
Just because a publishing house caught some hapless, bunbling true believers making up stuff doesn’t mean we should stop funding research. Indeed, it shows the process works.
DS
Fear of ionizing radiation led to lengthy do nothing studies regarding the use of cell phones. But why only cell phones?……You know how many devices emit ionizing radiation? All of them. We get radiation every single day of our lives from womb to coffin. We emit radiation because our human bodies despite being mostly water also contain minerals which are mildly radioactive and those minerals absorb radiation. A sunburn is radiation damage, we just don’t call it that. In Canada a reactor operating will be shut down by 2015 which is tragic because it’s the only reactor that currently supplies two radioactive elements that are essential to medicine. Molybdenum-99 which is crucial for radiology for high resolution detailed medical scans. And Bismuth-213 which is being pursued as a cancer destroying radiological therapy. Because it’s half life is so short it only has opportunity to destroy cancer cells and does it with alpha particles which is way more precise than current techniques that use beta particles.
Building a nuclear reactor in Canada requires three years of environmental assessment and review. In America, over a decade. Building a coal plant only takes a few weeks. Coal and gas plants are able to release radioactive materials into the air in far, far greater amounts than what the EPA would conside acceptable if it were a nuclear plant. Only because it’s considered NORM (naturally occurring radioactive materials) Whenever you frack underground shale for natural gas , you release plenty of radon, burn the gas, the radon gets released into the atmosphere. No one counts that radon against burning natural gas or uranium and thorium particulates when burning coal. Coal outputs more radioactive materials. If the regulations existed, the agency would shut down the gas and coal plant and they’ve spent a lot of money so agencies don’t regulate NORM compared to nuclear power. The radioactive release for example of replacing every coal plant in America with a nuclear plant would be heavily reduced.
During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 “landmark” publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development.
Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.
“It was shocking,” said Begley, now senior vice president of privately held biotechnology company TetraLogic, which develops cancer drugs. “These are the studies the pharmaceutical industry relies on to identify new targets for drug development. But if you’re going to place a $1 million or $2 million or $5 million bet on an observation, you need to be sure it’s true. As we tried to reproduce these papers we became convinced you can’t take anything at face value.”
http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328
It goes without saying that social “science” studies are not reproducible, even with the ridiculously low confidence level those publications believe in.
“Go into any scientific meeting claiming this is all a hoax and everything you say will be ignored.” The word “any” implies that all scientists agree–that is far from the case.
But I agree, calling this a “hoax” is wrong. Reasonable people should be able to disagree without calling others, or their beliefs, names. But go into any global warming scientist meeting and ask any question that doesn’t fit the party line – and you will be vilified.
go into any global warming scientist meeting and ask any question that doesn’t fit the party line – and you will be vilified.
Sounds like a myth to me. Any evidence for this assertion?
I have regularly attended scientific meetings. If one has good data to back up one’s position at scientific meetings one will be taken seriously, if not agreed with. If one has not studied the issue and try to just say any position is a hoax without data one will be ignored. Studying the issue means not just reading newspaper articles about the research, as the Antiplanner points out. it is necessary to have some coursework or background from reading many refereed papers on the issue. One then has to make in informed decision on the papers.
The position of showing up to meetings claiming something is a hoax is too bad and detrimental to ones position. For example in the San Francisco Bay area where the “Plan Bay Area” is being fought there is plenty of data to show that this plan does not create any significant decrease in global warming gases, its claimed effect. The plan is incorrect and non-scientific. Unfortunately some opponents show up carrying signs saying “global warming is a myth” and are ignored as it is assumed they do not know what they are talking about. If they want to believe that and show up to a meeting claiming that, fine. However, they are unlikely to get the plan changed.
Instead, pointing out that the plan makes no significant difference to production of global warming gases and the plan is either incompetent or at worst deliberate lying is much more effective. This is a powerful argument that sways most whatever they think or believe on global warming gases.
Here is a much more sober (and less biased) take on the Rogoff-Reinhardt controversy. Interestingly enough, the author’s advice includes many of the caveats Randal mentions in his post. They are well worth reading.
Just because a publishing house caught some hapless, bunbling true believers making up stuff doesn’t mean we should stop funding research. Indeed, it shows the process works.
I disagree that this would be evidence that “the process works”. More often than not, it is not the publishing house or an initial reviewer that identifies a flawed study. It is a subsequent reader (and often one who does not share the author’s leanings) who notices something that sounds suspicious. Even then, it is rare that a reader will go to the effort of contacting a journal or the study’s author in order to try to reproduce the results. This all feeds into the problems of confirmation bias and publication bias that exist in many fields.
I think Larry Summers makes some good points about the importance of not only good research design and attention to detail, but also reproducability before any piece of research is used to push a policy agenda.
Paul commented:
…For example in the San Francisco Bay area where the “Plan Bay Area” is being fought there is plenty of data to show that this plan does not create any significant decrease in global warming gases, its claimed effect. The plan is incorrect and non-scientific. Unfortunately some opponents show up carrying signs saying “global warming is a myth” and are ignored as it is assumed they do not know what they are talking about…
…Instead, pointing out that the plan makes no significant difference to production of global warming gases and the plan is either incompetent or at worst deliberate lying is much more effective.
This sort of John Birch Society style rhetoric doesn’t help the “everything is OK, we can keep driving our cars as we’ve have in the past” cause, either:
…Heather Gass, a Danville realtor, carried a sign that said, “ABAG and MTC don’t speak for me. This is a rigged meeting.”
“Stop lying to the public. This is about socially engineering our lives,” Gass told the panel of MTC and ABAG representatives.
She was one of four with Danville connections who spoke at the meeting.
Terry Thompson of Alamo also spoke.
“This is all about central planning. It didn’t work in the Soviet Union and it won’t work here,” he said. “There’s no such thing as regional government.”
From http://danvilleexpress.com/news/2013/07/19/plan-bay-area-passes.
Paul also commented:
…Instead, pointing out that the plan makes no significant difference to production of global warming gases and the plan is either incompetent or at worst deliberate lying is much more effective….
Yes, lying I suppose but not in the way you mean.
See http://blogs.kqed.org/newsfix/2013/07/16/103764/
http://blogs.kqed.org/newsfix/2013/07/16/103764/
Bay Area Climate Change Plan Won’t Actually Reduce Emissions
…Bay Area planning officials say efforts to encourage dense development will reduce greenhouse gas emissions from vehicles.
But what they rarely mention publicly is that their goal, a 15 percent per capita reduction of carbon dioxide from cars and light trucks by 2035, actually represents an overall emissions increase.
Essentially, it’s a math trick: The per capita figure hides a predicted regional population growth of 28 percent. That means total passenger vehicle emissions regionwide would actually rise by 9.1 percent — an indication that regional planning is not helping California’s efforts to become a model in combating climate change.
Senate President Pro Tem Darrell Steinberg said the decision by state regulators to present greenhouse gas emissions on a per-person basis instead of overall contradicts the spirit of his signature climate bill, the Sustainable Communities and Climate Protection Act of 2008….
The yammerings of most of the Anti Plan Bay Area activists aside, the deception is that to actually reduce overall emissions 15% vs. the 28% increase in population projected, per capita emissions would have to be reduced 34%-35% by 2035, perhaps doable would requires major technological and behaviorial changes.
Beyond some of the modest reductions in VMT resulting from demographics due to an aging population and the current 20-somethings’ growing indifference to motor vehicles, actual results in GHG emission reductions as noted here would require things like congestion charges and daily parking fees, investment in transit improvements that actually had a major impact on ridership (as opposed to porkbarrel projects such as S.F.’s Central Subway), as well as whole host of other things, both in transportation and elsewhere. No wonder the politicians chickened out across the board, and environmentalists and social justice advocates thought Plan Bay Area didn’t go nearly far enough.
On Plan Bay Area, would be more accurate to claim that the POLITICIANS certainly lied, but also that the bureaucrats and consultants DID EXACTLY WHAT THE POLITICIANS WANTED, so I strongly the Plan Bay Area numbers and data are mostly accurate, pr se.
But of course, pointing out that Plan Bay Area was under-ambitious doesn’t serve the purposes of Heather Gass, Citizen Marin and the many other individuals and groups who didn’t want the thing at all. I suspect they’d rage against ANY regional plan that didn’t accept what they think the accurate data and projections are, too.
I disagree that this would be evidence that “the process works”. More often than not, it is not the publishing house or an initial reviewer that identifies a flawed study. It is a subsequent reader (and often one who does not share the author’s leanings) who notices something that sounds suspicious.
A subsequent reader would be part of the process that works.
I hear there are meetings going on to come up with a better method than peer review, and that things are in the works and processes being developed to overcome the few papers a year that make it through….
Oh, wait: no I don’t.
DS
It is nice of the Bay Area to do so much so they can believe they delay global warming by 20-30 minutes a hundred years from now. I am sure it is worth it to them for the smug factor.
http://www.southparkstudios.com/full-episodes/s10e02-smug-alert
Once the environmentalists get smart enough to assign CO2 “costs” to the financial sector, then suddenly SF and NYC will lose their smugness.
Sandy Teal:
…Once the environmentalists get smart enough to assign CO2 “costs” to the financial sector, then suddenly SF and NYC will lose their smugness.
No, they’ll be collecting the commissions on it all, even more so than now. And they’ll have much less to pay for GHG emission costs relatively speak than the far suburbs, which produce 2-5x as much per capita GHG as San Francisco, New York City, and other dense urban centers.
“the far suburbs…produce 2-5x as much per capita GHG as…dense urban centers”
Thanks for admitting that you and Dan have a bigger CO2 footprint than I do and are contributing more to climate change.
Well, Frank thought he had a “gotcha” but he doesn’t.
The “remote rural area” I live in for family and other reasons–like no rent since our farm pays the mortgage–generates a lot less GHG per person than the “far suburbs” where there are often more cars than adult drivers. Since the other family members are semi-retired, our total GHG footprint is quite small. Can’t say that about some of our neighbors who often drive to town twice per day per adult, e.g., 12-18 miles each way depending on exactly where they are headed in Napa.
Frank, you’re a smart guy and reasonable on a few issues, but next time you try to make a specific accusation about someone you hardly know, please, “put brain in gear before engaging mouth.”
Frank, if you want to get into a pissing contest over who is more virtuous about emitting GHG, here is the starting point: http://coolclimate.berkeley.edu/documentation.
Here is the same source’s map gateway including to their peer-reviewed paper that recently got a lot of press attention: http://coolclimate.berkeley.edu/maps.
I am not sure what the point of a pissing contest would be, nor what it would really prove, given that many of us have many fewer options about where we can afford to live in the wake of the 2008 financial crisis. And what I spend on gasoline each month is a tiny fraction of the $3,000+ rents for even one bedroom apartments a decent San Francisco neighborhood, or $2,000+ in Oakland or Berkeley–for housing products obviously in extreme demand.
msetty – Are you saying that the carbon footprint of the maid and seven children who has to commute a long distance to her job in the rich apartments of the Upper East Side should be charged to the far suburbs, or to the Upper East Side?
Are you saying the extremely wealthy lifestyle of Dubai is a relatively small carbon footprint, because you know there is no manufacturing there?
Isn’t financing part of every carbon producing industry? How much?
A subsequent reader would be part of the process that works.
No, it wouldn’t, because that reader is not part of the process. The reader the consumer of the finished product. Would you argue that a food processing firm’s quality control process ‘worked’ because it only resulted in a few recalls or only sickened/killed a few dozen people?
And by way, even if it were true that only a few papers in a given journal required retractions/errata in a given year, how many more flawed submissions made it through that weren’t detected (either by reviewers or interested readers)?
On Plan Bay Area, would be more accurate to claim that the POLITICIANS certainly lied, but also that the bureaucrats and consultants DID EXACTLY WHAT THE POLITICIANS WANTED, so I strongly the Plan Bay Area numbers and data are mostly accurate, pr se.
So you admit that planners either a) lied themselves or, b) were complicit in promoting someone else’s lie. You are a real credit to your profession.
No, it wouldn’t, because that reader is not part of the process. The reader the consumer of the finished product.
Um, no. That implication and incorrect description of the process is the exact opposite of how the world works. Scientific inquiry is iterative.
Thanks!
DS