Author Topic: (Editorial) Big Science is broken  (Read 863 times)

0 Members and 1 Guest are viewing this topic.

rangerrebew

  • Guest
(Editorial) Big Science is broken
« on: April 18, 2016, 11:58:39 pm »
Big Science is broken
Pascal-Emmanuel Gobry
Eva Bee/Ikon Images/Corbis
April 18, 2016

http://theweek.com/articles/618141/big-science-broken
 

Science is broken.

That's the thesis of a must-read article in First Things magazine, in which William A. Wilson accumulates evidence that a lot of published research is false. But that's not even the worst part.

Advocates of the existing scientific research paradigm usually smugly declare that while some published conclusions are surely false, the scientific method has "self-correcting mechanisms" that ensure that, eventually, the truth will prevail. Unfortunately for all of us, Wilson makes a convincing argument that those self-correcting mechanisms are broken.

For starters, there's a "replication crisis" in science. This is particularly true in the field of experimental psychology, where far too many prestigious psychology studies simply can't be reliably replicated. But it's not just psychology. In 2011, the pharmaceutical company Bayer looked at 67 blockbuster drug discovery research findings published in prestigious journals, and found that three-fourths of them weren't right. Another study of cancer research found that only 11 percent of preclinical cancer research could be reproduced. Even in physics, supposedly the hardest and most reliable of all sciences, Wilson points out that "two of the most vaunted physics results of the past few years — the announced discovery of both cosmic inflation and gravitational waves at the BICEP2 experiment in Antarctica, and the supposed discovery of superluminal neutrinos at the Swiss-Italian border — have now been retracted, with far less fanfare than when they were first published."

What explains this? In some cases, human error. Much of the research world exploded in rage and mockery when it was found out that a highly popularized finding by the economists Ken Rogoff and Carmen Reinhardt linking higher public debt to lower growth was due to an Excel error. Steven Levitt, of Freakonomics fame, largely built his career on a paper arguing that abortion led to lower crime rates 20 years later because the aborted babies were disproportionately future criminals. Two economists went through the painstaking work of recoding Levitt's statistical analysis — and found a basic arithmetic error.

Then there is outright fraud. In a 2011 survey of 2,000 research psychologists, over half admitted to selectively reporting those experiments that gave the result they were after. The survey also concluded that around 10 percent of research psychologists have engaged in outright falsification of data, and more than half have engaged in "less brazen but still fraudulent behavior such as reporting that a result was statistically significant when it was not, or deciding between two different data analysis techniques after looking at the results of each and choosing the more favorable."

Then there's everything in between human error and outright fraud: rounding out numbers the way that looks better, checking a result less thoroughly when it comes out the way you like, and so forth.

Still, shouldn't the mechanism of independent checking and peer review mean the wheat, eventually, will be sorted from the chaff?

Well, maybe not. There's actually good reason to believe the exact opposite is happening.

The peer review process doesn't work. Most observers of science guffaw at the so-called "Sokal affair," where a physicist named Alan Sokal submitted a gibberish paper to an obscure social studies journal, which accepted it. Less famous is a similar hoodwinking of the very prestigious British Medical Journal, to which a paper with eight major errors was submitted. Not a single one of the 221 scientists who reviewed the paper caught all the errors in it, and only 30 percent of reviewers recommended that the paper be rejected. Amazingly, the reviewers who were warned that they were in a study and that the paper might have problems with it found no more flaws than the ones who were in the dark.

This is serious. In the preclinical cancer study mentioned above, the authors note that "some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis."

This gets into the question of the sociology of science. It's a familiar bromide that "science advances one funeral at a time." The greatest scientific pioneers were mavericks and weirdos. Most valuable scientific work is done by youngsters. Older scientists are more likely to be invested, both emotionally and from a career and prestige perspective, in the regnant paradigm, even though the spirit of science is the challenge of regnant paradigms.

Why, then, is our scientific process so structured as to reward the old and the prestigious? Government funding bodies and peer review bodies are inevitably staffed by the most hallowed (read: out of touch) practitioners in the field. The tenure process ensures that in order to further their careers, the youngest scientists in a given department must kowtow to their elders' theories or run a significant professional risk. Peer review isn't any good at keeping flawed studies out of major papers, but it can be deadly efficient at silencing heretical views.

All of this suggests that the current system isn't just showing cracks, but is actually broken, and in need of major reform. There is very good reason to believe that much scientific research published today is false, there is no good way to sort the wheat from the chaff, and, most importantly, that the way the system is designed ensures that this will continue being the case.

As Wilson writes:

    Even if self-correction does occur and theories move strictly along a lifecycle from less to more accurate, what if the unremitting flood of new, mostly false, results pours in faster? Too fast for the sclerotic, compromised truth-discerning mechanisms of science to operate? The result could be a growing body of true theories completely overwhelmed by an ever-larger thicket of baseless theories, such that the proportion of true scientific beliefs shrinks even while the absolute number of them continues to rise. Borges' Library of Babel contained every true book that could ever be written, but it was useless because it also contained every false book, and both true and false were lost within an ocean of nonsense. [First Things]

This is a big problem, one that can't be solved with a column. But the first step is admitting you have a problem.

Science, at heart an enterprise for mavericks, has become an enterprise for careerists. It's time to flip the career track for science on its head. Instead of waiting until someone's best years are behind her to award her academic freedom and prestige, abolish the PhD and grant fellowships to the best 22-year-olds, giving them the biggest budgets and the most freedoms for the first five or 10 years of their careers. Then, with only few exceptions, shift them away from research to teaching or some other harmless activity. Only then can we begin to fix Big Science.
« Last Edit: April 19, 2016, 12:00:01 am by rangerrebew »

Offline Weird Tolkienish Figure

  • Technical
  • *****
  • Posts: 18,225
Re: (Editorial) Big Science is broken
« Reply #1 on: April 19, 2016, 12:03:22 am »
Non-replicable science isn't science at all.

Bill Cipher

  • Guest
Re: (Editorial) Big Science is broken
« Reply #2 on: April 19, 2016, 02:32:33 am »
Perhaps there's too much pressure to publish brand-new, but not enough pressure to publish replication.

rangerrebew

  • Guest
Re: (Editorial) Big Science is broken
« Reply #3 on: April 19, 2016, 01:44:24 pm »
Perhaps there's too much pressure to publish brand-new, but not enough pressure to publish replication.

Perhaps the real pressure is the government money available to toe their line. :whistle:

Offline Weird Tolkienish Figure

  • Technical
  • *****
  • Posts: 18,225
Re: (Editorial) Big Science is broken
« Reply #4 on: April 19, 2016, 01:49:30 pm »
Perhaps the real pressure is the government money available to toe their line. :whistle:

Probably a little of both.

Bill Cipher

  • Guest
Re: (Editorial) Big Science is broken
« Reply #5 on: April 19, 2016, 02:46:05 pm »
Perhaps the real pressure is the government money available to toe their line. :whistle:

Maybe in some areas; but not in many areas that are still afflicted with the problem. 

Offline Smokin Joe

  • Hero Member
  • *****
  • Posts: 57,002
  • I was a "conspiracy theorist". Now I'm just right.
Re: (Editorial) Big Science is broken
« Reply #6 on: April 20, 2016, 05:51:51 pm »
Big Science is broken
Pascal-Emmanuel Gobry
Eva Bee/Ikon Images/Corbis
April 18, 2016

http://theweek.com/articles/618141/big-science-broken
 

Science is broken.

That's the thesis of a must-read article in First Things magazine, in which William A. Wilson accumulates evidence that a lot of published research is false. But that's not even the worst part.

Advocates of the existing scientific research paradigm usually smugly declare that while some published conclusions are surely false, the scientific method has "self-correcting mechanisms" that ensure that, eventually, the truth will prevail. Unfortunately for all of us, Wilson makes a convincing argument that those self-correcting mechanisms are broken.

For starters, there's a "replication crisis" in science. This is particularly true in the field of experimental psychology, where far too many prestigious psychology studies simply can't be reliably replicated. But it's not just psychology. In 2011, the pharmaceutical company Bayer looked at 67 blockbuster drug discovery research findings published in prestigious journals, and found that three-fourths of them weren't right. Another study of cancer research found that only 11 percent of preclinical cancer research could be reproduced. Even in physics, supposedly the hardest and most reliable of all sciences, Wilson points out that "two of the most vaunted physics results of the past few years — the announced discovery of both cosmic inflation and gravitational waves at the BICEP2 experiment in Antarctica, and the supposed discovery of superluminal neutrinos at the Swiss-Italian border — have now been retracted, with far less fanfare than when they were first published."

What explains this? In some cases, human error. Much of the research world exploded in rage and mockery when it was found out that a highly popularized finding by the economists Ken Rogoff and Carmen Reinhardt linking higher public debt to lower growth was due to an Excel error. Steven Levitt, of Freakonomics fame, largely built his career on a paper arguing that abortion led to lower crime rates 20 years later because the aborted babies were disproportionately future criminals. Two economists went through the painstaking work of recoding Levitt's statistical analysis — and found a basic arithmetic error.

Then there is outright fraud. In a 2011 survey of 2,000 research psychologists, over half admitted to selectively reporting those experiments that gave the result they were after. The survey also concluded that around 10 percent of research psychologists have engaged in outright falsification of data, and more than half have engaged in "less brazen but still fraudulent behavior such as reporting that a result was statistically significant when it was not, or deciding between two different data analysis techniques after looking at the results of each and choosing the more favorable."

Then there's everything in between human error and outright fraud: rounding out numbers the way that looks better, checking a result less thoroughly when it comes out the way you like, and so forth.

Still, shouldn't the mechanism of independent checking and peer review mean the wheat, eventually, will be sorted from the chaff?

Well, maybe not. There's actually good reason to believe the exact opposite is happening.

The peer review process doesn't work. Most observers of science guffaw at the so-called "Sokal affair," where a physicist named Alan Sokal submitted a gibberish paper to an obscure social studies journal, which accepted it. Less famous is a similar hoodwinking of the very prestigious British Medical Journal, to which a paper with eight major errors was submitted. Not a single one of the 221 scientists who reviewed the paper caught all the errors in it, and only 30 percent of reviewers recommended that the paper be rejected. Amazingly, the reviewers who were warned that they were in a study and that the paper might have problems with it found no more flaws than the ones who were in the dark.

This is serious. In the preclinical cancer study mentioned above, the authors note that "some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis."

This gets into the question of the sociology of science. It's a familiar bromide that "science advances one funeral at a time." The greatest scientific pioneers were mavericks and weirdos. Most valuable scientific work is done by youngsters. Older scientists are more likely to be invested, both emotionally and from a career and prestige perspective, in the regnant paradigm, even though the spirit of science is the challenge of regnant paradigms.

Why, then, is our scientific process so structured as to reward the old and the prestigious? Government funding bodies and peer review bodies are inevitably staffed by the most hallowed (read: out of touch) practitioners in the field. The tenure process ensures that in order to further their careers, the youngest scientists in a given department must kowtow to their elders' theories or run a significant professional risk. Peer review isn't any good at keeping flawed studies out of major papers, but it can be deadly efficient at silencing heretical views.

All of this suggests that the current system isn't just showing cracks, but is actually broken, and in need of major reform. There is very good reason to believe that much scientific research published today is false, there is no good way to sort the wheat from the chaff, and, most importantly, that the way the system is designed ensures that this will continue being the case.

As Wilson writes:

    Even if self-correction does occur and theories move strictly along a lifecycle from less to more accurate, what if the unremitting flood of new, mostly false, results pours in faster? Too fast for the sclerotic, compromised truth-discerning mechanisms of science to operate? The result could be a growing body of true theories completely overwhelmed by an ever-larger thicket of baseless theories, such that the proportion of true scientific beliefs shrinks even while the absolute number of them continues to rise. Borges' Library of Babel contained every true book that could ever be written, but it was useless because it also contained every false book, and both true and false were lost within an ocean of nonsense. [First Things]

This is a big problem, one that can't be solved with a column. But the first step is admitting you have a problem.

Science, at heart an enterprise for mavericks, has become an enterprise for careerists. It's time to flip the career track for science on its head. Instead of waiting until someone's best years are behind her to award her academic freedom and prestige, abolish the PhD and grant fellowships to the best 22-year-olds, giving them the biggest budgets and the most freedoms for the first five or 10 years of their careers. Then, with only few exceptions, shift them away from research to teaching or some other harmless activity. Only then can we begin to fix Big Science.

While I take no issue with the writer's lament that there are definite problems with manipulation of data, worse in some fields than others and motivated by funding and ego, the greatest threat is in the peer review process.

However that threat may not be what the writer thinks. In the classical peer-review, the reviewers ('the old school') would steadfastly adhere to existing ideas in science: the burden of proof laid strictly on the person who proposed a new idea. In a way, this is a good thing, in that those reviewers would scrutinize the evidence for flaws, the methodology for inconsistency, and the conclusions would have to be supported by the evidence to the point that (sometimes long held) conventional scientific thought would have to be discarded and replaced with a new paradigm. So it was with Continental Drift and Plate Tectonics. Evidence needed to overcome the intellectual inertia present in geosynclinal theory was overwhelming, cutting edge, and yet there were fistfights over which was correct in at least one convention. Now, Plate Tectonics is well accepted.
 Where all this becomes a problem, however, is when the peer-review process becomes hijacked. When the reviewers agree with the theory at the onset of the process, they do not lend the sort of scrutiny to the methodology, the purity and completeness of data, and the logic by which conclusions are developed that someone who disagreed with those conclusions might, being hostile to the conclusions of the study.

Instead, we see papers reviewed by people who are in agreement with the concepts contained therein, and a varying amount of scientific laziness in assuring the veracity and repeatability of the data.
With that, comes the reference penumbra. If paper 1 has a conclusion not supported by data or just logically flawed, but passes review, others may use that paper as a reference, citing only conclusions not data. that second generation of papers will be colored by the conclusions of the first, and the false precept continues to cast its shadow through generations of studies, each referencing the conclusions of earlier studies. I have encountered this concept firsthand reviewing literature which dealt with social costs of motorcycle accidents, and found the assumptions drawn from conclusions of paper after paper, through several generations finally led back to a study done by the US Army on dispatch riders in wartime wearing helmets--although I could not find a copy of that study. I can see where the Army might find that wearing a helmet while riding a dispatch motorcycle in wartime might be safer, especially near front lines. Carrying that conclusion over to a different environment and different style of helmet might not be valid, however. (Opinions vary on the subject, but I do not intend to have that debate here). Just suffice it to say that those conclusions were carried through generations of studies, other flaws crept in, and the flaws in conclusions mounted with the prejudices of the review staff and the researchers.

For example, the idea of Anthropogenic Climate Change will be far harder to ferret out and remove from literature when the time comes, and the resistance to doing so will mount as there are more and more layers of 'research' predicated on and in agreement with original flawed or massaged data (still unavailable for review), and based on methodology which remains questionable.

Add to this the overwhelming speed with which a theory and its proponents can reach rock star status in the popular press, and thus, in political circles if those conclusions may be exploited for political aims, and those roaring young lions become prestigious, well funded (from public coffers), and decide the fate of those who follow--especially the 'heretics'--by denying the publication of papers opposing the very viewpoints which gave them status, not because those papers are scientifically flawed, but because that status is threatened. For the process to work correctly, time is a factor: time to fully vett the data collection, the data, and the methodology, as well as the logical follow through to the conclusions. The pressure to publish to maintain tenure only increases the tendency to publish the first thing that fits might well miss other, better, theories and more valid cause-effect relationships, and leaves science bleeding out in the gutter, slashed by Occam's Razor.

The unfortunate part in all this stems from the use of public funds to promote agendae which commonly only result in the usurpation of political power by the very agencies which supply the funding. While those same agencies roundly decry the validity of research conducted by independent or industry related institutions or researchers as being tainted by the source of their funding, they will accept no such mantle themselves, claiming to be the only proponents of the "Public Good", despite the potential for the political exploitation of that research. It isn't the same thing as doing fundamental research which seeks basic laws which apply over a wide spectrum of disciplines and can promote future technological or medical developments, instead, the purpose of this sort of research is to justify the limitation of development, the transfer of wealth from one part of a population to another, or the accumulation of power at the expense of human rights.

There, especially, science is being abused.
How God must weep at humans' folly! Stand fast! God knows what he is doing!
Seventeen Techniques for Truth Suppression

Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience.

C S Lewis