Take a photo of a barcode or cover
123 reviews for:
Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science
Stuart Ritchie
4.33 AVERAGE
123 reviews for:
Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science
Stuart Ritchie
4.33 AVERAGE
Succinct and often funny diagnosis of the currently broken scientific system. Bad incentives combine with human drive towards status and prestige and intersect with organizational desire for profit. A recipe for the disaster that has been unfolding for at least last 20 years. Not much here is new to those familiar with modern scientific system but still worth a read for a condensed and comprehensive overview of the issues.
Prior to reading this book my I had a sceptical stance towards most published research with a view of "anything published might be true", right now I think my stance is closer to "anything published is possibly false unless replicated"
Worth a read.
Prior to reading this book my I had a sceptical stance towards most published research with a view of "anything published might be true", right now I think my stance is closer to "anything published is possibly false unless replicated"
Worth a read.
informative
inspiring
reflective
fast-paced
Very clear and insightful. A very necessary read for everyone involved in (quantitative) science. As for qualitative science, seek elsewhere.
challenging
hopeful
informative
medium-paced
adventurous
challenging
hopeful
informative
inspiring
reflective
tense
fast-paced
Is This An Overview?
Science is a collaborative effort in error correcting information and improving on the knowledge that is available. As a collaborative effort, as a social field, the research needs to be shared and people convinced. Scientists are humans themselves, who have human biases. Scientists choose how to approach their research, they choose how to interpret their research and competing research, choose whether to publish or not, and choose how to persuade others. Each choice contains biases that can and has led to the spread of misinformation.
Scientists have been trusted, and are trusting themselves, but the system has enabled those who can exploit the system of science to wield power. The scientific community has perverse incentives as those who are untrustworthy are more likely to be promoted for they are willing to compromise the research process, than the trustworthy who seek to improve the knowledge base. Incentives that reduce the reliability of research.
Research is shared through a publication, but what is wanted for publishing is not necessarily what is needed to be published. What often gets published are the exciting results, exaggerated, misleading, and often wrong. The research that challenges or replicates other research are not welcome in publishing, even though they are needed to provide the limitations and legitimacy for the claims. Not publishing seemingly unimportant research, distorts the scientific record and enables harmful outcomes. There are costs to time, effort, and money when using and providing research that is uninformative.
The practice of science has been corrupted. Rather than error correcting, science enables misinformation to spread. Science needs to change how it is practiced to enable trust in the community. This book provides guidance on how science has been exploited, and methods to improve the practice of science.
Is Science An Ideal Field?
Science depends on a communal process to find errors and faults to determine whether claims are reliable and important. Being a communal process, requires persuading peers. But by focusing too much on persuading peers, scientists lose track of the purpose of science which is to get closer to truth. Persuading peers can take on various human biases that reduce the validity of the scientific process.
Skepticism is supposed to be the basic norm of science, but has enabled incompetence, delusion, lies, and self-deception. The very ideal that scientists hold about science, that of an error correcting system, has given space to research done with human biases while claiming to be objective and unbiased.
Which Research Is Published?
Scientific studies need to be replicated to prove that the results did not come by chance, fraud, or equipment error. Replication is meant to prevent false findings, bad experiments, and inappropriate data. But replication is not taken seriously, and studies are not often replicated. Claims are accepted without checking for replication. There are barely any attempts to replicate prior results. Creating a replication crisis, in various fields. When replication is attempted, many results fail to replicate. Various research results are used to make policy and health choices that have immediate negative consequences when the results have not been replicated.
News and journals focus on the new and exciting research, which tend to be primarily positive results with a few null results. Positive results are those in which discoveries are made, while null results are those in which no discovery is made. Repeat studies are usually rejected from publications, even if they show a different or contradictory result than the original. Scientist choose to publish results when they have positive research while not publishing null results. As positive, flashy, novel, newsworthy results are rewarded much more, scientist are incentivized to produce those results, and convince others that the research has the wanted attributes. Creating a publication bias. By failing to publish null results, there is an exaggerated importance of effects that create misleading beliefs. Publication bias distorts the information that is used to make decisions, leading to making decisions based on partial information. Decisions that are liable to create problems.
To get hired and promoted, scientists need published papers with appropriate journals. Universities are ranked by the papers they produce, which results in a publish or perish mentality. As scientists have limited time to publish papers along with the rest of their responsibilities, the scientific standards become bypassed. Quantity matters more than quality. Scientists can split their research into many papers, providing an artificially better CV. Without knowing the content of the papers, readers of one or few can think there is more evidence for results than there actually is. Low citation count can be an underappreciated work, but scientists are willing to publish useless works to secure jobs and grants rather than advance science.
Hype can be very harmful in science. Many press releases give recommendations to change behavior based on results that the research could not support. Press releases are important because journalists are time-pressed and therefore closely copy the language of the press release. This is known as churnalism. The problem with hyped science is that while the hyped research gets a lot of attention, the refutations are barely able to catch up. The scientific system incentivizes the lack of caution, restraint, and skepticism.
Peer review is enough to prevent flawed ideas from being published. Peer-review researchers can prevent alternative conclusions from being published. The h-index ranks citations based on number of studies, but this measure can be corrupted. Reviewers created conditions to make sure that papers they published listed the reviewer’s papers. Researchers have even created a citation cartel with editors collaborating with others for citations.
There are even problems with reproducibility. Results do not reproduce using the same data. Often because the method of reporting was not clear enough, or steps were left out of the report.
Papers that have been proved to be wrong are retracted. They remain in the literature with a retracted mark indicating that the paper is no longer considered legitimate.
How Can Science Go Wrong?
Not even highly respected scientific institutions are exempt from protecting their reputation by protecting the activities of fraudsters. Fraud comes about by exploiting trust. There will always be those who want fame and success above other concerns. Fraud does disproportionate damage to science because it takes time to investigate the findings, which takes researchers away from their own research. Fraud also wastes money through theft, people spending money trying to obtain results that were never real, and researches waste their funds trying to replicate fraudulent research. Fraud damages the reputation of scientists.
Although relatively few papers are retracted, for various reasons that include fraud. Anonymous surveys asking scientists if they committed fraud results in a relatively large portion of scientists admitting to fraud. Worse, as the portion of fraud increased when asked about known other researchers committing fraud. The actual numbers are higher, because not everyone would be willing to admit to fraud even anonymously.
Researchers can put in fake numbers into their papers to make their paper appear more attractive than it actually is. But that means that everyone who is looking at the paper and making use of the paper, are using wrong information. There are instances when measurements are accidentally incorrectly recorded, known as measurement error. There is an expectation that numbers are noisy. But, made up numbers do not have the properties of genuinely collected data.
There is sampling error which means generating wrong interpretations about the population from the sample. The different samples can have different averages, along with chance providing very different averages.
P-value indicates the potential randomness of getting a result if the hypothesis was not true. It does not indicate if the result is true or important. Statistical significance is given a p-value of 0.05, which is an arbitrary number. Significance does not indicate a worthy result. Scientists can also p-hack. They can run a plethora of tests until they find a test that is statistically significant. Alternatively retroactively come up with a hypothesis after they find a result they approve of. Both versions of p-hacking invalidates the p-value as they create methods of getting results through random chance. Running many tests increases the likelihood of getting a significant result by random chance. Without sharing the results that were not significant, leads to people being convinced of fake results. More opportunities means more chances for false-positive results. P-hacking is a way to make noise appear valuable.
How To Improve Science?
What is measured gets focused on. Creating conditions that make the metric meaningless, which overrides genuine scientific progress. Removing arbitrary measures is not necessarily going to resolve bad research practices, for that might introduce other sources of subjectivity.
Pre-registration enables researchers to be accountable to what they are planning to do. If a paper has the condition of being published no matter the results, as long as they maintain the pre-registration plan, then that eliminates many incentives for bias and fraud.
Caveats?
The listed problems of science are common in life. What the author does is reference the problems with scientists as their source. This book is critical of how science operates, for by knowing where science can go wrong, can science be corrected.
The author references the lack of publications on replication and null results from which no discoveries are made. Both types are needed in science, but they can also be corrupted.
Scientific Meta-Hype
Here’s a rough summary of Science Fictions:
1. There is no officially established procedure called ‘scientific method,’ by which to judge the quality of research results.
2. The process by which the results of scientific research are validated for consideration by the scientific community cannot ensure the reliability of these results either.
3. Consequently what circulates at any given time as scientific fact is mostly wrong or misleading. It takes time to discover errors.
4. Steps can be taken, mostly by non-scientists and a new kind of science, to reduce if not eliminate the amount of junk science currently being produced.
In other words science works, when it does, not because of how experimentation, theorisation, and analysis are carried out, or how the findings of individual scientists are publicised or criticised by colleagues, or because these findings are proven wrong, but because most of what is publicised will eventually be ignored as irrelevant. This Ritchie finds disturbing.
A key word in the above is ‘eventually.’ For science to be science, everything that is known is tentative. And centuries of scientific experience shows that everything known at any time will be ignored at some future time except as a kind of intellectual fossil. This is as close to an accurate existential definition of Science as one is likely to get.
I don’t think Stuart Ritchie would disagree with this assessment. Science, like politics, is extremely messy. That is to say, Science is inherently inefficient (I use caps to designate the modern institution). It does not progress according to any definable logic since it is constantly reviewing the logic it has previously adopted. Therefore, looking back from any point in time, the resources engaged in scientific efforts - money, talent, time, administration - have largely been spent in a demonstrably fruitless way.
This waste is essentially what Ritchie is writing about. A large part of his book is devoted to the errors, frauds, and bloopers in scientific research ranging from his own field of psychology to cancer research and molecular physics. Eventually these mistakes mostly are not refuted but buried by further research. In the meantime the scientific community has wasted effort. And, he says, this waste has serious impact because of delay in acquiring important knowledge for health, social policy, and the general well-being of society. The waste can be reduced, he says, and he has suggestions about how to do that.
Ritchie calls our current situation a “crisis.” He believes the existing institutions of Science are “corrupt.” He cites compelling evidence that “any given published scientific article is more likely to be false than true.” There have been, he says, “over 18,000 retractions in the scientific literature since the 1970’s,” largely due to forgery, conflict of interest, self-promotion or even criminal intentions. In cancer research Ritchie cites a study in 2017 which:
“… scoured the literature for studies using known misidentified cell lines found an astonishing 32,755 papers that used so-called impostor cells, and over 500,000 papers that cited those contaminated studies”
So serious business. Perhaps the UK government, which was purportedly “following the science assiduously” at the outset of the COVID pandemic in 2020 should have read Ritchie’s book immediately it was published. That might have saved some lives, relieved marital strife during lockdown, and avoided the immensely costly track and trace boondoggle. So what is it that the world should do to lessen the incidence of junk science, avoidably stupid science, not to mention criminal science? Surely this is an issue deserving of further investigation by the proper authorities.
Well a part of Ritchie’s solution is something somewhat more trivial than the problem he describes. Essentially his first recommendation is that SCIENTISTS MUST BECOME MORE VIGILANT. In more detail, this means brushing up a bit on their statistics, taking their job as peer reviewers of professional articles more seriously, mitigating the hype surrounding unusual research findings, being more watchful for professional fads, and being a little more suspicious of whatever they read in print. Hardly revolutionary, and somewhat condescending.
“Become more vigilant” is about all he can say to fellow scientists if he wants to maintain credibility. Anything else, like government supervision or professional regulations about how to conduct proper science, would destroy science itself. So he directs his next directives to non-scientists - universities, research institutes, journal editors and foundations. He would like them to stop providing incentives to scientists and academics that promote a lack of vigilance - number of published articles, citation intensity, implicit funding demands to overstate expected research results, organisational promotion etc.
But it is at this point that Ritchie’s ship of a new science runs aground and founders. He admits that scientists themselves are complicit in the web of incentives he abhors. In fact they want them:
Of course they are. So Ritchie’s killer app is an extraordinary proposal for the establishment of an essentially new profession of the “meta-scientist,” that is a group of scientists who study the work of other scientists. Part of this proposal are suggestions for new journals devoted to this meta-science, including the reporting of results of research flops, so called null result studies, which didn’t lead anywhere. He also wants public “pre-registration” of research intentions and expectations, as well as “Open Source” free access to registered research and its results. He thereby cleverly keeps scientific regulation in the family, as it were, away from politicians, government bureaucrats, and the un-lettered masses.
Ah yes, Dr. Ritchie, may one ask who controls the controllers? Will the world need meta-meta-science in a few years time. And isn’t your idea of pre-registration just a teensy bit bureaucratic and of unproven scientific worth. It’s an idea that may be suitable for big government-funded drug studies simply because of the fortunes to be gained. But for evaluating the reaction of mice to increased testosterone, for example, such regulation seems highly inappropriate. Then there’s the issue of the scientific police who would enforce the registration and supervision of research. Would their approval be necessary for changing a study’s direction mid-stream? And would the penalties for non-compliance be civil or criminal do you think?
Is it too much to assert that the condition in which science finds itself today is no different than it found itself when the Royal Society was founded in 1660, or for that matter in the ancient groves of Grecian academe. In fact I’m willing to bet that there are proportionately fewer scientific hacks in the world today than there has ever been thanks to modern procedures of accreditation and the spread of information through modern technology.
So what is the point of Ritchie’s proposals? Every example of error or malfeasance that Ritchie cites is an instance of the current community of scientists exposing and discounting flakey results. More will certainly be uncovered. Isn’t that the important fact - they will be uncovered? Not as fast as Ritchie would like apparently. But then can he demonstrate scientifically how much quicker good results will be available? And at what cost? And given that eventually all scientific conclusions will be subject to correction, is it possible that he’s just blowing smoke?
Here’s a rough summary of Science Fictions:
1. There is no officially established procedure called ‘scientific method,’ by which to judge the quality of research results.
2. The process by which the results of scientific research are validated for consideration by the scientific community cannot ensure the reliability of these results either.
3. Consequently what circulates at any given time as scientific fact is mostly wrong or misleading. It takes time to discover errors.
4. Steps can be taken, mostly by non-scientists and a new kind of science, to reduce if not eliminate the amount of junk science currently being produced.
In other words science works, when it does, not because of how experimentation, theorisation, and analysis are carried out, or how the findings of individual scientists are publicised or criticised by colleagues, or because these findings are proven wrong, but because most of what is publicised will eventually be ignored as irrelevant. This Ritchie finds disturbing.
A key word in the above is ‘eventually.’ For science to be science, everything that is known is tentative. And centuries of scientific experience shows that everything known at any time will be ignored at some future time except as a kind of intellectual fossil. This is as close to an accurate existential definition of Science as one is likely to get.
I don’t think Stuart Ritchie would disagree with this assessment. Science, like politics, is extremely messy. That is to say, Science is inherently inefficient (I use caps to designate the modern institution). It does not progress according to any definable logic since it is constantly reviewing the logic it has previously adopted. Therefore, looking back from any point in time, the resources engaged in scientific efforts - money, talent, time, administration - have largely been spent in a demonstrably fruitless way.
This waste is essentially what Ritchie is writing about. A large part of his book is devoted to the errors, frauds, and bloopers in scientific research ranging from his own field of psychology to cancer research and molecular physics. Eventually these mistakes mostly are not refuted but buried by further research. In the meantime the scientific community has wasted effort. And, he says, this waste has serious impact because of delay in acquiring important knowledge for health, social policy, and the general well-being of society. The waste can be reduced, he says, and he has suggestions about how to do that.
Ritchie calls our current situation a “crisis.” He believes the existing institutions of Science are “corrupt.” He cites compelling evidence that “any given published scientific article is more likely to be false than true.” There have been, he says, “over 18,000 retractions in the scientific literature since the 1970’s,” largely due to forgery, conflict of interest, self-promotion or even criminal intentions. In cancer research Ritchie cites a study in 2017 which:
“… scoured the literature for studies using known misidentified cell lines found an astonishing 32,755 papers that used so-called impostor cells, and over 500,000 papers that cited those contaminated studies”
So serious business. Perhaps the UK government, which was purportedly “following the science assiduously” at the outset of the COVID pandemic in 2020 should have read Ritchie’s book immediately it was published. That might have saved some lives, relieved marital strife during lockdown, and avoided the immensely costly track and trace boondoggle. So what is it that the world should do to lessen the incidence of junk science, avoidably stupid science, not to mention criminal science? Surely this is an issue deserving of further investigation by the proper authorities.
Well a part of Ritchie’s solution is something somewhat more trivial than the problem he describes. Essentially his first recommendation is that SCIENTISTS MUST BECOME MORE VIGILANT. In more detail, this means brushing up a bit on their statistics, taking their job as peer reviewers of professional articles more seriously, mitigating the hype surrounding unusual research findings, being more watchful for professional fads, and being a little more suspicious of whatever they read in print. Hardly revolutionary, and somewhat condescending.
“Become more vigilant” is about all he can say to fellow scientists if he wants to maintain credibility. Anything else, like government supervision or professional regulations about how to conduct proper science, would destroy science itself. So he directs his next directives to non-scientists - universities, research institutes, journal editors and foundations. He would like them to stop providing incentives to scientists and academics that promote a lack of vigilance - number of published articles, citation intensity, implicit funding demands to overstate expected research results, organisational promotion etc.
But it is at this point that Ritchie’s ship of a new science runs aground and founders. He admits that scientists themselves are complicit in the web of incentives he abhors. In fact they want them:
“What’s particularly disconcerting is that the people entangled in this thicket of worthless numbers are scientists: they’re supposed to be the very people who are most au fait with statistics, and most critical of their misuse. And yet somehow, they find themselves working in a system where these hollow and misleading metrics are prized above all else. ”
Of course they are. So Ritchie’s killer app is an extraordinary proposal for the establishment of an essentially new profession of the “meta-scientist,” that is a group of scientists who study the work of other scientists. Part of this proposal are suggestions for new journals devoted to this meta-science, including the reporting of results of research flops, so called null result studies, which didn’t lead anywhere. He also wants public “pre-registration” of research intentions and expectations, as well as “Open Source” free access to registered research and its results. He thereby cleverly keeps scientific regulation in the family, as it were, away from politicians, government bureaucrats, and the un-lettered masses.
Ah yes, Dr. Ritchie, may one ask who controls the controllers? Will the world need meta-meta-science in a few years time. And isn’t your idea of pre-registration just a teensy bit bureaucratic and of unproven scientific worth. It’s an idea that may be suitable for big government-funded drug studies simply because of the fortunes to be gained. But for evaluating the reaction of mice to increased testosterone, for example, such regulation seems highly inappropriate. Then there’s the issue of the scientific police who would enforce the registration and supervision of research. Would their approval be necessary for changing a study’s direction mid-stream? And would the penalties for non-compliance be civil or criminal do you think?
Is it too much to assert that the condition in which science finds itself today is no different than it found itself when the Royal Society was founded in 1660, or for that matter in the ancient groves of Grecian academe. In fact I’m willing to bet that there are proportionately fewer scientific hacks in the world today than there has ever been thanks to modern procedures of accreditation and the spread of information through modern technology.
So what is the point of Ritchie’s proposals? Every example of error or malfeasance that Ritchie cites is an instance of the current community of scientists exposing and discounting flakey results. More will certainly be uncovered. Isn’t that the important fact - they will be uncovered? Not as fast as Ritchie would like apparently. But then can he demonstrate scientifically how much quicker good results will be available? And at what cost? And given that eventually all scientific conclusions will be subject to correction, is it possible that he’s just blowing smoke?
informative
medium-paced
If I had to assign this book a genre, I would probably go with something like nonfiction informative horror. There are hundreds of citations which all help the author to prove that something has gone very wrong with science. I think my favorite part though is the end where some hope is provided that the replication crisis can be solved, thus leaving the world in a better place. The hopeful ending note and advice on methods for fixing current scientific methodology issues are necessary after the flaws in our system have been explained. I would recommend this book to essentially any audience. While it is perfectly acceptable not to be interested in science as a career or even occasional reading, it is extremely important to know how you are being impacted by current research. Essentially, if you have any interest in knowing what to believe and what to disbelieve, you should read Science Fictions.
Excellent, and a must read for anyone in the sciences, whether STEM or Social Sciences. Many themes are highly applicable to Arch/Anth.
2nd read:
I read this book when it first came out in 2020, and Stuart was actually the first guest on my podcast. I learned about issues with what’s presented to us as science from Last Week Tonight, and I wanted to learn more, so Stuart’s book enlightened me to all of the issues happening within the scientific community. Two years later, this book holds up, and now that I know more about the topic, a lot of what he discusses in the book makes much more sense. I really think this should be read by the general public because we blindly trust most of the research that comes our way, and we should be much more skeptical.
Conspiracies have become much worse since this book was first published, and at the end of the book, Ritchie discusses how people questioned whether or not he should be writing this book. Their concern was that this would give people like conspiracy theorists a way to further discredit science, but I think Stuart does a fantastic job explaining why that should be the least of our concerns. So, go get this book if you haven’t yet, and start teaching others how to be better at spotting bogus studies.
1st read:
Incredible book that I binged in a day. As an influencer who often references psychological studies but also knows how much bad science is out there, I’m always trying to learn more about this subject.
This author did a great job not just giving examples of bad science, but he explains WHY it’s happening and offers solutions. Absolutely loved this book and hope some journalists read it as well before they keep reporting on hyped up science.
I read this book when it first came out in 2020, and Stuart was actually the first guest on my podcast. I learned about issues with what’s presented to us as science from Last Week Tonight, and I wanted to learn more, so Stuart’s book enlightened me to all of the issues happening within the scientific community. Two years later, this book holds up, and now that I know more about the topic, a lot of what he discusses in the book makes much more sense. I really think this should be read by the general public because we blindly trust most of the research that comes our way, and we should be much more skeptical.
Conspiracies have become much worse since this book was first published, and at the end of the book, Ritchie discusses how people questioned whether or not he should be writing this book. Their concern was that this would give people like conspiracy theorists a way to further discredit science, but I think Stuart does a fantastic job explaining why that should be the least of our concerns. So, go get this book if you haven’t yet, and start teaching others how to be better at spotting bogus studies.
1st read:
Incredible book that I binged in a day. As an influencer who often references psychological studies but also knows how much bad science is out there, I’m always trying to learn more about this subject.
This author did a great job not just giving examples of bad science, but he explains WHY it’s happening and offers solutions. Absolutely loved this book and hope some journalists read it as well before they keep reporting on hyped up science.
quite good summary of issues with publication bias
does contradict himself a bit about p values not being important but being how you should read a paper (per the appendix) also shits a lot on small sample sizes in biomed science without talking about how expensive and inhumane larger sample size experiments are in animal studies where using 10 rather than 100 mice will be a sufficiently powered study larger sample sizes or requiring a higher threshold p value— which would under most cases Require a larger sample size in some case requiring much larger (more expensive and more animals in some research)
the main thing i’m surprised by is that he mentions paying to read papers and that the people writing and peer reviewing the research are basically volunteers and says journals are basically getting paid 3 times in free labor of reviewers, tax payer $ funding most studies, and tax payers/schools buying the articles on the back end but barely mentions that in addition to all these other things that journals themselves are Not Adding value with researchers also have to Pay to get their articles published in most journals like over 3-5k just to have the paper published which is insane.
does contradict himself a bit about p values not being important but being how you should read a paper (per the appendix) also shits a lot on small sample sizes in biomed science without talking about how expensive and inhumane larger sample size experiments are in animal studies where using 10 rather than 100 mice will be a sufficiently powered study larger sample sizes or requiring a higher threshold p value— which would under most cases Require a larger sample size in some case requiring much larger (more expensive and more animals in some research)
the main thing i’m surprised by is that he mentions paying to read papers and that the people writing and peer reviewing the research are basically volunteers and says journals are basically getting paid 3 times in free labor of reviewers, tax payer $ funding most studies, and tax payers/schools buying the articles on the back end but barely mentions that in addition to all these other things that journals themselves are Not Adding value with researchers also have to Pay to get their articles published in most journals like over 3-5k just to have the paper published which is insane.