Science is a process that begins with observation that leads to the formulation of a theory that is tested experimentally. Even if the result confirms the theory, there is another critically step that must be completed before the theory can be validated. The result must be independently reproduced. Reproducibility is a foundation of science. If an experimental result cannot be reproduced by an independent lab, the theory is unreliable and cannot be accepted as part of scientific knowledge.
Peer review is the process by which scientists review and evaluate the work of other scientists that is published in professional scientific journals.
A recent study published in the professional journal, Science, concluded that only 36% of 100 experimental results published in the top three professional journals in the field of Psychology could be independently reproduced. This result means that much of what psychologists assume to be true is not reliable.
In the Abstract of a research article titled, Estimating the Reproducibility of Psychological Science, the authors conclude,
No single indicator sufficiently describes replication success, and the five indicators examined here are not the only ways to evaluate reproducibility. Nonetheless, collectively these results offer a clear conclusion: A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes. Moreover, correlational evidence is consistent with the conclusion that variation in the strength of initial evidence (such as original P value) was more predictive of replication success than variation in the characteristics of the teams conducting the research (such as experience and expertise). The latter factors certainly can influence replication success, but they did not appear to do so here.
Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that “we already know this” belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not. This project provides accumulating evidence for many findings in psychological research and suggests that there is still more work to do to verify whether we know what we think we know.
The American Psychological Assocoation (APA) is already under fire for its continuing support of the use of
torture enhanced interrogation techniques, even though the organization and the war criminals people who use these techniques have never been able to show that they have produced accurate information not already known. The results of this study cast further doubt on whether Psychology is a science or junk science.
There has been a lot of talk this year about the need to reform the criminal justice system. No doubt we will hear a lot more talk about the subject in the run-up to the election next year. Most of the discussion has focused on releasing nonviolent offenders from our overcrowded prisons. I agree that we should do that and I have also proposed that we officially end the failed War on Drugs and decriminalize the use and possession of all drugs, including heroin and cocaine, by following the successful model established by Portugal.
Based on my extensive experience as a criminal defense attorney specializing in death-penalty defense and forensics, I can assure my readers that we also have to tighten the foundational requirements for expert witnesses in Evidence Rule 702, as it relates to self-described mental health experts, particularly psychologists. One especially troublesome area is the prediction of future violence. Despite the absence of any evidence that a psychologist can more accurately predict whether a particular individual will commit a future act of violence compared to merely flipping a coin, prosecutors in death cases still put on their
whore psychologist who predicts the defendant will commit future acts of violence. Jurors, who don’t know any better, respond with death verdicts. This crap needs to stop.
I will leave for another time a discussion about the woeful state of forensic laboratory work. Indeed, forensic fraud is one of the major causes of wrongful convictions. Just to provide a quick for example, Kentucky crime lab ‘experts’ still testify that a hair found on a victim’s body at a crime scene ‘matches’ a defendant’s known hair. The so-called match is based on a visual hair comparison using a stereoscopic microscope. They do this even though we have known for 20 years that the most can say about a visual hair comparison is that two hairs were contributed by two belonging to the same race. Further differentiation is impossible unless a nuclear DNA test (assuming a root is attached) or mitochondrial DNA (assuming a follicle with no root) is performed. Because the visual appearance of a hair differs substantially depending on what part of the hair is examined under the microscope. it’s impossible for an examiner to accurately conclude, unless he makes a lucky guess, that two segments of one hair follicle came from the same individual (assuming he does not know that the hair follicle was cut into two lengths).