Table of contents
Are you aware that utilizing a significance threshold of p=0.05 to indicate a discovery means you could be incorrect at least 30% of the time?
Navigating Statistical Significance without Missteps
In his seminal work, "An Investigation of the False Discovery Rate and the Misinterpretation of p-Values," Dr. David Colquhoun emphasizes that:
"... the function of significance tests is to prevent you from making a fool of yourself, and not to make unpublishable results publishable ..."
In other words, the purpose of significance tests is to prevent misguided conclusions rather than to render non-significant results publishable. He provocatively suggests that misinterpretation occurs when:
a) a discovery is claimed based on what is merely random variation, or
b) a genuine effect goes undetected, though this is less damaging to one's reputation.
Thus, adopting a threshold of p<0.05 as 'statistically significant' carries a substantial risk of error.
Understanding the False Discovery Rate (FDR)
The False Discovery Rate (FDR) represents the expected proportion of Type I errors, or false positives, which pose a significant issue during the confirmatory phase of research but are less critical during exploratory analysis.
Andrew Gelman, a noted political scientist and statistician at Columbia University, advocates for distinct approaches to exploratory and confirmatory analyses, emphasizing the importance of initial exploratory studies to identify potential findings without excessive concern for false positives. He says:
“… In this approach, exploratory and confirmatory analyses are approached differently and clearly labelled. … Researchers would first do two small exploratory studies and gather potentially interesting findings without worrying too much about false alarms. Then, on the basis of these results, the authors would decide exactly how they planned to confirm the findings.”
If you repeat a test enough times, you will always get a number of false positives. The goal in multiple testing is to control the FDR, thereby reducing the incidence of erroneous results.
The Benjamini-Hochberg Procedure
The Benjamini-Hochberg (BH) Procedure is an effective statistical method for reducing the FDR, offering a preferable alternative to the traditionally stringent Bonferroni correction. The BH procedure, which focuses on limiting the FDR, is particularly suited for biomedical research and has been incorporated into AutoDiscovery since its conception.
AutoDiscovery's Approach to the False Discovery Rate
AutoDiscovery incorporates the BH procedure, enabling it to identify statistically significant exploratory associations with clinical relevance. This integration facilitates the research process by distinguishing between:
a) Exploratory results: associations with a p-value equal or greater than the BH threshold and
b) High significance results: associations with a p-value lower than the BH threshold,
thereby accelerating the publication of findings.
This enhancement affects several aspects of the software:
- The Discovery Process: AutoDiscovery calculates the BH p-value threshold, categorizing associations into exploratory and high significance groups.
- The Discovery Map Tool: This feature now accentuates statistically significant associations for easier focus.
- The Hypo Booster Tool: It allows for the refinement of exploration by filtering for associations of high significance, streamlining the research process.