Low statistical power in biomedical science: a review of three human research domains (2024)

Associated Data

Supplementary Materials

Data Availability Statement

Abstract

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

Keywords: statistical power, reproducibility, neurology, psychiatry, somatic disease

1. Background

There is growing consideration of the possibility that many published research findings may be false, indicate effects that are in the opposite direction to any true effect, or indicate effects that are substantially inflated compared to any true effect[1]. All of these will increase the likelihood that research findings will prove difficult to replicate. Within the widely used null hypothesis significance testing (NHST) framework, one factor that contributes to poor reproducibility is low statistical power[2–4]. However, the relationship between statistical power and the likelihood that a statistically significant finding represents a false-positive result is under-appreciated[5]. Low statistical power (arising, for example, from low sample size of studies, small effects being investigated, or both) adversely impacts on the likelihood that a statistically significant finding actually reflects a true effect and (if the effect is indeed real) increases the likelihood that the estimate of the magnitude of that effect is inflated (also known as a type M or magnitude error) or in the opposite direction (a type S or sign error) relative to the true effect[6].

Despite the central place of statistical power within the NHST framework, there is evidence that it remains poorly understood and is often not considered at the study design stage[5,7]. A survey by Vankov et al.[5] of authors who had published in a leading psychology journal indicated that approximately one-third held beliefs that would, on average, serve to reduce statistical power. This included using accepted norms within their field to decide on sample size, in the belief that this would be sufficient to replicate previous results. Given evidence that many published findings report p-values close to the conventional 5% threshold for statistical significance[8], this belief is likely to be unwarranted. If an experiment finds an effect with a p-value close to 0.05, and if we assume the observed effect size is accurate, an exact replication of the experiment with the same sample size will on average reproduce that finding only 50% of the time[5]. In practice, power is likely to be lower than 50% because the effect size estimate observed in the original study will probably be an overestimate[2,9]. However, in this survey, over one-third of respondents inaccurately believed that in this scenario the finding would replicate over 80% of the time[5]. Button et al.[2] recently reported that studies in the neuroscience literature have a median statistical power of approximately between 8 and 31%. However, there is no particular reason to believe that the situation in the neuroscience literature is any better or worse than the situation across the biomedical sciences in general. Nevertheless, this has not been formally investigated across different domains of biomedical science to date.

We, therefore, conducted a review of meta-analyses of studies of the association of biological, environmental or cognitive/behavioural parameters with neurological, psychiatric and somatic diseases (excluding treatment studies to ensure that included studies were more directly comparable). Our intention was to estimate the average statistical power across these domains, and explore whether studies in particular domains tend, on average, to have higher or lower statistical power than in other domains. Our selection of disease domains was partly arbitrary, although we excluded, for example, stroke from neurological diseases because it is also a cardiovascular disease, and traumatic brain injury because it is accidental. Our selection of neurological diseases included four severe diseases with the highest prevalence, while our selection of four psychiatric disorders was less systematic and included two less prevalent disorders (autism and schizophrenia) and two more prevalent disorders (ADHD and unipolar depression). Finally, our selection of four somatic diseases was also largely arbitrary; these were selected because their biological causes and risk factors are still poorly understood, and because their prevalence is similar to that of the other domains.

We applied a similar methodology to that used previously by Button et al.[2], taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and estimated the average statistical power of studies in the meta-analysis to obtain p < 0.05 given that effect size (i.e. to reject the null hypothesis of no effect at p < 0.05). Whereas Button et al. selected meta-analyses published in a single year (2011), we selected meta-analyses from a longer period (2008–2012).

2. Material and methods

3. Results

4. Discussion

Our results indicate a striking similarity in average statistical power across three distinct biomedical research domains, and align with similar findings we have previously reported in the neuroscience literature[2], with a median statistical power in the region of 20%. In other words, approximately 50% of studies have power in the 0–10% or 11–20% range, well below the minimum statistical power of 80% that is considered conventional. Low statistical power, therefore, appears to be a problem that is endemic to the biomedical sciences, at least in the specific subject areas captured by our search strategy. This is despite large differences in average sample size—indeed, there is a clear and strong inverse relationship between average sample size in a field and the average effect size observed in that field. We also see evidence, again consistent with our previous work, of a bimodal distribution of the power of individual studies—while the majority of studies appear to have very low power, a minority appears to be very well powered. An important strength of the present study is that we attempted to ensure that the meta-analyses we included, focused on biological, environmental or cognitive correlates of disease, were relatively hom*ogeneous (e.g. by excluding treatment studies). We also sampled a longer period, including meta-analyses published between 2008 and 2012, thereby capturing a much larger number of relevant studies. However, the use of meta-analyses to provide a best estimate of the likely underlying population effect size meant that we were restricted to a subset of the literature within each domain of interest, limiting the extent to which our findings can be generalized to fields where meta-analysis is uncommon.

We conducted a sensitivity analysis excluding meta-analyses where the pooled effect size estimate was non-significant. This was done to reduce the possibility that we were including cases where the true effect size is zero or close to zero. Unsurprisingly average power was higher in this case, although still considerably lower than the 80% that is often conventionally regarded as adequate. On the one hand, including non-significant meta-analyses is likely to reduce our estimate of average power because this will lead to the inclusion of areas where the true effect size is zero or nearly zero. On the other hand, effect size estimates from meta-analyses may be inflated by publication bias against non-significant results. Study quality may also impact on the meta-analytic estimate if there is a systematic relationship between, for example, study sample size and study quality; we used the meta-analytic estimate obtained under a fixed effects model, which places greater weight on larger studies, but larger sample size may be achieved at the expense of measurement precision. An alternative approach would have been to specify what would constitute a biologically or clinically meaningful effect size for each meta-analysis, and exclude those where the pooled effect size estimate was smaller than this. However, this would have been extremely challenging—not only would this have to have been done for each meta-analysis, but it is not clear that such an effect size could be specified in every case. Using a p-value threshold instead is somewhat arbitrary but at least practical and transparent. Taken together, we feel that our results are likely to provide a plausible upper and lower bound of the average statistical power within the fields and methodologies sampled. Interestingly, this range aligns well with the results of a recent model that attempted to predict the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study[14]. This showed that researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conducting small studies that have only 10–40% statistical power, even though this means that half of the studies they publish will report erroneous conclusions.

We also observed strong patterning of average statistical power by research methodology. This was not explored in depth by Button et al., although high median power (ranging from 51 to 96%) was observed in five meta-analyses dealing with cognitive/behavioural measures[2]. Consistent with this, studies in the present analysis that used cognitive/behavioural measures showed high average statistical power. By contrast, we previously estimated the median power of structural brain imaging studies to be 8%[2], whereas in this study brain imaging studies had intermediate power around 30%. The reasons for this discrepancy are not clear and warrant further investigation. Those using other measures had average power around or below 50%, while genetic association studies showed very low statistical power. This suggests that the very low power of genetic association studies may contribute to the overall pattern of results we observed across research domains. Importantly, the genetic studies included were exclusively candidate gene studies, and it is now widely acknowledged that results from these studies are not robust, due to a combination of factors such as small effect sizes and small sample sizes[15]. Whole-genome approaches (e.g. genome-wide association) use a combination of very large sample sizes, stringent statistical standards and replication, and the results of these studies are accordingly much more robust[15]. One difficulty in interpreting these results is that certain methodologies (e.g. cognitive/behavioural) were almost exclusively restricted to studies in one research domain (psychiatry). A more comprehensive analysis of the relationship between research methodology and average statistical power is warranted.

The clear linear relationship between effect size and sample size is, on the one hand, to be expected—if researchers conduct power analyses, then sample size should be related to the effect size of the phenomenon being studied. However, this straightforward explanation ignores the fact that average statistical power appears to be low. An alternative explanation is that sample size is governed largely by other factors, such as convention and constraints on resources. If publication bias is operating, whereby only statistically significant findings are published (either because authors choose not to write up non-significant findings, or because journals choose not to publish them, or both), then the filtering effect of a significance threshold (e.g. an α level of 0.05) will mean that, for a given average sample size, only effect sizes above a certain magnitude will achieve statistical significance and be published. This would also result in a relationship between observed effect size and sample size. Factors such as analytic flexibility might increase the likelihood of obtaining a statistically significant result in an underpowered study, so that conducting underpowered studies represents a trade-off between the limited resources available and the likelihood of obtaining publishable results. It is worth considering the extent to which incentive structures may shape scientists' behaviour. Publication remains the primary currency of science, and both the quantity and quality of publications by a scientist contribute to their career advancement, success in obtaining grant funding, and so on. Given limited resources, scientists may therefore (consciously or unconsciously) make strategic decisions to maximize the return on those resources, for example, by conducting a relatively large number of small studies (many of which will be publishable, despite those findings potentially being unreliable), while also conducting a smaller number of large, confirmatory studies[14]. However, these factors may differ considerably across countries with different funding systems and incentive structures.

There are some limitations to our analysis that should be considered when interpreting these results. First, and perhaps most importantly, the concept of statistical power is inherently tied to the NHST framework, and the use of a p-value threshold (e.g. 0.05) to support statistical inference. This approach, and in particular, a reliance on p-value thresholds, has been criticized widely[16], not least because it relies on inferring that a phenomenon exists when statistical evidence passes an arbitrary threshold (typically p < 0.05), which in turn implies a demarcation between adequate and inadequate sample size (and power)[17,18]. Nevertheless, the use of p-value thresholds to denote whether a result is ‘significant’ or ‘non-significant’ remains a common approach in the biomedical sciences. Second, our method requires the use of information reported in meta-analyses, but meta-analyses may focus only on certain study types. For example, they might focus on the literatures where the answer to the underlying research question remains unclear, in which case our estimates of average power might be underestimates (for example, if that uncertainty is due to small studies being prevalent in that area). On the other hand, they might only include data from high quality studies (as recommended by the Cochrane Collaboration), in which case our estimates might be overestimates. Third, given the very large number of meta-analyses included, we could not check whether any individual studies contributed to more than one meta-analysis. However, this is likely to be uncommon and therefore should not influence our results substantially. Fourth, our method depends critically on the assumption that the effect size indicated by a meta-analysis is the best estimate of any true effect size. We believe that this assumption is valid, not least because accurately estimating effect size is a key purpose of meta-analysis. Relatedly, however, our method assumes that power should always be evaluated at the true effect size. This is invalid if the true effect size is zero (in which case power calculations are meaningless). We attempted to address this by excluding estimates from meta-analyses where the pooled effect size estimate was non-significant, but future studies should also attempt to estimate the power of individual studies to obtain p < 0.05 for a minimal biologically or theoretically important effect. Fifth, we relied on the results of fixed effects meta-analysis. This assumes that all studies are drawn from the same underlying population, and attempts to estimate the effect size within that population. Random effects meta-analysis, on the other hand, does not make this assumption, allowing studies to be drawn from multiple underlying populations, and therefore attempting to estimate the range of effect sizes that may exist across these populations. We believe that the assumptions of our analysis align best with those of fixed effects meta-analysis. Sixth, by excluding treatment studies to ensure greater comparability of included studies we have selected for studies where researchers have less control over sample size, or studies that are more exploratory in nature. However, given the range of sample sizes observed in studies within domains, we do not think the first possibility is likely. Also, included studies typically reported inferential statistics, and therefore were (ostensibly at least) hypothesis-testing, rather than exploratory. Seventh, we did not register a protocol for this review prior to conducting the study. However, the largely descriptive nature of our results means, in our opinion, that the scope for analytical flexibility is limited.

There are a number of ways in which research practices might be improved, taking into account the results we report here. First, sample size considerations should feature prominently in the design of studies, using clearer a priori estimates of likely effect size that are either based on similar studies in the literature, or considerations of what would be considered clinically or biologically meaningful. The pre-registration of study protocols may help to make these assumptions more transparent, and correctly place an emphasis on statistical power as a pre-study or design consideration[19]. Second, researchers should consider alternative ways of improving statistical power. Here we have focused on sample size, but this will not always be feasible or cost-effective; it may be more appropriate to increase the precision of outcome measures by reducing measurement error (e.g. by collecting more data on each participant). Third, other statistical frameworks or approaches exist that do not rely on the concept of statistical power. While NHST remains common in biomedical science, there are increasing calls to instead use estimation approaches, which focus on estimating the likely effect size and quantifying the precision of this estimate (e.g. using a confidence interval)[20]. This has the particular advantage of not requiring the use of arbitrary thresholds to determine the inferences one should draw from one's data[17]. Another alternative is to use Bayesian methods, which have the particular advantage of providing the probability of a specific hypothesis, given the observed data (rather than the probability of having obtained the observed data, given the null hypothesis). Not only is this potentially more informative, but it also allows evidence for the null hypothesis to be quantified (unlike in NHST, where the interpretation of a non-significant p-value can be difficult). In our view, studies should be designed to be informative. Exactly what this means will depend on the approach adopted (NHST, estimation, Bayesian, etc.), and the assumptions that underpin that approach, but certainly within the NHST framework statistical power is a critical design consideration when planning a study. A large body of very small studies providing inconclusive evidence is problematic.

5. Conclusion

Our results indicate that low statistical power may be a problem across the biomedical sciences—we find no clear evidence that the average statistical power differs across research into psychiatric, neurological or somatic diseases, but we do find evidence that research methodology may play an important role. This warrants further investigation. The role of current incentive structures in shaping the behaviour of individual scientists in a manner that serves to reduce statistical power also warrants further investigation. If research funders wish to support high-quality research, they should consider funding larger studies, which in turn may entail collaboration across multiple research groups. However, even in the absence of substantial grant funding, researchers can begin to shift research culture towards more powerful research through research training and wider collaboration[21].

Supplementary Material

Supplementary Material

Supplementary Material

Data accessibility

The data underlying the analyses reported in this study are included as the electronic supplementary material.

Authors' contributions

E.D.-M., K.S.B., T.B., F.G. and M.R.M. conceived the study. E.D.-M. and F.G. extracted the data, and K.S.B. checked the data. E.D.-M. and K.S.B. conducted the analyses. All authors contributed to the interpretation of the results and drafting of the manuscript.

Competing interests

The authors have no competing interests.

Funding

M.R.M. is a member of the UK Centre for Tobacco and Alcohol Studies, a UKCRC Public Health Research: Centre of Excellence. Funding from British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council and the National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged. K.S.B. was funded by the National Institute for Health Research School for Primary Care Research (NIHR SPCR). The NIHR SPCR is a partnership between the Universities of Birmingham, Bristol, Keele, Manchester, Nottingham, Oxford, Southampton and University College London. E.D.-M. was funded by a postdoctoral fellowship from the Region Aquitaine (C2SM research programme). This study was supported, in part, by the Medical Research Council and the University of Bristol (MC_UU_12013/6).

Disclaimer

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health of the United Kingdom.

References