Why should you plan your statistical plan prior to collecting data?
Humans are prone to cognitive biases such as confirmation bias (seeking information that supports existing beliefs) and hindsight bias (believing outcomes were predictable after the fact). In research, these biases can distort findings, especially when researchers make analytic decisions after seeing results. Although statistical testing typically accepts a 5% false positive rate, “researcher degrees of freedom” — choices about data collection, exclusions, transformations, sample size, covariates, etc. — can dramatically inflate false positives when decisions are made post hoc. Practices like increasing sample size until reaching statistical significance, selectively removing outliers, or trying multiple analytic strategies increase the likelihood of false-positive results. See how easy it is to find false “significant” results by using our p-hacking tool.
The core problem is that analyses guided by observed outcomes allow biases to influence decisions, making many reported effects unreliable. A key remedy is transparency and preregistration.
Benefits of preregistration
Preregistration, that is, specifying hypotheses, methods, and analysis plans before data collection or analysis, limits bias in confirmatory testing while still allowing exploratory analyses, clearly distinguishing robust hypothesis tests from hypothesis-generating work. This improves credibility, limits false positives, and often leads to better study design through early methodological feedback.
Preregistration can be beneficial for various type of studies, including:
- experimental studies (i.e. studies with a manipulated variable): it will define what will be your confirmatory analysis and strengthens your claim
- observational or exploratory studies: it will help you move along the exploratory-confirmatory continuum
- qualitative studies: it will provide a way to document e.g. your positionality towards a subject in the course of a project.
What is included in a preregistration?
Several preregistration templates exist. While the standard Open Science Framework (OSF) preregistration template is most commonly used, some are tailored for specific field or specific methods (e.g. systematic review, qualitative work, secondary data analysis).
Your preregistration will define your study’s:
- Hypothesis and predictions
- Data collection procedures
- Sample size and stopping rule
- Variables (manipulated, measured, indices)
- Statistical method (model, dependent and independent variables, covariables, transformations)
- Data exclusion criteria
- How to deal with missing data
A great tool to create your statistical plan, especially for early career researchers still learning statistics and needing feedback from supervisors, collaborators, or statisticians on their design, is to simulate data, and write the possible statistical tests to analyze that data (see 1.4.2. Simulation of data and 1.4.3. Power analyses). Including an analyses script (developed on simulated data) with your preregistration is optional but recommended.
To get support with planning your analytical approach, you can book a consultation with the LMU statistical consulting unit (StaBLab).
Publishing process
Once your study plan is finalized:
- Submit your preregistration before collecting new or analyzing existing data. You can do so on discipline specific registries (see 1.1.2. Preregistrations) or discipline agnostic repositories such as the OSF.
- Embargo your plan if you are concerned about scooping. On the OSF, your preregistration can be kept private for a predetermined amount of time, and for a maximum of 4 years.
- Include your preregistration’s DOI in your manuscript. Make your registration public upon the publication of your manuscript.
Creating a preregistration improves transparency and allows for valuable early feedback from collaborators. An even stronger approach is submitting preregistrations directly to journals (then called “Registered Reports”), enabling peer review at a stage where methodological adjustments are still possible.
Registered Reports
Registered Reports are a publication format, now adopted by over 300 journals (see participating journals), where preregistrations are peer-reviewed before data collection. Reviewers evaluate the hypotheses, methods, and planned analyses, allowing methodological improvements. If the plan is approved, the journal grants in-principle acceptance, meaning publication is guaranteed provided researchers follow the protocol.
After completing the study, authors add results and discussion sections, clearly separating preregistered confirmatory analyses from exploratory ones. Final review focuses on adherence to the approved plan and the validity of conclusions, not on whether results are significant. This model shifts incentives toward asking important questions and using rigorous methods rather than chasing striking or ‘novel’ outcomes.
LEARN MORE
TOOLS & RESOURCES
In our context, a computer simulation is the generation of artificial data to build up an understanding of real data and the statistical models we use to analyze them. You can simulate data to:
Test your statistical intuition or demonstrate mathematical properties you cannot easily anticipate.
Example: Check whether there are more than 5% significant effects (assuming \(\alpha = .05\)) when random data from \(H_0\) are generated.
Understand sampling theory and probability distributions or test whether you understand the underlying processes of your system.
Example: See whether simulated data drawn from specific distributions is comparable to real data.
Perform power analyses.
Example: Assess whether the sample size (within a simulation repetition) is high enough to detect a simulated effect in more than 80% of the cases. (see 1.4.3. Power analyses)
Prepare a pre-analysis plan.
Example: To strengthen your planned confirmatory analyses before collecting data, consider sharing a simulated dataset with a statistician or mentor. This allows for specific feedback on suitable statistical tests. The resulting analysis code can accompany your preregistration or registered report (see 1.4.1. Pre-analysis planning) so reviewers can clearly see your intended approach. When real data are collected, they can be directly substituted into the code to generate results.
Generating an artificial dataset in R (see our simulation tutorial) is much easier than you might think and is often very helpful, even when you need to make assumptions about variable distribution or when the parameter space is not well known.
Power analysis is relevant whether you are designing a project from scratch or running an analysis on already existing data. There are two main types of power analyses:
A priori power analysis
Simulate data to calculate the smallest sample size required to detect the smallest effect of interest. See our advanced power analyses tutorial using R.
For a very basic power calculation, you can use simple R functions if you know 3 out of 4 of these parameters:
- required sample size n (usually the one missing)
- desired power 1 - β (default 0.80)
- the alpha level α (default 0.05)
- the expected effect size (has to be estimated or extracted from the literature on the form of d, f, etc.)
To get support with pre-analysis planning, you can book a consultation with the LMU statistical consulting unit (StaBLab).
Post-hoc power analysis
Compute a post-hoc power when you are not be able to control the sample size for your project. Beware: This power computation comes in two flavors - one is legitimate, and one is flawed and not defensible.
The legitimate post-hoc power is computed with your actual n, and the same effect size that you plugged into your a-priori power analysis. This analysis gives you the achieved power to detect your assumed effect.
The flawed version of post-hoc power is called “observed power”: If an analysis yields a non-significant result, some researchers calculate the post-hoc power, but plug in the observed effect size. “Observed power”, however, is just a one‑to‑one function of the p‑value (a non-significant p-value returns a low power < 50 %, a just significant p‑value of .05 always yields a power of exactly 50%). Observed power adds no new information to the p‑value and is essentially meaningless. Do not compute this type of post-hoc power!
