Computational reproducibility, replicability, and transparent reporting

This page adopts the National Academies' 2019 convention. The terms reproducibility and replicability are used differently across fields, so the convention is stated explicitly here.

Computational reproducibility means obtaining consistent results using the same input data, computational steps, methods and code, and conditions of analysis. Input data are the records supplied to the analysis after the stated curation steps. Analysis conditions include the computational environment when it can affect the result.

Replicability means obtaining consistent results across studies aimed at answering the same scientific question, each using its own newly obtained data. A second study is a replication attempt; new data alone do not establish that the results are consistent. The comparison must account for relevant measurement or sampling uncertainty and for material differences in methods. "Each study's own data" identifies study-specific observations; it does not assert probabilistic independence of the datasets or require different investigators.

Generalizability concerns whether findings apply consistently in other contexts or populations. It is a separate question from reproducing a computation or comparing studies that address the same question.

Transparent reporting identifies the data and code or other computational artifacts where sharing is lawful and ethical, and otherwise states the access restrictions or route. It records the relevant computational environment; data collection, cleaning, grouping, exclusions, and curation; methods; analysis decisions and when they were made; whether analyses were exploratory or confirmatory; and the uncertainty and limitations needed to interpret the results. These are definitions and reporting recommendations, not a theorem guaranteeing correctness or future replication. Reproducing a computation can repeat an error, and careful, correct, transparent work can still fail to replicate.