Reproducibility and Repeatability

Reproducibility

Reproducibility is the ability of an entire experiment or study to be duplicated, either by the same researcher or by someone else working independently. Reproducing an experiment is called replicating it. Reproducibility is one of the main principles of the scientific method. (from https://en.wikipedia.org/wiki/Reproducibility)

Repeatability

Repeatability or test–retest reliability is the variation in measurements taken by a single person or instrument on the same item, under the same conditions, and in a short period of time. A less-than-perfect test–retest reliability causes test–retest variability. Such variability can be caused by, for example, intra-individual variability and intra-observer variability. A measurement may be said to be repeatable when this variation is smaller than a pre-determined acceptance criteria. (https://en.wikipedia.org/wiki/Repeatability)

Reproducible Research Design

Reproducible research design is a framework or process for the design of research projects. The goal of reproducible research design is to standardize and regularize the design of research projects. We want to be able to describes and document the tasks that need to be done during the design of a project from the earliest concept to the commencement of the project's operational phase. The project should be designed as completely as possible as early as possible, with a minimum of mistakes and omissions.

  • Capture the questions and decisions that occur during the design.
    • Meeting notes/minutes
    • choices that are made and reasons
    • choices that are not selected and reasons
    • etc.
  • Research question - concise, informative question to be answered. Guides methods, population, etc. Outcomes, alternate outcomes.
    • What do you specifically want to learn about?
    • What do you not know about that you would like to learn?
    • What questions will you try to learn about and their relation to other questions?
  • Rationale - Implications of the results, references, background information
  • Study Design - type of study (randomized clinical trial, prospective cohort study, etc.)
  • Nature of Intervention(s) - drug, counseling program, etc.
  • Ethics -
  • Disclosures
    • sponsors
    • conflicts
    • etc.
  • Institutional review -
  • Study populations - characteristics of study population; inclusion/exclusion criteria
  • Study size or power - total number of study participants needed (in each arm if applicable)
  • Hypothesis - Assumption taken to be true for purposes of investigation. Expected outcome.
  • Feasibility - Ability to answer question fully, resources, barriers, timeline, supplies, scheduling, data collection, analyses, etc.
  • Budget - How much will all this cost?
  • Data - Type of data to be collected (interviews, observation, survey, etc), database schema and design, data privacy protection, blinding protection, how to distribute study data sets.
  • Instruments - Tools needed to measure data (forms, verified tools, machines, etc.)
  • Implementation - Plan for conduct of the study, e.g. hiring, printing, meetings, travel, acquiring study material, infrastructure, disaster recovery, recruitment, miscellaneous supplies, office support, management
  • Risks - risk of failure, risk to institution, risk to subjects, risk to reputation
  • Methods of analysis - statistical methods, types of reports, graphics, etc.
  • Reporting and deliverables - Types of reports, frequency of reports, adverse event reporting, interim reports, final reports, white papers, published reports, web accessibility, web applications, apps

Reproducible Research

The term reproducible research refers to the idea that the ultimate product of academic research is the paper along with the full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research. (from https://en.wikipedia.org/wiki/Reproducibility)

Reproducible Manuscripts and Collaborative Editing

Reproducible Data

Topic revision: r3 - 19 Jul 2017, FrankHarrell
 

This site is powered by FoswikiCopyright © 2013-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Vanderbilt Biostatistics Wiki? Send feedback