Procedures for Scientific Observational Research

For observational research to be fully respected, it must be carried out rigorously. This can be done by copying how experimental research is done, as described by Rubin ¹. One step in operationalizing this approach is to formulate full analysis plans before looking at the data, with the only modifications to the plan resulting from finding too much missing data or unreliability in variables involved, through objective examination [these kinds of examinations can also be done before the analysis plan is written.]. The analysis plan needs to be stored in a central place (e.g., our wiki under the central projects area ) and any changes to it are documented with date and reason.

Some research is highly exploratory. It is generally not sufficient to end an article with a "need to replicate" statement. The exploration procedure should be fully pre-specified, and the biased caused by exploration should be estimated (e.g., using the bootstrap or 100 repeats of 10-fold cross-validation), taking into account (by repeating) all analysis steps for each resample.

Frank's notes from faculty meeting:

• Need to preface with overall goal in terms of ethical conduct of research and a blurb from the ASA code of ethics
• Can lay out other approaches
  • Pre-specify strategy for formulating the final analysis
  • Specify a priority order for analyses and reporting of same
• Get buy-in from senior collaborators