Guidelines for Article Preparation for Submission
Preparing a Registered Report
The publication and review process for Registered Reports is divided into two stages. In Stage 1, reviewers assess published Study Protocols before data is collected. In Stage 2, reviewers consider the full published study as a Research Article, including results and interpretation. These articles are denoted by a Registered Report badge. Please also refer to Gates Open Research’s editorial policies.
Criteria
Articles submitted to Gates Open Research will first be published on its associated preprint server, VeriXiv. The articles will also undergo peer review on VeriXiv and only once they have passed peer review will the version of record article be published on Gates Open Research and indexed accordingly.
Stage 1: Study Protocol submission, publication, and peer review
The cornerstone of the Gates Open Research Registered Reports format is that a significant part of the manuscript will be published and peer reviewed prior to data collection (Stage 1). Initial Stage 1 submissions should take the format of a Study Protocol and include a description of the key research question and background literature, hypotheses, experimental procedures, analysis pipeline, a statistical power analysis (or Bayesian equivalent), and pilot data (where applicable). We also welcome protocols that propose secondary analyses of existing datasets (see Secondary registrations). All protocols for randomised clinical trials must follow the SPIRIT guidelines; ethical approval for the study must have been already granted. Protocols for systematic reviews should be registered and must follow the PRISMA-P guidelines. In addition to the requirements for a Study Protocol, Stage 1 submissions should include:
1. Title
The title must begin with Stage 1 Registered Report.
2. Introduction
The introduction should provide a review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. Please note that once peer review is complete, the Introduction section cannot be altered apart from correction of factual errors and typographic errors and altering of tense from future to past for Stage 2.
3. Methods
The methods must include a full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g. outlier extraction). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.
All experimental procedures must be described in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly in the subsequent experiments or both the Stage 1 and Stage 2 manuscripts can lose their Registered Report status.
Please also include the proposed analysis pipeline, including all pre-processing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Only pre-planned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned exploratory analyses will be admissible in a separate section of the Results.
Studies involving Neyman-Pearson inference must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature or theory. Since publication bias overinflates published estimates of effect size, power analysis must be based on the lowest available or meaningful estimate of the effect size. For frequentist analysis plans, the a priori power must be 0.9 or higher for all proposed hypothesis tests. In the case of highly uncertain effect sizes, a variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined.
Methods involving Bayesian hypothesis testing are encouraged. For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified. For example, will you use a uniform up to some specified maximum, or a normal/half-normal to represent a likely effect size, or a JZS/Cauchy with a specified scaling constant? For inference by Bayes factors, authors must be able to guarantee data collection until the Bayes factor is at least 6 times in favour of the experimental hypothesis over the null hypothesis (or vice versa). Authors with resource limitations are permitted to specify a maximum feasible sample size at which data collection must cease regardless of the Bayes factor; however to be eligible for advance acceptance this number must be sufficiently large that inconclusive results at this sample size would nevertheless be an important message for the field. For further advice on Bayes factors or Bayesian sampling methods, prospective authors are encouraged to read this key article by Schönbrodt and Wagenmakers.
- Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.
- Timeline for completion of the study and proposed submission date if Stage 1 peer review is successful. Extensions to this deadline can be negotiated with the editorial team.
- Any description of prospective methods or analysis plans should be written in future tense.
4. Data Availability
All articles must include a Data Availability statement, even where there is no data associated with the article - see our data guidelines and policies for more information. Pilot data files should be appropriately time stamped. Other than pilot data published and approved as part of the Stage 1 protocol, no data acquired prior to the date of publication of the Study Protocol is admissible in the Stage 2 submission. Raw data must be accompanied by guidance notes, where required, to assist other scientists in replicating the analysis pipeline.
5. Peer Review
Initial Stage 1 submissions will undergo a rapid initial check by the in-house editorial team before being published with the status Awaiting Peer Review and labelled with a Registered Report badge. Expert reviewers are invited on the authors’ behalf. Reviewers are asked to consider the following questions:
- Is the rationale for, and objectives of, the study clearly described?
- Is the study design appropriate for the research question (including statistical power analysis, where appropriate)?
- Are sufficient details of the methods provided to allow replication by others?
- Have the authors pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks?
- Are the datasets clearly presented in a useable and accessible format?
The peer review is entirely transparent: The reviewers’ names and affiliations, their peer review report and the approval status they choose are published alongside the article. Peer review reports are posted as soon as they are received and the peer review status of the article is updated with every published report.
Publications that receive Approved and/or Approved with Reservations reports will retain the Registered Report badge; authors will be encouraged to revise the paper to address any concerns raised by the reviewers. Once the reviewers and authors are in agreement that the methods and proposed analyses are adequate to proceed with data collection, the Stage 1 Study Protocol will be frozen and no further changes will be allowed (for exceptions, please see Incremental registrations). The authors will then proceed to conduct the study, adhering exactly to the peer-reviewed protocol. When the study is complete the authors will submit their finalised Research Article for publication and peer review (Stage 2).
Publications that receive a Not Approved report will lose the Registered Report badge permanently and Registered Report will be removed from the title of the paper. Following this, the protocol will be treated as any other Study Protocol on Gates Open Research; it will remain published and the authors are welcome to revise.
Stage 2: Research Article submission, publication, and peer review
Once the study is complete, authors may submit their full manuscript for publication and peer review. The manuscript should take the format of a Research Article with the following additions:
1. Title
The title should begin with Stage 2 Registered Report.
2. Abstract
The abstract must include a link to the approved Stage 1 protocol on Gates Open Research.
3. Background, Rationale and Methods
Apart from minor stylistic revisions, the Introduction cannot be altered from the approved Stage 1 protocol, and the stated hypotheses cannot be amended or appended. At Stage 2, any description of the rationale or proposed methodology that was written in future tense in the Stage 1 publication should be changed to past tense. Any textual changes to the Introduction or Methods (e.g. correction of typographic errors) must be clearly marked in the Stage 2 submission. Any relevant literature that appeared following the date of publication of the Stage 1 protocol should be covered in the Discussion.
4. Results & Discussion
The outcome of all analyses outlined in the Stage 1 Study Protocol must be reported in the manuscript, except in rare instances where an approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, reviewers must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but omitted with justification from the Results.
It is reasonable that authors may wish to include additional analyses that were not included in the Study Protocol. For instance, a new analytic approach might become available between publication of the Study Protocol and Stage 2, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc analyses.
Authors reporting null hypothesis significance tests are required to report exact p values and effect sizes for all inferential analyses.
5. Data Availability
All articles must include a Data Availability statement, even where there is no data associated with the article - see our data guidelines and policies for more information. Data files should be appropriately time stamped to show that data was collected after peer review of the Study Protocol. Other than pilot data published and approved as part of the Stage 1 protocol, no data acquired prior to the date of publication of the Study Protocol is admissible in the Stage 2 submission. Raw data must be accompanied by guidance notes, where required, to assist other scientists in replicating the analysis pipeline.
6. Peer Review
Initial Stage 2 submissions will undergo a rapid initial check by the in-house editorial team before being published with the status Awaiting Peer Review, labelled with the Registered Report badge, and linked to the Stage 1 Study Protocol. Expert reviewers are invited on the authors’ behalf and will most likely be the same reviewers as in Stage 1, but could also be new reviewers. Reviewers will be asked:
- Are the data able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls)?
- Are the introduction, rationale and stated hypotheses the same as the approved Stage 1 submission? (required)
- Did the authors adhere precisely to the registered experimental procedures? If not, has an explanation been provided regarding any change?
- Are any unregistered post hoc analyses added by the authors justified, methodologically sound and informative?
- Is the work clearly and accurately presented and does it cite the current literature?
- Is the study design appropriate and does the work have academic merit?
- Are sufficient details of methods and analysis provided to allow replication by others?
- If applicable, is the statistical analysis and its interpretation appropriate?
- Are all the source data underlying the results available to ensure full reproducibility?
- Are the conclusions drawn adequately supported by the results?
The peer review is entirely transparent: The reviewers’ names and affiliations, their peer review report and the approval status they choose are published alongside the article. Peer review reports are posted as soon as they are received and the peer review status of the article is updated with every published report.
Publications that receive ‘Approved’ and/or ‘Approved with Reservations’ reports will retain the Registered Report badge; authors will be encouraged to revise the paper to address any concerns raised by the reviewers.
Publications that receive a Not Approved report will lose the Registered Report badge permanently and Registered Report will be removed from the title of the paper. The Registered Report status of the Stage 1 Study Protocol will also be withdrawn (see Withdrawn registrations). Following this, both the Stage 1 Study Protocol and the Stage 2 Research Article will be treated as any other article on Gates Open Research; each will remain published and the authors are welcome to revise.
Authors are reminded that any deviation from the published protocol, regardless of how minor it may seem to the authors, could lead to removal of the Registered Report badge from both the Stage 1 and Stage 2 publications. In cases where the Stage 1 protocol is altered after peer review completion due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the editorial director immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted per editorial discretion. If the authors wish to alter the experimental procedures more substantially following Stage 1 peer review but still wish to publish their article as a Registered Report then the manuscript must be revised and re-reviewed as a Stage 1 submission (see also Incremental registrations). Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in the final Research Article.
Secondary registrations
Gates Open Research welcomes Stage 1 Study Protocols proposing secondary analyses of existing datasets, provided authors can supply sufficient evidence (e.g. self-certification; letter from independent gatekeeper) to confirm that they have had no prior access to the data in question.
Incremental registrations
Authors have the option to add experiments to published and peer reviewed Stage 1 Protocols. This option may be particularly appropriate where an initial experiment reveals a major serendipitous finding that warrants follow-up within the same paper. In such cases, the authors will be able to propose additional experiments for Stage 1 consideration using our versioning system; these experiments must extend the approved protocol as opposed to being part of a new submission. The revised publication will then undergo peer review. In cases where a Stage 1 incremented registration is rejected, authors will retain the option of publishing the most recently approved version of the manuscript. For further advice on specific scenarios for incremental registration, authors are invited to contact the editorial team.
Withdrawn registrations
It is possible that authors who publish a Stage 1 Study Protocol may wish to withdraw their Registered Report status following or during data collection. Possible reasons could include a major technical error, an inability to complete the study due to other unforeseen circumstances, or the desire to submit the results to a different journal. In all such cases, the Registered Report status can of course be withdrawn at the authors’ discretion. When the Registered Report status is withdrawn, all associated publications will lose the Registered Report badge permanently and Registered Report will be removed from the title(s). In addition, removal of the Registered Report badge will be accompanied by a brief note from Gates Open Research publicly explaining the reason for removal (for example: submission to another journal, substantial changes to methodology, unforeseen methodological/technical difficulties).