Registered report is a relatively new article type that involves a 2-stage submission process:
- Stage 1, pre-data collection, initial manuscript submission and review
- Stage 2, after study completion, full manuscript is submitted for re-review
The main benefits of this two-stage approach are:
- Once the methods and proposed analyses are provisionally accepted in Stage 1, the journal will commit to publishing the results regardless of the outcome, provided the final study conforms to the initially approved proposal and meets all quality checks. This means that publication bias is reduced as negative results will not prevent publication.
- Peer review of the research proposal provides an opportunity for the authors to receive constructive critical feedback that may help them to fine-tune the study design prior to conducting the experiment.
- This process can help reduce researcher bias.
- This process may enhance the credibility of the work.
Stage 0: Initial Proposal
Authors are strongly encouraged to submit an initial proposal prior to submitting their stage 1 registered report. This proposal will be shared with the senior editors for registered reports for initial assessment. While optional, this extra step is beneficial for authors because it helps ensure the planned study is appropriate for the registered report format and that reviewers are available to expedite the peer review process upon submission of the Stage 1 manuscript.
The initial proposal should contain:
- A brief description of the planned study – background, research question, basic proposed methodology (1 paragraph).
- A justification for why a Registered Report (RR) is preferred over traditional submission types for the proposed study (1-3 sentences). Authors are encouraged to refer to the likely replication value of the research. High-value replication studies are welcomed in addition to novel studies.
- Sufficient detail to evaluate whether the proposed study is likely to be of high quality and appropriate for the RR format. Authors may want to consult the Stage 1 Development Checklist [INSERT LINK] and upload a copy of the completed design planner as part of their initial proposal.
- Additional details, including a full list of potential authors, the estimated date when the stage 1 manuscript will be ready to be submitted, and the target date that data collections is expected to begin.
After receipt of the Stage 0 proposal, the senior editor writes back to invite the author to submit a Stage 1 manuscript, or to recommend the traditional publication route. Decision-making is based on whether the authors make a good case for the study to be conducted as a RR and whether the proposed manuscript is a good fit with the mission and scope of the journal. No commitments are made at this stage.
Stage 1: Initial Manuscript Submission and Review
At stage 1, the author submits
- Cover letter
- Preliminary data/studies
The Stage 1 cover letter should include:
- A brief scientific case for consideration. Authors are encouraged to refer to the likely replication value of the research. Replication studies are welcome in addition to novel studies.
- An anticipated timeline for completing the study if the initial submission is accepted.
During Stage 1 submission, authors will be asked to affirm the following:
- All necessary support (e.g. funding, facilities) and approvals (e.g. ethics) are in place for the proposed research. Note that manuscripts will be generally considered only for studies that are able to commence immediately; however, authors with alternative plans are encouraged to contact the journal office for advice.
- The authors agree to share their raw data, any digital study materials, and analysis code as appropriate.
- Following Stage 1 in principle acceptance, the authors agree to register their approved protocol on the Open Science Framework (https://osf.io/) or other recognized repository, either publicly or under private embargo until submission of the Stage 2 manuscript.
- If the authors later withdraw their paper, they agree to the Journal of Speech, Language, and Hearing Research publishing a short summary of the pre-registered study under a section “Withdrawn Registrations.”
Following initial screening, the Stage 1 submission will be sent for in-depth peer review. In considering papers at the registration stage, reviewers will be asked to assess:
- The importance of the research question(s).
- The logic, rationale, and plausibility of the proposed hypotheses.
- The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
- Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.
- Whether the authors have pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks.
After Stage 1 peer review, manuscripts may be offered in principle acceptance (IPA). Following IPA, the authors will then proceed to conduct the study, adhering exactly to the peer-reviewed procedures. Once the study is complete, authors prepare and resubmit their manuscript for re-review (Stage 2). The review process is illustrated in the flow chart below.
Authors are reminded that any deviation from the stated experimental procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2. In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the editorial board immediately for advice, and prior to the completion of data collection.
Stage 2: Full Manuscript Review
In stage 2, the author submits the completed manuscript, including results and discussion, which is then sent for an expedited peer review. Please note that no substantive changes can be made to the introduction and methods submitted in Stage 1. Pending quality checks, the journal will commit to publishing the results.
The resubmission will most likely be considered by the same reviewers as in Stage 1, but could also be assessed by new reviewers. In considering papers at Stage 2, reviewers will be asked to decide:
- Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls)
- Whether the Introduction, rationale and stated hypotheses are the same as the approved Stage 1 submission (required)
- Whether the authors adhered precisely to the registered experimental procedures
- Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative
- Whether the authors’ conclusions are justified given the data
Reviewers are informed that editorial decisions will not be based on the perceived importance, novelty or conclusiveness of the results. Thus, while reviewers are free to enter such comments on the record, they will not influence editorial decisions. Reviewers at Stage 2 may suggest that authors report additional post hoc tests on their data; however, authors are not obliged to do so unless such tests are necessary to satisfy one or more of the Stage 2 review criteria.
Once accepted at this stage, the full manuscript is published.
More details about Registered Reports is available from the Center for Open Science at https://www.cos.io/initiatives/registered-reports
The Review Process for Registered Reports
Registered Report Q&A
In Stage 1, reviewers evaluate the importance of the research question and quality of the proposed design. At the conclusion of this stage, the editor may reject the submission, request revisions that may be sent back to reviewers, or accept the submission “in principle.”
If a paper is accepted in principle, the author(s) are asked to register the report on a registry (e.g., ClinicalTrials.gov, OSF) and proceed to conduct the study, data analysis, and write up the results.
In Stage 2, researchers resubmit completed papers that have been accepted in principle after the proposed research has been conducted. Reviewers are asked to evaluate whether the researchers carried out the research in the way they proposed in Stage 1 or adequately justified any divergences from the pre-analysis plan. ASHA ojurnals will make every effort to recruit the same reviewers from Stage 1, but if reviewers are unavailable, we may recruit a new reviewer and provide this person with all of the details from Stage 1. Submissions that clear this stage are accepted for publication and subjected to the same publication process as other submissions (e.g., checks for reproducible results, etc.)
Like a traditional paper, the Stage 1 review process could take 3-6 months and more if extensive revisions are requested, and author(s) should account for the review process when planning their study timeline.
In Stage 1, reviewers are asked to review the registered report along five dimensions:
- Importance of the research question
- Soundness of proposed hypothesis
- Soundness and feasibility of proposed design (including statistical power)
- Whether proposed experiment offers an adequate and appropriate test of hypothesis
- Whether proposed methods are appropriate and sufficiently detailed
Like all manuscripts submitted to ASHA journals, the editors have the option of immediately rejecting the registered report if they believe that it is fundamentally lacking on any of the five dimensions. Registered report sent out for peer review will be considered in a similar way. The editor may conditionally accept the registered report, ask for revisions, or reject it. Editors are encouraged to set a high bar for registered reports. The proposed revisions should be doable and straightforward as well as promise to be reasonably successful at addressing reviewers’ concerns.
Registered reports that are accepted-in-principle are subject to a second stage review where, to the extent possible, the original reviewers are re-contacted and asked to assess the manuscript along four dimensions listed below.
- Research question and rationale for hypothesis did not change from Stage 1.
- Experimental procedures detailed in the registered report were followed closely and any departures are noted and justified.
- Unregistered post hoc analyses are clearly labeled, justified, methodologically sound, and informative.
- Conclusions are justified by the data. (Considerations of data quality fall here).
Reviewers are asked not to consider the perceived importance, novelty, or clarity of the empirical results.
If the reviewers and editor conclude that the manuscript did not adhere to the registered report, the manuscript will be either rejected or, in the case that the editor deems a revision to be eminently doable, given a chance to make revisions.
Accepted registered reports must meet guidelines for publishing other manuscripts in the ASHA journals.
A registered report only differs from a traditional research paper in that the proposed experiment has not yet been conducted. Otherwise, a registered report should look very much like a traditional research paper.
The suggested maximum page limit for registered reports is 40 manuscript pages. Authors are encouraged to use an online appendix as a way to provide details that reviewers need to evaluate proposed experimental designs and supplemental materials for , as applicable.
No. The proposed experiment(s) can only be conducted after the Stage 1 review process is complete and the proposed design reflects the reviewers’ comments.
Yes. Previous research (registered or not) can be incorporated in a registered report in the same way it would be in a traditional paper — as a preliminary study in the main text or pilot study in the appendix. It is important that authors make clear in the text which results were registered and which were not.
Yes, but this should be made clear in the main text and an anonymized version of registered pre-analysis plan for the larger project should be provided in the online appendix, so that reviewers can consider the hypotheses to be tested in the registered report in the broader context of other proposed tests in the larger pre-analysis plan.
As detailed as possible. Please use an appendix as a way to provide full details of experimental conditions, measures, and the like. Report everything that you would in a traditional paper, including power analyses, experimental stimuli, question wording, details about measurement (e.g., recoding, factor analysis, etc.), rationales for excluding data, details about statistical models, thresholds for inferential statistics, etc.
As is the case with a traditional paper, the burden is on the author(s) to organize the information in a way that is clear and accessible to reviewers.
If my registered report is rejected, but I run the study anyway, can I submit it later as a traditional research article that features a registered analysis plan?
Yes. We recommend that the author(s) explain in the cover letter how the pre-analysis plan (and thus study) was modified in light of reviews from Stage 1.
One year is typically suggested, but researcher(s) can request a longer schedule in their Stage 1 submission cover letter when providing their anticipated timeline for completing the study.
Experiments must be registered before conducting the study on an established and recognized public registry for pre-analysis plans (e.g., ClinicalTrials.gov, OSF, etc.).
What if I need to make unforeseen adjustments to experiments in a registered report once I am in the field?
You will need to note and justify in the main text of the registered report any divergences from the pre-analysis plan that was accepted in principle.
If unforeseen circumstances cause the authors to deviate from the design and protocol detailed in the accepted-in-principle registered report, authors are encouraged to reach out to the editorial office for guidance. Senior Editors have the option of consulting with the first stage reviewers.
That is fine! The strength of your paper should rest on the importance of the research question and quality of the research design (assessed in Stage 1 of the review process), not the perceived novelty of the results.
Can I conduct analyses that I did not register in the first stage of the registered report review process?
It is best to register all of the planned analyses. Nonetheless, if an additional analysis only becomes apparent after the experiment is conducted, it is permissible to do additional post-hoc analyses, as long as you offer a defensible substantive or statistical rationale for the additional analysis and flag these as post hoc when writing up the data analysis. One of the strengths of registered research is that it makes transparent the distinction between confirmatory and exploratory analyses.
You may register contingencies in your pre-analysis plan (e.g., if A result is obtained, B method will be used).
Any divergences from the proposed pre-analysis plans that weaken the research design (e.g., reduce its power, vitiate the causal identification strategy, or alters the estimand) will likely lead to a negative outcome.
The Open Science Framework offers a number of resources, including checklists, for registered reports: https://cos.io/prereg/
Nosek, et al. (https://www.pnas.org/content/115/11/2600) offer helpful reflections on challenges to registered work in practice.
Declare Design help researchers formalize their research design and simulate data (https://declaredesign.org/).