The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network

Moshontz, Hannah, Lorne Campbell, Charles R. Ebersole, Hans IJzerman, Heather L. Urry, Patrick S. Forscher, Jon E. Grahe, et al. 2018. ‘The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network’. Advances in Methods and Practices in Psychological Science 1 (4): 501–15.

preprint available here

« The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions, or attempt to replicate prior research, in large, diverse samples. » (Moshontz et al., 2018, p. 2)

« The initiative quickly grew into a network with over 300 data collection labs, an organized governance structure, and a set of policies for evaluating, preparing, conducting, and disseminating studies. » (Moshontz et al., 2018, p. 3)

« […] the PSA provides an ideal context in which to train early-career psychological scientists, and in which psychological scientists of all career stages can learn about new methodological practices and paradigms. » (Moshontz et al., 2018, p. 18)

Phase 1: Submission & Evaluation

« Proposing authors submit a description of the proposed study background, desired participant characteristics, materials, procedures, hypotheses, effect-size estimates, and data-analysis plan, including an analysis script and simulated data when possible, much like a Stage 1 manuscript submitted under a Registered Reports model. These submissions are then masked and evaluated according to a process overseen by the Study Selection Committee . » (Moshontz et al., 2018, p. 11)

Phase 2: Preparation

« Next, the Methodology and Data Analysis Committee , whose members are selected on the basis of methodological and statistical expertise, evaluates and suggests revisions of the selected studies to help prepare the protocols for implementation. » (Moshontz et al., 2018, p. 12)

« Next, the Logistics Committee identifies specific labs willing and able to run the specific protocols, bundling multiple studies into single laboratory sessions to maximize data collection efficiency when possible. The Logistics Committee then matches data collection labs to projects. Not every network lab participates in every study. » (Moshontz et al., 2018, p. 12‑13)

Phase 3: Implementation

« We expect implementation to be the most time-intensive and variable phase. This process begins with pre-registering the hypotheses and confirmatory or exploratory research questions , the data-collection protocol, and the analysis plan developed in Phase 2, with instructional resources and support provided to the proposing authors as needed by the Project Management Committee. Pre-registration of confirmatory analysis plans, methods, and hypotheses is a minimum requirement of the PSA. The PSA encourages exploratory research and exploratory analyses, as long as these are transparently reported as such.» (Moshontz et al., 2018, p. 13)

Phase 4: Analysis and Dissemination

« The proposing authors will complete confirmatory data analyses, as described in their pre-registration. Once the confirmatory analyses have been conducted, the proposing authors will draft the empirical report . Drafting authors will be encouraged to write the manuscript as a dynamic document , for example using R Markdown. All contributing labs and other authors (e.g., those involved in designing and implementing the project) will be given the opportunity to provide feedback and approve the manuscript with reasonable lead time prior to submission. Following the principle of transparency, the PSA prefers publishing in open-access outlets or as open-access articles.» (Moshontz et al., 2018, p. 14)

« When the project is concluded, all data, analytic code and meta-data will be posted in full and made public on the OSF by default or on another public and stable repository on a case-by-case basis. These data will be available for other researchers to conduct exploratory and planned secondary analyses. Data release will be staged such that a “train” dataset will be publicly released quickly after data collection and preparation, and the remaining “test” dataset will be released later, following a wide and early (e.g., one year out) public announcement (e.g., as in Klein et al., 2018). » (Moshontz et al., 2018, p. 15)

1 Like

Hi Sabrina,

thanks for the fish. It looks like some Psychology research scientists are well organized… I wonder whether such approach would translate/make sense in computer science. Unfortunately, I don’t have access to this document. Is it a proposal or an effective organization ? Are there already studies that have been conducted with the PSA protocol ? Do you know how the committee handle conflicts of interests at the submission phase ?



Hi Arnaud,

by advance, sorry for the clumsy English !

the preprint is available here

and you can email this author to get the accepted version : hmonshontz/at/gmail/dot/com

Yes, psychologists are very well organized ; have a look at this :
really impressive to me …

Indeed, one of the authors is a research at Grenoble University : Hans Ijzerman
his email is on the first page of the preprint