Reproducible Autism Research (2): Preregistration

My studies normally take me about a year from getting ethical approval to the point where I am ready to start analysing my data. In the past, I’d get my data into SPSS, get a cup of tea, and then be poised to press the analysis buttons, nervously. Make or break time. Will p < .05? In other words, will my results be ‘statistically significant’? If so, perhaps I’d get my paper into a high-impact journal. If not, I’d probably get it published somewhere, but it’d be more difficult.
 
This doesn’t seem right. If the study was well-designed and well-conducted, then the findings should be important regardless of the outcome. Registered reports are a format offered by some journals that get around this issue. The journal reviews your introduction and methods before you collect the data, including your hypotheses and analysis plan, and then once the reviewers are happy, you can go and collect the data knowing that the journal will definitely publish your paper if you stick to your plan. As well as getting useful feedback on your proposed research when it really matters, you also no longer have that nervous make-or-break moment. And you can’t be tempted to conduct multiple analyses until you get the results you want.
 
I’ve not done this yet. I really want to, but so far it has just not worked out. I think that registered reports work best when you have some flexibility in when a study can start – to allow you time in case the reviewers take a while to respond, to allow you to make changes to your protocol and, if necessary, to go back-and-forth with the ethics committee. However, as a developmental researcher, my studies are constrained by school terms. Some studies are school-based, so I need to arrange with a school when I can come to visit children – often far in advance. Other studies are conducted in the lab, which means participants have to come in outside of school hours and so school holidays are the most efficient times to run studies. Therefore, there is often little flexibility in when a study starts. For me, this has been exacerbated recently by periods of leave that can’t be moved (including maternity leave and research visits). Getting a journal to review a study before data collection has just not been feasible. One day, I’ll be more organised (and less pregnant) and this will all come together. But so far, I’ve not managed it.
 
Instead, I have pre-registered my hypotheses and analysis plans for a couple of studies on the Open Science Framework. This way doesn’t guarantee your paper will be accepted by a journal, but it is a far less involved process and still means that you must state your hypotheses and analysis plan up-front. My first experience of this was when my MSc student, Furtuna Tewolde, entered the Open Science Framework’s Pre-registration Challenge with me and co-supervisor Dorothy Bishop. We had to answer a series of questions for our study, including questions about our sample size and analysis plan, and then the Open Science Framework reviewed it for statistical soundness (this only took a day or two). We were then ready to collect our data – and as long as the study was published in a participating journal while the Challenge was still running, the project would be awarded $1000 (and indeed it was – a nice incentive for a graduate student!). I really enjoyed this process, and the analysis and write-up was a breeze. It was definitely the shortest delay I’ve ever had between the end of data collection and acceptance from a journal. I am fairly sure that the process of pre-registration made the review process easier, too.
 
But I hadn’t quite thought through how pre-registration would affect the supervision process. Traditionally, this might involve coding up an experiment, giving the student some reading suggestions, and then having a couple of months break while the student collects the data, before coming back to you to work on the analysis scripts. Here, all of this needed to be done at the start.
 
When we came to analysing the data, we found a couple of annoying inconsistencies in the pre-registration document. For example, there was a point where we said we would exclude trials that had response times “over 3 times the maximum occlusion duration, i.e., 8s”. But the longest occlusion duration for our stimuli was 4s, so our exclusion criterion of 8s was only two times the longest occlusion duration. I panicked – thinking all of our work pre-registering had been wasted. I told Dorothy (who is very wise in these matters), and she said I needn’t panic, and to contact the Open Science Framework. I did, and they told me to just post a document outlining the inconsistencies and to link to this in the published paper. Quite rightly, there is an understanding that sometimes researchers make mistakes.
 
The study I am working on now is a cognitive modelling study, looking at differences in perceptual decision-making between autistic children and typically developing children. For this, it is hard to specify all analysis decisions up-front. Some things depend on the data – for example there may be some participants whose data cannot be well-fit by the cognitive model – in a way that you may not have predicted. We are dealing with this by having a ‘blind modeller’ who will make analysis decisions such as excluding outliers, applying transformations and modelling contaminant processes using a version of the dataset where the information about the group membership of participants (autistic or typically developing) is mixed up, so that all modelling decisions will be made without bias in respect to the hypotheses under test. This was suggested to me by my collaborator, EJ Wagenmakers, who has used this type of blinding before with Gilles Dutilh and others.
 
I think there are many benefits to preregistration, which far outweigh the challenges. Even if you think you can’t preregister your analyses, there may be creative solutions (like using blind-modellers) to overcome these obstacles.