Registration: Design and Mock Report 03/2011
The importance of registering studies is now broadly recognized among researchers working on the political economy of development. But registrations remain rare in practice.
Registration can help reduce various types of biases (research biases, reporting biases, publication biases) that can affect empirical research. Perhaps the most important of these are the biases that can arise when analysis plans are developed after examining actual outcome data. The basic problem is that, deliberately or not, attention can easily get refocused on areas in which results are strongest. This leads to an overall false evaluation of the evidence.
To avoid these biases in this study we specified the primary hypotheses in 2007 prior to the launch of the DRC program. In addition we developed and circulated (on 7 March 2011 to members of the EGAP network) a detailed analysis plan prior to completion of data collection. This analysis plan includes a complete “mock report” in which we undertake the core analysis using “scrambled” data–doing so forces us to lay out our exact tests and specifications without being influenced by results. Both of these documents are published on this page below.
- DESIGN DOCUMENT (ODS): Our Outcomes and Data Sources (4 March 2011) document lists our hypotheses and also provides links for every hypotheses to the survey or behavioral item used to measure them. ODS also describes how we plan to analyze results, what controls we will be using and so on.
- MOCK REPORT: Our Mock Report (4 March 2011) report runs our analysis for a core set of primary measures to test a set of 15 primary hypotheses. This is what our main evaluation report will look like. This report was constructed using real data collected so far but with data on the treatment variable randomized prior to analysis to force us to go through the whole process of developing analysis code and selecting measures etc prior to actually looking at the results. (We got this nearly right but sadly not fully right; for a small set of measures—marked with a cross in ODS and highlighted in the Mock Report—the first tables produced actually used real though of course very incomplete data).
Further relevant registration information (responses to questions from proposed EGAP registration standard protocol):
- Has this research received Institutional Review Board (IRB) or ethics committee approval? This study received IRB approval: Columbia IRB PRotocol IRB-AAAC5860 (Approved: 08/04/2007) & IRB-AAAF4550 Protocol Overview (Approved 09/12/2010; last modified 02/01/2011).
- Was a power analysis conducted prior to data collection? Power analyses were conducted prior to launch and included in the initial design document.
- Will the intervention be implemented by the researcher or a third party? If a third party, please provide the name. Data collection is implemented by the Universities of Bukavu and Lubumbashi under contract with the IRC. Both Columbia and IRC are overseeing teh quality of data collection.
- Did any of the research team receive remuneration from the implementing agency for taking part in this research? No.
- If relevant, is there an advance agreement with the implementation group that all results can be published? Yes. Moreover for the purpose of the evaluation the implementing agency (IRC) has approved the analysis plan without any access of any form to outcome data.