Shokraneh, Farhad
(2020)
Study-based registers of randomised controlled trials: the premise and increasing sophistication of data supply for evidence synthesis.
PhD thesis, University of Nottingham.
Abstract
Extended Abstract
Summary
Although narrative reviews remain important, overviewing literature often now takes some form of systematic approach. Pivotal to being systematic is the searching and, with that, the role of Information Specialists. To offset a common criticism of the time-consuming nature of systematic reviewing the Information Specialist must evolve real-world solutions for highly sensitive and specific searches and efficient supply of complete, valid, and accessible data.
This work describes the five-year evolution of a unique and powerful relational study-based register – of randomised controlled trials (RCTs) (Paper 1). Meta-data from 19,964 RCTs has been extracted and a controlled language created to allow accurate classification and identification of only relevant studies for any given review (Paper 6). This advanced system almost eradicates the need for reviewers of trials to search for themselves, saving the usual waste in review preparation or grant application (Paper4).
The umbrella-term ‘meta-data’ may include complete datasets – randomised trials’ ‘big data’. Although increasing numbers of individual patient datasets (IPD) exist, by far the most common data are the qualitative and quantitative information extracted - by hand or machine - from each study’s set of publications. To be rigorous, this process of data extraction must be possible to verify- with each tiny piece of data being traceable to its source. This should also prevent the continuous repetition of the same data extraction by successive generations of reviewers. Paper 2 describes pioneering work in creating an easy system to make this possible. Paper 3 calls for wide access to publically funded datasets of extracted data from trials and Paper 8 and Paper 9 describe why openness is important for reproducibility and how we could enhance the reproducibility of systematic reviews and make them a role model for other study designs.
Furthermore, a register working at this level of sophistication lends itself to semi-automation of the systematic reviewing process (Paper 4, Paper 5) and novel uses of these data – including increasing the rigour in the methodology of the analyses of systematic reviews (Paper 5).These registers greatly facilitate to new insights into research activity(presented in Paper 7). This paper reports patterns and trends that could support decisions about the future of the register, the process of systematic reviews and the direction of research overall.
This work represents a step-change in sophistication of the role of Information Specialists in systematic reviewing. The investment of effort of the last half-decade results in a database with unparalleled functionality and completeness, with rich research-potential, already relating to reliable, accurate datasets that can be supplied to any person or machine.
The body of work presented in this thesis is a weave of four papers placed within ‘background and developing novel methods’ – although parts of these papers do also report results and conclusions. Those four ‘background’ papers lead to another two articles largely reporting results, and finally three papers focus on ‘conclusions and impact on policy’.
Background and developing novel methods
Paper 1: This introduces the idea of two types of registers to Information Science:
1. Reference-based register - based on the bibliographic data of separate, disconnected, multiple reports of a study; and
2. Study-based register - based on the entire data of one study, including its connected reports, and its associated meta-data and bibliographic details; and, within this
a. Automated study-based register - in which data and meta-data are widely available so that the systematic reviews could start with meta-analysis.
The paper is the first to discuss the necessity, rationale, and steps for the development, utilization and maintenance of study-based registers as well as the challenges and gains for organizations supporting systematic reviews. Finally, the paper presents an example of structured data in machine-readable XML and human-friendly tabular format encouraging sharing of data, meta-data and the locations of extracted and tabulated data in the original reports.
Paper 2: This follows the arguments from paper one and describes three methods of locating data in the original reports. The paper, for the first time, compares the advantages and disadvantages of each method. The paper develops the argument to describe the practicalities of how actual tabular data records - including meta-data and the exact location of every small piece of qualitative and quantitative data - were created (work supported by HTA NIHR Programme grant HTA-14/27/02). Paper 2 ends with a call for open access sharing of this type of research data.
Paper 4: This describes use of a sophisticated trials register with a particular focus on saving time/effort/money. This describes and quantifies – including though a flow diagram - the processes of how tasks that usually take months to complete can be undertaken [better] in minutes through use of a well-constructed and maintained study-based register. The paper discusses – and tries to quantify - the avoidable waste in the process of systematic reviewing and a radical approach to study search and screening.
Paper 5: This describes use of a sophisticated trials register with a particular focus on novel analysis and easy-to-use quantifiable means of increasing methodological rigour in network meta-analyses. High-grade registers are used not only to identify all relevant studies but also all relevant comparisons within those studies. This work presents, for the first time, a simple mathematical formula that accurately predicts the number of potential comparisons within a single RCT or, more importantly, a network meta-analysis. For example, a single trial with two interventions generates one comparison; a three-arm trial –three; and an eight-arm trial no less than 28. Within the increasingly prevalent network meta-analyses, many arms exist for potential indirect comparisons and the tested formula accurately enumerates this number. Those embarking on a network meta-analysis can pre-state which potential comparisons are of interest rather than doing this post hoc. Where a shortfall in the number of comparisons actually utilised or reported occurs - this is a considerable opportunity for the inclusion of bias, that can be, at least partially guarded against by use of the pre hoc simple formula.
Results
Paper 6: This documents the detailed, classification of all pharmacological interventions used in all schizophrenia RCTs. Data relating to interventions extracted from 19,964 RCTs were, for the first time, carefully categorised using a [necessarily] novel controlled language derived from WHO ATC. This initiative now allows uniquely accurate searching for intervention with resulting searches of ultra-high, pinpoint accuracy and no redundancy. Quantification of the workload involved in systematically reviewing an area or topic becomes noticeably more accurate, further magnified by supply of full datasets.
Paper 7: Using the curated register, I illustrate how new insights into publication, research and care can gained from even the relatively simple analysis of the now less confused body of trial evidence maintained within the study-based register.
Conclusion and Impact on Policy
Paper 3: To help the move toward full access to all data extracted from trials by people who are publically funded, I planned, instigated, led and co-ordinated this international and senior collaborative authorship. The paper encouraged the Cochrane Collaboration to develop global policy and take action regarding data sharing, referring to successful examples of such sharing from systematic reviews. This call did help move the argument forward within this largest producer of maintained reviews worldwide (Appendix A).
Paper 8 and 9: Study-based registers can directly assist in the crisis over irreproducibility within research. Systematic review methods do have certain strengths because of the need to use two or three reviewers and through development of automation. Unlike many who suggest adding new reproducibility tests into the systematic review process – to increase transparency but also making the process even more time-consuming - I discuss seven suggested strategies to enhance the reproducibility of systematic reviews: pre-registration, open methods, open data, collaboration, automation, reporting guidelines, and post-publication reviews. These two papers complement Paper 3’s call for data sharing policy in Cochrane Collaboration. Furthermore, Paper 8 & 9 expand on the idea that, because systematic reviews are often updated and have existing protocols, and also because relevant automation tools are developing or in existence – allowing replication of processes in seconds - systematic reviews can be a role model of reproducibility for other research designs.
References
Paper 1: Shokraneh F, Adams CE. Study-based registers of randomized controlled trials: Starting a systematic review with data extraction or meta-analysis. BioImpacts 2017; 7(4): 209-217. https://doi.org/10.15171/bi.2017.25
Paper 2: Shokraneh F, Adams CE. Increasing value and reducing waste in data extraction for systematic reviews: tracking data in data extraction forms. Systematic Reviews 2018; 6: 153. https://doi.org/10.1186/s13643-017-0546-z
Paper 3: Shokraneh F, Adams CE, Clarke M, Amato L, Bastian H, Beller E, et al. Why Cochrane should prioritise sharing data. BMJ 2018; 362:k3229. https://doi.org/10.1136/bmj.k3229
Paper 4: Shokraneh F, Adams CE. Study-based registers reduce waste in systematic reviewing: discussion and case report. Systematic Reviews 2019; 8:129 https://doi.org/10.1186/s13643-019-1035-3
Paper 5: Shokraneh F, Adams CE. A simple formula for enumerating comparisons in trials and network meta-analysis. F1000Research 2019; 8:38. https://doi.org/10.12688/f1000research.17352.1
Paper 6: Shokraneh F, Adams CE. Classification of all pharmacological interventions tested in trials relevant to people with schizophrenia: A study-based analysis. Health Information and Libraries Journal [Revised]
Paper 7: Shokraneh F, Adams CE. Cochrane Schizophrenia Group’s Study-Based Register of Randomized Controlled Trials: Development and Content Analysis. Schizophrenia Bulletin 2020 [Submitted]
Paper 8: Shokraneh F. Reducing waste and increasing value through embedded replicability and reproducibility in systematic review process and automation. Journal of Clinical Epidemiology 2019; 112: 98-9. https://doi.org/10.1016/j.jclinepi.2019.04.008
Paper 9: Shokraneh F. Reproducibility and replicability of systematic reviews. World Journal of Meta-Analysis 2019;7(3):66-71. http://dx.doi.org/10.13105/wjma.v7.i3.66
| Item Type: |
Thesis (University of Nottingham only)
(PhD)
|
| Supervisors: |
Adams, Clive E. |
| Keywords: |
Systematic Reviews; Evidence Synthesis; Cochrane Collaboration; Schizophrenia; Study-Based Registers; Databases; Open Data; Reproducibility; Data Sharing; Data Extraction; Screening; Systematic Searching; Classification; Pharmacotherapy; Pharmacological Interventions; Network Meta-Analysis; Data Science; Register-Based Study |
| Subjects: |
W Medicine and related subjects (NLM Classification) > W Health professions |
| Faculties/Schools: |
UK Campuses > Faculty of Medicine and Health Sciences > School of Medicine |
| Item ID: |
59619 |
| Depositing User: |
Shokraneh, Farhad
|
| Date Deposited: |
23 Jul 2021 15:21 |
| Last Modified: |
13 Jan 2023 13:41 |
| URI: |
https://eprints.nottingham.ac.uk/id/eprint/59619 |
Actions (Archive Staff Only)
 |
Edit View |
Tools
Tools
![[img]](/style/images/fileicons/application_pdf.png)