In this presentation, Adam Dunn, PhD from the University of Sydney, presents on research conducted over the past decade (often in collaboration with Dr. Florence Bourgeois from Harvard Medical School). This talk was given as part of a symposium on advancing methods and technologies to improve evidence synthesis, put on by the Agency for Healthcare Quality and Research (AHRQ) in March of 2021.

This session focuses on deploying AI techniques, such as Natural Language Processing (NLP), to analyze clinicaltrials.gov and PubMed data, with the end goal of addressing complex problems related to systematic reviews of clinical trials. While discussing a variety of topics and use cases, two areas of particular interest are tools and methods to (1) identify publications associated with a trial registration in cases where the built-in linkages between clinicaltrials.gov and PubMed do not exist; and (2) lay the groundwork for automating the numerous steps associated with updating a published systematic review.

In order to identify publications associated with trial registrations, Dr. Dunn and Dr. Bourgeois mined 27,280 pairs of trials and articles that had metadata links between PubMed and clinicaltrials.gov. Using this information, they trained a classifier to rank articles based on the likelihood that an article reports on the trial described in its registration. For over 80% of trial registrations, the article ranked first matched the trial in question. Furthermore, for 90% of registrations, the resulting article was found on the first page of results.

In the second half of the talk, Dr. Dunn discusses groundwork for a tool that reviews new trial registrations on clinicaltrials.gov, flags them if reflective of trials contained in an already published review, and sends the records to the review author for approval. If approved, when results become available on clinicaltrials.gov, they are automatically extracted and included in an existing systematic review, and if appropriate, meta-analysis. To date they have tested this workflow using a variety of AI methods such as document similarity, hierarchical agglomerative clustering, and hierarchy traversals. They plan to continue exploring and refining this work in the future.

As doers and promoters of fast, intelligent, evidence syntheses, we find the work of Dr. Dunn and Dr. Bourgeois to be not only compelling, but perfectly aligned with our ethos here at Prism. As data is produced at an unprecedented rate, it is clear that traditional manual approaches to evidence synthesis are falling short. By employing a tool that identifies publications from clinical trial registrations, reviewers can cut down on the amount of time they spend searching for publications and instead, spend time doing the hard work of making inferences and decisions.

At Prism, we draw on clinicaltrials.gov data to help our clients make important decisions such as which trial to plan next or decide which outcomes to measure. A tool, such as the one described in this talk, could allow our clients to ask a different set of questions and more readily compare what was reported on the trial registration with the published results.

Finally, in order to make decisions like what research to fund or which drug to approve, decision makers need access to the most up to date information in real time. As discussed in the talk, and daily at Prism, traditional approaches to evidence synthesis are lengthy, prone to duplication, and rarely updated; all realities that result in an unsettling amount of waste. Much like the Prism approach to evidence synthesis, the tools described by Dr. Dunn cut down on waste and promote the curation of high quality, living evidence landscapes.

As we continue to grow and offer new tools and methods to our clients, we look forward to continuing to learn from experts like Dr. Dunn and Dr. Bourgeois. If you have a talk or article you'd like to share, drop us a line at info@prism.bio

Additional Resources

Check out Dr. Dunn and Dr. Bourgeois’ Evidence Surveillance Synthesis and Sharing (ES3) web platform where the output of this research is put into real world action.

To see other talks included in this symposium, check out this page on the University of Colorado's website.