Artificial Intelligence (AI) and the literature review process: Searching

Generic Chatbots

AI chat bots such as ChatGPT show potential in generating Boolean search queries. But there are caveats that you need to be aware of:

Most language models are trained with data originating from the Internet but this data is not always "up-to-date". For instance, GPT 3.5 was trained on data up until January 2022.

Some chatbots can access external data sources via plug-ins or extensions. This ability to connect to live data sources reduces the risk of "hallucinations" in the generated answers. Plugins such as PubMed Research ChatGPT and ScholarAI largely overcome the issues of hallucination and currency that is often commented upon in academic literature but require a monthly subscription.

Copilot and Chat GPT 4 can search the internet while Bard uses Google's crawler to continuously update its training data.

CopilotChatGPT 4Bard

Example prompts for Chatbots

Prompt - Step 1

Follow my instructions precisely to develop a highly effective Boolean query for a medical systematic review literature search. Do not explain or elaborate. Only respond with exactly what I request. First, Given the following statement and text from a relevant study, please identify 50 terms

or phrases that are relevant. The terms you identify should be used to retrieve more relevant studies, so be careful that the terms you choose are not too broad. You are not allowed to have duplicates in your list.
Statement: "Prevalence of Differentiated Thyroid Cancer in Autopsy Studies
Over Six Decades: A Meta-Analysis". 
Text omitted.

Prompt - Step 2

For each item in the list you created in step 1, classify it into as of three categories: terms relating to health conditions (A), terms relating to a treatment (B), terms relating to types of study design (C). When an item does not fit one of these categories, mark it as (N/A). Each item needs to be categorised into (A), (B), (C), or (N/A).

Prompt - Step 3

Using the categorised list you created in step 2, create a Boolean query that can be submitted to PubMed which groups together items from each category. For example: ((itemA1[Title/Abstract] OR itemA2[Title/Abstract] or itemA3[Title/Abstract])
AND
 (itemB1[Title/Abstract] OR itemB2[Title/Abstract] OR itemB3[Title/Abstract])
AND
(itemC1[Title/Abstract] OR itemC2[Title/Abstract] OR itemC3[Title/Abstract])).

Evidence

ChatGPT is not trained on PubMed so the results of a search for medical items may not find relevant studies. The search generated by ChatGPT for Altmae et al. (2023) did not find any articles from PubMed or Google. In addition, the articles retrieved were not on topic. They all looked plausible but 6 of the 8 generated (75%) were entirely fictional.

Wang et al. (2023) was the first research study to evaluate the effectiveness of using ChatGPT for creating Boolean queries. Their findings are instructive:

• Generating a good prompt is critical. See the advice in the section on prompt engineering.
• Providing a sample high quality Boolean query, from a previous systematic literature review, in the prompt is beneficial.
• Multiple prompts and interactions with ChatGPT are far better than a single interaction.

ChatGPT generated a higher precision (studies which were relevant) compared to current-state-of-the-art automatic query formulation methods. This increased precision was often at the expense of recall (finding all studies which were relevant) which was lower than current automatic methods. Systematic literature review methods require high recall to make sure that all relevant studies have been retrieved. ChatGPT may therefore be "best suited when time is limited e.g. for rapid reviews" (Wang et al, 2023: 9).

There are three key caveats:

• ChatGPT generated queries that contained medical subject headings (MeSH) which were incorrect: they were not MeSH subject headings.
• repeating the same prompt will result in different queries being generated – and their effectiveness can differ. This is an issue in a systematic literature review where the searches need to be repeatable.
• in a separate study, the search string generated by ChatGPT looks plausible but did not actually retrieve any results. ChatGPT also applied a filter to search by publication type for clinical trials but this excluded trials not tagged by this type (Qureishi et al., 2023).

Please be aware that the content produced by AI tools such as ChatGPT may score highly on plagiarism detection software if you submit this work without alteration. Aydin and Karaarslan (2022) found that the abstracts generated by ChatGPT, when asked to summarize the literature on the basis of abstracts provided from Google Scholar, matched highly when using iThenticate. Similarly, the content generated by ChatGPT to answer questions on the topic had matches which would raise questions of whether text had been plagiarised. This university uses Turnitin software to detect plagiarism. Turnitin has incorporated AI detection to help identify when AI writing tools such as ChatGPT have been used in students’ submissions.

Semantic Scholar

Semantic Scholar is a free, AI-powered search and discovery tool that helps researchers discover and understand scientific literature most relevant to their work.

Semantic Scholar sources its content via web indexing and from partnerships with scientific journals, indexes, and content providers. These include leading publishers such as Springer Nature, Taylor & Francis, Wiley and Sage and institutions such as the IEEE and ACM. It does not contain content from Elsevier, Emerald or Oxford University Press. There is a list of its publisher partners.

Semantic Scholar does not support Boolean searching or wildcards. Its tutorial pages tell you that Semantic Scholar covers papers across all subjects including biomedicine, computer science, business, history, and economics. Its Ask This Paper and Generative Term Understanding features employ generative AI.

Semantic Scholar

Using Semantic Scholar

Semantic Scholar uses one search box in which you can enter your search terms. Your search keywords goes to Elasticsearch and the top 1000 results are reranked by a machine learning ranker. You can also enter the title of a paper.

The results of the search can be filtered by fields of study, date range, whether there is a pdf of the article, author and key journals and conference titles. Click on the title to view the paper page.

The results display provides a TLDR (too long, didn't read) short summary of the paper, an abstract, figures and tables from the paper if allowed by the publisher and the citation network.

The paper page displays if any papers are highly influential i.e. any citations where the cited publication has a significant impact on the citing publication, making it easier to understand how publications build upon and relate to each other.

Semantic Scholar uses AI models to classify the intent and predict the influence of each citation.

The citation type filter allows you to identify papers that cite the background, cite the methods, or cite the results. Its tutorial pages give you more explanation.

Evidence

Semantic Scholar has often been used as a database to search in systematic literature reviews in addition to databases such as PubMed, Cochrane Database, Scopus, Web of Science and Science Direct (see the review of the use of iodoform in jaw lesions by Arangaraju et al. (2023); the review of the use of total temporomandibular joint replacement by Khattak et al., 2023; and the efficacy of auditory verbal therapy in children with cochlear implantation by Noel et al. (2023) for recent examples).

These AI tools have been used in the academic literature to find papers similar to papers already found.

Connected Papers

Connected Papers (Smolyansky, 2020) is based on the Semantic Scholar database. Its premise is that two papers that have highly overlapping citations and references are presumed to have a higher chance of treating a related subject matter.

The graphs that are produced are designed to highlight the most important and relevant papers immediately.

Limits: You will only get 5 graphs per month for free.

Connected Papers

Using Connected Papers

• Enter the title details in Connected Papers.
• Choose the correct paper from a list.
  This will then allow you to produce a graph of similar papers.
• Screen the titles, abstracts, and full texts of all the connected papers to identify other relevant studies.
• Clicking on prior works shows you papers that were most commonly cited by the papers in the graph.
  This usually means that they are important works in this field so it would be useful for you to be familiar with them. Selecting a prior work will highlight all graph papers referencing it. Selecting a graph paper will highlight all referenced prior work.
• Clicking on derivate works will show you the papers that cited many of the papers in the graph.
  This usually means that they are either surveys of the literature or recent relevant works which have cited as evidence many papers in the graph.

Evidence

Connected Papers has been used in literature reviews in several disciplines alongside searching traditional databases to identify other relevant studies. Its key advantage is that you do not need to know all the keywords and alternative terms to search as with manual searching. Example reviews include understanding the identity of lived experience researchers and providers (Gupta et al., 2023), positive parenting interventions and body weight (Kong et al., 2023), components and characteristics of effective interventions involving men and boys in family planning in low- and middle-income countries (Aventin et al., 2023) and diagnostic windows in non-neoplastic diseases (Whitfield et al., 2023).

Research Rabbit

Research Rabbit also uses Semantic Scholar to search for papers but also uses PubMed to find biomedical and life sciences papers. It is committed to remaining free to researchers.

ResearchRabbit

Using Research Rabbit

For Research Rabbit, the prompt is to provide details of a seed paper in which you are interested.

Research Rabbit uses PubMed records in the biological and life sciences and uses Semantic Scholar for all other subject areas. It provides you with a list from which you select the correct paper.

Directly add papers to your collection with the Add Papers button or use the Add By Search button to find papers through PubMed.

Once you have added your papers, Research Rabbit will start generating recommendations for you in the buttons at the bottom of the panel. As you add more papers, Research Rabbit better understands your interests and generates better recommendations.

Research Rabbit allows you to:

• Explore papers (either similar work or listing the references or the articles citing it).
• Explore the authors or suggested authors.
• Search for similar results in the included articles.

Clicking on All Citations under the Explore Papers section will open a new panel showing the connections between the original paper and others. The panels are like a breadcrumb trail showing each step you take along a research rabbit hole. Each hop opens a new panel to the right, so you can see the path you have taken.

Create new collections and drag papers to the appropriate collection. Papers can be in more than one collection. All collections in which a paper is found are shown with yellow tags. There is no institutional access to papers. Research Rabbit is relying on papers to be freely available to provide you with any copies. The FAQ page for Research Rabbit shows you how to import and export references to reference management software.

Evidence

Similar to Connected Papers, Research Rabbit has been used in literature reviews to find papers similar to the original work you supply.

These include community mental healthcare for people with severe mental illness (van Genk et al., 2023), the circular economy, urban design and urban planning (Bortolotti et al., 2023), reviewing imaging techniques for measuring the small intestine (Chacon and Wilson, 2023) and ice detection sensors (DiLorenzo ad Yu, 2023).

Consensus

Consensus uses Semantic Scholar as its data source and aims to use AI to make expert information accessible to all. It provides citations for research papers​ and clearly lays out the findings of the specific paper along with the abstract of the paper and access to the full text.

Consensus uses large language models to find and synthesize insights from academic research papers. As with the above, there is no chat function but it uses artificial intelligence to help make the research process more efficient. Details of how it does this are on its FAQs.

Its free account enables you to conduct unlimited searches, unlimited research quality indicators and 20 AI credits per month for its most powerful features: GPT-4 Summaries, Consensus Meters, and Study Snapshots.

In November 2023, Consensus announced the launch of ResearchGPT,  a custom GPT plugin for ChatGPT, that allows users to find answers, search for papers, and draft content by searching the Consensus database directly within the ChatGPT interface. As a plugin, ResearchGPT requires you have a ChatGPT Plus account which requires subscription.

Consensus

Using Consensus

Best practice for Consensus advises to search for a subject on which research papers have been written. Consensus works best with:

• Yes/No questions - are genetically modified foods safe?; 
• Ask about the relationship between concepts: Does raising the minimum wage increase unemployment?”
• Ask about the effects, impact, or benefits of a concept: What are the effects of immigration on the economy?”
• Input an “open-ended phrase” about your subject of interest: "avocado health effects".

Consensus presents the search results with a tile representing a single paper:

• The title is the bold text at the very top.
• Other information like journal, authors, and research quality indicators are found at the bottom.
• The grey box contains an an insight from the paper.

If you asked a question, it will generate an answer from the paper.

If you searched with keywords, this text will be the "Key Takeaway" that is extracted from every paper.

The Study Snapshots extract key details such as population, sample size, methods and outcomes measured. Apply filters to restrict the search by study design or journal quality.

Evidence

Consensus has been used to develop a search strategy and assess its comprehensiveness. Consensus was one of the AI tools used to create a list of 20 benchmark studies, highly relevant papers, that the search strategy needed to retrieve in a systematic review protocol created to answer the research question: How effective are protected areas for reducing threats to biodiversity? (Pulido-Chadid et al., 2023).

Scopus

Scopus contains a ScopusAI option which allows you to search using natural language rather than the traditional line-by-line approach.
This generates a summary in response to your query. There are options to show the references and to visualise the result. This may help with providing a focus for your research.
Currently there is no interaction, no follow-up you can ask in response. 

Scopus

Web of Science

The AI research assistant is currently in beta testing and was set to go live in December 2023 (see Clarivate’s blog, 7 November).
This will bring semantic search, typeaheads, autocorrect functions to improve search queries.
Web of Science is currently using AI to create suggested keywords to add to your query: consider using them but it is probably best to add them manually.

Web of Science