Systematic Reviews: Medical Literature Databases to search

  • Types of literature review, methods, & resources
  • Protocol and registration
  • Search strategy
  • Medical Literature Databases to search
  • Study selection and appraisal
  • Data Extraction/Coding/Study characteristics/Results
  • Reporting the quality/risk of bias
  • Manage citations using RefWorks This link opens in a new window
  • GW Box file storage for PDF's This link opens in a new window

How to document your literature search

You should always  document how you have searched each database, what keywords or index terms were used, the date on which the search was performed, how many results you retrieved, and if you use RefWorks to deduplicate results record how many were removed as duplicates and the final number of discrete studies you subjected to your first sift through of study selection.  Here is an example of how to document a literature search on an Excel spreadsheet , this example records a search of the hematology literature for articles about sickle cell disease. Here is another example of  how to document a literature search, this time on one page of a Word document , this example records a search of the medical literature for a poster on Emergency Department throughput.  The numbers recorded can then be used to populate the PRISMA flow diagram summarizing the literature search.

In the final report add as an appendix the full electronic search strategy for each database searched for the literature review e.g. MEDLINE with MeSH terms, keywords & limits

In the final report in the methods section:

PRISMA checklist Item 7 information sources will be reported as:

  • What databases/websites you searched, the name of the database search platform and the start/end dates the index covers if relevant e.g. OVID MEDLINE (1950-present, or just PubMed
  • Who developed & conducted the searches
  • Date each database/website was last searched
  • Supplementary sources - what other websites did you search? What journal titles were hand searched, whether reference lists were checked, what trial registries or regulatory agency websites were searched, were manufacturers or other authors contacted to obtain unpublished or missing information on study methods or results.

PRISMA checklist Item 8 search will be reported as:

  • In text: describe the principal keywords used to search databases, websites & trials registers

What databases/indexes should you search?

At a minimum you need to search MEDLINE ,  EMBASE , and the  Cochrane CENTRAL  trials register .  This is the recommendation of three medical and public health research organizations: the U.S.  Agency for Healthcare Research and Quality ( AHRQ ), the U.K. Centre for Reviews and Dissemination ( CRD ), and the International Cochrane Collaboration (Source:  Institute of Medicine (2011) Finding What Works in Healthcare: Standards for Systematic Reviews  Table E-1, page 267).  Some databases have an alternate version, linked in parentheses below, that search the same records sets, ie the content of MEDLINE is in PubMed and Scopus, while the content of EMBASE is in Scopus. You should reformat your search for each database as appropriate, contact your librarian if you want help on how to search each database.  

Begin by searching:

1.        MEDLINE  (or  PubMed )

2.       EMBASE (or  Scopus )  Please note Himmelfarb Library does not have a subscription to EMBASE. The content is in the Scopus  database that you can search using keywords, but it is not possible to perform an EMTREE theasaurus search in Scopus.

3.        Cochrane Central Trials Register  (or  Cochrane Library ). In addition Cochrane researchers recommend you search the clinicaltrials.gov and ICTRP clinical trial registries due to the low sensitivity of the Cochrane CENTRAL index because according to Hunter et al (2022) "register records as they appear in CENTRAL are less comprehensive than the original register entry, and thus are at a greater risk than other systems of being missed in a search."

The Polyglot Search Translator is a very useful tool for translating search strings from PubMed or Medline via Ovid across multiple databases, developed by the Institute for Evidence-Based Healthcare at Bond University. But please note Polyglot does not automatically map subject terms across databases (e.g. MeSH terms to Emtree terms) so you will need to manually edit the search syntax in a text editor to change to the actual subject terms used by another database.

The Yale Mesh Analyzer is another very useful tool you can copy and paste in a list of up to 20 PMID numbers for records in the PubMed database, the Yale Mesh Analyzer will then display the Mesh Medical Subject Headings for those 20 articles as a table so you can identify and compare what Mesh headings they have in common, this can suggest additional search terms for your PubMed search.

The MedSyntax tool is another useful tool, for parsing out very long searches with many levels of brackets. This would be useful if you are trying to edit a pre-existing search strategy with many levels of parentheses.

Some sources for pre-existing database search filters or "hedges" include:

  • CADTH Search Filters Database ,
  • McMaster University Health Information Research Unit ,
  • University of York Centre for Reviews and Dissemination InterTASC Information Specialists' Sub-Group ,
  • InterTASC Population Specific search filters  (particularly useful for identifying Latinx, Indigenous people's, LGBTQ, Black & Minority ethnic)
  • CareSearch Palliative Care PubMed search filters  (bereavement, dementia, heart failure, lung cancer, cost of care, and Palliative Care)
  • Low and Middle Income countries filter at https://epoc.cochrane.org/lmic-filters . 
  • Search Pubmed for another validated search filter using some variation of a search like this, possibly adding your discipline or search topic keywords: ("Databases, Bibliographic"[Mesh] OR "Search Engine"[Mesh]) AND ("Reproducibility of Results"[Mesh] OR "Sensitivity and Specificity"[Mesh] OR validat*) AND (filter OR hedge) .
  • Search MEDLINE (or PubMed), preferably using a peer reviewed search strategy per protocol and apply any relevant methodology filters.
  • Search EMBASE (or Scopus) and the Cochrane Central trials register using appropriately reformatted search versions for those databases, and any other online resources. 
  • You should also search other subject specific databases that index the literature in your field.  Use our Himmelfarb Library  research guides  to identify other  subject specific databases . 
  • Save citations in Covidence to deduplicate citations prior to screening.
  • After screening export citations to  RefWorks database when you are ready to write up your manuscript. The Covidence and Refworks databases should be shared with all members of the investigative team.

Supplementary resources to search

Other member of your investigative team may have ideas about databases, websites, and journals they think you should search. Searching these sources is not required to perform a systematic review. You may need to reformat your search keywords.

Researchers at GW should check our subject research guides for suggestions, or check the libguides community for a guide on your subject.

In addition you may wish to search one or more of the following resources:

  • Google Scholar
  • BASE  academic search engine is useful for searching in University Institutional Repositories
  • Cochrane Database of Systematic Reviews  to search for a pre-existing systematic review on your topic
  • Epistemonikos database, has a matrix of evidence table so you can see what citations are shared in common across existing systematic reviews of the same topic. This feature might help identify sentinel or 'don't miss' articles.

You might also consider searching one or more of the following websites depending on your topic:

Clinical trial registers. The Cochrane Collaboration recommends for a systematic review to search both clinicaltrials.gov and the WHO ICTRP (See http://handbook.cochrane.org/ section 4.3):

  • ClinicalTrials.gov  - also contains study population characteristics and results data of FDA regulated drugs and medical devices in NIH funded studies produced after January 18, 2017.
  • WHO ICTRP  - trials register
  • TRIP  - searchable index of clinical trials, guidelines,and regulatory guidance
  • CenterWatch
  • Current Controlled Trials
  • European Clinical Trials Register
  • ISRCTN Register
  • COMPARE - tracks outcome switching in clinical trials
  • OpenTrials - aims to match published trials with the underlying data where this is publicly available in an open source 
  • ECRI Guidelines Trust

Grey literature resources:

  • WONDER - CDC data and reports
  • FDSys - search federal government publications
  • Science.gov
  • NRR Archive
  • NIH Reporter
  • re3data registry of data repositories
  • Data Repositories (listed by the Simmons Open Access Directory)
  • OpenDOAR  search academic open access research repositories
  • f1000research search open access repositories of articles, slides, and research posters, in the life sciences, public health, education, and communication.
  • RAND Health Reports
  • National Academy of Medicine Publications
  • Kaiser Family Foundation 
  • Robert Wood Johnson Foundation health and medical care data archive
  • Milbank Memorial Fund reports and issue briefs
  • Also search the resources listed in the CADTH (2019) Grey Matters checklist.

Preprints 

  • See our Himmelfarb preprints guide page on finding preprints , a useful database for searching Health Sciences preprints is  Europe PMC

Dissertations and Theses:

  • Proquest Dissertations and Theses Online 
  • Networked Digital Library of Theses and Dissertations
  • Open Access Theses and Dissertations
  • WorldCat and change Content: from Any Content to Thesis/dissertations

Conference proceedings:

Most conference proceedings are difficult to find because they may or may not be published. Only select individual papers may be made available in print as a book, journal, or series, rather than all of the presented items. Societies and Associations may only publish abstracts, or extended abstracts, from a conference, often in an annual supplement to an issue of the journal of record of that professional society.  Often posters are not published, if they are they may be made available only to other conference registrants at that meeting or online. Authors may "publish" their conference papers or posters on personal or institutional websites.  A limited set of conference proceedings databases include the following:

  • BASE  academic search engine, has an Advanced Search feature with a Limit by Type to 'Conference Objects', this is useful for searching for conference posters and submissions stored in University Institutional Repositories.
  • Web of Science - click All Databases and select Core Collection - under More Settings limit to the Conference Proceedings Citation Index (CPCI) - searches a limited set of conferences on Science, Social Science and Humanities from 1990-present.
  • Scopus - Limit Document Type to Conference Paper or Conference Review.
  • Proquest  - Limit search results to conference papers &/or proceedings under Advanced Search.
  • BioMed Central Proceedings  - searches a limited set of biomedical conference proceedings, including bioinformatics, genetics, medical students, and data visualization.
  • F1000 Research - browse by subject and click the tabs for articles, posters, and slides - which searches a limited number of biology and medical society meetings/conferences. This is a voluntary self-archive repository.

Individual Journals 

  • You may choose to "hand search" select journals where the research team reads the Table of Contents of each issue for a chosen period of time.  You can look for the names of high impact journal titles in a particular field indexed in Journal Citation Reports  (JCR). Please note as of August 2021 ISI are linking to a new version of JCR that currently does not have the particularly helpful 'Browse by Category' link working, so I recommend you click the Products link in the top right corner and select Journal Citation Reports (Classic) to switch back to the old version to get that functionality back.
  • The AllTrials petition aims to motivate health care researchers to petition regulators and research bodies to require the results and data of all clinical trials be published.
  • << Previous: Search strategy
  • Next: Study selection and appraisal >>

Creative Commons License

  • Last Updated: Nov 16, 2023 9:12 AM
  • URL: https://guides.himmelfarb.gwu.edu/systematic_review

GW logo

  • Himmelfarb Intranet
  • Privacy Notice
  • Terms of Use
  • GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
  • Himmelfarb Health Sciences Library
  • 2300 Eye St., NW, Washington, DC 20037
  • Phone: (202) 994-2850
  • [email protected]
  • https://himmelfarb.gwu.edu

Login into Epistemonikos

Your browser is ancient! Upgrade to a different browser to experience this site.

Combines the best of Evidence-Based Health Care , information technologies and a network of experts to provide a unique tool for people making decisions concerning clinical or health-policy questions.

Epistemonikos database in a nutshell

One-stop shop.

All of the evidence relevant for health decision-making, including world's largest systematic review database, curated and annotated by our network of collaborators.

Interconnected evidence

Articles are connected, so you can easily move from any article to all the evidence answering the same question. You can see the forest, not only the trees!

Powerful search engine

Find what you are looking for using our multilingual foolproof search tools. Searching for the best available evidence has never been easier!

Testimonials

Picture of Gordon Guyatt

Epistemonikos has become one of the largest databases of health evidence, a user friendly source that includes over 100,000 systematic reviews and hundreds of thousands of individual studies. Apart from the amount of information, Epistemonikos is unique in a number of ways. First, it caters to non-English speaking users - you can conduct your search in any of 9 languages. Second, a "matrix of evidence" tool facilitates quick searching and updating. The tool identifies all systematic reviews that address a particular question, and all the studies included in those reviews, displaying a cross-tabulation of reviews and studies. It also allows you to stay up to date: after you save your search, any new review will appear automatically.

Gordon Guyatt Distinguished University Professor at McMaster University in Hamilton, Ontario. Pioneer in evidence-based medicine, coined the term "evidence-based medicine" in 1990.

Picture of Andy Oxman

With over 100,000 systematic reviews plus the studies included in those reviews plus overviews of reviews, Epistemonikos is not only the world's largest database of systematic reviews, it is one of the easiest to use. In addition to making it easy to quickly find them, it makes it easy to compare systematic reviews. Epistemonikos has quickly become an essential tool for anyone interested in using evidence to inform healthcare decisions. It also is a unique platform for creating specialised databases, such as PDQ-Evidence , a database of systematic reviews that provides quick access to the best evidence for informing decisions about health systems.

Andy Oxman Research Director, Global Health Unit, Norwegian Institute of Public Health

Picture of Users' Guides to the Medical Literature

...for example, Epistemonikos connects systematic reviews and their included studies, and thus allows clustering of systematic reviews based on the primary studies they have in common. Epistemonikos is also unique in offering an appreciable multilingual user interface, multilingual search, and translation of abstracts in more than 9 languages.

Users' Guides to the Medical Literature

Picture of Iain Chalmers

Iain Chalmers Founder of the Cochrane Collaboration 1st Cochrane Lecture, 20th Cochrane Colloquium

U.S. flag

An official website of the United States government

Here's how you know

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Literature Search: Databases and Gray Literature

The literature search.

  • A systematic review search includes a search of databases, gray literature, personal communications, and a handsearch of high impact journals in the related field.  See our list of recommended databases and gray literature sources on this page.
  • a comprehensive literature search can not be dependent on a single database, nor on bibliographic databases only.
  • inclusion of multiple databases helps avoid publication bias (georaphic bias or bias against publication of negative results).
  • The Cochrane Collaboration recommends PubMed, Embase and the Cochrane Central Register of Controlled Trials (CENTRAL) at a minimum.     
  • NOTE:  The Cochrane Collaboration and the IOM recommend that the literature search be conducted by librarians or persons with extensive literature search experience. Please contact the NIH Librarians for assistance with the literature search component of your systematic review. 

Cochrane Library

A collection of six databases that contain different types of high-quality, independent evidence to inform healthcare decision-making. Search the Cochrane Central Register of Controlled Trials here.

European database of biomedical and pharmacologic literature.

PubMed comprises more than 21 million citations for biomedical literature from MEDLINE, life science journals, and online books.

Largest abstract and citation database of peer-reviewed literature and quality web sources. Contains conference papers.

Web of Science

World's leading citation databases. Covers over 12,000 of the highest impact journals worldwide, including Open Access journals and over 150,000 conference proceedings. Coverage in the sciences, social sciences, arts, and humanities, with coverage to 1900.

Subject Specific Databases

APA PsycINFO

Over 4.5 million abstracts of peer-reviewed literature in the behavioral and social sciences. Includes conference papers, book chapters, psychological tests, scales and measurement tools.

CINAHL Plus

Comprehensive journal index to nursing and allied health literature, includes books, nursing dissertations, conference proceedings, practice standards and book chapters.

Latin American and Caribbean health sciences literature database

Gray Literature

  • Gray Literature  is the term for information that falls outside the mainstream of published journal and mongraph literature, not controlled by commercial publishers
  • hard to find studies, reports, or dissertations
  • conference abstracts or papers
  • governmental or private sector research
  • clinical trials - ongoing or unpublished
  • experts and researchers in the field     
  • Library catalogs
  • Professional association websites
  • Google Scholar  - Search scholarly literature across many disciplines and sources, including theses, books, abstracts and articles.
  • Dissertation Abstracts - dissertation and theses database - NIH Library biomedical librarians can access and search for you.
  • NTIS  - central resource for government-funded scientific, technical, engineering, and business related information.
  • AHRQ  - agency for healthcare research and quality
  • Open Grey  - system for information on grey literature in Europe. Open access to 700,000 references to the grey literature.
  • World Health Organization  - providing leadership on global health matters, shaping the health research agenda, setting norms and standards, articulating evidence-based policy options, providing technical support to countries and monitoring and assessing health trends.
  • New York Academy of Medicine Grey Literature Report  - a bimonthly publication of The New York Academy of Medicine (NYAM) alerting readers to new gray literature publications in health services research and selected public health topics. NOTE: Discontinued as of Jan 2017, but resources are still accessible.
  • Gray Source Index
  • OpenDOAR - directory of academic repositories
  • International Clinical Trials Registery Platform  - from the World Health Organization
  • Australian New Zealand Clinical Trials Registry
  • Brazilian Clinical Trials Registry
  • Chinese Clinical Trial Registry - 
  • ClinicalTrials.gov   - U.S.  and international federally and privately supported clinical trials registry and results database
  • Clinical Trials Registry  - India
  • EU clinical Trials Register
  • Japan Primary Registries Network  
  • Pan African Clinical Trials Registry
  • Selected Databases

Core Resources for Systematic Reviews

It is a good practice to search several databases when conducting a systematic review. Include broad coverage databases as well as subject-specific databases.

Scholarly Articles and Other Documents

   Source:  University of Maryland Health Sciences and Human Services Library

PubMed: Find Articles on a Topic  An interactive tutorial to help you search PubMed for articles on a specific topic.

More PubMed Training

Cochrane Library

  • Cochrane Central Register of Controlled Trials (CENTRAL) A highly concentrated source of reports of randomized and quasi-randomized controlled trials. Most CENTRAL records are taken from bibliographic databases (mainly PubMed and Embase), but records are also derived from other published and unpublished sources, including CINAHL, ClinicalTrials.gov and the WHO's International Clinical Trials Registry Platform.

   Basic search tutorial on the  Cochrane Database of Systematic Reviews  from Drexel University Library.

Cochrane Handbook: Searching for and Selecting Studies Appendix of Resources  A list of resources for review authors. It is provided to support and complement the  Technical supplement  to Chapter 4: Searching for and selecting studies. Maintained by Julie Glanville ( [email protected] ).

  • JBI This link opens in a new window Joanna Briggs Institute EBP Database is an online resource for healthcare professionals to rapidly access the best available evidence on a wide range of clinical topics at the point of care. Globally acknowledged as a leading producer of evidence-based care, practice guidance and other resources, JBI includes 4500+ JBI Evidence Summaries, Recommended Practices and Best Practice Information Sheets.

Multidisciplinary Subject Coverage Databases

News and Magazines

  • Basic Searching in ProQuest

How to Conduct a Basic Search Tutorial  A video tutorial on searching Scopus from the Scopus Access and Use Support Center.

How to Use Advanced Search Tutorial A video tutorial on advanced searching from the Scopis Access and Use Support Center.

  • << Previous: Search Strategies
  • Next: Grey Literature >>
  • Types of Questions
  • Key Features and Limitations
  • Is a Systematic Review Right for Your Research?
  • Integrative Review
  • Scoping Review
  • Rapid Review
  • Meta-Analysis/Meta-Synthesis
  • Reducing Bias
  • Guidelines for Student Researchers
  • Register Your Protocol
  • Handbooks & Manuals
  • Reporting Guidelines
  • PRESS 2015 Guidelines
  • Search Strategies
  • Grey Literature
  • Handsearching
  • Citation Searching
  • Study Types & Terminology
  • Quantitative vs. Qualitative Research
  • Critical Appraisal of Studies
  • Broad Functionality Programs & Tools
  • Search Strategy Tools
  • Deduplication Tools
  • CItation Screening
  • Critical Appraisal Tools
  • Quality Assessment/Risk of Bias Tools
  • Data Collection/Extraction
  • Meta Analysis Tools
  • Books on Systematic Reviews
  • Finding Systematic Review Articles in the Databases
  • Systematic Review Journals
  • More Resources
  • Evidence-Based Practice Research in Nursing
  • Citation Management Programs
  • Last Updated: Jan 26, 2024 3:26 PM
  • URL: https://libguides.adelphi.edu/Systematic_Reviews

Memorial Sloan Kettering Library

systematic review health care databases

Systematic Review (SR) Service: Databases, Resources & Tools

  • What is an SR?
  • Getting Started
  • Databases, Resources & Tools
  • Critical Appraisal
  • SR Publications (97)
  • Evidence Based Practice

The National Academies states that a common error is to rely solely on a limited number of bibliographic databases in conducting SRs. Below are some of the large well-known databases commonly searched for SRs. A librarian can recommend other sources relevant to your topic, including grey literature (conference proceedings and government documents, for example).

  • Canadian Agency For Drugs and Technologies in Health (CADTH) Grey Matter This open access resource is not exhaustive but is updated regularly to include content from Canadian and International Health Technology Assessment Agencie
  • ClinicalTrials.gov Registry and results database of publicly and privately supported clinical studies of human participants conducted around the world.
  • The Cochrane Library Provides reliable and up-to-date information from seven different databases on the effects of interventions in healthcare. Published on a quarterly basis and designed to provide information and evidence to support decisions taken in healthcare and to inform those receiving care.
  • Cumulative Index to Nursing & Allied Health (CINAHL) This major indexing resource for nursing literature covering over 2,900 hundred English language nursing journals, as well as primary journals for the allied health disciplines. Abstracts are included for selected journal titles. The database also contains references to books, dissertations, selected conference proceedings, and standards of professional practice including publications of the American Nurses'Association and the National League for Nursing.
  • EMBASE EMBASE.com is updated daily and contains biomedical and pharmacological information. This database allows you to search both EMBASE and unique MEDLINE records simultaneously. Duplicate references are automatically removed and will not appear in search results. Subject areas covered include Drug Research; Pharmacology; Pharmaceutics; Pharmacy; Side Effects, Interactions and Toxicology.
  • Epistemonikos A collaborative, multilingual database of health evidence containing systematic reviews relevant for health-decision making, and a large source of other types of scientific evidence. This database is aimed at health professionals, researchers and health decision-makers.
  • Health and Psychosocial Instruments (HAPI) Provides information about articles on questionnaires, interview schedules, checklists, index measures, coding schemes and manuals, rating scales, and other measurement instruments in the fields of health, psychosocial sciences, organizational behavior, and library and information science. This database is updated quarterly and links to full text articles (Fulltext@MSK).
  • MeSH Browser This tool allows for searches of MeSH terms, text-word searches of the Annotation and Scope Note, and searches of various fields for chemicals.
  • MeSH on Demand This resource identifies MeSH terms in your text using the NLM Medical Text Indexer (MTI) program. The results provided by MeSH on Demand are a simple list of MeSH Terms that MTI identifies as being relevant to your text. Each of the identified MeSH Terms has a link to the corresponding MeSH Browser Web page for that MeSH Term. Can be a helpful tool in determining controlled vocabulary to use within your systematic review search strategy.
  • Open Access Theses Dissertations Good resource for finding open access graduate theses and dissertations published around the world. Content comes from many colleges, universities, and research institutions. See list of sites that contribute records: http://oatd.org/oatd-publishers.html
  • PsycINFO Covers the professional and academic literature in psychology and related disciplines including medicine, psychiatry, nursing, sociology, education, pharmacology, physiology, linguistics, management, business, social sciences, neuroscience, law and social work. Coverage is worldwide and includes references and abstracts to over 2,000 journals, dissertations, book chapters and books in the English language.
  • PubMed PubMed is a search service of the National Library of Medicine. It includes references from MEDLINE and MEDLINE in Process, as well as a small number of references not included in these databases. It features built-in search filters for diagnosis, therapy, etiology, and prognosis and includes links to databanks of DNA/protein sequence and 3-D structure data.
  • SRDR+ (Systematic Review Data Repository) This open source repository serves as an archive and data extraction tool, and facilitate evidence reviews. Registration is required.
  • Web of Science Two citation databases. Science Citation Index Expanded includes about 5,900 major journals across 150 scientific disciplines on biology; medicine; legal medicine; chemistry; cell biology; microbiology; medical informatics; oncology, psychiatry; toxicology; pharmacology and pharmacy; and physics. Social Sciences Citation Index covers about 1,725 journals across 50 social sciences disciplines on geriatrics and gerontology; health policy and services; nursing; rehabilitation; public health.

Systematic Reviews: Standards, Guidelines & Resources

  • AHRQ - Agency for Healthcare Research and Quality AHRQ offers practical, research-based tools and other resources to help a variety of health care organizations, providers, and others make care safer in all health care settings.
  • CEBD: Centre For Evidence Based Dentistry Based in the UK, this organization was established in 1995 and is an independent body whose aim is to promote the teaching, learning, practice and evaluation of evidence-based dentistry world-wide. This page provides an overview of the Systematic Review process.
  • Centre for Cognitive Ageing and Cognitive Epidemiology This Centre based at The University of Edinburgh provides a systematic reviews and meta-analyses step by step guide which is based on guidance from the Cochrane Collaboration and the Centre for Reviews and Dissemination at York.
  • Cochrane Handbook For Systematic Reviews of Interventions, Version 6.3, 2022 This edition of the Handbook is divided into four parts. The first section (available only online) addresses issues specific to working with Cochrane. The second describes the core methods applicable to systematic reviews of interventions, from framing the question through to interpreting the results. The third and fourth parts address specific perspectives and methodological issues that are relevant to some, though not all, reviews, such as non-randomized studies, qualitative evidence and economics evidence.
  • CRD's Guidance for Undertaking Reviews in Health Care This is the third edition of the Centre for Reviews and Dissemination (CRD) guidance for undertaking systematic reviews in health care. CRD is part of the National Institute for Health Research (NIHR) and is a department of the University of York.
  • EQUATOR Organization that brings together researchers, medical journal editors, peer reviewers, developers of reporting guidelines, research funding bodies and other collaborators with mutual interest in improving the quality of research publications and of research itself.
  • Finding Out What Works in Health Care: Standards for Systematic Reviews, from the National Academies Press A 2011 consensus report, reflecting the views and recommendations of the authors.
  • MeSH Analyzer (Yale) A bookmark tool for your browser that simplifies the creation of a MeSH analysis grid. A MeSH analysis grid can help identify the problems in a search strategy by presenting the ways articles are indexed in the MEDLINE database in an easy-to-scan tabular format. Creating a MeSH analysis grid manually is a tedious, time-consuming task. Metadata for each article must be manually retrieved, extracted, and pasted into a grid, and then the MeSH terms must be manually sorted and grouped alphabetically.
  • Methods Guide for Medical Test Reviews In an effort to improve the transparency, consistency, and scientific rigor of the work of the Effective Health Care (EHC) Program, the Agency for Healthcare Research and Quality (AHRQ), the Scientific Resource Center, and the Evidence-based Practice Centers (EPCs), have developed this textbook to serve as a resource for the EPCs as well as for other investigators interested in conducting systematic reviews on medical tests.
  • PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Explanation & Elaboration The PRESS Guideline provides a set of recommendations concerning the information that should be used by librarians and other information specialists when they are asked to evaluate these electronic search strategies. This guideline updates and expands upon the 2008 CADTH report PRESS: Peer Review of Electronic Search Strategies, as well as An Evidence Based Checklist for the Peer Review of Electronic Search Strategies (PRESS EBC), published in the Evidence Based Library and Information Practice journal in 2010.
  • PRISMA 2020 Evidence-based minimum set of items for reporting in systematic reviews and meta-analyses; primarily focuses on the reporting of reviews evaluating the effects of interventions, but can also be used as a basis for reporting systematic reviews with objectives other than evaluating interventions (e.g. prevalence, diagnosis or prognosis).
  • The PRISMA 2020 Statement An Updated Guideline For Reporting Systematic Reviews Learn more about the updated guideline, view video presentation (1:04:03).
  • PROSPERO International prospective register of systematic reviews. This site includes protocol details for systematic reviews relevant to health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome.
  • Why prospective registration of systematic reviews makes sense (Published: 09 Feb 2012) This editorial supports why develop and registering your systematic review protocol is important.
  • RAMESES - Realist and Meta-Narrative Evidence Synthesis Evolving Standards Realist and meta-narrative review are relatively new approaches to systematic review whose overall place in the secondary research toolkit is not yet fully established. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies.
  • RLO: Steps in Conducting a Systematic Review From the School of Nursing and Academic Division of Midwifery, University of Nottingham, this page provides an overview for conducting a systematic review.
  • Systematic Reviews This journal encompasses all aspects of the design, conduct and reporting of systematic reviews and aims to publish high quality systematic review products including systematic review protocols, systematic reviews related to a very broad definition of health, rapid reviews, updates of already completed systematic reviews, and methods research related to the science of systematic reviews, such as decision modeling.
  • Systematic Reviews: CRD's guidance for undertaking reviews in health care (Jan 2009) Optimized for Internet Explorer and Firefox and may not display correctly in other browsers.
  • Systematic Review Toolbox Community-driven resource and a searchable web-based catalogue of tools that support the systematic review process across multiple domains. Aims to help reviewers find appropriate tools based on how they provide support for the systematic review process. Users can perform a simple keyword search (e.g. Quick Search) to locate tools, or a more detailed search (e.g. Advanced Search) allowing the selection of various criteria to find specific types of tools or submit new tools to the database.
  • The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies (2012) Impact of language on systematic reviews.
  • English-Language Restriction When Conducting Systematic Review-based Meta-analyses: Systematic Review of Published Studies (Feb 2009) Authored by the Canadian Agency for Drugs and Technologies in Health. Impact of language on systematic reviews.

Covidence is a tool that we have integrated within our Systematic Review Service.  Our goal is to support our users, improve efficiency for the systematic review team, and reduce the time required to produce a systematic review.  

ACCESSING COVIDENCE You can create your personal sign-in information with Covidence before or after joining the institutional subscription. To request access to the i nstitutional account in Covidence , you must use your current MSK email address.  If you have already joined the MSK Library’s Covidence account, then you can log into Covidence with your email and password and start using this reference review management tool right away!

Helpful tips and tools for getting started. Learn about systematic reviews and how to use this application from Covidence Academy .

AMSTAR - Acronym stands for A Measurement Tool to Assess Systematic Reviews

Rayyan - An application developed at Qatar Computing Research Institute which aids systematic review authors in performing their research in a quick and easy fashion. Authors can collaborate together to determine papers that will be included/excluded and label articles. There is no cost associated with this tool.

ROBIS -  Tool for assessing the risk of bias in systematic reviews (rather than in primary studies) and is currently aimed at four broad categories of reviews mainly within healthcare settings: interventions, diagnosis, prognosis and aetiology. The target audience for this tool is primarily guideline developers, authors of overviews of systematic reviews (“reviews of reviews”) and review authors who might want to assess or avoid risk of bias in their reviews.tool designed specifically to assess the risk of bias in systematic reviews.  To learn more about this resource, review the 2016 publication in the Journal of Epidemiology entitled " ROBIS : A new tool to assess risk of bias in systematic reviews was developed ."

  • << Previous: Getting Started
  • Next: Critical Appraisal >>
  • Last Updated: Jan 22, 2024 10:18 AM
  • URL: https://libguides.mskcc.org/SR

Systematic Reviews (in the Health Sciences)

  • Introduction
  • Library Help & Tools
  • Standards & Guidelines
  • Getting Started
  • Research Question
  • Search Development
  • Search Techniques

Journal Article Databases

Database a-z list, database tutorials, google scholar, cited references & cited by, hand searching.

  • Grey Literature
  • Citation Management
  • Screening & Selection
  • Retrieving Full Text
  • Data Extraction
  • Meta-Analysis

You need to search databases that cover the topic of interest. Different questions require the use of different databases.

These journal article databases are frequently used for health sciences systematic reviews. If necessary, you can also view the Database A-Z List below to find additional databases relevant for your topic.

Databases include different journals. You need to search multiple databases published by different companies and agencies in order to find the full scope of relevant literature published.

PubMed and Ovid Medline are both MEDLINE. You only need to search one or the other. Not both. 

USC login required

  • PubMed Online Training
  • The New PubMed: Trainer's Toolkit
  • Using the CINAHL/MeSH Headings
  • Cochrane Library Training Hub
  • PsycINFO on Proquest YouTube Playlist

How do I find more database guides & tutorials?

  • Search Google or YouTube for the name of the database plus training/help/tutorial. 
  • Go into the database and look for a help section or question mark (?) icon. 

You can search Google Scholar in an effort to capture journal articles missed from other database searches.

  • Google Scholar will retrieve thousands of results. In general, include only the first 200 citations. 
  • Google Scholar has limited search features and no controlled vocabulary.
  • Google Scholar has a 256 character search limit. Google Scholar search strategy should be a simplified version of your other database search strategies. 
  • Google Scholar Search Help

SR teams should examine the reference lists and citing articles of studies included in the review to find additional relevant articles missed by database searches. 

For a more detailed explanation, see ITEM 2C: CITATION SEARCHING of the PRISMA-S Explanation and Elaboration document. 

  • PRISMA-S (Searching)

Hand searching refers to reading through specific journal table of contents and selecting articles relevant to the research topic to find additional relevant articles missed by database searches. 

For a more detailed explanation, see ITEM 2B: MANUAL SEARCHING of the PRISMA-S Explanation and Elaboration document. 

  • << Previous: Search Techniques
  • Next: Grey Literature >>
  • Last Updated: Jan 17, 2024 1:33 PM
  • URL: https://libguides.usc.edu/healthsciences/systematicreviews

Northeastern University Library

  • Northeastern University Library
  • Research Subject Guides
  • Guides for Library Services
  • Systematic Reviews and Evidence Syntheses
  • Evidence Synthesis Service
  • Types of Systematic Reviews in the Health Sciences
  • Beginning Your Project
  • Standards & Guidance
  • Critical Appraisal
  • Evidence-Based Assignments
  • Tips for a Successful Review Team
  • Training and Tutorials

Systematic Reviews and Evidence Syntheses : Databases

You will want to search at least three databases for your systematic review. Three databases alone does  not  complete the search standards for systematic review requirements. You will also have to complete a search of the grey literature and complete additional hand searches.  Which databases you should search is highly dependent on your systematic review topic, so it is recommended you  meet with a librarian . 

Commonly Used Health Sciences Databases

Commonly used social sciences databases, commonly used education databases.

  • Resources for Finding Systematic Reviews

You will want to search at least three databases for your systematic review. Three databases alone does  not  complete the search standards for systematic review requirements as you will also have additional searches of the grey literature and hand searches to complete.  Which databases you search is highly dependent on your systematic review topic, so it is recommended you  meet with a librarian . 

Cochrane, which is considered the gold standard for clinical systematic reviews, recommends searching the following three databases, at a minimum: PubMed, Embase, and Cochrane Central Register of Controlled Trials (CENTRAL).

Northeastern login or email required

You will want to search at least three databases for your systematic review. Three databases alone does  not  complete the search standards for systematic review requirements as you will also have additional searches of the grey literature and hand searches to complete.  Which databases you search is highly dependent on your systematic review topic, so it is recommended you  meet with a librarian . 

  • ERIC (Education Resources Institute) This link opens in a new window Citations to education information, including scholarly articles, professional literature, education dissertations, and books, plus grey literature such as curriculum guides, conference proceedings, government publications, and white papers. Covers 1966 to the present. more... less... Sponsored by the U.S. Department of Education.

Looking to Find Systematic Reviews?

There are a number of places to look for systematic reviews, including within the commonly used databases listed on this page. Some other resources to consider are:

  • Systematic Review Repository - International Initiative for Impact Evaluation The systematic review repository from International Initiative for Impact Evaluation is an essential resource for policymakers and researchers who are looking for synthesized evidence on the effects of social and economic interventions in low- and middle- income countries.
  • Epistemonikos Epistemonikos is a collaborative, multilingual database of health evidence. It is the largest source of systematic reviews relevant for health-decision making, and a large source of other types of scientific evidence. PLEASE NOTE: Epistemonikos is a systematic reviews focused database. It pulls in systematic reviews from a number of different international sources and pulls in the studies those reviews. While you will find randomized controlled trials and other primary studies in this database, they are only added in because of their association with a systematic review. Therefore, searching here for randomized controlled trials or other primary studies would NOT be considered a comprehensive search.
  • << Previous: Types of Systematic Reviews in the Health Sciences
  • Next: Resources for Completing Evidence Syntheses >>
  • Ask a Librarian
  • Last Updated: Feb 12, 2024 6:45 PM
  • URL: https://subjectguides.lib.neu.edu/systematicreview

Systematic Literature Reviews: Database Comparison Charts

  • Types of Reviews
  • Database Comparison Charts

Choices, choices, choices!

The Rowan-Virtua SOM Health Sciences Library offers access to more than 95 databases, with another 300 available through the Campbell Library on the main campus.  A large percentage of the citations in the databases include full-text.  Since these are paid subscription resources, the system will require you to use Duo authentication.  You'll find the list of these and other databases at the  Health Sciences Library web site  under the button for  "Medical Resources" .

Interested in "gray literature"?  Scroll down for a list of some of the best sites to find those resources.   Happy searching! 

Feel free to email me at [email protected] if you need help.

Choosing Your Main Databases

Grey literature: what is it and where do i get it.

Materials and research produced by organizations outside of the traditional commercial or academic publishing and distribution channels. Common grey literature publication types include reports (annual, research, technical, project, etc.), working papers, government documents, white papers and evaluations. Organizations that produce grey literature include government departments and agencies, civil society or non-governmental organizations, academic centres and departments, and private companies and consultants.  (Wikipedia).  Below you will find some places to look for grey literature.

  • ClinicalTrials.gov Explore 350,601 research studies in all 50 states and in 216 countries. Provided by the National Library of Medicine.
  • Drugs@FDA Search a drug to find approval information, letters, reviews and more.
  • Devices@FDA Documents relating to approval, safety, effectiveness and clinical trials.
  • Grey Literature Report New York Academy of Medicine, covering a number of topics in urban medicine.
  • Dissertations & Thesis Global Subscription through Rowan's Campbell Library

Other Databases to Consider

  • << Previous: Types of Reviews
  • Last Updated: Jun 21, 2023 9:42 AM
  • URL: https://rowanmed.libguides.com/systematicreviews
  • Systematic review
  • Open access
  • Published: 19 February 2024

‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice

  • Annette Boaz   ORCID: orcid.org/0000-0003-0557-1294 1 ,
  • Juan Baeza 2 ,
  • Alec Fraser   ORCID: orcid.org/0000-0003-1121-1551 2 &
  • Erik Persson 3  

Implementation Science volume  19 , Article number:  15 ( 2024 ) Cite this article

990 Accesses

55 Altmetric

Metrics details

The gap between research findings and clinical practice is well documented and a range of strategies have been developed to support the implementation of research into clinical practice. The objective of this study was to update and extend two previous reviews of systematic reviews of strategies designed to implement research evidence into clinical practice.

We developed a comprehensive systematic literature search strategy based on the terms used in the previous reviews to identify studies that looked explicitly at interventions designed to turn research evidence into practice. The search was performed in June 2022 in four electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched from January 2010 up to June 2022 and applied no language restrictions. Two independent reviewers appraised the quality of included studies using a quality assessment checklist. To reduce the risk of bias, papers were excluded following discussion between all members of the team. Data were synthesised using descriptive and narrative techniques to identify themes and patterns linked to intervention strategies, targeted behaviours, study settings and study outcomes.

We identified 32 reviews conducted between 2010 and 2022. The reviews are mainly of multi-faceted interventions ( n  = 20) although there are reviews focusing on single strategies (ICT, educational, reminders, local opinion leaders, audit and feedback, social media and toolkits). The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Furthermore, a lot of nuance lies behind these headline findings, and this is increasingly commented upon in the reviews themselves.

Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been identified. We need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of research perspectives (including social science) in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed.

Peer Review reports

Contribution to the literature

Considerable time and money is invested in implementing and evaluating strategies to increase the implementation of research into clinical practice.

The growing body of evidence is not providing the anticipated clear lessons to support improved implementation.

Instead what is needed is better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice.

This would involve a more central role in implementation science for a wider range of perspectives, especially from the social, economic, political and behavioural sciences and for greater use of different types of synthesis, such as realist synthesis.

Introduction

The gap between research findings and clinical practice is well documented and a range of interventions has been developed to increase the implementation of research into clinical practice [ 1 , 2 ]. In recent years researchers have worked to improve the consistency in the ways in which these interventions (often called strategies) are described to support their evaluation. One notable development has been the emergence of Implementation Science as a field focusing explicitly on “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice” ([ 3 ] p. 1). The work of implementation science focuses on closing, or at least narrowing, the gap between research and practice. One contribution has been to map existing interventions, identifying 73 discreet strategies to support research implementation [ 4 ] which have been grouped into 9 clusters [ 5 ]. The authors note that they have not considered the evidence of effectiveness of the individual strategies and that a next step is to understand better which strategies perform best in which combinations and for what purposes [ 4 ]. Other authors have noted that there is also scope to learn more from other related fields of study such as policy implementation [ 6 ] and to draw on methods designed to support the evaluation of complex interventions [ 7 ].

The increase in activity designed to support the implementation of research into practice and improvements in reporting provided the impetus for an update of a review of systematic reviews of the effectiveness of interventions designed to support the use of research in clinical practice [ 8 ] which was itself an update of the review conducted by Grimshaw and colleagues in 2001. The 2001 review [ 9 ] identified 41 reviews considering a range of strategies including educational interventions, audit and feedback, computerised decision support to financial incentives and combined interventions. The authors concluded that all the interventions had the potential to promote the uptake of evidence in practice, although no one intervention seemed to be more effective than the others in all settings. They concluded that combined interventions were more likely to be effective than single interventions. The 2011 review identified a further 13 systematic reviews containing 313 discrete primary studies. Consistent with the previous review, four main strategy types were identified: audit and feedback; computerised decision support; opinion leaders; and multi-faceted interventions (MFIs). Nine of the reviews reported on MFIs. The review highlighted the small effects of single interventions such as audit and feedback, computerised decision support and opinion leaders. MFIs claimed an improvement in effectiveness over single interventions, although effect sizes remained small to moderate and this improvement in effectiveness relating to MFIs has been questioned in a subsequent review [ 10 ]. In updating the review, we anticipated a larger pool of reviews and an opportunity to consolidate learning from more recent systematic reviews of interventions.

This review updates and extends our previous review of systematic reviews of interventions designed to implement research evidence into clinical practice. To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. [ 9 ] and Boaz, Baeza and Fraser [ 8 ] overview articles. To ensure optimal retrieval, our search strategy was refined with support from an expert university librarian, considering the ongoing improvements in the development of search filters for systematic reviews since our first review [ 11 ]. We also wanted to include technology-related terms (e.g. apps, algorithms, machine learning, artificial intelligence) to find studies that explored interventions based on the use of technological innovations as mechanistic tools for increasing the use of evidence into practice (see Additional file 1 : Appendix A for full search strategy).

The search was performed in June 2022 in the following electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched for articles published since the 2011 review. We searched from January 2010 up to June 2022 and applied no language restrictions. Reference lists of relevant papers were also examined.

We uploaded the results using EPPI-Reviewer, a web-based tool that facilitated semi-automation of the screening process and removal of duplicate studies. We made particular use of a priority screening function to reduce screening workload and avoid ‘data deluge’ [ 12 ]. Through machine learning, one reviewer screened a smaller number of records ( n  = 1200) to train the software to predict whether a given record was more likely to be relevant or irrelevant, thus pulling the relevant studies towards the beginning of the screening process. This automation did not replace manual work but helped the reviewer to identify eligible studies more quickly. During the selection process, we included studies that looked explicitly at interventions designed to turn research evidence into practice. Studies were included if they met the following pre-determined inclusion criteria:

The study was a systematic review

Search terms were included

Focused on the implementation of research evidence into practice

The methodological quality of the included studies was assessed as part of the review

Study populations included healthcare providers and patients. The EPOC taxonomy [ 13 ] was used to categorise the strategies. The EPOC taxonomy has four domains: delivery arrangements, financial arrangements, governance arrangements and implementation strategies. The implementation strategies domain includes 20 strategies targeted at healthcare workers. Numerous EPOC strategies were assessed in the review including educational strategies, local opinion leaders, reminders, ICT-focused approaches and audit and feedback. Some strategies that did not fit easily within the EPOC categories were also included. These were social media strategies and toolkits, and multi-faceted interventions (MFIs) (see Table  2 ). Some systematic reviews included comparisons of different interventions while other reviews compared one type of intervention against a control group. Outcomes related to improvements in health care processes or patient well-being. Numerous individual study types (RCT, CCT, BA, ITS) were included within the systematic reviews.

We excluded papers that:

Focused on changing patient rather than provider behaviour

Had no demonstrable outcomes

Made unclear or no reference to research evidence

The last of these criteria was sometimes difficult to judge, and there was considerable discussion amongst the research team as to whether the link between research evidence and practice was sufficiently explicit in the interventions analysed. As we discussed in the previous review [ 8 ] in the field of healthcare, the principle of evidence-based practice is widely acknowledged and tools to change behaviour such as guidelines are often seen to be an implicit codification of evidence, despite the fact that this is not always the case.

Reviewers employed a two-stage process to select papers for inclusion. First, all titles and abstracts were screened by one reviewer to determine whether the study met the inclusion criteria. Two papers [ 14 , 15 ] were identified that fell just before the 2010 cut-off. As they were not identified in the searches for the first review [ 8 ] they were included and progressed to assessment. Each paper was rated as include, exclude or maybe. The full texts of 111 relevant papers were assessed independently by at least two authors. To reduce the risk of bias, papers were excluded following discussion between all members of the team. 32 papers met the inclusion criteria and proceeded to data extraction. The study selection procedure is documented in a PRISMA literature flow diagram (see Fig.  1 ). We were able to include French, Spanish and Portuguese papers in the selection reflecting the language skills in the study team, but none of the papers identified met the inclusion criteria. Other non- English language papers were excluded.

figure 1

PRISMA flow diagram. Source: authors

One reviewer extracted data on strategy type, number of included studies, local, target population, effectiveness and scope of impact from the included studies. Two reviewers then independently read each paper and noted key findings and broad themes of interest which were then discussed amongst the wider authorial team. Two independent reviewers appraised the quality of included studies using a Quality Assessment Checklist based on Oxman and Guyatt [ 16 ] and Francke et al. [ 17 ]. Each study was rated a quality score ranging from 1 (extensive flaws) to 7 (minimal flaws) (see Additional file 2 : Appendix B). All disagreements were resolved through discussion. Studies were not excluded in this updated overview based on methodological quality as we aimed to reflect the full extent of current research into this topic.

The extracted data were synthesised using descriptive and narrative techniques to identify themes and patterns in the data linked to intervention strategies, targeted behaviours, study settings and study outcomes.

Thirty-two studies were included in the systematic review. Table 1. provides a detailed overview of the included systematic reviews comprising reference, strategy type, quality score, number of included studies, local, target population, effectiveness and scope of impact (see Table  1. at the end of the manuscript). Overall, the quality of the studies was high. Twenty-three studies scored 7, six studies scored 6, one study scored 5, one study scored 4 and one study scored 3. The primary focus of the review was on reviews of effectiveness studies, but a small number of reviews did include data from a wider range of methods including qualitative studies which added to the analysis in the papers [ 18 , 19 , 20 , 21 ]. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. In this section, we discuss the different EPOC-defined implementation strategies in turn. Interestingly, we found only two ‘new’ approaches in this review that did not fit into the existing EPOC approaches. These are a review focused on the use of social media and a review considering toolkits. In addition to single interventions, we also discuss multi-faceted interventions. These were the most common intervention approach overall. A summary is provided in Table  2 .

Educational strategies

The overview identified three systematic reviews focusing on educational strategies. Grudniewicz et al. [ 22 ] explored the effectiveness of printed educational materials on primary care physician knowledge, behaviour and patient outcomes and concluded they were not effective in any of these aspects. Koota, Kääriäinen and Melender [ 23 ] focused on educational interventions promoting evidence-based practice among emergency room/accident and emergency nurses and found that interventions involving face-to-face contact led to significant or highly significant effects on patient benefits and emergency nurses’ knowledge, skills and behaviour. Interventions using written self-directed learning materials also led to significant improvements in nurses’ knowledge of evidence-based practice. Although the quality of the studies was high, the review primarily included small studies with low response rates, and many of them relied on self-assessed outcomes; consequently, the strength of the evidence for these outcomes is modest. Wu et al. [ 20 ] questioned if educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes. Although based on evaluation projects and qualitative data, their results also suggest that positive changes on patient outcomes can be made following the implementation of specific evidence-based approaches (or projects). The differing positive outcomes for educational strategies aimed at nurses might indicate that the target audience is important.

Local opinion leaders

Flodgren et al. [ 24 ] was the only systemic review focusing solely on opinion leaders. The review found that local opinion leaders alone, or in combination with other interventions, can be effective in promoting evidence‐based practice, but this varies both within and between studies and the effect on patient outcomes is uncertain. The review found that, overall, any intervention involving opinion leaders probably improves healthcare professionals’ compliance with evidence-based practice but varies within and across studies. However, how opinion leaders had an impact could not be determined because of insufficient details were provided, illustrating that reporting specific details in published studies is important if diffusion of effective methods of increasing evidence-based practice is to be spread across a system. The usefulness of this review is questionable because it cannot provide evidence of what is an effective opinion leader, whether teams of opinion leaders or a single opinion leader are most effective, or the most effective methods used by opinion leaders.

Pantoja et al. [ 26 ] was the only systemic review focusing solely on manually generated reminders delivered on paper included in the overview. The review explored how these affected professional practice and patient outcomes. The review concluded that manually generated reminders delivered on paper as a single intervention probably led to small to moderate increases in adherence to clinical recommendations, and they could be used as a single quality improvement intervention. However, the authors indicated that this intervention would make little or no difference to patient outcomes. The authors state that such a low-tech intervention may be useful in low- and middle-income countries where paper records are more likely to be the norm.

ICT-focused approaches

The three ICT-focused reviews [ 14 , 27 , 28 ] showed mixed results. Jamal, McKenzie and Clark [ 14 ] explored the impact of health information technology on the quality of medical and health care. They examined the impact of electronic health record, computerised provider order-entry, or decision support system. This showed a positive improvement in adherence to evidence-based guidelines but not to patient outcomes. The number of studies included in the review was low and so a conclusive recommendation could not be reached based on this review. Similarly, Brown et al. [ 28 ] found that technology-enabled knowledge translation interventions may improve knowledge of health professionals, but all eight studies raised concerns of bias. The De Angelis et al. [ 27 ] review was more promising, reporting that ICT can be a good way of disseminating clinical practice guidelines but conclude that it is unclear which type of ICT method is the most effective.

Audit and feedback

Sykes, McAnuff and Kolehmainen [ 29 ] examined whether audit and feedback were effective in dementia care and concluded that it remains unclear which ingredients of audit and feedback are successful as the reviewed papers illustrated large variations in the effectiveness of interventions using audit and feedback.

Non-EPOC listed strategies: social media, toolkits

There were two new (non-EPOC listed) intervention types identified in this review compared to the 2011 review — fewer than anticipated. We categorised a third — ‘care bundles’ [ 36 ] as a multi-faceted intervention due to its description in practice and a fourth — ‘Technology Enhanced Knowledge Transfer’ [ 28 ] was classified as an ICT-focused approach. The first new strategy was identified in Bhatt et al.’s [ 30 ] systematic review of the use of social media for the dissemination of clinical practice guidelines. They reported that the use of social media resulted in a significant improvement in knowledge and compliance with evidence-based guidelines compared with more traditional methods. They noted that a wide selection of different healthcare professionals and patients engaged with this type of social media and its global reach may be significant for low- and middle-income countries. This review was also noteworthy for developing a simple stepwise method for using social media for the dissemination of clinical practice guidelines. However, it is debatable whether social media can be classified as an intervention or just a different way of delivering an intervention. For example, the review discussed involving opinion leaders and patient advocates through social media. However, this was a small review that included only five studies, so further research in this new area is needed. Yamada et al. [ 31 ] draw on 39 studies to explore the application of toolkits, 18 of which had toolkits embedded within larger KT interventions, and 21 of which evaluated toolkits as standalone interventions. The individual component strategies of the toolkits were highly variable though the authors suggest that they align most closely with educational strategies. The authors conclude that toolkits as either standalone strategies or as part of MFIs hold some promise for facilitating evidence use in practice but caution that the quality of many of the primary studies included is considered weak limiting these findings.

Multi-faceted interventions

The majority of the systematic reviews ( n  = 20) reported on more than one intervention type. Some of these systematic reviews focus exclusively on multi-faceted interventions, whilst others compare different single or combined interventions aimed at achieving similar outcomes in particular settings. While these two approaches are often described in a similar way, they are actually quite distinct from each other as the former report how multiple strategies may be strategically combined in pursuance of an agreed goal, whilst the latter report how different strategies may be incidentally used in sometimes contrasting settings in the pursuance of similar goals. Ariyo et al. [ 35 ] helpfully summarise five key elements often found in effective MFI strategies in LMICs — but which may also be transferrable to HICs. First, effective MFIs encourage a multi-disciplinary approach acknowledging the roles played by different professional groups to collectively incorporate evidence-informed practice. Second, they utilise leadership drawing on a wide set of clinical and non-clinical actors including managers and even government officials. Third, multiple types of educational practices are utilised — including input from patients as stakeholders in some cases. Fourth, protocols, checklists and bundles are used — most effectively when local ownership is encouraged. Finally, most MFIs included an emphasis on monitoring and evaluation [ 35 ]. In contrast, other studies offer little information about the nature of the different MFI components of included studies which makes it difficult to extrapolate much learning from them in relation to why or how MFIs might affect practice (e.g. [ 28 , 38 ]). Ultimately, context matters, which some review authors argue makes it difficult to say with real certainty whether single or MFI strategies are superior (e.g. [ 21 , 27 ]). Taking all the systematic reviews together we may conclude that MFIs appear to be more likely to generate positive results than single interventions (e.g. [ 34 , 45 ]) though other reviews should make us cautious (e.g. [ 32 , 43 ]).

While multi-faceted interventions still seem to be more effective than single-strategy interventions, there were important distinctions between how the results of reviews of MFIs are interpreted in this review as compared to the previous reviews [ 8 , 9 ], reflecting greater nuance and debate in the literature. This was particularly noticeable where the effectiveness of MFIs was compared to single strategies, reflecting developments widely discussed in previous studies [ 10 ]. We found that most systematic reviews are bounded by their clinical, professional, spatial, system, or setting criteria and often seek to draw out implications for the implementation of evidence in their areas of specific interest (such as nursing or acute care). Frequently this means combining all relevant studies to explore the respective foci of each systematic review. Therefore, most reviews we categorised as MFIs actually include highly variable numbers and combinations of intervention strategies and highly heterogeneous original study designs. This makes statistical analyses of the type used by Squires et al. [ 10 ] on the three reviews in their paper not possible. Further, it also makes extrapolating findings and commenting on broad themes complex and difficult. This may suggest that future research should shift its focus from merely examining ‘what works’ to ‘what works where and what works for whom’ — perhaps pointing to the value of realist approaches to these complex review topics [ 48 , 49 ] and other more theory-informed approaches [ 50 ].

Some reviews have a relatively small number of studies (i.e. fewer than 10) and the authors are often understandably reluctant to engage with wider debates about the implications of their findings. Other larger studies do engage in deeper discussions about internal comparisons of findings across included studies and also contextualise these in wider debates. Some of the most informative studies (e.g. [ 35 , 40 ]) move beyond EPOC categories and contextualise MFIs within wider systems thinking and implementation theory. This distinction between MFIs and single interventions can actually be very useful as it offers lessons about the contexts in which individual interventions might have bounded effectiveness (i.e. educational interventions for individual change). Taken as a whole, this may also then help in terms of how and when to conjoin single interventions into effective MFIs.

In the two previous reviews, a consistent finding was that MFIs were more effective than single interventions [ 8 , 9 ]. However, like Squires et al. [ 10 ] this overview is more equivocal on this important issue. There are four points which may help account for the differences in findings in this regard. Firstly, the diversity of the systematic reviews in terms of clinical topic or setting is an important factor. Secondly, there is heterogeneity of the studies within the included systematic reviews themselves. Thirdly, there is a lack of consistency with regards to the definition and strategies included within of MFIs. Finally, there are epistemological differences across the papers and the reviews. This means that the results that are presented depend on the methods used to measure, report, and synthesise them. For instance, some reviews highlight that education strategies can be useful to improve provider understanding — but without wider organisational or system-level change, they may struggle to deliver sustained transformation [ 19 , 44 ].

It is also worth highlighting the importance of the theory of change underlying the different interventions. Where authors of the systematic reviews draw on theory, there is space to discuss/explain findings. We note a distinction between theoretical and atheoretical systematic review discussion sections. Atheoretical reviews tend to present acontextual findings (for instance, one study found very positive results for one intervention, and this gets highlighted in the abstract) whilst theoretically informed reviews attempt to contextualise and explain patterns within the included studies. Theory-informed systematic reviews seem more likely to offer more profound and useful insights (see [ 19 , 35 , 40 , 43 , 45 ]). We find that the most insightful systematic reviews of MFIs engage in theoretical generalisation — they attempt to go beyond the data of individual studies and discuss the wider implications of the findings of the studies within their reviews drawing on implementation theory. At the same time, they highlight the active role of context and the wider relational and system-wide issues linked to implementation. It is these types of investigations that can help providers further develop evidence-based practice.

This overview has identified a small, but insightful set of papers that interrogate and help theorise why, how, for whom, and in which circumstances it might be the case that MFIs are superior (see [ 19 , 35 , 40 ] once more). At the level of this overview — and in most of the systematic reviews included — it appears to be the case that MFIs struggle with the question of attribution. In addition, there are other important elements that are often unmeasured, or unreported (e.g. costs of the intervention — see [ 40 ]). Finally, the stronger systematic reviews [ 19 , 35 , 40 , 43 , 45 ] engage with systems issues, human agency and context [ 18 ] in a way that was not evident in the systematic reviews identified in the previous reviews [ 8 , 9 ]. The earlier reviews lacked any theory of change that might explain why MFIs might be more effective than single ones — whereas now some systematic reviews do this, which enables them to conclude that sometimes single interventions can still be more effective.

As Nilsen et al. ([ 6 ] p. 7) note ‘Study findings concerning the effectiveness of various approaches are continuously synthesized and assembled in systematic reviews’. We may have gone as far as we can in understanding the implementation of evidence through systematic reviews of single and multi-faceted interventions and the next step would be to conduct more research exploring the complex and situated nature of evidence used in clinical practice and by particular professional groups. This would further build on the nuanced discussion and conclusion sections in a subset of the papers we reviewed. This might also support the field to move away from isolating individual implementation strategies [ 6 ] to explore the complex processes involving a range of actors with differing capacities [ 51 ] working in diverse organisational cultures. Taxonomies of implementation strategies do not fully account for the complex process of implementation, which involves a range of different actors with different capacities and skills across multiple system levels. There is plenty of work to build on, particularly in the social sciences, which currently sits at the margins of debates about evidence implementation (see for example, Normalisation Process Theory [ 52 ]).

There are several changes that we have identified in this overview of systematic reviews in comparison to the review we published in 2011 [ 8 ]. A consistent and welcome finding is that the overall quality of the systematic reviews themselves appears to have improved between the two reviews, although this is not reflected upon in the papers. This is exhibited through better, clearer reporting mechanisms in relation to the mechanics of the reviews, alongside a greater attention to, and deeper description of, how potential biases in included papers are discussed. Additionally, there is an increased, but still limited, inclusion of original studies conducted in low- and middle-income countries as opposed to just high-income countries. Importantly, we found that many of these systematic reviews are attuned to, and comment upon the contextual distinctions of pursuing evidence-informed interventions in health care settings in different economic settings. Furthermore, systematic reviews included in this updated article cover a wider set of clinical specialities (both within and beyond hospital settings) and have a focus on a wider set of healthcare professions — discussing both similarities, differences and inter-professional challenges faced therein, compared to the earlier reviews. These wider ranges of studies highlight that a particular intervention or group of interventions may work well for one professional group but be ineffective for another. This diversity of study settings allows us to consider the important role context (in its many forms) plays on implementing evidence into practice. Examining the complex and varied context of health care will help us address what Nilsen et al. ([ 6 ] p. 1) described as, ‘society’s health problems [that] require research-based knowledge acted on by healthcare practitioners together with implementation of political measures from governmental agencies’. This will help us shift implementation science to move, ‘beyond a success or failure perspective towards improved analysis of variables that could explain the impact of the implementation process’ ([ 6 ] p. 2).

This review brings together 32 papers considering individual and multi-faceted interventions designed to support the use of evidence in clinical practice. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been conducted. As a whole, this substantial body of knowledge struggles to tell us more about the use of individual and MFIs than: ‘it depends’. To really move forwards in addressing the gap between research evidence and practice, we may need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of perspectives, especially from the social, economic, political and behavioural sciences in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed. Harvey et al. [ 53 ] suggest that when context is likely to be critical to implementation success there are a range of primary research approaches (participatory research, realist evaluation, developmental evaluation, ethnography, quality/ rapid cycle improvement) that are likely to be appropriate and insightful. While these approaches often form part of implementation studies in the form of process evaluations, they are usually relatively small scale in relation to implementation research as a whole. As a result, the findings often do not make it into the subsequent systematic reviews. This review provides further evidence that we need to bring qualitative approaches in from the periphery to play a central role in many implementation studies and subsequent evidence syntheses. It would be helpful for systematic reviews, at the very least, to include more detail about the interventions and their implementation in terms of how and why they worked.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Before and after study

Controlled clinical trial

Effective Practice and Organisation of Care

High-income countries

Information and Communications Technology

Interrupted time series

Knowledge translation

Low- and middle-income countries

Randomised controlled trial

Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30. https://doi.org/10.1016/S0140-6736(03)14546-1 .

Article   PubMed   Google Scholar  

Green LA, Seifert CM. Translation of research into practice: why we can’t “just do it.” J Am Board Fam Pract. 2005;18:541–5. https://doi.org/10.3122/jabfm.18.6.541 .

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1–3. https://doi.org/10.1186/1748-5908-1-1 .

Article   PubMed Central   Google Scholar  

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:2–14. https://doi.org/10.1186/s13012-015-0209-1 .

Article   Google Scholar  

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:1–8. https://doi.org/10.1186/s13012-015-0295-0 .

Nilsen P, Ståhl C, Roback K, et al. Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Sci. 2013;8:2–12. https://doi.org/10.1186/1748-5908-8-63 .

Rycroft-Malone J, Seers K, Eldh AC, et al. A realist process evaluation within the Facilitating Implementation of Research Evidence (FIRE) cluster randomised controlled international trial: an exemplar. Implementation Sci. 2018;13:1–15. https://doi.org/10.1186/s13012-018-0811-0 .

Boaz A, Baeza J, Fraser A, European Implementation Score Collaborative Group (EIS). Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4:212. https://doi.org/10.1186/1756-0500-4-212 .

Article   PubMed   PubMed Central   Google Scholar  

Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior – an overview of systematic reviews of interventions. Med Care. 2001;39 8Suppl 2:II2–45.

Google Scholar  

Squires JE, Sullivan K, Eccles MP, et al. Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implement Sci. 2014;9:1–22. https://doi.org/10.1186/s13012-014-0152-6 .

Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Development of an efficient search filter to retrieve systematic reviews from PubMed. J Med Libr Assoc. 2021;109:561–74. https://doi.org/10.5195/jmla.2021.1223 .

Thomas JM. Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evid Based Med. 2013;1:1–6.

Effective Practice and Organisation of Care (EPOC). The EPOC taxonomy of health systems interventions. EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services; 2016. epoc.cochrane.org/epoc-taxonomy . Accessed 9 Oct 2023.

Jamal A, McKenzie K, Clark M. The impact of health information technology on the quality of medical and health care: a systematic review. Health Inf Manag. 2009;38:26–37. https://doi.org/10.1177/183335830903800305 .

Menon A, Korner-Bitensky N, Kastner M, et al. Strategies for rehabilitation professionals to move evidence-based knowledge into practice: a systematic review. J Rehabil Med. 2009;41:1024–32. https://doi.org/10.2340/16501977-0451 .

Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–8. https://doi.org/10.1016/0895-4356(91)90160-b .

Article   CAS   PubMed   Google Scholar  

Francke AL, Smit MC, de Veer AJ, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:1–11. https://doi.org/10.1186/1472-6947-8-38 .

Jones CA, Roop SC, Pohar SL, et al. Translating knowledge in rehabilitation: systematic review. Phys Ther. 2015;95:663–77. https://doi.org/10.2522/ptj.20130512 .

Scott D, Albrecht L, O’Leary K, Ball GDC, et al. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7:1–17. https://doi.org/10.1186/1748-5908-7-70 .

Wu Y, Brettle A, Zhou C, Ou J, et al. Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review. Nurse Educ Today. 2018;70:109–14. https://doi.org/10.1016/j.nedt.2018.08.026 .

Yost J, Ganann R, Thompson D, Aloweni F, et al. The effectiveness of knowledge translation interventions for promoting evidence-informed decision-making among nurses in tertiary care: a systematic review and meta-analysis. Implement Sci. 2015;10:1–15. https://doi.org/10.1186/s13012-015-0286-1 .

Grudniewicz A, Kealy R, Rodseth RN, Hamid J, et al. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci. 2015;10:2–12. https://doi.org/10.1186/s13012-015-0347-5 .

Koota E, Kääriäinen M, Melender HL. Educational interventions promoting evidence-based practice among emergency nurses: a systematic review. Int Emerg Nurs. 2018;41:51–8. https://doi.org/10.1016/j.ienj.2018.06.004 .

Flodgren G, O’Brien MA, Parmelli E, et al. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD000125.pub5 .

Arditi C, Rège-Walther M, Durieux P, et al. Computer-generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2017. https://doi.org/10.1002/14651858.CD001175.pub4 .

Pantoja T, Grimshaw JM, Colomer N, et al. Manually-generated reminders delivered on paper: effects on professional practice and patient outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD001174.pub4 .

De Angelis G, Davies B, King J, McEwan J, et al. Information and communication technologies for the dissemination of clinical practice guidelines to health professionals: a systematic review. JMIR Med Educ. 2016;2:e16. https://doi.org/10.2196/mededu.6288 .

Brown A, Barnes C, Byaruhanga J, McLaughlin M, et al. Effectiveness of technology-enabled knowledge translation strategies in improving the use of research in public health: systematic review. J Med Internet Res. 2020;22:e17274. https://doi.org/10.2196/17274 .

Sykes MJ, McAnuff J, Kolehmainen N. When is audit and feedback effective in dementia care? A systematic review. Int J Nurs Stud. 2018;79:27–35. https://doi.org/10.1016/j.ijnurstu.2017.10.013 .

Bhatt NR, Czarniecki SW, Borgmann H, et al. A systematic review of the use of social media for dissemination of clinical practice guidelines. Eur Urol Focus. 2021;7:1195–204. https://doi.org/10.1016/j.euf.2020.10.008 .

Yamada J, Shorkey A, Barwick M, Widger K, et al. The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review. BMJ Open. 2015;5:e006808. https://doi.org/10.1136/bmjopen-2014-006808 .

Afari-Asiedu S, Abdulai MA, Tostmann A, et al. Interventions to improve dispensing of antibiotics at the community level in low and middle income countries: a systematic review. J Glob Antimicrob Resist. 2022;29:259–74. https://doi.org/10.1016/j.jgar.2022.03.009 .

Boonacker CW, Hoes AW, Dikhoff MJ, Schilder AG, et al. Interventions in health care professionals to improve treatment in children with upper respiratory tract infections. Int J Pediatr Otorhinolaryngol. 2010;74:1113–21. https://doi.org/10.1016/j.ijporl.2010.07.008 .

Al Zoubi FM, Menon A, Mayo NE, et al. The effectiveness of interventions designed to increase the uptake of clinical practice guidelines and best practices among musculoskeletal professionals: a systematic review. BMC Health Serv Res. 2018;18:2–11. https://doi.org/10.1186/s12913-018-3253-0 .

Ariyo P, Zayed B, Riese V, Anton B, et al. Implementation strategies to reduce surgical site infections: a systematic review. Infect Control Hosp Epidemiol. 2019;3:287–300. https://doi.org/10.1017/ice.2018.355 .

Borgert MJ, Goossens A, Dongelmans DA. What are effective strategies for the implementation of care bundles on ICUs: a systematic review. Implement Sci. 2015;10:1–11. https://doi.org/10.1186/s13012-015-0306-1 .

Cahill LS, Carey LM, Lannin NA, et al. Implementation interventions to promote the uptake of evidence-based practices in stroke rehabilitation. Cochrane Database Syst Rev. 2020. https://doi.org/10.1002/14651858.CD012575.pub2 .

Pedersen ER, Rubenstein L, Kandrack R, Danz M, et al. Elusive search for effective provider interventions: a systematic review of provider interventions to increase adherence to evidence-based treatment for depression. Implement Sci. 2018;13:1–30. https://doi.org/10.1186/s13012-018-0788-8 .

Jenkins HJ, Hancock MJ, French SD, Maher CG, et al. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ. 2015;187:401–8. https://doi.org/10.1503/cmaj.141183 .

Bennett S, Laver K, MacAndrew M, Beattie E, et al. Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies. Int Psychogeriatr. 2021;33:947–75. https://doi.org/10.1017/S1041610220001702 .

Noonan VK, Wolfe DL, Thorogood NP, et al. Knowledge translation and implementation in spinal cord injury: a systematic review. Spinal Cord. 2014;52:578–87. https://doi.org/10.1038/sc.2014.62 .

Albrecht L, Archibald M, Snelgrove-Clarke E, et al. Systematic review of knowledge translation strategies to promote research uptake in child health settings. J Pediatr Nurs. 2016;31:235–54. https://doi.org/10.1016/j.pedn.2015.12.002 .

Campbell A, Louie-Poon S, Slater L, et al. Knowledge translation strategies used by healthcare professionals in child health settings: an updated systematic review. J Pediatr Nurs. 2019;47:114–20. https://doi.org/10.1016/j.pedn.2019.04.026 .

Bird ML, Miller T, Connell LA, et al. Moving stroke rehabilitation evidence into practice: a systematic review of randomized controlled trials. Clin Rehabil. 2019;33:1586–95. https://doi.org/10.1177/0269215519847253 .

Goorts K, Dizon J, Milanese S. The effectiveness of implementation strategies for promoting evidence informed interventions in allied healthcare: a systematic review. BMC Health Serv Res. 2021;21:1–11. https://doi.org/10.1186/s12913-021-06190-0 .

Zadro JR, O’Keeffe M, Allison JL, Lembke KA, et al. Effectiveness of implementation strategies to improve adherence of physical therapist treatment choices to clinical practice guidelines for musculoskeletal conditions: systematic review. Phys Ther. 2020;100:1516–41. https://doi.org/10.1093/ptj/pzaa101 .

Van der Veer SN, Jager KJ, Nache AM, et al. Translating knowledge on best practice into improving quality of RRT care: a systematic review of implementation strategies. Kidney Int. 2011;80:1021–34. https://doi.org/10.1038/ki.2011.222 .

Pawson R, Greenhalgh T, Harvey G, et al. Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10Suppl 1:21–34. https://doi.org/10.1258/1355819054308530 .

Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implementation Sci. 2012;7:1–10. https://doi.org/10.1186/1748-5908-7-33 .

Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open. 2015;5:e008592. https://doi.org/10.1136/bmjopen-2015-008592 .

Metz A, Jensen T, Farley A, Boaz A, et al. Is implementation research out of step with implementation practice? Pathways to effective implementation support over the last decade. Implement Res Pract. 2022;3:1–11. https://doi.org/10.1177/26334895221105585 .

May CR, Finch TL, Cornford J, Exley C, et al. Integrating telecare for chronic disease management in the community: What needs to be done? BMC Health Serv Res. 2011;11:1–11. https://doi.org/10.1186/1472-6963-11-131 .

Harvey G, Rycroft-Malone J, Seers K, Wilson P, et al. Connecting the science and practice of implementation – applying the lens of context to inform study design in implementation research. Front Health Serv. 2023;3:1–15. https://doi.org/10.3389/frhs.2023.1162762 .

Download references

Acknowledgements

The authors would like to thank Professor Kathryn Oliver for her support in the planning the review, Professor Steve Hanney for reading and commenting on the final manuscript and the staff at LSHTM library for their support in planning and conducting the literature search.

This study was supported by LSHTM’s Research England QR strategic priorities funding allocation and the National Institute for Health and Care Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust. Grant number NIHR200152. The views expressed are those of the author(s) and not necessarily those of the NIHR, the Department of Health and Social Care or Research England.

Author information

Authors and affiliations.

Health and Social Care Workforce Research Unit, The Policy Institute, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE, UK

Annette Boaz

King’s Business School, King’s College London, 30 Aldwych, London, WC2B 4BG, UK

Juan Baeza & Alec Fraser

Federal University of Santa Catarina (UFSC), Campus Universitário Reitor João Davi Ferreira Lima, Florianópolis, SC, 88.040-900, Brazil

Erik Persson

You can also search for this author in PubMed   Google Scholar

Contributions

AB led the conceptual development and structure of the manuscript. EP conducted the searches and data extraction. All authors contributed to screening and quality appraisal. EP and AF wrote the first draft of the methods section. AB, JB and AF performed result synthesis and contributed to the analyses. AB wrote the first draft of the manuscript and incorporated feedback and revisions from all other authors. All authors revised and approved the final manuscript.

Corresponding author

Correspondence to Annette Boaz .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix a., additional file 2: appendix b., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Boaz, A., Baeza, J., Fraser, A. et al. ‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implementation Sci 19 , 15 (2024). https://doi.org/10.1186/s13012-024-01337-z

Download citation

Received : 01 November 2023

Accepted : 05 January 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s13012-024-01337-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Implementation
  • Interventions
  • Clinical practice
  • Research evidence
  • Multi-faceted

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

systematic review health care databases

  • Open access
  • Published: 17 February 2024

Global prevalence of intimate partner violence during the COVID-19 pandemic among women: systematic review and meta-analysis

  • Mearg Eyasu Kifle 1 ,
  • Setognal Birara Aychiluhm 2 &
  • Etsay Woldu Anbesu 2  

BMC Women's Health volume  24 , Article number:  127 ( 2024 ) Cite this article

827 Accesses

Metrics details

During the coronavirus pandemic, people faced strict preventive measures, including staying at home and maintaining social distance, which led to increasing rates of intimate partner violence. Women have been facing dual health emergencies, including COVID-19 and domestic violence. Despite this, there is a lack of representative data on intimate partner violence during the COVID-19 pandemic and inconsistent findings.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines were used to develop the systematic review and meta-analysis. All English-language studies conducted between 31 December 2019 and May 15/2022 were extracted from databases such as PubMed/Medline, CINAHL, and Google Scholar. The quality of the articles was assessed using the Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI). The I 2 was used to assess heterogeneity among studies. Publication bias was assessed using funnel plot inspection and Egger’s test. A random effect model was used for the analysis using RevMan and STATA 14 software.

A total of 5065 studies were retrieved, and 14 studies were included in the final meta-analysis. The pooled prevalence of intimate partner violence was 31% (95% CI: 22, 40). Subgroup analysis based on region showed that the highest prevalence of intimate partner violence was in developing regions (33, 95% CI: 23.0, 43.0) compared to developed regions (14, 95% CI: 11.0, 17.0). Subgroup analysis based on country showed that Uganda had the highest prevalence of IPV 68% (95% CI: 62.0, 72.0), and the lowest was in the USA 10% (95% CI: 7.0, 15.0).

Nearly one in three women experienced intimate partner violence during the COVID-19 pandemic. Subgroup analysis based on region showed that the highest prevalence of intimate partner violence was in developing regions (33%). All forms of intimate partner violence (physical, sexual, emotional, and economic) were prevalent. Thus, available interventions should be implemented to alleviate women’s intimate partner violence during the COVID-19 pandemic and similar emerging and remerging pandemics, particularly in developing countries.

Trial registration

PROSPERO registration number: CRD42022334613 .

Peer Review reports

Introduction

Gender-based violence (GBV) is any cruelty directed at individuals based on their sex, gender identity, or socially defined way of femaleness and maleness [ 1 , 2 , 3 ]. Violence against women is the primary form of GBV and is a basic violation of women’s human rights [ 1 , 2 , 3 , 4 , 5 ]. Threats, coercion, and denial of liberty against women are some of the violence against women [ 5 , 6 , 7 ]. The main actors of violence against women are male partners, including husbands, fiancées, or ex-partners, often referred to as intimate partners [ 5 , 6 , 7 , 8 ]. The World Health Organization (WHO) defines intimate partner violence as any behaviour within an intimate relationship by an intimate partner that causes physical, psychological, and sexual harm to those in the relationship, and it is one of the most common types of violence experienced by women [ 7 , 8 , 9 ].

Intimate partner violence is a serious, highly prevalent, preventable public health problem that violates women’s rights [ 10 ]. It has been exacerbated during the COVID-19 pandemic following control and prevention actions such as isolation, stay-at-home, and movement restrictions, targeted at reducing the pandemic have brought vulnerable women and potential perpetrators under the confines of the home setting and have increased the risk of IPV [ 11 , 12 , 13 ]. Globally, one in three women experiences physical, sexual, or psychological harm from an intimate partner or ex-partner [ 14 , 15 ]. The World Health Organization (WHO) and European Commission evidence indicated a ‘shadow pandemic’, with the strong potential of increased IPV across the globe as seen during the Ebola pandemic. At the beginning of the pandemic (March–April), community-based victim organizations reported a 25–50% increase in hotline calls, up to a 150% increase in website traffic, and a 12.5% increase in IPV-related police activity [ 16 , 17 , 18 , 19 ].

Lockdown declarations following the COVID-19 pandemic in several countries of developed countries increased intimate partner violence by 20%, 21–35%, 32–36%, and 30–50% [ 12 , 20 ]. In Africa, approximately 36.6% of women experience lifetime physical or sexual IPV [ 7 ]. During the COVID-19 pandemic, in Kenya (35%), Somalia (50%), South Africa (10660), Niger (499 cases) and Ethiopia (12.9%), Intimate partner violence has been reported to be as high as before [ 21 ]. In Ethiopia, more than 100 girls have been raped during COVID-19 within less than 2 months, and some of them are close family members [ 13 ].

Intimate partner violence has a complex and multifaceted health outcome, including physical, mental, sexual, and reproductive health issues, which, in turn, result in a high degree of women’s morbidity and mortality [ 22 ]. A study performed by the WHO showed that women who experienced violence were twice as likely to have an abortion and doubled their likelihood of falling into depression [ 23 ]. Approximately 41% of female IPV survivors experience some form of physical injury [ 24 ]. IPV can also extend beyond physical injury and result in death. Data from U.S. crime reports suggest that 16% of murder victims are killed by an intimate partner and that over 40% of female homicide victims in the U.S. are killed by an intimate partner [ 25 ].

There are policies and strategies implemented to overcome the problem at the global or local level just before and after the pandemic, including teaching safe and healthy relationship skills, engaging influential adults and peers, disrupting developmental pathways toward IPV, creating protective environments, strengthening economic support for families, and supporting survivors. Increased safety and lessened harm, commitment, cooperation, and leadership from numerous sectors, including public health, education, justice, health care, social services, business and labor, and government [ 26 , 27 , 28 , 29 , 30 , 31 ]. Despite this intervention, intimate partner violence remains a major public health problem during the COVID-19 pandemic. Moreover, there is a lack of representative data on intimate partner violence during the COVID-19 pandemic and inconsistent findings. Therefore, this systematic review and meta-analysis aimed to estimate the pooled prevalence of intimate partner violence during the COVID-19 pandemic among women.

Protocol and registration

These systematic reviews and meta-analyses were registered with the International Prospective Register of Systematic Reviews PROSPERO with an ID number (CRD42022334613) available at https://www.crd.york.ac.uk/prospero/#myprospero .

Form of violence

Physical violence includes slapping, hitting, kicking and beating.

Sexual violence includes forced sexual intercourse and other forms of sexual coercion.

Emotional (psychological) abuse includes insults, belittling, constant humiliation, intimidation (e.g., destroying things), threats of harm, and threats to take away children.

Controlling behaviours , including isolating a person from family and friends; monitoring their movements; and restricting access to financial resources, employment, education or medical care.

Search strategy and appraisal of studies

All published studies conducted in different countries that reported intimate partner violence during COVID-19 from December 2019 to May 2022 were included. The search was limited to peer-reviewed, indexed scientific journals and written in English. “The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 32 ] were used to develop the present systematic review using PRISMA checklist − 2020 (supplementary material file  1 ).

Article searches were conducted in databases including PubMed/MEDLINE, CINAHL, and Google Scholar. Medical Subject Headings (MeSH) terms and entry terms were used to search studies, and amendments were made based on the types of databases. The key terms and entry terms were connected by Boolean operators (supplementary material file  2 ). Screening was conducted independently by both authors (ME and EW), and disagreements were resolved through discussion with the third author (SB). Using a snowballing technique, references of eligible studies and relevant reviews were also searched.

Eligibility criteria

The study included studies performed in different countries (globally), observational study designs included cross-sectional and cohort studies, published and unpublished studies, studies that reported the prevalence of intimate partner violence during COVID-19, only quantitative results for studies that reported both quantitative and qualitative results, English published articles, women having intimate partner violence, and studies conducted since COVID-19 identified in Wuhan, China from December 2019 to May 2022 (databases search date may 15–30/2022) were included. However, studies other than English, articles with no available full text and no response for relevant missing data after email contact with the corresponding author, case and conference reports, reviews, letters, and qualitative results for studies that reported both quantitative and qualitative results were excluded.

CoCoPop/PEO

Condition : Women’s intimate partner violence during the COVID-19 pandemic.

Context: worldwide.

Population: women with partners.

Exposure of interest : exposure is a determinant that increases or decreases the likelihood of intimate partner violence during the COVID-19 pandemic. The determinants can be but are not limited to age, residence, husbands’ educational status, decision-making power, social support, wealth index, history of abortion, arranged marriage, history of child death, controlling behaviour of the husband, and COVID-19 pandemic.

Outcome/condition : The outcome of the study was the pooled prevalence of intimate partner violence during COVID-19. Intimate partner violence includes physical, sexual, and emotional abuse and controlling behaviours by an intimate partner [ 7 ].

Study selection

Two independent reviewers (ME and EW) screened the searched studies. Duplicate articles were removed, assessments of articles using their titles and abstracts were performed, and irrelevant titles and abstracts were removed. A full-text review of relevant studies was performed before the inclusion of studies in the final meta-analysis. Disagreements among reviewers during the review process were resolved through discussion with the third author (SB). Endnote reference manager software [ 33 ] was used to collect and remove duplicate, irrelevant titles and abstracts.

Quality assessment

During the screening process, two independent reviewers (ME and EW.) performed the quality assessment and evaluated the risk of bias in eligible studies. The “Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI)” tool was used to critically appraise the quality of the studies (supplementary material file  3 ) [ 34 ]. The components of the quality assessment were clear inclusion criteria, study population and setting , measurement criteria, event and exposure measurements, and appropriate statistical analysis . During the critical quality appraisal of the studies, any disagreement among the authors was resolved by discussion with the third author (SB).

Data extraction

Data were extracted independently by two authors (ME and EW) using a pilot test data extraction Excel sheet and RevMan software. The outcome data extraction format contains the authors’ names, publication year, countries, study design, study setting, and sample size. Any disagreement was resolved through discussion with the third author (SB). In the case of incomplete results, email contact with the corresponding author was made, and articles were excluded if no response was made.

Statistical analysis

The final included studies were imported to STATA version 14 to determine the pooled prevalence. The results were reported in narrative descriptions, tables, and graphs. A random-effects model was used to estimate the true effect at the 95% CI [ 35 ].

The results were reported using a forest plot with respective odds ratios and 95% CIs. Heterogeneity among the included studies was assessed by visual graphical inspection of the forest plot [ 36 ] and statistically using the I 2 statistic [ 37 ]. I 2 statistics of 25, 50, and 75% indicated low, moderate, and increased levels of heterogeneity, respectively, with p  < 0.05.

Publication bias was identified using visual inspection of the funnel plot. In addition, evidence of publication bias was assessed statistically using Egger’s tests [ 38 ] at p  < 0.05. The differences in heterogeneity between the studies were performed by subgroup analysis and meta-regression [ 39 ] based on country, study area (developed/developing), and sample size.

A total of 5065 articles were collected from PubMed/MED-LINE, CINAHL, and Google Scholar. All articles were imported into EndNote software (version X8; Thomson Reuters, New York, NY), and 37 articles were excluded due to duplications. A total of 4884 articles were excluded after a review of their titles and abstracts. A total of 144 articles were assessed for eligibility based on the preset criteria. A total of 130 articles were excluded because the outcome of interest was not reported, and qualitative studies were excluded. Finally, 14 articles were eligible and included in this meta-analysis (Fig.  1 ).

figure 1

Flow chart of study selection for meta-analysis of IPV among women during the COVID-19 pandemic, 2022

Quality appraisal

All the included studies met a minimum of four out of eight (50% and above) JBI critical appraisal scores. The criteria for inclusion were clearly defined in all studies. Strategies to address confounding factors and appropriate statistics were made in all included studies. However, since all the included studies were cross-sectional studies, the identification of confounding factors was not applicable for this study (supplementary material file  4 ).

Characteristics of included studies

All 14 studies in this systematic review and meta-analysis were cross-sectional studies; eight of them were conducted in Ethiopia [ 4 , 13 , 40 , 41 , 42 , 43 , 44 , 45 ], one was conducted in Congo [ 46 ], one was conducted in Uganda [ 47 ], one was conducted in Bangladesh [ 48 ], one was conducted in Arab countries [ 49 ], one was conducted in Canada [ 50 ], and one was conducted in the USA [ 51 ]. A total of 8335 women with intimate partners were involved in our study. The sample size of the studies ranged from 216 [ 50 ] to 2002 [ 46 ]. In this review, the lowest prevalence (7.1%) of intimate partner violence was in St. Paul’s Hospital, Ethiopia [ 45 ], while the highest prevalence (68%) of IPV was reported in Uganda 48 (Table  1 ).

Pooled prevalence of any form of intimate partner violence among women during COVID-19

The pooled prevalence of intimate partner violence among women was 31% (95% CI: 22, 40)). In this review, the lowest prevalence (7.1%) of IPV was in St. Paul’s Hospital, Ethiopia [ 45 ], while the highest prevalence (68%) of intimate partner violence was reported in Uganda [ 51 ]. The included studies exhibited significant heterogeneity (I 2  = 99.07, p  < 0.001) (Fig.  2 ).

figure 2

Forest plot showing the pooled prevalence of IPV among women during the COVID-19 pandemic

Subgroup analysis

Subgroup analysis was performed based on region (developed/developing) and country to identify the possible source of heterogeneity. Subgroup analysis based on region showed that the highest prevalence of intimate partner violence was in developing regions (33, 95% CI: 23.0, 43.0) compared to developed regions (14, 95% CI: 11.0, 17.0). High heterogeneity was reported in developing countries (I 2 = 99.19; p  < 0.001) (Fig.  3 ).

figure 3

Forest plot showing subgroup analysis on IPV among women during the COVID-19 pandemic by region

Subgroup analysis based on country showed that Uganda had the highest prevalence of intimate partner violence among women (68, 95% CI: 62.0, 72.0), and the lowest was in the USA (10, 95% CI: 7.0, 15.0%). High heterogeneity was reported in studies performed in Ethiopia (I 2 = 98.78; p  < 0.001). Ethiopia had the highest weight of 57.30, and the possible reason may be the high number of studies performed and included in that area, and the lowest weight was in Canada, 7.02 (Fig.  4 ).

figure 4

Forest plot subgroup prevalence of IPV among women during the COVID-19 pandemic by country

Metaregression

Meta-regression was performed to identify the source of heterogeneity across the studies by considering continuous and categorical variables, including region (developed/developing), country, and sample size. Meta-regression indicated that no heterogeneity was observed ( p value> 0.05) (Table  2 ).

Publication bias: On visual inspection, asymmetry was observed in the funnel plots since there were six studies on the right and eight studies on the left (Fig.  5 ). However, the results from Egger’s regression test did not show statistical significance ( p  = 0.345) (Table  3 ).

figure 5

Funnel plot for publication bias, IPV among women during the COVID-19 pandemic

Sensitivity analysis

The results showed that no single study unduly influenced the overall estimate of intimate partner violence during the COVID-19 pandemic and its associated factors (supplementary Figure file S 1 ) .

Forms of intimate partner violence

In this study, the prevalence was calculated for each form of intimate partner violence.

Controlling violence

The prevalence of controlling violence in one study during the pandemic was 54% (95% CI: 49, 60) [ 47 ] (Fig.  6 ).

figure 6

Forest plot showing the prevalence of controlling violence among women during the COVID-19 pandemic

Verbal violence

The pooled prevalence of verbal violence faced by women with intimate partners during the pandemic in two studies was 53% (95% CI: 51, 56) [ 46 , 49 ] (Fig.  7 ).

figure 7

Forest plot showing the pooled prevalence of verbal among women during the COVID-19 pandemic

Emotional violence

The pooled prevalence of emotional violence faced by women with intimate partners during the pandemic in 13 studies was 25% (95% CI: 17, 32) [ 4 , 13 , 40 , 41 , 42 , 43 , 44 , 45 , 47 , 48 , 49 , 50 , 52 ] (Fig.  8 ).

figure 8

Forest plot showing the pooled prevalence of emotional violence among women during the COVID-19 pandemic

Economic violence

The pooled prevalence of economic violence faced by women with intimate partners during the pandemic in two studies was 17% (95% CI: 15, 20) [ 47 , 49 ] (Fig.  9 ).

figure 9

Sexual violence

The pooled prevalence of sexual violence faced by women with intimate partners during the pandemic in 14 studies was 14% (95% CI: 10, 18) [ 4 , 13 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 52 ] (Fig.  10 ).

figure 10

Forest plot showing the pooled prevalence of sexual violence among women during the COVID-19 pandemic

Physical violence

The pooled prevalence of physical violence faced by women with intimate partners during the pandemic in 14 studies was 14% (95% CI: 9, 18) [ 4 , 13 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 52 ] (Fig.  11 ).

figure 11

Forest plot showing the pooled prevalence of physical violence among women during the COVID-19 pandemic

This systematic review and meta-analysis aimed to estimate the pooled prevalence of intimate partner violence during the COVID-19 pandemic. To the best of our knowledge, no systematic review or meta-analysis has been conducted on the pooled prevalence of intimate partner violence during the COVID-19 pandemic. Moreover, there is a lack of representative data on intimate partner violence during the COVID-19 pandemic, and there are also inconsistent findings. Therefore, this systematic review and meta-analysis will help policy-makers, programmers, planners, clinicians, and researchers design appropriate strategies.

The pooled prevalence of any form of IPV among women during the COVID-19 pandemic was 31% (95% CI: 22–40). This prevalence was comparable to a systematic review performed before the pandemic of 30% [ 53 ] and 27% [ 54 ] and during the pandemic of 31% [ 55 ] and 33.4% [ 56 ]. However, the pooled prevalence was higher than that in studies performed before the pandemic: sub-Saharan Africa, 20% [ 54 ]; northern Africa, 15% [ 57 ]; southern Asia, 19% [ 14 ]; western Asia, 13% [ 7 ]; African countries, 15.23% [ 58 ]; China, 7.7% [ 59 ]; and France, 7% [ 60 ]. Moreover, it was also higher than studies performed in the United States, 18.0% [ 17 ], Ethiopia, 26.6% [ 40 ], and Arab countries, 22.2% [ 49 ], during the pandemic. Our finding was lower than those of a study conducted in Peru (48.0% [ 61 ]), New Orleans (59% [ 62 ]), Jordan (40% [ 63 ]), Iran (65.4% [ 64 ]), and Bangladesh (45.29% [ 48 ]). The difference might be due to differences in sample size, study setting, study period, availability, and access to health services, reproductive health information, geographical areas, and the cultures of study subjects.

In this study, the prevalence of each component of IPV during the pandemic was also determined as controlling violence, verbal violence, emotional violence, economic violence, sexual violence and physical violence, which are the prevalent forms of violence faced during the pandemic by women with intimate partners.

The limitation of this study is that it includes only articles published in the English language. Databases such as Scopus and EMBASE were not considered due to the lack of free access, and we recommend funding to expand the database search source. Additionally, all included studies in this meta-analysis were cross-sectional; as a result, the outcome variables could be affected by other confounding variables, and cause and effect relationships could not be determined. Furthermore, studies from seven countries fulfilled the eligibility criteria and may not be representative. Despite these limitations, searching, selection and data extraction of the studies were performed based on eligibility criteria independently by two authors, and ambiguity was resolved by a third author.

Conclusions

Availability of data and materials.

All data generated or analysed during the current study are included in this manuscript and its supplementary information files.

Abbreviations

Coronavirus

Cumulative Index to Nursing and Allied Health Literature

Intimate Partner Violence

Medical Search Headings

Preferred Reporting Items for Systematic Review & Meta-analysis

Review Manager Software

Goldfarb S, Goldscheid J. International human rights law on violence against women and children and its impact on domestic law and action. In: Women and children as victims and offenders: background, prevention, reintegration. Springer; 2016. p. 3–45.

Google Scholar  

Russo NF, Pirlott A. Gender-based violence: concepts, methods, and findings. Ann N Y Acad Sci. 2006;1087(1):178–205.

Article   ADS   PubMed   Google Scholar  

Hossain M, McAlpine A. Gender based violence research methodologies in humanitarian settings: an evidence review and recommendations; 2017.

Ehitemariyam T. Assessment of intimate partner violence against women of reproductive age and associated factors during covid 19 pandemic in Debre Berhan Town, Ethiopia 2021: a community–based cross-sectional study 2021.

Garcia-Moreno C, Jansen HAFM, Ellsberg M, Heise L, Watts C. WHO multicountry study on women’s health and domestic violence against women: initial results on prevalence, health outcomes and women’s responses. Geneva: World Health Organization; 2005.

WHO P. Understanding and addressing violence against women. Geneva, Switzerland: The World Health Organization; 2012.

García-Moreno C, Pallitto C, Devries K, Stöckl H, Watts C, Abrahams N. Global and regional estimates of violence against women: prevalence and health effects of intimate partner violence and non-partner sexual violence. World Health Organization; 2013.

Krug EG, Mercy JA, Dahlberg LL, Zwi AB. The world report on violence and health. Lancet. 2002;360(9339):1083–8.

Article   PubMed   Google Scholar  

Benebo FO, Schumann B, Vaezghasemi M. Intimate partner violence against women in Nigeria: a multilevel study investigating the effect of women’s status and community norms. BMC Womens Health. 2018;18(1):1–17.

Article   Google Scholar  

Tochie JN, Ofakem I, Ayissi G, Endomba FT, Fobellah NN, Wouatong C, et al. Intimate partner violence during the confinement period of the COVID-19 pandemic: exploring the French and Cameroonian public health policies. Pan Afr Med J. 2020;35(Suppl 2)

Duncan TK, Weaver JL, Zakrison TL, Joseph B, Campbell BT, Christmas AB, et al. Domestic violence and safe storage of firearms in the COVID-19 era. Ann Surg. 2020;272(2):e55.

Hatchimonji JS, Swendiman RA, Seamon MJ, Nance ML. Trauma does not quarantine: violence during the COVID-19 pandemic. Ann Surg. 2020;272(2):e53.

Shitu S, Yeshaneh A, Abebe H. Intimate partner violence and associated factors among reproductive age women during COVID-19 pandemic in southern Ethiopia, 2020. Reprod Health. 2021;18(1):1–10.

Devries KM, Mak JY, Garcia-Moreno C, Petzold M, Child JC, Falder G, et al. The global prevalence of intimate partner violence against women. Science. 2013;340(6140):1527–8.

Article   ADS   CAS   PubMed   Google Scholar  

Gaines C. Commentary on a cochrane review of screening for intimate partner violence in health care settings. Nurs Womens Health. 2017;21(6):439–41.

Organization WH. COVID-19 and violence against women: what the health sector/system can do, 7 April 2020. World Health Organization; 2020.

Jetelina KK, Knell G, Molsberry RJ. Changes in intimate partner violence during the early stages of the COVID-19 pandemic in the USA. Inj Prev. 2021;27(1):93–7.

Committee IR. “Everything on her shoulders”. Rapid assessment on gender and violence against women and girls in the Ebola outbreak in Beni, DRC. New York: International Rescue Committee Accessed March. 2019;8:2021.

Piquero AR, Riddell JR, Bishopp SA, Narvey C, Reid JA, Piquero NL. Staying home, staying safe? A short-term analysis of COVID-19 on Dallas domestic violence. Am J Crim Justice. 2020;45(4):601–35.

Article   PubMed   PubMed Central   Google Scholar  

Bradley NL, DiPasquale AM, Dillabough K, Schneider PS. Health care practitioners’ responsibility to address intimate partner violence related to the COVID-19 pandemic. CMAJ. 2020;192(22):E609–E10.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Forum ACP, editor Under siege impact of COVID-19 on girls in Africa2020: PLAN International/African Child Policy Forum (ACPF).

Sere Y, Roman NV, Ruiter RA. Coping with the experiences of intimate partner violence among south African women: systematic review and meta-synthesis. Front Psychiatry. 2021:522.

Usher K, Bhullar N, Durkin J, Gyamfi N, Jackson D. Family violence and COVID-19: increased vulnerability and reduced options for support. Int J Ment Health Nurs. 2020;

Breiding MJ, Chen J, Black MC. Intimate partner violence in the United States--2010; 2014.

Cooper A, Smith EL. Homicide trends in the United States, 1980–2008. Washington, DC: Bureau of Justice Statistics; 2011.

Capaldi DM, Knoble NB, Shortt JW, Kim HK. A systematic review of risk factors for intimate partner violence. Partn Abus. 2012;3(2):231–80.

Browning CR. The span of collective efficacy: extending social disorganization theory to partner violence. J Marriage Fam. 2002;64(4):833–50.

Matjasko JL, Niolon PH, Valle LA. The role of economic factors and economic support in preventing and escaping from intimate partner violence. J Policy Anal Manag. 2013;32(1):122.

Banyard VL. Toward the next generation of bystander prevention of sexual and relationship violence: action coils to engage communities. Springer; 2015.

Book   Google Scholar  

Wasserman G, Seracini A, Loeber R, Farrington D. Child delinquents: development, intervention, and service needs. Thousand Oaks, CA: Sage Publication; 2001.

McMahon S, Stepleton K, Cusano J, O’Connor J, Gandhi K, McGinty F. Beyond sexual assault surveys: a model for comprehensive campus climate assessments. J Stud Aff Res Pract. 2018;55(1):78–90.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88:105906.

Agrawal A, Rasouli M. EndNote 1–2-3 easy!: reference management for the professional. Springer Nature; 2019.

Munn Z, Tufanaru C, Aromataris E. JBI's systematic reviews: data extraction and synthesis. American J Nurs. 2014;114(7):49–54.

Berkey CS, Hoaglin DC, Mosteller F, Colditz GA. A random-effects regression model for meta-analysis. Stat Med. 1995;14(4):395–411.

Article   CAS   PubMed   Google Scholar  

Rücker G, Schwarzer G. Beyond the forest plot: the drapery plot. Res Synth Methods. 2021;12(1):13–9.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.

Pustejovsky JE, Rodgers MA. Testing for funnel plot asymmetry of standardized mean differences. Res Synth Methods. 2019;10(1):57–71.

Spineli LM, Pandis N. Problems and pitfalls in subgroup analysis and meta-regression. Am J Orthod Dentofac Orthop. 2020;158(6):901–4.

Gebrewahd GT, Gebremeskel GG, Tadesse DB. Intimate partner violence against reproductive age women during COVID-19 pandemic in northern Ethiopia 2020: a community-based cross-sectional study. Reprod Health. 2020;17(1):1–8.

Fetene G, Alie MS, Girma D, Negesse Y. Prevalence and its predictors of intimate partner violence against pregnant women amid COVID-19 pandemic in Southwest Ethiopia, 2021: a cross-sectional study. SAGE Open Med. 2022;10:20503121221079317.

Getinet W, Azale T, Getie E, Salelaw E, Amare T, Demilew D, et al. Intimate partner violence among reproductive-age women in Central Gondar zone, northwest, Ethiopia: a population-based study. BMC Womens Health. 2022;22(1):109.

Shewangzaw Engda A, Dargie Wubetu A, Kasahun Amogne F, Moltot KT. Intimate partner violence and COVID-19 among reproductive age women: a community-based cross-sectional survey, Ethiopia. Womens Health. 2022;18:17455065211068980.

CAS   Google Scholar  

Tadesse AW, Tarekegn SM, Wagaw GB, Muluneh MD, Kassa AM. Prevalence and associated factors of intimate partner violence among married women during COVID-19 pandemic restrictions: a community-based study. J Interpers Violence. 2022;37(11–12):NP8632–NP50.

Teshome A, Gudu W, Bekele D, Asfaw M, Enyew R, Compton SD. Intimate partner violence among prenatal care attendees amidst the COVID-19 crisis: the incidence in Ethiopia. Int J Gynecol Obstet. 2021;153(1):45–50.

Article   CAS   Google Scholar  

Ditekemena JD, Luhata C, Mavoko HM, Siewe Fodjo JN, Nkamba DM, Van Damme W, et al. Intimate partners violence against women during a COVID-19 lockdown period: results of an online survey in 7 provinces of the Democratic Republic of Congo. Int J Environ Res Public Health. 2021;18(10)

Katushabe E, Chinweuba A, Omieibi A, Asiimwe JB. Prevalence and determinants of intimate-partner violence among pregnant women attending a City health Centre IV, south western Uganda, during the COVID-19 pandemic: a cross-sectional study. Stud J Health Res Afr. 2022;3(3):17.

Rayhan I, Akter K. Prevalence and associated factors of intimate partner violence (IPV) against women in Bangladesh amid COVID-19 pandemic. Heliyon. 2021;7(3):e06619.

El-Nimr NA, Mamdouh HM, Ramadan A, El Saeh HM, Shata ZN. Intimate partner violence among Arab women before and during the COVID-19 lockdown. J Egypt Public Health Assoc. 2021;96(1):1–8.

Muldoon KA, Denize KM, Talarico R, Boisvert C, Frank O, Harvey AL, et al. COVID-19 and perinatal intimate partner violence: a cross-sectional survey of pregnant and postpartum individuals in the early stages of the COVID-19 pandemic. BMJ Open. 2021;11(5):e049295.

Cannon CE, Ferreira R, Buttell F, First J. COVID-19, intimate partner violence, and communication ecologies. Am Behav Sci. 2021;65(7):992–1013.

Spencer CM, Gimarc C, Durtschi J. COVID-19 specific risk markers for intimate partner violence perpetration. J Fam Violence. 2022;37(6):881–91.

García-Moreno C, Stöckl H. Violence against women, its prevalence and health consequences. Violence against Women and Mental Health. 178: Karger Publishers; 2013. p. 1–11.

Sardinha L, Maheu-Giroux M, Stöckl H, Meyer SR, García-Moreno C. Global, regional, and national prevalence estimates of physical or sexual, or both, intimate partner violence against women in 2018. Lancet. 2022;399(10327):803–13.

Berniell I, Facchini G. COVID-19 lockdown and domestic violence: evidence from internet-search behavior in 11 countries. Eur Econ Rev. 2021;136:103775.

Boxall H, Morgan A. Intimate partner violence during the COVID-19 pandemic: a survey of women in Australia. Australia's National Research Organisation for Women's Safety; 2021.

Maheu-Giroux M, Sardinha L, Stöckl H, Meyer SR, Godin A, Alexander M, et al. A framework to model global, regional, and national estimates of intimate partner violence. BMC Med Res Methodol. 2022;22(1):1–17.

Shamu S, Abrahams N, Temmerman M, Musekiwa A, Zarowsky C. A systematic review of African studies on intimate partner violence against pregnant women: prevalence and risk factors. PLoS One. 2011;6(3):e17591.

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Wang T, Liu Y, Li Z, Liu K, Xu Y, Shi W, et al. Prevalence of intimate partner violence (IPV) during pregnancy in China: a systematic review and meta-analysis. PLoS One. 2017;12(10):e0175108.

Peraud W, Quintard B, Constant A. Factors associated with violence against women following the COVID-19 lockdown in France: results from a prospective online survey. PLoS One. 2021;16(9):e0257193.

Agüero JM. COVID-19 and the rise of intimate partner violence. World Dev. 2021;137:105217.

Buttell F, Ferreira RJ. The hidden disaster of COVID-19: intimate partner violence. Psychol Trauma Theory Res Pract Policy. 2020;12(S1):S197.

Abuhammad S. Violence against Jordanian women during COVID-19 outbreak. Int J Clin Pract. 2021;75(3):e13824.

Fereidooni R, Mootz J, Sabaei R, Khoshnood K, Heydari ST, Moradian MJ, et al. The COVID-19 pandemic, socioeconomic effects, and intimate partner violence against women: A population-based cohort study in Iran. Maryam and Molavi Vardanjani, Hossein, The COVID-19 Pandemic, Socioeconomic Effects, and Intimate Partner Violence Against Women: A Population-Based Cohort Study in Iran. 2021.

Download references

Acknowledgements

We would like to thank Samara University for the HINARY database website and network access.

No funding.

Author information

Authors and affiliations.

Family Guidance Association, Logia, Afar, Ethiopia

Mearg Eyasu Kifle

Department of Public Health, College of Medical and Health Sciences, Samara University, Samara, Ethiopia

Setognal Birara Aychiluhm & Etsay Woldu Anbesu

You can also search for this author in PubMed   Google Scholar

Contributions

ME and EW are participated on screened ,data extraction and quality appraisal and SB a mediator that solves any disagrmeent with in the two authors.

Corresponding author

Correspondence to Mearg Eyasu Kifle .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., additional file 2., additional file 3., additional file 4., additional file 5., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kifle, M.E., Aychiluhm, S.B. & Anbesu, E.W. Global prevalence of intimate partner violence during the COVID-19 pandemic among women: systematic review and meta-analysis. BMC Women's Health 24 , 127 (2024). https://doi.org/10.1186/s12905-023-02845-8

Download citation

Received : 27 December 2022

Accepted : 14 December 2023

Published : 17 February 2024

DOI : https://doi.org/10.1186/s12905-023-02845-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Intimate partner violence
  • Pooled prevalence

BMC Women's Health

ISSN: 1472-6874

systematic review health care databases

Benefits and barriers associated with the use of smart home health technologies in the care of older persons: a systematic review

Affiliations.

  • 1 Institute for Biomedical Ethics, University of Basel, Basel, 4056, Switzerland. [email protected].
  • 2 Institute for Biomedical Ethics, University of Basel, Basel, 4056, Switzerland.
  • 3 Centre d'excellence sur le vieillissement de Québec, VITAM- Research Center on Sustainable Health, Laval University, Quebec City, QC, Canada.
  • 4 Division of Geriatrics, Department of Medicine, Laval University, Quebec City, QC, Canada.
  • 5 School of nursing sciences, La Source, HES-SO University of Applied Sciences and Arts of Western Switzerland, Lausanne, Switzerland.
  • PMID: 38355464
  • PMCID: PMC10865618
  • DOI: 10.1186/s12877-024-04702-1

Background: Smart home health technologies (SHHTs) have been discussed in the frame of caregiving to enable aging-in-place and independence. A systematic review was conducted in accordance with the PRISMA guidelines to gather the up-to-date knowledge on the benefits and barriers of using SHHTs in the care of older persons from the perspective of older persons and their caregivers.

Methods: Ten electronic databases were reviewed for empirical peer-reviewed literature published from 01.01.2000 to 31.12.2021 in English, German, and French reporting on experimental, qualitative, quantitative, and other empirical study designs were included. Included studies contained user-feedback from older persons over 65 years of age or their caregivers (formal and informal). We used an extraction document to collect relevant data from all included studies and applied narrative synthesis to analyze data related to benefits and barriers of SHHTs.

Results: 163 empirical peer-reviewed articles were included, the majority of those published between 2014 and 2021. Five first-order categories of benefits and five of barriers were found with individual sub-themes. SHHTs could be useful in the care context where continuous monitoring is needed. They improve self-management and independent living of older persons. Barriers currently exist with respect to ease of usability, social acceptance, and cost.

Conclusions: SHHTs could be useful in the care context but are not without concerns. Researchers and policy makers can use the information as a starting point to better understand how the roles and outcomes of SHHTs could be improved for the care of older persons, while caregivers of older adults could use our findings to comprehend the scope of SHHTs and to decide when and where such technology could best address their individual family needs. Limitations lie in the possible exclusion of relevant articles published outside the inclusion criteria as well as the fact that due to digital divide, our review represents opinions of those who could and wanted to participate in the included 163 studies.

Trial registration: This review has been registered as PROSPERO CRD42021248543. A protocol was completed in March 2021 with the PRISMA-P guidance. We have extended the review period from 2000 to 2020 since the registration of the protocol to 2000-2021.

Keywords: Caregiving; Home; Independent living; Older people; Smart home health technologies.

© 2024. The Author(s).

Publication types

  • Aged, 80 and over
  • Caregivers*
  • Independent Living
  • Meta-Analysis as Topic
  • Palliative Care*
  • Systematic Reviews as Topic

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • PMC10248995

Logo of sysrev

Guidance to best tools and practices for systematic reviews

Kat kolaski.

1 Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC USA

Lynne Romeiser Logan

2 Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY USA

John P. A. Ioannidis

3 Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA USA

Associated Data

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.

A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.

Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13643-023-02255-9.

Part 1. The state of evidence synthesis

Evidence syntheses are commonly regarded as the foundation of evidence-based medicine (EBM). They are widely accredited for providing reliable evidence and, as such, they have significantly influenced medical research and clinical practice. Despite their uptake throughout health care and ubiquity in contemporary medical literature, some important aspects of evidence syntheses are generally overlooked or not well recognized. Evidence syntheses are mostly retrospective exercises, they often depend on weak or irreparably flawed data, and they may use tools that have acknowledged or yet unrecognized limitations. They are complicated and time-consuming undertakings prone to bias and errors. Production of a good evidence synthesis requires careful preparation and high levels of organization in order to limit potential pitfalls [ 1 ]. Many authors do not recognize the complexity of such an endeavor and the many methodological challenges they may encounter. Failure to do so is likely to result in research and resource waste.

Given their potential impact on people’s lives, it is crucial for evidence syntheses to correctly report on the current knowledge base. In order to be perceived as trustworthy, reliable demonstration of the accuracy of evidence syntheses is equally imperative [ 2 ]. Concerns about the trustworthiness of evidence syntheses are not recent developments. From the early years when EBM first began to gain traction until recent times when thousands of systematic reviews are published monthly [ 3 ] the rigor of evidence syntheses has always varied. Many systematic reviews and meta-analyses had obvious deficiencies because original methods and processes had gaps, lacked precision, and/or were not widely known. The situation has improved with empirical research concerning which methods to use and standardization of appraisal tools. However, given the geometrical increase in the number of evidence syntheses being published, a relatively larger pool of unreliable evidence syntheses is being published today.

Publication of methodological studies that critically appraise the methods used in evidence syntheses is increasing at a fast pace. This reflects the availability of tools specifically developed for this purpose [ 4 – 6 ]. Yet many clinical specialties report that alarming numbers of evidence syntheses fail on these assessments. The syntheses identified report on a broad range of common conditions including, but not limited to, cancer, [ 7 ] chronic obstructive pulmonary disease, [ 8 ] osteoporosis, [ 9 ] stroke, [ 10 ] cerebral palsy, [ 11 ] chronic low back pain, [ 12 ] refractive error, [ 13 ] major depression, [ 14 ] pain, [ 15 ] and obesity [ 16 , 17 ]. The situation is even more concerning with regard to evidence syntheses included in clinical practice guidelines (CPGs) [ 18 – 20 ]. Astonishingly, in a sample of CPGs published in 2017–18, more than half did not apply even basic systematic methods in the evidence syntheses used to inform their recommendations [ 21 ].

These reports, while not widely acknowledged, suggest there are pervasive problems not limited to evidence syntheses that evaluate specific kinds of interventions or include primary research of a particular study design (eg, randomized versus non-randomized) [ 22 ]. Similar concerns about the reliability of evidence syntheses have been expressed by proponents of EBM in highly circulated medical journals [ 23 – 26 ]. These publications have also raised awareness about redundancy, inadequate input of statistical expertise, and deficient reporting. These issues plague primary research as well; however, there is heightened concern for the impact of these deficiencies given the critical role of evidence syntheses in policy and clinical decision-making.

Methods and guidance to produce a reliable evidence synthesis

Several international consortiums of EBM experts and national health care organizations currently provide detailed guidance (Table ​ (Table1). 1 ). They draw criteria from the reporting and methodological standards of currently recommended appraisal tools, and regularly review and update their methods to reflect new information and changing needs. In addition, they endorse the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system for rating the overall quality of a body of evidence [ 27 ]. These groups typically certify or commission systematic reviews that are published in exclusive databases (eg, Cochrane, JBI) or are used to develop government or agency sponsored guidelines or health technology assessments (eg, National Institute for Health and Care Excellence [NICE], Scottish Intercollegiate Guidelines Network [SIGN], Agency for Healthcare Research and Quality [AHRQ]). They offer developers of evidence syntheses various levels of methodological advice, technical and administrative support, and editorial assistance. Use of specific protocols and checklists are required for development teams within these groups, but their online methodological resources are accessible to any potential author.

Guidance for development of evidence syntheses

Notably, Cochrane is the largest single producer of evidence syntheses in biomedical research; however, these only account for 15% of the total [ 28 ]. The World Health Organization requires Cochrane standards be used to develop evidence syntheses that inform their CPGs [ 29 ]. Authors investigating questions of intervention effectiveness in syntheses developed for Cochrane follow the Methodological Expectations of Cochrane Intervention Reviews [ 30 ] and undergo multi-tiered peer review [ 31 , 32 ]. Several empirical evaluations have shown that Cochrane systematic reviews are of higher methodological quality compared with non-Cochrane reviews [ 4 , 7 , 9 , 11 , 14 , 32 – 35 ]. However, some of these assessments have biases: they may be conducted by Cochrane-affiliated authors, and they sometimes use scales and tools developed and used in the Cochrane environment and by its partners. In addition, evidence syntheses published in the Cochrane database are not subject to space or word restrictions, while non-Cochrane syntheses are often limited. As a result, information that may be relevant to the critical appraisal of non-Cochrane reviews is often removed or is relegated to online-only supplements that may not be readily or fully accessible [ 28 ].

Influences on the state of evidence synthesis

Many authors are familiar with the evidence syntheses produced by the leading EBM organizations but can be intimidated by the time and effort necessary to apply their standards. Instead of following their guidance, authors may employ methods that are discouraged or outdated 28]. Suboptimal methods described in in the literature may then be taken up by others. For example, the Newcastle–Ottawa Scale (NOS) is a commonly used tool for appraising non-randomized studies [ 36 ]. Many authors justify their selection of this tool with reference to a publication that describes the unreliability of the NOS and recommends against its use [ 37 ]. Obviously, the authors who cite this report for that purpose have not read it. Authors and peer reviewers have a responsibility to use reliable and accurate methods and not copycat previous citations or substandard work [ 38 , 39 ]. Similar cautions may potentially extend to automation tools. These have concentrated on evidence searching [ 40 ] and selection given how demanding it is for humans to maintain truly up-to-date evidence [ 2 , 41 ]. Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) and studies related to COVID-19, [ 2 , 42 ] but such tools are not yet commonly used [ 43 ]. The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.

Editorials about unreliable or misleading systematic reviews highlight several of the intertwining factors that may contribute to continued publication of unreliable evidence syntheses: shortcomings and inconsistencies of the peer review process, lack of endorsement of current standards on the part of journal editors, the incentive structure of academia, industry influences, publication bias, and the lure of “predatory” journals [ 44 – 48 ]. At this juncture, clarification of the extent to which each of these factors contribute remains speculative, but their impact is likely to be synergistic.

Over time, the generalized acceptance of the conclusions of systematic reviews as incontrovertible has affected trends in the dissemination and uptake of evidence. Reporting of the results of evidence syntheses and recommendations of CPGs has shifted beyond medical journals to press releases and news headlines and, more recently, to the realm of social media and influencers. The lay public and policy makers may depend on these outlets for interpreting evidence syntheses and CPGs. Unfortunately, communication to the general public often reflects intentional or non-intentional misrepresentation or “spin” of the research findings [ 49 – 52 ] News and social media outlets also tend to reduce conclusions on a body of evidence and recommendations for treatment to binary choices (eg, “do it” versus “don’t do it”) that may be assigned an actionable symbol (eg, red/green traffic lights, smiley/frowning face emoji).

Strategies for improvement

Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [ 46 , 53 ]. Informing them about research methodology could increase the likelihood they will apply rigorous methods [ 25 , 33 , 45 ]. We tackle this challenge, from both a theoretical and a practical perspective, by offering guidance applicable to any specialty. It is based on recent methodological research that is extensively referenced to promote self-study. However, the information presented is not intended to be substitute for committed training in evidence synthesis methodology; instead, we hope to inspire our target audience to seek such training. We also hope to inform a broader audience of clinicians and guideline developers influenced by evidence syntheses. Notably, these communities often include the same members who serve in different capacities.

In the following sections, we highlight methodological concepts and practices that may be unfamiliar, problematic, confusing, or controversial. In Part 2, we consider various types of evidence syntheses and the types of research evidence summarized by them. In Part 3, we examine some widely used (and misused) tools for the critical appraisal of systematic reviews and reporting guidelines for evidence syntheses. In Part 4, we discuss how to meet methodological conduct standards applicable to key components of systematic reviews. In Part 5, we describe the merits and caveats of rating the overall certainty of a body of evidence. Finally, in Part 6, we summarize suggested terminology, methods, and tools for development and evaluation of evidence syntheses that reflect current best practices.

Part 2. Types of syntheses and research evidence

A good foundation for the development of evidence syntheses requires an appreciation of their various methodologies and the ability to correctly identify the types of research potentially available for inclusion in the synthesis.

Types of evidence syntheses

Systematic reviews have historically focused on the benefits and harms of interventions; over time, various types of systematic reviews have emerged to address the diverse information needs of clinicians, patients, and policy makers [ 54 ] Systematic reviews with traditional components have become defined by the different topics they assess (Table 2.1 ). In addition, other distinctive types of evidence syntheses have evolved, including overviews or umbrella reviews, scoping reviews, rapid reviews, and living reviews. The popularity of these has been increasing in recent years [ 55 – 58 ]. A summary of the development, methods, available guidance, and indications for these unique types of evidence syntheses is available in Additional File 2 A.

Types of traditional systematic reviews

Both Cochrane [ 30 , 59 ] and JBI [ 60 ] provide methodologies for many types of evidence syntheses; they describe these with different terminology, but there is obvious overlap (Table 2.2 ). The majority of evidence syntheses published by Cochrane (96%) and JBI (62%) are categorized as intervention reviews. This reflects the earlier development and dissemination of their intervention review methodologies; these remain well-established [ 30 , 59 , 61 ] as both organizations continue to focus on topics related to treatment efficacy and harms. In contrast, intervention reviews represent only about half of the total published in the general medical literature, and several non-intervention review types contribute to a significant proportion of the other half.

Evidence syntheses published by Cochrane and JBI

a Data from https://www.cochranelibrary.com/cdsr/reviews . Accessed 17 Sep 2022

b Data obtained via personal email communication on 18 Sep 2022 with Emilie Francis, editorial assistant, JBI Evidence Synthesis

c Includes the following categories: prevalence, scoping, mixed methods, and realist reviews

d This methodology is not supported in the current version of the JBI Manual for Evidence Synthesis

Types of research evidence

There is consensus on the importance of using multiple study designs in evidence syntheses; at the same time, there is a lack of agreement on methods to identify included study designs. Authors of evidence syntheses may use various taxonomies and associated algorithms to guide selection and/or classification of study designs. These tools differentiate categories of research and apply labels to individual study designs (eg, RCT, cross-sectional). A familiar example is the Design Tree endorsed by the Centre for Evidence-Based Medicine [ 70 ]. Such tools may not be helpful to authors of evidence syntheses for multiple reasons.

Suboptimal levels of agreement and accuracy even among trained methodologists reflect challenges with the application of such tools [ 71 , 72 ]. Problematic distinctions or decision points (eg, experimental or observational, controlled or uncontrolled, prospective or retrospective) and design labels (eg, cohort, case control, uncontrolled trial) have been reported [ 71 ]. The variable application of ambiguous study design labels to non-randomized studies is common, making them especially prone to misclassification [ 73 ]. In addition, study labels do not denote the unique design features that make different types of non-randomized studies susceptible to different biases, including those related to how the data are obtained (eg, clinical trials, disease registries, wearable devices). Given this limitation, it is important to be aware that design labels preclude the accurate assignment of non-randomized studies to a “level of evidence” in traditional hierarchies [ 74 ].

These concerns suggest that available tools and nomenclature used to distinguish types of research evidence may not uniformly apply to biomedical research and non-health fields that utilize evidence syntheses (eg, education, economics) [ 75 , 76 ]. Moreover, primary research reports often do not describe study design or do so incompletely or inaccurately; thus, indexing in PubMed and other databases does not address the potential for misclassification [ 77 ]. Yet proper identification of research evidence has implications for several key components of evidence syntheses. For example, search strategies limited by index terms using design labels or study selection based on labels applied by the authors of primary studies may cause inconsistent or unjustified study inclusions and/or exclusions [ 77 ]. In addition, because risk of bias (RoB) tools consider attributes specific to certain types of studies and study design features, results of these assessments may be invalidated if an inappropriate tool is used. Appropriate classification of studies is also relevant for the selection of a suitable method of synthesis and interpretation of those results.

An alternative to these tools and nomenclature involves application of a few fundamental distinctions that encompass a wide range of research designs and contexts. While these distinctions are not novel, we integrate them into a practical scheme (see Fig. ​ Fig.1) 1 ) designed to guide authors of evidence syntheses in the basic identification of research evidence. The initial distinction is between primary and secondary studies. Primary studies are then further distinguished by: 1) the type of data reported (qualitative or quantitative); and 2) two defining design features (group or single-case and randomized or non-randomized). The different types of studies and study designs represented in the scheme are described in detail in Additional File 2 B. It is important to conceptualize their methods as complementary as opposed to contrasting or hierarchical [ 78 ]; each offers advantages and disadvantages that determine their appropriateness for answering different kinds of research questions in an evidence synthesis.

An external file that holds a picture, illustration, etc.
Object name is 13643_2023_2255_Fig1_HTML.jpg

Distinguishing types of research evidence

Application of these basic distinctions may avoid some of the potential difficulties associated with study design labels and taxonomies. Nevertheless, debatable methodological issues are raised when certain types of research identified in this scheme are included in an evidence synthesis. We briefly highlight those associated with inclusion of non-randomized studies, case reports and series, and a combination of primary and secondary studies.

Non-randomized studies

When investigating an intervention’s effectiveness, it is important for authors to recognize the uncertainty of observed effects reported by studies with high RoB. Results of statistical analyses that include such studies need to be interpreted with caution in order to avoid misleading conclusions [ 74 ]. Review authors may consider excluding randomized studies with high RoB from meta-analyses. Non-randomized studies of intervention (NRSI) are affected by a greater potential range of biases and thus vary more than RCTs in their ability to estimate a causal effect [ 79 ]. If data from NRSI are synthesized in meta-analyses, it is helpful to separately report their summary estimates [ 6 , 74 ].

Nonetheless, certain design features of NRSI (eg, which parts of the study were prospectively designed) may help to distinguish stronger from weaker ones. Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [ 79 , 80 ] This process is facilitated by a study design feature checklist; guidance on using the checklist is included with developers’ description of the tool [ 73 , 74 ]. Authors collect information about these design features during data extraction and then consider it when making final study selection decisions and when performing RoB assessments of the included NRSI.

Case reports and case series

Correctly identified case reports and case series can contribute evidence not well captured by other designs [ 81 ]; in addition, some topics may be limited to a body of evidence that consists primarily of uncontrolled clinical observations. Murad and colleagues offer a framework for how to include case reports and series in an evidence synthesis [ 82 ]. Distinguishing between cohort studies and case series in these syntheses is important, especially for those that rely on evidence from NRSI. Additional data obtained from studies misclassified as case series can potentially increase the confidence in effect estimates. Mathes and Pieper provide authors of evidence syntheses with specific guidance on distinguishing between cohort studies and case series, but emphasize the increased workload involved [ 77 ].

Primary and secondary studies

Synthesis of combined evidence from primary and secondary studies may provide a broad perspective on the entirety of available literature on a topic. This is, in fact, the recommended strategy for scoping reviews that may include a variety of sources of evidence (eg, CPGs, popular media). However, except for scoping reviews, the synthesis of data from primary and secondary studies is discouraged unless there are strong reasons to justify doing so.

Combining primary and secondary sources of evidence is challenging for authors of other types of evidence syntheses for several reasons [ 83 ]. Assessments of RoB for primary and secondary studies are derived from conceptually different tools, thus obfuscating the ability to make an overall RoB assessment of a combination of these study types. In addition, authors who include primary and secondary studies must devise non-standardized methods for synthesis. Note this contrasts with well-established methods available for updating existing evidence syntheses with additional data from new primary studies [ 84 – 86 ]. However, a new review that synthesizes data from primary and secondary studies raises questions of validity and may unintentionally support a biased conclusion because no existing methodological guidance is currently available [ 87 ].

Recommendations

We suggest that journal editors require authors to identify which type of evidence synthesis they are submitting and reference the specific methodology used for its development. This will clarify the research question and methods for peer reviewers and potentially simplify the editorial process. Editors should announce this practice and include it in the instructions to authors. To decrease bias and apply correct methods, authors must also accurately identify the types of research evidence included in their syntheses.

Part 3. Conduct and reporting

The need to develop criteria to assess the rigor of systematic reviews was recognized soon after the EBM movement began to gain international traction [ 88 , 89 ]. Systematic reviews rapidly became popular, but many were very poorly conceived, conducted, and reported. These problems remain highly prevalent [ 23 ] despite development of guidelines and tools to standardize and improve the performance and reporting of evidence syntheses [ 22 , 28 ]. Table 3.1  provides some historical perspective on the evolution of tools developed specifically for the evaluation of systematic reviews, with or without meta-analysis.

Tools specifying standards for systematic reviews with and without meta-analysis

a Currently recommended

b Validated tool for systematic reviews of interventions developed for use by authors of overviews or umbrella reviews

These tools are often interchangeably invoked when referring to the “quality” of an evidence synthesis. However, quality is a vague term that is frequently misused and misunderstood; more precisely, these tools specify different standards for evidence syntheses. Methodological standards address how well a systematic review was designed and performed [ 5 ]. RoB assessments refer to systematic flaws or limitations in the design, conduct, or analysis of research that distort the findings of the review [ 4 ]. Reporting standards help systematic review authors describe the methodology they used and the results of their synthesis in sufficient detail [ 92 ]. It is essential to distinguish between these evaluations: a systematic review may be biased, it may fail to report sufficient information on essential features, or it may exhibit both problems; a thoroughly reported systematic evidence synthesis review may still be biased and flawed while an otherwise unbiased one may suffer from deficient documentation.

We direct attention to the currently recommended tools listed in Table 3.1  but concentrate on AMSTAR-2 (update of AMSTAR [A Measurement Tool to Assess Systematic Reviews]) and ROBIS (Risk of Bias in Systematic Reviews), which evaluate methodological quality and RoB, respectively. For comparison and completeness, we include PRISMA 2020 (update of the 2009 Preferred Reporting Items for Systematic Reviews of Meta-Analyses statement), which offers guidance on reporting standards. The exclusive focus on these three tools is by design; it addresses concerns related to the considerable variability in tools used for the evaluation of systematic reviews [ 28 , 88 , 96 , 97 ]. We highlight the underlying constructs these tools were designed to assess, then describe their components and applications. Their known (or potential) uptake and impact and limitations are also discussed.

Evaluation of conduct

Development.

AMSTAR [ 5 ] was in use for a decade prior to the 2017 publication of AMSTAR-2; both provide a broad evaluation of methodological quality of intervention systematic reviews, including flaws arising through poor conduct of the review [ 6 ]. ROBIS, published in 2016, was developed to specifically assess RoB introduced by the conduct of the review; it is applicable to systematic reviews of interventions and several other types of reviews [ 4 ]. Both tools reflect a shift to a domain-based approach as opposed to generic quality checklists. There are a few items unique to each tool; however, similarities between items have been demonstrated [ 98 , 99 ]. AMSTAR-2 and ROBIS are recommended for use by: 1) authors of overviews or umbrella reviews and CPGs to evaluate systematic reviews considered as evidence; 2) authors of methodological research studies to appraise included systematic reviews; and 3) peer reviewers for appraisal of submitted systematic review manuscripts. For authors, these tools may function as teaching aids and inform conduct of their review during its development.

Description

Systematic reviews that include randomized and/or non-randomized studies as evidence can be appraised with AMSTAR-2 and ROBIS. Other characteristics of AMSTAR-2 and ROBIS are summarized in Table 3.2 . Both tools define categories for an overall rating; however, neither tool is intended to generate a total score by simply calculating the number of responses satisfying criteria for individual items [ 4 , 6 ]. AMSTAR-2 focuses on the rigor of a review’s methods irrespective of the specific subject matter. ROBIS places emphasis on a review’s results section— this suggests it may be optimally applied by appraisers with some knowledge of the review’s topic as they may be better equipped to determine if certain procedures (or lack thereof) would impact the validity of a review’s findings [ 98 , 100 ]. Reliability studies show AMSTAR-2 overall confidence ratings strongly correlate with the overall RoB ratings in ROBIS [ 100 , 101 ].

Comparison of AMSTAR-2 and ROBIS

a ROBIS includes an optional first phase to assess the applicability of the review to the research question of interest. The tool may be applicable to other review types in addition to the four specified, although modification of this initial phase will be needed (Personal Communication via email, Penny Whiting, 28 Jan 2022)

b AMSTAR-2 item #9 and #11 require separate responses for RCTs and NRSI

Interrater reliability has been shown to be acceptable for AMSTAR-2 [ 6 , 11 , 102 ] and ROBIS [ 4 , 98 , 103 ] but neither tool has been shown to be superior in this regard [ 100 , 101 , 104 , 105 ]. Overall, variability in reliability for both tools has been reported across items, between pairs of raters, and between centers [ 6 , 100 , 101 , 104 ]. The effects of appraiser experience on the results of AMSTAR-2 and ROBIS require further evaluation [ 101 , 105 ]. Updates to both tools should address items shown to be prone to individual appraisers’ subjective biases and opinions [ 11 , 100 ]; this may involve modifications of the current domains and signaling questions as well as incorporation of methods to make an appraiser’s judgments more explicit. Future revisions of these tools may also consider the addition of standards for aspects of systematic review development currently lacking (eg, rating overall certainty of evidence, [ 99 ] methods for synthesis without meta-analysis [ 105 ]) and removal of items that assess aspects of reporting that are thoroughly evaluated by PRISMA 2020.

Application

A good understanding of what is required to satisfy the standards of AMSTAR-2 and ROBIS involves study of the accompanying guidance documents written by the tools’ developers; these contain detailed descriptions of each item’s standards. In addition, accurate appraisal of a systematic review with either tool requires training. Most experts recommend independent assessment by at least two appraisers with a process for resolving discrepancies as well as procedures to establish interrater reliability, such as pilot testing, a calibration phase or exercise, and development of predefined decision rules [ 35 , 99 – 101 , 103 , 104 , 106 ]. These methods may, to some extent, address the challenges associated with the diversity in methodological training, subject matter expertise, and experience using the tools that are likely to exist among appraisers.

The standards of AMSTAR, AMSTAR-2, and ROBIS have been used in many methodological studies and epidemiological investigations. However, the increased publication of overviews or umbrella reviews and CPGs has likely been a greater influence on the widening acceptance of these tools. Critical appraisal of the secondary studies considered evidence is essential to the trustworthiness of both the recommendations of CPGs and the conclusions of overviews. Currently both Cochrane [ 55 ] and JBI [ 107 ] recommend AMSTAR-2 and ROBIS in their guidance for authors of overviews or umbrella reviews. However, ROBIS and AMSTAR-2 were released in 2016 and 2017, respectively; thus, to date, limited data have been reported about the uptake of these tools or which of the two may be preferred [ 21 , 106 ]. Currently, in relation to CPGs, AMSTAR-2 appears to be overwhelmingly popular compared to ROBIS. A Google Scholar search of this topic (search terms “AMSTAR 2 AND clinical practice guidelines,” “ROBIS AND clinical practice guidelines” 13 May 2022) found 12,700 hits for AMSTAR-2 and 1,280 for ROBIS. The apparent greater appeal of AMSTAR-2 may relate to its longer track record given the original version of the tool was in use for 10 years prior to its update in 2017.

Barriers to the uptake of AMSTAR-2 and ROBIS include the real or perceived time and resources necessary to complete the items they include and appraisers’ confidence in their own ratings [ 104 ]. Reports from comparative studies available to date indicate that appraisers find AMSTAR-2 questions, responses, and guidance to be clearer and simpler compared with ROBIS [ 11 , 101 , 104 , 105 ]. This suggests that for appraisal of intervention systematic reviews, AMSTAR-2 may be a more practical tool than ROBIS, especially for novice appraisers [ 101 , 103 – 105 ]. The unique characteristics of each tool, as well as their potential advantages and disadvantages, should be taken into consideration when deciding which tool should be used for an appraisal of a systematic review. In addition, the choice of one or the other may depend on how the results of an appraisal will be used; for example, a peer reviewer’s appraisal of a single manuscript versus an appraisal of multiple systematic reviews in an overview or umbrella review, CPG, or systematic methodological study.

Authors of overviews and CPGs report results of AMSTAR-2 and ROBIS appraisals for each of the systematic reviews they include as evidence. Ideally, an independent judgment of their appraisals can be made by the end users of overviews and CPGs; however, most stakeholders, including clinicians, are unlikely to have a sophisticated understanding of these tools. Nevertheless, they should at least be aware that AMSTAR-2 and ROBIS ratings reported in overviews and CPGs may be inaccurate because the tools are not applied as intended by their developers. This can result from inadequate training of the overview or CPG authors who perform the appraisals, or to modifications of the appraisal tools imposed by them. The potential variability in overall confidence and RoB ratings highlights why appraisers applying these tools need to support their judgments with explicit documentation; this allows readers to judge for themselves whether they agree with the criteria used by appraisers [ 4 , 108 ]. When these judgments are explicit, the underlying rationale used when applying these tools can be assessed [ 109 ].

Theoretically, we would expect an association of AMSTAR-2 with improved methodological rigor and an association of ROBIS with lower RoB in recent systematic reviews compared to those published before 2017. To our knowledge, this has not yet been demonstrated; however, like reports about the actual uptake of these tools, time will tell. Additional data on user experience is also needed to further elucidate the practical challenges and methodological nuances encountered with the application of these tools. This information could potentially inform the creation of unifying criteria to guide and standardize the appraisal of evidence syntheses [ 109 ].

Evaluation of reporting

Complete reporting is essential for users to establish the trustworthiness and applicability of a systematic review’s findings. Efforts to standardize and improve the reporting of systematic reviews resulted in the 2009 publication of the PRISMA statement [ 92 ] with its accompanying explanation and elaboration document [ 110 ]. This guideline was designed to help authors prepare a complete and transparent report of their systematic review. In addition, adherence to PRISMA is often used to evaluate the thoroughness of reporting of published systematic reviews [ 111 ]. The updated version, PRISMA 2020 [ 93 ], and its guidance document [ 112 ] were published in 2021. Items on the original and updated versions of PRISMA are organized by the six basic review components they address (title, abstract, introduction, methods, results, discussion). The PRISMA 2020 update is a considerably expanded version of the original; it includes standards and examples for the 27 original and 13 additional reporting items that capture methodological advances and may enhance the replicability of reviews [ 113 ].

The original PRISMA statement fostered the development of various PRISMA extensions (Table 3.3 ). These include reporting guidance for scoping reviews and reviews of diagnostic test accuracy and for intervention reviews that report on the following: harms outcomes, equity issues, the effects of acupuncture, the results of network meta-analyses and analyses of individual participant data. Detailed reporting guidance for specific systematic review components (abstracts, protocols, literature searches) is also available.

PRISMA extensions

PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses

a Note the abstract reporting checklist is now incorporated into PRISMA 2020 [ 93 ]

Uptake and impact

The 2009 PRISMA standards [ 92 ] for reporting have been widely endorsed by authors, journals, and EBM-related organizations. We anticipate the same for PRISMA 2020 [ 93 ] given its co-publication in multiple high-impact journals. However, to date, there is a lack of strong evidence for an association between improved systematic review reporting and endorsement of PRISMA 2009 standards [ 43 , 111 ]. Most journals require a PRISMA checklist accompany submissions of systematic review manuscripts. However, the accuracy of information presented on these self-reported checklists is not necessarily verified. It remains unclear which strategies (eg, authors’ self-report of checklists, peer reviewer checks) might improve adherence to the PRISMA reporting standards; in addition, the feasibility of any potentially effective strategies must be taken into consideration given the structure and limitations of current research and publication practices [ 124 ].

Pitfalls and limitations of PRISMA, AMSTAR-2, and ROBIS

Misunderstanding of the roles of these tools and their misapplication may be widespread problems. PRISMA 2020 is a reporting guideline that is most beneficial if consulted when developing a review as opposed to merely completing a checklist when submitting to a journal; at that point, the review is finished, with good or bad methodological choices. However, PRISMA checklists evaluate how completely an element of review conduct was reported, but do not evaluate the caliber of conduct or performance of a review. Thus, review authors and readers should not think that a rigorous systematic review can be produced by simply following the PRISMA 2020 guidelines. Similarly, it is important to recognize that AMSTAR-2 and ROBIS are tools to evaluate the conduct of a review but do not substitute for conceptual methodological guidance. In addition, they are not intended to be simple checklists. In fact, they have the potential for misuse or abuse if applied as such; for example, by calculating a total score to make a judgment about a review’s overall confidence or RoB. Proper selection of a response for the individual items on AMSTAR-2 and ROBIS requires training or at least reference to their accompanying guidance documents.

Not surprisingly, it has been shown that compliance with the PRISMA checklist is not necessarily associated with satisfying the standards of ROBIS [ 125 ]. AMSTAR-2 and ROBIS were not available when PRISMA 2009 was developed; however, they were considered in the development of PRISMA 2020 [ 113 ]. Therefore, future studies may show a positive relationship between fulfillment of PRISMA 2020 standards for reporting and meeting the standards of tools evaluating methodological quality and RoB.

Choice of an appropriate tool for the evaluation of a systematic review first involves identification of the underlying construct to be assessed. For systematic reviews of interventions, recommended tools include AMSTAR-2 and ROBIS for appraisal of conduct and PRISMA 2020 for completeness of reporting. All three tools were developed rigorously and provide easily accessible and detailed user guidance, which is necessary for their proper application and interpretation. When considering a manuscript for publication, training in these tools can sensitize peer reviewers and editors to major issues that may affect the review’s trustworthiness and completeness of reporting. Judgment of the overall certainty of a body of evidence and formulation of recommendations rely, in part, on AMSTAR-2 or ROBIS appraisals of systematic reviews. Therefore, training on the application of these tools is essential for authors of overviews and developers of CPGs. Peer reviewers and editors considering an overview or CPG for publication must hold their authors to a high standard of transparency regarding both the conduct and reporting of these appraisals.

Part 4. Meeting conduct standards

Many authors, peer reviewers, and editors erroneously equate fulfillment of the items on the PRISMA checklist with superior methodological rigor. For direction on methodology, we refer them to available resources that provide comprehensive conceptual guidance [ 59 , 60 ] as well as primers with basic step-by-step instructions [ 1 , 126 , 127 ]. This section is intended to complement study of such resources by facilitating use of AMSTAR-2 and ROBIS, tools specifically developed to evaluate methodological rigor of systematic reviews. These tools are widely accepted by methodologists; however, in the general medical literature, they are not uniformly selected for the critical appraisal of systematic reviews [ 88 , 96 ].

To enable their uptake, Table 4.1  links review components to the corresponding appraisal tool items. Expectations of AMSTAR-2 and ROBIS are concisely stated, and reasoning provided.

Systematic review components linked to appraisal with AMSTAR-2 and ROBIS a

CoI conflict of interest, MA meta-analysis, NA not addressed, PICO participant, intervention, comparison, outcome, PRISMA-P Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols, RoB risk of bias

a Components shown in bold are chosen for elaboration in Part 4 for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors; and/or 2) the component is evaluated by standards of an AMSTAR-2 “critical” domain

b Critical domains of AMSTAR-2 are indicated by *

Issues involved in meeting the standards for seven review components (identified in bold in Table 4.1 ) are addressed in detail. These were chosen for elaboration for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors based on consistent reports of their frequent AMSTAR-2 or ROBIS deficiencies [ 9 , 11 , 15 , 88 , 128 , 129 ]; and/or 2) the review component is judged by standards of an AMSTAR-2 “critical” domain. These have the greatest implications for how a systematic review will be appraised: if standards for any one of these critical domains are not met, the review is rated as having “critically low confidence.”

Research question

Specific and unambiguous research questions may have more value for reviews that deal with hypothesis testing. Mnemonics for the various elements of research questions are suggested by JBI and Cochrane (Table 2.1 ). These prompt authors to consider the specialized methods involved for developing different types of systematic reviews; however, while inclusion of the suggested elements makes a review compliant with a particular review’s methods, it does not necessarily make a research question appropriate. Table 4.2  lists acronyms that may aid in developing the research question. They include overlapping concepts of importance in this time of proliferating reviews of uncertain value [ 130 ]. If these issues are not prospectively contemplated, systematic review authors may establish an overly broad scope, or develop runaway scope allowing them to stray from predefined choices relating to key comparisons and outcomes.

Research question development

a Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Hulley SB, Cummings SR, Browner WS, editors. Designing clinical research: an epidemiological approach; 4th edn. Lippincott Williams & Wilkins; 2007. p. 14–22

b Doran, GT. There’s a S.M.A.R.T. way to write management’s goals and objectives. Manage Rev. 1981;70:35-6.

c Johnson BT, Hennessy EA. Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc Sci Med. 2019;233:237–51

Once a research question is established, searching on registry sites and databases for existing systematic reviews addressing the same or a similar topic is necessary in order to avoid contributing to research waste [ 131 ]. Repeating an existing systematic review must be justified, for example, if previous reviews are out of date or methodologically flawed. A full discussion on replication of intervention systematic reviews, including a consensus checklist, can be found in the work of Tugwell and colleagues [ 84 ].

Protocol development is considered a core component of systematic reviews [ 125 , 126 , 132 ]. Review protocols may allow researchers to plan and anticipate potential issues, assess validity of methods, prevent arbitrary decision-making, and minimize bias that can be introduced by the conduct of the review. Registration of a protocol that allows public access promotes transparency of the systematic review’s methods and processes and reduces the potential for duplication [ 132 ]. Thinking early and carefully about all the steps of a systematic review is pragmatic and logical and may mitigate the influence of the authors’ prior knowledge of the evidence [ 133 ]. In addition, the protocol stage is when the scope of the review can be carefully considered by authors, reviewers, and editors; this may help to avoid production of overly ambitious reviews that include excessive numbers of comparisons and outcomes or are undisciplined in their study selection.

An association with attainment of AMSTAR standards in systematic reviews with published prospective protocols has been reported [ 134 ]. However, completeness of reporting does not seem to be different in reviews with a protocol compared to those without one [ 135 ]. PRISMA-P [ 116 ] and its accompanying elaboration and explanation document [ 136 ] can be used to guide and assess the reporting of protocols. A final version of the review should fully describe any protocol deviations. Peer reviewers may compare the submitted manuscript with any available pre-registered protocol; this is required if AMSTAR-2 or ROBIS are used for critical appraisal.

There are multiple options for the recording of protocols (Table 4.3 ). Some journals will peer review and publish protocols. In addition, many online sites offer date-stamped and publicly accessible protocol registration. Some of these are exclusively for protocols of evidence syntheses; others are less restrictive and offer researchers the capacity for data storage, sharing, and other workflow features. These sites document protocol details to varying extents and have different requirements [ 137 ]. The most popular site for systematic reviews, the International Prospective Register of Systematic Reviews (PROSPERO), for example, only registers reviews that report on an outcome with direct relevance to human health. The PROSPERO record documents protocols for all types of reviews except literature and scoping reviews. Of note, PROSPERO requires authors register their review protocols prior to any data extraction [ 133 , 138 ]. The electronic records of most of these registry sites allow authors to update their protocols and facilitate transparent tracking of protocol changes, which are not unexpected during the progress of the review [ 139 ].

Options for protocol registration of evidence syntheses

a Authors are advised to contact their target journal regarding submission of systematic review protocols

b Registration is restricted to approved review projects

c The JBI registry lists review projects currently underway by JBI-affiliated entities. These records include a review’s title, primary author, research question, and PICO elements. JBI recommends that authors register eligible protocols with PROSPERO

d See Pieper and Rombey [ 137 ] for detailed characteristics of these five registries

e See Pieper and Rombey [ 137 ] for other systematic review data repository options

Study design inclusion

For most systematic reviews, broad inclusion of study designs is recommended [ 126 ]. This may allow comparison of results between contrasting study design types [ 126 ]. Certain study designs may be considered preferable depending on the type of review and nature of the research question. However, prevailing stereotypes about what each study design does best may not be accurate. For example, in systematic reviews of interventions, randomized designs are typically thought to answer highly specific questions while non-randomized designs often are expected to reveal greater information about harms or real-word evidence [ 126 , 140 , 141 ]. This may be a false distinction; randomized trials may be pragmatic [ 142 ], they may offer important (and more unbiased) information on harms [ 143 ], and data from non-randomized trials may not necessarily be more real-world-oriented [ 144 ].

Moreover, there may not be any available evidence reported by RCTs for certain research questions; in some cases, there may not be any RCTs or NRSI. When the available evidence is limited to case reports and case series, it is not possible to test hypotheses nor provide descriptive estimates or associations; however, a systematic review of these studies can still offer important insights [ 81 , 145 ]. When authors anticipate that limited evidence of any kind may be available to inform their research questions, a scoping review can be considered. Alternatively, decisions regarding inclusion of indirect as opposed to direct evidence can be addressed during protocol development [ 146 ]. Including indirect evidence at an early stage of intervention systematic review development allows authors to decide if such studies offer any additional and/or different understanding of treatment effects for their population or comparison of interest. Issues of indirectness of included studies are accounted for later in the process, during determination of the overall certainty of evidence (see Part 5 for details).

Evidence search

Both AMSTAR-2 and ROBIS require systematic and comprehensive searches for evidence. This is essential for any systematic review. Both tools discourage search restrictions based on language and publication source. Given increasing globalism in health care, the practice of including English-only literature should be avoided [ 126 ]. There are many examples in which language bias (different results in studies published in different languages) has been documented [ 147 , 148 ]. This does not mean that all literature, in all languages, is equally trustworthy [ 148 ]; however, the only way to formally probe for the potential of such biases is to consider all languages in the initial search. The gray literature and a search of trials may also reveal important details about topics that would otherwise be missed [ 149 – 151 ]. Again, inclusiveness will allow review authors to investigate whether results differ in gray literature and trials [ 41 , 151 – 153 ].

Authors should make every attempt to complete their review within one year as that is the likely viable life of a search. (1) If that is not possible, the search should be updated close to the time of completion [ 154 ]. Different research topics may warrant less of a delay, for example, in rapidly changing fields (as in the case of the COVID-19 pandemic), even one month may radically change the available evidence.

Excluded studies

AMSTAR-2 requires authors to provide references for any studies excluded at the full text phase of study selection along with reasons for exclusion; this allows readers to feel confident that all relevant literature has been considered for inclusion and that exclusions are defensible.

Risk of bias assessment of included studies

The design of the studies included in a systematic review (eg, RCT, cohort, case series) should not be equated with appraisal of its RoB. To meet AMSTAR-2 and ROBIS standards, systematic review authors must examine RoB issues specific to the design of each primary study they include as evidence. It is unlikely that a single RoB appraisal tool will be suitable for all research designs. In addition to tools for randomized and non-randomized studies, specific tools are available for evaluation of RoB in case reports and case series [ 82 ] and single-case experimental designs [ 155 , 156 ]. Note the RoB tools selected must meet the standards of the appraisal tool used to judge the conduct of the review. For example, AMSTAR-2 identifies four sources of bias specific to RCTs and NRSI that must be addressed by the RoB tool(s) chosen by the review authors. The Cochrane RoB-2 [ 157 ] tool for RCTs and ROBINS-I [ 158 ] for NRSI for RoB assessment meet the AMSTAR-2 standards. Appraisers on the review team should not modify any RoB tool without complete transparency and acknowledgment that they have invalidated the interpretation of the tool as intended by its developers [ 159 ]. Conduct of RoB assessments is not addressed AMSTAR-2; to meet ROBIS standards, two independent reviewers should complete RoB assessments of included primary studies.

Implications of the RoB assessments must be explicitly discussed and considered in the conclusions of the review. Discussion of the overall RoB of included studies may consider the weight of the studies at high RoB, the importance of the sources of bias in the studies being summarized, and if their importance differs in relationship to the outcomes reported. If a meta-analysis is performed, serious concerns for RoB of individual studies should be accounted for in these results as well. If the results of the meta-analysis for a specific outcome change when studies at high RoB are excluded, readers will have a more accurate understanding of this body of evidence. However, while investigating the potential impact of specific biases is a useful exercise, it is important to avoid over-interpretation, especially when there are sparse data.

Synthesis methods for quantitative data

Syntheses of quantitative data reported by primary studies are broadly categorized as one of two types: meta-analysis, and synthesis without meta-analysis (Table 4.4 ). Before deciding on one of these methods, authors should seek methodological advice about whether reported data can be transformed or used in other ways to provide a consistent effect measure across studies [ 160 , 161 ].

Common methods for quantitative synthesis

CI confidence interval (or credible interval, if analysis is done in Bayesian framework)

a See text for descriptions of the types of data combined in each of these approaches

b See Additional File 4  for guidance on the structure and presentation of forest plots

c General approach is similar to aggregate data meta-analysis but there are substantial differences relating to data collection and checking and analysis [ 162 ]. This approach to syntheses is applicable to intervention, diagnostic, and prognostic systematic reviews [ 163 ]

d Examples include meta-regression, hierarchical and multivariate approaches [ 164 ]

e In-depth guidance and illustrations of these methods are provided in Chapter 12 of the Cochrane Handbook [ 160 ]

Meta-analysis

Systematic reviews that employ meta-analysis should not be referred to simply as “meta-analyses.” The term meta-analysis strictly refers to a specific statistical technique used when study effect estimates and their variances are available, yielding a quantitative summary of results. In general, methods for meta-analysis involve use of a weighted average of effect estimates from two or more studies. If considered carefully, meta-analysis increases the precision of the estimated magnitude of effect and can offer useful insights about heterogeneity and estimates of effects. We refer to standard references for a thorough introduction and formal training [ 165 – 167 ].

There are three common approaches to meta-analysis in current health care–related systematic reviews (Table 4.4 ). Aggregate meta-analyses is the most familiar to authors of evidence syntheses and their end users. This standard meta-analysis combines data on effect estimates reported by studies that investigate similar research questions involving direct comparisons of an intervention and comparator. Results of these analyses provide a single summary intervention effect estimate. If the included studies in a systematic review measure an outcome differently, their reported results may be transformed to make them comparable [ 161 ]. Forest plots visually present essential information about the individual studies and the overall pooled analysis (see Additional File 4  for details).

Less familiar and more challenging meta-analytical approaches used in secondary research include individual participant data (IPD) and network meta-analyses (NMA); PRISMA extensions provide reporting guidelines for both [ 117 , 118 ]. In IPD, the raw data on each participant from each eligible study are re-analyzed as opposed to the study-level data analyzed in aggregate data meta-analyses [ 168 ]. This may offer advantages, including the potential for limiting concerns about bias and allowing more robust analyses [ 163 ]. As suggested by the description in Table 4.4 , NMA is a complex statistical approach. It combines aggregate data [ 169 ] or IPD [ 170 ] for effect estimates from direct and indirect comparisons reported in two or more studies of three or more interventions. This makes it a potentially powerful statistical tool; while multiple interventions are typically available to treat a condition, few have been evaluated in head-to-head trials [ 171 ]. Both IPD and NMA facilitate a broader scope, and potentially provide more reliable and/or detailed results; however, compared with standard aggregate data meta-analyses, their methods are more complicated, time-consuming, and resource-intensive, and they have their own biases, so one needs sufficient funding, technical expertise, and preparation to employ them successfully [ 41 , 172 , 173 ].

Several items in AMSTAR-2 and ROBIS address meta-analysis; thus, understanding the strengths, weaknesses, assumptions, and limitations of methods for meta-analyses is important. According to the standards of both tools, plans for a meta-analysis must be addressed in the review protocol, including reasoning, description of the type of quantitative data to be synthesized, and the methods planned for combining the data. This should not consist of stock statements describing conventional meta-analysis techniques; rather, authors are expected to anticipate issues specific to their research questions. Concern for the lack of training in meta-analysis methods among systematic review authors cannot be overstated. For those with training, the use of popular software (eg, RevMan [ 174 ], MetaXL [ 175 ], JBI SUMARI [ 176 ]) may facilitate exploration of these methods; however, such programs cannot substitute for the accurate interpretation of the results of meta-analyses, especially for more complex meta-analytical approaches.

Synthesis without meta-analysis

There are varied reasons a meta-analysis may not be appropriate or desirable [ 160 , 161 ]. Syntheses that informally use statistical methods other than meta-analysis are variably referred to as descriptive, narrative, or qualitative syntheses or summaries; these terms are also applied to syntheses that make no attempt to statistically combine data from individual studies. However, use of such imprecise terminology is discouraged; in order to fully explore the results of any type of synthesis, some narration or description is needed to supplement the data visually presented in tabular or graphic forms [ 63 , 177 ]. In addition, the term “qualitative synthesis” is easily confused with a synthesis of qualitative data in a qualitative or mixed methods review. “Synthesis without meta-analysis” is currently the preferred description of other ways to combine quantitative data from two or more studies. Use of this specific terminology when referring to these types of syntheses also implies the application of formal methods (Table 4.4 ).

Methods for syntheses without meta-analysis involve structured presentations of the data in any tables and plots. In comparison to narrative descriptions of each study, these are designed to more effectively and transparently show patterns and convey detailed information about the data; they also allow informal exploration of heterogeneity [ 178 ]. In addition, acceptable quantitative statistical methods (Table 4.4 ) are formally applied; however, it is important to recognize these methods have significant limitations for the interpretation of the effectiveness of an intervention [ 160 ]. Nevertheless, when meta-analysis is not possible, the application of these methods is less prone to bias compared with an unstructured narrative description of included studies [ 178 , 179 ].

Vote counting is commonly used in systematic reviews and involves a tally of studies reporting results that meet some threshold of importance applied by review authors. Until recently, it has not typically been identified as a method for synthesis without meta-analysis. Guidance on an acceptable vote counting method based on direction of effect is currently available [ 160 ] and should be used instead of narrative descriptions of such results (eg, “more than half the studies showed improvement”; “only a few studies reported adverse effects”; “7 out of 10 studies favored the intervention”). Unacceptable methods include vote counting by statistical significance or magnitude of effect or some subjective rule applied by the authors.

AMSTAR-2 and ROBIS standards do not explicitly address conduct of syntheses without meta-analysis, although AMSTAR-2 items 13 and 14 might be considered relevant. Guidance for the complete reporting of syntheses without meta-analysis for systematic reviews of interventions is available in the Synthesis without Meta-analysis (SWiM) guideline [ 180 ] and methodological guidance is available in the Cochrane Handbook [ 160 , 181 ].

Familiarity with AMSTAR-2 and ROBIS makes sense for authors of systematic reviews as these appraisal tools will be used to judge their work; however, training is necessary for authors to truly appreciate and apply methodological rigor. Moreover, judgment of the potential contribution of a systematic review to the current knowledge base goes beyond meeting the standards of AMSTAR-2 and ROBIS. These tools do not explicitly address some crucial concepts involved in the development of a systematic review; this further emphasizes the need for author training.

We recommend that systematic review authors incorporate specific practices or exercises when formulating a research question at the protocol stage, These should be designed to raise the review team’s awareness of how to prevent research and resource waste [ 84 , 130 ] and to stimulate careful contemplation of the scope of the review [ 30 ]. Authors’ training should also focus on justifiably choosing a formal method for the synthesis of quantitative and/or qualitative data from primary research; both types of data require specific expertise. For typical reviews that involve syntheses of quantitative data, statistical expertise is necessary, initially for decisions about appropriate methods, [ 160 , 161 ] and then to inform any meta-analyses [ 167 ] or other statistical methods applied [ 160 ].

Part 5. Rating overall certainty of evidence

Report of an overall certainty of evidence assessment in a systematic review is an important new reporting standard of the updated PRISMA 2020 guidelines [ 93 ]. Systematic review authors are well acquainted with assessing RoB in individual primary studies, but much less familiar with assessment of overall certainty across an entire body of evidence. Yet a reliable way to evaluate this broader concept is now recognized as a vital part of interpreting the evidence.

Historical systems for rating evidence are based on study design and usually involve hierarchical levels or classes of evidence that use numbers and/or letters to designate the level/class. These systems were endorsed by various EBM-related organizations. Professional societies and regulatory groups then widely adopted them, often with modifications for application to the available primary research base in specific clinical areas. In 2002, a report issued by the AHRQ identified 40 systems to rate quality of a body of evidence [ 182 ]. A critical appraisal of systems used by prominent health care organizations published in 2004 revealed limitations in sensibility, reproducibility, applicability to different questions, and usability to different end users [ 183 ]. Persistent use of hierarchical rating schemes to describe overall quality continues to complicate the interpretation of evidence. This is indicated by recent reports of poor interpretability of systematic review results by readers [ 184 – 186 ] and misleading interpretations of the evidence related to the “spin” systematic review authors may put on their conclusions [ 50 , 187 ].

Recognition of the shortcomings of hierarchical rating systems raised concerns that misleading clinical recommendations could result even if based on a rigorous systematic review. In addition, the number and variability of these systems were considered obstacles to quick and accurate interpretations of the evidence by clinicians, patients, and policymakers [ 183 ]. These issues contributed to the development of the GRADE approach. An international working group, that continues to actively evaluate and refine it, first introduced GRADE in 2004 [ 188 ]. Currently more than 110 organizations from 19 countries around the world have endorsed or are using GRADE [ 189 ].

GRADE approach to rating overall certainty

GRADE offers a consistent and sensible approach for two separate processes: rating the overall certainty of a body of evidence and the strength of recommendations. The former is the expected conclusion of a systematic review, while the latter is pertinent to the development of CPGs. As such, GRADE provides a mechanism to bridge the gap from evidence synthesis to application of the evidence for informed clinical decision-making [ 27 , 190 ]. We briefly examine the GRADE approach but only as it applies to rating overall certainty of evidence in systematic reviews.

In GRADE, use of “certainty” of a body of evidence is preferred over the term “quality.” [ 191 ] Certainty refers to the level of confidence systematic review authors have that, for each outcome, an effect estimate represents the true effect. The GRADE approach to rating confidence in estimates begins with identifying the study type (RCT or NRSI) and then systematically considers criteria to rate the certainty of evidence up or down (Table 5.1 ).

GRADE criteria for rating certainty of evidence

a Applies to randomized studies

b Applies to non-randomized studies

This process results in assignment of one of the four GRADE certainty ratings to each outcome; these are clearly conveyed with the use of basic interpretation symbols (Table 5.2 ) [ 192 ]. Notably, when multiple outcomes are reported in a systematic review, each outcome is assigned a unique certainty rating; thus different levels of certainty may exist in the body of evidence being examined.

GRADE certainty ratings and their interpretation symbols a

a From the GRADE Handbook [ 192 ]

GRADE’s developers acknowledge some subjectivity is involved in this process [ 193 ]. In addition, they emphasize that both the criteria for rating evidence up and down (Table 5.1 ) as well as the four overall certainty ratings (Table 5.2 ) reflect a continuum as opposed to discrete categories [ 194 ]. Consequently, deciding whether a study falls above or below the threshold for rating up or down may not be straightforward, and preliminary overall certainty ratings may be intermediate (eg, between low and moderate). Thus, the proper application of GRADE requires systematic review authors to take an overall view of the body of evidence and explicitly describe the rationale for their final ratings.

Advantages of GRADE

Outcomes important to the individuals who experience the problem of interest maintain a prominent role throughout the GRADE process [ 191 ]. These outcomes must inform the research questions (eg, PICO [population, intervention, comparator, outcome]) that are specified a priori in a systematic review protocol. Evidence for these outcomes is then investigated and each critical or important outcome is ultimately assigned a certainty of evidence as the end point of the review. Notably, limitations of the included studies have an impact at the outcome level. Ultimately, the certainty ratings for each outcome reported in a systematic review are considered by guideline panels. They use a different process to formulate recommendations that involves assessment of the evidence across outcomes [ 201 ]. It is beyond our scope to describe the GRADE process for formulating recommendations; however, it is critical to understand how these two outcome-centric concepts of certainty of evidence in the GRADE framework are related and distinguished. An in-depth illustration using examples from recently published evidence syntheses and CPGs is provided in Additional File 5 A (Table AF5A-1).

The GRADE approach is applicable irrespective of whether the certainty of the primary research evidence is high or very low; in some circumstances, indirect evidence of higher certainty may be considered if direct evidence is unavailable or of low certainty [ 27 ]. In fact, most interventions and outcomes in medicine have low or very low certainty of evidence based on GRADE and there seems to be no major improvement over time [ 202 , 203 ]. This is still a very important (even if sobering) realization for calibrating our understanding of medical evidence. A major appeal of the GRADE approach is that it offers a common framework that enables authors of evidence syntheses to make complex judgments about evidence certainty and to convey these with unambiguous terminology. This prevents some common mistakes made by review authors, including overstating results (or under-reporting harms) [ 187 ] and making recommendations for treatment. This is illustrated in Table AF5A-2 (Additional File 5 A), which compares the concluding statements made about overall certainty in a systematic review with and without application of the GRADE approach.

Theoretically, application of GRADE should improve consistency of judgments about certainty of evidence, both between authors and across systematic reviews. In one empirical evaluation conducted by the GRADE Working Group, interrater reliability of two individual raters assessing certainty of the evidence for a specific outcome increased from ~ 0.3 without using GRADE to ~ 0.7 by using GRADE [ 204 ]. However, others report variable agreement among those experienced in GRADE assessments of evidence certainty [ 190 ]. Like any other tool, GRADE requires training in order to be properly applied. The intricacies of the GRADE approach and the necessary subjectivity involved suggest that improving agreement may require strict rules for its application; alternatively, use of general guidance and consensus among review authors may result in less consistency but provide important information for the end user [ 190 ].

GRADE caveats

Simply invoking “the GRADE approach” does not automatically ensure GRADE methods were employed by authors of a systematic review (or developers of a CPG). Table 5.3 lists the criteria the GRADE working group has established for this purpose. These criteria highlight the specific terminology and methods that apply to rating the certainty of evidence for outcomes reported in a systematic review [ 191 ], which is different from rating overall certainty across outcomes considered in the formulation of recommendations [ 205 ]. Modifications of standard GRADE methods and terminology are discouraged as these may detract from GRADE’s objectives to minimize conceptual confusion and maximize clear communication [ 206 ].

Criteria for using GRADE in a systematic review a

a Adapted from the GRADE working group [ 206 ]; this list does not contain the additional criteria that apply to the development of a clinical practice guideline

Nevertheless, GRADE is prone to misapplications [ 207 , 208 ], which can distort a systematic review’s conclusions about the certainty of evidence. Systematic review authors without proper GRADE training are likely to misinterpret the terms “quality” and “grade” and to misunderstand the constructs assessed by GRADE versus other appraisal tools. For example, review authors may reference the standard GRADE certainty ratings (Table 5.2 ) to describe evidence for their outcome(s) of interest. However, these ratings are invalidated if authors omit or inadequately perform RoB evaluations of each included primary study. Such deficiencies in RoB assessments are unacceptable but not uncommon, as reported in methodological studies of systematic reviews and overviews [ 104 , 186 , 209 , 210 ]. GRADE ratings are also invalidated if review authors do not formally address and report on the other criteria (Table 5.1 ) necessary for a GRADE certainty rating.

Other caveats pertain to application of a GRADE certainty of evidence rating in various types of evidence syntheses. Current adaptations of GRADE are described in Additional File 5 B and included on Table 6.3 , which is introduced in the next section.

Concise Guide to best practices for evidence syntheses, version 1.0 a

AMSTAR A MeaSurement Tool to Assess Systematic Reviews, CASP Critical Appraisal Skills Programme, CERQual Confidence in the Evidence from Reviews of Qualitative research, ConQual Establishing Confidence in the output of Qualitative research synthesis, COSMIN COnsensus-based Standards for the selection of health Measurement Instruments, DTA diagnostic test accuracy, eMERGe meta-ethnography reporting guidance, ENTREQ enhancing transparency in reporting the synthesis of qualitative research, GRADE Grading of Recommendations Assessment, Development and Evaluation, MA meta-analysis, NRSI non-randomized studies of interventions, P protocol, PRIOR Preferred Reporting Items for Overviews of Reviews, PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PROBAST Prediction model Risk Of Bias ASsessment Tool, QUADAS quality assessment of studies of diagnostic accuracy included in systematic reviews, QUIPS Quality In Prognosis Studies, RCT randomized controlled trial, RoB risk of bias, ROBINS-I Risk Of Bias In Non-randomised Studies of Interventions, ROBIS Risk of Bias in Systematic Reviews, ScR scoping review, SWiM systematic review without meta-analysis

a Superscript numbers represent citations provided in the main reference list. Additional File 6 lists links to available online resources for the methods and tools included in the Concise Guide

b The MECIR manual [ 30 ] provides Cochrane’s specific standards for both reporting and conduct of intervention systematic reviews and protocols

c Editorial and peer reviewers can evaluate completeness of reporting in submitted manuscripts using these tools. Authors may be required to submit a self-reported checklist for the applicable tools

d The decision flowchart described by Flemming and colleagues [ 223 ] is recommended for guidance on how to choose the best approach to reporting for qualitative reviews

e SWiM was developed for intervention studies reporting quantitative data. However, if there is not a more directly relevant reporting guideline, SWiM may prompt reviewers to consider the important details to report. (Personal Communication via email, Mhairi Campbell, 14 Dec 2022)

f JBI recommends their own tools for the critical appraisal of various quantitative primary study designs included in systematic reviews of intervention effectiveness, prevalence and incidence, and etiology and risk as well as for the critical appraisal of systematic reviews included in umbrella reviews. However, except for the JBI Checklists for studies reporting prevalence data and qualitative research, the development, validity, and reliability of these tools are not well documented

g Studies that are not RCTs or NRSI require tools developed specifically to evaluate their design features. Examples include single case experimental design [ 155 , 156 ] and case reports and series [ 82 ]

h The evaluation of methodological quality of studies included in a synthesis of qualitative research is debatable [ 224 ]. Authors may select a tool appropriate for the type of qualitative synthesis methodology employed. The CASP Qualitative Checklist [ 218 ] is an example of a published, commonly used tool that focuses on assessment of the methodological strengths and limitations of qualitative studies. The JBI Critical Appraisal Checklist for Qualitative Research [ 219 ] is recommended for reviews using a meta-aggregative approach

i Consider including risk of bias assessment of included studies if this information is relevant to the research question; however, scoping reviews do not include an assessment of the overall certainty of a body of evidence

j Guidance available from the GRADE working group [ 225 , 226 ]; also recommend consultation with the Cochrane diagnostic methods group

k Guidance available from the GRADE working group [ 227 ]; also recommend consultation with Cochrane prognostic methods group

l Used for syntheses in reviews with a meta-aggregative approach [ 224 ]

m Chapter 5 in the JBI Manual offers guidance on how to adapt GRADE to prevalence and incidence reviews [ 69 ]

n Janiaud and colleagues suggest criteria for evaluating evidence certainty for meta-analyses of non-randomized studies evaluating risk factors [ 228 ]

o The COSMIN user manual provides details on how to apply GRADE in systematic reviews of measurement properties [ 229 ]

The expected culmination of a systematic review should be a rating of overall certainty of a body of evidence for each outcome reported. The GRADE approach is recommended for making these judgments for outcomes reported in systematic reviews of interventions and can be adapted for other types of reviews. This represents the initial step in the process of making recommendations based on evidence syntheses. Peer reviewers should ensure authors meet the minimal criteria for supporting the GRADE approach when reviewing any evidence synthesis that reports certainty ratings derived using GRADE. Authors and peer reviewers of evidence syntheses unfamiliar with GRADE are encouraged to seek formal training and take advantage of the resources available on the GRADE website [ 211 , 212 ].

Part 6. Concise Guide to best practices

Accumulating data in recent years suggest that many evidence syntheses (with or without meta-analysis) are not reliable. This relates in part to the fact that their authors, who are often clinicians, can be overwhelmed by the plethora of ways to evaluate evidence. They tend to resort to familiar but often inadequate, inappropriate, or obsolete methods and tools and, as a result, produce unreliable reviews. These manuscripts may not be recognized as such by peer reviewers and journal editors who may disregard current standards. When such a systematic review is published or included in a CPG, clinicians and stakeholders tend to believe that it is trustworthy. A vicious cycle in which inadequate methodology is rewarded and potentially misleading conclusions are accepted is thus supported. There is no quick or easy way to break this cycle; however, increasing awareness of best practices among all these stakeholder groups, who often have minimal (if any) training in methodology, may begin to mitigate it. This is the rationale for inclusion of Parts 2 through 5 in this guidance document. These sections present core concepts and important methodological developments that inform current standards and recommendations. We conclude by taking a direct and practical approach.

Inconsistent and imprecise terminology used in the context of development and evaluation of evidence syntheses is problematic for authors, peer reviewers and editors, and may lead to the application of inappropriate methods and tools. In response, we endorse use of the basic terms (Table 6.1 ) defined in the PRISMA 2020 statement [ 93 ]. In addition, we have identified several problematic expressions and nomenclature. In Table 6.2 , we compile suggestions for preferred terms less likely to be misinterpreted.

Terms relevant to the reporting of health care–related evidence syntheses a

a Reproduced from Page and colleagues [ 93 ]

Terminology suggestions for health care–related evidence syntheses

a For example, meta-aggregation, meta-ethnography, critical interpretative synthesis, realist synthesis

b This term may best apply to the synthesis in a mixed methods systematic review in which data from different types of evidence (eg, qualitative, quantitative, economic) are summarized [ 64 ]

We also propose a Concise Guide (Table 6.3 ) that summarizes the methods and tools recommended for the development and evaluation of nine types of evidence syntheses. Suggestions for specific tools are based on the rigor of their development as well as the availability of detailed guidance from their developers to ensure their proper application. The formatting of the Concise Guide addresses a well-known source of confusion by clearly distinguishing the underlying methodological constructs that these tools were designed to assess. Important clarifications and explanations follow in the guide’s footnotes; associated websites, if available, are listed in Additional File 6 .

To encourage uptake of best practices, journal editors may consider adopting or adapting the Concise Guide in their instructions to authors and peer reviewers of evidence syntheses. Given the evolving nature of evidence synthesis methodology, the suggested methods and tools are likely to require regular updates. Authors of evidence syntheses should monitor the literature to ensure they are employing current methods and tools. Some types of evidence syntheses (eg, rapid, economic, methodological) are not included in the Concise Guide; for these, authors are advised to obtain recommendations for acceptable methods by consulting with their target journal.

We encourage the appropriate and informed use of the methods and tools discussed throughout this commentary and summarized in the Concise Guide (Table 6.3 ). However, we caution against their application in a perfunctory or superficial fashion. This is a common pitfall among authors of evidence syntheses, especially as the standards of such tools become associated with acceptance of a manuscript by a journal. Consequently, published evidence syntheses may show improved adherence to the requirements of these tools without necessarily making genuine improvements in their performance.

In line with our main objective, the suggested tools in the Concise Guide address the reliability of evidence syntheses; however, we recognize that the utility of systematic reviews is an equally important concern. An unbiased and thoroughly reported evidence synthesis may still not be highly informative if the evidence itself that is summarized is sparse, weak and/or biased [ 24 ]. Many intervention systematic reviews, including those developed by Cochrane [ 203 ] and those applying GRADE [ 202 ], ultimately find no evidence, or find the evidence to be inconclusive (eg, “weak,” “mixed,” or of “low certainty”). This often reflects the primary research base; however, it is important to know what is known (or not known) about a topic when considering an intervention for patients and discussing treatment options with them.

Alternatively, the frequency of “empty” and inconclusive reviews published in the medical literature may relate to limitations of conventional methods that focus on hypothesis testing; these have emphasized the importance of statistical significance in primary research and effect sizes from aggregate meta-analyses [ 183 ]. It is becoming increasingly apparent that this approach may not be appropriate for all topics [ 130 ]. Development of the GRADE approach has facilitated a better understanding of significant factors (beyond effect size) that contribute to the overall certainty of evidence. Other notable responses include the development of integrative synthesis methods for the evaluation of complex interventions [ 230 , 231 ], the incorporation of crowdsourcing and machine learning into systematic review workflows (eg the Cochrane Evidence Pipeline) [ 2 ], the shift in paradigm to living systemic review and NMA platforms [ 232 , 233 ] and the proposal of a new evidence ecosystem that fosters bidirectional collaborations and interactions among a global network of evidence synthesis stakeholders [ 234 ]. These evolutions in data sources and methods may ultimately make evidence syntheses more streamlined, less duplicative, and more importantly, they may be more useful for timely policy and clinical decision-making; however, that will only be the case if they are rigorously reported and conducted.

We look forward to others’ ideas and proposals for the advancement of methods for evidence syntheses. For now, we encourage dissemination and uptake of the currently accepted best tools and practices for their development and evaluation; at the same time, we stress that uptake of appraisal tools, checklists, and software programs cannot substitute for proper education in the methodology of evidence syntheses and meta-analysis. Authors, peer reviewers, and editors must strive to make accurate and reliable contributions to the present evidence knowledge base; online alerts, upcoming technology, and accessible education may make this more feasible than ever before. Our intention is to improve the trustworthiness of evidence syntheses across disciplines, topics, and types of evidence syntheses. All of us must continue to study, teach, and act cooperatively for that to happen.

Acknowledgements

Michelle Oakman Hayes for her assistance with the graphics, Mike Clarke for his willingness to answer our seemingly arbitrary questions, and Bernard Dan for his encouragement of this project.

Authors’ contributions

All authors participated in the development of the ideas, writing, and review of this manuscript. The author(s) read and approved the final manuscript.

The work of John Ioannidis has been supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University.

Declarations

The authors declare no competing interests.

This article has been published simultaneously in BMC Systematic Reviews, Acta Anaesthesiologica Scandinavica, BMC Infectious Diseases, British Journal of Pharmacology, JBI Evidence Synthesis, the Journal of Bone and Joint Surgery Reviews , and the Journal of Pediatric Rehabilitation Medicine .

Publisher’ s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Study Protocol

Remote measurement based care (RMBC) interventions for mental health—Protocol of a systematic review and meta-analysis

Contributed equally to this work with: Felix Machleid, Twyla Michnevich

Roles Conceptualization, Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Department of Psychiatry and Psychotherapy, Charité Campus Mitte, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany, Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Berlin, Germany

ORCID logo

Affiliation Department of Psychiatry and Psychotherapy, Charité Campus Mitte, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany

Roles Conceptualization, Writing – review & editing

Affiliation Department of Infectious Diseases and Respiratory Medicine, Charité Campus Virchow-Klinikum, Charité – Universitätsmedizin Berlin, Berlin, Germany

Affiliation Clinic for Psychiatry, Psychotherapy and Psychosomatics, Krankenhaus am Urban, Berlin, Germany

Affiliations Clinics for Psychiatry and Psychotherapy, Clinics at the Theodor-Wenzel-Werk, Berlin, Germany, Recovery Cat GmbH, Berlin, Germany

Roles Conceptualization, Writing – original draft

Affiliation Recovery Cat GmbH, Berlin, Germany

Affiliations Department of Psychiatry and Psychotherapy, Charité Campus Mitte, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany, Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Berlin, Germany, Recovery Cat GmbH, Berlin, Germany

  • Felix Machleid, 
  • Twyla Michnevich, 
  • Leu Huang, 
  • Louisa Schröder-Frerkes, 
  • Caspar Wiegmann, 
  • Toni Muffel, 
  • Jakob Kaminski

PLOS

  • Published: February 16, 2024
  • https://doi.org/10.1371/journal.pone.0297929
  • Peer Review
  • Reader Comments

Poor management of mental illnesses is associated with lower treatment adherence, chronification, avoidable re-hospitalisations, and high costs. Remote measurement based care (RMBC) interventions have gained increasing relevance due to its potential in providing a comprehensive and patient-centric approach to mental health management.

The systematic review and meta-analysis aims to provide a comprehensive overview and analysis of existing evidence on the use of RMBC for patients with mental illness and to examine the effectiveness of RMBC interventions in alleviating disorder-specific symptoms, reducing relapse and improving recovery-oriented outcomes, global functioning, and quality of life.

Methods and analysis

Our multidisciplinary research team will develop a comprehensive search strategy, adapted to each electronic database (PubMed, Medline, Embase, and PsychINFO) to be examined systematically. Studies with patients formally diagnosed by the International Classification of Diseases or the Diagnostic and Statistical Manual of Mental Disorders which include assessment of self-reported psychiatric symptoms will be included. Publications will be reviewed by teams of independent researchers. Quality of studies will be assessed using the Cochrane Collaboration’s tool for assessing risk of bias. Outcomes cover symptom-focused or disease-specific outcomes, relapse, recovery-focused outcomes, global functioning, quality of life and acceptability of the intervention. Further data that will be extracted includes study characteristics, target population, intervention, and tracking characteristics. Data will be synthesised qualitatively, summarising findings of the systematic review. Randomised controlled trials (RCTs) will be considered for meta-analysis if data is found comparable in terms of mental illness, study design and outcomes. Cumulative evidence will be evaluated according to the Grading of Recommendations Assessment, Development and Evaluation framework.

Trial registration

Trial registration number : PROSPERO CRD42022356176 .

Citation: Machleid F, Michnevich T, Huang L, Schröder-Frerkes L, Wiegmann C, Muffel T, et al. (2024) Remote measurement based care (RMBC) interventions for mental health—Protocol of a systematic review and meta-analysis. PLoS ONE 19(2): e0297929. https://doi.org/10.1371/journal.pone.0297929

Editor: Qin Xiang Ng, Singapore General Hospital, SINGAPORE

Received: January 23, 2023; Accepted: January 14, 2024; Published: February 16, 2024

Copyright: © 2024 Machleid et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.

Funding: Felix Machleid is funded by the Junior Digital Clinician Scientist Program of the Berlin Institute of Health. Jakob Kaminski was supported by the digital health accelerator that facilitated the spin-off process of Recovery Cat (a software company dedicated to digital health in psychiatry) from Charité. No additional specific grant from any funding agency in the public, commercial or not-for-profit sectors was obtained. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: Jakob Kaminski is shareholder and managing director of Recovery Cat GmbH. Toni Muffel is an employee at Recovery Cat GmbH, Caspar Wiegmann received honorary from Recovery Cat for consulting. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

Introduction

Mental illnesses are estimated to be one of the highest global burdens of disease [ 1 ]. Managing these illnesses presents challenges to patients and clinicians alike. Often, subjective symptom reporting [ 2 , 3 ], memory bias [ 4 ], complex treatment interactions, poorly coordinated transitions from inpatient to outpatient settings [ 5 ], and short, infrequent appointments in outpatient and private practices [ 6 ] result in the loss of information about symptom progression and treatment side effects. Poor continuity of care increases the risk of lower treatment adherence [ 7 ], worsening of disease [ 7 , 8 ], avoidable re-hospitalisations [ 5 , 8 ], and associated costs [ 8 ].

To address these challenges, there has been a surge in development of diagnostic and therapeutic mobile mental health (MMH) technologies. One of the leading use cases for MMH is remote measurement based care (RMBC), which involves asynchronous assessment of (electronic) patient reported outcomes (ePROs) outside of clinical encounters and their use for clinical decision-making and scheduling [ 9 , 10 ]. In addition to ’traditional’ retrospective PRO assessment formats (e.g. validated questionnaires), ambulatory and diary methods have gained interest, as they offer unique insights into patients’ experience of symptoms and well-being in their natural surroundings [ 11 ]. Such methods, termed ecological momentary assessment (EMA), have evolved with technological advances that allow self-reporting of symptoms in an internet or mobile context, including web/online, text messaging or phone-call based methods [ 12 ].

RMBC interventions have gained significance in the context of mental health management because of their ability to continuously track PROs. They offer the potential to improve the detection of deterioration, to personalise treatment plans, and improve accessibility to high quality mental health services [ 9 ].

As technology rapidly advances, studies have shown strong evidence in favour of RMBC, which can improve clinical outcomes and increase treatment adherence [ 13 – 15 ]. For example, a study including 6,424 participants with various psychiatric diagnoses found that continuous feedback to therapists on the course of symptoms was associated with a doubling of therapeutic effects related to individual (or symptomatic) functioning, interpersonal relationships, and social role performance [ 16 ].

Moreover, the benefits of measurement-based care (MBC) during in-person clinical encounters have been recognised before the rise of RMBC. MBC has been linked to faster remission [ 17 , 18 ] compared to treatment as usual (TAU) and fewer missed outpatient appointments [ 19 , 20 ]. MBC can also support clinicians in adjusting treatment quickly and effectively [ 10 , 21 ]. Patients consider MBC helpful [ 22 ], and it may improve doctor patient communication or increase treatment motivation [ 23 , 24 ].

Despite these advantages, asynchronous MBC using digital solutions is not yet widespread in clinical practice resulting in various implementation efforts [ 25 ]. Although there is limited robust scientific evidence from randomised controlled trials or longitudinal studies [ 9 ], there is a significant number of pilot and feasibility studies on RMBC systems. However these studies often have the typical limitations of academic research such as insufficient power or limited bias reduction strategies [ 26 , 27 ]. This leads to heterogeneity in available data and necessitates regular systematic evaluations identifying general trends or effects to increase the adoption of MBC and MMH in clinical practice.

In a 2018 systematic review by Goldberg et al. containing 13 RCTs on RMBC the results were promising [ 9 ]. The review highlighted the short-term feasibility and acceptability of RMBC. However, only three studies isolated the effects of RMBC experimentally, with one reporting greater symptom improvement in the RMBC group and two finding no differences between intervention group and controls. In the planned systematic review and meta-analysis, we aim to provide a comprehensive overview of the current evidence on RMBC in psychiatric care and update the above mentioned findings by Goldberg and colleagues [ 9 ]. Specifically, we will focus on interventions that aim to alleviate disorder-specific symptoms, reduce relapse, and improve recovery-oriented outcomes, global functioning, and quality of life. As the data allows, we will also conduct meta-analyses of relevant outcomes.

Protocol and registration

This protocol follows the guidelines of PRISMA-P (preferred reporting items for systematic review and explanation meta-analysis protocols) checklist ( S1 Table , [ 8 ]). The systematic review and meta-analysis were registered at PROSPERO (reference CRD42022356176).

Eligibility criteria

The research question and inclusion and exclusion criteria ( S2 Table ) were formulated using the PICOS (population, intervention, comparison, outcome, study) framework [ 28 ].

The review will include studies published before August 24th, 2022 examining adults (≥18 years) with mental health disorders according to the International Statistical Classification of Diseases and Related Health Problems (ICD) and/ or the Diagnostic and Statistical Manual (DSM). Interventions delivered solely to family members (either targets of the intervention or in addition to patients) will be excluded.

Interventions

We will include interventions that allow the assessment of self-reported symptoms of mental illness and other aspects of well-being. This feature is not required to be the predominant element of the intervention. Thus, studies that include an evidence-based form of therapy (e.g. cognitive-behavioural therapy, psychodynamic therapies, behaviour therapy or behaviour modification, etc.) plus an RMBC element will also be included.

Due to the variety of included study designs, no specific comparison to a control group is intended.

Studies must report quantitative data, such as changes in symptom-related outcomes, recovery-related outcomes (e.g. empowerment, self-efficacy, hope, social connectedness), (global) level of functioning, relapse, or quality of life. Symptom tracking entirely derived from passive sensing or passive monitoring, as well as quantitative data in the form of correlative data analyses and statistical prediction models, will be excluded.

Study designs

We will comprehensively consider randomised studies, including RCTs, cluster RCTs or factorial RCTs, non-randomised studies, including observational, cohort, cross-sectional or case-control studies; mixed methods studies, and feasibility or pilot studies with available full texts written in English or German.

Information sources and search strategy

The multidisciplinary research team developed a comprehensive search strategy by screening reference lists of scientific and grey literature, including MeSH terms related to (1) mental disorders and psychological distress, (2) measurement-based care, and (3) digital technologies. The syntax of each database search will be slightly modified due to database specifications and the searches will be conducted using PubMed, Medline, Embase, and PsychINFO. The search strategy for each database can be found in S3 Table .

Study selection

For all reports included in the synthesis, data will be collected, extracted, and reviewed in a Google Sheets spreadsheet developed by the research team. In an initial screening, titles and abstracts will each be evaluated in batches by three subgroups of independent researchers (TwM&LS, FM&LH, JK&CW&ToM). In a second screening step, full-text articles of the selected records will be retrieved and imported to the Zotero reference management software. The subgroups will assess a different batch of full-text articles than the ones they formerly screened. Disagreements between the researchers will be discussed to reach a consensus. When no consensus can be reached within the pair/trio, the discussion will be conducted within the entire research team. The rationale for the exclusion of each full text will be provided.

Data extraction

We plan to extract various data: study identification (authors, year of publication, doi, URL), population (e.g. number of cases and controls, diagnosis, age, gender, years pre-university education), tracking- (e.g. mode, items, frequency) and study characteristics (e.g. design, hypotheses, study site, duration, randomisation, post-assessment period, follow-up, outcomes, response rate). Outcomes will be categorised into six categories: (1) symptom-focused or disease-specific outcomes, (2) relapse, (3) recovery-focused outcomes, (4) (global) functioning, (5) quality of life and (6) acceptability. We will consider metrics and timing of measurements for each outcome measurement tool and instrument. We will code p-values and standardised effect sizes (e.g. Cohen’s d, hazard ratio, odds ratio) when available. One researcher will extract all study features, and an additional researcher will review the coding. Discrepancies will be resolved through discussions with a third researcher.

Assessment of bias

Risk of bias in randomised studies will be examined using the Cochrane Collaboration’s tool version 2.0 [ 29 ]. We will use Funnel plot methods to examine publication bias in analyses with more than 10 studies included [ 30 ].

Data synthesis

For all studies included, a detailed description of the data items extracted and relevant results will be provided in narrative synthesis and respective tables.

Meta-analysis

Data analysis will be conducted using RStudio [ 31 ]. All types of randomised clinical studies will be considered for meta-analysis. To account for potential variations in true effects in the studies due to differences in study populations, interventions, and target behaviours, all meta-analyses will be conducted as a random-effects analyses [ 32 ].

Different diagnostic groups such as psychosis, depression or mania will be examined separately. If k>2 studies report the same outcome, they will be meta-analytically pooled. When a single study uses multiple measures to report the same outcome, the one defined as the primary outcome of the study will be favoured, or the more common measure in other studies will be chosen. Same outcomes will be examined across all diagnostic groups, with the inclusion of the diagnosis factor as a covariate in statistical analysis. When different measures are used across studies for similar outcomes, effect sizes will be standardised for pooling [ 32 ]. We will analyse studies that include treatment as usual and active control groups together. If reported, we will use intention to treat data for our analyses.

We will express effect sizes for continuous measures as standardised mean differences (SMDs) and their 95% confidence intervals calculated using the pooled standard deviation of the interventions. SMDs will be presented as values of Hedges’ g . For dichotomous measures, we will calculate risk ratios and combine the studies using the Mantel-Haenszel method. The number needed to treat (NNT) will be calculated to further illustrate the clinical relevance of RMBC interventions.

In the case of cluster RCTs, we consider each cluster as a distinct entity, using summary measures from each individual cluster for meta-analysis. In instances where the RCT does not provide adequate details for analysis, we rely on cluster-specific information, such as the intraclass correlation coefficient, to perform an approximate analysis [ 30 ]. In factorial RCTs, data from each treatment arm is extracted separately and treated as an individual study when relevant to meta-analysis. All findings will be reported transparently [ 30 ].

Sensitivity analyses will be carried out to examine the effect of a specific study on the pooled outcomes. Subgroup analyses separating population characteristics (e.g. mean age, gender, years of pre-university education, severity of illness), study characteristics or RMBC intervention type will be conducted to reduce heterogeneity of pooled estimates.

Heterogeneity

We will assess heterogeneity using the I 2 statistic, p -value of χ 2 test and visual inspection of forest plots.

Confidence in cumulative evidence

Cumulative evidence will be evaluated according to the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework.

Dissemination

We will publish the results of our study in a peer-reviewed journal in the field of psychiatry, digital health or telemedicine, and present them at international conferences and workshops.

Our study will summarise existing evidence on the use of RMBC interventions in mental health care and offer insights into their impact on clinical-, health service utilisation-, recovery- and quality of life related outcomes.

Supporting information

S1 table. prisma-p 2015 checklist..

https://doi.org/10.1371/journal.pone.0297929.s001

S2 Table. Inclusion and exclusion criteria related to the PICOS design.

https://doi.org/10.1371/journal.pone.0297929.s002

S3 Table. Search syntax, date of searches, and number of results returned for each database, number of references for review.

https://doi.org/10.1371/journal.pone.0297929.s003

Acknowledgments

We want to thank our colleagues at Recovery Cat for their support and thoughtful input.

  • View Article
  • PubMed/NCBI
  • Google Scholar
  • 30. Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Cochrane Collaboration; 2008.
  • 31. RStudio Team. In: RStudio: Integrated Development for R [Internet]. 2022 [cited 24 Dec 2022]. http://www.rstudio.com/ .
  • 32. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta-analysis. John Wiley & Sons; 2021.
  • Open access
  • Published: 19 February 2024

Effects of game-based physical education program on enjoyment in children and adolescents: a systematic review and meta-analysis

  • Weilong Mo 1 , 2 ,
  • Jamalsafri Bin Saibon 1 ,
  • Yaming LI 1 ,
  • Jiequan Li 3 &
  • Yanwu He 4  

BMC Public Health volume  24 , Article number:  517 ( 2024 ) Cite this article

231 Accesses

Metrics details

The objective of this study was to conduct a systematic review to summarize and assess the advancements lately made on the enjoyable impacts of game-based physical education interventions on children and adolescents. Additionally, it attempted to identify the effects and variables influencing the enjoyable outcomes of children and adolescents’ engagement in physical education games, through meta-analysis.

This study involves a comprehensive search of different databases like Web of Science, PubMed, Embase, EBSCOhost, Cochrane, and Scopus. Specific criteria are established for the selection process to make sure the relevant literature included. The quality assessment of the included researches is conducted based on the guidelines outlined in the Cochrane 5.1 handbook. Review Manager 5.3 software is employed to synthesis the effect sizes. Additionally, bias is assessed using funnel plots, and to identify potential sources of heterogeneity, subgroup analyses are performed.

A total of 1907 academic papers, out of which 2 articles were identified via other data sources. The present study examined the impact of a pedagogical intervention involving physical education games on the enjoyment experienced by children and adolescents. The results indicated a significant positive effect (MD = 0.53, 95%CI:[0.27,0.79], P  < 0.05) of this intervention on enjoyment. Subgroup analyses further revealed that both boys (MD = 0.31, 95%CI:[0.13,0.50], P  < 0.05) and girls (MD = 0.28, 95%CI:[0.05,0.51], P  < 0.05) experienced increased pleasure compared to traditional physical education. Additionally, children under 12 years of age (MD = 0.41, 95%CI:[0.17,0.64], P  < 0.05) benefited from sessions lasting at least 30 minutes or more per session (MD = 0.40, 95%CI:[0.19,0.60], P  < 0.05), occurring 1 to 3 times per week (MD = 0.28, 95%CI:[0.16,0.40], P  < 0.05), and lasting for more than 3 weeks (MD = 0.81, 95%CI:[0.29,1.34], P  < 0.05). These findings suggest that the implementation of physical education games can be an effective approach to teaching this subject.

Conclusions

1) Interventions using physical games have been shown to yield beneficial outcomes in terms of enhancing the enjoyment experienced by children and adolescents. 2) The effectiveness of treatments aimed at promoting enjoyment among children and adolescents is influenced by several aspects, including gender, age, duration and frequency of physical activity, as well as the specific cycle of activity used.

Peer Review reports

Introduction

Enjoyment is a subjective experience with pleasant emotions, such as pleasure, like, and fun [ 1 ]. Children and adolescents are naturally motivated by enjoyable experiences during the learning process, which enhances their academic achievement, involvement and effort in learning [ 2 , 3 , 4 ], this, in turn, leads to more effective and long-lasting learning [ 5 , 6 ]. In contrast, falling enjoyment can diminish their interest and engagement. Therefore, the cultivation of enjoyable feelings in children and adolescents has a crucial role in enhancing educational achievements.

Studies proved that physical education has a beneficial influence on the psychological and physical health of children and adolescents [ 7 , 8 ], as well as on the prevention of disease problems [ 9 ]. The influence of physical education on the enjoyment of children and adolescents, particularly in relation to emotions, has clear benefits [ 10 ]. Physical education activities for children and adolescents not only have the power to enhance individual happiness, but also foster a positive team atmosphere and promote collaboration and socialization [ 11 ]. Thus, it is essential to explore the positive impact of these activities on enjoyment and mental health.

Compared with traditional physical education, physical education games have several advantages. The implementation of physical play interventions has the potential to facilitate the acquisition of knowledge and skills among children and adolescents [ 12 , 13 ], enabling them to get enjoyment from the process of learning. A combination of entertainment components into traditional physical education (PE) is effective in motivating non-athletic students to actively engage in PE lessons, which cannot be achieved through organized sports [ 14 ]. Liao et al. [ 15 ] further explain that games not only enhance students’ satisfaction with PE lessons, but also facilitate skill development, create a relaxed play environment, foster interpersonal interactions, and offer opportunities for cooperation and socialization.

Hence, the implementation of physical games teaching offers a new and innovative approach within the context of traditional physical education classes [ 16 ]. This approach to learning is not only pleasurable for students, but also meets their requirement for social and physical engagement in the educational process and, most notably, contributes to a key part in sustaining the involvement of children and adolescents in physical education and sports [ 17 ].

Through a comprehensive analysis of 16 academic studies, it has been noticed that further research is needed regarding the efficacy of applying physical education games to enhance enjoyment in children and adolescents. Since the majority of studies show a beneficial effect on enjoyment [ 1 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ]. However, five studies remain uncertain about the impact of these games [ 14 , 27 , 28 , 29 , 30 ], and there is even a case where teaching with games seems to reduce students’ enjoyment [ 27 ]. Currently, there is not enough comprehensive analysis on how physical game teaching impacts the enjoyable feelings of children and adolescents, and also on the potential factors that may influence those effects (such as gender, age, and duration of the interventions).

The aim of this study was to explore the following topics with a meta-analysis: 1) whether teaching games in physical education has a beneficial influence on enjoyment experienced by children and adolescents, and 2) how other essential elements mitigate the influence of games teaching in physical education on enjoyment experienced by children and adolescents. Due to inconsistent findings from earlier research, there is no consensus on the link of physical games for enjoyment in children and adolescents; yet there is optimism that teaching physical play may have a favorable impact on enjoyment.

Search strategy and standards for selection

This research is conducted under the guidance of Cochrane Handbook for the Systematic Review of Interventions [ 31 ] and the PRISMA Statement Specification for Systematic Review and Meta-analysis [ 32 ].

This research explores six databases, namely Web of Science, PubMed, Embase, EBSCOhost, Scopus and Cochrane. PubMed is primarily adopted to identify medical terms (Mesh). The search period spans from the initiation of database collection until July 25, 2023. Both subject terms and the free word approach are included in the search. The following table consists of many columns. This study firstly focuses on the research object and then emphasizes the intervention strategy. In the third line, the result index is built by connecting the search words using the logical operator “or” inside each group of search terms. Additionally, the search phrases are linked by “or” between each set of search terms. Table  1 displays the whole search words used in the six databases.

Criteria for inclusion and exclusion

Research that meet the following requirements are selected for systematic evaluation: (1) the intervention modality is physical game teaching; (2) the subjects are children and adolescents (3–18 years old); (3) the outcome indicator is the inclusion of enjoyment related MeSH and Entry Terms; (4) the use of a control group; and (5) the articles are written in English.

Researches reach the following standards are omitted from systematic evaluation: (1) the intervention method is not physical game teaching; (2) the experimental subjects are infants, adults, animals, and specific populations (psychiatric patients), etc.; (3) the outcome indicator is the absence of pleasure-related subject terms and free words; (4) no control group is included; (5) the articles are written in other non-English languages; (6) review article; and (7) conference articles.

Screening process

Upload the relevant literature to Endnote (version X9) for organization. Following this, duplicate results are screened by two authors (MWL and LYM) independently. The screening process include reviewing titles, review articles, conference papers, and animal experiments. Read the abstracts to exclude articles that fail to meet such criteria as study subjects or interventions. Finally, read the full text of selected articles to exclude those that are inaccessible, non-English and does not provide end point indicators. The process involves an initial screening of eligible articles, a discussion of any discrepancy and reaching a consensus with the third author (LJQ). Ultimately, 16 articles are selected for the systematic analysis. Detailed information about these steps are presented in the PRISMA flowchart (refer to Fig.  1 ).

figure 1

Flowchart for inclusion and exclusion of studies

Extraction of data and quality evaluation

Three authors (MWL, LYM, and LJQ) extract data from eligible papers in an impartial manner. Any divergence is resolved through discussion until an agreement is reached. The extracted information is then placed in the publications respectively [ 1 ]. the extracted information primarily includes the name of the first author and the publication year [ 2 ]. the subjects’ features encompass the total sample size, age, and gender [ 3 ]. detailed data, like duration, frequency, and cycle, about the teaching process of physical education and sport games are included [ 4 ]. the intervention tool employed in the study is a questionnaire or scale designed to measure the degree of pleasure, satisfaction and motivation before and after physical education [ 5 ]. the intervention items utilized in the study are game items specifically employed for physical education [ 6 ]. the outcome indicators encompass various factors, including the level of pleasure, satisfaction, and motivation before and after physical education and sport activities. Pleasure may be defined as the state of experiencing gratification, enjoyment, satisfaction, delight, or fun [ 7 ]. the writers emphasize the importance of significant results.

The Cochrane 5.1 handbook is applied to assess the quality of bias. The evaluation includes one aspect, namely the random allocation procedure and the concealing of allocation schemes. In this research, the evaluation criteria are: 3) participant blinding and outcome assessment, 4) complete outcome data, 5) selective report findings, and 6) bias from other sources. Each criterion is assessed as having either a low risk (an indication for meeting the criterion), a high risk (an indication for not satisfying the criterion), or a medium risk (if not mentioned), with a note explaining the reason for this assessment. Two researchers assesses the article quality independently. Any divergence in their evaluations are solved by discussing with another author. Figure  2 shows the evaluation findings and the detailed information. A sensitivity analysis is conducted whereby each article is systematically removed one at a time. The analysis reveals that the findings are mostly unchanged, which suggests that the results are robust and reliable.

figure 2

Cochrane risk of bias evaluation

Statistical analysis

Review Manager 5.3, a statistical software, is applied to merge effect sizes and assess for bias. In this analysis, the indicator continuous variable is incorporated, making the results presented as mean ± standard deviation (Mean ± SD). The I 2 and Q tests are adopted to evaluate heterogeneity between studies. The fixed-effects model is applied as the I 2  < 50% or p  > 0.1, indicating the studies lack statistical heterogeneity. In contrast, a random-effects model is adopted to evaluate publication bias using funnel plots and to examine the reliability of the findings.

Search results

A comprehensive search for 1907 articles in total is conducted. The databases for search are Web of Science (1473 articles), PubMed (33 articles), Embase (75 articles), EBSCO host (154 articles), Cochrane (95 articles), and Scopus (75 articles). All identified articles are uploaded to Endnote (version X9), a reference management software. By examining the article titles, a total of 207 duplicate items are eliminated from further analysis. The articles are uploaded to Endnote (version X9), and after examining the titles, a total of 207 duplicates are removed. The dataset comprises 321 conference papers, 44 review articles, 31 articles with inconsistent subjects, 1 article applying inconsistent measurement tools, 94 articles employing inconsistent interventions, 66 articles featuring different endpoints as determined after reading the full text. 14 articles presenting results deviate from the mean ± standard deviation format. A total of 27 publications without a control group are identified, and 8 articles written in non-English languages are excluded. Additionally, the full texts of 26 articles are incomplete or unavailable. Hence, a final set of 16 eligible articles is included in the meta-analysis.

Basic features of the included articles

The analysis encompasses 16 articles, which collectively examines 17 studies. The total sample size has 2181 participants, with 1139 individuals assigned to the experimental group and 1042 individuals assigned to the control group. There were 1096 male participants and 1048 female ones. The age of the samples ranges from 4.9 to 15.62 years. The intervention duration covers from 15 to 90 minutes, with frequencies from 1 to 9 times each week. The intervention cycles span from 2 to 14 weeks. The interventions are concentrated on sports and games programs. The assessed outcome indicators include enjoyment and satisfaction, intervention instruments and major findings. See Table 2 for detailed information.

Quality assessment

This research examines the literature about the random assignment process and specifically focuses on six studies that meet the inclusion criteria [ 20 , 22 , 23 , 26 , 28 , 33 ]. The remaining research do not provide details about the randomization process. None of the 17 studies mention whether the allocation is concealed or not in the allocation scheme concealment. In terms of blinding, researchers on the subjects and inform them about the tests. Thus, the subjects are not blinded. Consequently, all 17 studies are deemed to have a high risk. The evaluation of the findings is featured with uncertainty. Two studies had a high incidence of staff turnover as for the completeness of the outcome data [ 20 , 27 ]. None of the 15 studies shows any subject or data loss, and all of them are considered to have low risk. The included studies show no further selective reporting or biases, and all of them are considered to have low risk of bias.

Tests for bias

This research includes outcome indicators for analysis, and the funnel plot demonstrates a distribution that is symmetrical, indicating the absence of publication bias, as seen in the Fig.  3 .

figure 3

Bias funnel plot

Efficacy tests

The relationship between teaching games in physical education and enjoyment of children and adolescents.

Heterogeneity tests were performed on the articles that were included in the analysis. Out of the total, 17 research (comprising 16 papers) indicated an altered association between the enjoyment experienced by children and adolescents in the context of teaching physical education games [ 1 , 14 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 33 ]. The researchers apply a random effects model to collect the findings about the articles’ outcome indicators. This study includes 17 studies and 2181 participants in total, with 1042 in the control group and 1139 in the experimental group. The present research provides evidence supporting the favorable impact of a physical education intervention using games on the positive emotions of children and adolescents in the experimental group (MD = 0.53, 95% CI: [0.27, 0.79], P  < 0.05), as depicted in Fig.  4 .

figure 4

Forest plot depicting the relation between physical game teaching and enjoyment in children and adolescents

Subgroup analyses

The combined impact size data for physical play teaching interventions on children and adolescents show a significant degree of variation. It is achieved by analyzing subgroups based on gender, age, duration, frequency, and cycling as potential influencing factors.

The results of subgroup analyses examining the influence of gender, age, duration, frequency, and cycling on the effects of games in physical education indicate that such games can enhance enjoyment of boys (MD = 0.31, 95% CI:[0.13,0.50], P  < 0.05) and positively affect girls (MD = 0.28, 95%CI:[0.05,0.51], P  < 0.05). Furthermore, it is observed that children aged 12 experienced an increasing enjoyment (MD = 0.41, 95% CI:[0.17,0.64], P  < 0.05), whereas adolescents aged 12 and above do not show a similar increase ( P  > 0.05). The duration of physical education sessions ranging from 30 to 60 minutes (MD = 0.40,95%CI:[0.19,0.60], P  < 0.05) can provide a favorable impact on enjoyment experienced by children and adolescents. Moreover, extending the duration of physical education beyond 60 minutes (MD = 0.55,95%CI:[0.10,1.00], P  < 0.05) may also improve their enjoyment. However, noticeably, durations shorter than 30 minutes do not show the same good effect ( P  > 0.05). It is more feasible to provide physical game teaching within a frequency range of 1 to 3 sessions per week (MD = 0.28,95%CI:[0.16,0.40], P  < 0.05) to elicit enjoyment among children and adolescents. Conversely, it is unsuitable to give physical game instructions, exceeding the threshold of 3 sessions per week ( P  > 0.05). The optimal duration for physical game teaching to elicit enjoyable outcomes in children and adolescents is between 3 to 6 weeks (MD = 0.81, 95%CI:[0.29,1.34], P  < 0.05), but durations beyond 6 weeks are also considered acceptable (MD = 0.29, 95%CI:[0.10,0.48], P  < 0.05). In contrast, it is not a proper option to be engaged in physical games for less than 3 weeks ( P  > 0.05). Hence, such factors as gender, age, duration, frequency, and cycle contribute significantly to the observed variation in satisfaction, as seen in Figs.  5 , 6 , 7 , 8 , 9 .

figure 5

Forest plot depicting gender subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 6

Forest plot depicting age subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 7

Forest plot depicting duration subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 8

Forest plot depicting frequency subgroup relationship between physical game teaching and enjoyment in children and adolescents

figure 9

Forest plot depicting cycle subgroup relation between physical game teaching and enjoyment in children and adolescents

Main research analyses

The results of this study, including the analysis of 17 studies, show that the adoption of physical education game-based intervention has a beneficial effect on the enjoyment levels of children and adolescents. Such corresponds to the idea offered by Tornero and Capella, 2017, which claims that playing games adjusts to the emotional state of children and adolescents [ 34 ]. This advantageous feeling state can further improve their engagement in school sports activities [ 35 , 36 ]. Physiological studies prove that engaging in physical activity or exercise causes the release of endorphins from the pituitary gland and subthalamus. Endorphins are hormones that induce feelings of calmness and pleasure, enhancing mood and creating an enjoyable experience for children and adolescents during physical education programs, including games [ 37 , 38 , 39 ]. Furthermore, when it comes to content, the teaching of physical education games appears to enhance the enjoyment experienced by children and adolescents to a greater extent than the classes of traditional physical education. In their study, Batez et al. (2021) discovered that adolescents in the experimental group who participated in the Teaching Games for Understanding (mini-volleyball) way experienced a greater sense of satisfaction compared to the control group during the post-test phase [ 23 ]. Lopez-Lemus et al. (2023) noticed that analyzing the pre-test and post-test results of both the experimental and control groups revealed that 67 students who were part of the Sport Education (SE)/Teaching for understanding (TGfU) experimental group, specifically focusing on handball, revealed enhancements in-game performance, enjoyment, perceptual skills, and intentions [ 21 ]. Similarly, researches on dance movement games and basketball games show superior levels of enjoyment compared to traditional teaching methods [ 19 , 40 ]. Hence, this research posits that including games into physical education courses may effectively enhance the enjoyment of children and adolescents, making it a recommended approach compared to programmes that do not use games.

Gender analysis

This study claims that practicing physical sports might affect enjoyment among individuals of different genders, with boys expressing a greater chance of experiencing enjoyment compared to girls. Research has shown that as they get older boys and girls display unique preferences. In a cultural analysis conducted by Joseph et al., 25 African American women were surveyed regarding their engagement in physical activities. The majority of these women reported positive and enjoyable experiences in childhood, but their feelings were not apparent during their youth [ 41 ]. Additionally, female adolescents had a lower frequency of pleasurable meets in physical exercise compared to male adolescents, and they also displayed negative emotions towards engaging in physical activity [ 42 ]. Even so, variations in the level of enjoyment based on gender are likely to be impacted by different types of sports games. Girls have a preference for cooperative activities, particularly dancing games [ 43 ], whereas boys seem to choose competitive fitness games [ 44 ]. In all, both males and females can experience enjoyment in physical education games, still, variations in the level of enjoyment may arise due to factors such as age and the specific type of game. It seems that gender alone is not the sole determinant of enjoyment, and further study is required to identify other contributing factors.

Analysis of age

According to this study, teaching physical education games has a major effect on the enjoyable feelings of children below the age of 12. And yet, it does not have a substantial influence on teenagers aged 12 and above. In the opinion of Velez & Garcia, children between the ages of 9 and 12 have better levels of individual feelings of happiness compared to teenagers aged 13 to 17 [ 45 ]. Play is an essential element in the development of children’s motor skills and is intrinsically linked to enjoyment, which serves as a motivation for children to engage in physical exercise [ 46 ]. According to Bremer et al., a study demonstrated that children between the ages of 6 and 13 with autism who enjoyed their physical education sessions were more likely to engage in other physical activities [ 47 ]. Academic competition at school is an important factor that hinders the development of enjoyable feelings in teens, this is mostly caused by the negative effects of stress-induced depression and anxiety [ 48 ]. Mangerud found that engaging in physical exercise has an impact on the positive emotions of adolescents with anxiety disorders, including their enjoyment of sports circumstances [ 49 ]. As a result, teaching youngsters under the age of 12 physical sports proves to be a more successful method for obtaining enjoyment compared to teenagers aged 12 and above.

Analysis of duration

The present research offers that applying a physical game lesson beyond a duration of 30 minutes has a favorable impact on the enjoyment of children and adolescents, but less than 30 minutes appears to have little to no effect. This corresponds to the findings of the Gil-Madrona, when children participated in 45 minutes of popular cooperative and cooperative-oppositional games [ 50 ]. Physical exercise in children and teenagers increases the release of neurotransmitters including dopamine and (−)-norepinephrine. These substances help to decrease depression and anxiety, leading to increased feelings of euphoria, achievement, and overall well-being, which improve over time [ 51 , 52 , 53 ]. Previous research has shown that children tend to get pleasure from short periods of intense physical activity followed by times of relaxation (similar to outside play), whereas adults may have a preference for lengthier activities [ 54 ]. However, Tobin et al. carried out experiments with participants of varying durations and determined that a 12-minute length of time was considered insufficient for players to become fully engaged in the game, thus serving merely as a warm-up period [ 55 ]. In addition, another study corroborated these findings by establishing that children exhibited diminished motivation and failed to experience enhanced enjoyment when engaging in sports games for a brief duration of 20 minutes [ 56 ]. In the end, engaging in sports games for no less than 30 minutes can lead to improved outcomes and heightened enjoyment for children and adolescents.

Analysis of frequency

The analyses suggested that a teaching intervention based on physical education and games held 1 to 3 times per week is suitable for children and adolescents to experience enjoyment. However, doing more than 3 sessions per week seems unsuitable. Studies indicate a correlation between the frequency of participating in physical activity and experiencing positive emotions [ 57 ]. Furthermore, sustaining a proper frequency of physical activity could promote the feeling of Feelings of happiness. For instance, Batez and Gil-Arias both applied the teaching games for understanding (TGfU) approach in a physical education program, results indicated that students’ level of enjoyment somewhat improved when the games were taught twice weekly [ 23 , 58 ]. However, excessive participation in game activities without sufficient time for rest and recovery can lead to the build-up of lactic acid in the muscles, resulting in increased physical fatigue and negatively impacting the individual’s mood, finally diminishing the enjoyment of the gaming experience [ 59 , 60 , 61 , 62 ]. Temporary breaks can effectively facilitate physical recovery during physical education games, it not only promotes bodily rejuvenation but also enhances the enjoyment of children and adolescents [ 63 ]. Therefore, it is advisable to offer 1–3 lessons per week to optimise the teaching of physical education games.

Analysis of cycles

This study stated that physical education game teaching interventions lasting between 3 to 6 weeks and more than 6 weeks are ideal for improving the enjoyable outcomes of children and adolescents. Conversely, interventions lasting less than 3 weeks are not advisable. This conclusion is supported by the findings of previous studies. Some curriculum interventions like Zetou et al. designed a 4-week ‘Play and Stay’ tennis teaching programme, Jones et al.’s 6-week Teaching Games for Understanding and Fernandez-Rio et al.’s 9-week Gamification [ 22 , 24 , 26 ]. Findings show an increase in students’ enjoyment, linked to the regular meeting of their intrinsic drive in the physical education classroom. Several studies indicated that short physical procedures lasting only 1 week do not effectively assess the intrinsic motivation of participants [ 64 , 65 ], thus posing difficulties with stimulating the generation of enjoyable sensations in persons. Moreover, extended periods of physical play may result in decreased intrinsic motivation or boredom in children and adolescents [ 66 , 67 ]. Fu et al. and Zeng et al. propose that while physical play at first brings joy, enjoyment diminishes over time [ 33 , 68 ]. In conclusion, children and adolescents should engage in playful activities for a minimum of 3 weeks, while also ensuring that the play program offers a variety of activities and rich content to enhance their enjoyment.

This study applies a meta-analysis to examine the significance of teaching games in physical education regarding emotional delight experienced by children and adolescents. Gender, age, duration, frequency and cycles may be the reasons for variances impacting the research outcomes. This research finds that male participants are more likely to show enjoyable behavior compared with their female counterparts as for games teaching in physical courses. However, it should be noted that gender disparities may be influenced by variables like age and the specific kind of sports taught in class. Besides,Children engage in a minimum of 30 minutes every session, attending 1 to 3 sessions per week, so guaranteeing that the physical education and games curriculum is delivered for a span exceeding 3 weeks. This approach aims to foster positive affective experiences among children, thereby facilitating the attainment of optimum outcomes.

Limitations and future research

Apart from the meaningful findings, this research also has some drawbacks. Firstly, it adopts a meta-analytical approach to examine the influence of games teaching in physical education on enjoyment in children and adolescent. It primarily focuses on the outcomes of curriculum and teaching implementation. Consequently, the results may not be applicable to other contexts, such as after-school physical game activities, community physical game activities, and family physical game activities. Furthermore, the 17 studies analyzed in this research have insufficient data on duration, frequency, and period. This insufficient information may influence the statistical accuracy of conducting effect size tests. Additionally, the 17 studies fail to offer any data about the intensity of the activities employed in games teaching in physical education, such as heart rate, oxygen uptake, and respiratory rate. Consequently, future studies can address this gap in knowledge. Additionally, the current research does not ascertain the ideal upper threshold for the duration of engagement in the activity. This aspect warrants further exploration in a later literature review. Additionally, it is crucial to evaluate other variables which may influence the research outcomes, such as the specific nature of the sports game being analyzed. For further study, it would be fruitful to classify different sorts of sports games to improve the whole quality of the research.

Availability of data and materials

All data generated or analysed during this study are included in this published article.

Gao Z, Zhang T, Stodden D. Children’s physical activity levels and psychological correlates in interactive dance versus aerobic dance. J Sport Health Sci. 2013;2(3):146–51.

Article   Google Scholar  

Sharif AF, Kasemy ZA, Alquraishi NA, Alshaikh KN, Alfaraj AH, Alibrahim EA, et al. Inserting an erroneous element in the answer key of grouped team readiness assurance test in team-based learning: a randomized controlled trial in clinical toxicology. Adv Med Educ Pract. 2022;13:865–82.

Article   PubMed   PubMed Central   Google Scholar  

Izadpanah S. The mediating role of academic passion in determining the relationship between academic self-regulation and goal orientation with academic burnout among English foreign language (EFL) learners. Front Psychol. 2022;13:1–12.

Google Scholar  

Zhang C, Mao L, Li N, Gu X. Chinese EFL Students’ social-emotional competence, grit, and academic engagement. Front Psychol. 2022;13:914759.

Trajkovik V, Malinovski T, Vasileva-Stojanovska T, Vasileva M. Traditional games in elementary school: relationships of student’s personality traits, motivation and experience with learning outcomes. PLoS One. 2018;13(8):e0202172.

Yu J, Huang C, Han Z, He T, Li M. Investigating the influence of interaction on learning persistence in online settings: moderation or mediation of academic emotions? Int J Environ Res Public Health. 2020;17(7):2320.

Yang J, Chen M. Construction of sports and health data resources and transformation of teachers’ orientation based on web database. J Healthc Eng. 2022;2022.

Triaca LM, Frio GS, França MTA. A gender analysis of the impact of physical education on the mental health of brazilian schoolchildren. SSM-Popul Health. 2019;8:100419.

Benka Wallén M, Franzén E, Nero H, Hagströmer M. Levels and patterns of physical activity and sedentary behavior in elderly people with mild to moderate Parkinson disease. Phys Ther. 2015;95(8):1135–41.

Article   PubMed   Google Scholar  

Fin G, Moreno-Murcia JA, León J, Baretta E, Nodari Júnior RJ. Teachers’ interpersonal style in physical education: exploring patterns of students’ self-determined motivation and enjoyment of physical activity in a longitudinal study. Front Psychol. 2019;9:2721.

Lim Y, Han A, Lee M, Kim M. The effects of an online-offline hybrid exercise program on the lives of children with cerebral palsy using wheelchairs during the COVID-19 pandemic in Korea. Int J Environ Res Public Health. 2022;19(12).

Simón-Piqueras JÁ, Cano-Noguera F, Zamorano-García D, Gil-Madrona P. Efecto de un programa de iniciación a la natación basado en juegos motores acuáticos sobre la autoestima corporal y socio afectiva en niñas y niños de 5 y 6 años de edad. Rev Electrón Interuniv Form Profr. 2023;26(3):75–88.

Simon-Piqueras JA, Prieto-Ayuso A, Gomez-Moreno E, Martinez-Lopez M, Gil-Madrona P. Evaluation of a program of aquatic motor games in the improvement of motor competence in children from 4 to 5 years old. Children (Basel, Switzerland). 2022;9(8).

Azlan AB, Ismail N, Fauzi NFM, Abd TR. Playing traditional games vs. free-play during physical education lesson to improve physical activity: a comparison study. Pedagogy Phys Cult Sports. 2021;25(3):178–87.

Liao CC, Hsu CH, Kuo KP, Luo YJ, Kao CC. Ability of the sport education model to promote healthy lifestyles in university students: a randomized controlled trial. Int J Environ Res Public Health. 2023;20(3).

Vaghetti CAO, Monteiro-Junior RS, Finco MD, Reategui E, da Costa S, Botelho S. Exergames experience in physical education: a review. Phys Cult Sport Stud Res. 2018;78(1):23–32.

Baysden E, Mendoza N, Callender C, Deng Z, Thompson D. Teen reactions to a self-representational avatar: a qualitative exploration. J Sport Health Sci. 2022;11(2):157–63.

Segura-Robles A, Fuentes-Cabrera A, Parra-Gonzalez ME, Lopez-Belmonte J. Effects on personal factors through flipped learning and gamification as combined methodologies in secondary education. Front Psychol. 2020;11.

Gil-Arias A, Harvey S, García-Herreros F, González-Víllora S, Práxedes A, Moreno A. Effect of a hybrid teaching games for understanding/sport education unit on elementary students’ self-determined motivation in physical education. Eur Phys Educ Rev. 2020;27(2):366–83.

Kolovelonis A, Goudas M. The effects of cognitively challenging physical activity games versus health-related fitness activities on Students’ executive functions and situational interest in physical education: a group-randomized controlled trial. Eur J Investig Health Psychol Educ. 2023;13(5):796–809.

PubMed   PubMed Central   Google Scholar  

Lopez-Lemus I, Del Villar F, Rodriguez-Gutierrez A, Gonzalez-Silva J, Moreno A. Could the hybridization of the SE/TGfU pedagogical models be an alternative for learning sports and promoting health? School context study. Children-Basel. 2023;10(5).

Fernandez-Rio J, Zumajo-Flores M, Flores-Aguilar G. Motivation, basic psychological needs and intention to be physically active after a gamified intervention Programme. Eur Phys Educ Rev. 2022;28(2):432–45.

Batez M, Petrusic T, Bogataj S, Trajkovic N. Effects of teaching program based on teaching games for understanding model on volleyball skills and enjoyment in secondary school students. Sustainability. 2021;13(2).

Jones RJA, Marshall S, Peters DM. Can we play a game now? The intrinsic benefits of TGfU. Eur J Phys Health Educ. 2010;4(2):57–63.

Navarro-Paton R, Basanta-Camino S, Gomez CA. Cooperative games: incidence in motivation, basic psychological needs and enjoyment in primary school. Sportis-Sci Tech J School Sport Phys Educ Psychomotricity. 2017;3(3):589–604.

Zetou E, Koronas V, Athanailidis I, Koussis P. Learning tennis skill through game play and stay in elementary pupils. J Hum Sport Exerc. 2012;7(2):560–72.

Quintas A, Bustamante JC. Effects of gamified didactic with exergames on the psychological variables associated with promoting physical exercise: results of a natural experiment run in primary schools. Phys Educ Sport Pedagog. 2023;28(5):467–81.

Engels ES, Freund PA. Effects of cooperative games on enjoyment in physical education-how to increase positive experiences in students? PLoS One. 2020;15(12):1–14.

Garcia-Castejon G, Camerino O, Castaner M, Manzano-Sanchez D, Jimenez-Parra JF, Valero-Valenzuela A. Implementation of a hybrid educational program between the model of personal and social responsibility (TPSR) and the teaching games for understanding (TGfU) in physical education and its effects on health: an approach based on mixed methods. Children (Basel, Switzerland). 2021;8(7):1–22.

Paton RN, Ferreiro MC, Nemina RE. Effect of a didactic unit based on competitive games in the motivation, basic psychological needs and enjoyment in primary education students. Sportis-Sci Tech J School Sport Phys Educ Psychomotricity. 2018;4(1):111–25.

Higgins JPT, Thomas J, Chandler J, Cumpston M, li TJ, Page MJ, Welch VA. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. Cochrane Collaboration; 2019. p. 13–655.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.

Fu Y, Burns RD, Constantino N, Zhang P. Differences in step counts, motor competence, and enjoyment between an exergaming group and a non-exergaming group. Games Health J. 2018;7(5):335–40.

Tornero M, Capella C. Change during psychotherapy through sand play tray in children that have been sexually abused. Front Psychol. 2017;8:617.

Bates LC, Zieff G, Stanford K, Moore JB, Kerr ZY, Hanson ED, et al. COVID-19 impact on behaviors across the 24-hour day in children and adolescents: physical activity, sedentary behavior, and sleep. Children (Basel, Switzerland). 2020;7(9):138.

PubMed   Google Scholar  

Gray S, Lorenzino L, Garcia BE. Not academic enough? Enjoyment of physical education and the arts and school engagement in early and middle adolescence. Retos. 2018;35:301–9.

Diggins AD, Hearn LE, Lechner SC, Annane D, Antoni MH, Whitehead NE. Physical activity in black breast cancer survivors: implications for quality of life and mood at baseline and 6-month follow-up. Psycho-Oncology. 2017;26(6):822–8.

Gitimoghaddam M, Vanderloo LM, Hung R, Ryce A, McKellin W, Miller A, et al. Impacts of participation in community-based physical activity programs on cognitive functions of children and youth with neurodevelopmental disabilities: a scoping review. Brain Sci. 2021;11(2):195.

Azevedo Da Silva M, Singh-Manoux A, Brunner EJ, Kaffashian S, Shipley MJ, Kivimaki M, et al. Bidirectional association between physical activity and symptoms of anxiety and depression: the Whitehall II study. Eur J Epidemiol. 2012;27(7):537–46.

Staiano AE, Marker AM, Beyl RA, Hsia DS, Katzmarzyk PT, Newton RL. A randomized controlled trial of dance exergaming for exercise training in overweight and obese adolescent girls. Pediatr Obes. 2017;12(2):120–8.

Article   CAS   PubMed   Google Scholar  

Joseph RP, Ainsworth BE, Mathis L, Hooker SP, Keller C. Utility of social cognitive theory in intervention Design for Promoting Physical Activity among African-American women: a qualitative study. Am J Health Behav. 2017;41(5):518–33.

Bigman G, Rajesh V, Koehly LM, Strong LL, Oluyomi AO, Strom SS, et al. Family cohesion and moderate-to-vigorous physical activity among Mexican origin adolescents: a longitudinal perspective. J Phys Act Health. 2015;12(7):1023–30.

Gao Z, Zhang P, Podlog LW. Examining elementary school children’s level of enjoyment of traditional tag games vs. interactive dance games. Psychol Health Med. 2014;19(5):605–13.

Buys R, Claes J, Walsh D, Cornelis N, Moran K, Budts W, et al. Cardiac patients show high interest in technology enabled cardiovascular rehabilitation. BMC Med Inform Decis Mak. 2016;16:95.

Velez CM, Garcia Garcia HI. Medicion de la calidad de vida en ninos. Iatreia. 2012;25(3):240–9.

Pesce C, Masci I, Marchetti R, Vazou S, Saakslahti A, Tomporowski PD. Deliberate play and preparation jointly benefit motor and cognitive development: mediated and moderated effects. Front Psychol. 2016;7:349.

Bremer E, Martin Ginis KA, Bassett-Gunter RL, Arbour-Nicitopoulos KP. Factors associated with participation in physical activity among Canadian school-aged children with autism Spectrum disorder: an application of the international classification of functioning, disability and health. Int J Environ Res Public Health. 2020;17(16):5925.

Lupien SJ, McEwen BS, Gunnar MR, Heim C. Effects of stress throughout the lifespan on the brain, behaviour and cognition. Nat Rev Neurosci. 2009;10(6):434–45.

Mangerud WL, Bjerkeset O, Lydersen S, Indredavik MS. Physical activity in adolescents with psychiatric disorders and in the general population. Child Adolesc Psychiatry Ment Health. 2014;8(1):2.

Gil-Madrona P, Pascual-Francés L, Jordá-Espi A, Mujica-Johnson F, Fernández-Revelles AB. Affectivity and motor interaction in popular motor games at school. Apunts Educ Fís Deportes. 2020;139:42–8.

Liu H, Liang J, Wang K, Zhang T, Liu S, Luo J. Mood status response to physical activity and its influence on performance: are Chronotype and exercise timing affect? Int J Environ Res Public Health. 2023;20(4):2822.

Motaghinejad O, Motaghinejad M, Motevalian M, Rahimi-Sharbaf F, Beiranvand T. The effect of maternal forced exercise on offspring pain perception, motor activity and anxiety disorder: the role of 5-HT2 and D2 receptors and CREB gene expression. J Exerc Rehabil. 2017;13(5):514–25.

Lott SA, Burghardt PR, Burghardt KJ, Bly MJ, Grove TB, Ellingrod VL. The influence of metabolic syndrome, physical activity and genotype on catechol-O-methyl transferase promoter-region methylation in schizophrenia. Pharmacogenomics J. 2013;13(3):264–71.

Kashikar-Zuck S, Myer G, Ting TV. Can behavioral treatments be enhanced by integrative neuromuscular training in the treatment of juvenile fibromyalgia? Pain Manag. 2012;2(1):9–12.

Tobin S, Bisson N, Grondin S. An ecological approach to prospective and retrospective timing of long durations: a study involving gamers. PLoS One. 2010;5(2):e9271.

Article   ADS   PubMed   PubMed Central   Google Scholar  

Taylor RW, Farmer VL, Cameron SL, Meredith-Jones K, Williams SM, Mann JI. School playgrounds and physical activity policies as predictors of school and home time activity. Int J Behav Nutr Phys Act. 2011;8:38.

Li L, Ouyang F, He J, Qiu D, Luo D, Xiao S. Associations of socioeconomic status and healthy lifestyle with incidence of dyslipidemia: a prospective Chinese governmental employee cohort study. Front Public Health. 2022;10:878126.

Gil-Arias A, Diloy-Pena S, Sevil-Serrano J, Garcia-Gonzalez L, Abos A. A hybrid TGfU/SE volleyball teaching unit for enhancing motivation in physical education: a mixed-method approach. Int J Environ Res Public Health. 2021;18(1).

Zhang ZT, Du XM, Ma XJ, Zong Y, Chen JK, Yu CL, et al. Activation of the NLRP3 inflammasome in lipopolysaccharide-induced mouse fatigue and its relevance to chronic fatigue syndrome. J Neuroinflammation. 2016;13(1):71.

Yang M, Tao L, Zhao CC, Wang ZL, Yu ZJ, Zhou W, et al. Antifatigue effect of Panax Notoginseng leaves fermented with microorganisms: in-vitro and in-vivo evaluation. Front Nutr. 2022;9:824525.

Ingrassia PL, Mormando G, Giudici E, Strada F, Carfagna F, Lamberti F, et al. Augmented reality learning environment for basic life support and defibrillation training: usability study. J Med Internet Res. 2020;22(5):e14910.

Taylor AK, Loades M, Brigden AL, Collin SM, Crawley E. It’s personal to me’: a qualitative study of depression in young people with CFS/ME. Clin Child Psychol Psychiatry. 2017;22(2):326–40.

Mascarenhas LPG, de Lima VA, Rebesco DB, Franca SN, Cordeiro GR, Mota J, et al. Acute changes in glucose induced by continuous or intermittent exercise in children and adolescents with type 1 diabetes. Arch Endocrinol Metab. 2022;66(2):176–81.

Liao Y, Basen-Engquist KM, Urbauer DL, Bevers TB, Hawk E, Schembre SMJCE, Biomarkers, et al. Using continuous glucose monitoring to motivate physical activity in overweight and obese adults: a pilot study. Cancer Epidemiol Biomarkers Prev. 2020;29(4):761–8.

Mullan B, Olivier C, Thogersen-Ntoumani C. Mind the gap: habit and self-determined motivation predict health behaviours in middle-aged and older adults. Br J Health Psychol. 2021;26(4):1095–113.

Chae H, Park J. The effects of Routinization on radical and incremental creativity: the mediating role of mental workloads. Int J Environ Res Public Health. 2023;20(4):3160.

Simons M, de Vet E, Chinapaw MJ, de Boer M, Seidell JC, Brug J. Personal, social, and game-related correlates of active and non-active gaming among dutch gaming adolescents: survey-based multivariable, multilevel logistic regression analyses. JMIR Serious Games. 2014;2(1):e4.

Zeng N, Gao X, Liu Y, Lee JE, Gao Z. Reliability of using motion sensors to measure Children’s physical activity levels in exergaming. J Clin Med. 2018;7(5):100.

Download references

Acknowledgements

We would like to express our deep appreciation to Professor Jamalsafri Bin Saibon from the School of Educational Studies, Universiti Sains Malaysia (USM), for providing advices and support for this project.

This study was not funded by any organization or institution.

Author information

Authors and affiliations.

Malaysia Postgraduate Department, School of Educational Studies, Universiti Sains Malaysia, 11800, Penang, Malaysia

Weilong Mo, Jamalsafri Bin Saibon & Yaming LI

Zhaoqing College, Zhaoqing, 526061, China

Dinghu First Experimental School, Zhaoqing, 526070, China

Huaqiao Junior High School, Zhaoqing, 526108, China

You can also search for this author in PubMed   Google Scholar

Contributions

MWL performed the experiment, LYM and LJQ performed the data analysis, HYW performs a final check of the data, MWL performed the formal analysis and wrote the manuscript, JBS helped perform the analysis with constructive discussions, All authors were involved in discussing the results.

Corresponding author

Correspondence to Jamalsafri Bin Saibon .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mo, W., Saibon, J.B., LI, Y. et al. Effects of game-based physical education program on enjoyment in children and adolescents: a systematic review and meta-analysis. BMC Public Health 24 , 517 (2024). https://doi.org/10.1186/s12889-024-18043-6

Download citation

Received : 07 November 2023

Accepted : 08 February 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s12889-024-18043-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Physical education
  • Children and adolescents
  • Meta-analysis

BMC Public Health

ISSN: 1471-2458

systematic review health care databases

  • Systematic review update
  • Open access
  • Published: 16 February 2024

The global economic burden of COVID-19 disease: a comprehensive systematic review and meta-analysis

  • Ahmad Faramarzi   ORCID: orcid.org/0000-0001-5661-8991 1 ,
  • Soheila Norouzi   ORCID: orcid.org/0000-0002-3028-7861 1 ,
  • Hossein Dehdarirad 2 ,
  • Siamak Aghlmand 1 ,
  • Hasan Yusefzadeh 1 &
  • Javad Javan-Noughabi 3  

Systematic Reviews volume  13 , Article number:  68 ( 2024 ) Cite this article

337 Accesses

Metrics details

The COVID-19 pandemic has caused a considerable threat to the economics of patients, health systems, and society.

This meta-analysis aims to quantitatively assess the global economic burden of COVID-19.

A comprehensive search was performed in the PubMed, Scopus, and Web of Science databases to identify studies examining the economic impact of COVID-19. The selected studies were classified into two categories based on the cost-of-illness (COI) study approach: top-down and bottom-up studies. The results of top-down COI studies were presented by calculating the average costs as a percentage of gross domestic product (GDP) and health expenditures. Conversely, the findings of bottom-up studies were analyzed through meta-analysis using the standardized mean difference.

The implemented search strategy yielded 3271 records, of which 27 studies met the inclusion criteria, consisting of 7 top-down and 20 bottom-up studies. The included studies were conducted in various countries, including the USA (5), China (5), Spain (2), Brazil (2), South Korea (2), India (2), and one study each in Italy, South Africa, the Philippines, Greece, Iran, Kenya, Nigeria, and the Kingdom of Saudi Arabia. The results of the top-down studies indicated that indirect costs represent 10.53% of GDP, while the total estimated cost accounts for 85.91% of healthcare expenditures and 9.13% of GDP. In contrast, the bottom-up studies revealed that the average direct medical costs ranged from US $1264 to US $79,315. The meta-analysis demonstrated that the medical costs for COVID-19 patients in the intensive care unit (ICU) were approximately twice as high as those for patients in general wards, with a range from 0.05 to 3.48 times higher.

Conclusions

Our study indicates that the COVID-19 pandemic has imposed a significant economic burden worldwide, with varying degrees of impact across countries. The findings of our study, along with those of other research, underscore the vital role of economic consequences in the post-COVID-19 era for communities and families. Therefore, policymakers and health administrators should prioritize economic programs and accord them heightened attention.

Peer Review reports

Coronavirus disease 2019 (COVID-19) is a respiratory infection instigated by the severe acute respiratory syndrome coronavirus 2 (SARS-COV-2), first identified in Wuhan, China, in December 2019. The disease has since proliferated globally at an alarming rate, prompting the World Health Organization (WHO) to declare a pandemic on March 11, 2020 [ 1 ]. As of February 21, 2023, the global total of confirmed COVID-19 cases stands at 757,264,511, with a death toll of 6,850,594 [ 2 ].

Patients afflicted with COVID-19 exhibit a range of symptoms, including flu-like manifestations, acute respiratory failure, thromboembolic diseases, and organ dysfunction or failure [ 3 ]. Moreover, these patients have had to adapt to significant changes in their environment, such as relocating for quarantine purposes, remote work or job loss, and air-conditioning [ 4 , 5 ].

The COVID-19 pandemic has imposed substantial direct and indirect costs on patients, families, healthcare systems, and communities. These costs fluctuate significantly based on socioeconomic factors, age, disease severity, and comorbidities [ 6 , 7 ]. For instance, a study conducted in the United States of America (USA) estimated the median direct medical cost of a single symptomatic COVID-19 case to be US $3045 during the infection period alone [ 8 ]. Additionally, indirect costs arising from the pandemic, such as lost productivity due to morbidity and mortality, reduced consumer spending, and supply chain disruptions, could be substantial in certain countries [ 9 ]. Studies by Maltezou et al. and Faramarzi et al. revealed that absenteeism costs accounted for a large proportion (80.4%) of total costs [ 10 ] and estimated an average cost of US $671.4 per patient [ 11 ], respectively. Furthermore, the macroeconomic impact of the COVID-19 pandemic is considerably more significant. Data from Europe indicates that the gross domestic product (GDP) fell by an average of 7.4% in 2020 [ 12 ]. Globally, the economic burden of COVID-19 was estimated to be between US $77 billion and US $2.7 trillion in 2019 [ 13 ]. Another study calculated the quarantine costs of COVID-19 to exceed 9% of the global GDP [ 14 ].

Evaluating the cost of COVID-19, encompassing both direct (medical and non-medical) and indirect costs, provides valuable insights for policymakers and healthcare managers to devise effective strategies for resource allocation and cost control, particularly in the post-COVID-19 era. Despite the abundance of literature on COVID-19, only a handful of studies have concentrated on its economic burden. Furthermore, the currency estimates provided in these articles is inconsistent. To address this gap, our study aimed to conduct a systematic review and meta-analysis of the global economic burden of COVID-19. The objectives of this study are twofold: firstly, to estimate the direct and indirect costs of COVID-19 as a percentage of GDP and health expenditure (HE) at the global level, and secondly, to estimate the direct medical costs based on the inpatient ward, which includes both the general ward and the intensive care unit (ICU).

This study was designed according to the Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA) guidelines [ 15 ].

Search strategy and data sources

We performed a comprehensive search in PubMed, Scopus, and Web of Science databases to retrieve studies on the economic burden of COVID-19 disease. To this objective, we conducted a comprehensive search by combining the search terms relating to COVID-19 (coronavirus, 2019-nCoV), as a class, with the terms relating to the economic burden and terms related to it (direct cost, indirect cost, productivity cost, morbidity cost, mortality cost, cost analysis, cost of illness, economic cost, noneconomic cost, financial cost, expenditure, spending). The search was limited to English language publications and human studies that were published before September 19, 2021. The search strategy was validated by a medical information specialist. All search strategies are available in the Additional file 1 .

Screening and selection

Two reviewers independently screened all distinct articles, focusing on the title and abstract and utilizing EndNote software. The reviewers were blinded to each other’s findings during the screening phase. Potential duplicates were identified and scrutinized to exclude identical entries. Any discrepancies between the reviewers were reconciled through consensus or by consulting a third reviewer. The final decision regarding inclusion was determined subsequent to a comprehensive review of the full-text article. The whole process of the study selection was outlined in a flow chart (Fig. 1 ).

figure 1

Flowchart depicting the selection of research studies

This systematic review included all original studies that addressed the economic burden of COVID-19, provided they (1) estimated all costs associated with COVID-19, including both direct (medical and non-medical) and indirect (morbidity and mortality) costs and (2) were designed as observational studies or controlled clinical trials. Studies were excluded based on the following criteria: (1) they were review articles, commentaries, editorials, protocols, case studies, case series, animal studies, book chapters, or theses, (2) they estimated costs for a specific disease or action during the COVID-19 pandemic, and (3) they were studies assessing budget impact or economic evaluations.

Data extraction

A specific data extraction template was developed to extract relevant information from every study that satisfied our eligibility criteria. The data extracted covered the general study characteristics (authors, study publication, geographical location of data collection), cost-related information (direct medical cost, direct nonmedical cost, indirect cost, total cost, years of costing, and currency), and participants-related data (sample size and population studied for estimation).

Outcome and quality assessment

The primary outcomes were documented as the standardized mean difference (SMD) accompanied by 95% confidence intervals, representing the direct medical costs borne in general wards as compared to ICU for patients diagnosed with COVID-19. Additionally, another outcome was the estimation of these costs as a proportion of the GDP and health expenditure (HE).

A quality assessment was conducted on all the included studies, utilizing the checklist formulated by Larg and Moss [ 16 ]. This checklist comprises three domains: analytic framework, methodology and data, and analysis and reporting. The quality assessment was independently corroborated by two reviewers. In case of any discrepancies in the quality assessment, resolution was ensured through consensus or consultation with a third reviewer.

Statistical analysis

To analyze the data, we utilized the cost-of-illness (COI) study approach, which involved categorizing the studies into two groups: top-down studies and bottom-up studies. Top-down studies were defined as population-based methods that estimated costs for a specific country or group of countries, while bottom-up studies were defined as person methods that estimated costs per person [ 16 ].

In our methodological approach to the top-down studies, we initially categorized the costs into direct and indirect types. The direct costs comprised both medical and nonmedical expenses, while the indirect costs were related to potential productivity losses stemming from mortality and morbidity. Subsequently, we undertook the adjustment of all costs to the 2020 US dollar value. This was achieved based on the principle of purchasing power parity (PPP), and we utilized the currency conversion factor as recommended by the World Bank for this purpose. We employed the method proposed by Konnopka and König to present the COVID-19 cost to top-down studies. This method, which expresses the costs as a proportion of the gross domestic product (GDP) and health expenditure (HE), eliminates the need for adjustments for inflation or differences in purchasing power [ 17 ]. Moreover, we computed the costs using both an unweighted mean and a population-weighted mean.

In the bottom-up studies, a random-effects model was employed for the meta-analysis, with the SMD serving as the measure of effect size. To mitigate the influence of heterogeneity, all costs were converted to 2020 US dollars based on PPP, utilizing the currency conversion factor suggested by the World Bank. The focus of our analysis was a comparison of the direct medical costs of patients admitted to the general ward versus those in ICU. The SMD was calculated as the measure of effect size, with the sample size acting as the weighting factor. Heterogeneity was assessed through Cochran’s Q test and the I 2 statistic. The Q -test, a classical measure with a chi-square distribution, is calculated as the weighted sum of squared differences between individual study effects and the pooled effects across studies. The I 2 statistic represents the percentage of variation across studies, with threshold values of 25%, 50%, and 75% indicating low, moderate, and high levels of heterogeneity, respectively. To assess possible publication or disclosure bias, we used funnel plots, the Begg-adjusted rank correlation test, and Egger’s test. All statistical analyses were performed using STATA version 14 (Stata Corp, College Station, TX, USA), and P -values less than 0.05 were considered as statistically significant.

The study selection process is illustrated in Figure 1 . The search strategy produced 3271 records (Scopus, 1450; PubMed, 1144; Web of Science, 677), from which 1358 duplicates were eliminated. Out of the remaining 1913 articles, a mere 101 satisfied the inclusion criteria and underwent a full-text review. During this full-text screening, 74 articles were excluded for various reasons, resulting in a final selection of 27 studies included in the systematic review. Among these, 20 were bottom-up studies [ 7 , 10 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ], and 7 were top-down studies [ 36 , 37 , 38 , 39 , 40 , 41 , 42 ].

Characteristics of included studies

Table 1 presents the general characteristics of the included studies. Out of the 27 studies, 5 were conducted in the USA; 5 in China; 2 each in Spain, Brazil, South Korea, and India; and 1 each in Italy, South Africa, the Philippines, Greece, Iran, Kenya, Nigeria, and the Kingdom of Saudi Arabia. Based on the methodology employed, 20 studies were categorized as bottom-up studies and seven as top-down studies.

Among the seven top-down studies, only three calculated direct medical costs [ 37 , 38 , 41 ], two studies examined the direct nonmedical costs [ 38 , 41 ], and all but Santos et al. [ 37 ], who did not report these costs, calculated indirect costs. Of the 20 bottom-up studies, all but 1 study [ 31 ] assessed the direct medical costs. Only four studies calculated the direct nonmedical costs [ 10 , 19 , 29 , 34 ], and seven studies reported the indirect costs [ 7 , 10 , 19 , 26 , 29 , 31 , 34 ].

Table 2 presents the specific characteristics of the top-down studies. These studies indicate that the direct costs of COVID-19 span from US $860 million to US $8,657 million, while indirect costs range from US $610 million to US $5,500,000 million. On average, top-down studies estimate the direct costs associated with COVID-19 to constitute 2.73% and 0.39% of healthcare expenditures, based on unweighted and weighted means, respectively. The results also reveal that, on average, indirect costs account for 10.53% of GDP, with a range of 0.02 to 30.90%. Furthermore, the total cost estimated by top-down studies comprises 85.91% of healthcare expenditure and 9.13% of GDP.

Table 3 outlines the specific characteristics of the bottom-up studies. Excluding two studies [ 23 , 27 ], all reported their sample sizes, which varied from 9 to 1,470,721. The mean estimate of direct medical costs ranged from US $1264 to US $79,315. Two studies reported values for direct nonmedical costs [ 19 , 29 ], with means of US $25 and US $71. The mean estimate of indirect costs ranged from US $187 to US $689,556.

Meta-analysis results

The results of the meta-analysis for the direct medical costs are shown in Figure 2 . The results indicate a significant association between the mean cost of direct medical services and the inpatient ward. Specifically, the analysis yielded a standardized mean difference (SMD) of 1.62 ( CI : 0.9–2.35) with a substantial degree of heterogeneity ( Q = 26170, p < 0.0001; I 2 = 100%).

figure 2

Mean direct medical cost for patient with COVID-19 based on disease severity

Assessment of publication bias

Figure 3 presents the information related to publication bias. The funnel plot, constructed from the studies included, does not suggest the presence of potential publication bias. Moreover, the application of Begg’s and Egger’s tests in the statistical analysis resulted in P-values of 0.788 and 0.789, respectively, indicating an absence of significant bias.

figure 3

The funnel plots, Begg’s test, and Egger’s test to assessment of publication bias for included studies that assessed the direct medical costs of patients hospitalized in the general ward versus those in the intensive care unit (ICU)

This investigation represents the initial systematic review and meta-analysis conducted on the topic of the global economic impact of COVID-19. Furthermore, it is the first study to evaluate economic burden research related to COVID-19 using both top-down and bottom-up approaches, and it has conducted a meta-analysis of medical direct expenses based on hospitalization wards. In general, studies examining the economic impact of COVID-19 are scarce, with a greater proportion of studies employing a bottom-up approach. More than 30% of these studies were conducted in the USA and China. Patients admitted to the ICU ward exhibited higher costs than those admitted to the general ward.

Admission to the ICU significantly escalated the medical expenditure associated with COVID-19 treatment. This study discovered that the medical costs for COVID-19 patients in the ICU were approximately twice as high as those for patients in general wards, with a range from 0.05 to 3.48 times higher. This finding aligns with existing literature, which suggests that ICU patients with COVID-19 are more likely to require expensive treatments such as mechanical ventilation and extracorporeal membrane oxygenation, compared to those in general wards [ 44 , 45 ]. Consistent with this, other studies have reported an increase in medical expenditures with the hospitalization of COVID-19 patients in the ICU. For instance, a study conducted in the USA found a fivefold increase in costs for patients in the ICU who required invasive mechanical ventilation (IMV), compared to those not in the ICU or without IMV [ 22 ]. Similarly, a study in China reported a 2.5-fold increase in costs for severe COVID-19 patients compared to mild cases [ 30 ]. Given the elevated medical costs associated with treating COVID-19 patients in the ICU or those with severe symptoms, health policymakers must concentrate on implementing programs that promote early diagnosis. Consequently, healthcare providers could initiate treatment at an earlier stage, potentially reducing the severity of the disease and associated costs.

Our research indicates that significant variations in estimated costs would be observed if these costs were reported in PPP, particularly in relation to direct medical expenses. The lowest value was calculated in India, amounting to US $1264, while the highest value was observed in the USA, reaching US $54,165. Furthermore, the calculated medical costs varied across countries. For example, in the USA, direct medical expenditures ranged from US $1701 to US $54,156 [ 21 , 35 ]. In contrast, in China, the reported costs fluctuated between US $5264 and US $79,315 [ 7 , 25 ]. Several factors contribute to this variation in the estimation of direct medical costs. Primarily, direct medical costs cover a spectrum of services, including diagnosis, medication, consumables, inpatient care, and consultation services. Consequently, each study may have estimated the direct medical costs for a subset or the entirety of these services, leading to differences in the estimated costs. For instance, Nguyen et al. demonstrated a nearly threefold increase in direct costs for COVID-19 patients managed with extracorporeal membrane oxygenation (ECMO) compared to patients not receiving ECMO [ 35 ]. This highlights the impact of specific treatments on the overall cost. Secondly, the sample size may vary between studies, resulting in different cost estimates. Larger sample sizes typically provide more accurate and reliable estimates, but they also require more resources to collect and analyze. Lastly, the studies may have estimated costs for patients with varying conditions, such as those in acute status, patients hospitalized in general wards, or those admitted to ICU wards.

In addition to direct medical expenditures, the indirect costs arising from productivity losses due to COVID-19 have substantial societal implications. This study discovered that direct medical expenses attributable to COVID-19 varied from US $860 million (representing 0.11% of China’s healthcare expenditure) as reported by Zhao et al. [ 38 ] in China to US $8657 million (equivalent to 7.4% of Spanish healthcare expenditure) as reported by Gonzalez Lopez et al. [ 41 ] in Spain. On a global scale, direct medical costs due to COVID-19 constituted 2.73% of healthcare expenditure and 0.25% of GDP. The results also unveiled that the indirect costs of the COVID-19 pandemic impacted different countries to varying extents. The minimum value of indirect costs was estimated in Italy [ 40 ] and India [ 39 ] at US $610 million and US $658 million, respectively. Interestingly, when reported as a percentage of GDP, India had a lower cost (0.02% of GDP) compared to China (0.03% of GDP). The maximum value of indirect costs was calculated in the USA at US $5,500,000 million, which accounted for approximately 26.32% of the USA’s GDP [ 36 ]. Despite the numerical value of indirect costs being lower in Spain than in the USA and China, it represented a higher percentage of GDP (30.90%). The resulting pooled estimate indicated that the indirect costs due to COVID-19 were responsible for 10.53% of global GDP. The review underscores the significant economic repercussions of COVID-19. The total costs in the USA accounted for about 157% of healthcare expenditure and 26% of GDP, in China for 80% of healthcare expenditure and 4.28% of GDP, and in Spain for approximately 345% of healthcare expenditure and 32% of GDP. Globally, the total costs of COVID-19 accounted for about 86% of healthcare expenditure and 9.13% of GDP. This highlights the profound economic impact of the pandemic on both healthcare systems and economies worldwide.

Strengths and limitation

Our study possesses several significant strengths. It is the inaugural meta-analysis of the worldwide costs associated with COVID-19, supplementing a systematic review conducted by Richards et al. on the economic burden studies of COVID-19 [ 12 ]. A considerable number of studies was conducted in the USA and China, but our analysis also incorporated studies from other high- and low-income countries, potentially enhancing the generalizability of our findings. Recognizing that economic burden studies often display significant heterogeneity, we endeavored to minimize this by distinguishing between bottom-up and top-down studies and standardizing currencies to US dollars in terms of PPP.

However, our study is not without limitations. As is typical with all meta-analyses of economic burden studies, the most substantial limitation is heterogeneity. This heterogeneity can originate from various factors, including differences in study design, the range of services included in individual studies, the year of estimation, the currencies used for estimation, the study population, among other factors. Our systematic review only incorporated studies that estimated costs for an actual population, thereby excluding a wide array of studies on the economic burden of COVID-19 that employed modeling techniques. Future research could potentially conduct systematic reviews and meta-analyses on cost estimation modeling studies for COVID-19. Lastly, while no publication bias was detected through statistical analysis, our study was limited to papers written in English. As a result, numerous papers published in other languages were inevitably excluded.

Our research indicates that the COVID-19 pandemic has imposed a substantial economic strain worldwide, with the degree of impact varying across nations. The quantity of studies examining the economic repercussions of COVID-19 is limited, with a majority employing a bottom-up methodology. The indirect costs ascribed to COVID-19 constituted 10.53% of the global GDP. In total, the costs linked to COVID-19 represented 9.13% of GDP and 86% of healthcare spending. Moreover, our meta-analysis disclosed that the direct medical expenses for COVID-19 patients in the ICU were almost twice those of patients in general wards. The results of our research, along with those of others, underscore the pivotal role of economic outcomes in the post-COVID-19 era for societies and families. Consequently, it is imperative for policymakers and health administrators to prioritize and pay greater attention to economic programs.

Availability of data and materials

Data sharing is not applicable as no new data were generated during the study. The data analysis file during this study is available from the corresponding author on reasonable request.

Lu H, Stratton CW, Tang YW. Outbreak of pneumonia of unknown etiology in Wuhan, China: the mystery and the miracle. J Med Virol. 2020;92(4):401–2.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Available from: https://www.who.int/emergencies/diseases/novel-coronavirus-2019 .

Gautier JF, Ravussin Y. A new symptom of COVID-19: loss of taste and smell. Obesity. 2020;28(5):848.

Article   CAS   PubMed   Google Scholar  

Delikhoon M, Guzman MI, Nabizadeh R, Norouzian BA. Modes of transmission of severe acute respiratory syndrome-coronavirus-2 (SARS-CoV-2) and factors influencing on the airborne transmission: a review. Int J Environ Res Public Health. 2021;18(2):395.

Alimohammadi M, Abolli S, Ghordouei ME. Perceiving effect of environmental factors on prevalence of SARS-Cov-2 virus and using health strategies: a review. J Adv Environ Health Res. 2022;10(3):187–96.

Article   Google Scholar  

Li XZ, Jin F, Zhang JG, Deng YF, Shu W, Qin JM, et al. Treatment of coronavirus disease 2019 in Shandong, China: a cost and affordability analysis. Infect Dis Poverty. 2020;9(1):78.

Jin H, Wang H, Li X, Zheng W, Ye S, Zhang S, et al. Economic burden of COVID-19, China, January–March, 2020: a cost-of-illness study. Bull World Health Organ. 2021;99(2):112.

Article   PubMed   Google Scholar  

Bartsch SM, Ferguson MC, McKinnell JA, O'Shea KJ, Wedlock PT, Siegmund SS, et al. The potential health care costs and resource use associated with COVID-19 in the United States. Health Aff (Millwood). 2020;39(6):927–35.

Juranek S, Paetzold J, Winner H, Zoutman F. Labor market effects of COVID-19 in Sweden and its neighbors: evidence from novel administrative data. NHH Dept of Business and Management Science Discussion Paper. (2020/8); 2020.

Google Scholar  

Maltezou H, Giannouchos T, Pavli A, Tsonou P, Dedoukou X, Tseroni M, et al. Costs associated with COVID-19 in healthcare personnel in Greece: a cost-of-illness analysis. J Hosp Infect. 2021;114:126–33.

Faramarzi A, Javan-Noughabi J, Tabatabaee SS, Najafpoor AA, Rezapour A. The lost productivity cost of absenteeism due to COVID-19 in health care workers in Iran: a case study in the hospitals of Mashhad University of Medical Sciences. BMC Health Serv Res. 2021;21(1):1169.

Article   PubMed   PubMed Central   Google Scholar  

Richards F, Kodjamanova P, Chen X, Li N, Atanasov P, Bennetts L, et al. Economic burden of COVID-19: a systematic review. Clinicoecon Outcomes Res. 2022;14:293–307.

Forsythe S, Cohen J, Neumann P, Bertozzi SM, Kinghorn A. The economic and public health imperatives around making potential coronavirus disease–2019 treatments available and affordable. Value Health. 2020;23(11):1427–31.

Rodela TT, Tasnim S, Mazumder H, Faizah F, Sultana A, Hossain MM. Economic impacts of coronavirus disease (COVID-19) in developing countries; 2020.

Book   Google Scholar  

Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group* t. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9.

Larg A, Moss JR. Cost-of-illness studies: a guide to critical evaluation. Pharmacoeconomics. 2011;29:653–71.

Konnopka A, König H. Economic burden of anxiety disorders: a systematic review and meta-analysis. Pharmacoeconomics. 2020;38:25–37.

Jeck J, Jakobs F, Kron A, Franz J, Cornely OA, Kron F. A cost of illness study of COVID-19 patients and retrospective modelling of potential cost savings when administering remdesivir during the pandemic “first wave” in a German tertiary care hospital. Infection. 2022;50(1):191–201.

Kotwani P, Patwardhan V, Pandya A, Saha S, Patel GM, Jaiswal S, et al. Valuing out-of-pocket expenditure and health related quality of life of COVID-19 patients from Gujarat, India. J Commun Dis (E-ISSN: 2581-351X & P-ISSN: 0019-5138). 2021;53(1):104–9.

Tsai Y, Vogt TM, Zhou F. Patient characteristics and costs associated with COVID-19–related medical care among Medicare fee-for-service beneficiaries. Ann Intern Med. 2021;174(8):1101–9.

Weiner JP, Bandeian S, Hatef E, Lans D, Liu A, Lemke KW. In-person and telehealth ambulatory contacts and costs in a large US insured cohort before and during the COVID-19 pandemic. JAMA Netw Open. 2021;4(3):e212618-e.

Di Fusco M, Shea KM, Lin J, Nguyen JL, Angulo FJ, Benigno M, et al. Health outcomes and economic burden of hospitalized COVID-19 patients in the United States. J Med Econ. 2021;24(1):308–17.

Edoka I, Fraser H, Jamieson L, Meyer-Rath G, Mdewa W. Inpatient care costs of COVID-19 in South Africa’s public healthcare system. Int J Health Policy Manag. 2022;11(8):1354.

PubMed   Google Scholar  

Tabuñar SM, Dominado TM. Hospitalization expenditure of COVID-19 patients at the university of the Philippines-Philippine general hospital (UP-PGH) with PhilHealth coverage. Acta Med Philipp. 2021;55(2)

Zhao J, Yao Y, Lai S, Zhou X. Clinical immunity and medical cost of COVID-19 patients under grey relational mathematical model. Results Phys. 2021;22:103829.

Ghaffari Darab M, Keshavarz K, Sadeghi E, Shahmohamadi J, Kavosi Z. The economic burden of coronavirus disease 2019 (COVID-19): evidence from Iran. BMC Health Serv Res. 2021;21(1):1–7.

Barasa E, Kairu A, Maritim M, Were V, Akech S, Mwangangi M. Examining unit costs for COVID-19 case management in Kenya. BMJ Glob Health. 2021;6(4):e004159.

Seon J-Y, Jeon W-H, Bae S-C, Eun B-L, Choung J-T, Oh I-H. Characteristics in pediatric patients with coronavirus disease 2019 in Korea. J Korean Med Sci. 2021;36(20):e148.

Banke-Thomas A, Makwe CC, Balogun M, Afolabi BB, Alex-Nwangwu TA, Ameh CA. Utilization cost of maternity services for childbirth among pregnant women with coronavirus disease 2019 in Nigeria’s epicenter. Int J Gynaecol Obstet. 2021;152(2):242–8.

Li X-Z, Jin F, Zhang J-G, Deng Y-F, Shu W, Qin J-M, et al. Treatment of coronavirus disease 2019 in Shandong, China: a cost and affordability analysis. Infect Dis Poverty. 2020;9(03):31–8.

CAS   Google Scholar  

Kirigia JM, Muthuri RNDK. The fiscal value of human lives lost from coronavirus disease (COVID-19) in China. BMC Res Notes. 2020;13:1–5.

Lee JK, Kwak BO, Choi JH, Choi EH, Kim J-H, Kim DH. Financial burden of hospitalization of children with coronavirus disease 2019 under the national health insurance service in Korea. J Korean Med Sci. 2020;35(24):e224.

Khan AA, AlRuthia Y, Balkhi B, Alghadeer SM, Temsah M-H, Althunayyan SM, et al. Survival and estimation of direct medical costs of hospitalized COVID-19 patients in the Kingdom of Saudi Arabia. Int J Environ Res Public Health. 2020;17(20):7458.

Mas Romero M, Avendaño Céspedes A, Tabernero Sahuquillo MT, Cortés Zamora EB, Gómez Ballesteros C, Sánchez-Flor Alfaro V, et al. COVID-19 outbreak in long-term care facilities from Spain. Many lessons to learn. PLoS One. 2020;15(10):e0241030.

Nguyen NT, Sullivan B, Sagebin F, Hohmann SF, Amin A, Nahmias J. Analysis of COVID-19 patients with acute respiratory distress syndrome managed with extracorporeal membrane oxygenation at US academic centers. Ann Surg. 2021;274(1):40.

Viscusi WK. Economic lessons for COVID-19 pandemic policies. South Econ J. 2021;87(4):1064–89.

Santos HLPC, Maciel FBM, Santos Junior GM, Martins PC, Prado NMBL. Public expenditure on hospitalizations for COVID-19 treatment in 2020, in Brazil. Rev Saude Publica. 2021;55:52.

Zhao J, Jin H, Li X, Jia J, Zhang C, Zhao H, et al. Disease burden attributable to the first wave of COVID-19 in China and the effect of timing on the cost-effectiveness of movement restriction policies. Value Health. 2021;24(5):615–24.

John D, Narassima M, Menon J, Rajesh JG, Banerjee A. Estimation of the economic burden of COVID-19 using disability-adjusted life years (DALYs) and productivity losses in Kerala, India: a model-based analysis. BMJ Open. 2021;11(8):e049619.

Nurchis MC, Pascucci D, Sapienza M, Villani L, D’Ambrosio F, Castrini F, et al. Impact of the burden of COVID-19 in Italy: results of disability-adjusted life years (DALYs) and productivity loss. Int J Environ Res Public Health. 2020;17(12):4233.

González López-Valcárcel B, Vallejo-Torres L. The costs of COVID-19 and the cost-effectiveness of testing. Appl Econ Anal. 2021;29(85):77–89.

Debone D, Da Costa MV, Miraglia SG. 90 days of COVID-19 social distancing and its impacts on air quality and health in Sao Paulo, Brazil. Sustainability. 2020;12(18):7440.

Article   CAS   Google Scholar  

Ghaffari Darab M, Keshavarz K, Sadeghi E, Shahmohamadi J, Kavosi Z. The economic burden of coronavirus disease 2019 (COVID-19): evidence from Iran. BMC Health Serv Res. 2021;21(1):132.

Oliveira TF, Rocha CAO, Santos AGG, Silva Junior LCF, Aquino SHS, Cunha EJO, et al. Extracorporeal membrane oxygenation in COVID-19 treatment: a systematic literature review. Braz J Cardiovasc Surg. 2021;36:388–96.

Petrilli CM, Jones SA, Yang J, Rajagopalan H, O’Donnell L, Chernyak Y, et al. Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study. BMJ. 2020;369:m1966.

Download references

There was no funding utilized in this study.

Author information

Authors and affiliations.

Department of Health Economics and Management, School of Public Health, Urmia University of Medical Sciences, Urmia, Iran

Ahmad Faramarzi, Soheila Norouzi, Siamak Aghlmand & Hasan Yusefzadeh

Department of Medical Library and Information Science, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran

Hossein Dehdarirad

Social Determinants of Health Research Center, Mashhad University of Medical Sciences, Mashhad, Iran

Javad Javan-Noughabi

You can also search for this author in PubMed   Google Scholar

Contributions

AF was principal investigators of designing the study, conducted analysis, and was major contributor in writing the manuscript. SN, HD, and SA conducted the literature search, conducted the screening and data extraction. HY and JJ contributed in designing the study and writing the manuscript. All authors contributed, reviewed, and approved this paper.

Corresponding author

Correspondence to Ahmad Faramarzi .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Ethics Committee of the Urmia University of Medical Sciences (IR.UMSU.REC.1400.121).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

Search strategies

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Faramarzi, A., Norouzi, S., Dehdarirad, H. et al. The global economic burden of COVID-19 disease: a comprehensive systematic review and meta-analysis. Syst Rev 13 , 68 (2024). https://doi.org/10.1186/s13643-024-02476-6

Download citation

Received : 13 August 2023

Accepted : 31 January 2024

Published : 16 February 2024

DOI : https://doi.org/10.1186/s13643-024-02476-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Cost of illness
  • Systematic review

Systematic Reviews

ISSN: 2046-4053

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

systematic review health care databases

Health Care Administration and Management

  • Find Articles/Databases
  • Reference Resources
  • Evidence Summaries & Clinical Guidelines
  • Health Data & Statistics
  • Grey Literature & Reports
  • Framing Research Questions
  • Selecting Databases
  • Crafting a Search
  • Narrowing / Filtering a Search
  • Expanding a Search
  • Cited Reference Searching
  • Saving Searches
  • Citing/Managing References
  • What are Literature Reviews?
  • Conducting & Reporting Systematic Reviews
  • Finding Systematic Reviews
  • Tutorials & Tools for Literature Reviews
  • Critical Appraisal Resources
  • Finding Full Text

Databases Containing Systematic Reviews

  • Cochrane Library (includes: CDSR, DARE, CCTR, HTA, NHSEED, CRG's) This link opens in a new window The Cochrane Library provides access to systematic reviews in addition to other sources of reliable information, from other systematic review abstracts, technology assessments, economic evaluations and individual clinical trials. Includes Cochrane Database of Systematic Reviews(CDSR), Database of Abstracts of Reviews of Effects (DARE), Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Methodology Register (CMR), NHS Economic Evaluation Database (NHSEED), and Health Technology Assessment Database (HTA).

Open Access

  • Joanna Briggs Institute EBP Database This link opens in a new window JBI EBP database gives comprehensive coverage of a wide range of medical, nursing, and health science specialties and includes a unique suite of information that’s been analyzed, appraised, and prepared by expert reviewers at JBI so you can integrate the world’s best evidence into your practice:
  • PROSPERO PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome. It aims to provide a comprehensive listing of systematic reviews, registered at inception, to help avoid duplication and reduce opportunity for reporting bias by enabling comparison completed review with what was planned in the protocol.

Finding Published Reviews:

In the subject-specific databases listed below, it is possible to limit a set of results to just systematic reviews and/or meta-analyses. Often this limit is found as an option in the facet for "methodology" or "article/publication type."

  • PsycNET (with PsycINFO) This link opens in a new window APA PsycNET is an integrated collection of databases from the American Psychological Association (APA), including the following: PsycINFO, PsycBOOKS, PsycTESTS, PsycTHERAPY, PsycARTICLES, and PsycEXTRA. PsycINFO, the most popular database in PsycNET, is a bibliographic index covering core literature in the psychological and behavioral sciences and their related disciplines. PsycBOOKS includes online versions of the APA Handbooks in Psychology Series, as well as other ebooks and online encyclopedias published by APA. PsycTESTS is a source of structured information about psychological tests and measures as well as a repository for the full text of select instruments. PsycTHERAPY provides access to streaming psychotherapy demonstration videos. Altogether, PsycNET provides information about and access to journal articles, books and ebooks, dissertations, conference presentations, tests and measures, videos, gray literature, and many other other publication types in the psychological, social, behavioral, and health sciences. More information less... To search only a specific database (e.g., PsycINFO) within the PsycNET interface, use the "Select Databases" features at the top of the PsycNET landing page and check (or uncheck) the database(s) that apply.
  • PsycINFO via Ovid This link opens in a new window PsycINFO via Ovid indexes core academic and professional literature in the psychological and behavioral sciences and their related disciplines. Coverage is from 1806 to the present. **Note that the database PsycINFO is also searchable on APA PsycNET platform.
  • Medline via PubMed This link opens in a new window PubMed comprises over 30 million citations for biomedical literature from MEDLINE, life science journals, and online books. PubMed citations and abstracts include the fields of biomedicine and health, covering portions of the life sciences, behavioral sciences, chemical sciences, and bioengineering. PubMed also provides access to additional relevant web sites and links to the other NCBI molecular biology resources.
  • Medline via Ovid This link opens in a new window The MEDLINE database is widely recognized as the premier source for bibliographic and abstract coverage of biomedical literature. MEDLINE encompasses information from Index Medicus, Index to Dental Literature, and International Nursing, as well as other sources of coverage in the areas of allied health, biological and physical sciences, humanities and information science as they relate to medicine and health care, communication disorders, population biology, and reproductive biology.
  • CINAHL Complete This link opens in a new window CINAHL indexes over 3,800 journals from the fields of nursing and allied health, with indexing for selected journals back to 1937. Topics covered are related to nursing, physical therapy, health education, social service/healthcare, occupational therapy, and related disciplines. CINAHL also provides access to healthcare books, nursing dissertations, selected conference proceedings, standards of professional practice, educational software and audiovisual materials in nursing. Dates of coverage: 1937 to present.
  • EMBASE: Excerpta Medica 1974 – present (updated daily) This link opens in a new window EMBASE is a biomedical and pharmaceutical database indexing over 3,500 international journals in the following fields: drug research, pharmacology, pharmaceutics, toxicology, clinical and experimental human medicine, health policy and management, public health, occupational health, environmental health, drug dependence and abuse, psychiatry, forensic medicine, and biomedical engineering/instrumentation. There is selective coverage for nursing, dentistry, veterinary medicine, psychology, and alternative medicine. Dates of coverage: 1974 to present (updated daily)
  • << Previous: Conducting & Reporting Systematic Reviews
  • Next: Tutorials & Tools for Literature Reviews >>
  • Last Updated: Feb 20, 2024 9:48 AM
  • URL: https://guides.nyu.edu/healthcaremanagement

Effects of internet-based digital health interventions on the physical activity and quality of life of colorectal cancer survivors: a systematic review and meta-analysis

  • Published: 20 February 2024
  • Volume 32 , article number  168 , ( 2024 )

Cite this article

  • Ching-Ching Su 1 , 6 ,
  • Su-Er Guo 1 , 2 , 3 , 4 &
  • Ya-Wen Kuo 1 , 5  

Explore all metrics

The recent trend of Internet-based digital health interventions has driven researchers to implement them to promote physical activity (PA) and improve patients’ health outcomes. This systematic review and meta-analysis aim to evaluate the effects of Internet-based digital health interventions on PA and quality of life (QoL) in colorectal cancer (CRC) survivors.

We searched for relevant studies investigating the effects of internet-based digital health interventions published until Dec. 2022 in electronic databases (PubMed, CINAHL, EMBASE, Cochrane Central Register of Controlled Trials, and CEPS) according to PRISMA guidelines. The Joanna Briggs Institute critical appraisal checklist was used to examine the quality of the included studies. We performed the fixed and random effects model for meta-analysis.

Among 746 identified studies, eight published between 2018 and 2022 were included. These covered 991 internet-based digital health interventions and 875 controls. After 6 months of internet-based digital health interventions, CRC survivors’ performance in PA (standardized mean difference (SMD) = 0.23, 95% confidence interval [CI] = 0.09—0.38) and QoL (SMD = 0.11, 95% CI = 0.01—0.22) indicators improved significantly.

Conclusions

Internet-based digital health improved the PA behaviour and QoL of patients with CRC. Because of differences in intervention outcomes, additional randomized controlled trials are warranted to provide suggestions for clinical practice. Internet-based digital health interventions are promising for promoting PA in CRC survivors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

systematic review health care databases

Data availability

Not applicable.

Code availability

Xi Y, Xu P (2021) Global colorectal cancer burden in 2020 and projections to 2040. Transl Oncol 14(10):101174. https://doi.org/10.1016/j.tranon.2021.101174

Article   PubMed   PubMed Central   Google Scholar  

Miller KD, Nogueira L, Mariotto AB, Rowland JH, Yabroff KR, Alfano CM et al (2019) (2019) Cancer treatment and survivorship statistics. CA Cancer J Clin 69(5):363–385. https://doi.org/10.3322/caac.21565

Article   PubMed   Google Scholar  

Sánchez-Jiménez A, Cantarero-Villanueva I, Delgado-García G, Molina-Barea R, Fernández-Lao C, Galiano-Castillo N et al (2015) Physical impairments and quality of life of colorectal cancer survivors: a case–control study. Eur J Cancer Care 24(5):642–649. https://doi.org/10.1111/ecc.12218

Article   Google Scholar  

Denlinger CS, Engstrom PF (2011) Colorectal cancer survivorship: movement matters. Cancer Prev Res 4(4):502–511. https://doi.org/10.1158/1940-6207

Gao R, Yu T, Liu L, Bi J, Zhao H, Tao Y et al (2020) Exercise intervention for post-treatment colorectal cancer survivors: a systematic review and meta-analysis. J Cancer Surviv 14(6):878–893. https://doi.org/10.1007/s11764-020-00900-z

Packel LB, Prehn AW, Anderson CL, Fisher PL (2015) Factors influencing physical activity behaviors in colorectal cancer survivors. Am J Health Promot 30(2):85–92. https://doi.org/10.4278/ajhp.140103-QUAN-7

Luo X, Gao L, Li J, Lin Y, Zhao J, Li Q (2020) A critical literature review of dyadic web-based interventions to support cancer patients and their caregivers, and directions for future research. Psychooncology 29(1):38–48. https://doi.org/10.1002/pon.5278

Gimenez L, Druel V, Roques S, Vasseur J, Grosclaude P, Delpierre C et al (2020) Inventory of tools for care coordination between general practice and hospital system for patients suffering from cancer in active phase of treatment: a scoping review. Eur J Cancer Care 29(6):e13319. https://doi.org/10.1007/s00520-017-3882-6

Goode AD, Lawler SP, Brakenridge CL, Reeves MM, Eakin EG (2015) Telephone, print, and Web-based interventions for physical activity, diet, and weight control among cancer survivors: a systematic review. J Cancer Surviv 9(4):660–682. https://doi.org/10.1007/s11764-015-0442-2

Haberlin C, O’Dwyer T, Mockler D, Moran J, O’Donnell DM, Broderick J (2018) The use of eHealth to promote physical activity in cancer survivors: a systematic review. Support Care Cancer 26(10):3323–3336. https://doi.org/10.1007/s00520-018-4305-z

Kanera IM, Willems RA, Bolman CA, Mesters I, Verboon P, Lechner L (2017) Long-term effects of a web-based cancer aftercare intervention on moderate physical activity and vegetable consumption among early cancer survivors: a randomized controlled trial. Int J Behav Nutr Phys Act 14(1):1–13. https://doi.org/10.1186/s12966-017-0474-2

Siegel RL, Miller KD, Goding Sauer A, Fedewa SA, Butterly LF, Anderson JC et al (2020) (2020) Colorectal cancer statistics. CA Cancer J Clin 70(3):145–164. https://doi.org/10.3322/caac.21601

Ayyoubzadeh SM, NiakanKalhori SR, Shirkhoda M, Mohammadzadeh N, Esmaeili M (2020) Supporting colorectal cancer survivors using eHealth: a systematic review and framework suggestion. Support Care Cancer 28(8):3543–3555. https://doi.org/10.1007/s00520-020-05372-6

Knobloch K, Yoon U, Vogt PM (2011) Preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement and publication bias. J Craniomaxillofac Surg 39(2):91–92. https://doi.org/10.1016/j.jcms.2010.11.001

Deeks J, Bossuyt P, Leeflang M, Leeflang Y, Flemyng E (2023) Cochrane handbook for systematic reviews of diagnostic test accuracy. https://training.cochrane.org/handbook-diagnostic-test-accuracy . Accessed 21 Feb 2023

JBI (2023) Critical appraisal tools. 2021. https://jbi.global/critical-appraisal-tools . Accessed January 14 2023

Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC (2023) Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.3 (updated February 2022). Cochrane. 2022. http://www.training.cochrane.org/handbook . Accessed 15 Jan 2023

Cheong IY, An SY, Cha WC, Rha MY, Kim ST, Chang DK et al (2018) Efficacy of mobile health care application and wearable device in improvement of physical performance in colorectal cancer patients undergoing chemotherapy. Clin Colorectal Cancer 17(2):e353–e362. https://doi.org/10.1016/j.clcc.2018.02.002

Mayer DK, Landucci G, Awoyinka L, Atwood AK, Carmack CL, Demark-Wahnefried W et al (2018) SurvivorCHESS to increase physical activity in colon cancer survivors: can we get them moving? J Cancer Surviv 12(1):82–94. https://doi.org/10.1007/s11764-017-0647-7

Van Blarigan EL, Dhruva A, Atreya CE, Kenfield SA, Chan JM, Milloy A et al (2022) Feasibility and acceptability of a physical activity tracker and text messages to promote physical activity during chemotherapy for colorectal cancer: pilot randomized controlled trial (Smart Pace II). JMIR Cancer 8(1):e31576

Yun YH, Lim CI, Lee ES, Kim YT, Shin KH, Kim YW et al (2020) Efficacy of health coaching and a web-based program on physical activity, weight, and distress management among cancer survivors: a multi-centered randomised controlled trial. Psychooncology 29(7):1105–1114. https://doi.org/10.1002/pon.5394

Golsteijn RHJ, Bolman C, Volders E, Peels DA, de Vries H, Lechner L (2018) Short-term efficacy of a computer-tailored physical activity intervention for prostate and colorectal cancer patients and survivors: a randomized controlled trial. Int J Behav Nutr Phys Act 15(1):1–14. https://doi.org/10.1186/s12966-018-0734-9

Rastogi S, Tevaarwerk AJ, Sesto M, Van Remortel B, Date P, Gangnon R et al (2020) Effect of a technology-supported physical activity intervention on health-related quality of life, sleep, and processes of behavior change in cancer survivors: a randomized controlled trial. Psychooncology 29(11):1917–1926. https://doi.org/10.1002/pon.5524

Van Blarigan EL, Chan H, Van Loon K, Kenfield SA, Chan JM, Mitchell E et al (2019) Self-monitoring and reminder text messages to increase physical activity in colorectal cancer survivors (Smart Pace): a pilot randomized controlled trial. BMC Cancer 19(1):1–9. https://doi.org/10.1186/s12885-019-5427-5

Van Der Hout A, van Uden-Kraan CF, Holtmaat K, Jansen F, Lissenberg-Witte BI, Nieuwenhuijzen GA (2020) Role of eHealth application Oncokompas in supporting self-management of symptoms and health-related quality of life in cancer survivors: a randomised, controlled trial. Lancet Oncol 21(1):80–94. https://doi.org/10.1016/S1470-2045(19)30675-8

Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ et al (2010) Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 14(8):1–220. https://doi.org/10.3310/hta14080

Nair AS (2019) Publication bias—importance of studies with negative results. Indian J Anaesth 63(6):505–507. https://doi.org/10.4103/ija.IJA_142_19

Cohen SR, Mount BM, Tomas JJN, Mount LF (1996) Existential well-being is an important determinant of quality of life: evidence from the McGill quality of life questionnaire. Cancer 77:576–586. https://doi.org/10.1002/(SICI)1097-0142(19960201)77:3%3c576::AID-CNCR22%3e3.0.CO;2-0

Cohen SR, Mount BM, Bruera E, Provost M, Rowe J, Tong K (1997) Validity of the McGill Quality of Life Questionnaire in the palliative care setting: a multi-centre Canadian study demonstrating the importance of the existential domain. Palliat Med 11(1):3–20

Pinar R (2005) Reliability and construct validity of the SF-36 in Turkish cancer patients. Qual Life Res 14(1):259–264. https://doi.org/10.1007/s11136-004-2393-3

Sprangers MAG, Bonnetain F (2014) EORTC QLQ-C30. In: Michalos AC (ed) Encyclopedia of quality of life and well-being research. Springer, Dordrecht, pp 1933–1935

Chapter   Google Scholar  

Yoo HJ, Kim JC, Eremenco S, Han OS (2005) Quality of life in colorectal cancer patients with colectomy and the validation of the Functional Assessment of Cancer Therapy-Colorectal (FACT-C), Version 4. J Pain Symptom Manage 30(1):24–32. https://doi.org/10.1016/j.jpainsymman.2004.12.009

Lu Y, Qu HQ, Chen FY, Li XT, Cai L, Chen S et al (2019) Effect of Baduanjin Qigong exercise on cancer-related fatigue in patients with colorectal cancer undergoing chemotherapy: a randomized controlled trial. Oncol Res Treat 42(9):431–439

McGettigan M, Cardwell CR, Cantwell MM, Tully MA (2020) Physical activity interventions for disease-related physical and mental health during and following treatment in people with non-advanced colorectal cancer. Cochrane Database Syst Rev 5(5):CD012864. https://doi.org/10.1002/14651858.CD012864.pub2

Osterlund P, Salminen T, Soveri LM, Kallio R, Kellokumpu I, Lamminm A et al (2021) Repeated centralized multidisciplinary team assessment of resectability, clinical behavior, and outcomes in 1086 Finnish metastatic colorectal cancer patients (RAXO): a nationwide prospective intervention study. Lancet Reg Health Eur 3:100049. https://doi.org/10.1016/j.lanepe.2021.100049

Kocarnik JM et al (2022) Cancer incidence, mortality, years of life lost, years lived with disability, and disability-adjusted life years for 29 cancer groups from 2010 to 2019: a systematic analysis for the global burden of disease study 2019. JAMA Oncol 8(3):420–444

Nakano J, Hashizume K, Fukushima T, Ueno K, Matsuura E, Ikio Y et al (2018) Effects of aerobic and resistance exercises on physical symptoms in cancer patients: a meta-analysis. Integr Cancer Ther 17(4):1048–1058

Hu W, Chu J, Chen X, Liu S, Sun N, Han Q et al (2022) The role of depression and physical activity in the association of between sleep quality, and duration with and health-related quality of life among the elderly: a UK Biobank cross-sectional study. BMC Geriatr 22(1):1–13

Cohen J (2013) Statistical power analysis for the behavioral sciences. Routledge, Oxfordshire

Book   Google Scholar  

Purcell SA, Elliott SA, Walter PJ, Preston T, Cai H, Skipworth RJE et al (2019) Total energy expenditure in patients with colorectal cancer: associations with body composition, physical activity, and energy recommendations. Am J Clin Nutr 110(2):367–376. https://doi.org/10.1093/ajcn/nqz112

Ho M, Ho JWC, Fong DYT, Lee CF, Macfarlane DJ, Cerin E et al (2020) Effects of dietary and physical activity interventions on generic and cancer-specific health-related quality of life, anxiety, and depression in colorectal cancer survivors: a randomized controlled trial. J Cancer Surviv 14(4):424–433. https://doi.org/10.1007/s11764-020-00864-0

Eyl RE, Xie K, Koch-Gallenkamp L, Brenner H, Arndt V (2018) Quality of life and physical activity in long-term (≥5 years post-diagnosis) colorectal cancer survivors—systematic review. Health Qual Life Outcomes 16(1):112. https://doi.org/10.1186/s12955-018-0934-7

Speed-Andrews AE, Courneya KS (2009) Effects of exercise on quality of life and prognosis in cancer survivors. Curr Sports Med Rep 8(4):176–181. https://doi.org/10.1249/JSR.0b013e3181ae98f3

Stacey FG, James EL, Chapman K, Courneya KS, Lubans DR (2015) A systematic review and meta-analysis of social cognitive theory-based physical activity and/or nutrition behavior change interventions for cancer survivors. J Cancer Surviv 9(2):305–338. https://doi.org/10.1007/s11764-014-0413-z

Phillips SM, McAuley E (2013) Social cognitive influences on physical activity participation in long-term breast cancer survivors. Psychooncology 22(4):783–791. https://doi.org/10.1002/pon.3074

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M (2008) Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337.  https://doi.org/10.1136/bmj.a1655

Furness K, Sarkies MN, Huggins CE, Croagh D, Haines TP (2020) Impact of the method of delivering electronic health behavior change interventions in survivors of cancer on engagement, health behaviors, and health outcomes: systematic review and meta-analysis. J Med Internet Res 22(6):e16112. https://doi.org/10.2196/16112

Dahlke DV, Fair K, Hong YA, Beaudoin CE, Pulczinski J, Ory MG (2015) Apps seeking theories: results of a study on the use of health behavior change theories in cancer survivorship mobile apps. JMIR mHealth uHealth 3(1):e3861. https://doi.org/10.2196/mhealth.3861

Marcolino MS, Oliveira JAQ, D’Agostino M, Ribeiro AL, Alkmim MBM, Novillo-Ortiz D (2018) The impact of mHealth interventions: systematic review of systematic reviews. JMIR mHealth uHealth 6(1):e23. https://doi.org/10.2196/mhealth.8873

Download references

This study was funded by Cheng Gung University of Sciences and Technology and the Ministry of Science and Technology, Taiwan (grant number: Ministry of Science and Technology 109–2410-H-255–003-MY2).

Author information

Authors and affiliations.

Department of Nursing and Graduate Institute of Nursing, College of Nursing, Chang Gung University of Science and Technology (CGUST), No. 2, Sec. W., Jiapu Rd., Puzi City, Chiayi County, 61363, Taiwan

Ching-Ching Su, Su-Er Guo & Ya-Wen Kuo

Chronic Diseases and Health Promotion Research Center, Chang Gung University of Science and Technology (CGUST), No. 2, Sec. W., Jiapu Rd., Puzi City, Chiayi County, 61363, Taiwan

Division of Pulmonary and Critical Care Medicine, Chang Gung Memorial Hospital, No. 6, West Sec., Jiapu Road, Puzi City, Chiayi County, 61363, Taiwan

Department of Safety Health and Environmental Engineering, Ming Chi University of Technology, No. 84 Gungjuan Rd., Taishan Dist., New Taipei City, 24301, Taiwan

Department of Neurology, Chang Gung Memorial Hospital, No. 6, West Sec., Jiapu Road, Puzi City, Chiayi County, 61363, Taiwan

Department of Hematology and Oncology, Chiayi Chang Gung Memorial Hospital, No. 6, West Sec., Jiapu Road, Puzi City, Chiayi County, 61363, Taiwan

Ching-Ching Su

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the study conception and design. C. S., S. G and Y. K. designed the study, collected data, interpreted data, and wrote the manuscript. Y. K. performed statistical analysis; C. S. and Y. K. interpreted data and critically reviewed the manuscript. S.G. contributed to subsequent manuscript discussion. C. S., S. G., and Y. K. wrote the first draft and final manuscript. All the authors have read and approved the final manuscript.

Corresponding authors

Correspondence to Su-Er Guo or Ya-Wen Kuo .

Ethics declarations

Ethical approval, consent of participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 26 KB)

Rights and permissions.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Su, CC., Guo, SE. & Kuo, YW. Effects of internet-based digital health interventions on the physical activity and quality of life of colorectal cancer survivors: a systematic review and meta-analysis. Support Care Cancer 32 , 168 (2024). https://doi.org/10.1007/s00520-024-08369-7

Download citation

Received : 07 July 2023

Accepted : 11 February 2024

Published : 20 February 2024

DOI : https://doi.org/10.1007/s00520-024-08369-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Physical activity
  • Quality of life
  • Colorectal cancer survivors
  • Internet-based digital health
  • Meta-analysis
  • Systematic review
  • Find a journal
  • Publish with us
  • Track your research

IMAGES

  1. Bibliographic databases

    systematic review health care databases

  2. (PDF) Systematic Review of Privacy-Preserving Distributed Machine

    systematic review health care databases

  3. (PDF) Rate of discontinuation and drug survival of biologic therapies

    systematic review health care databases

  4. (PDF) Systematic Reviews and Meta-analysis: Understanding the Best

    systematic review health care databases

  5. 3 Systematic Reviews and Meta-Analyses

    systematic review health care databases

  6. Systematic Reviews in Health Care

    systematic review health care databases

VIDEO

  1. Health Information Module

  2. Introduction to Systematic Review of Research

  3. Center for Health Discovery and Well Being Case Study

  4. Clinical research

  5. The Future of Clinical Care: Advanced Databases in Healthcare

  6. The effect of ealry and systematic integration of palliative care in oncoogy on Quality of life

COMMENTS

  1. Cochrane Database of Systematic Reviews

    The Cochrane Database of Systematic Reviews ( CDSR) is the leading database for systematic reviews in health care. The CDSR includes Cochrane Reviews (systematic reviews) and protocols for Cochrane Reviews as well as editorials and supplements.

  2. Systematic Reviews: Medical Literature Databases to search

    At a minimum you need to search MEDLINE, EMBASE, and the Cochrane CENTRAL trials register.This is the recommendation of three medical and public health research organizations: the U.S. Agency for Healthcare Research and Quality (AHRQ), the U.K. Centre for Reviews and Dissemination (CRD), and the International Cochrane Collaboration (Source: Institute of Medicine (2011) Finding What Works in ...

  3. Cochrane Reviews

    The Cochrane Library is a collection of high-quality, independent evidence to inform healthcare decision-making, including the Cochrane Database of Systematic Reviews and the CENTRAL register of controlled trials.

  4. Systematically Reviewing the Literature: Building the Evidence for

    Introduction Systematic reviews that summarize the available information on a topic are an important part of evidence-based health care. There are both research and non-research reasons for undertaking a literature review.

  5. Database combinations to retrieve systematic reviews in overviews of

    MEDLINE and Epistemonikos, complemented by reference checking of included studies, is the best database combination to identify systematic reviews on health-related topics. Keywords: Review methods, Overview of reviews, Umbrella review, Search strategy, Databases, Systematic reviews Go to: Highlights What is already known

  6. Finding Systematic Reviews

    PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome.

  7. Epistemonikos: Database of the best Evidence-Based Health Care

    One-stop shop All of the evidence relevant for health decision-making, including world's largest systematic review database, curated and annotated by our network of collaborators. Interconnected evidence Articles are connected, so you can easily move from any article to all the evidence answering the same question.

  8. Databases

    The primary resource, the Cochrane Database of Systematic Reviews, includes more than 4,650 reviews and 2,015 protocols which have been prepared by a highly credentialed group to determine if an intervention for a specific disease or other problem is effective.

  9. Literature Search: Databases and Gray Literature

    A systematic review search includes a search of databases, gray literature, personal communications, and a handsearch of high impact journals in the related field. See our list of recommended databases and gray literature sources on this page.

  10. Selected Databases

    Cochrane Library is the leading journal and database for finding systematic reviews in health care. It is a powerful evidence-based tool to enhance your healthcare knowledge and decision making. Cochrane reviews are systematic reviews undertaken by members of the Cochrane Collaboration, an international not-for-profit organization.

  11. Systematic Review (SR) Service: Databases, Resources & Tools

    A collaborative, multilingual database of health evidence containing systematic reviews relevant for health-decision making, and a large source of other types of scientific evidence. ... (CRD) guidance for undertaking systematic reviews in health care. CRD is part of the National Institute for Health Research (NIHR) and is a department of the ...

  12. Systematic Reviews and Meta-analysis: Understanding the Best Evidence

    Cochrane Primary Health Care Field is a systematic review of primary healthcare research on prevention, treatment, rehabilitation, and diagnostic test accuracy. ... This can be done by going through scientific resources such as electronic database, controlled clinical trials registers, other biomedical databases, non-English literatures ...

  13. Databases

    These journal article databases are frequently used for health sciences systematic reviews. If necessary, you can also view the Database A-Z List below to find additional databases relevant for your topic. Databases include different journals.

  14. Systematic Reviews and Evidence Syntheses : Databases

    You will want to search at least three databases for your systematic review. Three databases alone does not complete the search standards for systematic review requirements. You will also have to complete a search of the grey literature and complete additional hand searches.

  15. Systematic Literature Reviews: Database Comparison Charts

    The Cochrane Database of Systematic Reviews (CDSR) is the leading resource for systematic reviews in health care. Cochrane Clinical Answers (CCAs) provide a readable, digestible, clinically-focused entry point to rigorous research from Cochrane Reviews. They are designed to be actionable and to inform point-of-care decision-making.

  16. 'It depends': what 86 systematic reviews tell us about what strategies

    Jamal A, McKenzie K, Clark M. The impact of health information technology on the quality of medical and health care: a systematic review. Health Inf Manag. 2009;38:26-37. ... Flodgren G, O'Brien MA, Parmelli E, et al. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2019. https: ...

  17. Systematic Review of Interventions Addressing Racial and Ethnic

    PURPOSE Cancer health disparities result from complex interactions among socioeconomic, behavioral, and biological factors, disproportionately affecting marginalized racial and ethnic groups. The objective of this review is to synthesize existing evidence on interventions addressing racial or ethnic disparities in cancer-related health care access and clinical outcomes. METHODS A comprehensive ...

  18. Global prevalence of intimate partner violence during the COVID-19

    The Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines were used to develop the systematic review and meta-analysis. All English-language studies conducted between 31 December 2019 and May 15/2022 were extracted from databases such as PubMed/Medline, CINAHL, and Google Scholar.

  19. Clinical care of childhood sexual abuse: a systematic review and

    Guidelines were identified in this review through a search of six international electronic databases, CINAHL, Embase, Medline, PsycINFO, TRIP, Web of Science, as well as searches of Google Scholar, GIN, WHO Global Health, SUMSearch2, the Health Management Information Consortium (HMIC), the National Institute for Health and Care Excellence (NICE ...

  20. Benefits and barriers associated with the use of smart home health

    Background: Smart home health technologies (SHHTs) have been discussed in the frame of caregiving to enable aging-in-place and independence. A systematic review was conducted in accordance with the PRISMA guidelines to gather the up-to-date knowledge on the benefits and barriers of using SHHTs in the care of older persons from the perspective of older persons and their caregivers.

  21. Hearing loss prevalence and years lived with disability, 1990-2019

    We did systematic reviews of population-representative surveys on hearing loss prevalence from 1990 to 2019. ... Because the burden of moderate-to-complete hearing loss is concentrated in countries with low health-care quality and access, stronger health-care provision mechanisms are needed to reduce the burden of unaddressed hearing loss in ...

  22. Guidance to best tools and practices for systematic reviews

    Methods and guidance to produce a reliable evidence synthesis. Several international consortiums of EBM experts and national health care organizations currently provide detailed guidance (Table (Table1). 1).They draw criteria from the reporting and methodological standards of currently recommended appraisal tools, and regularly review and update their methods to reflect new information and ...

  23. Remote measurement based care (RMBC) interventions for mental health

    Background Poor management of mental illnesses is associated with lower treatment adherence, chronification, avoidable re-hospitalisations, and high costs. Remote measurement based care (RMBC) interventions have gained increasing relevance due to its potential in providing a comprehensive and patient-centric approach to mental health management. Objectives The systematic review and meta ...

  24. Systematic review of Aboriginal and Torres Strait ...

    The findings of this review, while specific for the Australian Aboriginal and Torres Strait Islander peoples' experiences in cancer care, could be extrapolated to other Indigenous groups in countries with similar publicly funded healthcare systems. Systematic reviews of cancer care elements in Canada's Indigenous populations have been ...

  25. Effects of game-based physical education program on ...

    Search strategy and standards for selection. This research is conducted under the guidance of Cochrane Handbook for the Systematic Review of Interventions [] and the PRISMA Statement Specification for Systematic Review and Meta-analysis [].This research explores six databases, namely Web of Science, PubMed, Embase, EBSCOhost, Scopus and Cochrane.

  26. The global economic burden of COVID-19 disease: a comprehensive

    The COVID-19 pandemic has caused a considerable threat to the economics of patients, health systems, and society. This meta-analysis aims to quantitatively assess the global economic burden of COVID-19. A comprehensive search was performed in the PubMed, Scopus, and Web of Science databases to identify studies examining the economic impact of COVID-19.

  27. Finding Systematic Reviews

    Databases Containing Systematic Reviews Cochrane Library (includes: CDSR, DARE, CCTR, HTA, NHSEED, CRG's) The Cochrane Library provides access to systematic reviews in addition to other sources of reliable information, from other systematic review abstracts, technology assessments, economic evaluations and individual clinical trials.

  28. Effects of internet-based digital health interventions on ...

    Purpose The recent trend of Internet-based digital health interventions has driven researchers to implement them to promote physical activity (PA) and improve patients' health outcomes. This systematic review and meta-analysis aim to evaluate the effects of Internet-based digital health interventions on PA and quality of life (QoL) in colorectal cancer (CRC) survivors. Methods We searched ...