Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

  • Formulate a research question
  • Develop a protocol
  • Search for all relevant studies
  • Apply the selection criteria
  • Extract the data
  • Synthesize the data
  • Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

  • Academic style
  • Vague sentences
  • Style consistency

See an example

a systematic literature review is mcq

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

  • A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
  • If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
  • Access to databases and journal archives. Often, your educational institution provides you with access.
  • Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
  • Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

  • They minimize research bias by considering all available evidence and evaluating each study for bias.
  • Their methods are transparent , so they can be scrutinized by others.
  • They’re thorough : they summarize all available evidence.
  • They can be replicated and updated by others.

Systematic reviews also have a few cons .

  • They’re time-consuming .
  • They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

  • Allow you to more effectively communicate your research to other researchers and practitioners
  • Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

  • Population(s) or problem(s)
  • Intervention(s)
  • Comparison(s)

You can rearrange these four components to write your research question:

  • What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

  • Type of study design(s)
  • The population of patients with eczema
  • The intervention of probiotics
  • In comparison to no treatment, placebo , or non-probiotic treatment
  • The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
  • Randomized control trials, a type of study design

Their research question was:

  • What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

  • Background information : Provide the context of the research question, including why it’s important.
  • Research objective (s) : Rephrase your research question as an objective.
  • Selection criteria: State how you’ll decide which studies to include or exclude from your review.
  • Search strategy: Discuss your plan for finding studies.
  • Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

  • Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
  • Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
  • Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
  • Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

  • Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
  • Handsearch: Conference proceedings and reference lists of articles
  • Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
  • Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

  • Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
  • Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

  • Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
  • Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

  • Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
  • Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

  • Abstract : A summary of the review
  • Introduction : Including the rationale and objectives
  • Methods : Including the selection criteria, search method, data extraction method, and synthesis method
  • Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
  • Discussion : Including interpretation of the results and limitations of the review
  • Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

  • Student’s  t -distribution
  • Normal distribution
  • Null and Alternative Hypotheses
  • Chi square tests
  • Confidence interval
  • Quartiles & Quantiles
  • Cluster sampling
  • Stratified sampling
  • Data cleansing
  • Reproducibility vs Replicability
  • Peer review
  • Prospective cohort study

Research bias

  • Implicit bias
  • Cognitive bias
  • Placebo effect
  • Hawthorne effect
  • Hindsight bias
  • Affect heuristic
  • Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved March 4, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Med Educ Online
  • v.27(1); 2022

Does developing multiple-choice Questions Improve Medical Students’ Learning? A Systematic Review

Youness touissi.

a Faculty of Medicine and Pharmacy of Rabat, Mohammed V University, Souissi, Rabat, Morocco

Ghita Hjiej

b Faculty of Medicine and Pharmacy of Oujda, Mohammed Premier University, Oujda, Morocco

Abderrazak Hajjioui

c Laboratory of Neurosciences, Faculty of Medicine and Pharmacy of Fes, Sidi Mohammed Ben Abdallah University, Fez, Morocco

Azeddine Ibrahimi

d Laboratory of Biotechnology, Mohammed V University, Souissi, Rabat, Morocco

Maryam Fourtassi

e Faculty of Medicine of Tangier, Abdelmalek Essaadi University, Tetouan, Morocco

Practicing Multiple-choice questions is a popular learning method among medical students. While MCQs are commonly used in exams, creating them might provide another opportunity for students to boost their learning. Yet, the effectiveness of student-generated multiple-choice questions in medical education has been questioned. This study aims to verify the effects of student-generated MCQs on medical learning either in terms of students’ perceptions or their performance and behavior, as well as define the circumstances that would make this activity more useful to the students. Articles were identified by searching four databases MEDLINE, SCOPUS, Web of Science, and ERIC, as well as scanning references. The titles and abstracts were selected based on a pre-established eligibility criterion, and the methodological quality of articles included was assessed using the MERSQI scoring system. Eight hundred and eighty-four papers were identified. Eleven papers were retained after abstract and title screening, and 6 articles were recovered from cross-referencing, making it 17 articles in the end. The mean MERSQI score was 10.42. Most studies showed a positive impact of developing MCQs on medical students’ learning in terms of both perception and performance. Few articles in the literature examined the influence of student-generated MCQs on medical students learning. Amid some concerns about time and needed effort, writing multiple-choice questions as a learning method appears to be a useful process for improving medical students’ learning.

Introduction

Active learning, where students are motivated to construct their understanding of things, and make connections between the information they grasp is proven to be more effective than passively absorb mere facts [ 1 ]. However, medical students, are still largely exposed to passive learning methods, such as lectures, with no active involvement in the learning process. In order to assimilate the vast amount of information they are supposed to learn, students adopt a variety of strategies, which are mostly oriented by the assessment methods used in examinations [ 2 ].

Multiple-choice questions (MCQs) represent the most common assessment tool in medical education worldwide [ 3 ]. Therefore, it is expected that students would favor practicing MCQs, either from old exams or commercial question banks, over other learning methods to get ready for their assessments [ 4 ]. Although this approach might seem practical for students as it strengthens their knowledge and gives them a prior exam experience, it might incite surface learning instead of constructing more elaborate learning skills, such as application and analysis [ 5 ].

Involving students in creating MCQs appears to be a potential learning strategy that combines students’ pragmatic approach and actual active learning. Developing good questions, in general, implies a deep understanding and a firm knowledge of the material that is evaluated [ 6 ]. Writing a good MCQ requires, in addition to a meticulously drafted stem, the ability to suggest erroneous but possible distractors [ 7 , 8 ]. It has been suggested that creating distractors may reveal misconceptions and mistakes and underlines when students have a defective understanding of the course material [ 6 , 9 ]. In other words, creating a well-constructed MCQ requires more cognitive abilities than answering one [ 10 ]. Several studies have shown that the process of producing questions is an efficient way to motivate students and enhance their performance, and linked MCQs generation to improve test performance [ 11–15 ]. Therefore, generating MCQs might develop desirable problem-solving skills and involve students in an activity that is immediately and clearly relevant to their final examinations.

In contrast, other studies indicated there was no considerable impact of this time-consuming MCQs development activity on students’ learning [ 10 ] or that question-generation might benefit only some categories of students [ 16 ].

Because of the conflicting conclusions about this approach in different studies, we conducted a systematic review to define and document evidence of the effect of writing MCQs activity on students learning, and understand how and under what circumstances it could benefit medical students, as to our knowledge, there is no prior systematic review addressing the effect of student-generated multiple-choice questions on medical students’ learning.

Study design

This systematic review was conducted following the guidelines of the Preferred Reporting Items for Systematic Review and Meta‐Analyses (PRISMA) [ 17 ]. Ethical approval was not required because this is a systematic review of previously published research, and does not include any individual participant information.

Inclusion and exclusion criteria

Table 1 summarizes the publications’ inclusion and exclusion criteria. The target population was undergraduate and graduate medical students. The intervention was generating MCQs of all types. The learning outcomes of the intervention had to be reported using validated or non-validated instruments. We excluded studies involving students from other health-related domains, those in which the intervention was writing questions other than MCQs, and also completely descriptive studies without an evaluation section of the learning outcome. Comparison to other educational interventions was not regarded as an exclusive criterion because much educational research in the literature is case-based.

Inclusion & exclusion criteria

Search strategy

On May 16 th, 2020, two reviewers separately conducted a systematic search on 4 databases, ‘Medline’ (via PubMed), ‘Scopus’, ‘Web of Science’ and ‘Eric’ using keywords as (Medical students, Multiple-choice questions, Learning, Creating) and their possible synonyms and abbreviations which were all combined by Boolean logic terms (AND, OR, NOT) with convenient search syntax for each database (Appendix 1). Then, all the references generated from the search were imported to a bibliographic tool (Zotero®) [ 18 ] used for the management of references. The reviewers also checked manually the references list of selected publications for more relevant papers. Sections as ‘Similar Articles’ below articles (e.g., PubMed) were also checked for possible additional articles. No restrictions regarding the publication date, language, or origin country were applied.

Study selection

The selection process was directed by two reviewers independently. It started with the screening of all papers generated with the databases search, followed by removal of all duplicates. All papers whose titles had a potential relation to the research subject were kept for an abstract screening, while those with obviously irrelevant titles were eliminated. The reviewers then conducted an abstract screening; all selected studies were retrieved for a final full-text screening. Any disagreement among the reviewers concerning papers inclusion was settled through consensus or arbitrated by a third reviewer if necessary.

Data collection

Two reviewers worked separately to create a provisional data extraction sheet, using a small sample made of 4 articles. Then, they met to finalize the coding sheet by adding, editing, and deleting sections, leading to a final template, implemented using Microsoft Excel® to ensure the consistency of collected data. Each reviewer then, extracted data independently using the created framework. Finally, the two reviewers compared their work to ensure the accuracy of the collected data. The items listed in the sheet were article authorship and year of publication, country, study design, participants, subject, intervention and co-interventions, MCQ type and quality, assessment instruments, and findings.

Assessment of study methodological quality

There are few scales to assess the methodological rigor and trustworthiness of quantitative research in medical education, to mention the Best Medical Education Evaluation global scale [ 19 ], Newcastle–Ottawa Scale [ 20 ], and Medical Education Research Study Quality Instrument (MERSQI) [ 21 ]. We chose the latter to assess quantitative studies because it provides a detailed list of items with specified definition, solid validity evidence, and its scores are correlated with the citation rate in the succeeding 3 years of publication, and with the journal impact factor [ 22 , 23 ]. MERSQI evaluates study quality based on 10 items: study design, number of institutions studied, response rate, data type, internal structure, content validity, relationship to other variables, appropriateness of data analysis, the complexity of analysis, and the learning outcome. The 10 items are organized into six domains, each with a maximum score of 3 and a minimum score of 1, not reported items are not scored, resulting in a maximum MERSQI score of 18 [ 21 ].

Each article was assessed independently by two reviewers; any disagreement between the reviewers about MERSQI scoring was resolved by consensus and arbitrated by a third reviewer if necessary. If a study reported more than one outcome, the one with the highest score was taken into account.

Study design and population characteristics

Eight hundred eighty-four papers were identified after the initial databases search, of which 18 papers were retained after title and abstract screening (see Figure 1 ). Seven of them didn’t fit in the inclusion criteria for reasons as the absence of learning outcome or the targeted population being other than medical students. Finally, only 11 articles were retained, added to another 6 articles retrieved by cross-referencing. For the 17 articles included, the two reviewers agreed about 16 articles, and only one paper was discussed and decided to be included.

An external file that holds a picture, illustration, etc.
Object name is ZMEO_A_2005505_F0001_OC.jpg

Flow-chart of the study selection.

The 17 included papers reported 18 studies, as one paper included two distinct studies. Thirteen out of the eighteen studies were single group studies representing the most used study design (See Table 2 ). Eleven of these single group studies were cross-sectional while two were pre-post-test studies. The second most frequent study design encountered was cohorts, which were adopted in three studies. The remaining two were randomized controlled trials (RCT). The studies have been conducted between 1996 and 2019 with 13 studies (79%) from 2012 to 2019.

Demographics, interventions, and outcome of the included studies

MCQs : Multiple-choice questions; N : Number; NR : Not reported; RCT : Randomized controlled trial

Regarding research methodology, 10 were quantitative studies, four were qualitative and four studies had mixed methods with a quantitative part and a qualitative one (students’ feedback).

Altogether, 2122 students participated in the 17 included papers. All participants were undergraduate medical students enrolled in the first five years of medical school. The preclinical stage was the most represented, with 13 out of the 17 papers including students enrolled in the first two years of medical studies.

Most studies used more than one data source, surveys were present as a main or a parallel instrument to collect data in eight studies. Other data sources were qualitative feedback (n = 8), qualitative feedback turned to quantitative data (n = 1), pre-post-test (n = 4), and post-test (n = 5).

Quality assessment

Overall, the MERSQI scores used to evaluate the quality of the 14 quantitative studies were relatively above average which is 10.7, with a mean MERSQI score of 10.75, ranging from 7 to 14 (see details of MERSQI score for each study in Table 3 ). Studies lost points on MERSQI for using single group design, limiting participants to a single institution, the lack of validity evidence for instrument (only two studies used valid instrument) in addition to measuring the learning outcome only in terms of students’ satisfaction and perceptions.

Methodological quality of included studies according to MERSQI

Details of MERSQI Scoring :

a. Study design: Single group cross-sectional/post-test only (1); single group pre- and post-test (1.5); nonrandomized 2 groups (2); randomized controlled experiment (3).

b. Sampling: Institutions studied: Single institution (0.5); 2 institutions (1); More than 2 institutions (1.5).

c. Sampling: Response rate: Not applicable (0); Response rate < 50% or not reported (0.5); Response rate 50–74% (1); Response rate > 75% (1.5).

d. Type of data: evaluation by study participants (1); Objective measurement (3).

e. Validity evidence for evaluation instrument scores: Content: Not reported/ Not applicable (0); Reported (1).

f. Validity evidence for evaluation instrument scores: Internal structure: Not reported/ Not applicable (0); Reported (1).

g. Validity evidence for evaluation instrument scores: Relationships to other variables: Not reported/ Not applicable (0); Reported (1).

h. Appropriateness of analysis: Inappropriate (0); appropriate (1)

i. Complexity of analysis: Descriptive analysis only (1); Beyond descriptive analysis (2).

j. Outcome: Satisfaction, attitudes, perceptions (1); Knowledge, skills (1.5); Behaviors (2); Patient/health care outcome (3)

The evaluation of the educational effect of MCQs writing was carried out using objective measures in 9 out of the 18 studies included, based on pre-post-tests or post-tests only. Subjective assessments as surveys and qualitative feedbacks were present as second data sources in 7 of these 9 studies, whereas they were the main measures in the remaining nine studies. Hence, 16 studies assessed the learning outcome in terms of students’ satisfaction and perceptions towards the activity representing the first learning level of the Kirkpatrick model which is a four-level model for analyzing and evaluating the results of training and educational programs [ 24 ]. Out of these 16 studies, 3 studies wherein students expressed dissatisfaction with the process and found it disadvantageous compared to other learning methods, whereas 4 studies found mixed results as students admitted the process value though they doubted its efficiency. On the other hand, nine studies provided favorable results of the exercise which was considered of immense importance and helped students consolidate their understanding and knowledge, although students showed reservations about the time expense of the exercise in three studies.

Regarding the nine studies that used objective measures to assess students’ skills and knowledge, which represent the second level of the Kirkpatrick model, six studies reported a significant improvement in students’ grades doing this activity, whereas two studies showed no noticeable difference in grades, and one showed a slight drop in grades.

One study suggested that students performed better when writing MCQs on certain modules compared to others. Two studies found the activity beneficial to all students’ categories while another two suggested the process was more beneficial for low performers.

Four Studies also found that writing and peer review combinations were more beneficial than solely writing MCQs. On the other hand, two studies revealed that peer-reviewing groups didn’t promote learning and one study found mixed results.

Concerning the quality of the generated multiple-choice questions, most studies reported that the MCQs were of good or even high quality when compared to faculty-written MCQs, except for two studies where students created MCQs of poor quality. However, only a few studies (n = 2) reported whether students wrote MCQs that tested higher-order skills such as application and analysis or simply tested recalling facts and concepts.

The majority of interventions required students to write single best answer MCQs (n = 6), three of which were vignettes MCQs. Assertion reason MCQs were present in two studies, and in one study, students were required to write only the stem of the MCQ, while in another study, students were asked to write distractors and the answer, while the rest of studies did not report the MCQs Type.

Data and methodology

This paper methodically reviewed 17 articles investigating the impact of writing multiple-choice questions by medical students on their learning. Several studies pointedly examined the effect of the activity inquired on the learning process, whereas it only represented a small section of the article, which was used for the review. This is due to the fact that many papers focused on other concepts like assessing the quality of students generated MCQs or the efficiency of online question platforms, reflecting the scarce research on the impact of a promising learning strategy (creating MCQs) in medical education.

The mean MERSQI score of quantitative studies was 10.75 which is slightly above the level suggestive of a solid methodology set to 10.7 or higher [ 21 ]. This indicates an acceptable methodology used by most of the studies included. Yet, only two studies [ 30 , 31 ] used a valid instrument in terms of internal structure, content, and relation to other variables, making the lack of the instrument validity, in addition to the use of a single institution and single group design, as the main identified methodological issues.

Furthermore, the studies assessing the outcome in terms of knowledge and skills scored higher than the ones appraising the learning outcome regarding perception and satisfaction. Hence, we recommend that future research should provide more details on the validity parameters of the assessment instruments, and also focus on higher learning outcome levels; precisely skills and knowledge as they are typically more linked with the nature of the studied activity.

Relation with existing literature

Apart from medical education, the impact of students’ generated questions has been a relevant research question in a variety of educational environments. Fu-Yun & Chun-Ping demonstrated through hundreds of papers that student-generated questions promoted learning and led to personal growth [ 32 ]. For example, in Ecology, students who were asked to construct multiple-choice questions significantly improved their grades [ 33 ]. Also, in an undergraduate taxation module, students who were asked to create multiple-choice questions significantly improved their academic achievement [ 34 ].

A previous review explored the impact of student-generated questions on learning and concluded that the process of constructing questions raised students’ abilities of recall and promoted understanding of essential subjects as well as problem-solving skills [ 35 ]. Yet, this review gave a general scope on the activity of generating questions, taking into consideration all questions formats. Thus, its conclusions will not necessarily concord with our review because medical students define a special students’ profile [ 36 ], along with the particularity of multiple-choice questions. As far as we know, this is the first systematic review made to appraise the pedagogical interest of the described process of creating MCQs in medical education.

Students’ satisfaction and perceptions

Students’ viewpoints and attitudes toward the MCQ generation process were evaluated in multiple studies, and the results were generally encouraging, despite a few exceptions where students expressed negative impressions of the process and favored other learning methods over it [ 4 , 10 ]. The most pronouncing remarks were essentially on the time-consumption limiting the process efficiency. This was mainly related to the complexity of the task given to students who were required to write MCQs in addition to other demanding assignments.

Since the most preferred learning method for students is learning by doing, they presumably benefit more when instructions are conveyed in shorter segments, and when introduced in an engaging format [ 37 ]. Thus, some researchers tried more flexible strategies as providing the MCQs distractors and asking students for the stem or better providing the stem and requesting distractors as these were considered to be the most challenging parts of the process [ 38 ].

Some authors used online platforms to create and share questions making the MCQs generation smoother. Another approach to motivate students was including some generated MCQs in examinations, to boost students’ confidence and enhance their reflective learning [ 39 ]. These measures, supposed to facilitate the task, were perceived positively by students.

Students’ performance

Regarding students’ performance, MCQs-generation exercise broadly improved students’ grades. However, not all studies have reported positive results. Some noted no significant effect of writing MCQs on students’ exam scores [ 10 , 31 ]. This was explained by the small number of participants, and the lack of instructors’ supervision. Moreover, students were tested on a broader material than the one they were instructed to write MCQs on, meaning that students might have effectively benefited from the process if they created a larger number of MCQs covering a wider range of material or if the process was aligned with the whole curriculum content. Besides, some studies reported that low performers benefited more from the process of writing MCQs, concordantly with the findings of other studies which indicate that activities promoting active learning advantage lower-performing students more than higher-performing ones [ 40 , 41 ]. Another suggested explanation was the fact that low achievers tried to memorize student-generated MCQs when these made part of their examinations, reversely favoring surface learning instead of the deep learning anticipated from this activity. This created a dilemma between enticing students to participate in this activity and the disadvantage of memorizing MCQs. Therefore, including modified student-generated MCQs after instructors’ input, rather than the original student-generated version in the examinations’ material, might be a reasonable option along with awarding extra points when students are more involved in the process of writing MCQs.

Determinant factors

Students’ performance tends to be related to their ability to generate high-quality questions. As suggested in preceding reviews [ 35 , 42 ], assisting students in constructing questions may enhance the quality of these students’ generated questions, encourage learning, and improve students’ achievement. Also, guiding students to write MCQs makes it possible to test higher-order skills as application and analysis besides recall and comprehension. Accordingly, in several studies, students were provided with instructions on how to write high-quality multiple-choice questions, resulting in high-quality student-generated MCQs [ 10 , 43–45 ]. Even so, such guidelines must take into account not making students’ job more challenging to maintain the process as pleasant.

Several papers discussed various factors that influence the learning outcome of the activity, as working in groups and peer checking MCQs, which were found to be associated with higher performance [ 30 , 38 , 43 , 44 , 46–49 ]. These factors were also viewed favorably by students because of their potential to broaden and deepen one’s knowledge, as well as to notice any misunderstandings or problems, according to many studies, that highlighted a variety of beneficial outcomes of peer learning approaches in the education community [ 42 , 50 , 51 ]. However, in other studies, students preferred to work alone and demanded that time devoted to peer-reviewing MCQs be reduced [ 38 , 45 ]. This was mostly due to students’ lack of trust in the quality of MCQs created by peers; thus, evaluating students’ MCQs by instructors was also a component of an effective intervention.

Strengths and limitations

The main limitation of the present review is the scarcity of studies in the literature. We used a narrowed inclusion criterion leading to the omission of articles published in non-indexed journals and papers from other health-care fields that may have been instructive. However, the choice of limiting the review scope to medical students only was motivated by the specificity of the medical education curricula and teaching methods compared to other health professions categories in most settings. Another limitation is the weak methodology of a non-negligible portion of studies included in this review which makes drawing and generalizing conclusions a delicate exercise. On the other hand, this is the first review to summarize data on the learning benefits of creating MCQs in medical education and to shed light on this interesting learning tool.

Writing multiple-choice questions as a learning method might be a valuable process to enhance medical students learning despite doubts raised on its real efficiency and pitfalls in terms of time and effort.

There is presently a dearth of research that examines the influence of student-generated MCQs on learning. Future research on the subject must use a strong study design, valid instruments, simple and flexible interventions, as well as measure learning based on performance and behavior, and explore the effect of the process on different students’ categories (eg. performance, gender, level), in order to reach the most appropriate circumstances for the activity to get the best out of it.

Appendix: Search strategy. 

  • Query: ((((Medical student) OR (Medical students)) AND (((Create) OR (Design)) OR (Generate))) AND ((((multiple-choice question) OR (Multiple-choice questions)) OR (MCQ)) OR (MCQs))) AND (Learning)
  • Results: 300
  • Query: ALL (medical PRE/0 students) AND ALL (multiple PRE/0 choice PRE/0 questions) AND ALL (learning) AND ALL (create OR generate OR design)
  • Results: 468
  • Query: (ALL = ‘Multiple Choice Questions’ OR ALL = ‘Multiple Choice Question’ OR ALL = MCQ OR ALL = MCQs) AND (ALL = ‘Medical Students’ OR ALL = ‘Medical Student’) AND (ALL = Learning OR ALL = Learn) AND (ALL = Create OR ALL = Generate OR ALL = Design)
  • Results: 109
  • Query: ‘Medical student’ AND ‘Multiple choice questions’ AND Learning AND (Create OR Generate OR Design)

Total = 884

After deleting double references : Number: 697

Funding Statement

The author(s) reported there is no funding associated with the work featured in this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

Systematic literature review

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

  • the definition of a systematic literature review
  • the purpose of a systematic literature review
  • the different types of systematic reviews
  • how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

  • Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
  • Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
  • Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
  • Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
  • Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
  • Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
  • Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
  • Prognostic reviews examine the course of a disease and its potential outcomes.
  • Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
  • Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

  • Decide on your team.
  • Formulate your question.
  • Plan your research protocol.
  • Search for the literature.
  • Screen the literature.
  • Assess the quality of the studies.
  • Extract the data.
  • Analyze the results.
  • Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

  • Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
  • Intervention : which main intervention or treatment are you considering for assessment?
  • Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
  • Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

  • the objectives of your project
  • the specific methods and processes that you’ll use
  • the eligibility criteria of the individual studies
  • how you plan to extract data from individual studies
  • which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

  • non-peer-reviewed journals
  • pharmaceutical industry files
  • conference proceedings
  • pharmaceutical company websites
  • internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

  • The databases you search and which years they cover
  • The dates you first run the searches, and when they’re updated
  • Which strategies you use, including search terms
  • The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

  • Pre-screening of all titles and abstracts, and selecting those appropriate
  • Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

  • Were the participants really randomly allocated to their groups?
  • Were the groups similar in terms of prognostic factors?
  • Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

  • Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
  • Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
  • Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

a systematic literature review is mcq

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

  • We're Hiring!
  • Help Center

paper cover thumbnail

Multiple choice question writing and medical students: a systematic literature review

Profile image of Thomas C Varkey

2023, MedEdPublish

Multiple-choice question (MCQ) tests have been a mainstay for ensuring fairness and ease of grading within school curricula for several years, and in many institutions are the primary way that comprehension is measured. Students, teachers, and researchers alike have developed methods and materials to improve performance using the MCQ format. One phenomenon of note has been the development of several question banks to help students study for major examinations such as the ACT, MCAT, and USMLE. In addition to the large emphasis on these questions within the curriculum, there is potential for the utilization of these questions as a metacognition technique for student learning . One practice of interest to the research team is students writing their own multiplechoice questions as a learning methodology. In utilizing this method, students are both required to retrieve the information that they utilized and create a convincing question with similar alternative answer choices, including...

Related Papers

BMC Medical Education

Felipe Lucena

a systematic literature review is mcq

Sapthagiri Institute of Medical Sciences Research Centre

Introduction: ​ There ​ are ​ totally different ​ ways to encourage students ​ improve their ​ learning outcome and understanding of ​ an issue​. This study was undertaken ​ assess the ​ impact of student­based construction of multiple ​ choice ​ questions (MCQs) as a ​ input for ​ learning outcome and understanding of topics in ​ biochemistry​. ​ Materials and Methods: The study was carried out at Sapthagiri institute of medical sciences, Department of Biochemistry. First year students were asked to construct MCQs. These students were pre­ and post­tested and the effect of the additional intervention (MCQ­construction) measured. They were then polled on their preferred methods of learning before and after a learning exercise in which they were asked to create some MCQs. ​ Results: The MCQ questions designed by the students were of a high standard and clearly displayed an understanding of the topic concerned. Student's opinion was significantly more favourable towards self­made MCQs than traditional learning methodologies (tutorials, books). Conclusions: Construction of MCQs as a learning tool is an unfamiliar exercise to most students and is an unpopular learning strategy. However, students are capable to prepare high quality questions, and the challenge for medical faculties is to utilize this initiative to the students' advantage.

International Journal of Medical Research and Review

siddharth banode

INTED Proceedings

Dr Jorge Reyna

For decades, medical education has used Multiple Choice Questions (MCQs) in undergraduate, postgraduate, and specialist training programs. With the development of Learning Management Systems (LMS) in the early 1990s, educators adapted paper-based MCQs for formative and summative assessments. Technology allowed automated feedback, question shuffling, instant marking, and 'branching' based on student responses. With the advent of e-learning authoring tools in the early 2000s, it became possible to create interactive online tests with images, animations, videos, drag-anddrop elements, blanks to fill in, and hotspots. Effective MCQ writing involves understanding educational concepts like learning taxonomies, constructive alignment, approaches to learning, cognitive load, and student motivation to learn. It is also essential when structuring a question to avoid ambiguity and to have the imagination to write MCQs that measure application of knowledge. Whether a basic or advanced topic, it is possible to design MCQs that measure higher-order thinking that require the student to apply their knowledge rather than simply recalling it. MCQs with hypothetical scenarios can measure higher-order thinking and promote deep learning. However, preparing the students for this type of examination is essential to enhance their learning experience. This article discusses the theoretical considerations involved in writing MCQs for medical education which encourage deep learning and improve the student learning experience.

Dr. VISHAL VARIA

Journal of the faculty of medicine baghdad

Zainab M . Alawad

Annals of the Academy of Medicine, Singapore

Edward Palmer

Medical Education

Eivind Meland

Indian Journal of Community Medicine

Pradeep Kumar

Trupti B . Naik

Introduction : Learning of medical students is assessed worldwide by using theory, practical and clinical examinations including written papers as well as objective structured clinical examinations (OSCE), case studies, and viva voce examinations. The written assessments are done with the help of multiple choice questions and subjective or descriptive open-ended questions. The descriptive questioning may be in the form of long essay questions (LEQ), structured essay questions (SEQ) and modified essay questions (MEQ). The objective of the study was to assess the preference of undergraduate medical students among the three methods of written assessment in open ended questions. Methodology : A cross sectional study was conducted among medical students at private medical college, Karnataka. A lecture was delivered on a topic. Test was conducted on the same topic and a question was asked in 3 different formats: LEQ, SEQ and MEQ. After the test, a pretested and semi structured questionnaire was used for data collection and analyzed using appropriate statistical methods. Results : Modified essay question was preferred for being: easier to answer, enjoyable, less time consuming, the best method to assess problem solving skills and clinical reasoning ability, to adequately assess the knowledge and a method, which is free of evaluator bias. Long essay question was considered as the best method to assess answer presentation skills and it covered greater spectrum of content. Conclusion : Assessment methods have both advantages and disadvantages. Finding the right assessment method depends on evaluators purpose to assess students in terms of skills, knowledge and understanding.

RELATED PAPERS

European Neuropsychopharmacology

Milica Prostran

JORGE ARMANDO ALBA SALINAS

Alan J. Yuter

College Teaching

Kristina Lambright

COLLOQUIUM AGRARIAE

Carlos Alberto Martins CORDEIRO

Michael Gershon

Laura Medialdea

Alex Vakaloudis

Food Analytical Methods

ambika bhardwaj

Rabiul Hosen

Revista musical chilena

Juan Manuel Bautista Gonzalez

African Journal of Pharmacy and Pharmacology

Suzan Matar

Soft Matter

Akram Hijazi

agatha priskila

Annals of Oncology

János Révész

Scientific Studies of Reading

Josefine Karlsson

Sedekah Al-qur'an

Ammar Muhammad Muhammad

Oscar Rafael Durán Peña

Ethiopian Journal of Health Development

Gideon Kruseman

Guilherme Sousa da Silva

Journal of Applied Geophysics

Jeferson De Souza

Journal of Clinical Oncology

Mohamad Eloubeidi

Alfonso Canul

Lecture Notes in Computer Science

Magdalena Zawisławska

IFAC Proceedings Volumes

Frontiers in plant science

Justice Norvienyeku

Lasse Berntzen

jhgjfgtf hgjfgdghf

The Future of Aging

Caraka: Jurnal Ilmiah Kebahasaan, Kesastraan, dan Pemelajarannya

Latif Anshori Kurniawan

RELATED TOPICS

  •   We're Hiring!
  •   Help Center
  • Find new research papers in:
  • Health Sciences
  • Earth Sciences
  • Cognitive Science
  • Mathematics
  • Computer Science
  • Academia ©2024

Research guidance, Research Journals, Top Universities

Understanding Research & Publication Ethics through MCQ

Research and Publication Ethics : In this blog post (MCQ on Research & Publication Ethics), you will understand Research ethics through various MCQs.

Topics Covered : Literature Review, Review Paper Writing, Writing a Research article, etc;

MCQs on Research & Publication Ethics

1. In the Literature review, systematic reorganization and reshuffling of the information to develop knowledge/ reasoning/ problem definition is referred to as-

 Synthesizing

 Summarizing

 Both 1 and 2

 None of these

2. Rationale of the study is

 Logic leading to methodology

 Logic leading to aim and objectives

 Logic behind Introduction

 All the above

3. Literature review for which body of knowledge is collected from the supervisor is –

 Literature Survey

 Experience survey

 Objective survey

 Both 2 and 1

4. A literature review is ………… of research.

  Foundation

 Last step

 a formality

 Both 2 and 3

5. Doing literature review from broader topic to a focal point is the-

 Literature search

 Convergent search

 Objective search

 Both b and c

6. Objectives of a study must be –

 Specific & Measurable

 Relevant & Time-bound

 1 & 2 both

 Easy to do

7. Literature review/survey is a –

 Continuous process

 Initial stage process

 Offline process

 All of the above

8. Curly bracket {} will search for the –

 Specific phrases

 Fussy phrases

 Both a and b

9. Systematic and organized compilation and critical study of a related body of knowledge is called

 Literature review

 Data analysis

 Statistical analysis

 Result & Discussion

10. Boolean search does not include-

11. SHODHGANGOTRI is the databases of-

 Thesis

  Synopsis

12. PUBMED is maintained in the US by the

  NCBI

13. SHODHGANGA is the database of –

  Thesis

 Synopsis

14. INFLIBNET is –

 Informatics and Library Network Centre

 Library and Information Network Centre

 Information and Library Network Centre

15. Which does not cover the social sciences database

 Scopus

  Pubmed

 Web of Science

16. The free national repository of Indian synopsis database is/are

 Shodhshudhi

 Shodhganga

 Eshodhsindhu

 None of the above

17. Literature management tool/(s)-

 Mendeley

 Endnote

 b & c both

18. DOAR stands for –

 Directory of open access repository

 Dictionary of open access repository

 Development of analytical report

 Development of access report

19. Which is a huge subscription-based scientific citation indexing service

 Shodganga

 Shodgangotri

20. Endnote supports importing in

 BiBtex

 EndNote

 a & b both

Research & Publication Ethics MCQs on Review Paper

21. Outline of the review paper

 drawn in the planning stage in consultation with mentor and research team

 should be tight and focussed

 unique summarising and synthesizing of idea

22. The review paper is different from the literature review in

 size, shape, and approach

 size only

 shape only

 size and shape

23. Systematic reviews are common in

 social sciences

 sciences

 life sciences

 sciences and life sciences

24. The section of the article which must not have subheadings

 methods

 introduction

 discussion

25. Writing review paper should be started

 in the early stage of planning your research topic

 in the later stage of planning of your research topic

 in the final stage of execution of your research work

 a & c both

26. The first rule of writing a body of the paper is

 include many references

 stick to your outline

 use effective English

27. In the conclusion section

 no separate headings are used

 the objective of the paper is restated

 the expected outcome may be included

28. Review papers are written for

 giving a new direction to existing research

 increasing the h index of author/researcher

 wide readership

29. Abstract is

 same as your Ph.D. research synopsis

 a comprehensive research summary with good length

 crisp, short, and representative summary of research work

30. Review paper writing is associated with a paradox

 Intensive task but not much recognition as compared to research papers

 Less effort is needed but more recognition as compared to research papers

 No lab work but more readership as compared to research papers

31. Which one of the following should not be acknowledged in an article

 Project grant

 Analytical facility provider

 Gift sample providers

 Senior authors

32. Identify which one of the following is a correct pair

 Acknowledgment – sources of study

 Materials and methods – output of the experiments

 The abstract – central idea

 Discussion – rationality of the work

33. Which is not a good a practice

 to cut the work into smaller pieces of work

 Plan the preliminary promising studies which yielded good and novel results

 Select journals with good indexing or Impact factor

34. IMRaD stands for

 introduction, material /methods, results and discussion

 investigation, methods results and discussion

 investigation, methods, results, analysis discussion

35. Title of the paper should be?

 Simple

 Reader-friendly

 Representative of study

36. Two key features of the Method section are

 short and catchy sentences

 clarity and the reproducibility

 accuracy and precision

37. In the introduction, you should

 give a brief background and brief literature review

 introduce the topic

 identify research gap, define problem & present rationality

38. In the result section

 Do not exaggerate the results

 Do not be afraid of reporting negative results

 Statistical support must be there

39. Infographics

 must be self-explanatory

 must have suitable legends or footnotes

40. The materials and methods section is also known as

 Methods

 Experimental

 Methodology

41. Major factors to be considered in right referencing are

 Quality

 Quantity

 Uniform Styles/ guidelines

 All of these

42. Sources of information that you are giving at the end of your manuscript is termed as

  Reference

 Results

 Discussions

43. When references are listed in alphabetical order stating the last name of the first author like Smith, G. (2008) is –

 APA style

 MLA style

 CMS style

44. Graphical abstract may contain –

 A graph

 A figure

 An image

45. Suggested reviewers should not include-

 Your supervisor

 Field experts

 Both A and B

46. Keywords of a manuscript can be related to the –

 Title

 Content

47. When “neither author nor peer reviewers know about the names of each other” This technique are called as –

 Double-blind

 Single Blind

48. The proofreading step is the final step –

 After acceptance

 Before acceptance

49. One should Discuss the results-

 Critically

 With Logical reasoning

 With general statements

50. The author should address the reviewer’s comments –

 Humbly

 Logically

Thanks for visiting our website to understand the Research & Publication Ethics through MCQs.

Share this:

2 thoughts on “understanding research & publication ethics through mcq”.

very much helpful and handy notes through mcq

Leave a Comment Cancel reply

Save my name, email, and website in this browser for the next time I comment.

Notify me of follow-up comments by email.

Notify me of new posts by email.

Banner

Systematic Reviews: Before you begin...

  • Before you begin...
  • Introducing systematic reviews
  • Step 1: Preparation
  • Step 2: Scoping
  • Step 3: Planning your search strategy
  • Step 4: Recording and managing results
  • Step 5: Selecting papers and quality assessment

Welcome to the Library’s Systematic Review Subject Guide

New to systematic reviews?

This Subject Guide page highlights things you should consider before and while undertaking your review. 

The systematic review process requires a lot of preparation, detailed searching, and analysing. It may take longer than you think to complete!

Image of a laptop and a notepad and pen on a wooden desk

Any questions? Contact your Subject Librarian.

Before you begin your review...

Please be assured that your Subject Librarian will support you as best they can. 

Subject Librarians are able to show QUB students and staff undertaking any type of literature search (e.g. literature review, scoping review, systematic review) how to:

  • Structure searches using AND/OR
  • Select appropriate databases
  • Search selected databases
  • Save and re-run searches
  • Export database results
  • Store and deduplicate results using EndNote
  • Identify grey literature (if required)

At peak periods of demand, Subject Librarians might not be able to deliver all of the above. Please contact your Subject Librarian for guidance on this.

QUB students and staff must provide Subject Librarians with a clear search topic or question, along with a selection of appropriate keywords and synonyms. Students should discuss these with their supervisor before contacting Subject Librarians.

Subject Librarians are unable to do the following for QUB students and staff:

  • Check review protocols
  • Peer review, or approve, search strategies        
  • Create search strategies from scratch
  • Search databases or grey literature sources
  • Deduplicate results
  • Screen results
  • Demonstrate systematic review tools (e.g. Covidence, Rayyan)
  • Create PRISMA flowcharts or similar documentation

Subject Librarians do not need to be listed as review co-authors. However, if reviewers wish to acknowledge the input of a specific Subject Librarian, they should contact the relevant Subject Librarian to ensure appropriate wording.

  • Next: Introducing systematic reviews >>
  • Last Updated: Dec 18, 2023 5:30 PM
  • URL: https://libguides.qub.ac.uk/systematicreviews

A Systematic Review of Automatic Question Generation for Educational Purposes

  • Open access
  • Published: 21 November 2019
  • Volume 30 , pages 121–204, ( 2020 )
  • Cite this article

You have full access to this open access article

  • Ghader Kurdi   ORCID: orcid.org/0000-0003-1745-5581 1 ,
  • Jared Leo 1 ,
  • Bijan Parsia 1 ,
  • Uli Sattler 1 &
  • Salam Al-Emari 2  

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

43k Accesses

144 Citations

12 Altmetric

Explore all metrics

While exam-style questions are a fundamental educational tool serving a variety of purposes, manual construction of questions is a complex process that requires training, experience, and resources. This, in turn, hinders and slows down the use of educational activities (e.g. providing practice questions) and new advances (e.g. adaptive testing) that require a large pool of questions. To reduce the expenses associated with manual construction of questions and to satisfy the need for a continuous supply of new questions, automatic question generation (AQG) techniques were introduced. This review extends a previous review on AQG literature that has been published up to late 2014. It includes 93 papers that were between 2015 and early 2019 and tackle the automatic generation of questions for educational purposes. The aims of this review are to: provide an overview of the AQG community and its activities, summarise the current trends and advances in AQG, highlight the changes that the area has undergone in the recent years, and suggest areas for improvement and future opportunities for AQG. Similar to what was found previously, there is little focus in the current literature on generating questions of controlled difficulty, enriching question forms and structures, automating template construction, improving presentation, and generating feedback. Our findings also suggest the need to further improve experimental reporting, harmonise evaluation metrics, and investigate other evaluation methods that are more feasible.

Similar content being viewed by others

a systematic literature review is mcq

The Promises and Challenges of Artificial Intelligence for Teachers: a Systematic Review of Research

Ismail Celik, Muhterem Dindar, … Sanna Järvelä

a systematic literature review is mcq

Word problems in mathematics education: a survey

Lieven Verschaffel, Stanislaw Schukajlow, … Wim Van Dooren

a systematic literature review is mcq

The Value of Using Tests in Education as Tools for Learning—Not Just for Assessment

Dillon H. Murphy, Jeri L. Little & Elizabeth L. Bjork

Avoid common mistakes on your manuscript.

Introduction

Exam-style questions are a fundamental educational tool serving a variety of purposes. In addition to their role as an assessment instrument, questions have the potential to influence student learning. According to Thalheimer ( 2003 ), some of the benefits of using questions are: 1) offering the opportunity to practice retrieving information from memory; 2) providing learners with feedback about their misconceptions; 3) focusing learners’ attention on the important learning material; 4) reinforcing learning by repeating core concepts; and 5) motivating learners to engage in learning activities (e.g. reading and discussing). Despite these benefits, manual question construction is a challenging task that requires training, experience, and resources. Several published analyses of real exam questions (mostly multiple choice questions (MCQs)) (Hansen and Dexter 1997 ; Tarrant et al. 2006 ; Hingorjo and Jaleel 2012 ; Rush et al. 2016 ) demonstrate their poor quality, which Tarrant et al. ( 2006 ) attributed to a lack of training in assessment development. This challenge is augmented further by the need to replace assessment questions consistently to ensure their validity, since their value will decrease or be lost after a few rounds of usage (due to being shared between test takers), as well as the rise of e-learning technologies, such as massive open online courses (MOOCs) and adaptive learning, which require a larger pool of questions.

Automatic question generation (AQG) techniques emerged as a solution to the challenges facing test developers in constructing a large number of good quality questions. AQG is concerned with the construction of algorithms for producing questions from knowledge sources, which can be either structured (e.g. knowledge bases (KBs) or unstructured (e.g. text)). As Alsubait ( 2015 ) discussed, research on AQG goes back to the 70’s. Nowadays, AQG is gaining further importance with the rise of MOOCs and other e-learning technologies (Qayyum and Zawacki-Richter 2018 ; Gaebel et al. 2014 ; Goldbach and Hamza-Lup 2017 ).

In what follows, we outline some potential benefits that one might expect from successful automatic generation of questions. AQG can reduce the cost (in terms of both money and effort) of question construction which, in turn, enables educators to spend more time on other important instructional activities. In addition to resource saving, having a large number of good-quality questions enables the enrichment of the teaching process with additional activities such as adaptive testing (Vie et al. 2017 ), which aims to adapt learning to student knowledge and needs, as well as drill and practice exercises (Lim et al. 2012 ). Finally, being able to automatically control question characteristics, such as question difficulty and cognitive level, can inform the construction of good quality tests with particular requirements.

Although the focus of this review is education, the applications of question generation (QG) are not limited to education and assessment. Questions are also generated for other purposes, such as validation of knowledge bases, development of conversational agents, and development of question answering or machine reading comprehension systems, where questions are used for training and testing.

This review extends a previous systematic review on AQG (Alsubait 2015 ), which covers the literature up to the end of 2014. Given the large amount of research that has been published since Alsubait’s review was conducted (93 papers over a four year period compared to 81 papers over the preceding 45-year period), an extension of Alsubait’s review is reasonable at this stage. To capture the recent developments in the field, we review the literature on AQG from 2015 to early 2019. We take Alsubait’s review as a starting point and extend the methodology in a number of ways (e.g. additional review questions and exclusion criteria), as will be described in the sections titled “ Review Objective ” and “ Review Method ”. The contribution of this review is in providing researchers interested in the field with the following:

a comprehensive summary of the recent AQG approaches;

an analysis of the state of the field focusing on differences between the pre- and post-2014 periods;

a summary of challenges and future directions; and

an extensive reference to the relevant literature.

Summary of Previous Reviews

There have been six published reviews on the AQG literature. The reviews reported by Le et al. 2014 , Kaur and Bathla 2015 , Alsubait 2015 and Rakangor and Ghodasara ( 2015 ) cover the literature that has been published up to late 2014 while those reported by Ch and Saha ( 2018 ) and Papasalouros and Chatzigiannakou ( 2018 ) cover the literature that has been published up to late 2018. Out of these, the most comprehensive review is Alsubait’s, which includes 81 papers (65 distinct studies) that were identified using a systematic procedure. The other reviews were selective and only cover a small subset of the AQG literature. Of interest, due to it being a systematic review and due to the overlap in timing with our review, is the review developed by Ch and Saha ( 2018 ). However, their review is not as rigorous as ours, as theirs only focuses on automatic generation of MCQs using text as input. In addition, essential details about the review procedure, such as the search queries used for each electronic database and the resultant number of papers, are not reported. In addition, several related studies found in other reviews on AQG are not included.

Findings of Alsubait’s Review

In this section, we concentrate on summarising the main results of Alsubait’s systematic review, due to its being the only comprehensive review. We do so by elaborating on interesting trends and speculating about the reasons for those trends, as well as highlighting limitations observed in the AQG literature.

Alsubait characterised AQG studies along the following dimensions: 1) purpose of generating questions, 2) domain, 3) knowledge sources, 4) generation method, 5) question type, 6) response format, and 7) evaluation.

The results of the review and the most prevalent categories within each dimension are summarised in Table  1 . As can be seen in Table  1 , generating questions for a specific domain is more prevalent than generating domain-unspecific questions. The most investigated domain is language learning (20 studies), followed by mathematics and medicine (four studies each). Note that, for these three domains, there are large standardised tests developed by professional organisations (e.g. Test of English as a Foreign Language (TOEFL), International English Language Testing System (IELTS) and Test of English for International Communication (TOEIC) for language, Scholastic Aptitude Test (SAT) for mathematics and board examinations for medicine). These tests require a continuous supply of new questions. We believe that this is one reason for the interest in generating questions for these domains. We also attribute the interest in the language learning domain to the ease of generating language questions, relative to questions belonging to other domains. Generating language questions is easier than generating other types of questions for two reasons: 1) the ease of adopting text from a variety of publicly available resources (e.g. a large number of general or specialised textual resources can be used for reading comprehension (RC)) and 2) the availability of natural language processing (NLP) tools for shallow understanding of text (e.g. part of speech (POS) tagging) with an acceptable performance, which is often sufficient for generating language questions. To illustrate, in Chen et al. ( 2006 ), the distractors accompanying grammar questions are generated by changing the verb form of the key (e.g. “write”, “written”, and “wrote” are distractors while “writing” is the key). Another plausible reason for interest in questions on medicine is the availability of NLP tools (e.g. named entity recognisers and co-reference resolvers) for processing medical text. There are also publicly available knowledge bases, such as UMLS (Bodenreider 2004 ) and SNOMED-CT (Donnelly 2006 ), that are utilised in different tasks such as text annotation and distractor generation. The other investigated domains are analytical reasoning, geometry, history, logic, programming, relational databases, and science (one study each).

With regard to knowledge sources, the most commonly used source for question generation is text (Table  1 ). A similar trend was also found by Rakangor and Ghodasara ( 2015 ). Note that 19 text-based approaches, out of the 38 text-based approaches identified by Alsubait ( 2015 ), tackle the generation of questions for the language learning domain, both free response (FR) and multiple choice (MC). Out of the remaining 19 studies, only five focus on generating MCQs. To do so, they incorporate additional inputs such as WordNet (Miller et al. 1990 ), thesaurus, or textual corpora. By and large, the challenge in the case of MCQs is distractor generation. Despite using text for generating language questions, where distractors can be generated using simple strategies such as selecting words having a particular POS or other syntactic properties, text often does not incorporate distractors, so external, structured knowledge sources are needed to find what is true and what is similar. On the other hand, eight ontology-based approaches are centred on generating MCQs and only three focus on FR questions.

Simple factual wh-questions (i.e. where the answers are short facts that are explicitly mentioned in the input) and gap-fill questions (also known as fill-in-the-blank or cloze questions) are the most generated types of questions with the majority of them, 17 and 15 respectively, being generated from text. The prevalence of these questions is expected because they are common in language learning assessment. In addition, these two types require relatively little effort to construct, especially when they are not accompanied by distractors. In gap-fill questions, there are no concerns about the linguistic aspects (e.g. grammaticality) because the stem is constructed by only removing a word or a phrase from a segment of text. The stem of a wh-question is constructed by removing the answer from the sentence, selecting an appropriate wh-word, and rearranging words to form a question. Other types of questions such as mathematical word problems, Jeopardy-style questions, Footnote 1 and medical case-based questions (CBQs) require more effort in choosing the stem content and verbalisation. Another related observation we made is that the types of questions generated from ontologies are more varied than the types of questions generated from text.

Limitations observed by Alsubait ( 2015 ) include the limited research on controlling the difficulty of generated questions and on generating informative feedback. Existing difficulty models are either not validated or only applicable to a specific type of question (Alsubait 2015 ). Regarding feedback (i.e. an explanation for the correctness/incorrectness of the answer), only three studies generate feedback along with the questions. Even then, the feedback is used to motivate students to try again or to provide extra reading material without explaining why the selected answer is correct/incorrect. Ungrammaticality is another notable problem with auto-generated questions, especially in approaches that apply syntactic transformations of sentences (Alsubait 2015 ). For example, 36.7% and 39.5% of questions generated in the work of Heilman and Smith ( 2009 ) were rated by reviewers as ungrammatical and nonsensical, respectively. Another limitation related to approaches to generating questions from ontologies is the use of experimental ontologies for evaluation, neglecting the value of using existing, probably large, ontologies. Various issues can arise if existing ontologies are used, which in turn provide further opportunities to enhance the quality of generated questions and the ontologies used for generation.

Review Objective

The goal of this review is to provide a comprehensive view of the AQG field since 2015. Following and extending the schema presented by Alsubait ( 2015 ) (Table  1 ), we have structured our review around the following four objectives and their related questions. Questions marked with an asterisk “*” are those proposed by Alsubait ( 2015 ). Questions under the first three objectives (except question 5 under OBJ3) are used to guide data extraction. The others are analytical questions to be answered based on extracted results.

Providing an overview of the AQG community and its activities

What is the rate of publication?*

What types of papers are published in the area?

Where is research published?

Who are the active research groups in the field?*

Summarising current QG approaches

What is the purpose of QG?*

What method is applied?*

What tasks related to question generation are considered?

What type of input is used?*

Is it designed for a specific domain? For which domain?*

What type of questions are generated?* (i.e., question format and answer format)

What is the language of the questions?

Does it generate feedback?*

Is difficulty of questions controlled?*

Does it consider verbalisation (i.e. presentation improvements)?

Identifying the gold-standard performance in AQG

Are there any available sources or standard datasets for performance comparison?

What types of evaluation are applied to QG approaches?*

What properties of questions are evaluated? Footnote 2 and What metrics are used for their measurement?

How does the generation approach perform?

What is the gold-standard performance?

Tracking the evolution of AQG since Alsubait’s review

Has there been any progress on feedback generation?

Has there been progress on generating questions with controlled difficulty?

Has there been progress on enhancing the naturalness of questions (i.e. verbalisation)?

One of our motivations for pursuing these objectives is to provide members of the AQG community with a reference to facilitate decisions such as what resources to use, whom to compare to, and where to publish. As we mentioned in the  Summary of Previous Reviews , Alsubait ( 2015 ) highlighted a number of concerns related to the quality of generated questions, difficulty models, and the evaluation of questions. We were motivated to know whether these concerns have been addressed. Furthermore, while reviewing some of the AQG literature, we made some observations about the simplicity of generated questions and about the reporting being insufficient and heterogeneous. We want to know whether these issues are universal across the AQG literature.

Review Method

We followed the systematic review procedure explained in (Kitchenham and Charters 2007 ; Boland et al. 2013 ).

Inclusion and Exclusion Criteria

We included studies that tackle the generation of questions for educational purposes (e.g. tutoring systems, assessment, and self-assessment) without any restriction on domains or question types. We adopted the exclusion criteria used in Alsubait ( 2015 ) (1 to 5) and added additional exclusion criteria (6 to 13). A paper is excluded if:

it is not in English

it presents work in progress only and does not provide a sufficient description of how the questions are generated

it presents a QG approach that is based mainly on a template and questions are generated by substituting template slots with numerals or with a set of randomly predefined values

it focuses on question answering rather than question generation

it presents an automatic mechanism to deliver assessments, rather than generating assessment questions

it presents an automatic mechanism to assemble exams or to adaptively select questions from a question bank

it presents an approach for predicting the difficulty of human-authored questions

it presents a QG approach for purposes other than those related to education (e.g. training of question answering systems, dialogue systems)

it does not include an evaluation of the generated questions

it is an extension of a paper published before 2015 and no changes were made to the question generation approach

it is a secondary study (i.e. literature review)

it is not peer-reviewed (e.g. theses, presentations and technical reports)

its full text is not available (through the University of Manchester Library website, Google or Google scholar).

Search Strategy

Data sources.

Six data sources were used, five of which were electronic databases (ERIC, ACM, IEEE, INSPEC and Science Direct), which were determined by Alsubait ( 2015 ) to have good coverage of the AQG literature. We also searched the International Journal of Artificial Intelligence in Education (AIED) and the proceedings of the International Conference on Artificial Intelligence in Education for 2015, 2017, and 2018 due to their AQG publication record.

We obtained additional papers by examining the reference lists of, and the citations to, AQG papers we reviewed (known as “snowballing”). The citations to a paper were identified by searching for the paper using Google Scholar, then clicking on the “cited by” option that appears under the name of the paper. We performed this for every paper on AQG, regardless of whether we had decided to include it, to ensure that we captured all the relevant papers. That is to say, even if a paper was excluded because it met some of the exclusion criteria (1-3 and 8-13), it is still possible that it refers to, or is referred to by, relevant papers.

We used the reviews reported by Ch and Saha ( 2018 ) and Papasalouros and Chatzigiannakou ( 2018 ) as a “sanity check” to evaluate the comprehensiveness of our search strategy. We exported all the literature published between 2015 and 2018 included in the work of Ch and Saha ( 2018 ) and Papasalouros and Chatzigiannakou ( 2018 ) and checked whether they were included in our results (both search results and snowballing results).

Search Queries

We used the keywords “question” and “generation” to search for relevant papers. Actual search queries used for each of the databases are provided in the Appendix under “ Search Queries ”. We decided on these queries after experimenting with different combinations of keywords and operators provided by each database and looking at the ratio between relevant and irrelevant results in the first few pages (sorted by relevance). To ensure that recall was not compromised, we checked whether relevant results returned using different versions of each search query were still captured by the selected version.

The search results were exported to comma-separated values (CSV) files. Two reviewers then looked independently at the titles and abstracts to decide on inclusion or exclusion. The reviewers skimmed the paper if they were not able to make a decision based on the title and abstract. Note that, at this phase, it was not possible to assess whether all papers had satisfied the exclusion criteria 2, 3, 8, 9, and 10. Because of this, the final decision was made after reading the full text as described next.

To judge whether a paper’s purpose was related to education, we considered the title, abstract, introduction, and conclusion sections. Papers that mentioned many potential purposes for generating questions, but did not state which one was the focus, were excluded. If the paper mentioned only educational applications of QG, we assumed that its purpose was related to education, even without a clear purpose statement. Similarly, if the paper mentioned only one application, we assumed that was its focus.

Concerning evaluation, papers that evaluated the usability of a system that had a QG functionality, without evaluating the quality of generated questions, were excluded. In addition, in cases where we found multiple papers by the same author(s) reporting the same generation approach, even if some did not cover evaluation, all of the papers were included but counted as one study in our analyses.

Lastly, because the final decision on inclusion/exclusion sometimes changed after reading the full paper, agreement between the two reviewers was checked after the full paper had been read and the final decision had been made. However, a check was also made to ensure that the inclusion/exclusion criteria were interpreted in the same way. Cases of disagreement were resolved through discussion.

Data Extraction

Guided by the questions presented in the “ Review Objective ” section, we designed a specific data extraction form. Two reviewers independently extracted data related to the included studies. As mentioned above, different papers that related to the same study were represented as one entry. Agreement for data extraction was checked and cases of disagreement were discussed to reach a consensus.

Papers that had at least one shared author were grouped together if one of the following criteria were met:

they reported on different evaluations of the same generation approach;

they reported on applying the same generation approach to different sources or domains;

one of the papers introduced an additional feature of the generation approach such as difficulty prediction or generating distractors without changing the initial generation procedure.

The extracted data were analysed using a code written in R markdown. Footnote 3

Quality Assessment

Since one of the main objectives of this review is to identify the gold standard performance, we were interested in the quality of the evaluation approaches. To assess this, we used the criteria presented in Table  2 which were selected from existing checklists (Downs and Black 1998 ; Reisch et al. 1989 ; Critical Appraisal Skills Programme 2018 ), with some criteria being adapted to fit specific aspects of research on AQG. The quality assessment was conducted after reading a paper and filling in the data extraction form.

In what follows, we describe the individual criteria (Q1-Q9 presented in Table  2 ) that we considered when deciding if a study satisfied said criteria. Three responses are used when scoring the criteria: “yes”, “no” and “not specified”. The “not specified” response is used when either there is no information present to support the criteria, or when there is not enough information present to distinguish between a “yes” or “no” response.

Q1-Q4 are concerned with the quality of reporting on participant information, Q5-Q7 are concerned with the quality of reporting on the question samples, and Q8 and Q9 describe the evaluative measures used to assess the outcomes of the studies.

When a study reports the exact number of participants (e.g. experts, students, employees, etc.) used in the study, Q1 scores a “yes”. Otherwise, it scores a “no”. For example, the passage “20 students were recruited to participate in an exam …” would result in a “yes”, whereas “a group of students were recruited to participate in an exam …” would result in a “no”.

Q2 requires the reporting of demographic characteristics supporting the suitability of the participants for the task. Depending on the category of participant, relevant demographic information is required to score a “yes”. Studies that do not specify relevant information score a “no”. By means of examples, in studies relying on expert reviews, those that include information on teaching experience or the proficiency level of reviewers would receive a “yes”, while in studies relying on mock exams, those that include information about grade level or proficiency level of test takers would also receive a “yes”. Studies reporting that the evaluation was conducted by reviewers, instructors, students, or co-workers without providing any additional information about the suitability of the participants for the task would be considered neglectful of Q2 and score a “no”.

For a study to score “yes” for Q3, it must provide specific information on how participants were selected/recruited, otherwise it receives a score of “no”. This includes information on whether the participants were paid for their work or were volunteers. For example, the passage “7th grade biology students were recruited from a local school.” would receive a score of “no” because it is not clear whether or not they were paid for their work. However, a study that reports “Student volunteers were recruited from a local school …” or “Employees from company X were employed for n hours to take part in our study… they were rewarded for their services with Amazon vouchers worth $n” would receive a “yes”.

To score “yes” for Q4, two conditions must be met: the study must 1) score “yes” for both Q2 and Q3 and 2) only use participants that are suitable for the task at hand. Studies that fail to meet the first condition score “not specified” while those that fail to meet the second condition score “no”. Regarding the suitability of participants, we consider, as an example, native Chinese speakers suitable for evaluating the correctness and plausibility of options generated for Chinese gap-fill questions. As another example, we consider Amazon Mechanical Turk (AMT) co-workers unsuitable for evaluating the difficulty of domain-specific questions (e.g. mathematical questions).

When a study reports the exact number of questions used in the experimentation or evaluation stage, Q5 receives a score of “yes”, otherwise it receives a score of “no”. To demonstrate, consider the following examples. A study reporting “25 of the 100 generated questions were used in our evaluation …” would receive a score of “yes”. However, if a study made a claim such as “Around half of the generated questions were used …”, it would receive a score of “no”.

Q6a requires that the sampling strategy be not only reported (e.g. random, proportionate stratification, disproportionate stratification, etc.) but also justified to receive a “yes”, otherwise, it receives a score of “no”. To demonstrate, if a study only reports that “We sampled 20 questions from each template …” would receive a score of “no” since no justification as to why the stratified sampling procedure was used is provided. However, if it was to also add “We sampled 20 questions from each template to ensure template balance in discussions about the quality of generated questions …” then this would be considered as a suitable justification and would warrant a score of “yes”. Similarly, Q6b requires that the sample size be both reported and justified.

Our decision regarding Q7 takes into account the following: 1) responses to Q6a (i.e. a study can only score “yes” if the score to Q6a is “yes”, otherwise, the score would be “not specified”) and 2) representativeness of the population. Using random sampling is, in most cases, sufficient to score “yes” for Q7. However, if multiple types of questions are generated (e.g. different templates or different difficulty levels), stratified sampling is more appropriate in cases in which the distribution of questions is skewed.

Q8 considers whether the authors provide a description, a definition, or a mathematical formula for the evaluation measures they used as well as a description of the coding system (if applicable). If so, then the study receives a score of “yes” for Q8, otherwise it receives a score of “no”.

Q9 is concerned with whether questions were evaluated by multiple reviewers and whether measures of the agreement (e.g., Cohen’s kappa or percentage of agreement) were reported. For example, studies reporting information similar to “all questions were double-rated and inter-rater agreement was computed…” receive a score of “yes”, whereas studies reporting information similar to “Each question was rated by one reviewer…” receive a score of “no” .

To assess inter-rater reliability, this activity was performed by two reviewers (the first and second authors), who are proficient in the field of AQG, independently on an exploratory random sample of 27 studies. Footnote 4 The percentage of agreement and Cohen’s kappa were used to measure inter-rater reliability for Q1-Q9. The percentage of agreement ranged from 73% to 100%, while Cohen’s kappa was above .72 for Q1-Q5, demonstrating “substantial to almost perfect agreement”, and equal to 0.42 for Q9, Footnote 5

Results and Discussion

Search and screening results.

Searching the databases and AIED resulted in 2,012 papers and we checked 974. Footnote 7 The difference is due to ACM which provided 1,265 results and we only checked the first 200 results (sorted by relevance) because we found that subsequent results became irrelevant. Out of the search results, 122 papers were considered relevant after looking at their titles and abstracts. After removing duplicates, 89 papers remained. This set was further reduced to 36 papers after reading the full text of the papers. Checking related work sections and the reference lists identified 169 further papers (after removing duplicates). After we read their full texts, we found 46 to satisfy our inclusion criteria. Among those 46, 15 were captured by the initial search. Tracking citations using Google Scholar provided 204 papers (after removing duplicates). After reading their full text, 49 were found to satisfy our inclusion criteria. Among those 49, 14 were captured by the initial search. The search results are outlined in Table  3 . The final number of included papers was 93 (72 studies after grouping papers as described before). In total, the database search identified 36 papers while the other sources identified 57. Although the number of papers identified through other sources was large, many of them were variants of papers already included in the review.

The most common reasons for excluding papers on AQG were that the purpose of the generation was not related to education or there was no evaluation. Details of papers that were excluded after reading their full text are in the Appendix under “ Excluded Studies ”.

Data Extraction Results

In this section, we provide our results and outline commonalities and differences with Alsubait’s results (highlighted in the “ Findings of Alsubait’s Review ” section). The results are presented in the same order as our research questions. The main characteristics of the reviewed literature can be found in the Appendix under “ Summary of Included Studies ”.

Rate of Publication

The distribution of publications by year is presented in Fig.  1 . Putting this together with the results reported by Alsubait ( 2015 ), we notice a strong increase in publication starting from 2011. We also note that there were three workshops on QG Footnote 8 in 2008, 2009, and 2010, respectively, with one being accompanied by a shared task (Rus et al. 2012 ). We speculate that the increase starting from 2011 is because workshops on QG have drawn researchers’ attention to the field, although the participation rate in the shared task was low (only five groups participated). The increase also coincides with the rise of MOOCs and the launch of major MOOC providers (Udacity, Udemy, Coursera and edX, which all started up in 2012 (Baturay 2015 )) which provides another reason for the increasing interest in AQG. This interest was further boosted from 2015. In addition to the above speculations, it is important to mention that QG is closely related to other areas such as NLP and the Semantic Web. Being more mature and providing methods and tools that perform well have had an effect on the quantity and quality of research in QG. Note that these results are only related to question generation studies that focus on educational purposes and that there is a large volume of studies investigating question generation for other applications as mentioned in the “ Search and Screening Results ” section.

figure 1

Publications per year

Types of Papers and Publication Venues

Of the papers published in the period covered by this review, conference papers constitute the majority (44 papers), followed by journal articles (32 papers) and workshop papers (17 papers). This is similar to the results of Alsubait ( 2015 ) with 34 conference papers, 22 journal papers, 13 workshop papers, and 12 other types of papers, including books or book chapters as well as technical reports and theses. In the Appendix, under “ Publication Venues ”, we list journals, conferences, and workshops that published at least two of the papers included in either of the reviews.

Research Groups

Overall, 358 researchers are working in the area (168 identified in Alsubait’s review and 205 identified in this review with 15 researchers in common). The majority of researchers have only one publication. In Appendix “ Active Research Groups ”, we present the 13 active groups defined as having more than two publications in the period of both reviews. Of the 174 papers identified in both reviews, 64 were published by these groups. This shows that, besides the increased activities in the study of AQG, the community is also growing.

Purpose of Question Generation

Similar to the results of Alsubait’s review (Table  1 ), the main purpose of generating questions is to use them as assessment instruments (Table  4 ). Questions are also generated for other purposes, such as to be employed in tutoring or self-assisted learning systems. Generated questions are still used in experimental settings and only Zavala and Mendoza ( 2018 ) have reported their use in a class setting, in which the generator is used to generate quizzes for several courses and to generate assignments for students.

Generation Methods

Methods of generating questions have been classified in the literature (Yao et al. 2012 ) as follows: 1) syntax-based, 2) semantic-based, and 3) template-based. Syntax-based approaches operate on the syntax of the input (e.g. syntactic tree of text) to generate questions. Semantic-based approaches operate on a deeper level (e.g. is-a or other semantic relations). Template-based approaches use templates consisting of fixed text and some placeholders that are populated from the input. Alsubait ( 2015 ) extended this classification to include two more categories: 4) rule-based and 5) schema-based. The main characteristic of rule-based approaches, as defined by Alsubait ( 2015 ), is the use of rule-based knowledge sources to generate questions that assess understanding of the important rules of the domain. As this definition implies that these methods require a deep understanding (beyond syntactic understanding), we believe that this category falls under the semantic-based category. However, we define the rule-based approach differently, as will be seen below. Regarding the fifth category, according to Alsubait ( 2015 ), schemas are similar to templates but are more abstract. They provide a grouping of templates that represent variants of the same problem. We regard this distinction between template and schema as unclear. Therefore, we restrict our classification to the template-based category regardless of how abstract the templates are.

In what follows, we extend and re-organise the classification proposed by Yao et al. ( 2012 ) and extended by Alsubait ( 2015 ). This is due to our belief that there are two relevant dimensions that are not captured by the existing classification of different generation approaches: 1) the level of understanding of the input required by the generation approach and 2) the procedure for transforming the input into questions. We describe our new classification, characterise each category and give examples of features that we have used to place a method within these categories. Note that these categories are not mutually exclusive.

Level of understanding

Syntactic: Syntax-based approaches leverage syntactic features of the input, such as POS or parse-tree dependency relations, to guide question generation. These approaches do not require understanding of the semantics of the input in use (i.e. entities and their meaning). For example, approaches that select distractors based on their POS are classified as syntax-based.

Semantic: Semantic-based approaches require a deeper understanding of the input, beyond lexical and syntactic understanding. The information that these approaches use are not necessarily explicit in the input (i.e. they may require reasoning to be extracted). In most cases, this requires the use of additional knowledge sources (e.g., taxonomies, ontologies, or other such sources). As an example, approaches that use either contextual similarity or feature-based similarity to select distractors are classified as being semantic-based.

Procedure of transformation

Template: Questions are generated with the use of templates. Templates define the surface structure of the questions using fixed text and placeholders that are substituted with values to generate questions. Templates also specify the features of the entities (either syntactic, semantic, or both), that can replace the placeholders.

Rule: Questions are generated with the use of rules. Rules often accompany approaches using text as input. Typically, approaches utilising rules annotate sentences with syntactic and/or semantic information. They then use these annotations to match the input to a pattern specified in the rules. These rules specify how to select a suitable question type (e.g. selecting suitable wh-words) and how to manipulate the input to construct questions (e.g. converting sentences into questions).

Statistical methods: This is where question transformation is learned from training data. For example, in Gao et al. ( 2018 ), question generation has been dealt with as a sequence-to-sequence prediction problem in which, given a segment of text (usually a sentence), the question generator forms a sequence of text representing a question (using the probabilities of co-occurrence that are learned from the training data). Training data has also been used in Kumar et al. ( 2015b ) for predicting which word(s) in the input sentence is/are to be replaced by a gap (in gap-fill questions).

Regarding the level of understanding, 60 papers rely on semantic information and only ten approaches rely only on syntactic information. All except three of the ten syntactic approaches (Das and Majumder 2017 ; Kaur and Singh 2017 ; Kusuma and Alhamri 2018 ) tackle the generation of language questions. In addition, templates are more popular than rules and statistical methods, with 27 papers reporting the use of templates, compared to 16 and nine for rules and statistical methods, respectively. Each of these three approaches has its advantages and disadvantages. In terms of cost, all three approaches are considered expensive. Templates and rules require manual construction, while learning from data often requires a large amount of annotated data which is unavailable in many specific domains. Additionally, questions generated by rules and statistical methods are very similar to the input (e.g. sentences used for generation), while templates allow the generating of questions that differ from the surface structure of the input, in the use of words for example. However, questions generated from templates are limited in terms of their linguistic diversity. Note that some of the papers were classified as not having a method of transforming the input into questions because they only focused on distractor generation or gap-fill questions for which the stem is the same input statement with a word or a phrase being removed. Readers interested in studies that belong to a specific approach are referred to the “ Summary of Included Studies ” in the Appendix.

Generation Tasks

Tasks involved in question generation are explained below. We grouped the tasks into the stages of preprocessing, question construction, and post-processing. For each task, we provide a brief description, mention its role in the generation process, and summarise different approaches that have been applied in the literature. The “ Summary of Included Studies ” in the Appendix shows which tasks have been tackled in each study.

Preprocessing

Two types of preprocessing are involved: 1) standard preprocessing and 2) QG-specific preprocessing. Standard preprocessing is common to various NLP tasks and is used to prepare the input for upcoming tasks; it involves segmentation, sentence splitting, tokenisation, POS tagging, and coreference resolution. In some cases, it also involves named entity recognition (NER) and relation extraction (RE). The aim of QG-specific preprocessing is to make or select inputs that are more suitable for generating questions. In the reviewed literature, three types of QG-specific preprocessing are employed:

Sentence simplification: This is employed in some text-based approaches (Liu et al. 2017 ; Majumder and Saha 2015 ; Patra and Saha 2018b ). Complex sentences, usually sentences with appositions or sentences joined with conjunctions, are converted into simple sentences to ease upcoming tasks. For example, Patra and Saha ( 2018b ) reported that Wikipedia sentences are long and contain multiple objects; simplifying these sentences facilitates triplet extraction (where triples are used later for generating questions). This task was carried out by using sentence simplification rules (Liu et al. 2017 ) and relying on parse-tree dependencies (Majumder and Saha 2015 ; Patra and Saha 2018b ).

Sentence classification: In this task, sentences are classified into categories, which is, according to Mazidi and Tarau ( 2016a ) and Mazidi and Tarau ( 2016b ), a key to determining the type of question to be asked about the sentence. This classification was carried out by analysing POS and dependency labels, as in Mazidi and Tarau ( 2016a ) and Mazidi and Tarau ( 2016b ) or by using a machine learning (ML) model and a set of rules, as in Basuki and Kusuma ( 2018 ). For example, in Mazidi and Tarau( 2016a , 2016b ), the pattern “S-V-acomp” is an adjectival complement that describes the subject and is therefore matched to the question template “Indicate properties or characteristics of S?”

Content selection: As the number of questions in examinations is limited, the goal of this task is to determine important content, such as sentences, parts of sentences, or concepts, about which to generate questions. In the reviewed literature, the majority approach is to generate all possible questions and leave the task of selecting important questions to exam designers. However, in some settings such as self-assessment and self-learning environments, in which questions are generated “on the fly”, leaving the selection to exam designers is not feasible.

Content selection was of interest for those approaches that utilise text more than for those that utilise structured knowledge sources. Several characterisations of important sentences and approaches for their selection have been proposed in the reviewed literature which we summarise in the following paragraphs.

Huang and He ( 2016 ) defined three characteristics for selecting sentences that are important for reading assessment and propose metrics for their measurement: keyness (containing the key meaning of the text), completeness (spreading over different paragraphs to ensure that test-takers grasp the text fully), and independence (covering different aspects of text content). Olney et al. ( 2017 ) selected sentences that: 1) are well connected to the discourse (same as completeness) and 2) contain specific discourse relations. Other researchers have focused on selecting topically important sentences. To that end, Kumar et al. ( 2015b ) selected sentences that contain concepts and topics from an educational textbook, while Kumar et al. ( 2015a ) and Majumder and Saha ( 2015 ) used topic modelling to identify topics and then rank sentences based on topic distribution. Park et al. ( 2018 ) took another approach by projecting the input document and sentences within it into the same n-dimensional vector space and then selecting sentences that are similar to the document, assuming that such sentences best express the topic or the essence of the document. Other approaches selected sentences by checking the occurrence of, or measuring the similarity to, a reference set of patterns under the assumption that these sentences convey similar information to sentences used to extract patterns (Majumder and Saha 2015 ; Das and Majumder 2017 ). Others (Shah et al. 2017 ; Zhang and Takuma 2015 ) filtered sentences that are insufficient on their own to make valid questions, such as sentences starting with discourse connectives (e.g. thus, also, so, etc.) as in Majumder and Saha ( 2015 ).

Still other approaches to content selection are more specific and are informed by the type of question to be generated. For example, the purpose of the study reported in Susanti et al. ( 2015 ) is to generate “closest-in-meaning vocabulary questions” Footnote 9 which involve selecting a text snippet from the Internet that contains the target word, while making sure that the word has the same sense in both the input and retrieved sentences. To this end, the retrieved text was scored on the basis of metrics such as the number of query words that appear in the text.

With regard to content selection from structured knowledge bases, only one study focuses on this task. Rocha and Zucker ( 2018 ) used DBpedia to generate questions along with external ontologies; the ontologies describe educational standards according to which DBpedia content was selected for use in question generation.

Question Construction

This is the main task and involves different processes based on the type of questions to be generated and their response format. Note that some studies only focus on generating partial questions (only stem or distractors). The processes involved in question construction are as follows:

Stem and correct answer generation: These two processes are often carried out together, using templates, rules, or statistical methods, as mentioned in the “ Generation Methods ” Section. Subprocesses involved are:

transforming assertive sentences into interrogative ones (when the input is text);

determination of question type (i.e. selecting suitable wh-word or template); and

selection of gap position (relevant to gap-fill questions).

universe84a

Systematic review and meta-analysis mcqs: multiple choice questions related to systematic review and meta-analysis theory, guidelines and softwares.

Systematic Review and Meta-Analysis MCQs: Multiple Choice Questions related to Systematic Review and Meta-Analysis Theory, Guidelines and Softwares

Systematic Review and Meta-Analysis MCQs

Systematic Review and Meta-Analysis MCQs: It is a  group of Multiple Choice Questions (MCQs) related to Systematic Review and Meta-Analysis Theory, Guidelines, and Softwares. Test your knowledge in Systematic Review and Meta-Analysis MCQs, in Theory, Guidelines, and Softwares of Systematic Review and Meta-Analysis by playing the “Systematic Review and Meta-Analysis MCQs” and raise your marks in examinations as well as elsewhere you need. There are multiple-choice questions (MCQs) . A question with four choices among them one right answer you have to chose.

A 20 MCQs set of Systematic Review and Meta-Analysis MCQs

  • Which of the following is not always required in systematic review?
  • Protocol development
  • Search strategy
  • Involvement of more than one author
  • Meta-analysis

Correct answer: Meta-analysis

  • A systematic review of evidence from qualitative studies is also known as a meta-analysis.
  • None of them

Correct answer: False

  • Which of the steps are included in the systematic review?
  • Formulate a question and develop a protocol
  • Conduct search
  • Select studies, assess study quality, and extract data
  • All of the above

Correct answer: All of the above

  • Where do we register the protocol of systematic review that will be conducted at the national level?
  •  Health Research Council of country
  • ClinicalTrial.gov

Correct answer: PROSPERO

  • What does “S” stand for in PICOS?
  • Systematic review

Correct answer: Study

  • Which is not the effect size based on continuous variables?
  • Absolute mean difference
  • Standardized mean difference
  • Response ratio

Correct answer: Odds ratio

  • A forest plot displays effect estimates and confidence intervals for both individual studies and meta-analyses. *

Correct answer: True

  • Which is/are the advantage/s of the meta-analyses?
  • To improve precision
  • To answer questions not posed by the individual studies
  • To settle controversies arising from apparently conflicting studies or to generate new hypotheses
  • In the inverse-variance method, larger studies are given more weight than smaller studies.
  • Which of the following method is done with the inverse-variance method?
  • Fixed-effect method for meta-analysis
  • Random-effects methods for meta-analysis

Correct answer: Both

  • Meta-regressions are similar in essence to simple regressions, in which an outcome variable is predicted according to the values of one or more explanatory variables.

Correct answer-True

  • Which of the following review does not necessarily require methodological quality assessment of included studies?
  • Narrative review
  • Scoping review
  • Both b and c

Correct answer: Both b and c

  • Which of the following is not related to selection bias?
  • Random sequence generation
  • Allocation concealment

Correct answer: Attrition

  • Which is true about GRADE?
  • A framework for developing and presenting summaries of evidence and providing a systematic approach for making clinical practice recommendations
  • Tool for grading the quality of evidence and for making recommendations
  • A reproducible and transparent framework for grading certainty in evidence
  • Which of the following checklist is used to report the systematic review?

Correct answer: PRISMA

  • Which of the following is the most rigorous and methodologically complex kind of review article?
  • Which of the following study provided the most robust evidence?
  • Randomized clinical trails
  • Cohort study
  • Cross-sectional analytical study
  • Systematic review and meta-analysis

Correct answer: Systematic review and meta-analysis

  • Steps in a meta-analysis include all of the following except:
  • Abstraction
  • Randomization

Correct answer: Randomization

  • A fixed-effects model is most appropriate in a meta-analysis when study findings are
  • Heterogenous
  • Either homogenous or heterogenous
  • Neither homogenous nor heterogenous

Correct answer: Homogenous

  • What is the full form of PRISMA?
  • Providing Results for Systematic Review and Meta-Analyses
  • Preferred Reporting Items for Systematic Reviews and Meta-Analyses
  • Providing Reporting Items for Systematic Reviews and Meta-Analyses
  • Provisional Reporting Items for Systematic Reviews and Meta-Analyses

Correct answer: Preferred Reporting Items for Systematic Reviews and Meta-Analyses

© 2024 Universe84a.com | All Rights Reserved

The comparison between effects of Taichi and conventional exercise on functional mobility and balance in healthy older adults: a systematic literature review and meta-analysis

Affiliations.

  • 1 School of Sport Medicine and Physical Therapy, Beijing Sport University, Beijing, China.
  • 2 Sports Coaching College, Beijing Sport University, Beijing, China.
  • 3 College of Physical Education and Health Science, Chongqing Normal University, Chongqing, China.
  • 4 Hebrew Senior Life Hinda and Arthur Marcus Institute for Aging Research, Harvard Medical School, Boston, MA, United States.
  • 5 China Institute of Sport and Health Science, Beijing Sport University, Beijing, China.
  • PMID: 38164444
  • PMCID: PMC10757983
  • DOI: 10.3389/fpubh.2023.1281144

Background: Taichi is beneficial for functional mobility and balance in older adults. However, such benefits of Taichi when comparing to conventional exercise (CE) are not well understood due to large variance in study protocols and observations.

Methods: We reviewed publications in five databases. Eligible studies that examined the effects of Taichi on the outcomes of functional mobility and balance in healthy older adults as compared to CE were included. Subgroup analyses compared the effects of different types of CE (e.g., single and multiple-type exercise) and different intervention designs (e.g., Taichi types) on those outcomes (Registration number: CRD42022331956).

Results: Twelve studies consisting of 2,901 participants were included. Generally, compared to CE, Taichi induced greater improvements in the performance of Timed-Up-and-Go (SMD = -0.18, [-0.33 to -0.03], p = 0.040, I 2 = 59.57%), 50-foot walking (MD = -1.84 s, [-2.62 to -1.07], p < 0.001, I 2 = 0%), one-leg stance with eyes open (MD = 6.00s, [2.97 to 9.02], p < 0.001, I 2 = 83.19%), one-leg stance with eyes closed (MD = 1.65 s, [1.35 to 1.96], p < 0.001, I 2 = 36.2%), and functional reach (SMD = 0.7, [0.32 to 1.08], p < 0.001, I 2 = 86.79%) tests. Subgroup analyses revealed that Taichi with relatively short duration (<20 weeks), low total time (≤24 h), and/or using Yang-style, can induce significantly greater benefits for functional mobility and balance as compared to CE. Uniquely, Taichi only induced significantly greater improvements in Timed-Up-and-Go compared to single- (SMD = -0.40, [-0.55 to -0.24], p < 0.001, I 2 = 6.14%), but not multiple-type exercise. A significant difference between the effects of Taichi was observed on the performance of one-leg stance with eyes open when compared to CE without balance (MD = 3.63 s, [1.02 to 6.24], p = 0.006, I 2 = 74.93%) and CE with balance (MD = 13.90s, [10.32 to 17.48], p < 0.001, I 2 = 6.1%). No other significant difference was shown between the influences of different CE types on the observations.

Conclusion: Taichi can induce greater improvement in functional mobility and balance in older adults compared to CE in a more efficient fashion, especially compared to single-type CE. Future studies with more rigorous design are needed to confirm the observations here.

Keywords: Taichi; balance; exercise prescription; functional mobility; older adults; protocol design; rehabilitative programs.

Copyright © 2023 Li, Liu, Zhou, Dong, Manor, Bao and Zhou.

Publication types

  • Meta-Analysis
  • Systematic Review
  • Exercise Therapy / methods
  • Health Status
  • Lower Extremity
  • Postural Balance*

Grants and funding

IMAGES

  1. Research Methodology MCQS

    a systematic literature review is mcq

  2. Systematic Literature Review

    a systematic literature review is mcq

  3. Systematic literature review phases.

    a systematic literature review is mcq

  4. Stages of a Systematic Review.

    a systematic literature review is mcq

  5. 10 Steps to Write a Systematic Literature Review Paper in 2023

    a systematic literature review is mcq

  6. Systematic Literature Review Methodology

    a systematic literature review is mcq

VIDEO

  1. Review of literature

  2. Introduction to Systematic Literature Review by Dr. K. G. Priyashantha

  3. SYSTEMATIC AND LITERATURE REVIEWS

  4. Systematic Literature Review, by Prof. Ranjit Singh, IIIT Allahabad

  5. Systematic Literature Review Paper

  6. Effective Review of Literature

COMMENTS

  1. Chapter 4: Multiple choice questions

    Chapter 4: Multiple choice questions. Instructions. ... A systematic literature review is: a) one which generates a literature review using a treasure hunt system. b) ... The whole point of a literature review is to show what we have read and what we thought about it. d) ...

  2. Top 20 MCQs on literature review with answers

    MCQs on literature review: The primary purpose of literature review is to facilitate detailed background of the previous studies to the readers on the topic of research.. In this blog post, we have published 20 MCQs on Literature Review (Literature Review in Research) with answers.. 20 Multiple Choice Questions on Literature Review. 1. Literature is a ...

  3. Multiple Choice Quiz

    Multiple Choice Quiz. Systematic reviews have all of the following in common EXCEPT: a. Systematic reviews are focused on a single topic. b. The author (s) of a systematic review must document the criteria used to include or exclude studies from the review. c.

  4. PDF SYSTEMATIC REVIEW Multiple choice question writing and ...

    systematic literature review [version 1; peer review: awaiting peer review] Jacob Lahti 1,2, Miriam Salamon3, Jordan Farhat3, Thomas Varkey 1,3 ... Multiple-choice question (MCQ) tests have been a ...

  5. Introduction to systematic review and meta-analysis

    A systematic review collects all possible studies related to a given topic and design, and reviews and analyzes their results [ 1 ]. During the systematic review process, the quality of studies is evaluated, and a statistical meta-analysis of the study results is conducted on the basis of their quality. A meta-analysis is a valid, objective ...

  6. How-to conduct a systematic literature review: A quick guide for

    Method details Overview. A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12].An SLR updates the reader with current literature about a subject [6].The goal is to review critical points of current knowledge on a ...

  7. Systematic Review

    A review is an overview of the research that's already been completed on a topic. What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias. The methods are repeatable, and the approach is formal and systematic: Formulate a research question. Develop a protocol.

  8. Guidance on Conducting a Systematic Literature Review

    Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...

  9. PDF Systematic Literature Reviews: an Introduction

    Systematic literature reviews (SRs) are a way of synthesising scientific evidence to answer a particular research question in a way that is transparent and reproducible, while seeking to include all published evidence on the topic and appraising the quality of th is evidence. SRs have become a major methodology

  10. How to Conduct a Systematic Review: A Narrative Literature Review

    Writing a research question is the first step in conducting a systematic review and is of paramount importance as it outlines both the need and validity of systematic reviews (Nguyen, et al., unpublished data). It also increases the efficiency of the review by limiting the time and cost of identifying and obtaining relevant literature [ 11 ].

  11. Multiple choice question writing and medical students: a systematic

    Multiple-choice questions (MCQs) are widely used in medical education and can promote surface learning strategies, but creating MCQs requires both in-depth content knowledge and sophisticated ...

  12. Does developing multiple-choice Questions Improve Medical Students

    Multiple-choice questions (MCQs) represent the most common assessment tool in medical ... we conducted a systematic review to define and document evidence of the effect of writing MCQs activity on students learning, and understand how and under what circumstances it could benefit medical students, as to our knowledge, there is no prior ...

  13. How to Do a Systematic Review: A Best Practice Guide for Conducting and

    The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information.

  14. Research Methodology MCQS

    Research Methodology MCQS writing literature review requires: planning clear writing research is: lab experiment report good writing all of the above systematic ... In systematic sampling, population is 200 and selected sample size is 50 then sampling interval is; a) 250 b) 0. ... A literature review is based on the assumption that: a) Copy ...

  15. How to write a systematic literature review [9 steps]

    Screen the literature. Assess the quality of the studies. Extract the data. Analyze the results. Interpret and present the results. 1. Decide on your team. When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis.

  16. Multiple choice question writing and medical students: a systematic

    This paper looks at the current body of literature written since 2018, utilizing the well-established PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) method3 to answer the research question, "what are the effects of the use of multiple-choice question writing on the performance of medical students on multiple ...

  17. Understanding Research & Publication Ethics through MCQ

    Research and Publication Ethics: In this blog post (MCQ on Research & Publication Ethics), you will understand Research ethics through various MCQs.. Topics Covered: Literature Review, Review Paper Writing, Writing a Research article, etc;. MCQs on Research & Publication Ethics. 1. In the Literature review, systematic reorganization and reshuffling of the information to develop knowledge ...

  18. Systematic Reviews: Before you begin...

    Before you begin your review... Please be assured that your Subject Librarian will support you as best they can. Subject Librarians are able to show QUB students and staff undertaking any type of literature search (e.g. literature review, scoping review, systematic review) how to: Structure searches using AND/OR. Select appropriate databases.

  19. Literature Review MCQ [Free PDF]

    Get Literature Review Multiple Choice Questions (MCQ Quiz) with answers and detailed solutions. Download these Free Literature Review MCQ Quiz Pdf and prepare for your upcoming exams Like Banking, SSC, Railway, UPSC, State PSC. ... Thus, A systematic review of the literature adopts explicit procedures is the correct answer.. ...

  20. A Systematic Review of Automatic Question Generation for Educational

    This review extends a previous systematic review on AQG (Alsubait 2015), which covers the literature up to the end of 2014. Given the large amount of research that has been published since Alsubait's review was conducted (93 papers over a four year period compared to 81 papers over the preceding 45-year period), an extension of Alsubait's ...

  21. Purpose of Literature Review MCQ Quiz

    Get Purpose of Literature Review Multiple Choice Questions (MCQ Quiz) with answers and detailed solutions. Download these Free Purpose of Literature Review MCQ Quiz Pdf and prepare for your upcoming exams Like Banking, SSC, Railway, UPSC, State PSC. ... Thus, A systematic review of the literature adopts explicit procedures is the correct answer

  22. A systematic literature review is:

    Solved Answer of MCQ A systematic literature review is: - (a) One which starts in your own library, then goes to on-line databases and, finally, to the internet - (b) A replicable, scientific and transparent process - (c) One which gives equal attention to the principal contributors to the area - (d) A responsible, professional process of time-management for research - Getting started ...

  23. Systematic Review and Meta-Analysis MCQs

    A 20 MCQs set of Systematic Review and Meta-Analysis MCQs. Which of the following is not always required in systematic review? Correct answer: Meta-analysis. A systematic review of evidence from qualitative studies is also known as a meta-analysis. Correct answer: False.

  24. The comparison between effects of Taichi and conventional ...

    The comparison between effects of Taichi and conventional exercise on functional mobility and balance in healthy older adults: a systematic literature review and meta-analysis Front Public Health . 2023 Dec 18:11:1281144. doi: 10.3389/fpubh.2023.1281144.