U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • HHS Author Manuscripts

Logo of nihpa

Design and Analysis of Multiple Events Case-Control Studies

Wenguang sun.

Department of Statistics, North Carolina State University, Raleigh, NC 27695, USA

Marshall M. Joffe

Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, PA 19104, USA

Steven M. Brunelli

Renal-Electrolyte and Hypertension Division, University of Pennsylvania, Philadelphia, PA 19104, USA

Associated Data

In case-control research where there are multiple case groups, standard analyses fail to make use of all available information. Multiple events case-control (MECC) studies provide a new approach to sampling from a cohort and are useful when it is desired to study multiple types of events in the cohort. In this design, subjects in the cohort who develop any event of interest are sampled, as well as a fraction of the remaining subjects. We show that a simple case-control analysis of data arising from MECC studies is biased and develop three general estimating-equation based approaches to analyzing data from these studies. We conduct simulation studies to compare the efficiency of the various MECC analyses with each other and with the corresponding conventional analyses. It is shown that the gain in efficiency by using the new design is substantial in many situations. We demonstrate the application of our approach to a nested case-control study of the effect of oral sodium phosphate use on chronic kidney injury with multiple case definitions.

1. Introduction

In large-scale cohort studies, investigators are often interested in assessing risk factors for several clinical endpoints. For example, the female textile workers cohort study ( Ray et al. 2007 ; Astrakianakis et al. 2007 ; Li et al. 2006 ) investigates occupational risk factors for several cancers observed in the cotton textile industry, including breast cancer, lung cancer, pancreatic cancer and liver cancer. The Nurses Health Study (NHS, Stampfer et al. 1985 ) investigates risk factors for several major chronic diseases in women including cancer, diabetes and cardiovascular disease. The chronic renal insufficiency cohort study (CRIC, Feldman et al 2003 ) investigates risk factors for the cardiovascular disease and the end-stage renal disease in a clinical population with chronic renal insufficiency. These endpoints will occur only in a minority of the study subjects over the course of follow-up. Due to the limited study budget, it is cost effective to obtain covariate histories on subjects who have developed outcomes of interest, and a subset of the remaining subjects.

In case-control research, investigators often wish to reuse a control group that has been used for studying associations with one outcome for studying associations with other outcomes. Reusing a single control group for multiple case groups can lead to savings in cost. However, a control group that is appropriate for one case group may not be appropriate for a different one. The case-cohort design circumvents this problem by selecting as a referent group a random sample of the entire underlying cohort; this random sample can then be used with any or multiple case groups ( Kupper, McMichael and Spirtas 1975 ; Prentice 1986 ; Rothman, Greenland, and Lash 2008 ). However, when there are multiple case groups, standard analyses of case-cohort studies fail to make use of all available information. In particular, subjects who are not selected for the referent group but who develop one of the outcomes of interest are ignored in the analysis of other outcomes.

To gain further insight into design and analysis of these studies, it is productive to consider the above paradigm as a distinct design, where sampling from the cohort is based not only on the outcome of interest (case status) for a particular analysis but also on events/outcomes auxiliary for that analysis. We term these studies multiple-events case-control (MECC) studies. In the MECC design, there is a known and enumerated cohort, from which subjects are selected for additional measurements; thus, the sampling proportions are known or can be estimated from the data.

The work on sampling from a cohort (e.g., Prentice 1986 ; Langholz and Thomas 1990 ) demonstrates that proper design and analysis for case-control and case-cohort studies can yield consistent estimates of the same measure of association in the underlying cohort studies. The MECC study can be conceptualized in this framework as well. However, this framework by itself can fail to use information available on the subjects not sampled. The missing data perspective on these studies supplements the cohort sampling one and allows use of data on all subjects in the cohort. For exposition of analytic methods, we adopt this view, and so consider and develop estimators based on the semi-parametric estimation theory by Robins, Rotnitzky and Zhao (1994) .

The main contributions of this paper include the following: a. We propose a new unified sampling and analysis framework when multiple event types are sampled from a cohort. b. The difficulty of implementing the semi-parametric estimation theory of Robins et al. (1994) is well known; we provide an accessible illustration on how to apply the theory to a new situation. The software is also made available. c. We compare our analytic methods taking appropriate account of the design with conventional analytic alternatives using simulations. We show that in many situations, the gain in efficiency and reduction in bias can be substantial by using the new design. d. We evaluate the performance of different estimators, including the weighted, weighted-augmented and semi-parametric efficient estimators, in the new design and give practical recommendations. e. The comparison (of different estimators) in Robins et al. (1994) considered a random sampling scheme from the underlying cohort, and only the efficiency for estimating the effect of the missing covariate was investigated. We consider a comparison using different sampling methods (nested case-control and MECC sampling from the underlying cohort), and extend the previous comparison of analytic approaches by including the always observed covariate as well.

The paper is organized as follows. In Section 2, we introduce the MECC design in the frameworks of sampling from a cohort and of missing data. Three estimating-equation based approaches to analyzing the MECC data are described in Section 3. Simulation studies are conducted in Section 4 to compare the efficiency of the three analytical approaches, as well as the efficiency of the MECC analyses versus conventional analyses. In Section 5, we illustrate the use of MECC analysis in a nested case-control study with multiple case definitions. The article concludes with a discussion of the results and some open problems. Derivations and additional simulation results are given in the web supplementary materials .

2. The Conceptual Framework for MECC Studies

Sampling from a cohort has long been applied in epidemiologic studies; the best known examples are case-control and case-cohort study designs ( Breslow and Day 1980 ; Prentice 1986 ; Rothman, Greenland, and Lash 2008 ). These designs allow cost effective inference for most or all population-level parameters of interest. Our newly proposed MECC study design also fits into this framework. This section briefly reviews the literature on sampling from a cohort and the missing-data view of this problem, then formulates the MECC design within these frameworks. Our subsequent treatment assumes that all outcomes are binary.

Let Y be the outcome of interest and S the auxiliary outcome/event. We are interested in studying associations between a collection of covariates X * = ( X , V ) and outcome Y, where X is the collection of covariates that are only available for a subset of the cohort and V the collection of covariates that are measured for the entire cohort. Denote by W = ( V , Y , S ) the data that are always observed.

We consider here simple versions of the case-cohort and nested case-control designs (NCC, here we refer to the unmatched case-control study with controls selected from event-free subjects) in which the outcome is a binary indicator rather than a censored failure-time outcome. The case-cohort and nested case-control designs apply to both survival analysis and logistic regression analysis (see Rothman et al. 2008 ), although they were originally proposed for and are better known as methods for survival data.

In the case-cohort design, a sample or subcohort of the entire cohort is chosen to obtain information on X . Let δ CC = 1 if a subject is in the subcohort, and let π V , C C ∗ = P ( δ C C = 1 ∣ V ) denote the probability of being selected into the subcohort; this probability may be a function of the always-measured baseline covariates V, but not of the sometimes unmeasured covariate X , nor of post-treatment variables Y and S . In a case-cohort study, information on X is also obtained on all subjects with Y = 1. Let Δ CC be an indicator of whether information on X is obtained in the simple case-cohort design. In this design, Δ CC = I ( δ CC = 1 or Y = 1), and the probability of inclusion in the study π V , Y , C C = P ( Δ C C = 1 ∣ X , V , S , Y ) = P ( Δ C C = 1 ∣ V , Y ) = Y + ( 1 − Y ) π V , C C ∗ . In nested case-control (NCC) studies, control subjects are selected from among the noncases; let Δ NCC indicate whether covariates X are measured for a given subject. Here, we have π V , Y , NCC = P ( Δ NCC = 1 ∣ X , V , S , Y ) = P ( Δ NCC = 1 ∣ V , Y ) = Y + ( 1 − Y ) π V , NCC ∗ , where π V , NCC ∗ is the sampling probability for the noncases. Thus, the sampling scheme is equivalent in the two designs (for simple binary outcomes). For binary outcomes, the main difference between the nested case-control design and the case-cohort design is that, in the case-control design, a random subcohort is selected in the case-cohort design while not in the nested case-control design. Two-phase designs ( Breslow and Holubkov 1997 ; Scott and Wild 1997 ) generalize the above designs by allowing the probability of gathering information on X to depend on both Y and V, and to be less than 1 even when Y = 1.

Multiple events case-control (MECC) studies generalize the above designs by allowing sampling probabilities to depend on auxiliary events or outcomes S as well as on the event of interest Y and baseline covariates V . For binary S , we will typically choose to sample (measure X ) for all subjects with S = 1, in addition to all subjects with Y = 1 and a subset of the remainder of the cohort. Let Δ MECC be an indicator of whether covariates X are measured for a given subject in this design. Typically, we have π V , Y , S , MECC = P ( Δ MECC = 1 ∣ X , V , S , Y ) = P ( Δ MECC = 1 ∣ V , S , Y ) = I ( Y = 1 or S = 1 ) + I ( Y = 0 and S = 0 ) π V , MECC ∗ , where π V , MECC ∗ is the sampling probability for noncases without the auxiliary outcome.

Early analytic methods for the case-cohort and nested case-control designs adopted the view that these designs are based on sampling from a cohort. In particular, subjects for whom Δ = 1 are sampled and so to be included in the analysis, whereas subjects for whom Δ = 0 are not sampled and so not included. This view may stem from the view of these designs as variants of standard case-control studies, in which the sampling frame is not always fully specified and the analysis is confined to subjects with Δ = 1.

An alternative view is that these are studies with data missing on X in the subset for whom Δ = 0, and so methods for dealing with missing data are appropriate here. An advantage of such missing-data approaches is that they may use information on always-measured covariates for subjects with Δ = 1 to obtain more efficient information about parameters of interest. Principled methods for dealing with missing data include multiple imputation, maximum likelihood, and estimating equations approaches. Likelihood-based approaches have been considered in this context in two-phase studies (e.g., Breslow and Cain (1988 ); Scott and Wild (1988; 1991 ; 1997 ); Breslow and Holubkov (1997) ). Robins, Rotnitzky, and Zhao (1994) proposed an approach involving weighted and possibly augmented estimating equations, and similar ideas have been considered for both survival and binary outcomes ( Fears and Brown 1986 ; Breslow and Chatterjee 1999 ; Lawless, Kalbfleisch and Wild 1999 ). Robins et al. (1994) explicitly considers the nested case-control sampling scheme and, in principle, also encompasses the MECC design outlined above. Even for the simpler case-cohort and NCC designs considered above where the nonparametric maximum likelihood approach ( Scott and Wild, 1997 ) is applicable, an advantage of this approach is that it can be extended to continuous V . In this paper, we adopt this approach for the analysis of MECC design as well as for the simpler case-cohort and NCC analyses.

Instead of using the somewhat more complicated approach to MECC data that we will outline in Section 3, MECC data may be analyzed as case-cohort studies. This may be done simply if a random subcohort is selected and so subjects with δ CC = 1 are identified even when Y = 0 and S = 1. Even in the absence of a random subcohort identified a priori, this may also be done by randomly sampling from this subset with the same probability as the selection probability for the noncases in the same stratum; i.e., P (Δ = 1| V , Y = 0, S = 1) = P (Δ = 1| V , Y = 0, S = 0). We will compare our methods for MECC analysis to the simpler analyses treating these as case-cohort studies (referred to as CC1) to consider the advantages of our approach over the simpler case-cohort approach.

Investigators contemplating a MECC study in which a single endpoint Y is the primary focus might want to consider a simpler case-cohort or NCC design in which the sampling is based on Y but not S . We might anticipate that such a design, while severely restricting our ability to study the associations of X and V with the auxiliary outcome S , might be more cost-efficient in studying the associations of these variables with the primary outcome Y . In such a design, we could increase the sampling fraction of subjects with Y = 0 from π V , MECC ∗ to π V , C C 2 ∗ = π V , MECC ∗ + P ( S = 1 , Y = 0 ) ( 1 − π V , MECC ∗ ) without increasing the number of subjects from whom data on X are to be collected. We call a nested case-control study with sampling fraction π C C 2 ∗ the CC2 alternative . The comparison of MECC with CC2 would be of interest, for example, if one has a fixed budget and wants to consider the tradeoff between the MECC design for studying two outcomes and a nested case-control design for studying one outcome of primary interest. The numerical results for comparison of MECC versus CC1 and CC2 are provided in Section 4.

3. The Analytic Approaches to MECC Studies

In this section, we outline three unbiased estimating equation approaches to analyzing data from MECC studies, based on the theory of Robins et al. (1994) : a simple approach using a weighted estimating function, a more complicated one in which the weighted estimating function is augmented, and the most complicated approach, using the optimal weighted and augmented function. In this approach, the observable and latent data ( X , V , S , Y , Δ) are taken to be an i.i.d random vector. We consider linear logistic regression models for the outcome on covariates measured at baseline: logit{ E ( Y | X * )} = X * β. X * can be replaced by q ( X * ), a known function of X * , to allow for interactions or nonlinearities in X * .

In case-control studies, a common approach is to perform a unweighted logistic regression analysis using subjects for whom full covariate information is available; i.e., logit{ P ( Y = 1| X , V , Δ CC = 1)} = X * γ CC . In these analyses, the estimate of the intercept is biased, but the estimates of the rest of the regression parameters are unbiased. However, in MECC studies, all estimates of the parameters γ MECC in the regression model logit{ P ( Y = 1| X , V , Δ MECC = 1)} = X * γ MECC are biased. This is illustrated in the simulation studies conducted in Section 4 (see Table 2 ).

Comparison of different estimators in MECC analyses for binary X: X ~ Bernoulli (0.3). The MLE is biased, whereas the WTE, WAE and SEE are asymptotically unbiased. The WTE is improved by the WAE, which is again improved by the SEE. p CC1 (p MECC ) is the average proportion of subjects with Δ CC1 = 1 (Δ MECC = 1).

Proper weighting methods may be applied through the estimating equation approach to eliminate the bias (e.g., Horvitz and Thompson 1952 ; Flanders and Greenland 1991 ; Zhao and Lipsitz 1992 ). Let U i ( β ) be the score function of the fully observable data for subject i and the unknown parameter β such that E { U i ( β )} = 0 when β is evaluated at its true value. In a logistic regression model, U i ( β ) is { Y i − E ( Y i ∣ X i ∗ ) } X i ∗ . Denote by π ( W i ) the sampling probability for subject i . Then it is easy to show that the weighted estimating equation

is unbiased. The estimator solving ( 1 ) is referred to as the weighted estimator (WTE). We summarize the weights used to estimate the regression parameters in MECC analysis in Table 1 , where we also contrast the MECC weights (also the sampling weights) with weights used in case-control analysis. It was shown by Robins et al. (1994) that estimating the selection probabilities can improve efficiency even when these probabilities are known. We hence use estimated weights in estimating equations (1) , (2) and (4) in subsequent simulation study and data analysis.

Weights used in the MECC and NCC sampling and analysis

The information of the subjects who are not selected to the MECC study is ignored in the analysis using ( 1 ). The weighted approaches are not fully efficient for the estimation of model parameters, and the general approach to improving the efficiency involves augmenting the weighted estimating equation by additional terms. Let φ ( W ) be a function of W and A ( φ ) = {Δ − π ( W )} φ ( W ) /π ( W ). We augment the estimating equation (1) to obtain

The estimator that solves ( 2 ) is referred to as the weighted-augmented estimator (WAE). The function φ does not depend on the missing covariates X , and for a given estimating function U ( β ), the optimal choice of φ ( W ) is φ ( W ) = E{ U ( β )| W }. The WAE is unbiased, and, by including an augmented term in the weighted estimating equation, we can reduce the variance of the weighted estimator by considering additional information in subjects with incomplete data. Our simulation studies in Section 4 show that the WAEs are, in general, more efficient than the WTEs.

In the presence of missing data, the most efficient approach involves choosing a new U i ( β ) and optimally augmenting the estimating function ( Robins et al. 1994 ). Consider a general non-linear regression model

Let h ( X * ) be a function of all covariates X * , U ( β , h ) = h ( X * ){ Y − g ( X * ; β )} an unbiased estimating function, φ ( W ) a function of W and A ( φ ) = {Δ − π ( W )} φ ( W ) /π ( W ). A class of estimators β ( h , φ h ), including the WTE and WAE as special cases, solves the following estimating equation

Denote by β ( h E , φ E ) the optimal estimator in this class. For a logistic regression model with linear predictors, we have h E = X * and φ E = 0 when no data are missing. For fixed h , the optimal choice for φ is φ h = E [ U ( β , h )| W ]. Robins et al. (1994) showed that the optimal estimator β ( h E , φ E ), also referred to as the semi-parametric efficient estimator (SEE), is unique in the class of estimators defined by ( 4 ), and achieves the semi-parametric efficiency bound. The WTE β wt and WAE β wa can be identified as special cases of β ( h , φ h ), that is, β wt = β ( X * , 0) and β wa = β ( X * , φ X * ). The derivation of the SEE is usually complicated, involving an iterative procedure to obtain h E and φ E . Illustration on how to adaptively obtain SEE in a simple logistic regression model is provided in the web supplementary materials .

4. Simulation

In this section, we conduct simulation studies to examine (i) the efficiency of various types of weighted, possibly augmented estimators (WTE, WAE, and SEE), both for MECC analyses and for CC analyses of MECC data, (ii) the efficiency gained by analyzing MECC data as MECC data rather than as case-cohort or nested case-control data (CC1), and (iii) the efficiency implications of using the MECC design instead of the single event case-control study with the same overall sampling proportion (CC2).

To allow our simulation model to be consistent with our model for analyzing P ( Y | X , V) , the following factorization of the joint distribution is chosen to simulate X , V , Y and S :

In our simulation, we shall consider both continuous and binary X (i.e., X ~ N (0, 1) and X ~ Bernoulli(0.3)). Other variables are binary and generated according to the following logistic regression models: logit { E ( V ∣ X ) } = logit ( p v ) = X v ∗ β v , logit { E ( Y ∣ X , V ) } = logit ( p y ) = X y ∗ β y , logit { E ( S ∣ X , V , Y ) } = logit ( p s ) = X s ∗ β s , where p v , p y and p s are the conditional expectations of V, Y and S , and X v ∗ = ( 1 , X ) , X y ∗ = ( 1 , X , V ) and X s ∗ = ( 1 , X , V , Y ) . The logistic regression model for the association between Y and X * = ( X , V) is logit[E( Y | X , V) ] = β 0 + β 1 X + β 2 V . The parameter of interest is β y = ( β 0 , β 1 , β 2 ), and other parameters ( β v , β s ) are treated as nuisance parameters. Various settings are considered to investigate how the comparison results vary according to the correlation between (i) X and V, (ii) Y and X , (iii) S and X , and (iv) Y and S . The parameters for simulation are chosen such that the populations of cases ( Y = 1) range from 4% to 15%. In all simulations, the cohort size is N = 800 and the number of replications is 1000. We noticed that for small cohort size and low event rate (cohort size=800, event rate<0.06), the SEE method fails to converge in about 5 percent out of the 1000 replications. The results from these replications have been excluded while producing our tables.

4.1 Comparison of Estimators in MECC Analysis

One naive MECC approach is to treat the MECC data as if they arose from a case-control study. In case-control studies, an unadjusted logistic regression analysis (or the maximum likelihood approach) produces unbiased estimates of covariate effects (but not of the intercept), as is noted by Prentice and Pyke (1979) . However, the maximum likelihood approach for the MECC model P ( Y | X , V , Δ MECC = 1) is biased since the controls from MECC studies, unlike those from conventional case-control studies, do not represent a random sample from the noncases. The estimate obtained using this biased approach is denoted by MLE MECC . Other estimators being considered for comparison include WTE, WAE and SEE.

We consider comparisons for both binary and continuous X . The simulation results for binary X are summarized in Table 2 . Additional comparisons of MLE, WTE and WAE for continuous X are provided in Table 3 (and Table 2 in supplementary materials ). We can see from Table 2 that the MLE MECC is biased, whereas the WTE, WAE and SEE (of the MECC analysis) are asymptotically unbiased. Relative efficiencies of different MECC estimators are also shown, with the WTE of CC1 analysis as the referent method. The following observations can be made based on the simulation results from Table 2 :

Comparison of different study designs/analyses for discrete X: X ~ Bernoulli (0.3). X and V are strongly correlated: β v = (= 2, 3). p CC1 (p CC2 , p MECC ) is the average proportion of subjects with Δ CC1 = 1 (Δ CC2 = 1, Δ MECC = 1). Table entries are the relative efficiencies of different estimators.

  • The WAE improves the WTE on estimation of β 0 and β 2 but not on estimation of β 1 . In fact, the WAE and WTE are asymptotically equivalent in estimating the effect of the incomplete covariate ( β 1 ) when estimated selection probabilities are used in the estimating equation, as is indicated by Corollary 6.1 in Robins et al. (1994) .
  • The efficiency for estimating the complete covariate ( β 2 ) is not discussed in Robins et al. (1994) . Table 2 shows that the efficiency gain of WAE over WTE in estimating β 2 is large when (a) X and V are weakly correlated, and (b) X and Y are strongly associated (Setting 1–3).
  • The SEE further improves the WAE in the estimation of all three parameters (including β 1 ). The efficiency gain of SEE over WAE is large when (a) Y and X are correlated (Setting 1–3, 6–8), and (b) S and X are uncorrelated (Setting 3–5).

4.2 Comparison of MECC design and analysis with alternatives

We report here comparisons of MECC analysis with corresponding CC1 analysis of the same data and with case-control analysis of the CC2 design. MECC versus CC1 is the primary comparison in this work.

The results for comparison are shown in Table 3 and Table 4 , respectively for binary X and continuous X (for which only WTE and WAE are considered). We consider a situation where V and X are strongly correlated. Simulation results for weakly correlated V and X are provided Table 1 and Table 2 in the supplementary materials . We mainly discuss the results in Table 3 , and similar conclusions can be drawn from other tables. Table entries are the relative efficiencies for estimating β y with the WTE of CC1 as the referent method. For each comparison, we also list the average proportion of subjects with Δ j = 1, where j may be MECC, CC1, or CC2. These proportions are denoted by p CC 1 , p MECC and p CC 2 , respectively for corresponding designs and analyses.

Comparison of different study designs/analyses for continuous X: X ~ N(0, 1). X and V are strongly correlated: β v = (−2, 3). p CC1 (p CC2 , p MECC ) is the average proportion of subjects with Δ CC1 = 1 (Δ CC2 = 1, Δ MECC = 1). Columns 2–7 are the relative efficiencies of different estimators.

The following observations can be made based on the simulation results:

  • MECC analysis is more efficient than CC1 analysis in estimating β 1 and β 2 for the corresponding analytic approaches (WTE, WAE and SEE). The gain in efficiency is large when (a) Y and X are moderately or strongly correlated (Setting 1–3, Table 3 ), and (b) S and X are strongly correlated (Setting 6–8, Table 3 )
  • If the goal is solely to estimate associations with the main outcome Y, CC2 analysis is more efficient than MECC analysis in most situations. The gain in efficiency is large when (a) Y and X are uncorrelated or weakly correlated (Setting 1–3, Table 3 ), (b) Y and S are moderately or strongly correlated, and (c) S and X are uncorrelated or weakly correlated.
  • It is interesting that MECC can be more efficient than CC2 in situations when both the associations between (a) Y and X , and (b) S and X are strong (Setting 3, Table 3 ; Setting 2–5, Table 4 ). This is because the subjects for whom Δ CC 1 = 0 and S = 1 provide more information about β 1 than the same number of subjects randomly sampled from the noncases ( Y = 0).

The confidence intervals of all estimators (obtained based on the estimated robust variance-covariance matrices), reported in Table 3 and 4 in the web supplementary materials , show that most confidence intervals have slightly higher coverage probabilities than the nominal level 95%. We also reported in Table 5 in the web supplementary materials for comparison of the empirical variance vs. estimated variance. The results show that the asymptotic approximation (used in the derivation of SEE) is improved slightly with increased sample sizes. For the simulation settings being considered (the numbers of events vary from 40 to 300), the asymptotic approximations are good.

5. MECC Analysis of a Nested Case-Control Study

So far we have discussed the use of MECC design in a cohort study with multiple types of events. However, “multiple” can be understood in a broader sense. In this section we discuss an interesting application of MECC design in a nested case-control study where only one type of event (chronic kidney injury) is of interest.

Colonoscopy is the diagnostic test of choice for many diseases of the lower gastrointestinal tract, and is among the preferred screening modalities for colorectal cancer. Prior to colonoscopy, patients need to undergo bowel preparation, and the sensitivity of colonoscopy is dependent upon the quality of this preparation. Bowel purging is most often achieved using either oral sodium phosphate (OSP)-based or polyethylene glycol (PEG)-based agents. The former is often preferred due to its greater efficacy, cost-effectiveness and patient tolerability. Recent case reports and series have suggested a potential association between OSP preparations and chronic kidney disease. Given the number of patients undergoing colonoscopy (14 million annually) and the clinical stakes of missed lesions, clarification of this potential risk is of great clinical importance.

Based on the need for manual abstraction of exposure data (OSP versus PEG), study of this association took the form of a nested case-control study ( Brunelli et al. 2007 ). Unfortunately, there is no consensus definition of incident chronic kidney disease in the clinical literature, necessitating that case status be adjudicated according to a clinically “reasonable” definition. In evaluating the association between OSP use and acute kidney injury, Hurst et al. (2007) determined that a 50% rise in serum creatinine (“strict case definition”) would represent a significant loss of kidney function. In contrast, Brunelli et al. (2007) determined cases to be a 25% rise in serum creatinine (“loose case definition”) based on: (i) a limited number of cases existed under the “strict case” definition, (ii) the consensus among study investigators that this still represented a clinically significant reduction in kidney function, and (iii) some precedent for this definition in the clinical literature. This “loose case” definition is hence used in selecting the subjects and subsequent association analysis ( Brunelli et al. 2007 ). Further investigations were undertaken to compare Brunelli’s study with Hurst’s study ( Brunelli et al. 2008 ). Various potential sources of discrepancy between the two studies were considered in the sensitivity analysis of Brunelli’s data. In particular, the investigators are interested in the impact of changing the case definition on subsequent analysis results.

Here we briefly describe the original sampling scheme and analytical methods using the “loose case” definition and an MECC reanalysis of data using the “strict case” definition. The nested case-control study conducted by Brunelli et al. (2007) enrolled 2237 subjects in the source cohort. The loose case definition was met in 141 instances; of these, bowel preparation data were available for 116. Among the 2096 colonoscopies for which the loose case definition was not met, 398 are randomly sampled and covariate data are available for 349. Patients for whom exposure data were not available did not differ from those for whom they were, see Table 1 in Brunelli et al. (2007) . Among the n = 465 subjects for whom the covariate data are available, the strict case definition is again met in 26. In our new analysis, the outcome of interest Y is the “strict case” and the auxiliary outcome S is the “loose case”. Although this is not a typical “multiple events” study (since Y = 1 implies S = 1), it naturally falls into the category of our MECC design, where information on OSP use is obtained on subjects with either the main outcome or the auxiliary outcome, as well as a random subset of subjects without either outcome.

Table 2 in Brunelli et al. (2007) presents a comparison of baseline characteristics of cases and controls, showing significant differences for gender and exposure to diuretics. Hence we consider the following logistic regression model:

Let X = (duretic, OSP) be the incomplete covariate (the exposure information of a patient to OSP or diuretic is only available for subjects who are selected) and V = Gender, the covariate available for all subjects.

In reanalysis with strict case definition, a naive approach is to fit a logistic regression model using all the n = 465 subjects collected for studying the loose case association (MLE MECC ). However, this analysis is biased for studying the strict case association. Alternately, we may randomly select from the 90 loose cases (for whom S = 1 and Y = 0) with probability 398/2096 (10 subjects were finally selected), and add them to the original 349 controls to recreate a random subset of the noncases (NCC/CC1). The NCC approach is unbiased but not efficient. An MECC analysis increases the efficiency by including in the study additional loose cases (80 subjects) who were not sampled in NCC. We implemented the weighted, weighted-augmented approaches in both loose and strict case analyses. The standard errors of the point estimates are obtained based on the robust variance-covariance estimates ( Web Appendix D in the supplementary materials ).

The results for all analyses are summarized in Table 5 . We can see that the standard errors of the estimators in the MECC analyses are slightly smaller than those in the corresponding NCC analyses. In addition, the weighted-augmented approach significantly improves the weighted approach in estimating all regression parameters. The new analysis indicates no association between OSP use and CKD, as is originally reported by Brunelli et al. (2007) using the loose case definition. However, the significantly reduced number of cases (from n = 116 to n = 26) may have limited the power of detecting the association.

The analysis of the data from a nested case-control study of risk factors for chronic kidney injury.

6. Discussion

The MECC design is useful when it is desirable to study more than one outcome in the cohort. It extends the traditional case-cohort design in that it not only uses the same subgroup of controls for studying several outcomes, but also uses subjects with one outcome to study other outcomes. A more general version of the design allows the sampling probabilities to depend on ( V , Y , S ). The case-control follow-up study ( Weiss and Lazovich 1996 ; Joffe 2003 ), where sampling is based on cancer diagnosis (an auxiliary outcome) but the outcome of interest is cancer mortality, may be viewed as a variant of the MECC study in which the sampling fraction may be unknown.

Our simulation results show that the new design, equipped with the analytical approaches in Robins et al. (1994) , can eliminate the bias of some traditional methods and improve the efficiency of the CC1 analysis of the same data. If the goal is solely to estimate associations with the main outcome Y, the MECC design is in general inferior compared to CC2 design unless the incomplete covariate ( X ) is strongly associated with both outcomes ( Y and S ). In addition, our simulation results indicate that augmenting the estimating equations is beneficial, requiring relatively little additional effort, whereas the semi-parametric efficient estimator, which requires substantial additional effort to obtain, sometimes provides little additional efficiency gain. The augmented approach makes use of information on the subjects with incomplete data, and can substantially improve the precision in estimating the effect of the always observable variables. The implementation of this approach is simple especially when W is discrete. However, this approach has scarcely been used in nested case-control research despite its advantages.

The MECC design is a sort of two-phase design (e.g., Breslow and Holubkov 1997 ; Scott and Wild 1997 ; White 1982 ; Breslow and Cain 1988 ; and Flanders and Greenland 1991 ), in which sampling depends on outcome Y and other variables. Other methods, including likelihood and pseudolikelihood-based approaches have been proposed for some such designs (e.g., Breslow and Holubkov 1997 ; Scott and Wild 1997 ). These approaches produce asymptotically efficient estimators and the implementations are not so complicated as the SEE. However, these methods require that the variables stratified for sampling (except the outcome variable) are independent of the disease outcome variable conditional on the covariates. This condition will typically fail to hold in the MECC design since the auxiliary outcome variable could still be correlated with the outcome variable of interest even after adjusting for other covariates (e.g., the example discussed in Section 5). It would be of interest for future research to implement these likelihood approaches to model sampling scheme that also depends on auxiliary outcomes.

Nested case-control and case-cohort studies often take place where a failure-time outcome rather than a simple binary event is of interest, and inference is often performed for parameters in semiparametric proportional hazards models. The simple weighting approach discussed here extends naturally to this setting. However, extensions of the more efficient augmented approaches will require further consideration.

Supplementary Material

Acknowledgments.

We thank the Associate Editor and the referee for detailed and constructive comments, which have greatly helped to improve the presentation of the paper.

7. Supplementary Materials

Web Appendices and Tables referenced in Sections 3 and 4 are available under the Paper Information link at the Biometrics website: http://www.biometrics.tibs.org .

Contributor Information

Wenguang Sun, Department of Statistics, North Carolina State University, Raleigh, NC 27695, USA.

Marshall M. Joffe, Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, PA 19104, USA.

Jinbo Chen, Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, PA 19104, USA.

Steven M. Brunelli, Renal-Electrolyte and Hypertension Division, University of Pennsylvania, Philadelphia, PA 19104, USA.

  • Astrakianakis G, Seixas N, Ray R, Camp J, Gao D, Feng Z, Li W, Wernli K, Fitzgibbons E, Thomas D, Checkoway H. Lung cancer risk among female textile workers exposed to endotoxin. Journal of the National Cancer Institute. 2007; 99 :357–64. [ PubMed ] [ Google Scholar ]
  • Breslow N, Day N. Statistical Methods in Cancer Research I. The Analysis of Case-Control Studies. Lyon, France: International Agency for Research on Cancer; 1980. [ Google Scholar ]
  • Breslow N, Cain K. Logistic Regression for Two-Stage Case-Control data. Biometrika. 1988; 75 :11–20. [ Google Scholar ]
  • Breslow N, Chatterjee N. Design and Analysis of Two-phase Studies with Binary Outcome Applied to Wilms Tumour Prognosis. Journal of the Royal Statistical Society, Seires C. 1999; 48 :457–468. [ Google Scholar ]
  • Breslow N, Holubkov R. Maximum likelihood estimation of logistic regression parameters under two-phase, outcome-dependent sampling. Journal of the Royal Statistical Society, Series B. 1997; 59 :447–461. [ Google Scholar ]
  • Brunelli SM, Lewis JD, Gupta M, Latif SM, Weiner MG, Feldman HI. Risk of kidney injury following oral phosphosoda bowel preparations. Journal of the American Society of Nephrology. 2007; 18 :3199–3205. [ PubMed ] [ Google Scholar ]
  • Brunelli SM, Lewis JD, Lynch KE, Joffe MM, Gupta M, Latif SM, Weiner MG, Feldman HI. Techinical Report. Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania; 2008. Further investigation of the association between oral sodium phosphate use and kidney injury. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Feldman H, Appel L, Chertow G, Cifelli D, Cizman B, Daugirdas J, Fink J, Franklin-Becker E, Go A, Hamm L, He J, Hostetter T, Hsu C, Jamerson K, Joffe M, Kusek W, Landis J, Lash J, Miller E, Mohler E, Muntner P, Ojo A, Rahman M, Townsend R, Wright J. The Chronic Renal Insufficiency Cohort (CRIC) Study: Design and Methods. Journal of the American Society of Nephrology. 2003; 14 :148–153. [ PubMed ] [ Google Scholar ]
  • Fears T, Brown C. Logistic Regression Methods for Retrospective Case-Contrl Studies Using Complex Sampling Procedures. Biometrics. 1986; 42 :955–960. [ PubMed ] [ Google Scholar ]
  • Flanders W, Greenland S. Analytic methods for two-stage case-control studies and other stratified designs. Statististics in Medcine. 1991; 10 :739–47. [ PubMed ] [ Google Scholar ]
  • Horvitz D, Thompson D. A generalization of sampling without replacement from a finite universe. Journal of the American Stastistical Association. 1952; 47 :663–685. [ Google Scholar ]
  • Hurst FP, Bohen EM, Osgard EM, Oliver DK, Das NP, Gao SW, Abbott KC. Association of oral sodium phosphate purgative use with acute kidney injury. Journal of the American Society of Nephrology. 2007; 18 :3192–3198. [ PubMed ] [ Google Scholar ]
  • Joffe M. A case-control follow-up study for disease-specific mortality. Biometrics. 2003; 59 :115–125. [ PubMed ] [ Google Scholar ]
  • Kupper L, McMichael A, Spirtas R. A hybrid epidemiologic study design useful in estimating relative risk. Journal of the American Statistical Association. 1975; 70 :524–528. [ Google Scholar ]
  • Langholz B, Thomas D. Nested case-control and case-cohort methods of sampling from a cohort: a critical comparison. American Journal of Epidemiology. 1990; 131 :169–176. [ PubMed ] [ Google Scholar ]
  • Lawless J, Kalbfleisch J, Wild C. Semiparametric methods for response-selective and missing data problems in regression. Journal of the Royal Statistical Society, Series B. 1999; 61 :413–438. [ Google Scholar ]
  • Li W, Ray M, Gao D, Fitzgibbons E, Seixas N, Camp J, Wernli K, Astrakianakis G, Feng Z, Thomas D, Checkoway H. Occupational risk factors for pancreatic cancer among female textile workers in Shanghai, China. Occupational and Environmental Medicine. 2006; 63 :788–93. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Prentice R. A case-cohort design for epidemiologic cohort studies and disease prevention trials. Biometrika. 1986; 73 :1–11. [ Google Scholar ]
  • Prentice RL, Pyke R. Logistic disease incidence models and case-control studies. Biometrika. 1979; 66 :403–411. [ Google Scholar ]
  • Ray M, Gao L, Li W, Wernli K, Astrakianakis G, Seixas S, Camp J, Fitzgibbons E, Feng Z, Thomas B, Checkoway H. Occupational exposures and breast cancer among women textile workers in Shanghai. Epidemiology. 2007; 18 :383–92. [ PubMed ] [ Google Scholar ]
  • Robins J, Rotnitzky A, Zhao L. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association. 1994; 89 :846–866. [ Google Scholar ]
  • Rothman K, Greenland S, Lash T. Modern Epidemiology. 3. Lippincott-Raven: Philadelphia; 2008. [ Google Scholar ]
  • Scott A, Wild C. Hypothesis Testing in Case-Control Studies. Biometrika. 1989; 76 :806–808. [ Google Scholar ]
  • Scott A, Wild C. Fitting logistic models in stratified case-control studies. Biometrics. 1991; 47 :497–510. [ Google Scholar ]
  • Scott A, Wild C. Fitting regression models to case-control data by maximum likelihood. Biometrika. 1997; 84 :57–71. [ Google Scholar ]
  • Stampfer M, Willett W, Colditz G, Rosner B, Speizer F, Hennekens C. A prospective study of postmenopausal estrogen theorapy and coronary heart disease. New England Journal of Medicine. 1985; 313 :1044–1049. [ PubMed ] [ Google Scholar ]
  • Weiss N, Lazovich D. Case-control studies of screening efficacy: the use of persons newly diagnosed with cancer who later sustain an unfavorable outcome. American Journal of Epidemiology. 1996; 143 :319–322. [ PubMed ] [ Google Scholar ]
  • White J. A two stage design for the study of the relationship between a rare exposure and a rare disease. American Journal of Epidemiology. 1982; 115 :119–128. [ PubMed ] [ Google Scholar ]
  • Zhao L, Lipsitz S. Designs and analysis of two-stage studies. Statistics in Medcine. 1992; 11 :769–82. [ PubMed ] [ Google Scholar ]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • What Is a Case-Control Study? | Definition & Examples

What Is a Case-Control Study? | Definition & Examples

Published on February 4, 2023 by Tegan George . Revised on June 22, 2023.

A case-control study is an experimental design that compares a group of participants possessing a condition of interest to a very similar group lacking that condition. Here, the participants possessing the attribute of study, such as a disease, are called the “case,” and those without it are the “control.”

It’s important to remember that the case group is chosen because they already possess the attribute of interest. The point of the control group is to facilitate investigation, e.g., studying whether the case group systematically exhibits that attribute more than the control group does.

Table of contents

When to use a case-control study, examples of case-control studies, advantages and disadvantages of case-control studies, other interesting articles, frequently asked questions.

Case-control studies are a type of observational study often used in fields like medical research, environmental health, or epidemiology. While most observational studies are qualitative in nature, case-control studies can also be quantitative , and they often are in healthcare settings. Case-control studies can be used for both exploratory and explanatory research , and they are a good choice for studying research topics like disease exposure and health outcomes.

A case-control study may be a good fit for your research if it meets the following criteria.

  • Data on exposure (e.g., to a chemical or a pesticide) are difficult to obtain or expensive.
  • The disease associated with the exposure you’re studying has a long incubation period or is rare or under-studied (e.g., AIDS in the early 1980s).
  • The population you are studying is difficult to contact for follow-up questions (e.g., asylum seekers).

Retrospective cohort studies use existing secondary research data, such as medical records or databases, to identify a group of people with a common exposure or risk factor and to observe their outcomes over time. Case-control studies conduct primary research , comparing a group of participants possessing a condition of interest to a very similar group lacking that condition in real time.

Prevent plagiarism. Run a free check.

Case-control studies are common in fields like epidemiology, healthcare, and psychology.

You would then collect data on your participants’ exposure to contaminated drinking water, focusing on variables such as the source of said water and the duration of exposure, for both groups. You could then compare the two to determine if there is a relationship between drinking water contamination and the risk of developing a gastrointestinal illness. Example: Healthcare case-control study You are interested in the relationship between the dietary intake of a particular vitamin (e.g., vitamin D) and the risk of developing osteoporosis later in life. Here, the case group would be individuals who have been diagnosed with osteoporosis, while the control group would be individuals without osteoporosis.

You would then collect information on dietary intake of vitamin D for both the cases and controls and compare the two groups to determine if there is a relationship between vitamin D intake and the risk of developing osteoporosis. Example: Psychology case-control study You are studying the relationship between early-childhood stress and the likelihood of later developing post-traumatic stress disorder (PTSD). Here, the case group would be individuals who have been diagnosed with PTSD, while the control group would be individuals without PTSD.

Case-control studies are a solid research method choice, but they come with distinct advantages and disadvantages.

Advantages of case-control studies

  • Case-control studies are a great choice if you have any ethical considerations about your participants that could preclude you from using a traditional experimental design .
  • Case-control studies are time efficient and fairly inexpensive to conduct because they require fewer subjects than other research methods .
  • If there were multiple exposures leading to a single outcome, case-control studies can incorporate that. As such, they truly shine when used to study rare outcomes or outbreaks of a particular disease .

Disadvantages of case-control studies

  • Case-control studies, similarly to observational studies, run a high risk of research biases . They are particularly susceptible to observer bias , recall bias , and interviewer bias.
  • In the case of very rare exposures of the outcome studied, attempting to conduct a case-control study can be very time consuming and inefficient .
  • Case-control studies in general have low internal validity  and are not always credible.

Case-control studies by design focus on one singular outcome. This makes them very rigid and not generalizable , as no extrapolation can be made about other outcomes like risk recurrence or future exposure threat. This leads to less satisfying results than other methodological choices.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

  • Student’s  t -distribution
  • Normal distribution
  • Null and Alternative Hypotheses
  • Chi square tests
  • Confidence interval
  • Quartiles & Quantiles
  • Cluster sampling
  • Stratified sampling
  • Data cleansing
  • Reproducibility vs Replicability
  • Peer review
  • Prospective cohort study

Research bias

  • Implicit bias
  • Cognitive bias
  • Placebo effect
  • Hawthorne effect
  • Hindsight bias
  • Affect heuristic
  • Social desirability bias

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

A case-control study differs from a cohort study because cohort studies are more longitudinal in nature and do not necessarily require a control group .

While one may be added if the investigator so chooses, members of the cohort are primarily selected because of a shared characteristic among them. In particular, retrospective cohort studies are designed to follow a group of people with a common exposure or risk factor over time and observe their outcomes.

Case-control studies, in contrast, require both a case group and a control group, as suggested by their name, and usually are used to identify risk factors for a disease by comparing cases and controls.

A case-control study differs from a cross-sectional study because case-control studies are naturally retrospective in nature, looking backward in time to identify exposures that may have occurred before the development of the disease.

On the other hand, cross-sectional studies collect data on a population at a single point in time. The goal here is to describe the characteristics of the population, such as their age, gender identity, or health status, and understand the distribution and relationships of these characteristics.

Cases and controls are selected for a case-control study based on their inherent characteristics. Participants already possessing the condition of interest form the “case,” while those without form the “control.”

Keep in mind that by definition the case group is chosen because they already possess the attribute of interest. The point of the control group is to facilitate investigation, e.g., studying whether the case group systematically exhibits that attribute more than the control group does.

The strength of the association between an exposure and a disease in a case-control study can be measured using a few different statistical measures , such as odds ratios (ORs) and relative risk (RR).

No, case-control studies cannot establish causality as a standalone measure.

As observational studies , they can suggest associations between an exposure and a disease, but they cannot prove without a doubt that the exposure causes the disease. In particular, issues arising from timing, research biases like recall bias , and the selection of variables lead to low internal validity and the inability to determine causality.

Sources in this article

We strongly encourage students to use sources in their work. You can cite our article (APA Style) or take a deep dive into the articles below.

George, T. (2023, June 22). What Is a Case-Control Study? | Definition & Examples. Scribbr. Retrieved February 19, 2024, from https://www.scribbr.com/methodology/case-control-study/
Schlesselman, J. J. (1982). Case-Control Studies: Design, Conduct, Analysis (Monographs in Epidemiology and Biostatistics, 2) (Illustrated). Oxford University Press.

Is this article helpful?

Tegan George

Tegan George

Other students also liked, what is an observational study | guide & examples, control groups and treatment groups | uses & examples, cross-sectional study | definition, uses & examples, what is your plagiarism score.

Case Control Studies

Affiliations.

  • 1 University of Nebraska Medical Center
  • 2 Spectrum Health/Michigan State University College of Human Medicine
  • PMID: 28846237
  • Bookshelf ID: NBK448143

A case-control study is a type of observational study commonly used to look at factors associated with diseases or outcomes. The case-control study starts with a group of cases, which are the individuals who have the outcome of interest. The researcher then tries to construct a second group of individuals called the controls, who are similar to the case individuals but do not have the outcome of interest. The researcher then looks at historical factors to identify if some exposure(s) is/are found more commonly in the cases than the controls. If the exposure is found more commonly in the cases than in the controls, the researcher can hypothesize that the exposure may be linked to the outcome of interest.

For example, a researcher may want to look at the rare cancer Kaposi's sarcoma. The researcher would find a group of individuals with Kaposi's sarcoma (the cases) and compare them to a group of patients who are similar to the cases in most ways but do not have Kaposi's sarcoma (controls). The researcher could then ask about various exposures to see if any exposure is more common in those with Kaposi's sarcoma (the cases) than those without Kaposi's sarcoma (the controls). The researcher might find that those with Kaposi's sarcoma are more likely to have HIV, and thus conclude that HIV may be a risk factor for the development of Kaposi's sarcoma.

There are many advantages to case-control studies. First, the case-control approach allows for the study of rare diseases. If a disease occurs very infrequently, one would have to follow a large group of people for a long period of time to accrue enough incident cases to study. Such use of resources may be impractical, so a case-control study can be useful for identifying current cases and evaluating historical associated factors. For example, if a disease developed in 1 in 1000 people per year (0.001/year) then in ten years one would expect about 10 cases of a disease to exist in a group of 1000 people. If the disease is much rarer, say 1 in 1,000,0000 per year (0.0000001/year) this would require either having to follow 1,000,0000 people for ten years or 1000 people for 1000 years to accrue ten total cases. As it may be impractical to follow 1,000,000 for ten years or to wait 1000 years for recruitment, a case-control study allows for a more feasible approach.

Second, the case-control study design makes it possible to look at multiple risk factors at once. In the example above about Kaposi's sarcoma, the researcher could ask both the cases and controls about exposures to HIV, asbestos, smoking, lead, sunburns, aniline dye, alcohol, herpes, human papillomavirus, or any number of possible exposures to identify those most likely associated with Kaposi's sarcoma.

Case-control studies can also be very helpful when disease outbreaks occur, and potential links and exposures need to be identified. This study mechanism can be commonly seen in food-related disease outbreaks associated with contaminated products, or when rare diseases start to increase in frequency, as has been seen with measles in recent years.

Because of these advantages, case-control studies are commonly used as one of the first studies to build evidence of an association between exposure and an event or disease.

In a case-control study, the investigator can include unequal numbers of cases with controls such as 2:1 or 4:1 to increase the power of the study.

Disadvantages and Limitations

The most commonly cited disadvantage in case-control studies is the potential for recall bias. Recall bias in a case-control study is the increased likelihood that those with the outcome will recall and report exposures compared to those without the outcome. In other words, even if both groups had exactly the same exposures, the participants in the cases group may report the exposure more often than the controls do. Recall bias may lead to concluding that there are associations between exposure and disease that do not, in fact, exist. It is due to subjects' imperfect memories of past exposures. If people with Kaposi's sarcoma are asked about exposure and history (e.g., HIV, asbestos, smoking, lead, sunburn, aniline dye, alcohol, herpes, human papillomavirus), the individuals with the disease are more likely to think harder about these exposures and recall having some of the exposures that the healthy controls.

Case-control studies, due to their typically retrospective nature, can be used to establish a correlation between exposures and outcomes, but cannot establish causation . These studies simply attempt to find correlations between past events and the current state.

When designing a case-control study, the researcher must find an appropriate control group. Ideally, the case group (those with the outcome) and the control group (those without the outcome) will have almost the same characteristics, such as age, gender, overall health status, and other factors. The two groups should have similar histories and live in similar environments. If, for example, our cases of Kaposi's sarcoma came from across the country but our controls were only chosen from a small community in northern latitudes where people rarely go outside or get sunburns, asking about sunburn may not be a valid exposure to investigate. Similarly, if all of the cases of Kaposi's sarcoma were found to come from a small community outside a battery factory with high levels of lead in the environment, then controls from across the country with minimal lead exposure would not provide an appropriate control group. The investigator must put a great deal of effort into creating a proper control group to bolster the strength of the case-control study as well as enhance their ability to find true and valid potential correlations between exposures and disease states.

Similarly, the researcher must recognize the potential for failing to identify confounding variables or exposures, introducing the possibility of confounding bias, which occurs when a variable that is not being accounted for that has a relationship with both the exposure and outcome. This can cause us to accidentally be studying something we are not accounting for but that may be systematically different between the groups.

Copyright © 2024, StatPearls Publishing LLC.

  • Introduction
  • Issues of Concern
  • Clinical Significance
  • Enhancing Healthcare Team Outcomes
  • Review Questions

Publication types

  • Study Guide

Logo for Open Educational Resources

9 Study Designs Revisited

Learning Objectives

After reading this chapter, you will be able to do the following:

  • Compare and contrast the strengths and limitations of cohort, case-control, cross-sectional, and randomized controlled trial studies
  • Describe ecologic studies and explain the ecologic fallacy
  • Describe the appropriate use of a systematic review and meta-analysis

Now that we have a firm understanding of potential threats to study validity, in this chapter we will revisit the 4 main epidemiologic study designs, focusing on strengths, weaknesses, and important details. I will also describe a few other study designs you may see, then end with a section on systematic reviews and meta-analyses, which are formal methods for synthesizing a body of literature on a given exposure/disease topic.

Recall from chapter 4 that a cohort study   consists of drawing an at-risk (nondiseased) sample from the population, assessing levels of exposure, and then following the cohort over time and watching for incident disease:

case control study multiple outcomes

Cohort studies are a very strong study design, meaning that they are less prone to bias and temporality-related logical errors than some other designs. First, because we begin with a non-diseased sample, for which we immediately assess exposure status, we know that the exposure came first. Because of this, cohorts are unlikely to have misclassification of exposure differentially by disease status because the exposure is measured before disease status is known (misclassification of disease status differentially by exposure, however, is still a risk).

Cohort Temporality and Latent Periods 

For diseases that have a long latent period—meaning that the biological onset of disease occurs long before the disease is detected and diagnosed—it is possible that some of our “nondiseased” sample are actually diseased but just have not been diagnosed yet. This could happen, for instance, for a cancer patient while their tumor is still too small to detect. When conducting studies on conditions with known or suspected long latent periods, epidemiologists will often exclude from the sample any participants in whom the disease is diagnosed during the first several months of follow-up, theorizing that those individuals were not truly disease-free at baseline.

Second, because cohort studies look for incident disease, they do not conflate the person’s having the disease with how long they have had it, as prevalence studies do (see chapter 2 for a discussion of the mathematical relationship among incidence, prevalence, and duration of disease).

Third, they are the only study design that can be used to assess rare exposures. If the exposure is uncommon within the target population (say, 10% or fewer people can be expected to be exposed), then cohort studies can deliberately sample exposed individuals to ensure sufficient statistical power (the smallest cell in the 2 × 2 table drives the power) without needing an unreasonably large sample. For example, if we are concerned about chemical exposures in a particular factory, we might enroll exposed workers from that factory as well as a unexposed group of workers from a different factory (checking first, of course, to make sure that the second factory is truly exposure-free) and follow both groups, looking for incident disease.

Which leads nicely to the fourth strength: multiple outcomes can be assessed in the same cohort. In our factory example above, the exposed workers from Factory 1 and the unexposed workers from Factory 2 can be followed for any reasonably common disease. (Just how common is a judgment call—we could also watch for and track uncommon diseases, as long as we acknowledge that those analyses would be underpowered.) We could look for new-onset heart disease, leukemia, fibromyalgia, diabetes, death, or anything else of interest. If looking at more than one outcome, then we also must measure all outcomes of interest in the sample at baseline. Then, for analyses of each specific outcome, we merely eliminate from the cohort the people who were not at risk of that outcome. For instance, if Person A joins our factory study, and at the beginning of the study they already have hypertension   but do not have melanoma, then we would not include that person in analyses where hypertension is the disease outcome. Their data could, however, be included in analyses where melanoma is the disease outcome, because they were at risk of melanoma at baseline.

Cohort studies can also be used to study multiple exposures, as long as these exposures are all common enough that we would not need to deliberately sample on exposure status. To do this, we would just grab a sample from the target population, and assess a multitude of exposures. If we want to also assess more than one outcome, then we need to measure all disease states of interest at baseline so that eventual analyses can be restricted to the population at risk, as discussed above. This ability to look at multiple outcomes—and potentially also multiple exposures—adds efficiency to cohort studies, as we can essentially conduct numerous studies all at once.

The Framingham Heart Study is a classic example of a cohort study that assessed multiple exposures and multiple outcomes. This study, a collaboration between the US National Heart, Lung, and Blood Institute (a division of the National Institutes of Health) and Boston University, began in 1948 by enrolling just over 5,000 adults living in Framingham, Massachusetts. Investigators measured numerous exposures and outcomes, then repeated the measurements every few years. As the cohort aged, their spouses, children, children’s spouses, and grandchildren have been enrolled. The Framingham study is responsible for much of our knowledge about heart disease, stroke, and related disorders, as well as of the intergenerational effects of some lifestyle habits. More information and a list of additional publications (more than 3,500 studies have been published using Framingham data) can be found here .

Cohort studies also have downsides. They cannot be used to study rare diseases because the cohort would need to be too large to be practical. For example, phenylketonuria is a genetic metabolic disorder affecting about 1 in 10,000 infants born in the US. 1 To get even 100 affected individuals, then, we would need to enroll one million pregnant women in our study—a number that is neither practical nor feasible.

Furthermore, prospective cohort studies are costly. Following people over time takes a fair bit of effort, which means that study personnel costs are high. Because of this, cohort studies cannot be used to study diseases with decades-long induction   or latent periods .

For example, it would be difficult to conduct a cohort study looking at whether adolescent dairy product consumption is associated with osteoporosis in 80-year-old women, because following current teenagers for 60 years or more would be extremely difficult. Along similar lines, selection biases related to lack of follow-up can be severe in studies with long durations: the longer we try to follow people, the more likely it is that they move, change phone numbers/email addresses, or get tired of filling out a survey every year and just stop participating. More troubling would be if people who start to feel ill are the ones who quit answering inquiries from the study team. What if these people were feeling ill because they were about to be diagnosed with the outcome under study? Despite this difficulty, a few long cohort studies such as Framingham exist and have yielded rich datasets and much knowledge about human health.

Randomized Controlled Trials

Recall from chapter 4 that an RCT is conceptually just like a cohort, with one difference:  the investigator determines exposure status.

case control study multiple outcomes

Thus all of the strengths and weaknesses of cohort studies apply also to RCTs, with one exception:  to study multiple exposures, one would need to re-randomize for each exposure. A few studies have successfully done this (the Women’s Health Initiative , for instance, randomized women to both hormone replacement therapy or placebo, and also, separately, to calcium supplements or placebo), but practically speaking RCTs are usually limited to one exposure.

One additional strength of a randomized trial (which does not apply to cohort studies) is that if the study is large enough (at least several hundred participants) and exposure allocation is truly random (i.e., not “every other person” or some other predictable scheme), then there will be no confounding. One can control, statistically, for measured confounders in a cohort study (see chapter 7), but what about any unknown and/or unmeasured confounders? The key feature of randomization is that it accounts for all confounders: known, unknown, measured, and unmeasured.

Recall from chapter 7 that for a variable to act as a confounder, it must satisfy these conditions:

case control study multiple outcomes

The variable must cause the outcome, be statistically associated with the exposure, and not be on the causal pathway (so the exposure does not cause the confounder). By randomly assigning the exposure, we have ensured that no variables exist that are associated with the exposure.

The picture now looks like this:

case control study multiple outcomes

Because no variables are more common in the exposed group than the unexposed group (or vice versa), we have gotten rid of all possible confounding. The benefits of this in terms of internal study validity cannot be overstated.

However, RCTs also have limitations, and these should not be overlooked. First and foremost, they are even more expensive than cohort studies. Second, there are often ethical considerations rendering the randomized trial design unusable. For instance, at this point, we could not ethically justify randomizing people to a smoking exposure (because its harms are so well-documented, we cannot ask people to begin smoking for our study). We also cannot randomize where people live, but certainly where people live has a profound effect on their health. ii Observational studies of these exposures, on the other hand, are ethically viable because people have already chosen whether to smoke and where to live, and the epidemiologist merely measures these existing exposures.

Third, RCTs often have generalizability issues because the kinds of people who are willing to participate in a study where they (the participant) do not get to choose which study group to be in are not a random subset of the overall population. For instance, if the only people who have time to participate in our physical activity intervention are people who are retired, then can we generalize to the (presumably younger) population who are still working? Perhaps—but perhaps not. Investigators conducting RCTs also sometimes overly restrict the inclusion criteria to the extent that results are not generalizable to the overall population. For instance, a well-known trial of blood pressure control in older adults excluded those with diabetes, cancer, and a host of other comorbidities. iii Given that most older people have at least one of these chronic diseases, to whom can we really apply the results?

Lastly, we have to precisely specify the exposure in an RCT. If we are doing a physical activity intervention, are we going to ask those randomized to the exposed group to walk? To take a yoga class? Do supervised strength training? If so, how much? How often? With how much intensity? For how many weeks or months? In a cohort study, we would assess the physical activity people are doing anyway, and there would be a huge variety of responses, which we could then categorize in any number of ways. With a randomized trial, we have to decide on all of the details. If we are wrong, or if we apply the intervention at the wrong time in the disease process, it could seem like there is no exposure/disease association, when really there is and our exposure was slightly off somehow.

Randomized trials are often called the “gold standard” of epidemiologic and clinical research because of their ability to minimize confounding. However, their drawbacks are substantial, and well-conducted observational studies should not necessarily be discounted merely because they are not RCTs. Nonetheless, RCTs play an enormous role particularly in medicine, as the Food and Drug Administration (FDA) requires multiple RCTs prior to approving new drugs and medical devices. Because of the FDA’s strict requirements, protocols for randomized trials must be registered (at clinicaltrials.gov ) prior to the start of any data collection.

Outside of pharmaceutical research and development, RCTs, because of their methodologic strengths, have the potential to change practice when evidence from new, large, well-designed studies emerge. For example, in 2005 Dr. Paul Ridker and colleagues absolutely changed the way physicians thought about heart disease prophylaxis in women. iv Prior to publication of this large (20,000 women in each group) trial, we assumed that, like men, older women should take a baby aspirin every other day to prevent heart attacks. However, the Ridker trial showed that aspirin acts differently in women (gender is an effect modifier!), and the aspirin-a-day-prevents-heart-attacks regimen will not work for most women.

Case-Control Studies

A case-control study is a retrospective design wherein we begin by finding a group of cases (people who have the disease under study) and a comparable group of controls (people who do not have the disease):

case control study multiple outcomes

A common mistake made by beginning epidemiology students is to state that “cases are people with the disease, who are exposed.” This is incorrect. Cases are people with the disease, and to avoid differential misclassification, it is important that both cases and controls be recruited without regard for exposure status. Once we have identified all cases and controls, then we assess which people were exposed.

Because they do not require following people over time, case-control studies are much cheaper to conduct than cohorts or randomized trials. They also provide an efficient way to study rare diseases and diseases with long induction and/or latent periods. Case-control studies can assess multiple exposures, though they are limited to one outcome by definition.

Case-control studies assess exposure in the past. Occasionally, these past exposure data come from existing records (e.g., medical records for a person’s blood pressure history), but usually we rely on questionnaires. Case-control studies are thus subject to recall bias, more so than prospective designs. Epidemiologists conducting case-control studies need to be particularly wary of differential recall by case status. It is plausible that people with a given condition will have spent time thinking about what might have caused it and thus be able to report past exposures with greater detail than members of the control group. Regardless of case status, the questions asked must be possible for people to answer. No one can say with certainty exactly what they ate on a particular day a decade ago; however, most people can probably recall what kinds of foods they usually ate on most days. Details are thus sacrificed in favor of bigger-picture accuracy (which may still be of questionable validity, depending on people’s memories). Remember from chapters 5 and 6: ask yourself, “Can people tell me this? Will people tell me this?”

The proper selection of controls is paramount in case-control studies, but unfortunately, who constitutes a “proper” control is not always immediately obvious. To avoid selection biases, cases and controls must come from the same target population—that is, if controls had been sick with the disease in question, they too would have been cases.

For instance, if cases are recruited from a particular hospital, then controls should be sampled from the population of people who also would have sought care at that hospital if necessary. This seems simple enough, but it is not always easy to translate into practice. If we are studying traumatic brain injury (TBI) in children in Oregon, a good place to find cases would be at Doernbecher Children’s Hospital in Portland. Other hospitals throughout the Pacific Northwest send kids with severe TBIs to Doernbecher, where a myriad of pediatric specialists are available to care for them; this hospital thus has a sufficient number of cases for our study.

Where would we get controls? One possibility would be to take as controls other children who are patients at Doernbecher, for a condition other than TBI. This satisfies the criterion that controls would also get care at this hospital, because they are  getting care at this hospital. However, to the extent that kids receiving care for other conditions might also  have unusual exposure histories, this could lead to biased estimates of association. Another option would be to designate as controls children who are not sick, sampled perhaps from a Portland neighborhood or two. However, this would also lead to selection bias, because Doernbecher is a referral hospital, receiving as patients children from a several-hundred-mile radius, not just children who live in Portland. If kids who live out in more rural areas are different than those who live in the city, we would have biased estimates of association.

The bottom line is that there is no perfect way to recruit controls, and epidemiologists love to poke holes in other people’s control groups for case-control studies v, vi (this is considered good sport at epidemiology conferences). One way to reduce bias from the control group is to recruit multiple control groups—perhaps one hospital-based and one community-based. If the results are not substantially different, then any selection biases that are operating are perhaps not overly influencing the results.

For long-lasting chronic diseases, the issue of disease duration again comes into play. To avoid temporality issues, we must know at a minimum the date of diagnosis and ensure that we are assessing exposures that happened well before that date. For conditions for which the induction and latent periods are unknown, investigators will sometimes conduct a case-control study that recruits incident cases of disease over a period of several months. Thus as soon as cases are recruited, we can ask about past exposures with the confidence that at least the case diagnosis occurred after those exposures. While a long latent period might still be an issue, one way around this would be to ask about exposures over multiple time periods—say, 0–5 years ago, 6–10 years ago, 11–15 years ago, and so on—and compare results across these windows.

Despite these difficulties, case-control studies have made substantial contributions to our knowledge about health over the years. The surgeon general’s 1964 report Smoking and Health vii , for instance, was based on literature that stemmed from a case-control study conducted by Richard Doll and Austin Bradford Hill. viii

Cross-Sectional Studies

Recall from chapter 4 that in a cross-sectional study, we draw a single sample from the target population and assess current exposure and disease status on everyone:

case control study multiple outcomes

The main strength of cross-sectional studies is that they are the fastest and cheapest studies to conduct. They are thus used for many surveillance activities—the National Health and Nutrition Examination Survey (NHANES), Pregnancy Risk Assessment Monitoring System (PRAMS), and Behavioral Risk Factor Surveillance System (BRFSS) are all cross-sectional studies that are repeated with a new sample each year (see chapter 3)—and in other situations where resources may be limited and/or immediate answers are required.

Cross-sectional studies are limited by the fact that we sample for neither exposure nor disease and that we instead “get what we get” when drawing our sample from the population. They thus cannot be used for either rare exposures or rare diseases.

Another limitation is that we have no data on temporality: we do not know whether the exposure or the disease came first because we are measuring the prevalence of both at the same point in time.

Cross-sectional studies along with surveillance (which looks only at measures of disease frequency, not at exposure/disease relationships) are thus limited to hypothesis generation activities. We cannot make (nonsurveillance) public health or clinical decisions based on evidence only from these studies.

Case Reports/Case Series

In the clinical literature, one often sees case reports. These are short blurbs reporting an interesting and unusual patient seen by a particular doctor or clinic. A case series is the same thing but describes more than one patient—usually only a few, ix,x but sometimes several hundred. xi   Case reports and case series have little value for epidemiologists because they are not studies per se; they have no comparison groups. If a case series is published saying that 45% of patients in this series with disease Y also have disease Z, this is not useful information for an epidemiologist. How many patients who do not have disease Y also have disease Z? Without data on a comparable group of patients who do not have disease Y, there is nothing to be done with the 45% data point given in the case series.

That said, case reports and case series can be extremely useful for public health professionals. Because by definition they present data from unusual patients, they can often act as a kind of sentinel surveillance, drawing our attention to a new, emerging public health threat. For example, in 1941, a physician from Australia noticed an increase in a kind of birth defect affecting infant eyes. He published this as a case series, xii   hypothesizing that maternal rubella infection was the cause. Other physicians from around the world chimed in that they, too, had seen a recent sudden increase in this birth defect in women whose pregnancies were complicated by rubella, xiii, xiv, xv   leading to our current practice of checking for rubella antibodies in all pregnant women and vaccinating those without immunity. As another example, in the early 1980s, a set of case series published by the Centers for Disease Control and Prevention (CDC) in its Morbidity and Mortality Weekly Report  drew our attention to unusual kinds of cancers and opportunistic infections occurring in otherwise young, healthy populations—our first inkling of the HIV/AIDS epidemic. x, xvi, xvii, xviii   More recently, in 2003, case reports detailing an unusual, deadly respiratory infection in people traveling to Hong Kong led to increased global public health and clinical awareness of this unusual set of symptoms, allowing immediate quarantine of affected individuals who had traveled back to Toronto. xix, xx, xxi   This quick action prevented SARS from becoming a global pandemic.

Ecologic Studies

Ecologic studies are those in which group-level data (usually geographic) are used to compare rates of disease and/or disease behaviors. For instance, this picture showing variation in seat belt use by state from chapter 1 is a kind of ecologic study:

case control study multiple outcomes

By comparing rates of seat belt use across different states, we are comparing group-level data, not data from individuals. While useful, this kind of picture can lead to many errors in logic. For example, it assumes that everyone in a given state is exactly the same—obviously this is not true. While it is true that on average, people in Oregon wear their seat belt more often than people from Idaho, this does not mean that everyone in Oregon wears their seat belt more often than everyone in Idaho. We could easily find someone in Oregon who never wears their seat belt and someone in Idaho who always does.

The above logical error—ascribing group-level numbers to any one individual—is an example of the ecologic fallacy . This also comes into play when looking at both exposure and disease patterns using group-level data, as in this example, looking at per-capita rice consumption and maternal mortality in each country:

case control study multiple outcomes

Figure 9-8. Created with data from here and here .

From looking at this graph, it appears that the more rice that is consumed by citizens of a particular country, the higher the maternal mortality rate. The ecologic fallacy here would stem from assuming that it is the rice consumers who are dying from complications related to pregnancy or birth, but we cannot know whether this is true using only group-level data.

With all of these problems, then, why conduct ecologic studies? Even more so than cross-sectional studies, they are quick and cheap. They also always use preexisting data—census estimates for per-county income; the amount of some product (such as rice) consumed by a given group of people (often tracked by sellers of that product); and recorded information on the prevalence of certain diseases (usually publicly available via the websites of health ministries for various countries or as by-country comparisons published by the World Health Organization). The use of ecologic studies is limited only to hypothesis generation, but they are so easy that they can be a good first step for a totally new research question.

Systematic Reviews and Meta-analyses

Because epidemiology relies on humans, it is more prone to both bias and confounding than other sciences. Does this render it useless? Absolutely not, though one must have a robust appreciation for the assumptions and limitations inherent in epidemiologic studies. One of these limitations is that barring exceptionally well-done, randomized controlled trials (as the Ridker trial, iv   mentioned previously), we rarely change public health or clinical policy based on just one epidemiologic study. Rather, we do one study, then another, and then another, using better and better study designs until eventually there is a body of evidence on a topic that stems from different populations, uses different study designs, perhaps measures the exposure in slightly different ways, and so on. If all these studies tend to show the same general results (as did all the early studies on smoking and lung cancer), then we start to think that the association might be causal (see chapter 10 for more detail on this) and implement public health or clinical changes.

When results of existing studies on a topic are more mixed, there is a formal way of synthesizing their results across all of them, to arrive at “the” answer: meta-analysis (or systematic review—they differ slightly, as discussed below). The procedure for either of these is the same:

  • Determine the topic—precisely. Do we care about correlates of physical activity in kids generally, or only in PE class at school? Only at home? Everywhere? Is our focus all children or only grade-school kids? Only adolescents? There often is no right answer, but as with defining our target population (see chapter 1), this needs to be decided ahead of time.
  • Systematically search the literature for relevant papers. By systematically, I mean using and documenting specific search terms and placing documented limits (language, publication date, etc.) on the search results. The key is to make the search replicable by others. It is not acceptable to just include papers that authors are aware of without searching the literature for others—doing so results in a biased sample of all the papers that should have been included.
  • Narrow down the search results to only those directly addressing the topic as determined in Step 1.
  • For each of the studies to be included, abstract key data: the exposure definition and measurement methods, the outcome definition and measurement methods, how the sample was drawn, the target population, the main results, and so on.
  • If they are, then researchers essentially combine all the data from all the included studies and generate an “overall” measure of association and 95% confidence interval.
  • If they are not, then the authors will synthesize the studies in other meaningful ways, comparing and contrasting their results, strengths, and weaknesses and arriving at an overall conclusion based on the existing literature. An overall measure of association is not calculated, but usually the authors are able to conclude that some exposure either is or is not associated with some outcome (and perhaps roughly the strength of that association).
  • Assess the likelihood of publication bias (again, there are formal statistical methods for this) xxii(pp197-200)  and the degree to which that may or may not have affected the results.
  • Publish the results!

Ideally, at least 2 different investigators will conduct steps 2–4 completely independently of each other, checking in after completion of each step and resolving any discrepancies, usually by consensus. xxiii   This provides a check against un- or subconscious bias on the part of the authors (remember: we’re all human and therefore all biased). For systematic reviews and meta-analyses conducted after 2015 or so, the protocol for the review (search strategy, exact topic, etc.) should be registered prior to step 2 with a central registry, such as PROSPERO . This provides a check against bias—authors who deviate from their preregistered protocols should provide very good reasons for doing so, and such studies should be interpreted with extreme caution.

Results from meta-analyses are often presented as forest plots that plot each included study’s main result (with the size of the square corresponding to sample size) and an overall estimate of association is indicated as a diamond at the bottom. Here is an example from a meta-analysis of chocolate consumption and systolic blood pressure (SBP, the top number in a blood pressure reading):

case control study multiple outcomes

Figure 9-9. Adapted from Reid et.al., BMC Medicine 2010

You can see from this forest plot that the majority of studies showed a decrease in SBP for people who ate more chocolate, though not all studies found this. Some point estimates are quite close to 0.0 (which is the “null” value here, because we’re looking at change in a single number, not a ratio), and 10 of the confidence intervals cross 0.0, indicating that they are not statistically significant. Nonetheless, 6 studies–the largest studies, since their confidence intervals are the narrowest– are statistically significant, and all of these in the direction of chocolate being beneficial. Indeed, the overall (or “pooled”) change in SBP and 95% CI shown at the bottom (the black diamond) indicates a small (approximately 3 mm Hg) reduciton in SBP for chocolate consumers. Does this mean we should all start eating lots of chocolate? Not necessarily:  a 3 mm Hg (“millimeters of mercury”–still the units in which we measure blood pressure, despite mercury not being involved for several decades now) drop in SBP is not clinically significant. A normal SBP is between 90 and 120, so a 3 mm Hg drop puts you at 87-117–likely not even a noticeable physiologic change.

As alluded to above, meta-analysis requires a certain similarity among the studies that will be pooled (e.g., they need to control for similar, if not identical, confounders). Often, this is not the case for a given body of literature—in which case, the authors will systematically examine all the evidence and do their best to come up with “an” answer, taking into consideration the quality of individual studies, the overall pattern of results, and so on. For example, in a systematic review of risk-reducing mastectomy (RRM), the prophylactic surgical removal of breasts in women who do not yet have breast cancer, but who have the BRCA-1 or BRCA-2 genes and thus are at very high risk (where BRRM refers to bilateral RRM—having both breasts removed). The authors described the overall results of this study as follows:

Twenty-one BRRM studies looking at the incidence of breast cancer or disease-specific mortality, or both, reported reductions after BRRM, particularly for those women with BRCA1/2 mutations.…Twenty studies assessed psychosocial measures; most reported high levels of satisfaction with the decision to have RRM but greater variation in satisfaction with cosmetic results. Worry over breast cancer was significantly reduced after BRRM when compared both to baseline worry levels and to the groups who opted for surveillance rather than BRRM, but there was diminished satisfaction with body image and sexual feelings. xxvii (p2)

The authors then concluded:

While published observational studies demonstrated that BRRM was effective in reducing both the incidence of, and death from, breast cancer, more rigorous prospective studies are suggested. [Because of risks associated with this surgery] BRRM should be considered only among those at high risk of disease, for example, BRCA1/2 carriers. xxvii (p2)

No overall “pooled” estimate of the protective effect associated with RRM is provided, but the authors are nonetheless able to convey the overall state of the literature, including where the body of literature is lacking.

Systematic reviews and meta-analyses are excellent resources for learning about a topic. Realistically, no one has the time to keep up with the literature in anything other than a very narrow topic area, and even then it is really only a boon to researchers in that field to take note of new individual studies. For public health professionals and clinicians not routinely engaging in research, relying on systematic reviews and meta-analyses provides a much better overall picture that is potentially less prone to the biases found in individual studies. However, care must be taken to read well-done reviews. The title of the paper should include either systematic review or meta-analysis , and the methods should mirror those outlined above. Be wary of review papers that are not explicitly systematic—they are extremely prone to biases on the part of the authors and probably should be ignored. [1]

Conclusions

The figure below is  a representation of the relative cost and internal validity of the study designs discussed in this chapter:

case control study multiple outcomes

There are many types of epidemiologic studies, from reports of a single, unusual patient up to formal meta-analyses of dozens of other studies. The relative validity of these in terms of using their evidence to shape policy varies widely, but with the exception of review papers, the “better” studies are the more expensive and time-consuming ones. Review papers in and of themselves are not particularly expensive, but they cannot be done until numerous other studies have been published, so if you include those as indirect costs, they take a lot of time and money. The 4 main study types (cross-sectional, case-control, cohort, and RCT) each have strengths and weaknesses, and readers of the epidemiologic literature should be aware of these. There are occasions, independent of cost or validity considerations, when one design or another is preferred (e.g., case-control for rare diseases).

i. Williams RA, Mamotte CD, Burnett JR. Phenylketonuria: an inborn error of phenylalanine metabolism. Clin Biochem Rev . 2008;29(1):31-41.

ii. Could where you live influence how long you live? RWJF. https://www.rwjf.org/en/library/interactives/whereyouliveaffectshowlongyoulive.html. Accessed February 19, 2019.

iii. A randomized trial of intensive versus standard blood-pressure control. N Engl J Med . 2017;377(25):2506. doi:10.1056/NEJMx170008

iv. Ridker PM, Cook NR, Lee I-M, et al. A randomized trial of low-dose aspirin in the primary prevention of cardiovascular disease in women. N Engl J Med . 2005;352(13):1293-1304. doi:10.1056/NEJMoa050613. ( ↵ Return 1 ) ( ↵ Return 2 )

v. Wacholder S, McLaughlin JK, Silverman DT, Mandel JS. Selection of controls in case-control studies, I: principles. Am J Epidemiol . 1992;135(9):1019-1028. ( ↵ Return )

vi. Wacholder S, Silverman DT, McLaughlin JK, Mandel JS. Selection of controls in case-control studies, II: types of controls. Am J Epidemiol . 1992;135(9):1029-1041. ( ↵ Return )

vii. Health CO on S and. Smoking and tobacco use: history of the Surgeon General’s Report. Centers for Disease Control and Prevention. 2017. http://www.cdc.gov/tobacco/data_statistics/sgr/history/. Accessed October 30, 2018. ( ↵ Return )

viii. Doll R, Hill AB. Smoking and carcinoma of the lung. Br Med J . 1950;2(4682):739-748. ( ↵ Return )

ix. Bowden K, Kessler D, Pinette M, Wilson E. Underwater birth: missing the evidence or missing the point? Pediatrics . 2003;112(4):972-973.

x. Centers for Disease Control and Prevention (CDC). A cluster of Kaposi’s sarcoma and Pneumocystis carinii pneumonia among homosexual male residents of Los Angeles and Orange counties, California.  MMWR Morb Mortal Wkly Rep . 1982;31(23):305-307.

xi. Cheyney M, Bovbjerg M, Everson C, Gordon W, Hannibal D, Vedam S. Outcomes of care for 16,924 planned home births in the United States: the Midwives Alliance of North America statistics project, 2004 to 2009. J Midwifery Womens Health . 2014;59(1):17-27. ( ↵ Return )

xii. Gregg NM. Congenital cataract following German measles in the mother. Trans Opthalmol Soc Aust . 1941;3:35-46. ( ↵ Return )

xiii. Greenberg M, Pellitteri O, Barton J. Frequency of defects in infants whose mothers had rubella during pregnancy. J Am Med Assoc . 1957;165(6):675-678. ( ↵ Return )

xiv. Manson M, Logan W, Loy R. Rubella and Other Virus Infections during Pregnancy . London: Her Royal Majesty’s Stationery Office; 1960. ( ↵ Return )

xv. Lundstrom R. Rubella during pregnancy: a follow-up study of children born after an epidemic of rubella in Sweden, 1951, with additional investigations on prophylaxis and treatment of maternal rubella. Acta Paediatr Suppl . 1962;133:1-110. ( ↵ Return )

xvi. Centers for Disease Control (CDC). Possible transfusion-associated acquired immune deficiency syndrome (AIDS)—California. MMWR Morb Mortal Wkly Rep . 1982;31(48):652-654. ( ↵ Return )

xvii. Centers for Disease Control (CDC). Pneumocystis pneumonia—Los Angeles. MMWR Morb Mortal Wkly Rep . 1981;30(21):250-252. ( ↵ Return )

xviii. Centers for Disease Control (CDC). Unexplained immunodeficiency and opportunistic infections in infants—New York, New Jersey, California. MMWR Morb Mortal Wkly Rep . 1982;31(49):665-667. ( ↵ Return )

xix. Centers for Disease Control and Prevention (CDC). Severe acute respiratory syndrome—Singapore, 2003. MMWR Morb Mortal Wkly Rep . 2003;52(18):405-411. ( ↵ Return )

xx. Centers for Disease Control and Prevention (CDC). Update: severe acute respiratory syndrome—United States, May 14, 2003. MMWR Morb Mortal Wkly Rep . 2003;52(19):436-438. ( ↵ Return )

xxi. Centers for Disease Control and Prevention (CDC). Cluster of severe acute respiratory syndrome cases among protected health-care workers—Toronto, Canada, April 2003. MMWR Morb Mortal Wkly Rep . 2003;52(19):433-436. ( ↵ Return )

xxii. Egger M, Smith GD, Altman DG, eds. Systematic Reviews in Health Care: Meta-analysis in Context . London: BMJ Publishing; 2001. ( ↵ Return 1 ) ( ↵ Return 2 )

xxiii. Harris JD, Quatman CE, Manring MM, Siston RA, Flanigan DC. How to write a systematic review. Am J Sports Med . 2014;42(11):2761-2768. doi:10.1177/0363546513497567 ( ↵ Return )

xxiv. Gilbert R, Salanti G, Harden M, See S. Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. Int J Epidemiol . 2005;34(4):874-887. doi:10.1093/ije/dyi088

xxv. CDC. Safe sleep for babies. Centers for Disease Control and Prevention. 2018. https://www.cdc.gov/vitalsigns/safesleep/index.html. Accessed January 10, 2019

xxvi. Task Force on Sudden Infant Death Syndrome, Moon RY. SIDS and other sleep-related infant deaths: expansion of recommendations for a safe infant sleeping environment. Pediatrics . 2011;128(5):1030-1039.

xxvii. Carbine NE, Lostumbo L, Wallace J, Ko H. Risk-reducing mastectomy for the prevention of primary breast cancer. Cochrane Database Syst Rev . 2018;4:CD002748. doi:10.1002/14651858.CD002748.pub4 ( ↵ Return 1 ) ( ↵ Return 2 )

  • Metasynthesis is a legitimate technique for systematic reviewing qualitative literature. The papers to watch out for are the ones called “integrative review,” “literature review,” or just “review”—anything that is not “systematic review.” ↵

An observational design. Usually prospective, in which case one selects a sample  of at-risk (non-diseased) people from the target population , assesses their exposure status, and then follows them over time looking for incident cases  of disease. Because we measure  incidence , the usual measure of association is either the risk ratio  or the rate ratio , though occasionally one will see odds ratios reported instead. If the exposure under study is common (>10%), one can just select a sample from the target population; however, if the exposure is rare, then exposed persons are sampled deliberately. (Cohort studies are the only design available for rare exposures.) This whole thing can be done in a retrospective manner if one has access to existing records (employment or medical records, usually) from which one can go back and "create" the cohort of at-risk folks, measure their exposure status at that time, and then "follow" them and note who became diseased.

The probability that your study will find something that is there. Power = 1 – β; beta is the  type II error  rate. Small studies, or studies of rare events, are typically under-powered.

High blood pressure, often abbreviated HTN.

The amount of time between an exposure and the biological onset of disease. Depending on the exposure/disease pair in question, can vary from minutes for some potent toxins to decades for many chronic diseases.

The amount of time between biological onset of disease and diagnosis. Depending on the disease, can be highly-variable in length, from hours to years. Duration of the latent period also varies depending on access to healthcare.

The extent to which a study’s methods are sufficiently correct that we can believe the findings as they apply that that study sample.

Treatment undertaken in an attempt to prevent a poor outcome. It is designed specifically to prevent, not to treat. For instance, in chapter 9, there is discussion of “risk-reducing mastectomy”—prophylactic removal of breasts in women at very high risk of breast cancer. The mastectomy occurs prior to the cancer, in an attempt to prevent the cancer from occurring. As another example, health care workers known to have been exposed to HIV (e.g., from an accidental needle stick) are offered prophylactic anti-retroviral drugs, in an attempt to prevent their bodies from seroconverting/becoming infected with HIV.

German measles.

A logical error that stems from applying group-level characteristics to individuals.

Bias  in the state of the literature on a particular topic that results from journals preferentially publishing papers with exciting results, rather than those showing no effect.

Foundations of Epidemiology Copyright © 2020 by Marit Bovbjerg is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Share This Book

  • En español – ExME
  • Em português – EME

Case-control and Cohort studies: A brief overview

Posted on 6th December 2017 by Saul Crandon

Man in suit with binoculars

Introduction

Case-control and cohort studies are observational studies that lie near the middle of the hierarchy of evidence . These types of studies, along with randomised controlled trials, constitute analytical studies, whereas case reports and case series define descriptive studies (1). Although these studies are not ranked as highly as randomised controlled trials, they can provide strong evidence if designed appropriately.

Case-control studies

Case-control studies are retrospective. They clearly define two groups at the start: one with the outcome/disease and one without the outcome/disease. They look back to assess whether there is a statistically significant difference in the rates of exposure to a defined risk factor between the groups. See Figure 1 for a pictorial representation of a case-control study design. This can suggest associations between the risk factor and development of the disease in question, although no definitive causality can be drawn. The main outcome measure in case-control studies is odds ratio (OR) .

case control study multiple outcomes

Figure 1. Case-control study design.

Cases should be selected based on objective inclusion and exclusion criteria from a reliable source such as a disease registry. An inherent issue with selecting cases is that a certain proportion of those with the disease would not have a formal diagnosis, may not present for medical care, may be misdiagnosed or may have died before getting a diagnosis. Regardless of how the cases are selected, they should be representative of the broader disease population that you are investigating to ensure generalisability.

Case-control studies should include two groups that are identical EXCEPT for their outcome / disease status.

As such, controls should also be selected carefully. It is possible to match controls to the cases selected on the basis of various factors (e.g. age, sex) to ensure these do not confound the study results. It may even increase statistical power and study precision by choosing up to three or four controls per case (2).

Case-controls can provide fast results and they are cheaper to perform than most other studies. The fact that the analysis is retrospective, allows rare diseases or diseases with long latency periods to be investigated. Furthermore, you can assess multiple exposures to get a better understanding of possible risk factors for the defined outcome / disease.

Nevertheless, as case-controls are retrospective, they are more prone to bias. One of the main examples is recall bias. Often case-control studies require the participants to self-report their exposure to a certain factor. Recall bias is the systematic difference in how the two groups may recall past events e.g. in a study investigating stillbirth, a mother who experienced this may recall the possible contributing factors a lot more vividly than a mother who had a healthy birth.

A summary of the pros and cons of case-control studies are provided in Table 1.

case control study multiple outcomes

Table 1. Advantages and disadvantages of case-control studies.

Cohort studies

Cohort studies can be retrospective or prospective. Retrospective cohort studies are NOT the same as case-control studies.

In retrospective cohort studies, the exposure and outcomes have already happened. They are usually conducted on data that already exists (from prospective studies) and the exposures are defined before looking at the existing outcome data to see whether exposure to a risk factor is associated with a statistically significant difference in the outcome development rate.

Prospective cohort studies are more common. People are recruited into cohort studies regardless of their exposure or outcome status. This is one of their important strengths. People are often recruited because of their geographical area or occupation, for example, and researchers can then measure and analyse a range of exposures and outcomes.

The study then follows these participants for a defined period to assess the proportion that develop the outcome/disease of interest. See Figure 2 for a pictorial representation of a cohort study design. Therefore, cohort studies are good for assessing prognosis, risk factors and harm. The outcome measure in cohort studies is usually a risk ratio / relative risk (RR).

case control study multiple outcomes

Figure 2. Cohort study design.

Cohort studies should include two groups that are identical EXCEPT for their exposure status.

As a result, both exposed and unexposed groups should be recruited from the same source population. Another important consideration is attrition. If a significant number of participants are not followed up (lost, death, dropped out) then this may impact the validity of the study. Not only does it decrease the study’s power, but there may be attrition bias – a significant difference between the groups of those that did not complete the study.

Cohort studies can assess a range of outcomes allowing an exposure to be rigorously assessed for its impact in developing disease. Additionally, they are good for rare exposures, e.g. contact with a chemical radiation blast.

Whilst cohort studies are useful, they can be expensive and time-consuming, especially if a long follow-up period is chosen or the disease itself is rare or has a long latency.

A summary of the pros and cons of cohort studies are provided in Table 2.

case control study multiple outcomes

The Strengthening of Reporting of Observational Studies in Epidemiology Statement (STROBE)

STROBE provides a checklist of important steps for conducting these types of studies, as well as acting as best-practice reporting guidelines (3). Both case-control and cohort studies are observational, with varying advantages and disadvantages. However, the most important factor to the quality of evidence these studies provide, is their methodological quality.

  • Song, J. and Chung, K. Observational Studies: Cohort and Case-Control Studies .  Plastic and Reconstructive Surgery.  2010 Dec;126(6):2234-2242.
  • Ury HK. Efficiency of case-control studies with multiple controls per case: Continuous or dichotomous data .  Biometrics . 1975 Sep;31(3):643–649.
  • von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Lancet 2007 Oct;370(9596):1453-14577. PMID: 18064739.

' src=

Saul Crandon

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

No Comments on Case-control and Cohort studies: A brief overview

' src=

Very well presented, excellent clarifications. Has put me right back into class, literally!

' src=

Very clear and informative! Thank you.

' src=

very informative article.

' src=

Thank you for the easy to understand blog in cohort studies. I want to follow a group of people with and without a disease to see what health outcomes occurs to them in future such as hospitalisations, diagnoses, procedures etc, as I have many health outcomes to consider, my questions is how to make sure these outcomes has not occurred before the “exposure disease”. As, in cohort studies we are looking at incidence (new) cases, so if an outcome have occurred before the exposure, I can leave them out of the analysis. But because I am not looking at a single outcome which can be checked easily and if happened before exposure can be left out. I have EHR data, so all the exposure and outcome have occurred. my aim is to check the rates of different health outcomes between the exposed)dementia) and unexposed(non-dementia) individuals.

' src=

Very helpful information

' src=

Thanks for making this subject student friendly and easier to understand. A great help.

' src=

Thanks a lot. It really helped me to understand the topic. I am taking epidemiology class this winter, and your paper really saved me.

Happy new year.

' src=

Wow its amazing n simple way of briefing ,which i was enjoyed to learn this.its very easy n quick to pick ideas .. Thanks n stay connected

' src=

Saul you absolute melt! Really good work man

' src=

am a student of public health. This information is simple and well presented to the point. Thank you so much.

' src=

very helpful information provided here

' src=

really thanks for wonderful information because i doing my bachelor degree research by survival model

' src=

Quite informative thank you so much for the info please continue posting. An mph student with Africa university Zimbabwe.

' src=

Thank you this was so helpful amazing

' src=

Apreciated the information provided above.

' src=

So clear and perfect. The language is simple and superb.I am recommending this to all budding epidemiology students. Thanks a lot.

' src=

Great to hear, thank you AJ!

' src=

I have recently completed an investigational study where evidence of phlebitis was determined in a control cohort by data mining from electronic medical records. We then introduced an intervention in an attempt to reduce incidence of phlebitis in a second cohort. Again, results were determined by data mining. This was an expedited study, so there subjects were enrolled in a specific cohort based on date(s) of the drug infused. How do I define this study? Thanks so much.

' src=

thanks for the information and knowledge about observational studies. am a masters student in public health/epidemilogy of the faculty of medicines and pharmaceutical sciences , University of Dschang. this information is very explicit and straight to the point

' src=

Very much helpful

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

""

Cluster Randomized Trials: Concepts

This blog summarizes the concepts of cluster randomization, and the logistical and statistical considerations while designing a cluster randomized controlled trial.

""

Expertise-based Randomized Controlled Trials

This blog summarizes the concepts of Expertise-based randomized controlled trials with a focus on the advantages and challenges associated with this type of study.

""

An introduction to different types of study design

Conducting successful research requires choosing the appropriate study design. This article describes the most common types of designs conducted by researchers.

Comparison of estimators in nested case–control studies with multiple outcomes

  • Published: 02 March 2012
  • Volume 18 , pages 261–283, ( 2012 )

Cite this article

  • Nathalie C. Støer 1 &
  • Sven Ove Samuelsen 1  

543 Accesses

28 Citations

Explore all metrics

Reuse of controls in a nested case–control (NCC) study has not been considered feasible since the controls are matched to their respective cases. However, in the last decade or so, methods have been developed that break the matching and allow for analyses where the controls are no longer tied to their cases. These methods can be divided into two groups; weighted partial likelihood (WPL) methods and full maximum likelihood methods. The weights in the WPL can be estimated in different ways and four estimation procedures are discussed. In addition, we address modifications needed to accommodate left truncation. A full likelihood approach is also presented and we suggest an aggregation technique to decrease the computation time. Furthermore, we generalize calibration for case-cohort designs to NCC studies. We consider a competing risks situation and compare WPL, full likelihood and calibration through simulations and analyses on a real data example.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Barlow WE (1994) Robust variance estimation for the case-cohort design. Biometrics 50(4): 1064–1072

Article   MATH   Google Scholar  

Borgan Ø, Goldstein L, Langholz B (1995) Methods for the analysis of samled cohort data in the Cox proportional hazards model. Ann Stat 23(5): 1749–1778

Article   MathSciNet   MATH   Google Scholar  

Breslow NE, Lumley T, Ballantyne CM, Chambless LE, Kulich M (2009a) Improved Horvitz–Thompson estimation of model parameters for two-phase stratified samples: applications in epidemiology. Stat Biosci 1(1): 32–49

Article   Google Scholar  

Breslow NE, Lumley T, Ballantyne CM, Chambless LE, Kulich M (2009b) Using the whole cohort in the analysis of case-cohort data. Am J Epidemiol 169(11): 1398–1405

Chen KN (2001) Generalized case-cohort sampling. J R Stat Soc B 63(4): 791–809

Deville JC, Särndal CE (1992) Calibration estimators in survey sampling. J Am Stat Assoc 87(418): 376–382

MATH   Google Scholar  

Deville JC, Särndal CE, Sautory O (1993) Generalized raking procedures in survey sampling. J Am Stat Assoc 88(423): 1013–1020

Kalbfleisch JD, Lawless JF (1988) Likelihood analysis of multi-state models for disease incidence and mortality. Stat Med 7(1–2): 149–160

Kulich M, Lin DY (2004) Improving the efficiency of relative-risk estimation in case-cohort studies. J Am Stat Assoc 99(467): 832–844

Lin DY, Wei LJ (1989) The robust inference for the Cox proportional hazards model. J Am Stat Assoc 84(408): 1074–1078

MathSciNet   MATH   Google Scholar  

Liu M, Lu W, Tseng CH (2010) Cox regression in nested case–control studies with auxiliary covariates. Biometrics 66(2): 374–381

Lumley T (2010) Complex surveys: a guide to analysis using R. Wiley series in survey mehodology. Wiley, Hoboken

Google Scholar  

Prentice RL (1986) A case-cohort design for epidemiologic cohort studies and disease prevention trials. Biometrika 73(1): 1–11

Saarela O, Kulathinal S (2007) Conditional likelihood inference in a case-cohort design: an application to haplotype analysis. Int J Biostat 3(1): 1

MathSciNet   Google Scholar  

Saarela O, Kulathinal S, Arjas E, Läärä E (2008) Nested case–control data utilized for multiple outcomes: a likelihood approach and alternatives. Stat Med 27(28): 5991–6008

Article   MathSciNet   Google Scholar  

Salim A, Hultman C, Sparén P, Reilly M (2009) Combining data from 2 nested case–control studies of overlapping cohorts to improve efficiency. Biostatistics 10(1): 70–79

Samuelsen SO (1997) A pseudolikelihood approach to analysis of nested case–control studies. Biometrika 84(2): 379–394

Samuelsen SO, Magnus P, Bakketeig LS (1998) Birth weight and mortality in childhood in Norway. Am J Epidemiol 148(10): 983–991

Samuelsen SO, Ånestad H, Skrondal A (2007) Stratified case-cohort analysis of general cohort sampling designs. Scand J Stat 34(1): 103–119

Scheike TH, Juul A (2004) Maximum likelihood estimation for Cox’s regression model under nested case–control sampling. Biostatistics 5(2): 193–206

Scott AJ, Wild CJ (1986) Fitting logistic models under case–control or choice based sampling. J R Stat Soc B 48(2): 170–182

Scott AJ, Wild CJ (1991) Fitting logistic regression models in stratified case–control studies. Biometrics 47(2): 497–510

Suissa S, Edwardes MD, Boivin JF (1998) External comparisons from nested-case control designs. Epidemiology 9(1): 72–78

Thomas DC (1977) Addendum to “methods of cohort analysis: appraisal by application to asbestos mining” by Liddell FDK, McDonald JC and Thomas DC. J R Stat Soc A 140: 469–491

Download references

Author information

Authors and affiliations.

Department of Mathematics, University of Oslo, P.O. Box 1053, 0316, Oslo, Norway

Nathalie C. Støer & Sven Ove Samuelsen

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Nathalie C. Støer .

Rights and permissions

Reprints and permissions

About this article

Støer, N.C., Samuelsen, S.O. Comparison of estimators in nested case–control studies with multiple outcomes. Lifetime Data Anal 18 , 261–283 (2012). https://doi.org/10.1007/s10985-012-9214-8

Download citation

Received : 29 April 2011

Accepted : 30 January 2012

Published : 02 March 2012

Issue Date : July 2012

DOI : https://doi.org/10.1007/s10985-012-9214-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Nested case–control
  • Competing risks
  • Weighted partial likelihood
  • Maximum likelihood for nested case–control
  • Calibration
  • Find a journal
  • Publish with us
  • Track your research

User Preferences

Content preview.

Arcu felis bibendum ut tristique et egestas quis:

  • Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris
  • Duis aute irure dolor in reprehenderit in voluptate
  • Excepteur sint occaecat cupidatat non proident

Keyboard Shortcuts

7.2.1 - case-cohort study design.

A case-cohort study is similar to a nested case-control study in that the cases and non-cases are within a parent cohort; cases and non-cases are identified at time \(t_1\), after baseline. In a case-cohort study, the cohort members were assessed for risk factors at any time prior to \(t_1\). Non-cases are randomly selected from the parent cohort, forming a subcohort. No matching is performed.

Advantages of Case-Cohort Study:

Similar to nested case-control study design:

  • Efficient– not all members of the parent cohort require diagnostic testing
  • Flexible– allows testing hypotheses not anticipated when the cohort was drawn \((t_0)\)
  • Reduces selection bias – cases and noncases sampled from the same population
  • Reduced information bias – risk factor exposure can be assessed with investigator blind to case status

Other advantages, as compared to nested case-control study design:

  • The subcohort can be used to study multiple outcomes
  • Risk can be measured at any time up to \(t_1\) (e.g. elapsed time from a variable event, such as menopause, birth)
  • Subcohort can be used to calculate person-time risk

Disadvantages of Case-Cohort Study:

As compared to nested case-control study design:

  • subcohort may have been established after \(t_0\)
  • exposure information collected at different times (e.g. potential for sample deterioration)

Statistical Analysis for Case-Cohort Study:

Weighted Cox proportional hazards regression model (we will look at proportional hazards regression later in this course)

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Case-control study of adverse childhood experiences and multiple sclerosis risk and clinical outcomes

Roles Conceptualization, Formal analysis, Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Division of Epidemiology and Biostatistics, Genetic Epidemiology and Genomics Laboratory, School of Public Health, University of California, Berkeley, CA, United States of America, Computational Biology Graduate Group, University of California, Berkeley, California, United States of America

ORCID logo

Roles Methodology, Software, Writing – review & editing

Affiliation California Institute for Quantitative Biosciences, University of California Berkeley, Berkeley, CA, United States of America

Roles Data curation, Methodology, Project administration, Resources, Software, Writing – review & editing

Affiliation Division of Epidemiology and Biostatistics, Genetic Epidemiology and Genomics Laboratory, School of Public Health, University of California, Berkeley, CA, United States of America

Roles Data curation, Project administration, Writing – review & editing

Affiliation Kaiser Permanente Division of Research, Oakland, CA, United States of America

Roles Data curation, Project administration, Resources, Writing – review & editing

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Writing – review & editing

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Writing – original draft, Writing – review & editing

  • Mary K. Horton, 
  • Shannon McCurdy, 
  • Xiaorong Shao, 
  • Kalliope Bellesis, 
  • Terrence Chinn, 
  • Catherine Schaefer, 
  • Lisa F. Barcellos

PLOS

  • Published: January 13, 2022
  • https://doi.org/10.1371/journal.pone.0262093
  • Reader Comments

Table 1

Adverse childhood experiences (ACEs) are linked to numerous health conditions but understudied in multiple sclerosis (MS). This study’s objective was to test for the association between ACEs and MS risk and several clinical outcomes.

We used a sample of adult, non-Hispanic MS cases (n = 1422) and controls (n = 1185) from Northern California. Eighteen ACEs were assessed including parent divorce, parent death, and abuse. Outcomes included MS risk, age of MS onset, Multiple Sclerosis Severity Scale score, and use of a walking aid. Logistic and linear regression estimated odds ratios (ORs) (and beta coefficients) and 95% confidence intervals (CIs) for ACEs operationalized as any/none, counts, individual events, and latent factors/patterns.

Overall, more MS cases experienced ≥1 ACE compared to controls (54.5% and 53.8%, respectively). After adjusting for sex, birthyear, and race, this small difference was attenuated (OR = 1.01, 95% CI: 0.87, 1.18). There were no trends of increasing or decreasing odds of MS across ACE count categories. Consistent associations between individual ACEs between ages 0–10 and 11–20 years and MS risk were not detected. Factor analysis identified five latent ACE factors, but their associations with MS risk were approximately null. Age of MS onset and other clinical outcomes were not associated with ACEs after multiple testing correction.

Despite rich data and multiple approaches to operationalizing ACEs, no consistent and statistically significant effects were observed between ACEs with MS. This highlights the challenges of studying sensitive, retrospective events among adults that occurred decades before data collection.

Citation: Horton MK, McCurdy S, Shao X, Bellesis K, Chinn T, Schaefer C, et al. (2022) Case-control study of adverse childhood experiences and multiple sclerosis risk and clinical outcomes. PLoS ONE 17(1): e0262093. https://doi.org/10.1371/journal.pone.0262093

Editor: Torsten Klengel, Harvard Medical School, UNITED STATES

Received: April 7, 2021; Accepted: December 16, 2021; Published: January 13, 2022

Copyright: © 2022 Horton et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly because there are several restrictions on sharing a de-identified data set from this study. When participants were enrolled in the study (8-15 years ago), they did not provide consent to having their individual-level data shared outside of Kaiser Permanente Northern California or UC Berkeley without the expressed permission of the study PIs and respective IRBs. This was due to the sensitive nature of some of the data collected (which included genetic data). This is enforced by both the Kaiser Permanente Northern California and UC Berkeley IRBs. Given this restriction, data requests may be sent to Lisa Barcellos ( [email protected] ) and Lynn Hollyer ( [email protected] ). Requests will be reviewed by the IRB and data may be shared upon approval.

Funding: This work was supported by the National Institute of Neurological Disorders and Stroke [grant numbers R01 NS 049510 to L.F.B. and F31 NS 108668 to M.K.H.] ( https://www.ninds.nih.gov/ ); the National Institute of Allergy and Infectious Diseases [grant number R01 AI 076544 to L.F.B.] ( https://www.niaid.nih.gov/ ); and National Institute of Environmental Health Sciences [grant number R01 ES 017080 to L.F.B.] ( https://www.niehs.nih.gov/ ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Adverse Childhood Experiences (ACEs) are potentially traumatic events that occur in childhood and can include physical, emotional, and sexual abuse and/or neglect and household disfunction [ 1 ]. They are common in the U.S.—occurring in about 58% of the population—and an important social determinant of health [ 2 ]. Childhood represents a particularly vulnerable period when body systems are developing. Excessive activation of stress response systems during this period can impact brain development, immunity, metabolic regulatory systems, and the cardiovascular system [ 3 ]. A large body of literature has linked ACEs to physical and mental health conditions in adulthood including heart disease, obesity, type 2 diabetes, cancer, and depression [ 4 ].

One particularly relevant downstream effect of excessive activation of stress response systems is dysregulation of the immune system. Numerous studies have shown in experimental and observational settings that psychosocial stressors can cause persistent inflammation and suppression of anti-inflammatory compounds [ 5 – 7 ]. Dysregulation of the immune system can lead to many serious health conditions, including autoimmune conditions such as multiple sclerosis (MS), lupus, and rheumatoid arthritis. The literature regarding the effects of ACEs on autoimmune disorders is limited but suggests an increased numbers of ACEs are associated with increased risk of autoimmune conditions overall and individually [ 8 ]. More studies are needed to fully understand this relationship, particularly among individual autoimmune conditions.

MS is one autoimmune condition where more work on this topic is needed. MS is a chronic, inflammatory autoimmune condition of the central nervous system and is the second most common neurological disorder among young adults [ 9 , 10 ]. Diagnosis is common relatively early in adulthood (ages 20 to 40 years) and among women (3:1 female-to-male ratio). Several studies have shown that risk factors (e.g., obesity, concussion, Epstein-Barr virus infection, and vitamin D/sun exposure), particularly during adolescence (ages 11–20 years), are associated with increased MS risk [ 11 , 12 ]. Given the relatively young age of MS diagnosis, support for adolescent exposures being important for MS risk, and the critical involvement of inflammation in MS disease processes, determining whether a role exists for ACEs in MS risk is important. Of the few studies that have examined the association between ACEs and MS risk, their results are inconsistent. Findings from the U.S.-based Nurses’ Health Study, which asked adult participants to quantify “physical or sexual abuse in childhood or adolescence”, suggested MS risk was not significantly associated with abuse [ 13 ]. A Danish study found that parent divorce, but not parent or sibling death, was associated with risk of MS [ 14 ]. A German study using a 28-item self-report questionnaire of childhood maltreatment found an increased risk of MS among domains of physical abuse, sexual abuse, emotional neglect, and severe abuse [ 15 ]. Inconsistencies in these findings are likely due to differences in specific ACEs and how they were quantified, as well as differing cultural and social contexts in each population, underscoring the challenges of this important work and need for further investigation.

There is even less knowledge about whether ACEs affect clinical outcomes of MS which may be influenced by early life stress and inflammation. The largest study, to date, to investigate this association utilized 217 MS cases and determined that physical abuse, emotional neglect, and severe abuse were associated with higher relapse rates but not age of onset or other indicators of physical or cognitive outcomes [ 15 ]. The only other (smaller) study to investigate this found that more ACEs were associated with younger age of MS onset and worse reading cognition [ 16 ]. Understanding the relationship between ACEs and MS risk and clinical outcomes may strengthen the argument for childhood screening of ACEs and interventions that prevent or modify the effects of ACEs and improve our understanding of MS etiology.

Our approach to studying ACEs was to interrogate how they might affect MS risk and clinical outcomes using multiple methodologies. For our study, ACEs included death of a parent or sibling, victim of a violent crime, loss of a home, and significant physical or verbal abuse or neglect, among others. It is common to analyze ACEs as individual events or summarized into any/none or count variables; however, these have several limitations. It is possible that individual ACEs (such as parent divorce shown by Riise et.al) may have different effects on MS risk, but ACEs (and social exposures more broadly) often co-occur and are not necessarily independent [ 17 ]. This limits the interpretability of assessing single events that are highly inter-related. In addition, single events may be rare and limit our power to examine associations with MS in all but very large studies. Quantifying ACEs as any/none may be meaningful if the hypothesis is that any adverse event impacts health. However, this dichotomy fails to consider the relative importance of different types of ACEs with varying impacts on chronic stress or behaviors and thus MS. The use of counts assumes the cumulative burden of ACEs affects health, rather than particular type, combination, or chronicity. These limitations highlight the challenges in studying ACEs and the need to consider them in multiple ways in order to understand their complex, nuanced relationships with health outcomes, particularly MS.

The aim of the current study was to estimate the association between ACEs and MS risk and several clinical outcomes, including age of onset, use of a walking aid, and Multiple Sclerosis Severity Scale score, in a case-control sample of 2607 adults in Northern California using multiple approaches including quantifying ACEs as individual events, any/none, and counts. We also included a factor analysis to evaluate variance of ACEs in order to identify “latent factors”, which are weighted linear combinations of variables, that represent patterns of ACEs that tend to co-occur. Collectively, this approach may help identify how ACEs are associated with MS.

Study population

Data were from the Kaiser Permanente Northern California (KPNC) MS Research Program which recruited non-Hispanic MS cases and controls from the KPNC Health Plan between 2006 and 2014. This membership includes over four million people, representing 25–30% of the 22-county service area population in Northern California. The broad goal of this study was to assess risk factors for MS across hundreds of genetic and environmental exposures. To achieve sufficient power for genetic analyses, study inclusion was limited to the largest subgroup of KPNC members which were largely non-Hispanic whites. Recruitment details are explained elsewhere [ 18 ]. Briefly, eligible cases were diagnosed with MS by a neurologist ( International Classification of Diseases , Ninth Revision , code 340.x), aged 18–69 years old, and a KPNC member at initial contact. For our analyses, cases were excluded if age of onset occurred before age 21 years to minimize the potential for reverse causality or MS onset occurring before ACEs (assessed up to age 20). Age of onset was determined by review of electronic health records and comprehensive interview data. Controls were KPNC members without a MS diagnosis or related condition (optic neuritis, transverse myelitis, or demyelinating disease) and were matched to cases by sex, age, and zip code. A total of 2607 participants (1422 cases and 1185 controls) were available for analyses.

Study protocols for participants were approved by the Institutional Review Boards of KPNC and the University of California, Berkeley. Written informed consent was obtained from all study participants.

Adverse Childhood Experiences (ACEs)

Participants were administered a comprehensive computer-assisted telephone interview (CATI) including hundreds of self-reported demographic, clinical, environmental, and lifestyle questions, as described elsewhere [ 19 ]. The CATI included nine ACE questions modified from Coddington’s Life Event Record [ 20 ] ( Table 1 ). Not all questions were included to reduce the length of the extensive CATI. Events included broadly overlap with the original Centers for Disease Control and Prevention-Kaiser ACE Study [ 1 ], but there are several differences. Our study does not ask about sexual abuse, household substance abuse, or incarcerated household members. It also combines physical and verbal abuse and adds questions about parent/sibling death and foster care/adoption. Participants indicated yes/no as to whether they experienced any of the events in either of two age periods: 0–10 and 11–20 years old (a total of 18 ACEs). These two time periods were chosen because studies have shown that relationships between several risk factors and MS differ in adolescence and childhood [ 11 , 12 ].

thumbnail

  • PPT PowerPoint slide
  • PNG larger image
  • TIFF original image

https://doi.org/10.1371/journal.pone.0262093.t001

MS clinical outcomes

As part of the CATI, MS cases were asked the year of their first MS symptom (i.e., “onset”), the type of MS they currently have (relapsing-remitting, secondary progressive, primary progressive, or relapsing-progressive), and an indication of their walking ability in the past four weeks. For each MS case, we calculated the Multiple Sclerosis Severity Scale (MSSS), which is an indicator of disease severity that uses the Expanded Disability Severity Scale and disease duration (time from onset to EDSS) [ 21 ]. We also created an indicator of whether a case had severe or mild MS based on MSSS scores (≥7.5 was severe and <3 was mild).

Demographic and clinical data collected from the CATI and considered confounders included sex, birth year, race, and years since MS onset. Race was categorized as white or non-white, noting 98.5% of non-whites identified as African American. Additional confounders considered in sensitivity analyses (see below) included education-level (bachelor’s degree or not), parent’s homeowner status when participant was 10 years old (rent vs own/other), and family history of MS (parent or sibling). These were not included in primary/secondary analyses to preserve statistical power and prevent over-stratification of models with already low frequency substrata (including rare ACEs, men, and non-whites).

Statistical analysis

Factor analysis was conducted among all participants to determine the latent factor structure of 18 total ACEs (nine ACEs at two time points). A tetrachoric correlation matrix, appropriate for binary data, was constructed. Zero-count cells were corrected by adding 0.1. Factors were extracted using maximum likelihood estimation in the polycor package and factanal in R Version 3.5 [ 22 ]. VARIMAX (orthogonal) rotation was used to increase interpretability of factors. Number of factors to extract was based on optimal coordinates and reduced if any factor loading was ≥1.0 [ 23 ]. Factor scores were calculated and standardized to a mean of zero and standard deviation of one [ 24 ].

Primary analyses tested the association between ACEs and MS risk using logistic regression to estimate odds ratios (ORs) and 95% confidence intervals (CIs). ACEs were expressed as: 1) any/none at each time period and overall, 2) 0, 1, 2, 3, or 4 or more at each time period and overall, 3) individually at each time period and overall, and 4) continuously for each factor score. The associations between individual ACEs and MS risk were only estimated for ACEs that occurred in at least 5% of the sample in order to achieve sufficient statistical power. To improve interpretability of ORs from models using continuous factor scores (where a 1-unit increase in respective factor score would represent nearly the entire range of values), beta coefficients and their standard errors were divided by ten. All models adjusted for sex, birthyear, and race. Multiple testing corrected false discovery rate (FDR) q values are presented for primary analyses [ 25 ]; they account for all primary models assessing MS risk simultaneously. All analyses used R Version 3.5 [ 22 ].

Secondary analyses investigated the association between ACEs and clinically relevant MS outcomes including MSSS, age of onset, progressive MS subtype, and current walking ability. We also included a sub-analysis comparing ACEs among individuals with mild and severe MS. For MSSS and age of onset outcomes, linear regression models were used to estimate beta coefficients and 95% CIs. Both models adjusted for sex and race. MSSS models additionally adjusted for birthyear. Age of onset was approximately normally distributed while MSSS was slightly right-skewed. We also conducted a sub-analysis utilized individuals only at the extreme ends of the MSSS scale (n = 818) where the outcome was severe or mild (reference) illness. For MS subtype, type of MS was categorized as relapsing (relapsing remitting or secondary progressive) (reference) or progressive (primary progressive or relapsing progressive). For current walking ability, individuals were classified according to whether they did or did not (reference) regularly use a walking aid (such as cane, walker, or wheelchair). For all binary outcomes, ORs and 95% CIs were estimated using logistic regression and adjusted for birthyear, sex, and race. Walking ability models additionally adjusted for years since MS onset. For all MS outcome models, ACEs were considered the independent variable and expressed as count categories (0, 1, 2, 3, or 4 or more) over the entire exposure period (0 through 20 years of age). Additional ACE classifications were not included to minimize the impact of multiple testing corrections on a reduced sample size (1422 MS cases). Results from secondary analyses were corrected for FDR and account for all secondary clinical outcome tests.

Sensitivity analyses

To evaluate whether socioeconomic factors independent of race might confound the observed primary associations between ACEs and MS risk, we included two additional logistic regression models which adjust for covariates in the original models plus 1) participant’s educational level or 2) parent’s homeowner status when participants were 10 years old and family history of MS. Family history was considered a potential confounder because the risk of MS is ~seven times higher among those who have a first degree relative with MS [ 26 ] and it may be a cause of parent or sibling illness or death (an ACE in our assessment).

Baseline characteristics were described in Table 2 . Among MS patients, 79.0% identified as female (81.5% for controls). The average years since MS onset was 17.1 (sd = 11.8), and the majority of MS cases had mild illness (MSSS <3) (47.6%). Cases had higher frequency of family history of MS (6.8%) compared to controls (1.6%), as expected. When participants were 10 years old, fewer parents of MS cases owned a home compared to controls (78.0% and 81.9%, respectively), as previously reported [ 19 ].

thumbnail

https://doi.org/10.1371/journal.pone.0262093.t002

The proportion of participants who experienced ≥1 ACE was higher among cases (54.5%) compared to controls (53.8%) ( Table 2 ). Among the entire sample, the most common ACE during ages 0–10 years was significant physical abuse/neglect (12.1%); it was also the most common ACE during ages 11–20 year (14.9%) ( Table 1 ). The distribution of individual ACEs was similar among cases and controls although fewer cases reported significant physical abuse/neglect or home loss during ages 0–10 years.

Overall, individuals who reported at least one ACE between ages 0–20 years did not have a significantly higher odds of MS compared to individuals who experienced none (OR = 1.01, 95% CI: 0.87, 1.18) ( Table 3 ). A similar non-significant effect was also observed for each age category separately. When ACE counts were categorized into 0, 1, 2, 3, or 4 or more, none of the categories were significantly associated with MS and there were no consistent trends where increased ACEs increased or decreased odds of MS. No individual ACEs were significantly associated with MS at an FDR q <0.05 except abuse (OR = 0.66, 95% CI: 0.52, 0.84) and home loss (OR = 0.61, 95% CI: 0.45, 0.82) between ages 0–10 years. These effect sizes were attenuated and not statistically significant at ages 11–20 years (OR abuse = 0.87, 95% CI: 0.70, 1.08 and OR home loss = 0.96, 95% CI: 0.71, 1.30). For secondary analyses pertaining to ACEs and clinical outcomes of MS, no associations were significant at FDR q <0.05 ( Table 4 ). Before adjusting for multiple testing comparisons, two associations were significant at p <0.05. These included a two year younger age of onset, on average, for MS cases who experienced at least four ACEs compared to those who experienced no ACEs (β = -1.99, 95% CI: -3.62, -0.37, p = 0.02), and a higher odds of needing to regularly use a walking aid among MS cases who experienced at least four ACEs compared to MS cases who experienced no ACEs (OR = 1.52, 95% CI: 1.03, 2.24, p = 0.03).

thumbnail

https://doi.org/10.1371/journal.pone.0262093.t003

thumbnail

https://doi.org/10.1371/journal.pone.0262093.t004

Optimal coordinates analysis identified five factors of co-occurring ACEs which explained 57.0% of the variance in 18 reported ACEs ( S1 Table ). For each factor, the following ACEs contributed the largest loadings: lost home or moved ages 0–10 and 11–20 years (Factor 1), parent divorce and parent remarriage ages 0–10 (Factor 2), physical or verbal abuse or neglect ages 0–10 and 11–20 years (Factor 3), placed in foster care and parents divorced ages 11–20 years (Factor 4), and parent or sibling death ages 0–10 years (Factor 5). Logistic regression using continuous factor scores did not yield statistically significant results ( Table 3 ). For all factors, a 0.1-unit increase in factor score had very small or null association with MS risk (e.g., Factor 1 OR = 0.98, 95% CI: 0.95, 1.02).

Sensitivity analyses for MS risk models yielded ORs and 95% CIs that did not substantially change when models additionally controlled for participant’s educational attainment, parent’s homeowner status, or family history of MS ( S2 and S3 Tables).

ACEs are associated with numerous adult health conditions [ 4 ], but the relationship between ACEs and MS has remained elusive. Understanding this relationship may be particularly relevant because one hypothesized biological mechanism linking ACEs and general poor adult health is inflammation [ 27 ], a key cause of neuronal damage in MS. Despite rich data and multiple approaches for operationalizing ACEs in the current study, no consistent and statistically significant effects were observed between ACEs with MS risk and clinical outcomes after correcting for multiple testing comparisons. This highlights the challenges of studying sensitive, retrospective events among adults that occurred decades before data collection. It also underscores the need for ACE assessments early in the MS disease course to overcome some of these challenges.

Our primary findings, which do not support the role of ACEs in risk of MS, both agree with and contradict past studies of MS and autoimmune disorders. Results from a large cohort study of U.S. nurses did not identify associations between MS and stressful life events, including physical and/or sexual abuse during childhood or adolescence [ 13 ]. Corresponding odds ratios ranged from 0.72 to 1.30 but were not statistically significant, which may be due to the small number of MS cases identified from the large cohort (n = 369). These findings align with the magnitude and insignificant nature of the current findings. Similar to our results, a Danish study (the largest study to date) found that risk of MS was not associated with parent death (OR = 1.04, 95% CI: 0.90, 1.21) or sibling death (OR = 1.04, 95% CI: 0.81, 1.32) [ 14 ]. However, this study did observe that parent divorce, specifically, was associated with increased risk of MS (OR = 1.13, 95% CI: 1.04, 1.23), which is not consistent with our results. Their results are likely highly accurate given that Danish registries capture all family relations and marital statuses for all Danish residents and capture all MS diagnoses since 1956. However, social structures, levels of inequities, and the demographic make-up of Denmark and the U.S. are very different, so these adverse events might not be expected to have the same effects in both countries. Our findings pertaining to physical abuse (and home loss) demonstrated a significant protective effect during childhood, but there is no reason to believe that physical abuse or home loss, but not other ACEs, would prevent MS. In fact, previous research contradicts this finding which identified an increased risk of MS among those who have experienced severe abuse (OR = 1.7) and null associations between physical abuse or neglect and MS risk [ 15 ]. Similarly, latent factors 1 or 3 were not associated with MS risk despite being the factors for which childhood abuse and home loss contributed the most.

Among other autoimmune conditions, increasing number of ACEs have been associated with first hospitalization of any autoimmune disease as well as rheumatic, Th1-type and Th2-type immunopathologies, and Systemic Lupus Erythematosus (SLE) [ 8 , 28 ]. In particular, physical and emotional abuse have been shown to be associated with over two times the risk of SLE [ 8 ]. These were not found to be associated in our study. The differing results may be a result of different associations between ACEs and specific autoimmune conditions or insufficient statistical power, measurement error, or selection bias within our study or others.

Our findings that a younger age of onset and regular use of a walking aid were more common among MS cases that had at least four ACEs were not significant after correcting for multiple testing comparisons. Current research on this topic is very limited, with only two small studies reporting their findings. Of these, age of onset was found to be inversely correlated with ACEs (r = −0.30, p = 0.04) [ 16 ] or not associated with ACEs [ 15 ]. In another autoimmune condition, SLE, higher ACE levels and ACE domains were associated with worse patient-reported disease activity, depression, and health status [ 29 ]. It is important to note that our analysis did not have available comprehensive clinical outcomes data; therefore, only several features were assessed. Additional analyses considering relapse rate, neuroimaging measures, symptom burden, fatigue, pain, cognitive impairment, health-related quality of life, and psychological impacts might reveal meaningful associations with ACEs and should be conducted in the future. Our findings should be explored further in a larger sample size to improve statistical power to identify whether a true relationship exists between clinical features of MS and ACEs.

A major challenge that may have contributed to inconsistencies between our results and other studies, as well as our generally null observed effects, is information bias. Particularly, retrospectively asking adults about ACEs that occurred decades in the past that are sensitive by nature and may be misremembered or repressed from memory could have led to underreporting. Comparing the frequency of several of our study’s ACEs to those in the Behavioral Risk Factor Surveillance System (BRFSS) (derived from the Kaiser-CDC ACEs study) provides evidence of this underreporting. For example, 28% and 34% of individuals in the BRFSS had their parents’ divorce/separate and experienced emotional abuse while 19% and 18% experienced these ACEs in our sample, respectively [ 30 ]. Recall of sensitive events may have been under-reported, specifically, among cognitively impaired MS patients. However, this is not consistent with knowledge that cognitive MS symptoms do not commonly affect recall of memories from the distant past but rather lead to trouble with recall due to deficits in ability to store new knowledge for future recall [ 31 , 32 ]. Alternatively, MS cases may have interpreted questions regarding home loss or abuse more conservatively than controls, not willing to report the event unless they considered it an extreme circumstance. This is unlikely given “recall bias” which often, but not always, leads to more accurate recall of particular events/exposures among case groups than control groups.

In addition to this potential retrospective reporting bias, there are several limitations that should be considered. First, the events utilized in this dataset are not, together, part of a standardized ACE index. Compared to the BRFSS, our events similarly included parent divorce/separation, but did not include substance use, parent incarceration, or sexual abuse. Exclusion of these sensitive, important topics may have contributed to observed null findings. This is particularly relevant given that household substance abuse is one of the more common ACEs in the BRFSS (26.8% reported experiencing this) [ 2 ]. Combining physical and verbal abuse into a single category may also have underestimated the impact of ACEs in our sample. We did, however, include important events not part of the BRFSS survey including parent death and life-threatening illness of parent or sibling. Second, using ACEs is an imperfect way of measuring childhood adversity. Individual events tend to be interrelated and the social environment and factors that may influence it are complex and challenging to disentangle. To improve upon individual ACE analyses (which also may suffer from reduced statistical power due to rarity of certain events), we utilized factor analysis to create unobserved “latent” variables to capture the relatedness of ACEs. The observed associations between each latent variable and MS risk were approximately null, but the extent to which these factors might represent true unobserved continuous variables remains unknown. These five factors captured a relatively small amount of variation in ACEs (57%), which also limits the effectiveness of estimating their associations with MS. Last, low income and African American individuals disproportionately experience a number of adverse experiences [ 30 , 33 ]. This demographic is under-represented in the current sample which may lead to limited generalizations of findings to more diverse populations or selection bias. It may also have led to the observed null findings given African Americans tend to have worse MS clinical outcomes compared to Whites [ 34 ]. Future studies should further explore relationships between ACEs and MS among African Americans, Hispanics, Asians, and other non-White populations. This work is currently underway. Future studies should also investigate the nuanced synergistic and/or cumulative relationships between ACEs, socioeconomic position, and MS. For example, the effect of ACEs on MS may be stronger among individuals whose parents rented rather than owned a home (indicator of socioeconomic position and associated with MS) or among those who also experienced stressful events as adults later in the lifespan [ 19 ].

Conclusions

Findings from the current study did not support an association between ACEs and development of MS or clinical feature of MS. While we cannot exclude the potential role of ACEs on MS, our results highlight how poor recall or even recall bias for reporting sensitive events in the past may be particularly challenging to overcome in the context of MS. Future studies should consider alternative tools for assessing ACEs and childhood trauma, such as biomarkers of stress, and/or obtain ACE information from MS patients as close to diagnosis as possible to reduce the number of years between exposure and outcome.

Supporting information

S1 table. factor loadings for a 5-factor model based on adverse childhood experiences data from the kaiser permanente northern california multiple sclerosis research program cases and controls, 2006–2014 (n = 2,607)..

https://doi.org/10.1371/journal.pone.0262093.s001

S2 Table. Sensitivity analysis of multivariable logistic regression models of the effect of adverse childhood experiences (ACEs) during two age periods on odds of multiple sclerosis accounting for educational attainment.

https://doi.org/10.1371/journal.pone.0262093.s002

S3 Table. Sensitivity analysis of multivariable logistic regression models of the effect of adverse childhood experiences (ACEs) during two age periods on odds of multiple sclerosis (MS) accounting for parent homeowner status and family history of MS.

https://doi.org/10.1371/journal.pone.0262093.s003

Acknowledgments

We thank all members and staff of the Kaiser Permanente Division of Research and the University of California, Berkeley, Genetic Epidemiology and Genomics Laboratory.

  • View Article
  • PubMed/NCBI
  • Google Scholar
  • 10. Atlas of MS 2013: Mapping multiple sclerosis around the world. London; 2013. Available: ttp:// www.msif.org/about-ms/publications-and-resources .
  • 22. R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2018.
  • Open access
  • Published: 19 February 2024

Sustaining the collaborative chronic care model in outpatient mental health: a matrixed multiple case study

  • Bo Kim 1 , 2 ,
  • Jennifer L. Sullivan 3 , 4 ,
  • Madisen E. Brown 1 ,
  • Samantha L. Connolly 1 , 2 ,
  • Elizabeth G. Spitzer 1 , 5 ,
  • Hannah M. Bailey 1 ,
  • Lauren M. Sippel 6 , 7 ,
  • Kendra Weaver 8 &
  • Christopher J. Miller 1 , 2  

Implementation Science volume  19 , Article number:  16 ( 2024 ) Cite this article

34 Accesses

1 Altmetric

Metrics details

Sustaining evidence-based practices (EBPs) is crucial to ensuring care quality and addressing health disparities. Approaches to identifying factors related to sustainability are critically needed. One such approach is Matrixed Multiple Case Study (MMCS), which identifies factors and their combinations that influence implementation. We applied MMCS to identify factors related to the sustainability of the evidence-based Collaborative Chronic Care Model (CCM) at nine Department of Veterans Affairs (VA) outpatient mental health clinics, 3–4 years after implementation support had concluded.

We conducted a directed content analysis of 30 provider interviews, using 6 CCM elements and 4 Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) domains as codes. Based on CCM code summaries, we designated each site as high/medium/low sustainability. We used i-PARIHS code summaries to identify relevant factors for each site, the extent of their presence, and the type of influence they had on sustainability (enabling/neutral/hindering/unclear). We organized these data into a sortable matrix and assessed sustainability-related cross-site trends.

CCM sustainability status was distributed among the sites, with three sites each being high, medium, and low. Twenty-five factors were identified from the i-PARIHS code summaries, of which 3 exhibited strong trends by sustainability status (relevant i-PARIHS domain in square brackets): “Collaborativeness/Teamwork [Recipients],” “Staff/Leadership turnover [Recipients],” and “Having a consistent/strong internal facilitator [Facilitation]” during and after active implementation. At most high-sustainability sites only, (i) “Having a knowledgeable/helpful external facilitator [Facilitation]” was variably present and enabled sustainability when present, while (ii) “Clarity about what CCM comprises [Innovation],” “Interdisciplinary coordination [Recipients],” and “Adequate clinic space for CCM team members [Context]” were somewhat or less present with mixed influences on sustainability.

Conclusions

MMCS revealed that CCM sustainability in VA outpatient mental health clinics may be related most strongly to provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators. These findings have informed a subsequent CCM implementation trial that prospectively examines whether enhancing the above-mentioned factors within implementation facilitation improves sustainability. MMCS is a systematic approach to multi-site examination that can be used to investigate sustainability-related factors applicable to other EBPs and across multiple contexts.

Peer Review reports

Contributions to the literature

We examined the ways in which the sustainability of the evidence-based Collaborative Chronic Care Model differed across nine outpatient mental health clinics where it was implemented.

This work demonstrates a unique application of the Matrixed Multiple Case Study (MMCS) method, originally developed to identify factors and their combinations that influence implementation, to investigate the long-term sustainability of a previously implemented evidence-based practice.

Contextual influences on sustainability identified through this work, as well as the systematic approach to multi-site examination offered by MMCS, can inform future efforts to sustainably implement and methodically evaluate an evidence-based practice’s uptake and continued use in routine care.

The sustainability of evidence-based practices (EBPs) over time is crucial to maximize the public health impact of EBPs implemented into routine care. Implementation evaluators focus on sustainability as a central implementation outcome, and funders of implementation efforts seek sustained long-term returns on their investment. Furthermore, practitioners and leadership at implementation sites face the task of sustaining an EBP’s usage even after implementation funding, support, and associated evaluation efforts conclude. The circumstances and influences contributing to EBP sustainability are therefore of high interest to the field of implementation science.

Sustainability depends on the specific EBP being implemented, the individuals undergoing the implementation, the contexts in which the implementation takes place, and the facilitation of (i.e., support for) the implementation. Hence, universal conditions that invariably lead to sustainability are challenging to establish. Even if a set of conditions could be identified as being associated with high sustainability “on average,” its usefulness is questionable when most real-world implementation contexts may deviate from “average” on key implementation-relevant metrics.

Thus, when seeking a better understanding of EBP sustainability, there is a critical need for methods that examine the ways in which sustainability varies in diverse contexts. One such method is Matrixed Multiple Case Study (MMCS) [ 1 ], which is beginning to be applied in implementation research to identify factors related to implementation [ 2 , 3 , 4 , 5 ]. MMCS capitalizes on the many contextual variations and heterogeneous outcomes that are expected when an EBP is implemented across multiple sites. Specifically, MMCS provides a formalized sequence of steps for cross-site analysis by arranging data into an array of matrices, which are sorted and filtered to test for expected factors and identify less expected factors influencing an implementation outcome of interest.

Although the MMCS represents a promising method for systematically exploring the “black box” of the ways in which implementation is more or less successful, it has not yet been applied to investigate the long-term sustainability of implemented EBPs. Therefore, we applied MMCS to identify factors related to the sustainability of the evidence-based Collaborative Chronic Care Model (CCM), previously implemented using implementation facilitation [ 6 , 7 , 8 ], at nine VA medical centers’ outpatient general mental health clinics. An earlier interview-based investigation of CCM provider perspectives had identified key determinants of CCM sustainability at the sites, yet characteristics related to the ways in which CCM sustainability differed at the sites are still not well understood. For this reason, our objective was to apply MMCS to examine the interview data to determine factors associated with CCM sustainability at each site.

Clinical and implementation contexts

CCM-based care aims to ensure that patients are treated in a coordinated, patient-centered, and anticipatory manner. This project’s nine outpatient general mental health clinics had participated in a hybrid CCM effectiveness-implementation trial 3 to 4 years prior, which had resulted in improved clinical outcomes that were not universally maintained post-implementation (i.e., after implementation funding and associated evaluation efforts concluded) [ 7 , 9 ]. This lack of aggregate sustainability across the nine clinics is what prompted the earlier interview-based investigation of CCM provider perspectives that identified key determinants of CCM sustainability at the trial sites [ 10 ].

These prior works were conducted in VA outpatient mental health teams, known as Behavioral Health Interdisciplinary Program (BHIP) teams. While there was variability in the exact composition of each BHIP team, all teams consisted of a multidisciplinary set of frontline clinicians (e.g., psychiatrists, psychologists, social workers, nurses) and support staff, serving a panel of about 1000 patients each.

This current project applied MMCS to examine the data from the earlier interviews [ 10 ] for the ways in which CCM sustainability differed at the sites and the factors related to sustainability. The project was determined to be non-research by the VA Boston Research and Development Service, and therefore did not require oversight by the Institutional Review Board (IRB). Details regarding the procedures undertaken for the completed hybrid CCM effectiveness-implementation trial, which serves as the context for this project, have been previously published [ 6 , 7 ]. Similarly, details regarding data collection for the follow-up provider interviews have also been previously published [ 10 ]. We provide a brief overview of the steps that we took for data collection and describe the steps that we took for applying MMCS to analyze the interview data. Additional file  1 outlines our use of the Consolidated Criteria for Reporting Qualitative Research (COREQ) Checklist [ 11 ].

Data collection

We recruited 30 outpatient mental health providers across the nine sites that had participated in the CCM implementation trial, including a multidisciplinary mix of mental health leaders and frontline staff. We recruited participants via email, and we obtained verbal informed consent from all participants. Each interview lasted between 30 and 60 min and focused on the degree to which the participant perceived care processes to have remained aligned to the CCM’s six core elements: work role redesign, patient self-management support, provider decision support, clinical information systems, linkages to community resources, and organizational/leadership support [ 12 , 13 , 14 ]. Interview questions also inquired about the participant’s perceived barriers and enablers influencing CCM sustainability, as well as about the latest status of CCM-based care practices. Interviews were digitally recorded and professionally transcribed. Additional details regarding data collection have been previously published [ 10 ].

Data analysis

We applied MMCS’ nine analytical steps [ 1 ] to the interview data. Each step described below was led by one designated member of the project team, with subsequent review by all project team members to reach a consensus on the examination conducted for each step.

We established the evaluation goal (step 1) to identify the ways in which sustainability differed across the sites and the factors related to sustainability, defining sustainability (step 2) as the continued existence of CCM-aligned care practices—namely, that care processes remained aligned with the six core CCM elements. Table  1 shows examples of care processes that align with each CCM element. As our prior works directly leading up to this project (i.e., design and evaluation of the CCM implementation trial that involved the very sites included in this project [ 6 , 15 , 16 ]) were guided by the Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework [ 17 ] and i-PARIHS positions facilitation (the implementation strategy that our trial was testing) as the core ingredient that drives implementation [ 17 ], we selected i-PARIHS’ four domains—innovation, recipients, context, and facilitation—as relevant domains under which to examine factors influencing sustainability (step 3). i-PARIHS posits that the successful implementation of an innovation and its sustained use by recipients in a context is enabled by facilitation (both the individuals doing the facilitation and the process used for facilitation). We examined the data on both sustainability and potentially relevant i-PARIHS domains (step 4) by conducting directed content analysis [ 18 ] of the recorded and professionally transcribed interview data. We used the six CCM elements and the four i-PARIHS domains as a priori codes.

Additional file  2 provides an overview of data input, tasks performed, and analysis output for MMCS steps 5 through 9 described below. We assessed sustainability per site (step 5) by generating CCM code summaries per site, and reached a consensus on whether each site exhibited high, medium, or low sustainability relative to other sites based on the summary data. We assigned a higher sustainability level for sites that exhibited more CCM-aligned care processes, had more participants consistently mention those processes, and considered those processes more as “just the way things are done” at the site. Namely, (i) high sustainability sites had concrete examples of CCM-aligned care processes (such as the ones shown in Table  1 ) for many of the six CCM elements, which multiple participants mentioned as central to how they deliver care, (ii) low sustainability sites had only a few concrete examples of CCM-aligned care processes, mentioned by only a small subset of participants and/or inconsistently practiced, and (iii) medium sustainability sites matched neither of the high nor low sustainability cases, having several concrete examples of CCM-aligned care process for some of the CCM elements, varying in whether they are mentioned by multiple participants or how consistently they are a part of delivering care. For the CCM code summaries per site, one project team member initially reviewed the coded data to draft the summaries including exemplar quotes. Each summary and relevant exemplar quotes were then reviewed by and refined with input from all six project team members during recurring team meetings to finalize the high, medium, or low sustainability designation to use in the subsequent MMCS steps. Reviewing and refining the summaries for the nine sites took approximately four 60-min meetings of the six project team members, with each site’s CCM code summary taking approximately 20–35 min to discuss and reach consensus on. We referred to lists of specific examples of how the six core CCM elements were operationalized in our CCM implementation trial [ 19 , 20 ]. Refinements occurred mostly around familiarizing the newer members of the project team (i.e., those who had not participated in our prior CCM-related work) with the examples and definitions. We aligned to established qualitative analysis methods for consensus-reaching discussions [ 18 , 21 ]. Recognizing the common challenge faced by such discussions in adequately accounting for everyone’s interpretations of the data [ 22 ], we drew on Bens’ meeting facilitation techniques [ 23 ] that include setting ground rules, ensuring balanced participation from all project team members, and accurately recording decisions and action items.

We then identified influencing factors per site (step 6), by generating i-PARIHS code summaries per site and identifying distinct factors under each domain of i-PARIHS (e.g., Collaborativeness and teamwork as a factor under the Recipients domain). For the i-PARIHS code summaries per site, one project team member initially reviewed the coded data to draft the summaries including exemplar quotes. They elaborated on each i-PARIHS domain-specific summary by noting distinct factors that they deemed relevant to the summary, proposing descriptive wording to refer to each factor (e.g., “team members share a commitment to their patients” under the Recipients domain). Each summary, associated factor descriptions, and relevant exemplar quotes were then reviewed and refined with input from all six project team members during recurring team meetings to finalize the relevant factors to use in the subsequent MMCS steps. Finalizing the factors included deciding which similar proposed factor descriptions from different sites to consolidate into one factor and which wording to use to refer to the consolidated factor (e.g., “team members share a commitment to their patients,” “team members collaborate well,” and “team members know each other’s styles and what to expect” were consolidated into the Collaborativeness and teamwork factor under the Recipients domain). It took approximately four 60-min meetings of the six project team members to review and refine the summaries and factors for the nine sites, with each site’s i-PARIHS code summary and factors taking approximately 20–35 min to discuss and reach consensus on. We referred to lists of explicit definitions of i-PARIHS constructs that our team members had previously developed and published [ 16 , 24 ]. We once again aligned to established qualitative analysis methods for consensus-reaching discussions [ 18 , 21 ], drawing on Bens’ meeting facilitation techniques [ 23 ] to adequately account for everyone’s interpretations of the data [ 22 ].

We organized the examined data (i.e., the assessed sustainability and identified factors per site) into a sortable matrix (step 7) using Microsoft Excel [ 25 ], laid out by influencing factor (row), sustainability (column), and site (sheet). We conducted within-site analysis of the matrixed data (step 8), examining the data on each influencing factor and designating whether the factor (i) was present, somewhat present, or minimally present [based on aggregate reports from the site’s participants; used “minimally present” when, considering all available data from a site regarding a factor, the factor was predominantly weak (e.g., predominantly weak Ability to continue patient care during COVID at a medium sustainability site); used “somewhat present” when, considering all available data from a site regarding a factor, the factor was neither predominantly strong nor predominantly weak (e.g., neither predominantly strong nor predominantly weak Collaborativeness and teamwork at a low sustainability site)], and (ii) had an enabling, hindering, or neutral/unclear influence on sustainability (designated as “neutral” when, considering all available data from a site regarding a factor, the factor had neither a predominantly enabling nor a predominantly hindering influence on sustainability). These designations of factors’ presence and influence are conceptually representative of what is commonly referred to as magnitude and valence, respectively, by other efforts that construct scoring for qualitative data (e.g., [ 26 , 27 ]). Like the team-based consensus approach of earlier MMCS steps, factors’ presence and type of influence per site were initially proposed by one project team member after reviewing the matrix’s site-specific data, then refined with input from all project team members during recurring team meetings that reviewed the matrix. Accordingly, similar to the earlier MMCS steps, we aligned to established qualitative methods [ 18 , 21 ] and meeting facilitation techniques [ 23 ] for these consensus-reaching discussions.

We then conducted a cross-site analysis of the matrixed data (step 9), assessing whether factors and their combinations were (i) present across multiple sites, (ii) consistently associated with higher or lower sustainability, and (iii) emphasized at some sites more than others. We noted that any factor may have not come up during interviews with a site because either it is not pertinent or it is pertinent but still did not come up, although we asked an open-ended question at the end of each interview about whether there was anything else that the participant wanted to share regarding sustainability. To adequately account for these possibilities, we decided as a team to regard a factor or a combination of factors as being associated with high/medium/low sustainability if it was identified at a majority (i.e., even if not all) of the sites designated as high/medium/low sustainability (e.g., if the Collaborativeness and teamwork factor is identified at a majority, even if not all, of the high sustainability sites, we would find it to be associated with high sustainability). Like the team-based consensus approach of earlier MMCS steps, cross-site patterns were initially proposed by one project team member after reviewing the matrix’s cross-site data, then refined with input from all project team members during recurring team meetings that reviewed the matrix. Accordingly, similar to the earlier MMCS steps, we aligned to established qualitative methods [ 18 , 21 ] and meeting facilitation techniques [ 23 ] for these consensus-reaching discussions. We acknowledged the potential existence of additional factors influencing sustainability that may not have emerged during our interviews and also may vary substantially between sites. For example, adaptation of the CCM, characteristics of the patient population, and availability of continued funding, which are factors that extant literature reports as being relevant to sustainability [ 28 , 29 ], were not seen in our interview data. To maintain our analytic focus on the factors seen in our data, we did not add these factors to our analysis.

For the nine sites included in this project, we found the degree of CCM sustainability to be split evenly across the sites—three high-, three medium-, and three low-sustainability. Twenty-five total influencing factors were identified under the i-PARIHS domains of Innovation (6), Recipients (6), Context (8), and Facilitation (5). Table  2 shows these identified influencing factors by domain. Figure  1 shows 11 influencing factors that were identified for at least two sites within a group of high/medium/low sustainability sites—e.g., the factor “consistent and strong internal facilitator” is shown as being present at high sustainability sites with an enabling influence on sustainability, because it was identified as such at two or more of the high sustainability sites. Of these 11 influencing factors, four were identified only for sites with high CCM sustainability and two were identified only for sites with medium or low CCM sustainability.

figure 1

Influencing factors that were identified for at least two sites within a group of high/medium/low sustainability sites

Key trends in influencing factors associated with high, medium, and/or low CCM sustainability

Three factors across two i-PARIHS domains exhibited strong trends by sustainability status. They were the Collaborativeness and teamwork and Turnover of clinic staff and leadership factors under the Recipients domain, and the Having a consistent and strong internal facilitator factor under the Facilitation domain.

Recipients-related factors

Collaborativeness and teamwork was present with an enabling influence on CCM sustainability at most high and medium sustainability sites, while it was only somewhat present with a neutral influence on CCM sustainability at most low sustainability sites. When asked what had made their BHIP team work well, a participant from a high sustainability site said,

“Just a collaborative spirit.” (Participant 604)

A participant from a medium sustainability site said,

“We joke that [the BHIP teams] are even family, that the teams really do function pretty tightly and they each have their own personality.” (Participant 201)

At the low sustainability sites, willingness to work as a team varied across team members; a participant from a low sustainability site said,

“… I think it has to be the commitment of the people who are on the team. So those that are regularly attending, we get a lot more out of it than those that probably don't ever come [to team meetings].” (Participant 904)

Collaborativeness and teamwork of BHIP team members were often perceived as the highlight of pursuing interdisciplinary care.

Turnover of clinic staff and leadership was present with a hindering influence on CCM sustainability at most high, medium, and low sustainability sites.

“We’ve lost a lot of really, really good providers here in the time I’ve been here …,” (Participant 102)

said a participant from a low-sustainability site that had to reconfigure its BHIP teams due to clinic staff shortages. Turnover of mental health clinic leadership made it difficult to maintain CCM practices, especially beyond the teams that participated in the original CCM implementation trial. A participant from a medium sustainability site said,

“Probably about 90 percent of the things that we came up with have fallen by the wayside. Within our team, many of those remain but again, that hand off towards the other teams that I think partly is due to the turnover rate with program managers, supervisors, didn’t get fully implemented.” (Participant 703)

Although turnover was an issue for high sustainability sites as well, there was also indication of the situation improving in recent years; a participant from a high sustainability site said,

“… our attrition rollover rate has dropped quite a bit and I would really attribute that to [the CCM being] more functional and more sustainable and tolerable for the providers.” (Participant 502)

As such, staff and leadership turnover was deemed a major challenge for CCM sustainability for all sites regardless of the overall level of sustainability.

Facilitation-related factor

Having a consistent and strong internal facilitator was present with an enabling influence on CCM sustainability at high sustainability sites, not identified as an influencing factor at most of the medium sustainability sites, and variably present with a hindering, neutral, or unclear influence on CCM sustainability at low sustainability sites. Participants from a high sustainability site perceived that it was important for the internal facilitator to understand different BHIP team members’ personalities and know the clinic’s history. A participant from another high sustainability site shared that, as an internal facilitator themselves, they focused on recognizing and reinforcing the progress of team members:

“… I'm often the person who kind of [starts] off with, ‘Hey, look at what we've done in this location,’ ‘Hey look at what the team's done this month.’” (Participant 402)

A participant from a low sustainability site had also served as an internal facilitator and recounted the difficulty and importance of readying the BHIP team to function in the long run without their assistance:

“I should have been able to get out sooner, I think, to get it to have them running this themselves. And that was just a really difficult process.” (Participant 301)

Participants, especially from the high and low sustainability sites, attributed their BHIP teams’ successes and challenges to the skills of the internal facilitator.

Influencing factors identified only for sites with high CCM sustainability

Four factors across four i-PARIHS domains were identified for high sustainability sites and not for medium or low sustainability sites. They were the factors Details about the CCM being well understood (Innovation domain), Interdisciplinary coordination (Recipients domain), Having adequate clinic space for CCM team members (Context domain), and Having a knowledgeable and helpful external facilitator (Facilitation domain).

Innovation-related factor

Details about the CCM being well understood was minimal to somewhat present with an unclear influence on CCM sustainability.

“We’ve … been trying to help our providers see the benefit of team-based care and the episodes-of-care idea, and I would say that is something our folks really have continued to struggle with as well,” (Participant 401)

said a participant from a high sustainability site. “What is considered CCM-based care?” continued to be a question on providers’ minds. A participant from a high sustainability site asked during the interview,

“Is there kind of a clearing house of some of the best practices for [CCM] that you guys have … or some other collection of resources that we could draw from?” (Participant 601)

Although such references are indeed accessible online organization-wide, participants were not always aware of those resources or what exactly CCM entails.

Recipients-related factor

Interdisciplinary coordination was somewhat present with a hindering, neutral, or unclear influence on CCM sustainability. Coordination between psychotherapy and psychiatry providers was deemed difficult by participants from high-sustainability sites. A participant said,

“We were initially kind of top heavy on the psychiatry so just making sure we have … therapy staff balancing that out [has been important].” (Participant 501)

Another participant perceived that BHIP teams were helpful in managing.

… ‘sibling rivalry’ between different disciplines … because [CCM] puts us all in one team and we communicate.” (Participant 505)

Interdisciplinary coordination was understood by the participants as being necessary for effective CCM-based care yet difficult to achieve.

Context-related factor

Having adequate clinic space for CCM team members was minimal to somewhat present with a hindering, neutral, or unclear influence on CCM sustainability. COVID-19 led to changes in how clinic space was used/assigned. A participant from a high sustainability site remarked,

“Pre-COVID everything was in a room instead of online. And now all our meetings are online and so it's actually really easy for the supervisors to be able to rotate through them and then, you know, they can answer programmatic questions ….” (Participant 402)

Participants from another high sustainability site found that issues regarding limited clinic space were both exacerbated and alleviated by COVID, with the mental health service losing space to vaccine clinics but more mental health clinicians teleworking and in less need of clinic space. Virtual connections were seen to alleviate some physical workspace-related concerns.

Having a knowledgeable and helpful external facilitator was variably present; when present, it had an enabling influence on CCM sustainability. Participants from a high sustainability site noted how many of the external facilitator’s efforts to change the BHIP team’s work processes very much remained over time. An example of a change was to have team meetings be structured to meet evolving patient needs. Team members came to meetings with the shared knowledge and expectation that,

“… we need to touch on folks who are coming out of the hospital, we need to touch on folks with higher acuity needs.” (Participant 402)

Implementation support that sites received from their external facilitator mostly occurred during the time period of the original CCM implementation trial; correspondence with the external facilitator after that trial time period was not common for sites. Participants still largely found the external facilitator to provide helpful guidance and advice on delivering CCM-based care.

Influencing factors identified only for sites with medium or low CCM sustainability

Two factors were identified for medium or low sustainability sites and not for high sustainability sites. They were the factors Ability to continue patient care during COVID and Adequate resources/capacity for care delivery . These factors were both under i-PARIHS’ Context domain, unlike the influencing factors above that were identified only for high sustainability sites, which spanned all four i-PARIHS domains.

Context-related factors

Ability to continue patient care during COVID had a hindering influence on CCM sustainability when minimally present. Participants felt that their CCM work was challenged when delivering care through telehealth was made difficult—e.g., at a medium sustainability site, site policies during the pandemic required a higher number of in-person services than the BHIP team providers expected or desired to deliver. On the other hand, this factor had an enabling influence on CCM sustainability when present. A participant at a low sustainability site mentioned the effect of telehealth on being able to follow up more easily with patients who did not show up for their appointments:

“… my no-show rate has dropped dramatically because if people don’t log on after a couple minutes, I call them. They're like ‘oh, I forgot, let me pop right on,’ whereas, you know, in the face-to-face space, you know, you wait 15 minutes, you call them, it’s too late for them to come in so then they're no shows.” (Participant 102)

The advantages of virtual care delivery, as well as the challenges of getting approvals to pursue it to varying extents, were well recognized by the participants.

Adequate resources/capacity for care delivery was minimally present at medium sustainability sites with a hindering influence on CCM sustainability. At a medium sustainability site, although leadership was supportive of CCM, resources were being used to keep clinics operational (especially during COVID) rather than investing in building new CCM-based care delivery processes.

“I think that if my boss came to me, [and asked] what could I do for [the clinics] … I would say even more staff,” (Participant 202)

said a participant from a medium sustainability site. At the same time, the participant, as many others we interviewed, understood and emphasized the need for BHIP teams to proceed with care delivery even when resources were limited:

“… when you’re already dealing with a very busy clinic, short staff and then you’re hit with a pandemic you handle it the best that you can.” (Participant 202)

Participants felt the need for basic resource requirements to be met in order for CCM-based care to be feasible.

In this project, we examined factors influencing the sustainability of CCM-aligned care practices at general mental health clinics within nine VA medical centers that previously participated in a CCM implementation trial. Guided by the core CCM elements and i-PARIHS domains, we conducted and analyzed CCM provider interviews. Using MMCS, we found CCM sustainability to be split evenly across the nine sites (three high, three medium, and three low), and that sustainability may be related most strongly to provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators.

In comparison to most high sustainability sites, participants from most medium or low sustainability sites did not mention a knowledgeable and helpful external facilitator who enabled sustainability. Participants at the high sustainability sites also emphasized the need for clarity about what CCM-based care comprises, interdisciplinary coordination in delivering CCM-aligned care, and adequate clinic space for BHIP team members to connect and collaborate. In contrast, in comparison to participants at most high sustainability sites, participants at most medium or low sustainability sites emphasized the need for better continuity of patient-facing activities during the COVID-19 pandemic and more resources/capacity for care delivery. A notable difference between these two groups of influencing factors is that the ones emphasized at most high sustainability sites are more CCM-specific (e.g., external facilitator with CCM expertise, knowledge, and structures to support delivery of CCM-aligned care), while the ones emphasized at most medium or low sustainability sites are factors that certainly relate to CCM sustainability but are focused on care delivery operations beyond CCM-aligned care (e.g., COVID’s widespread impacts, limited staff availability). In short, an emphasis on immediate, short-term clinical needs in the face of the COVID-19 pandemic and staffing challenges appeared to sap sites’ enthusiasm for sustaining more collaborative, CCM-consistent care processes.

Our previous qualitative analysis of these interview data suggested that in order to achieve sustainability, it is important to establish appropriate infrastructure, organizational readiness, and mental health service- or department-wide coordination for CCM implementation [ 10 ]. The findings from the current project augment these previous findings by highlighting the specific factors associated with higher and lower CCM sustainability across the project sites. This additional knowledge provides two important insights into what CCM implementation efforts should prioritize with regard to the previously recommended appropriate infrastructure, readiness, and coordination. First, for knowledge retention and coordination during personnel changes (including any changes in internal facilitators through and following implementation), care processes and their specific procedures should be established and documented in order to bring new personnel up to speed on those care processes. Management sciences, as applied to health care and other fields, suggest that such organizational knowledge retention can be maximized when there are (i) structures set up to formally recognize/praise staff when they share key knowledge, (ii) succession plans to be applied in the event of staff turnover, (iii) opportunities for mentoring and shadowing, and (iv) after action reviews of conducted care processes, which allow staff to learn about and shape the processes themselves [ 30 , 31 , 32 , 33 ]. Future CCM implementation efforts may thus benefit from enacting these suggestions alongside establishing and documenting CCM-based care processes and associated procedures.

Second, efforts to implement CCM-aligned practices into routine care should account for the extent to which sites’ more fundamental operational needs are met or being addressed. That information can be used to appropriately scope the plan, expectations, and timeline for implementation. For instance, ongoing critical staffing shortages or high turnover [ 34 ] at a site are unlikely to be resolved through a few months of CCM implementation. In fact, in that situation, it is possible that CCM implementation efforts could lead to reduced team effectiveness in the short term, given the effort required to establish more collaborative and coordinated care processes [ 35 ]. Should CCM implementation move forward at a given site, implementation goals ought to be set on making progress in realms that are within the implementation effort’s control (e.g., designing CCM-aligned practices that take staffing challenges into consideration) [ 36 , 37 ] rather than on factors outside of the effort’s control (e.g., staffing shortages). As healthcare systems determine how to deploy support (e.g., facilitators) to sites for CCM implementation, they would benefit from considering whether it is primarily CCM expertise that the site needs at the moment, or more foundational organizational resources (e.g., mental health staffing, clinical space, leadership enhancement) [ 38 ] to first reach an operational state that can most benefit from CCM implementation efforts at a later point in time. There is growing consensus across the field that the readiness of a healthcare organization to innovate is a prerequisite to successful innovation (e.g., CCM implementation) regardless of the specific innovation [ 39 , 40 ]. Several promising strategies specifically target these organizational considerations for implementing evidence-based practices (e.g., [ 41 , 42 ]). Further, recent works have begun to more clearly delineate leadership-related, climate-related, and other contextual factors that contribute to organizations’ innovation readiness [ 43 ], which can inform healthcare systems’ future decisions regarding preparatory work leading to, and timing of, CCM implementation at their sites.

These considerations informed by MMCS may have useful implications for implementation strategy selection and tailoring for future CCM implementation efforts, especially in delineating the target level (e.g., system, organizational, clinic, individual) and timeline of implementation strategies to be deployed. For instance, of the three factors found to most notably trend with CCM sustainability, Collaborativeness and teamwork may be strengthened through shorter-term team-building interventions at the organizational and/or clinic levels [ 38 ], Turnover of clinic staff and leadership may be mitigated by aiming for longer-term culture/climate change at the system and/or organizational levels [ 44 , 45 , 46 ], and Having a consistent and strong internal facilitator may be ensured more immediately by selecting an individual with fitting expertise/characteristics to serve in the role [ 15 ] and imparting innovation/facilitation knowledge to them [ 47 ]. Which of these factors to focus on, and through what specific strategies, can be decided in partnership with an implementation site—for instance, candidate strategies can be identified based on ones that literature points to for addressing these factors [ 48 ], systematic selection of the strategies to move forward can happen with close input from site personnel [ 49 ], and explicit further specification of those strategies [ 50 ] can also happen in collaboration with site personnel to amply account for site-specific contexts [ 51 ].

As is common for implementation projects, the findings of this project are highly context-dependent. It involves the implementation of a specific evidence-based practice (the CCM) using a specific implementation strategy (implementation facilitation) at specific sites (BHIP teams within general mental health clinics at nine VA medical centers). For such context-dependent findings to be transferable [ 52 , 53 ] to meaningfully inform future implementation efforts, sources of variation in the findings and how the findings were reached must be documented and traceable. This means being explicit about each step and decision that led up to cross-site analysis, as MMCS encourages, so that future implementation efforts can accurately view and consider why and how findings might be transferable to their own work. For instance, beyond the finding that Turnover of clinic staff and leadership was a factor present at most of the examined sites, MMCS’ traceable documentation of qualitative data associated with this factor at high sustainability sites also allowed highlighting the perception that CCM implementation is contributing to mitigating turnover of providers in the clinic over time, which may be a crucial piece of information that fuels future CCM implementation efforts.

Furthermore, to compare findings and interpretations across projects, consistent procedures for setting up and conducting these multi-site investigations are indispensable [ 54 , 55 , 56 ]. Although many projects involve multiple sites and assess variations across the sites, it is less common to have clearly delineated protocols for conducting such assessments. MMCS is meant to target this very gap, by offering a formalized sequence of steps that prompt specification of analytical procedures and decisions that are often interpretive and left less specified. MMCS uses a concrete data structure (the matrix) to traceably organize information and knowledge gained from a project, and the matrix can accommodate various data sources and conceptual groundings (e.g., guiding theories, models, and frameworks) that may differ from project to project – for instance, although our application of MMCS aligned to i-PARIHS, other projects applying MMCS [ 2 , 5 ] use different conceptual guides (e.g., Consolidated Framework for Implementation Research [ 57 ], Theoretical Domains Framework [ 58 ]). Therefore, as more projects align to the MMCS steps [ 1 ] to identify factors related to implementation and sustainability, better comparisons, consolidations, and transfers of knowledge between projects may become possible.

This project has several limitations. First, the high, medium, and low sustainability assigned to the sites were based on the sites’ CCM sustainability relative to one another, rather than based on an external metric of sustainability. As measures of sustainability such as the Program Sustainability Assessment Tool [ 59 , 60 ] and the Sustainment Measurement System Scale [ 61 ] become increasingly developed and tested, future projects may consider the feasibility of incorporating such measures to assess each site’s sustainability. In our case, we worked on addressing this limitation by using a consensus approach within our project team to assign sustainability levels to sites, as well as by confirming that the sites that we designated as high sustainability exhibited CCM elements that we had previously observed at the end of their participation in the original CCM implementation trial [ 19 ]. Second, we did not assign strict thresholds above/below which the counts or proportions of data regarding a factor would automatically indicate whether the factor (i) was present, somewhat present, or minimally present and (ii) had an enabling, hindering, or neutral/unclear influence on sustainability. This follows widely accepted qualitative analytical guidance that discourages characterizing findings solely based on the frequency with which a notion is mentioned by participants [ 62 , 63 , 64 ], in order to prevent unsubstantiated inferences or conclusions. We sought to address this limitation in two ways: We carefully documented the project team’s rationale for each consensus reached, and we reviewed all consensuses reached in their entirety to ensure that any two factors with the same designation (e.g., “minimally present”) do not have associated rationale that conflict across those factors. These endeavors we undertook closely adhere to established case study research methods [ 65 ], which MMCS builds on, that emphasize strengthening the validity and reliability of findings through documenting a detailed analytic protocol, as well as reviewing data to ensure that patterns match across analytic units (e.g., factors, interviewees, sites). Third, our findings are based on three sites each for high/medium/low sustainability, and although we identified single factors associated with sustainability, we found no specific combinations of factors’ presence and influence that were repeatedly existent at a majority of the sites designated as high/medium/low sustainability. Examining additional sites on the factors identified through this work (as we will for our subsequent CCM implementation trial described below) will allow more opportunities for repeated combinations and other factors to emerge, making possible firmer conclusions regarding the extent to which the currently identified factors and absence of identified combinations are applicable beyond the sites included in this study. Fourth, the identified influencing factor “leadership support for CCM” (under the Context domain of the i-PARIHS framework) substantially overlaps in concept with the core “organizational/leadership support” element of the CCM. To avoid circular reasoning, we used leadership support-related data to inform our assignment of sites’ high, medium, or low CCM sustainability, rather than as a reason for the sites’ CCM sustainability. In reality, strong leadership support may both result from and contribute to implementation and sustainability [ 16 , 66 ], and thus causal relationships between the i-PARIHS-aligned influencing factors and the CCM elements (possibly with feedback loops) warrant further examination to most appropriately use leadership support-related data in future analyses of CCM sustainability. Fifth, findings may be subject to both social desirability bias in participants providing more positive than negative evidence of sustainability (especially participants who are responsible for implementing and sustaining CCM-aligned care at their site) and the project team members’ bias in interpreting the findings to align to their expectations of further effort being necessary to sustainably implement the CCM. To help mitigate this challenge, the project interviewers strove to elicit from participants both positive and negative perceptions and experiences related to CCM-based care delivery, both of which were present in the examined interview data.

Future work stemming from this project is twofold. Regarding CCM implementation, we will conduct a subsequent CCM implementation trial involving eight new sites to prospectively examine how implementation facilitation with an enhanced focus on these findings affects CCM sustainability. We started planning for sustainability prior to implementation, looking to this work for indicators of specific modifications needed to the previous way in which we used implementation facilitation to promote the uptake of CCM-based care [ 67 ]. Findings from this work suggest that sustainability may be related most strongly to (i) provider collaboration, (ii) knowledge retention during staff/leadership transitions, and (iii) availability of skilled internal facilitators. Hence, we will accordingly prioritize developing procedures for (i) regular CCM-related information exchange amongst BHIP team members, as well as between the BHIP team and clinic leadership, (ii) both translating knowledge to and keeping knowledge documented at the site, and (iii) supporting the sites’ own personnel to take the lead in driving CCM implementation.

Regarding MMCS, we will continuously refine and improve the method by learning from other projects applying, testing, and critiquing MMCS. Outside of our CCM-related projects, examinations of implementation data using MMCS are actively underway for various implementation efforts including that of a data dashboard for decision support on transitioning psychiatrically stable patients from specialty mental health to primary care [ 2 ], a peer-led healthy lifestyle intervention for individuals with serious mental illness [ 3 ], screening programs for intimate partner violence [ 4 ], and a policy- and organization-based health system strengthening intervention to improve health systems in sub-Saharan Africa [ 5 ]. As MMCS is used by more projects that differ from one another in their specific outcome of interest, and especially in light of our MMCS application that examines factors related to sustainability, we are curious whether certain proximal to distal outcomes are more subject to heterogeneity in influencing factors than other outcomes. For instance, sustainability outcomes, which are tracked following a longer passage of time than some other outcomes, may be subject to more contextual variations that occur over time and thus could particularly benefit from being examined using MMCS. We will also explore MMCS’ complementarity with coincidence analysis and other configurational analytical approaches [ 68 ] for examining implementation phenomena. We are excited about both the step-by-step traceability that MMCS can bring to such methods and those methods’ computational algorithms that can be beneficial to incorporate into MMCS for projects with larger numbers of sites. For example, Salvati and colleagues [ 69 ] described both the inspiration that MMCS provided in structuring their data as well as how they addressed MMCS’ visualization shortcomings through their innovative data matrix heat mapping, which led to their selection of specific factors to include in their subsequent coincidence analysis. Coincidence analysis is an enhancement to qualitative comparative analysis and other configurational analytical methods, in that it is formulated specifically for causal inference [ 70 ]. Thus, in considering improved reformulations of MMCS’ steps to better characterize examined factors as explicit causes to the outcomes of interest, we are inspired by and can draw on coincidence analysis’ approach to building and evaluating causal chains that link factors to outcomes. Relatedly, we have begun to actively consider the potential contribution that MMCS can make to hypothesis generation and theory development for implementation science. As efforts to understand the mechanisms through which implementation strategies work are gaining momentum [ 71 , 72 , 73 ], there is an increased need for methods that help decompose our understanding of factors that influence the mechanistic pathways from strategies to outcomes [ 74 ]. Implementation science is facing the need to develop theories, beyond frameworks, which delineate hypotheses for observed implementation phenomena that can be subsequently tested [ 75 ]. The methodical approach that MMCS offers can aid this important endeavor, by enabling data curation and examination of pertinent factors in a consistent way that allows meaningful synthesis of findings across sites and studies. We see these future directions as concrete steps toward elucidating the factors related to sustainable implementation of EBPs, especially leveraging data from projects where the number of sites is much smaller than the number of factors that may matter—which is indeed the case for most implementation projects.

Using MMCS, we found that provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators may be most strongly related to CCM sustainability in VA outpatient mental health clinics. Informed by these findings, we have a subsequent CCM implementation trial underway to prospectively test whether increasing the aforementioned factors within implementation facilitation enhances sustainability. The MMCS steps used here for systematic multi-site examination can also be applied to determining sustainability-related factors relevant to various other EBPs and implementation contexts.

Availability of data and materials

The data analyzed during the current project are not publicly available because participant privacy could be compromised.

Abbreviations

Behavioral Health Interdisciplinary Program

Collaborative Chronic Care Model

Consolidated Criteria for Reporting Qualitative Research

coronavirus disease

evidence-based practice

Institutional Review Board

Integrated Promoting Action on Research Implementation in Health Services

Matrixed Multiple Case Study

United States Department of Veterans Affairs

Kim B, Sullivan JL, Ritchie MJ, Connolly SL, Drummond KL, Miller CJ, et al. Comparing variations in implementation processes and influences across multiple sites: What works, for whom, and how? Psychiatry Res. 2020;283:112520.

Article   PubMed   Google Scholar  

Hundt NE, Yusuf ZI, Amspoker AB, Nagamoto HT, Kim B, Boykin DM, et al. Improving the transition of patients with mental health disorders back to primary care: A protocol for a partnered, mixed-methods, stepped-wedge implementation trial. Contemp Clin Trials. 2021;105:106398.

Tuda D, Bochicchio L, Stefancic A, Hawes M, Chen J-H, Powell BJ, et al. Using the matrixed multiple case study methodology to understand site differences in the outcomes of a Hybrid Type 1 trial of a peer-led healthy lifestyle intervention for people with serious mental illness. Transl Behav Med. 2023;13(12):919–27.

Adjognon OL, Brady JE, Iverson KM, Stolzmann K, Dichter ME, Lew RA, et al. Using the Matrixed Multiple Case Study approach to identify factors affecting the uptake of IPV screening programs following the use of implementation facilitation. Implement Sci Commun. 2023;4(1):145.

Article   PubMed   PubMed Central   Google Scholar  

Seward N, Murdoch J, Hanlon C, Araya R, Gao W, Harding R, et al. Implementation science protocol for a participatory, theory-informed implementation research programme in the context of health system strengthening in sub-Saharan Africa (ASSET-ImplementER). BMJ Open. 2021;11(7):e048742.

Bauer MS, Miller C, Kim B, Lew R, Weaver K, Coldwell C, et al. Partnering with health system operations leadership to develop a controlled implementation trial. Implement Sci. 2016;11:22.

Bauer MS, Miller CJ, Kim B, Lew R, Stolzmann K, Sullivan J, et al. Effectiveness of implementing a Collaborative Chronic Care Model for clinician teams on patient outcomes and health status in mental health: a randomized clinical trial. JAMA Netw Open. 2019;2(3):e190230.

Ritchie MJ, Dollar KM, Miller CJ, Smith JL, Oliver KA, Kim B, et al. Using Implementation Facilitation to Improve Healthcare (Version 3): Veterans Health Administration, Behavioral Health Quality Enhancement Research Initiative (QUERI). 2020.

Google Scholar  

Bauer MS, Stolzmann K, Miller CJ, Kim B, Connolly SL, Lew R. Implementing the Collaborative Chronic Care Model in mental health clinics: achieving and sustaining clinical effects. Psychiatr Serv. 2021;72(5):586–9.

Miller CJ, Kim B, Connolly SL, Spitzer EG, Brown M, Bailey HM, et al. Sustainability of the Collaborative Chronic Care Model in outpatient mental health teams three years post-implementation: a qualitative analysis. Adm Policy Ment Health. 2023;50(1):151–9.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Von Korff M, Gruman J, Schaefer J, Curry SJ, Wagner EH. Collaborative management of chronic illness. Ann Intern Med. 1997;127(12):1097–102.

Article   Google Scholar  

Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q. 1996;74(4):511–44.

Article   CAS   PubMed   Google Scholar  

Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the chronic care model in the new millennium. Health Aff (Millwood). 2009;28(1):75–85.

Connolly SL, Sullivan JL, Ritchie MJ, Kim B, Miller CJ, Bauer MS. External facilitators’ perceptions of internal facilitation skills during implementation of collaborative care for mental health teams: a qualitative analysis informed by the i-PARIHS framework. BMC Health Serv Res. 2020;20(1):165.

Kim B, Sullivan JL, Drummond KL, Connolly SL, Miller CJ, Weaver K, et al. Interdisciplinary behavioral health provider perceptions of implementing the Collaborative Chronic Care Model: an i-PARIHS-guided qualitative study. Implement Sci Commun. 2023;4(1):35.

Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. 2016;11:33.

Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

Sullivan JL, Kim B, Miller CJ, Elwy AR, Drummond KL, Connolly SL, et al. Collaborative Chronic Care Model implementation within outpatient behavioral health care teams: qualitative results from a multisite trial using implementation facilitation. Implement Sci Commun. 2021;2(1):33.

Miller CJ, Sullivan JL, Kim B, Elwy AR, Drummond KL, Connolly S, et al. Assessing collaborative care in mental health teams: qualitative analysis to guide future implementation. Adm Policy Ment Health. 2019;46(2):154–66.

Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook: sage. 1994.

Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Bens I. Facilitating with Ease!: core skills for facilitators, team leaders and members, managers, consultants, and trainers. Hoboken: John Wiley & Sons; 2017.

Ritchie MJ, Drummond KL, Smith BN, Sullivan JL, Landes SJ. Development of a qualitative data analysis codebook informed by the i-PARIHS framework. Implement Sci Commun. 2022;3(1):98.

Excel: Microsoft. Available from: https://www.microsoft.com/en-us/microsoft-365/excel . Accessed 15 Feb 2024.

Madrigal L, Manders OC, Kegler M, Haardörfer R, Piper S, Blais LM, et al. Inner and outer setting factors that influence the implementation of the National Diabetes Prevention Program (National DPP) using the Consolidated Framework for Implementation Research (CFIR): a qualitative study. Implement Sci Commun. 2022;3(1):104.

Wilson HK, Wieler C, Bell DL, Bhattarai AP, Castillo-Hernandez IM, Williams ER, et al. Implementation of the Diabetes Prevention Program in Georgia Cooperative Extension According to RE-AIM and the Consolidated Framework for Implementation Research. Prev Sci. 2023;Epub ahead of print.

Proctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10:88.

Fathi LI, Walker J, Dix CF, Cartwright JR, Joubert S, Carmichael KA, et al. Applying the Integrated Sustainability Framework to explore the long-term sustainability of nutrition education programmes in schools: a systematic review. Public Health Nutr. 2023;26(10):2165–79.

Guptill J. Knowledge management in health care. J Health Care Finance. 2005;31(3):10–4.

PubMed   Google Scholar  

Gammelgaard J. Why not use incentives to encourage knowledge sharing. J Knowledge Manage Pract. 2007;8(1):115–23.

Liebowitz J. Knowledge retention: strategies and solutions. Boca Raton: CRC Press; 2008.

Ensslin L, CarneiroMussi C, RolimEnsslin S, Dutra A, Pereira Bez Fontana L. Organizational knowledge retention management using a constructivist multi-criteria model. J Knowledge Manage. 2020;24(5):985–1004.

Peterson AE, Bond GR, Drake RE, McHugo GJ, Jones AM, Williams JR. Predicting the long-term sustainability of evidence-based practices in mental health care: an 8-year longitudinal analysis. J Behav Health Serv Res. 2014;41(3):337–46.

Miller CJ, Griffith KN, Stolzmann K, Kim B, Connolly SL, Bauer MS. An economic analysis of the implementation of team-based collaborative care in outpatient general mental health clinics. Med Care. 2020;58(10):874–80.

Silver SA, Harel Z, McQuillan R, Weizman AV, Thomas A, Chertow GM, et al. How to begin a quality improvement project. Clin J Am Soc Nephrol. 2016;11(5):893–900.

Dixon-Woods M. How to improve healthcare improvement-an essay by Mary Dixon-Woods. BMJ. 2019;367:l5514.

Miller CJ, Kim B, Silverman A, Bauer MS. A systematic review of team-building interventions in non-acute healthcare settings. BMC Health Serv Res. 2018;18(1):146.

Robert G, Greenhalgh T, MacFarlane F, Peacock R. Organisational factors influencing technology adoption and assimilation in the NHS: a systematic literature review. Report for the National Institute for Health Research Service Delivery and Organisation programme. London; 2009.

Kelly CJ, Young AJ. Promoting innovation in healthcare. Future Healthc J. 2017;4(2):121–5.

PubMed   PubMed Central   Google Scholar  

Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11.

Ritchie MJ, Parker LE, Kirchner JE. Facilitating implementation of primary care mental health over time and across organizational contexts: a qualitative study of role and process. BMC Health Serv Res. 2023;23(1):565.

van den Hoed MW, Backhaus R, de Vries E, Hamers JPH, Daniëls R. Factors contributing to innovation readiness in health care organizations: a scoping review. BMC Health Serv Res. 2022;22(1):997.

Melnyk BM, Hsieh AP, Messinger J, Thomas B, Connor L, Gallagher-Ford L. Budgetary investment in evidence-based practice by chief nurses and stronger EBP cultures are associated with less turnover and better patient outcomes. Worldviews Evid Based Nurs. 2023;20(2):162–71.

Jacob RR, Parks RG, Allen P, Mazzucca S, Yan Y, Kang S, et al. How to “start small and just keep moving forward”: mixed methods results from a stepped-wedge trial to support evidence-based processes in local health departments. Front Public Health. 2022;10:853791.

Aarons GA, Conover KL, Ehrhart MG, Torres EM, Reeder K. Leader-member exchange and organizational climate effects on clinician turnover intentions. J Health Organ Manag. 2020;35(1):68–87.

Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC. Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med. 2014;29 Suppl 4(Suppl 4):904–12.

Strategy Design: CFIR research team-center for clinical management research. Available from: https://cfirguide.org/choosing-strategies/ . Accessed 15 Feb 2024.

Kim B, Wilson SM, Mosher TM, Breland JY. Systematic decision-making for using technological strategies to implement evidence-based interventions: an illustrated case study. Front Psychiatry. 2021;12:640240.

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

Lewis CC, Scott K, Marriott BR. A methodology for generating a tailored implementation blueprint: an exemplar from a youth residential setting. Implement Sci. 2018;13(1):68.

Maher C, Hadfield M, Hutchings M, de Eyto A. Ensuring rigor in qualitative data analysis: a design research approach to coding combining NVivo with traditional material methods. Int J Qual Methods. 2018;17(1):1609406918786362.

Holloway I. A-Z of qualitative research in healthcare. 2nd ed. Oxford: Wiley-Blackwell; 2008.

Reproducibility and Replicability in Research: National Academies. Available from: https://www.nationalacademies.org/news/2019/09/reproducibility-and-replicability-in-research . Accessed 15 Feb 2024.

Chinman M, Acosta J, Ebener P, Shearer A. “What we have here, is a failure to [Replicate]”: ways to solve a replication crisis in implementation science. Prev Sci. 2022;23(5):739–50.

Vicente-Saez R, Martinez-Fuentes C. Open Science now: a systematic literature review for an integrated definition. J Bus Res. 2018;88:428–36.

Consolidated Framework for Implementation Research: CFIR Research Team-Center for Clinical Management Research. Available from: https://cfirguide.org/ . Accessed 15 Feb 2024.

Atkins L, Francis J, Islam R, O’Connor D, Patey A, Ivers N, et al. A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implement Sci. 2017;12(1):77.

Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability Assessment Tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11:130184.

Calhoun A, Mainor A, Moreland-Russell S, Maier RC, Brossart L, Luke DA. Using the Program Sustainability Assessment Tool to assess and plan for sustainability. Prev Chronic Dis. 2014;11:130185.

Palinkas LA, Chou CP, Spear SE, Mendon SJ, Villamar J, Brown CH. Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale. Implement Sci. 2020;15(1):71.

Sandelowski M. Real qualitative researchers do not count: the use of numbers in qualitative research. Res Nurs Health. 2001;24(3):230–40.

Wood M, Christy R. Sampling for Possibilities. Qual Quant. 1999;33(2):185–202.

Chang Y, Voils CI, Sandelowski M, Hasselblad V, Crandell JL. Transforming verbal counts in reports of qualitative descriptive studies into numbers. West J Nurs Res. 2009;31(7):837–52.

Yin RK. Case study research and applications. Los Angeles: Sage; 2018.

Bauer MS, Weaver K, Kim B, Miller C, Lew R, Stolzmann K, et al. The Collaborative Chronic Care Model for mental health conditions: from evidence synthesis to policy impact to scale-up and spread. Med Care. 2019;57 Suppl 10 Suppl 3(10 Suppl 3):S221-s7.

Miller CJ, Sullivan JL, Connolly SL, Richardson EJ, Stolzmann K, Brown ME, et al. Adaptation for sustainability in an implementation trial of team-based collaborative care. Implement Res Pract. 2024;5:26334895231226197.

Curran GM, Smith JD, Landsverk J, Vermeer W, Miech EJ, Kim B, et al. Design and analysis in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. 3 ed. New York: Oxford University Press; In press.

Salvati ZM, Rahm AK, Williams MS, Ladd I, Schlieder V, Atondo J, et al. A picture is worth a thousand words: advancing the use of visualization tools in implementation science through process mapping and matrix heat mapping. Implement Sci Commun. 2023;4(1):43.

Whitaker RG, Sperber N, Baumgartner M, Thiem A, Cragun D, Damschroder L, et al. Coincidence analysis: a new method for causal inference in implementation science. Implement Sci. 2020;15(1):108.

Lewis CC, Powell BJ, Brewer SK, Nguyen AM, Schriger SH, Vejnoska SF, et al. Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda. BMJ Open. 2021;11(10):e053474.

Kilbourne AM, Geng E, Eshun-Wilson I, Sweeney S, Shelley D, Cohen DJ, et al. How does facilitation in healthcare work? Using mechanism mapping to illuminate the black box of a meta-implementation strategy. Implement Sci Commun. 2023;4(1):53.

Kim B, Cruden G, Crable EL, Quanbeck A, Mittman BS, Wagner AD. A structured approach to applying systems analysis methods for examining implementation mechanisms. Implementation Sci Commun. 2023;4(1):127.

Geng EH, Baumann AA, Powell BJ. Mechanism mapping to advance research on implementation strategies. PLoS Med. 2022;19(2):e1003918.

Luke DA, Powell BJ, Paniagua-Avila A. Bridges and mechanisms: integrating systems science thinking into implementation research. Annu Rev Public Health. In press.

Download references

Acknowledgements

The authors sincerely thank the project participants for their time, as well as the project team members for their guidance and support. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

This project was funded by VA grant QUE 20–026 and was designed and conducted in partnership with the VA Office of Mental Health and Suicide Prevention.

Author information

Authors and affiliations.

Center for Healthcare Organization and Implementation Research (CHOIR), VA Boston Healthcare System, 150 South Huntington Avenue, Boston, MA, 02130, USA

Bo Kim, Madisen E. Brown, Samantha L. Connolly, Elizabeth G. Spitzer, Hannah M. Bailey & Christopher J. Miller

Harvard Medical School, 25 Shattuck Street, Boston, MA, 02115, USA

Bo Kim, Samantha L. Connolly & Christopher J. Miller

Center of Innovation in Long Term Services and Supports (LTSS COIN), VA Providence Healthcare System, 385 Niagara Street, Providence, RI, 02907, USA

Jennifer L. Sullivan

Brown University School of Public Health, 121 South Main Street, Providence, RI, 02903, USA

VA Rocky Mountain Mental Illness Research, Education and Clinical Center (MIRECC), 1700 N Wheeling Street, Aurora, CO, 80045, USA

Elizabeth G. Spitzer

VA Northeast Program Evaluation Center, 950 Campbell Avenue, West Haven, CT, 06516, USA

Lauren M. Sippel

Geisel School of Medicine at Dartmouth, 1 Rope Ferry Road, Hanover, NH, 03755, USA

VA Office of Mental Health and Suicide Prevention, 810 Vermont Avenue NW, Washington, DC, 20420, USA

Kendra Weaver

You can also search for this author in PubMed   Google Scholar

Contributions

Concept and design: BK, JS, and CM. Acquisition, analysis, and/or interpretation of data: BK, JS, MB, SC, ES, and CM. Initial drafting of the manuscript: BK. Critical revisions of the manuscript for important intellectual content: JS, MB, SC, ES, HB, LS, KW, and CM. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Bo Kim .

Ethics declarations

Ethics approval and consent to participate.

This project was determined to be non-research by the VA Boston Research and Development Service, and therefore did not require oversight by the Institutional Review Board (IRB).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

COREQ (COnsolidated criteria for REporting Qualitative research) Checklist.

Additional file 2.

Data input, tasks performed, and analysis output for MMCS Steps 5 through 9.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kim, B., Sullivan, J.L., Brown, M.E. et al. Sustaining the collaborative chronic care model in outpatient mental health: a matrixed multiple case study. Implementation Sci 19 , 16 (2024). https://doi.org/10.1186/s13012-024-01342-2

Download citation

Received : 14 June 2023

Accepted : 21 January 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s13012-024-01342-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Collaborative care
  • Implementation
  • Interdisciplinary care
  • Mental health
  • Sustainability

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

case control study multiple outcomes

  • Study Protocol
  • Open access
  • Published: 20 February 2024

The PEP++ study protocol: a cluster-randomised controlled trial on the effectiveness of an enhanced regimen of post-exposure prophylaxis for close contacts of persons affected by leprosy to prevent disease transmission

  • Duane C. Hinders 1 ,
  • Anneke T. Taal 1 ,
  • Suchitra Lisam 2 ,
  • Aymée M. da Rocha 3 ,
  • Nand Lal Banstola 4 ,
  • Prativa Bhandari 4 ,
  • Abhijit Saha 5 ,
  • Jugal Kishore 6 ,
  • Virginia O. Fernandes 7 ,
  • Abu Sufian Chowdhury 5 ,
  • Anna T. van ‘t Noordende 1 ,
  • Liesbeth Mieras 1 ,
  • Jan Hendrik Richardus 8 &
  • Wim H. van Brakel 1  

BMC Infectious Diseases volume  24 , Article number:  226 ( 2024 ) Cite this article

Metrics details

Leprosy is an infectious disease with a slow decline in global annual caseload in the past two decades. Active case finding and post-exposure prophylaxis (PEP) with a single dose of rifampicin (SDR) are recommended by the World Health Organization as measures for leprosy elimination. However, more potent PEP regimens are needed to increase the effect in groups highest at risk (i.e., household members and blood relatives, especially of multibacillary patients). The PEP++ trial will assess the effectiveness of an enhanced preventive regimen against leprosy in high-endemic districts in India, Brazil, Bangladesh, and Nepal compared with SDR-PEP.

The PEP++ study is a cluster-randomised controlled trial in selected districts of India, Brazil, Bangladesh, and Nepal. Sub-districts will be allocated randomly to the intervention and control arms. Leprosy patients detected from 2015 − 22 living in the districts will be approached to list their close contacts for enrolment in the study. All consenting participants will be screened for signs and symptoms of leprosy and tuberculosis (TB). In the intervention arm, eligible contacts receive the enhanced PEP++ regimen with three doses of rifampicin (150 − 600 mg) and clarithromycin (150 − 500 mg) administered at four-weekly intervals, whereas those in the control arm receive SDR-PEP. Follow-up screening for leprosy will be done for each individual two years after the final dose is administered. Cox’ proportion hazards analysis and Poisson regression will be used to compare the incidence rate ratios between the intervention and control areas as the primary study outcome.

Past studies have shown that the level of SDR-PEP effectiveness is not uniform across contexts or in relation to leprosy patients. To address this, a number of recent trials are seeking to strengthen PEP regimens either through the use of new medications or by increasing the dosage of the existing ones. However, few studies focus on the impact of multiple doses of chemoprophylaxis using a combination of antibiotics. The PEP++ trial will investigate effectiveness of both an enhanced regimen and use geospatial analysis for PEP administration in the study communities.

Trial registration

NL7022 on the Dutch Trial Register on April 12, 2018. Protocol version 9.0 updated on 18 August 2022 https://www.onderzoekmetmensen.nl/en/trial/23060

Peer Review reports

Leprosy, or Hansen’s disease, is an infectious disease caused by Mycobacterium leprae [ 1 ]. Although the transmission is not fully understood, the main mode is human-to-human via respiratory aerosols through coughing and sneezing. Prolonged and frequent close contact with an infectious person who has not started treatment is necessary for transmission [ 1 , 2 ]. Therefore, individuals living in the same household as a person affected by the disease, neighbours and other family members are at a higher risk of being infected [ 2 ].

In 2022, the number of new leprosy patients reported globally was 174,087. The majority of the world’s leprosy patients (78%) live in three countries: India (60%), Brazil (11%) and Indonesia (7%). An additional 17% come from the next 20 global priority countries, including Bangladesh and Nepal [ 3 ]. Between 2010 and 2019, the annual decrease in new leprosy cases globally was a modest 2% per year [ 4 ]. Therefore, new interventions, such as more precise active case finding and chemoprophylaxis, are essential to reduce the global caseload.

Trials to study the effectiveness of chemoprophylaxis to prevent leprosy have been conducted for decades. The first successful leprosy chemoprophylaxis trials were conducted with dapsone administered weekly or biweekly for months or years with limited effectiveness [ 5 ]. More research was required for more powerful single dose regimens. Rifampicin (RMP), recognised as the most effective bactericidal agent against M . leprae , was subsequently researched [ 6 ]. A single dose of rifampicin as leprosy post-exposure prophylaxis (SDR-PEP) was administered to close contacts of leprosy patients to prevent the disease in several studies [ 7 , 8 , 9 ].

The effectiveness of SDR-PEP as chemoprophylaxis was most firmly established in COLEP, a randomised, placebo-controlled, double-blind trial in Bangladesh [ 7 ]. The COLEP study found an overall reduction in leprosy incidence of 57% among contacts in the intervention group in the first two years [ 10 ]. Feasibility and acceptability of implementing SDR-PEP in routine leprosy control programmes was investigated thereafter in eight countries in the leprosy post-exposure prophylaxis (LPEP) programme [ 8 , 11 ]. As a result of these successful studies, the screening of close contacts of new leprosy patients combined with administration of SDR-PEP has been included in the World Health Organization (WHO) Guidelines for the Diagnosis, Treatment and Prevention of Leprosy [ 12 ]. Despite the overall protective effect of 57%, it was much lower among blood-related contacts and household members of multibacillary (MB) leprosy cases [ 10 ]. Therefore, a more potent post-exposure prophylaxis (PEP) regimen is needed to prevent disease especially among those at greater risk of developing the disease.

In 2016, NLR developed the idea for a large multi-country trial testing the effectiveness of an enhanced PEP regimen to significantly reduce the new case detection and stop the transmission of leprosy. An international expert meeting recommended a regimen consisting of three doses of a combination of two highly bactericidal and accessible antibiotics: RMP and moxifloxacin (MXF) for adults and RMP and clarithromycin (CLR) for children [ 13 ]. In 2018, however, the European Medicines Agency (EMA) restricted the use of MXF as preventive treatment because of potential long-lasting and disabling side effects. In response to these restrictions, leprosy experts recommended to use the combination of RMP and CLR for both adults and children. This combination therapy using repeated doses has been tested in a nude mouse model [ 14 ]. Results showed that the PEP++ combination (RMP/CLR) has a greater effect compared to any single antibiotic. This increased effectiveness has not, however, been tested in human populations in endemic countries to date. In this trial, we hypothesise that the leprosy incidence will be reduced more substantially in areas where the enhanced regimen is administered, thus demonstrating the effectiveness of the enhanced regimen.

Methods and design

The primary objective of the PEP++ randomised controlled trial (RCT) is to provide evidence of the effectiveness of an enhanced post-exposure prophylaxis regimen (PEP++) compared to the currently recommended regimen of SDR-PEP. Each study district will have an adverse events committee to monitor the frequency and severity of such events in each study arm and ensure participant safety.

Secondary study objectives seek to provide evidence of:

The acceptability and cost-effectiveness of PEP++ as a preventive intervention

The accuracy of geospatial methods to identify target areas for blanket campaigns, and

The effectiveness of community education and behaviour change (CEBC) interventions to change the perception of leprosy and reduce stigma

Study design

The study will use a cluster-randomised non-blinded controlled trial design with two arms to compare the effectiveness of three doses of RMP/CLR (the PEP++ regimen) with SDR-PEP in the prevention of leprosy disease among contacts of newly diagnosed leprosy patients. The randomisation units will sub-district divisions in each country context that will be randomly allocated to the two study arms.

All eligible leprosy patients in the districts will be asked to enumerate their close contacts. These individuals will subsequently be approached and enrolled in the study. Those contacts who provide informed consent are screened for signs of leprosy and tuberculosis and for inclusion eligibility. Contacts of leprosy patients living in the intervention arm receive the PEP++ regimen while those in the control arm receive SDR-PEP. All participants included in the study will be followed up two years after receiving the final dose of PEP++ or SDR-PEP.

The residences of the leprosy cases approached for the study will be geocoded in order to develop epidemiological maps that identify clusters of leprosy cases. After the regimen trial intake is concluded, blanket campaigns with SDR-PEP will be implemented in these clusters (high-risk areas) to reduce leprosy incidence at the population level. The study schedule is presented in Fig.  1 . A checklist of Recommendations for Clinical Intervention Trials (SPIRIT) is also available (Supplementary file 1 ).

figure 1

Study schedule of enrolment, interventions, and assessments

Study setting

This study will be conducted in two districts in India, Brazil, and Bangladesh and three districts in Nepal. Together these four countries accounted for 128,727 new leprosy cases in 2022, or 73.9% of the global caseload of 174,087 [ 3 ]. The districts/municipalities were selected per country based on the number of new leprosy cases detected in recent years, availability of contact screening and diagnostic services for leprosy, and logistical feasibility. For ease of study implementation and operational considerations, the districts are located in a single state or province in each country. A mix of urban, rural, and peri-urban settings was sought in order to show the replicability of the intervention across contexts in the future.

Participants

In this trial, we approach recently detected leprosy patients (index cases) as derived from leprosy programme registries in the four countries. The total number of new leprosy cases per study site is presented in Table  1 . These patients must have been detected in 2015 or later, give informed consent to participate in the study, and be willing to list their close contacts. The target population for preventive treatment is comprised of household contacts, family members, neighbours, and other social contacts (jointly denominated as ‘close contacts’) as listed by the index cases. These contacts are individuals who have had intensive contact with a leprosy patient for approximately 20 h per week during at least three months in the year before the index case was diagnosed. From previous studies, it is expected that approximately 20 close contacts per index case will be listed, depending on the study setting. All close contacts listed by an index case and located by the research staff are enrolled in the study with a unique identification code (UIC) and receive information on the study.

The contacts who consent to participate in the trial will first be screened by the research staff for signs and symptoms of leprosy and tuberculosis (TB) and checked for eligibility criteria. Exclusion criteria for the administration of preventive treatment are refusal to provide informed consent, a history of liver, renal or cardiac disease or a known allergy to RMP and/or CLR. Contacts are also temporarily ineligible if: pregnant or breastfeeding, under the age of two years, received RMP for any reason in the last two years or using contraindicated medicines for non-chronic use. If these conditions change during the trial intake period, they may still be included in the study. In the presence of any possible signs of leprosy, the participant will be referred to the closest qualified health centre for confirmation of diagnosis. Those who are confirmed as a new leprosy case are subsequently requested to list their close contacts for enrolment in the study.

Randomisation

The allocation of intervention and control arms in the countries has been done blindly using stratified randomisation. For each country, the randomisation unit has been determined via consultation with local government officials and project staff. This resulted in the use of territories/neighbourhoods in Brazil, blocks in India, municipalities in Nepal and unions in Bangladesh. These randomisation units were divided into two strata based on the number of clusters and the number of cases in clusters (based on geospatial analysis results with data from 2015–2020) followed by random sampling into intervention or control arms.

Sample size

The calculation of the sample size is based on the primary objective to find differences in new case detection rates (NCDR). The first is a difference in NCDR in the population in the intervention area after four years, compared to the baseline rate in 2019. The year 2019 is selected as the baseline to avoid any effect of the COVID-19 pandemic. The second is the difference in incidence rates in the contact groups between the close contacts who have received PEP++ and controls who have received SDR-PEP.

The sample size calculation for the difference in rates in the close contacts is based on the NCDR found in contacts in the COLEP trial sample. The NCDR in the SDR intervention group was 291/100,000 over 2 years, or an annual rate of 146/100,000. To reduce this rate by 50%, using a power of 90%, a significance level of 0.05, design effect of 1.5, correction for non-eligibility of 25% and loss to follow-up of 25%, we aimed to enrol 202,360 close contacts. This should result in giving PEP to at least 162,000 participants.

Outcome measures

The primary outcome measures of the trial will be the number of new leprosy cases detected, the number of child cases detected, and the incidence rate ratios of each arm compared with 2019 baseline. These will be measured at the two-year follow-up of the trial. In addition, the cost-effectiveness and acceptability of PEP++ as a preventive intervention will be measured as separate side studies. The added value of geospatial methods will be measured by the proportion of new leprosy cases in clusters and non-clusters.

Intervention implementation

Post-exposure prophylaxis will be offered to all eligible participants. Those in the intervention arm will receive three doses of RMP/CLR (PEP++ regimen) with four-week intervals. To increase the acceptability and reduce the potential adverse events, extended-release clarithromycin will be offered where available for purchase. An additional four-week tolerance exists for each repeated dose so that an interval of eight weeks between doses is valid. Participants in the control arm will receive SDR-PEP. The medication dosages of the PEP++ and SDR-PEP regimens per age group are presented in Table  2 . Both regimens will be provided only under supervision of the research staff and/or medical officers. The date, dosage and type of PEP will be recorded for each eligible person in the study.

The risk of inducing rifampicin resistance in M. leprae or M. tuberculosis after providing a single dose of rifampicin is considered negligible because only repeated doses of a single antibiotic will increase the risk of resistance [ 15 ]. Moreover, participants that have received rifampicin in the last two years or that show any signs or symptoms of TB or leprosy will not receive PEP. Contacts confirmed to have TB or leprosy will be treated according to the national guidelines.

Leprosy perception

A person’s perception of leprosy can negatively affect health-seeking behaviour and the acceptance of new interventions [ 16 ]. Therefore, contextualised community education and behaviour change (CEBC) materials are developed as part of the study to change the perception (knowledge, attitudes, beliefs, and emotions) regarding leprosy and reduce stigma, as well as to increase the community acceptance of preventive treatment. First, a leprosy perception study was conducted in each country to investigate the perceptions of leprosy patients, contacts, community members and health workers regarding leprosy, i.e., the way people see leprosy, what they know about leprosy and their attitudes, beliefs and reported behaviour towards persons affected by leprosy [ 17 , 18 ]. A mixed-method approach will be used to measure the perception, including in-depth interviews, focus group discussions (FGDs), the knowledge, attitudes, and practices (KAP) tool, the Explanatory Model Interview Catalogue Community Stigma Scale (EMIC-CSS) and the Social Distance Scale (SDS). During a workshop, leprosy affected persons and other key stakeholders will use the outcomes of the perception study to develop messages for the CEBC materials. The CEBC materials will be distributed throughout the implementation areas and piloted before the start of the trial [ 19 ].

Geospatial methods

Prior to the trial implementation, geospatial analysis was done to develop epidemiological maps and determine the target areas for blanket campaigns. The latitude and longitude of all patient’s residents registered from 2015 to 2021 were collected using the mobile application MapitPro (version 7.6.0). All data points were checked for clustering in open-source Quantum Geographic Information System (QGIS) version 3.4.1 (QGIS Developer team, Madeira (2018)). A contextualised spatial analysis approach was developed to identify small and precise clusters in each implementation area. This approach included non-statistical geospatial methods combined with expert consultation. During the expert consultations, country-specific definitions of a cluster were determined considering the local context [ 20 ].

  • Blanket campaigns

As predicted by mathematical modelling, a larger reduction in new cases detected can be achieved by implementing additional blanket campaigns in the identified clusters. For each index case in a cluster, the households not listed as close (neighbour) contacts within a radius of approximately 100 m will be visited until 80 to 100 participants (blanket contacts) are included in the study. If this results in over 80% of the cluster area being covered, then the entire population will be invited for participation. These blanket contacts are, similar to the close contacts, asked for consent to participate in the trial, screened for signs and symptoms of leprosy and when eligible, are provided SDR-PEP.

Data collection

Data collection will be done offline using the Research Electronic Data Capture (REDCap) system created at Vanderbilt University in the United States. Specific forms and modules were developed by the study teams, translated to the local languages, and downloaded to tablets and smartphones for field usage. First, a unique identification code (UIC) will be created for each close contact in the trial to anonymise the data. The UIC is linked to the index case and will be used to link the different data collection forms that will be created for each contact. Contacts are then visited at home and asked to sign a consent form if they are willing to participate. Disclosure of the identity of the index case is avoided whenever possible when approaching neighbours and social contacts.

From all contacts, we will collect demographic data (e.g., name, age, gender), information on the relationship with the index case and the results from the screening and eligibility questions in the close contact registration form. The date and dosage of the PEP administered is recorded in either the SDR-PEP form or PEP++ forms (i.e., first, second and third dose). Moreover, any side effects or adverse events due to SDR-PEP or PEP++ will be registered in the adverse event form. During the trial, the GPS coordinates of all participants and new leprosy patients will be collected to determine the spatial effectiveness of PEP. all case records and GPS data are stored temporarily on the mobile study devices and uploaded daily to a national server in each country.

Data analysis

Data from the PEP++ trial will be analysed primarily through quantitative methods using descriptive analysis for all variables. Table 3 outlines the tools to be used to measure and analyse the study outcomes.

Dissemination

Study outcomes are expected to be applicable for wider replication and scaling up throughout the four study countries, as well as in other countries with highly endemic regions. The national and international study teams will write extensively on the outcomes of the study. All publications will go before a project publications committee with the approval of the local Principal Investigator before submission to open-access, peer-reviewed journals. It will also be well represented in international congresses and events by the study teams in the four countries involved. Communications with the WHO and relevant Ministries of Health will take place throughout the project with the long-term sustainability of the approach in mind.

Although SDR-PEP distribution to household contacts is currently the standard WHO-recommended preventive treatment for leprosy, it may not be sufficient to stop leprosy infection in contacts who are already incubating leprosy disease. In the COLEP trial, the effectiveness of SDR-PEP was lowest among blood-relatives and household contacts [ 10 ]. To meet the needs of this group, recent studies with stronger regimens and/or different antibiotics have been conducted or are ongoing. The PEOPLE project in the Comoros and Madagascar assessed the effectiveness of a single double dose of rifampicin (SDDR) among household contacts [ 21 ]. Preliminary study data show that SDDR is effective in reducing the risk of leprosy even among household contacts and also at the population level (Hasker et al., accepted for publication). However, the effectiveness was still limited and the authors suggested that other stronger regimens should be tested. Therefore, the same study group is currently evaluating the effectiveness of a new enhanced PEP regimen consisting of bedaquiline and rifampicin in a four-year trial in the Comoros [ 22 ]. Bedaquiline is a more potent and longer acting antibiotic, which is used for latent TB infection and multidrug resistant pulmonary TB. It has not been used as leprosy preventive treatment before and is therefore undergoing a series of drug trials as part of this study. Rifapentine is another potent and longer-acting antibiotic that has been known as a promising candidate in the fight against leprosy for many years. A recent study in Southwest China showed that a single dose of rifapentine reduced the cumulative incidence of leprosy among household contacts by 84% compared with an untreated control group. It was considerably more effective than single-dose rifampicin in the target group [ 23 ]. If made available at affordable rates in all leprosy endemic countries, these powerful bactericidal agents offer hope of better single-dose protective regimens in the future.

Nevertheless, the study by Lenz et al. (2020) of M. leprae -infected nude mice showed that four multi-drug, multi-dose regimens were more effective to stop bacterial growth than any single-dose intervention, even those using multiple antibiotics with rifapentine [ 14 ]. The team concluded that ‘multi-dose PEP may be required to control infection in highly susceptible individuals with subclinical leprosy to prevent disease and decrease transmission.’ The PEP++ trial seeks to provide evidence to back up these laboratory findings and show that an enhanced multi-dose regimen of existing antibiotics is a powerful tool to reduce the risk of leprosy across a range of health systems under real-life field conditions.

Finally, several studies show that targeting the households of leprosy patients only is insufficient to stop transmission [ 10 , 21 , 24 ]. Community members within a 100-m radius of the leprosy patient’s house are also at a higher risk to develop leprosy. Targeted population-wide approaches or focal mass drug administration campaigns are therefore also needed. Little evidence has been published on the impact and cost-effectiveness of these approaches. The five-islands study in Indonesia by Bakker et al. (2005) compared a population-wide SDR-PEP approach with a contact-based approach [ 6 ]. They found a larger decrease in the risk of leprosy on the island using the population-wide approach. However, this was an island setting with low mobility of the population. Therefore, to collect additional evidence on the effectiveness of population-wide approaches, we will conduct blanket campaigns in identified clusters targeting 80 to 100 community members per index case.

A large-scale reduction in transmission across a large area will require a comprehensive application of ‘PEP services’, including SDR-PEP distribution to neighbours and social contacts, active case detection, community education and capacity strengthening of health workers. Following the individual- and population-level data analysis of this study, we expect to have evidence that it is possible to accelerate the reduction in leprosy incidence within a few years through implementation of the enhanced PEP++ regimen combined with SDR-PEP blanket campaigns and health system strengthening components as appropriate.

Availability of data and materials

No datasets were generated or analysed during the current study.

Abbreviations

Community education and behaviour change

  • Clarithromycin

Contact transmission and chemoprophylaxis in leprosy study

European Medicines Agency

Explanatory Model Interview Catalogue Community Stigma Scale

Geographic Information System

Global Positioning System

Knowledge, Attitudes and Practices measure

Leprosy post-exposure prophylaxis programme

Multi-drug therapy

  • Post-exposure prophylaxis

Enhanced multi-drug, multi-dose regimen leprosy post-exposure prophylaxis

Quantum Geographic Information System

Randomised controlled trial

Research Electronic Data Capture System

Single dose of rifampicin as leprosy post-exposure prophylaxis

Social Distance Scale

Stochastic individual-based model for transmission and control of leprosy

Tuberculosis

Unique identification code

World Health Organization

Fischer M. Leprosy – an overview of clinical features, diagnosis, and treatment. JDDG J Ger Soc Dermatology. 2017;15:801–27. https://doi.org/10.1111/ddg.13301 .

Article   Google Scholar  

Walker SL, Lockwood DNJ. The clinical and immunological features of leprosy. Br Med Bull. 2006;77–78(1):103–21. https://doi.org/10.1093/bmb/ldl010 .

Article   PubMed   Google Scholar  

World Health Organization. Global leprosy (Hansen disease) update, 2022: new paradigm - control to elimination. Wkly Epidemiol Rec. 2023;98:409–30.

Google Scholar  

World Health Organization. Global leprosy (Hansen disease) update, 2019: time to step-up prevention initiatives. Wkly Epidemiol Rec. 2020;95:417–40.

Smith CM, Smith WCS. Chemoprophylaxis is effective in the prevention of leprosy in endemic countries: a systematic review and meta-analysis. J Infect. 2000;41:137–42. https://doi.org/10.1053/jinf.2000.0698 .

Article   CAS   PubMed   Google Scholar  

Bakker MI, Hatta M, Kwenang A, Van Benthem BHB, Van Beers SM, Klatser PR, et al. Prevention of leprosy using rifampicin as chemoprophylaxis. Am J Trop Med Hyg. 2005;72:443–8 PMID: 15827283.

Moet FJ, Oskam L, Faber R, Pahan D, Richardus JH. A study on transmission and a trial of chemoprophylaxis in contacts of leprosy patients: design, methodology and recruitment findings of COLEP. Lepr Rev. 2004;75(4):376–88 PMID: 15682975.

Barth-Jaeggi T, Steinmann P, Mieras L, van Brakel WH, Richardus JH, Tiwari A, Bratschi MW, Cavaliero A, vander Plaetse B, Mirza F, Aerts A. Leprosy Post-Exposure Prophylaxis (LPEP) programme: study protocol for evaluating the feasibility and impact on case detection rates of contact tracing and single dose rifampicin. BMJ Open. 2016;6:e013633. https://doi.org/10.1136/bmjopen-2016 .

Article   PubMed   PubMed Central   Google Scholar  

Richardus R, Alam K, Kundu K, Roy JC, Zafar T, Chowdhury AS, et al. Effectiveness of single-dose rifampicin after BCG vaccination to prevent leprosy in close contacts of patients with newly diagnosed leprosy: A cluster randomized controlled trial. Int J Infect Dis. 2019;88:65–72.

Moet FJ, Pahan D, Oskam L, Richardus JH. Effectiveness of single dose rifampicin in preventing leprosy in close contacts of patients with newly diagnosed leprosy: cluster randomised controlled trial. BMJ. 2008;336:761–4. https://doi.org/10.1136/bmj.39500.885752.BE .

Richardus JH, Tiwari A, Barth-Jaeggi T, Arif MA, Banstola NL, Baskota R, et al. Leprosy post-exposure prophylaxis with single-dose rifampicin (LPEP): an international feasibility programme. Lancet Glob Health. 2020;9(1):e81–90. https://doi.org/10.1016/S2214-109X(20)30396-X .

World Health Organization. Guidelines for the Diagnosis, Treatment and Prevention of Leprosy. 1st ed. Geneva: WHO; 2018. p. 106.

Mieras LF, Taal AT, van Brakel WH, Cambau E, Saunderson PR, Smith WCS, et al. An enhanced regimen as post-exposure chemoprophylaxis for leprosy: PEP++. BMC Infect Dis. 2018;18:1–8. https://doi.org/10.1186/s12879-018-3402-4 .

Article   CAS   Google Scholar  

Lenz SM, Collins JH, Ray NA, Hagge DA, Lahiri R, Adams LB. Post-exposure prophylaxis (PEP) efficacy of rifampin, rifapentine, moxifloxacin, minocycline and clarithromycin in a susceptible-subclinical model of leprosy. PLoS Negl Trop Dis. 2020;14(9):e0008583. https://doi.org/10.1371/journal.pntd.0008583 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mieras L, Anthony R, van Brakel W, Bratschi MW, van den Broek J, Cambau E, et al. Negligible risk of inducing resistance in Mycobacterium tuberculosis with single-dose rifampicin as post-exposure prophylaxis for leprosy. Infect Dis Poverty. 2016;5:1–5. https://doi.org/10.1186/s40249-016-0140-y .

Nicholls PG, Ross L, Smith WCS. Promoting early detection in leprosy - a literature review to identify proven and potential interventions addressing patient-related delay. Lepr Rev. 2006;77:298–310.

van’t Noordende AT, Korfage IJ, Lisam S, Arif MA, Kumar A, van Brakel WH. The role of perceptions and knowledge of leprosy in the elimination of leprosy: A baseline study in Fatehpur district, northern India. PLoS Negl Trop Dis. 2019;13(4):e0007302. https://doi.org/10.1371/journal.pntd.0007302 .

van’t Noordende AT, Lisam S, Ruthindartri P, Sadiq A, Singh V, Arifin M, Korfage IJ. Leprosy perceptions and knowledge in endemic districts in India and Indonesia: differences and commonalities. PLoS Negl Trop Dis. 2021;15(1):e0009031. https://doi.org/10.1371/journal.pntd.0009031 .

van’t Noordende AT, Lisam S, Singh V, Sadiq A, Agarwal A, Hinders DC, Korfage IJ. Changing perception and improving knowledge of leprosy: An intervention study in Uttar Pradesh, India. PLoS Negl Trop Dis. 2021;15(8):e0009654. https://doi.org/10.1371/journal.pntd.0009654 .

Taal AT, Garg A, Lisam S, Agarwal A, Barreto JG, van Brakel WH, Richardus JH, Blok DJ. Identifying clusters of leprosy patients in India: A comparison of methods. PLoS Negl Trop Dis. 2022;16(12):e0010972. https://doi.org/10.1371/journal.pntd.0010972 .

Ortuno-Gutierrez N, Younoussa A, Randrianantoandro A, Braet S, Cauchoix B, Ramboarina S, et al. Protocol, rationale and design of PEOPLE (Post ExpOsure Prophylaxis for LEprosy in the Comoros and Madagascar): a cluster randomized trial on effectiveness of different modalities of implementation of post-exposure prophylaxis of leprosy contacts. BMC Infect Dis. 2019;19(1):1–7. https://doi.org/10.1186/S12879-019-4649-0 .

Younoussa A, Samidine SN, Bergeman AT, Piubello A, Attoumani N, Grillone SH, et al. Protocol, rationale and design of BE-PEOPLE (Bedaquiline enhanced exposure prophylaxis for LEprosy in the Comoros): a cluster randomized trial on effectiveness of rifampicin and bedaquiline as post-exposure prophylaxis of leprosy contacts. BMC Infect Dis. 2023;23(1):310. https://doi.org/10.1186/s12879-023-08290-0 . PMID:37161571;PMCID:PMC10169125.

Wang L, Wang H, Yan L, Yu M, Yang J, Jinlan Li, et al. Single-dose rifapentine in household contacts of patients with leprosy. N Engl J Med. 2023;388(20):1843–52. https://doi.org/10.1056/NEJMoa2205487 .

van Beers SM, Hatta M, Klatser PR. Patient contact is the major determinant in incident leprosy: implications for future control. Int J Lepr Other Mycobact Dis. 1999;67(2):119–28.

PubMed   Google Scholar  

Download references

Acknowledgements

Our appreciation goes to all those involved in the PEP++ research consortium (listed individually below), the study participants, and their communities. We are grateful for the support, time and dedication of the following partners involved in each country:

* Bangladesh: The Leprosy Mission International Bangladesh (study coordinator); Federal Ministry of Health, Dhaka; District Governments of Nilphamari and Rangpur

* Brazil: NHR Brasil (study coordinator), the Federal University of Ceará (UFC) – Clinical Research Unit, Federal Ministry of Health, State Health Secretariat of Ceará, Municipal Health Secretariats of Fortaleza and Sobral, School of Family Health (Sobral), MORHAN

* India: NLR India (study coordinator), Vardhman Mahavir Medical College/Safdarjung Hospital, Federal Ministry of Health and Family Welfare, Central Leprosy Division; Uttar Pradesh State Leprosy Office, Chandauli and Fatehpur District Leprosy Offices, The Leprosy Mission Trust India (TLMTI), APAL

* Nepal: NLR Nepal (study coordinator), B.P. Koirala Institute of Health Sciences, Federal Ministry of Health and Population, Madesh Pradesh Provincial Government, Dhanusha and Mahottari District Governments, Nepal Leprosy Trust (NLT), The Leprosy Mission Nepal (TLMN)

* Netherlands: Erasmus University Medical Center, Rotterdam

* United Kingdom: The Leprosy Mission International, Brentford.

PEP++ Research Consortium

Bangladesh: Abu Sufian Chowdhury, Abhijit Saha, Shaikh A. Shahed Hossain, Salomon Sumon Halder, Surendra Nath Singh, Rishad Choudhury Robin, Sohel Marndi, Md. Anwar Hossain, Md. Rashidul Hasan, Johan Chandra Roy

Brazil: Virginia O. Fernandes, Aymée M da Rocha, Marcos Virmond, Zoica Bakirtzief Pereira, José A. Menezes da Silva, Juliana Cavalcante Ribeiro Ramos, Alberto Novaes Ramos Jr, Jaqueline Caracas Barbosa, Josafá Gonçalves Barreto, Adriana Reis, Nagila Nathaly Lima Ferreira, Renato Lima

France: Emmanuelle Cambau

India: Jugal Kishore, Suchitra Lisam, Ashok Agarwal, Danish Suhail, Anil Kumar, Sudarshan Mandal, Jaya Dehalvi, Hemanta K. Kar, Atif Sadiq, Sanjay K. Srivastava, Akshat Garg

Indonesia: Cita Rosita Prakoeswa, Asken Sinaga, Teky Budiawan, Linda Astari, Md. Atoillah Isfandiari, Ulfah Abqari

Nepal: Nand Lal Banstola, Prativa Bhandari, Bikash M. Singh, Nilamber Jha, Andriyash Magar, Bipin K. Yadav, Milan Tamang, Yugal K. Singh

Netherlands: Wim H. van Brakel, Duane C. Hinders, Anneke T. Taal, Anna T. van ’t Noordende, Liesbeth Mieras, Jan Hendrik Richardus, Tom Hambridge, Daan Nieboer

Norway: Paul Saunderson

United Kingdom: Jannine Ebenso, Cairns Smith

Funding for the study in Brazil, India and Nepal has been put forward by the Dutch Postcode Lottery (NPL) with counterpart funding made available by NLR, Amsterdam. Financial support for the intervention in Bangladesh has been provided by The Leprosy Mission International (TLMI) and its global fellowship members. The funders had no role in development, implementation, analysis of results, or preparation of the manuscript.

Author information

Authors and affiliations.

NLR, Amsterdam, The Netherlands

Duane C. Hinders, Anneke T. Taal, Anna T. van ‘t Noordende, Liesbeth Mieras & Wim H. van Brakel

NLR India, Delhi, India

Suchitra Lisam

NHR Brasil, Fortaleza, Brazil

Aymée M. da Rocha

NLR Nepal, Kathmandu, Nepal

Nand Lal Banstola & Prativa Bhandari

TLMI Bangladesh, Dhaka, Bangladesh

Abhijit Saha & Abu Sufian Chowdhury

Vardhman Mahavir Medical College/Safdarjung Hospital, Delhi, India

Jugal Kishore

Federal University of Ceará, Fortaleza, Brazil

Virginia O. Fernandes

Erasmus MC, University Medical Center, Rotterdam, The Netherlands

Jan Hendrik Richardus

You can also search for this author in PubMed   Google Scholar

Contributions

DH, ATT, ATN, LM, WB designed the study. DH, ATT, ATN, LM and WB drafted the manuscript. All authors revised and approved the manuscript.

Corresponding author

Correspondence to Duane C. Hinders .

Ethics declarations

Ethics approvals and consent to participate.

The PEP++ study has gone before and been approved by institutional review boards in Bangladesh, Brazil, India, and Nepal under the names of the national Principal Investigators. It has also been reviewed favourably by the Bangladesh Medical Research Council, Indian Council of Medical Research, and Nepal Health Research Council. Each index case, close contact and blanket contact approached in the project receives verbal and written study information from the research assistants and be requested to sign a paper informed consent form (ICF). Participants receive details on possible side effects from the antibiotics and have the right to refuse participation or withdraw informed consent at any point in the study. When dealing with a child/adolescent under the age of 18 or a person with mental disabilities, the ICF will be presented to and signed by the legal guardian. The ICFs will be stored and catalogued according to the national requirements of each country. The PEP++ clinical trial and project are registered on Dutch Trial Register, under trial registration NL7022 (formerly NTR7221) with the title Stop the Transmission of Leprosy. It was originally registered on April 12, 2018, and last updated on August 18, 2022. https://www.onderzoekmetmensen.nl/en/trial/23060 .

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Hinders, D.C., Taal, A.T., Lisam, S. et al. The PEP++ study protocol: a cluster-randomised controlled trial on the effectiveness of an enhanced regimen of post-exposure prophylaxis for close contacts of persons affected by leprosy to prevent disease transmission. BMC Infect Dis 24 , 226 (2024). https://doi.org/10.1186/s12879-024-09125-2

Download citation

Received : 10 January 2024

Accepted : 12 February 2024

Published : 20 February 2024

DOI : https://doi.org/10.1186/s12879-024-09125-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • High-endemic areas

BMC Infectious Diseases

ISSN: 1471-2334

case control study multiple outcomes

Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • Current issue
  • Hosted content
  • BMJ Journals More You are viewing from: Google Indexer

You are here

  • Online First
  • Proton pump inhibitors and the risk of inflammatory bowel disease: a Mendelian randomisation study
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

Download PDF

  • Hongjin An 1 ,
  • Min Zhong 1 ,
  • http://orcid.org/0000-0002-5736-1283 Huatian Gan 2 , 3
  • 1 Department of Gastroenterology and Hepatology, West China Hospital, Sichuan University , Chengdu , China
  • 2 Department of Geriatrics and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University , Chengdu , China
  • 3 Department of Gastroenterology and Laboratory of Inflammatory Bowel Disease, the Center for Inflammatory Bowel Disease, Clinical Institute of Inflammation and Immunology, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University , Chengdu , China
  • Correspondence to Dr Huatian Gan, West China Hospital of Sichuan University, Chengdu, Sichuan, China; ganhuatian123{at}163.com

https://doi.org/10.1136/gutjnl-2024-331904

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

  • INFLAMMATORY BOWEL DISEASE

We read with great interest the population-based cohort study by Abrahami D et al , 1 in which they found that the use of proton pump inhibitors (PPIs) was not associated with an increased risk of inflammatory bowel disease (IBD). However, the assessment of causality in observational studies is often challenging due to the presence of multiple confounding factors. The existence of a causal relationship between PPIs and IBD remains unclear at present. Mendelian randomisation (MR) is a method of generating more reliable evidence using exposure-related genetic variants to assess causality, limiting the bias caused by confounders. 2 Therefore, we used a two-sample MR analysis to investigate the association between the use of PPIs and IBD including Crohn’s disease (CD) and ulcerative colitis (UC).

Supplemental material

Here, we mainly used the inverse-variance weighted 8 method for MR analysis with weighted median, 9 MR-Egger 10 and MR-PRESSO 5 as complementary approaches. Furthermore, we applied a series of sensitivity analyses to ensure the robustness of our results, with Cochran’s Q test to assess heterogeneity and the intercept of an MR-Egger regression to assess horizontal pleiotropy. The genetic prediction of omeprazole, esomeprazole, lansoprazole and rabeprazole use, as depicted in figure 1 , demonstrated no significant association with an increased risk of IBD after excluding pleiotropic SNPs (omeprazole, OR, 1.05; 95% CI, 0.88 to 1.25; p=0.587; esomeprazole, OR, 0.99; 95% CI, 0.92 to 1.07; p=0.865; lansoprazole, OR, 1.06; 95% CI, 0.89 to 1.26; p=0.537; and rabeprazole, OR, 1.00; 95% CI, 0.95 to 1.04; p=0.862). The IBD subtype analyses also did not reveal any evidence of an increased risk of CD or UC associated with the use of PPIs ( figure 1 ). These findings were robustly confirmed through complementary approaches employing rigorous methodologies that consistently yielded similar point estimates ( figure 1 ). Further sensitivity analyses showed the absence of heterogeneity (All P heterogeneity >0.05) and pleiotropy (All P pleiotropy >0.05), again demonstrating the robustness of the conclusions ( figure 1 ).

  • Download figure
  • Open in new tab
  • Download powerpoint

Mendelian randomisation estimates the associations between the use of different types of proton pump inhibitors and inflammatory bowel disease. IBD, inflammatory bowel disease; CD, Crohn’s disease; UC, ulcerative colitis; PPIs, proton pump inhibitors; IVW, inverse-variance weighted; MR, Mendelian randomisation.

In conclusion, the MR results corroborate Abrahami D et al ’s findings that PPIs were not associated with an increased risk of IBD. Nonetheless, further research is needed to elucidate the effects of more types, drug dosage, frequency and duration on IBD.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

  • Abrahami D ,
  • Pradhan R ,
  • Yin H , et al
  • Kathiresan S
  • Fang H , et al
  • van Sommeren S ,
  • Huang H , et al
  • Verbanck M ,
  • Neale B , et al
  • Tilling K ,
  • Davey Smith G
  • Brion M-JA ,
  • Shakhbazov K ,
  • Visscher PM
  • Burgess S ,
  • Timpson NJ , et al
  • Davey Smith G ,
  • Haycock PC , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

  • Data supplement 1

HA and MZ contributed equally.

Contributors All authors conceived and designed the study. HA and MZ did the statistical analyses and wrote the manuscript. HG revised the manuscript and is the guarantor. HA and MZ have contributed equally to this study.

Funding The present work was supported by the National Natural Science Foundation of China (No. 82070560) and 1.3.5 Project for Disciplines of Excellence, West China Hospital, Sichuan (No. ZYGD23013).

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

IMAGES

  1. Case-control and Cohort studies: A brief overview

    case control study multiple outcomes

  2. PPT

    case control study multiple outcomes

  3. what is a case study control

    case control study multiple outcomes

  4. Multiple Case Study Method

    case control study multiple outcomes

  5. PPT

    case control study multiple outcomes

  6. PPT

    case control study multiple outcomes

VIDEO

  1. Case Control Study (Lecture

  2. Understanding of Program Outcome, Program Specific Outcome and Course Outcome

  3. Case Studies

  4. Case Control Study Part 1

  5. Case-control study design

  6. case control study part 2 || epidemiology|| PSM|| @Sudarshan263

COMMENTS

  1. Design and Analysis of Multiple Events Case-Control Studies

    Multiple events case-control (MECC) studies provide a new approach to sampling from a cohort and are useful when it is desired to study multiple types of events in the cohort. In this design, subjects in the cohort who develop any event of interest are sampled, as well as a fraction of the remaining subjects.

  2. A Practical Overview of Case-Control Studies in Clinical Practice

    In a case-control study the researcher identifies a case group and a control group, with and without the outcome of interest. Such a study design is called observational because the researcher does not control the assignment of a subject to one of the groups, unlike in a planned experimental study.

  3. 5

    Case-control studies can involve more than two outcome groups, enabling us to estimate and compare exposure-outcome associations across groups. Studies may involve multiple case subtypes and a single control group, or one case group and two or more control groups, for example.

  4. A Practical Overview of Case-Control Studies in Clinical Practice

    In a classic case-control study, the outcome of interest is known, and prior exposure histories are obtained (ie, a retrospective design). Case-control studies are different from a retrospective cohort study design.

  5. A Practical Overview of Case-Control Studies in Clinical Practice

    The following questions provide clinicians with a general guideline about how to proceed with a case-control study: 1. Consider the nature of the case-control study. a. If assessing multiple disease outcomes, consider a case-cohort study design. b. If there are transient risk factors, consider a case-crossover study design. c.

  6. What Is a Case-Control Study?

    A case-control study is an experimental design that compares a group of participants possessing a condition of interest to a very similar group lacking that condition. Here, the participants possessing the attribute of study, such as a disease, are called the "case," and those without it are the "control."

  7. Case Control Studies

    A case-control study is a type of observational study commonly used to look at factors associated with diseases or outcomes. The case-control study starts with a group of cases, which are the individuals who have the outcome of interest.

  8. Analysis of matched case-control studies

    Options for analysing case-control studies. Unmatched case-control studies are typically analysed using the Mantel-Haenszel method10 or unconditional logistic regression.4 The former involves the familiar method of producing a 2×2 (exposure-disease) stratum for each level of the confounder (eg, if there are five age groups and two sex groups, then there will be 10 2×2 tables, each showing ...

  9. Case-control study: Design, measures, classic example

    A case-control study is an investigation involving two groups of patients: a study ("case") group who have known medical condition (s), disease state, and a control group of similar patients without the focus of inquiry. 1. Exposure data are retrospectively collected for both groups. The primary outcome of a case-control study is the ...

  10. 6.3

    In a cohort study of risk factors for suicide, Agerbo et al. (2002), enrolled 496 young people who had committed suicide during 1981-97 in Denmark matched for sex, age, and time to 24,800 controls. Read how they matched each case to a representative random subsample of 50 people born the same year! Case-Cohort Study Design

  11. Study Designs Revisited

    Case-control studies can assess multiple exposures, though they are limited to one outcome by definition. Case-control studies assess exposure in the past. Occasionally, these past exposure data come from existing records (e.g., medical records for a person's blood pressure history), but usually we rely on questionnaires. Case-control studies ...

  12. Case-control and Cohort studies: A brief overview

    Case-control studies should include two groups that are identical EXCEPT for their outcome / disease status. As such, controls should also be selected carefully. It is possible to match controls to the cases selected on the basis of various factors (e.g. age, sex) to ensure these do not confound the study results.

  13. Nested Case-Control Studies in Cohorts with Competing Events

    The nested case-control design is the most widely used method for sampling from epidemiologic cohorts when investigators need to collect additional data in a reduced sample. 1 Using incidence density sampling, the potential impact of exposures on disease occurrence can be studied by hazard ratios in a reduced data set. 1, 2 Furthermore, the ...

  14. Case-control study

    A case-control study (also known as case-referent study) is a type of observational study in which two existing groups differing in outcome are identified and compared on the basis of some supposed causal attribute.

  15. Comparison of estimators in nested case-control studies with multiple

    Reuse of controls in a nested case-control (NCC) study has not been considered feasible since the controls are matched to their respective cases. However, in the last decade or so, methods have been developed that break the matching and allow for analyses where the controls are no longer tied to their cases. These methods can be divided into two groups; weighted partial likelihood (WPL ...

  16. 7.2.1

    The subcohort can be used to study multiple outcomes; Risk can be measured at any time up to \(t_1\) (e.g. elapsed time from a variable event, such as menopause, birth) Subcohort can be used to calculate person-time risk; Disadvantages of Case-Cohort Study: As compared to nested case-control study design: Increased potential for information ...

  17. Case-control study of adverse childhood experiences and multiple ...

    Background Adverse childhood experiences (ACEs) are linked to numerous health conditions but understudied in multiple sclerosis (MS). This study's objective was to test for the association between ACEs and MS risk and several clinical outcomes. Methods We used a sample of adult, non-Hispanic MS cases (n = 1422) and controls (n = 1185) from Northern California. Eighteen ACEs were assessed ...

  18. Sustaining the collaborative chronic care model in outpatient mental

    One such method is Matrixed Multiple Case Study (MMCS) , which is beginning to be applied in implementation research to identify factors related to implementation [2,3,4,5]. MMCS capitalizes on the many contextual variations and heterogeneous outcomes that are expected when an EBP is implemented across multiple sites.

  19. Cochlear-facial dehiscence

    Outcomes. The primary outcome was the presence of a CFD on the preoperative CT scan of the implanted ear. ... The otosclerosis/OI factor had little effect on the OR for CFD in the multiple logistic regression model when compared to the univariate logistic regression model, ... We present the first case-control study of the prevalence of CFD ...

  20. The PEP++ study protocol: a cluster-randomised controlled trial on the

    Background Leprosy is an infectious disease with a slow decline in global annual caseload in the past two decades. Active case finding and post-exposure prophylaxis (PEP) with a single dose of rifampicin (SDR) are recommended by the World Health Organization as measures for leprosy elimination. However, more potent PEP regimens are needed to increase the effect in groups highest at risk (i.e ...

  21. Proton pump inhibitors and the risk of inflammatory bowel disease: a

    For this study, different cohort data sources were used for exposure and outcome to avoid sample overlap (online supplemental material). We used summary statistics from the medication use case-control genome-wide association studies conducted among UK Biobank study participants to generate genetic instruments for …