Introduction

Conducting and Reporting an Experimental Research Study

Important Terms

• Internal validity is the extent to which the results of a study are true. That is, it really is the intervention that caused the change in behavior, and not some other extraneous factor, such as differences in assessment procedures between intervention and control conditions.

  For example...
  
  

• External validity is the extent to which the results can be generalized to a population of interest. For the purposes of this module, the "population of interest" is up to you to determine. We will point out factors that you should take into account when deciding how well the results apply to your population.

• Bias is the systematic distortion of the real, true effect that results from the way a study is conducted. This can lead to invalid conclusions about whether an intervention works. Bias in research can make a treatment look better or worse than it really is. Bias can occur at almost any stage in the research process.

For example...

What is a Controlled Trial?

Sample Selection – Excerpt

Sample Selection – Target Population

• Is the target population identified?

  A good quality trial:

  • Specifies the target population clearly. (The researchers did this: they specified the target population as residents of rural counties in Arizona.)
  • Pulls the intervention and control participants from the same population.

  In theory, the researchers targeted the same population for both the intervention and control conditions in this study. But in reality, they recruited intervention and control participants from two different locations. This increases the risk that any difference in the study outcome (infant motor vehicle injuries) will reflect pre-existing differences in the kinds of people assigned to the study conditions. This is known as selection bias. For example, what if the control country has a higher proportion of Native American women who give birth at home? A consequence is that a smaller proportion of the new mothers in that particular county get exposed to any intervention delivered in a hospital. The study results might occur not because the intervention was effective, but rather, because a hospital-based intervention in one county reaches more of the population than it does in the other.

• Are inclusion or exclusion criteria clearly explained?

  A good quality trial tells the reader any characteristics that people needed to have (or not have) to be eligible for the study. Specification of enrollment criteria shows that the researchers had a clearly-identified target population and were thoughtful and systematic in their sample selection. It also helps readers to determine how well the sample reflects the population of interest to them.

  The Arizona researchers did not list inclusion criteria other than giving birth in one of the study hospitals. Conceivably, the researchers may really have attempted to counsel every woman (which would increase the generalizability of the research). We wonder, though, whether they implemented—but did not report—certain inclusion rules regarding the women's spoken language, vision, or access to transportation.

Selecting the Sample

• Who do they invite?

  A good quality trial specifies exactly how the researchers selected who to invite to participate in the research. Did they invite everyone in the population that met their inclusion criteria? Just a sample of those eligible?

  The Arizona researchers say that they recruited all women giving birth in hospitals located in the study counties. Confidence in the internal validity of a study lessens if the selection process is not described. That omission increases the likelihood that the selection was not systematic and therefore biased. For example, did the county hospitals all have nurse educators available seven days per week to recruit new mothers into the study? If not, then mothers discharged on days without a nurse educator may have been missed. We do not think recruitment omissions constituted a serious problem in the Arizona study, however, since nurse educators in the county hospitals contacted all but 2% of mothers of newborns (61/2959).

• Were the same procedures followed to recruit people to the intervention and control conditions?

  To minimize selection bias, a study can:

  • Select intervention and control participants from the same location (e.g., county, clinic, school, classroom), or recruit participants from more than one location for each study group

  If multiple locations are used, then a good quality trial:

  • Matches similar locations and randomly assigns the locations in each pair to their study arms
  • Assesses the characteristics of the locations and makes sure they are comparable

  The Arizona study recruited intervention and control participants from two different counties—one for each treatment group—which can be a concern. They did attempt to select two counties that were similar on some relevant characteristics, and the fact that they reported no differences in baseline characteristics of enrolled participants in the two counties was reassuring. However, since every county has unique features, we would have been more confident about the study's internal validity if they had randomized multiple counties to each study condition.

• Were the recruitment staff blind to group assignment?

  In a good quality trial, the person selecting study candidates is unable to figure out the treatment arm to which they will be assigned if found eligible. This is called "allocation concealment." If the recruiter is not blind to treatment allocation, this can bias the results of a study.

  The researchers did not say whether the person recruiting Arizona counties for the study knew which county would be assigned to the child safety seat intervention. It would, however, constitute a serious problem of bias if the researcher knew the condition to which a county would be assigned. That would introduce the chance that the recruiter might bias sample selection intentionally or inadvertently, by recruiting counties that give the intervention a better chance of succeeding.

  In the Arizona study, the nurse educators who were delivering the intervention also recruited participants into the study, so they were not blind to treatment assignment. This is a concern. However, it is not a major flaw for two reasons. First, the nurse educator did not determine whether the participant was eligible, but instead attempted to recruit everyone she saw. There is a much greater risk of selection bias if the recruiter is required to make a judgment about the participant's suitability for enrollment. Second, since the proportion who did not receive the intervention is low and similar between the two groups, the risk of bias in enrollment is minimal.

• How was the eligibility assessment conducted?

  A good quality trial conducts eligibility assessment in such a way as to maximize accuracy. For example, if family members were able to see or hear assessments, the participant may not have felt comfortable answering all questions honestly.

  This might have been a problem if the researchers targeted the intervention only to women who reported that their significant others were unsafe drivers. However, in this study, no eligibility assessment was conducted since all women in the study hospitals were considered eligible.

• How many participants were selected?

  A good quality trial provides:

  • A power calculation, showing the number of subjects needed for an expected effect size
  • A reasonable estimate of the intervention's expected effect size

  This study did not report a power calculation. However, since they attempted to enroll all eligible women (rather than selecting a subgroup), who are assumed to be nearly the entire eligible population, power calculations are not critical.

Sample Selection – Validity and Representativeness

• Is the target population representative of the population to which we want to generalize?

  A good quality trial states inclusion and exclusion rules to help us decide how well the study population mirrors our own.

• Is the sample a random (unbiased) representation of the population of interest?

  Even if the sample matches our population of interest perfectly in terms of demographics such as age and ethnicity, sometimes recruitment procedures lead the study sample to be a select, non-representative subgroup of the population.

  For example...

• What proportion of the eligible invitees agreed to enroll and were randomized?

  A good quality trial enrolls a high proportion of participants who are invited and known to be eligible for the study. If a large number of potentially eligible participants refused or were not randomized for some other reason, then the study sample may be a select, non-representative subgroup of the larger population.

  If we were public health officials in a rural U.S. county, we would probably conclude that sample selection procedures in the excerpted study had no detrimental effect on how well their study sample applies to our county. The degree to which a study in rural Arizona is applicable to our county is purely subjective.

For example...

Group Assignment – Excerpt

• From the Methods section:

  Once eligibility was confirmed and consent was received, families were randomized to either the group-based family intervention or single-family intervention arm. Randomization was computer-generated and took place off-site at the coordinating center, which notified both the participants and the project staff of the randomization results. Staff who conducted either intake interviews or outcomes assessment were blind to the treatment condition of the participants. Separate randomization tables were created for families where the index patient was a boy vs. a girl in order to maintain balanced groups.

• From the Results section:

  Baseline characteristics are presented in Table 1 and were compared between the two treatment arms. The average family size was larger in the group treatment arm (4.2 vs. 3.8), but no other baseline group differences were found. Therefore, family size was included as a covariate in all multivariate adjusted regression models. The average age of the index child was 11.2, and 69% of the index children were boys.

Group Assignment – Questions

• Allocation Concealment

  A good quality trial conceals treatment allocation when initial assessments are being made. As already noted, it is crucial that the person conducting initial assessments is blind to the participant's likely treatment assignment, especially if there are any judgments to be made about the participants' eligibility for the study. If allocation is not concealed, this is a serious concern. Assessment staff may unconsciously evaluate and treat participants differently depending on the condition to which they expect the person to be assigned.

  A strength of the ADHD study is that research staff conducting baseline assessments were blind to treatment allocation.

• Assignment to Treatment Groups

  A good quality trial provides enough information about the treatment assignment process to reassure the reader that intake assessors could not influence treatment assignment.

  • For example, a good quality trial explains how the group assignment was generated. Ideally the researcher assigns participants using a random number generator programmed into a computer or a random number table. Sometimes researchers use some other attributes like the person's name or social security number or birth date, or assign them to treatment groups based on the day of the week when the intake was conducted. The concern with these methods is the assignment may not be truly random. In that case, the person conducting the intake interview may be able to tell which group the prospective participant will be assigned to. That, in turn, can introduce bias.

  Group assignment in the ADHD study was excellent, inspiring confidence in the study's internal validity. Independent, off-site, research staff generated the random sequence via computer and kept on-site research staff and participants blind to treatment assignment.

• Stratified Randomization

  A good quality trial assesses and controls for differences in study locations and other important variables that may be related to the success of the intervention.

  • If there are known differences in demographics between the recruitment locations, it is helpful to stratify participant randomization within levels of these factors. For example...

  Since the researchers knew that there would be many more eligible boys than girls, they had separate randomization tables for boys and girls. This helped reduce the likelihood that sex would be a confounding factor. If one group were to wind up with a greater proportion of boys and had a less successful outcome, then it would be difficult to determine if the difference in results was due to the different interventions or the difference in gender distribution in the study arms.

  The researchers did not explicitly state whether they had separate randomization tables for each site. However, one purpose of a coordinating center is to ensure high quality, consistent data between study sites. So although we aren't given specific information about whether separate tables were used at each study site, the presence of a coordinating center and off-site randomization is reassuring since it suggests attention to the quality of random assignment.

• Baseline Characteristics

  A good quality trial tests for differences in the baseline characteristics of the treatment groups and, if found, accounts for these in the study analyses. Commonly examined baseline characteristics include age, gender, ethnicity, socio-economic status, level of severity and treatment history for the condition of interest.

  The ADHD study reported baseline characteristics of the groups and found only one, probably minor, difference between them. We are reassured by the lack of differences found on the many characteristics the researchers examined.

For example...

Assessment and Data Collection – Excerpt

• From the Methods section:

  Assessments were performed at baseline, and at 6, 12, and 18-months post-baseline. Data were collected in face-to-face interviews by staff who were not involved in the intervention and were unaware of the treatment group assignment of the participants. The primary outcome was the Social Functioning Scale, developed and validated on people diagnosed with schizophrenia. The seven subscales of the Social Functioning Scale measure withdrawal, relationships, social activities, recreation, independence competence, independence performance, and employment.

• From the Results section:

  There were no differences between the intervention and usual care groups in attrition at any of the followup assessments. Overall, 92% completed the 6-month assessment, 83% completed the 12-month assessment, and 82% completed the 18-month assessment.

Assessment and Data Collection – Questions

• Are the procedures the same for all treatment groups?

  A good quality trial assesses all treatment groups with the same instruments, in the same settings, in the same way. Any differences in assessment procedures between groups could bias the results. That is, the differences in procedures could be the real reason why the groups differ at the end of the study.

  The schizophrenia study appears to have used the same assessment procedures in the intervention and control groups.

• Are the assessment staff blind to treatment allocation?

  A good quality trial ensures that assessment staff are unaware of the treatment group assignment of the participants they assess. Knowing which treatment group a participant is in could subtly bias an interviewer.

  The researchers who conducted the schizophrenia study explicitly state that assessment staff were unaware of patients' treatment group assignment.

• Did they use valid and reliable assessment methods?

  A good quality trial provides references or information describing what is known about the reliability and validity of its data collection instruments. It can be a concern if the researcher uses instruments that have not been developed and tested previously.

  Researchers should take special care to describe the validity and reliability of instruments they use to assess behaviors that are complex or difficult to measure, or that have strong cultural influences. The researchers in this study used an established instrument (the Social Functioning Scale) to assess their primary outcome. However, it appears that their assessment tool was a symptom rating scale that requires a trained evaluator. It would have been preferable if the researchers had described their assessment staff's training and interrater reliability with the measurement tool.

• Is the amount of loss-to-followup reasonable and comparable between the groups?

  A good quality trial has low overall loss-to-followup (also known as attrition) and similar loss-to-followup between groups. If attrition is high, then we can't adequately assess the intervention's impact on the sample as a whole. Any systematic differences between groups in attrition introduces bias and reduces confidence that group differences at the end of the study are caused by the intervention. For example...

  In the schizophrenia study, attrition was within reasonable bounds. Also the researchers specifically tested for between group differences in attrition and found none.

For example...

Assessment and Data Collection – External Validity and Representativeness

Intervention – Excerpt

Intervention – Excerpt (continued)

Intervention – Questions

• Was the intervention clearly described?

  A good quality trial clearly describes all of the intervention conditions and control conditions. The description should include the overarching treatment model, specific topics covered and/or techniques used.

  • For individual-level interventions, the study should also describe the length and number of treatment sessions, and what kinds of materials were used (treatment manuals, patient hand-outs, video presentations, media used and content of media ads, etc.), who provided the intervention, and what kind of training they received on the study intervention. The setting in which the intervention took place should also be described.

  In the media intervention study to prevent adolescent smoking, the researchers clearly described all components of the intervention, including its dates. Dates can be important information for large-scale public health campaigns because news events or changes in laws and public policies may either enhance or interfere with the success of the intervention.

• Was the intervention delivered as intended?

  A good quality trial provides information on whether and how researchers assessed if the intervention was delivered as intended. This is sometimes known as treatment fidelity or treatment integrity.

  • In individual-level treatment studies, reports that interventionists were observed and rated or supervised provides some evidence that efforts were made to ensure treatment fidelity. If treatment fidelity was poor, then a study may not have provided an adequate test of the intervention.

  Treatment fidelity is usually not problematic for media campaigns like that used in the smoking prevention study, although researchers should monitor to ensure that advertisements aired.

• Was there adequate exposure or participation?

  A good quality trial will provide information on how well the intervention reached the target audience.

  • For media campaigns like “The Rest of the Story,” some measure of exposure to the campaign is analogous to participant participation or compliance. I.e., how many people saw one of the media announcements or visited a study website?
  • In studies involving direct intervention with individuals, researchers should quantify exposure via the average number of sessions completed by participants, and completion of homework, if homework was an integral part of the intervention. If only 15% of participants participated in one important component of the treatment, then the study may not have provided an adequate test of the intervention.

  The "Rest of the Story" study did not provide information on exposure, which is a flaw. If a treatment effect is not found, we will not know if it is because people didn’t see the ads or if they saw them and did not find them compelling. The researchers could have described the number of billboards per capita, the frequency and duration of television and radio announcements, and the number of hits on the website. They could also have surveyed teenagers in the community to assess what proportion saw or heard about the videos.

Intervention – External Validity

Data Analysis/Results – Excerpt

Data Analysis/Results – Excerpt (continued)

Data Analysis/Results – Questions

• How do the analyses address missing data?

  A good quality trial has minimal loss-to-followup and uses only conservative, appropriate methods for data imputation (substitution) when missing data are present. Data imputation methods allow you to estimate responses for those with missing data. They can be useful because they allow you to retain more people in your analysis, rather than dropping cases with missing data. If data imputation methods are used, the researcher should discuss how the results differ from those using the original data. A thorough discussion of data imputation methods is beyond the scope of this module, but if they are used, you should think carefully about how they might bias study results.

  Ms. Biggs’ study had attrition of about 12%, which is not excessive. Because the followup assessment was a paper-and-pencil test administered in the classroom, missing data were likely due to school absences and not related to the students’ experiences with the intervention. This study used the last observation carried forward method (LOCF), which substitutes the last known observation for missing data. A more conservative approach is to substitute the student’s baseline value for missing data, which assumes that the treatment had had no impact, rather than the last observation, which assumes that changes will be sustained. However, since in this case the LOCF analysis showed smaller effects than the “completers” analysis, we can conclude that the imputation methods did not exaggerate the effect size.

• Is attrition similar between the treatment groups?

  Differential attrition should be revisited when appraising the quality of the data analysis, especially if data imputation methods are used.

• Were reasonable data analysis methods used?

  For most study data there are multiple legitimate approaches to analysis. Some approaches have more limitations than others, and it is useful to have a good grounding in basic statistical approaches.

  In her study, Ms. Biggs conducted repeated measures ANOVAs. In most statistical packages, this analysis technique drops cases that have any missing observations, which is why Ms. Biggs used the LOCF method. Other more flexible statistical techniques can handle missing data better and might have slightly more power to detect group differences, but the ANOVA analysis was acceptable. Ms Biggs appropriately reported F- and p-values and related degrees of freedom.

  Click here to read about two common errors in data analysis.

• Did the researchers provide enough detail for you to be able to evaluate the strength or size of the effect as well as its direction?

  A good quality trial provides information about whether the effect could be considered clinically significant or meaningful, or offers benchmarks to help gauge the magnitude of the effect.

  Ms. Biggs did not provide information on the likely clinical significance of the group differences she reported. It would have been useful if she would have reported the average change in the scores for each group and some information about what that amount of change means. For example...

  For further information see, Statistical Significance Testing in Section III of the Randomized Clinical Trials Module.

• Are there concerns about reporting bias?

  A good quality trial reports clear aims and has a rationale for selecting the specific, limited number of outcome measures they chose.

  Does it seem as if the researcher examined a multitude of variables and only reported the ones with statistically significant results (also known as cherrypicking)? If many statistical tests were run on a wide variety of outcomes, it is likely that the researchers capitalized on chance associations. In that case, our confidence is reduced that the intervention was the reason for the change in the outcome.

  If you are unsure whether the hypotheses being tested in the publication you are evaluating really were the ones the study was originally designed to test, one way to identify the original intention is to find the study at the website clinicaltrials.gov. When researchers register their controlled trials at this site, they must identify the aims and primary outcomes of the study.

  Ms. Biggs’ study reported on just two outcomes which were directly relevant to answering her research question. Reporting bias is unlikely in this study.

• What is the external validity?

  As has already been discussed, high attrition is the main concern related to external validity that comes up during data analysis.

Common errors

For example...

What are Time Series Research Designs?

Basic Terms

• Baseline

  Baseline refers to a period of time in which the target behavior or outcome (dependent variable) is observed and recorded as it occurs before introducing a new intervention. The baseline behavior provides the frame of reference against which future behavior is compared. In some designs, the term baseline can also refer to a period of time following a treatment in which conditions match what was present in the original baseline.

• Treatment Condition

  Treatment condition or treatment phase in these designs describes the period of time during which the experimental manipulation is introduced and the target behavior continues to be observed and recorded.

Design Types

Design Types

• Interrupted Time Series Design

  The simplest type of time series design is the interrupted time series. This design is typically used to evaluate the impact of a population-wide policy or intervention. It involves a single treatment group which is measured many times before and after the start of the intervention. It is called "interrupted" time series because the researcher graphs the data before and after the intervention, and looks for an interruption in the line or curve where the intervention was introduced. The interruption could be a change in:

  • Level or height of the curve (as in Figure TS-1a, which shows that midway through 1993 the prevalence of smoking suddenly fell to a new, lower level)
  • Trend or slope of the curve (as in Figure TS-1b), which shows that the prevalence of HIV in a very high-risk population rose by about 1% every 2 years in the 8-year period between 1985 and 1993 before the intervention was implemented. After the initiation of the intervention, the rate of increase in HIV prevalence slowed substantially: the entire next 7-year period showed less than a 1% increase in HIV prevalence

  To be able to see the natural pattern of the behavior change, it is crucial to this type of design to have a sufficiently long period of baseline observation that includes usual seasonal or cyclical changes.

• Reversal/ABAB Design

  Another general family of time series designs are ABAB or reversal designs,. These are usually used with a single participant or a few individuals. The "A" refers to the baseline and "B" refers to the treatment. In this design the treatment is systematically provided and withdrawn. After a stable baseline is established (A), whereby the target behavior remains relatively constant, the treatment or intervention is provided (B). If the treatment is successful, we would expect to see a change in the target behavior in the predicted direction. After a period of treatment, the intervention is withdrawn. At that time the target behavior is expected to return to baseline levels (A). Treatment is once again initiated (B), and, if successful, should initiate a similar treatment response.

  This design provides an opportunity to repeatedly demonstrate, or replicate, the relationship between the treatment and outcomes of interest with a single participant. Figure SSRD-1 shows an example of this type of design. The panels show the effect of holding a favorite toy on the verbal utterances of an autistic boy. Without the toy, the boy does not speak. When he is given the toy, he speaks; when the toy is taken away he ceases talking, and so on.

• Multiple Baseline Designs

  Other time series studies can be characterized as multiple baseline designs. Reversal designs are not always feasible. Once treated, a behavior may not deteriorate back to baseline after treatment stops. Ethically, once a patient has responded to treatment, it may be considered too harmful to discontinue it. The multiple baseline design incorporates a baseline and an intervention condition (an A and a B) across multiple participants, behaviors, or contexts. The greater the number of replications, the more confident one can be that the treatment produced the observed changes.

Sample Selection – Excerpt

Sample Selection – Target Population

• Is the target population identified?

  A good quality time series study will clearly define the target population ahead of time. This increases the chances that participants were selected because they are typical of the target population, rather than being selected for idiosyncratic reasons such as openness to trying new treatment approaches.

  In the excerpted study, the target population was described as Hispanic women from a low-income neighborhood in Florida. The New York family service workers also work with Hispanic immigrants chiefly from Puerto Rico. However, they wonder whether the women in the Florida sample were also immigrants and parents. They also worry that the women from Florida might be of Cuban or Mexican origins, rather than Puerto Rican. These factors could influence whether the Florida intervention fits the needs of the New York population.

• How many participants are included?

  In time series designs, participants serve as their own control or comparison, so traditional power calculations are not applicable. In this study, confidence that any change in depression symptoms is due to the intervention increases when the effect replicates consistently across multiple participants. In population-based studies a community may be considered one participant.

  Some studies include only a single participant; our clinical example study evaluated three.

• What are the inclusion and exclusion criteria?

  Just as in experimental designs, inclusion and exclusion criteria help to describe how participants were selected into the study. A clear description of these criteria increases the likelihood that participants were selected because they are representative of the target population, rather than for idiosyncratic reasons which will not be replicable.

  The Florida example study clearly specified some inclusion and exclusion criteria regarding ethnicity, diagnosis, and geographic setting.

• How were individual participants chosen?

  Were participants chosen at random or through some other method (e.g., convenience)? Non-random methods have the potential to introduce bias, especially when study samples are very small.

  In our Florida study, it isn’t clear how the three women were specifically chosen, except that all were patients of the same social worker. If they were not randomly selected out of a larger number of eligible participants who also met inclusion and exclusion criteria, we should be concerned about the potential for bias.

  Population studies sometimes use summary statistics for an entire community to inform policy or public health initiatives for that population. However, some population-based studies do select a sample from the population to help assess the impact of an intervention. In these cases, random selection of a sample improves the likelihood that the sample is representative of the population. In addition, it is helpful if the researcher compares the characteristics of the study sample with those of the larger population to help determine how well the sample reflects the community.

Group (Factorial) Assignment in Single Subject Time Series Designs

For example...

Group (Factorial) Assignment in Single Subject Time Series Designs

Assessment and Data Collection – Excerpt

Assessment and Data Collection – Questions

• A good quality time series study uses an adequate number of data points in each phase (baseline and intervention) for each participant or community. The number of assessment points is adequate if the outcome measures have attained stability.

  In the sample study, the researchers provided a rationale for the years of data they collected, including an effort to establish a stable baseline with five years of data. In many cases, five observations might not be sufficient for data to stabilize. In this case, though, traffic fatalities are relatively stable, so five data points were sufficient.

• A good quality time series study provides sufficient detail about data collection procedures. Information about quality assurance methods is especially important when secondary data sources are used.

  Sometimes population-level studies are dependent on data sources that were created for other purposes, such as tracking traffic accidents. In such cases, researchers have little control over the collection of the data. Therefore the data are usually less trustworthy than data collected specifically for research purposes.

  An additional concern with using pre-existing data sources is that sometimes reporting systems change. When this happens it can appear as if the outcome is changing when it actually is not. For example...

  In the sample study, the authors provide little information about the quality of the data. It would be preferable if they described who was responsible for populating the data, whether clear instructions are provided, and what quality assurance checks are performed on the database.

For Example...

Data Analysis/Results – Excerpt

Data Analysis/Results – Questions

• Predictions

  A good quality time series design specifies hypotheses about the expected intervention response in targeted outcomes. For example, the researchers may predict:

  • Direction of the change;
  • How soon response to the intervention will be seen; or
  • Which targeted behavior will show the greatest responsiveness.

  The more the results reflect a priori (pre-stated) predictions, the more confident we are that the results reflect the impact of the intervention and not some other factor.

  In this study the authors hypothesized that the ban would result in a reduction in the heart attack rate, which was confirmed by the results.

• Visual Analysis

  A good quality time series design:

  • Reports appropriate visual analysis that addresses level, trend and variability
  • Clearly labels x- and y-axes, indicates baseline and treatment phases with vertical lines, and uses consistent scales that accurately reflect the variability in the data

  Level refers to the frequency or intensity of the behavior in a particular phase of the study. In some studies the mean of each outcome is summarized for each baseline and treatment study phase.

  Trend refers to a pattern of change observed across data points. For example, if the outcome targeted by the intervention trends toward improvement during the baseline phase, before the intervention is introduced, this suggests that the outcome might have improved even in the absence of treatment.

  Variability refers to the consistency of the behavior or outcome. If it is highly variable during the baseline or treatment phases, stability may not have been attained, making it hard to determine the treatment response.

  In our sample study...

• Statistical Analysis

  The application of statistical analysis in time series studies is relatively new. Many existing studies do not use statistics to analyze results. However, statistical procedures may be particularly helpful in certain contexts. Examples of statistics used with time series studies include ARIMA models, celeration line approach, two-standard deviation band method, C-statistic, or other.

  Visual analyses are subjective as there is no standardized procedure for this approach to data analysis. Instances in which visual outcomes can be especially difficult to evaluate include studies where a stable baseline is not established, tests of new interventions where treatment effects are relatively unknown, or cases where treatment effects may be delayed or subtle.

  The use of both visual and statistical analyses enhances the quality of the smoking ban study. The advocacy group has some evidence that a smoking ban may have an impact on heart attacks.

In our sample study...

What are Systematic Reviews?

Defining and Searching for Relevant Literature – Excerpt

Sample Selection – Questions

• Is the relevant body of literature clearly defined?

  A good quality systematic review clearly describes through detailed inclusion and exclusion rules what kinds of studies will be used to address the question. These rules specify study characteristics, such as population characteristics, interventions, comparison groups, outcomes, study design, publication date, and the required duration of follow-up. If those rules are not clearly defined (usually by a protocol) then two immediate problems can result:

  • Searching may not adequately locate all studies that are in the relevant body of literature
  • Articles whose characteristics fall into a gray area may be handled inconsistently or in a biased manner when it comes to deciding whether a study should be included in the review. For example...

  Mr. Cohen’s review protocol specified required intervention elements and the research design types they would consider. They did not mention limits regarding language (can they translate non-English language publications?) or location of the study (are studies conducted outside of the U.S. within their scope of work?), or what types of comparison conditions are acceptable. It would have been preferable to have these factors specifically described.

• Are search methods adequate?

  A good quality systematic review:

  • Searches all relevant databases. Usually, more than one database is needed to capture all relevant studies. In addition, there should be evidence that the investigators utilized other methods such as hand-searching key journals and bibliographies of relevant articles and contacting researchers in the field to identify studies.
  • Provides information about the specific search terms used. Searching electronic databases can be a high-level task requiring specialized knowledge of databases and how indexed terms are used in each one. Input from a librarian can be extremely helpful, in many cases essential.

  Mr. Cohen’s systematic review searched a number of relevant databases, used a specially-trained librarian to develop the search strategy (thus increasing the odds that the appropriate terms were selected and used correctly), and reported additional, non-electronic search strategies. All of these instill confidence that the searching process was systematic and comprehensive.

• How is the external validity affected?

  As described under "Search Methods," the roster of databases searched should be as comprehensive as possible. Narrow searches can bias the results by excluding studies conducted by researchers from certain disciplines or geographic regions. Study inclusion and exclusion criteria also have a direct impact and should be carefully reviewed. If inclusion and exclusion rules are not clearly specified, then readers should carefully examine the characteristics of the individual trials for information about how the study samples reflect the population of interest to them.

For example...

Selecting Studies and Collecting Data – Excerpt

Selecting Studies and Collecting Data – Questions

• Are the procedures to evaluate abstracts and articles for inclusion in the review clear and free of apparent bias?

  In a good quality systematic review, abstracts and articles are reviewed by at least two people to ensure that no important studies are excluded and that inclusion/exclusion criteria are applied accurately. Optimally, articles describe whether reviewers were working independently (i.e., blind to the other reviewers’ results) and how disagreements were resolved when they arose.

  Some reviews use a blind assessment procedure whereby author and journal information are stripped from the copy of the article being assessed. This further reduces the risk of bias, but is not necessary in most cases, particularly if inclusion and exclusion criteria are well detailed.

  The Activation and Commitment Therapy systematic review provided adequate detail regarding the process of deciding which studies to include, including the numbers of abstracts and articles reviewed.

• Were studies assessed for quality?

  A good quality systematic review:

  • Critically appraises the quality of the included studies and reports the results
  • Describes how information on study quality has been used in the review. Some researchers exclude poor quality studies and others conduct analyses to see what (if any) effect study quality has on the treatment effect. Either method can be appropriate.
  • Describes the specific threats to validity they evaluated, or provides references for instruments they used to assess study quality. A tremendous amount of information is available about assessing study quality. Some quality rating instruments have been developed and may be used. Other reviewers prefer to develop quality rating instruments specific to their review question.

  The authors of the Activation and Commitment Therapy systematic review failed to provide information about what aspects of study quality they examined. They should have either provided an appendix listing the threats to validity that they assessed, or a reference if they used methods described elsewhere.

• Are the procedures used to assess the quality of the articles clear and free of apparent bias?

  Ideally, reviewers will be blind to other reviewers’ assessment. This is particularly important for the assessment of quality, which involves some subjective judgment. For that reason, quality assessment is prone to bias or to being influenced by knowledge of other reviewers’ conclusions. Ideally the researchers will also report procedures for resolving disagreements in quality ratings.

  The description of the quality assessment process in this review is weak. The reviewer does not say whether quality was assessed by more than one person for each study, much less whether the quality assessment was conducted with or without the knowledge of others’ ratings.

• Is data abstraction checked/verified?

  A good quality systematic review clearly describes data quality assurance measures, such as having all or some data elements checked by a person other than the person who abstracted the data. This is especially important for primary outcomes and any data used in meta-analysis. It is very easy to make transcription errors, especially when there is a large volume of data.

  The authors appropriately described their data abstraction process as involving a standard form, and it is reassuring that data abstraction was checked by a rater other than the abstractor.

Data Analysis/Results

Data Analysis/Results – Excerpt

• From the Methods section:

  The primary outcomes of interest were the proportion of participants who fell during the followup period and the proportion of participants who had a fall-related fracture. Because of the high degree of heterogeneity in patient populations, intervention approach, setting, and method of assessing the outcomes, we used random effects models for all meta-analyses rather than fixed effects. We analyzed the relative risks (RR) of falls and fractures. If the study did not provide the RR or the related standard error (SE), we calculated the missing statistics from raw data (i.e., numbers of events in each group and numbers randomized to each group). If they provided both unadjusted and adjusted RRs, the RR from the adjusted model was used. We calculated the I² statistic to assess statistical heterogeneity of the trials in each analysis.

• From the Results section:

  Six trials assessed fall-related fractures and found that physical activity interventions did not reduce the probability of a fracture. The pooled relative risk was 0.83 (95% CI 0.61, 1.14) with low heterogeneity (I² = 16.3% percent). However, the thirteen trials reporting the proportion of participants with a fall found that physical activity interventions did reduce the probability of falling (RR=0.76, 95% CI 0.63, 0.93). Heterogeneity was low for this analysis as well (I2=5.2%).

Data Analysis/Results – Questions

• If studies were statistically combined in a meta-analysis, were they sufficiently alike to warrant doing so?

  Three types of heterogeneity should be evaluated: clinical (variability in factors such populations, interventions, outcomes, settings), design (variability in the design features such as presence of random assignment and time to followup), and statistical. The first two are evaluated subjectively by the reviewer, and the third is evaluated using statistical tests.

  A good quality review:

  • Only combines studies statistically that are reasonably similar (What does "reasonably similar" mean?)
  • Uses statistical models that account for the degree of heterogeneity that is present (More on heterogeneity)
  • Provides a measure of statistical heterogeneity, such as I² (In this study...)
  • Provides enough information to the reader to evaluate how similar the studies are

  If the studies in a review are very different from each other, then an average may be meaningless. For example...

• How many studies were included in the analysis and what was the quality of the studies?

  Higher quality studies yield more trustworthy results. A large number of studies is also usually preferable to a smaller number, though there are some exceptions. Two to three very good quality, large trials would usually provide more trustworthy results than 10 small, fair-quality trials.

• Do they describe which outcome(s) they chose?

  A good quality systematic review:

  • Describes which specific measures were accepted as outcomes and which statistics were entered into the meta-analysis (e.g., relative risks, odds ratios, mean differences)
  • Reports formulae, if they calculated outcome measures that were not reported in the individual studies
  • Reports which outcomes were chosen if a study reported more than one relevant outcome, such as self-report vs. medical chart data, any fracture vs. only non-vertebral fractures, and falls in 6-month vs. 12-month window

• Do they describe how they handled studies with three or more treatment arms?

  Designs with multiple intervention and/or control groups are very common. Reviewers should not simply include multiple comparisons in the same meta-analysis as if they were from different studies. Two commonly used strategies are:

  • Select the intervention arm with the highest intensity
  • Take an average of the data across treatment arms

• Do they provide a visual display of the data?

  If they have multiple studies with similar outcomes, systematic reviews commonly include forest plots (see Figure SR-1 for an example), which show the average effect size for each study along with “whiskers” that show the 95% confidence interval. The size of the symbol showing the effect size is usually proportional to the size of the study. If the studies are sufficiently similar to warrant combining them statistically, then an average effect size is usually presented as well, either for the entire group of studies or for logical subgroups of studies. The plots help the reader understand how consistent the results are and whether one or two studies are exerting a strong influence on the summary statistics.

  The authors of the falls prevention review did not provide forest plot, despite having enough data to run a meta-analysis.

• Are there concerns about reporting bias?

  A good quality systematic review describes a priori a limited number of outcomes they will report.

  Sometimes an outcome is reported in only a few of the studies included in a systematic review. When that occurs, the possibility is raised that this outcome is reported selectively: i.e., it is more likely to be reported when the results are statistically significant. This review limited itself to two primary outcomes: falls and fall-related fractures, which reduced the risk of reporting bias.

• Are there concerns about publication bias?

  Publication bias occurs when studies with statistically significant results are more likely to be published than those without statistically significant results. A good quality systematic review reports what they did to examine the potential for publication bias and whether there is any reason for concern. Certain ways of plotting the data (such as a funnel plot or L’abbe plot) are useful for assessing the likelihood that positive studies in an area were more likely to have been published. A funnel plot should have been provided or described for the falls prevention review. The plot could have helped the reader gauge the likelihood that unidentified, unpublished negative trials exist but were not captured by the review.

• Are the reviewer’s conclusions warranted by the data?

  Sometimes a reviewer over-generalizes the findings of the review or uses statistical methods that exaggerate the size of the effect. In this example, the reviews report only relative risk statistics. So, we know that the risk of falling is 17% lower for people engaged in falls prevention interventions, but we need to know the absolute difference in risk in order to put the results in perspective. Reporting on relative risks can be misleading.

• What is the impact on external validity?

  The methods used for analyzing data have a limited impact on external validity. Readers may want to look for analyses of subgroups of the trials whose samples best match their population of interest. For example, a systematic review may include studies conducted in both residential and outpatient settings. If you work in a residential setting, then a meta-analysis limited to the subset of trials performed in residential settings may provide the more relevant information for your context.

For example...

Reasonably similar

Heterogeneity

In this study...

Critical Appraisal of Guidelines and Recommendations

Critical Appraisal of Guidelines and Recommendations

Grading of Recommendations Assessment, Development, and Evaluation (GRADE)

Conclusions

Conclusions

Resources

Glossary