Quasi-experimental design in the evaluation of outcomes of health services in New Zealand

Abstract

Background: The New Zealand health system constantly needs to adapt its services to address changes in population needs, technology, or models of care. It is important to be able to determine if the changes made are achieving the outcomes desired. Frequently a randomised controlled trial is not acceptable, feasible, or the best choice for undertaking outcome evaluations. A quasi-experimental evaluation may be an alternative, but there are important concerns about their internal validity. Outcome evaluations that are not accurate may lead to poor decisions which will adversely impact our health services. Aims: This thesis aims to determine the role Quasi-Experimental approaches should take in improving information for making decisions in New Zealand health services. Its goal is to answer four questions: 1. What is the place of quasi-experimental outcome evaluation in the New Zealand health service? 2. What is current New Zealand health service quasi-experimental outcome evaluation practice? 3. Do quasi-experimental methods lead to results with strong internal validity, and in what circumstances? 4. How should we undertake quasi-experimental outcome evaluations, and how should we assess our evaluations internal validity? The thesis has a focus on internal validity, whilst acknowledging that other validities, in particular construct validity and statistical conclusion validity, also play a vital role in determining the usefulness of quasi-experimental approaches. Methods: Three substantial pieces of work are described in the thesis. Each of these provides information to answer one or more of the above questions. Firstly, current New Zealand practice of quasi-experimental outcome evaluation is examined through a review of 50 evaluations. The review determines the types of study designs used, and then critiques each study for risk of bias against a 15-item tool developed by the author. Secondly, it reviews the literature on quasi-experiments; in particularly undertaking a systematic review of the published within-study-comparison studies. These studies provide an empirical way of examining quasi-experiments’ internal validity by comparing their results with Randomised Controlled Trials (RCTs) done under similar conditions. I examine the degree to which different quasi-experimental study designs can achieve unbiased intervention effect estimates, and what other design features are associated with reducing bias. Finally, I report on six quasi-experimental outcome evaluations, that I have led in Waitemata and Auckland District Health Boards (DHBs). These are analysed as case-studies using a structure similar to the tool discussed above. In particular, the case studies examine the utility of these evaluations, and the difficulties of undertaking quasi-experiments in the New Zealand health service setting. Results: Currently, most New Zealand quasi-experimental outcome evaluations use weak designs, particularly uncontrolled before-and-after designs. They also frequently use methodology that cannot effectively address bias risks. There is a considerable risk that these evaluations provide incorrect information to decision makers. Strong study designs; including controlled before-and-after designs, interrupted time series, and regression discontinuity designs, will usually produce intervention effect estimates that are similar to an RCT. Controlled-after only studies are at greater risk of bias. However, not uncommonly, even these strong designs will deliver biased estimates. There is considerable evidence on how to best undertake these designs to reduce risk of bias. However, it is not possible to say, from the evidence presented, that following this advice will guarantee unbiased results. The case studies found that quasi-experimental outcome evaluations are acceptable, feasible and valued by the DHBs. It was possible to use strong designs and follow recommended methodology, with reasonable expectation of strong internal validity in some cases. However, a wide range of difficulties were also experienced. Particularly important limitations are the lack of opportunity for the evaluator to influence the project being studied to strengthen the proposed evaluation, and limitations in the type of outcome data available for evaluation. Conclusions: Quasi-experimental approaches should play an important role in evaluating the outcomes of changes in New Zealand. Whilst they cannot reach the degree of certainty of internal validity of an RCT, the use of strong designs and good methodology provides useful estimates of intervention effects. There is evidence that current practice usually does not meet these requirements. Chapter 13 of this thesis provides advice on when and how to undertake a quasi-experimental outcome evaluation, or on how to assess an evaluation done by others. All designs must attribute variation in outcomes between the intervention and comparison group, or variation in outcomes over time, or, preferably, both to the intervention. Better evaluations can be achieved with more outcome data over groups and time, good design, and, by using good methodology. Achieving internal validity through the use of quasi-experimental studies is complex. Therefore every evaluation must systematically consider the elements of the evaluation (participants, intervention, comparison, outcomes, and time), the data and design to be used, and what threats to interval validity are likely. Limitations and policy implications are discussed.