The intent of the current endeavor was to perform a formative evaluation [23,24] of the CCTs effectiveness. We recognized there would be analytic challenges and limits to such early-stage analyses. Nonetheless, we believed it was vital, especially given the cost of the CCTs and the growing financial pressure on health care networks more globally, to determine the preliminary effectiveness of the CCTs’, care management interventions and, if possible, suggest improvements to the intervention. As intended in formative evaluations, the evaluation is ongoing. Future analyses will bring more rigor to the evaluation and solve the data and analytic obstacles that affected the results of this first round of analyses.
A major learning of the early-stage analyses was the difficulty in developing comparison groups that are equivalent to the intervention groups at baseline. A number of matching schemes were attempted at the patient level in addition to the one presented here, but they were equally problematic. Avenues for creating more valid comparison groups in the future include the use of propensity score matching as well as drawing comparison groups from data from other networks. In addition, as time passes and more than 1 follow-up point is available for analysis, multilevel modeling can be employed which can specify different intercepts (baseline values) for groups. Still, it’s worth noting that constructing appropriate comparison groups is challenging even with those approaches: most health networks do not collect data on the most relevant matching variables (eg, health literacy, social economic status, social isolation/support) due to their cost and burden to both practices and patients.
Another major gap revealed by the early-stage analyses was the need to improve the strategy used for selecting patients for CCT intervention. In addition to many physician referrals, there was a large number of patients on the high-risk registry who required intervention relative to the small CCT staff. Various strategies to prioritize the list were attempted, including cost-related analyses. As plagued the formation of comparisons groups, it seemed the variables most critical to risk stratification were unavailable in administrative datasets. Appreciating that data collection is costly and patients and busy practices already have survey fatigue, the evaluation team examined the empirical literature for a single useful tool for ranking patients as well as constructing better matched comparison groups. This search indicated that a measure of patient activation [25–27] would be particularly helpful not only for selecting the riskiest and costliest patients for CCT intervention but also for tailoring CCT services to different types of patients. Since the implementation of the CCTs the network has also contracted with a predictive analytics company to provide risk scores for the patients.
The current formative evaluation was an important learning journey, laying important ground work for better evaluating the CCTs’ effectiveness specifically and eventually becoming an accountable care provider more generally. If the network is to provide health care more effectively and efficiently, it will need to bring greater rigor to evaluations of its various interventions and other ACO endeavors. The current formative evaluation was a valuable demonstration to non-scientists of the weakness of single group pre-post designs and how more rigorous evaluations, which include comparison groups and address confounding variables, can enhance the validity of the analytic results. This learning journey also highlighted the limitations of administrative databases and the necessity of both primary data collection and mixed methods. For example, it seems that some practices may require educational interventions to take full advantage of the CCT and qualitative assessments on practice readiness seem a necessary addition to the quantitative practice assessment to identify the specific characteristics that need strengthening. In addition, the evaluation team also recently added a qualitative sub-study on high-risk patients’ experience with the CCT to overcome locally low CAHPS response rates and capture themes broader than patient satisfaction. Upcoming rounds of analyses will also tackle other aspects of formative evaluations including the study of the CCT implementation as more practices receive CCTs and determining if process and fidelity measures of the PCMH pillars are linked to better outcomes. Furthermore, future analytic plans include identifying the active ingredients and optimal doses of the CCT intervention as well as determine the most effective matches between different types of patients and different CCT interventions (eg, behavioral, care management, social, pharmacy). While we appreciate that barriers still remain and require solutions, we hope the current evaluation highlights the utility of performing such formative evaluations.