The Rationale for Program Evaluation

Program evaluation can be a simple way for clinic leaders to measure their clinic’s Primary Care Behavioral Health (PCBH) successes and identify areas for improvement. Periodic program evaluation can assist with maintaining clinic and command support as well as help identify areas for improvement. Attending to necessary changes better meets the needs of various stakeholders, including patients, primary care managers (PCMs), PCBH staff and program leaders.

Monitoring key variables related to PCBH implementation can be accomplished at the clinic level. Such local program evaluation activities can assist clinic leaders in identifying strengths of PCBH implementation in their clinic, as well as areas in which performance or outcomes fall below expected or desired levels. Ultimately, program evaluation results can shape process improvement plans to enhance the delivery of care.

Selecting key topics/useful metrics to monitor or evaluate: 

The Military Health System (MHS) is guided by the Quadruple Aim, with emphasis on improving population health, enhancing the experience of care, responsibly managing health care costs, and maintaining mission readiness. These four areas represent key outcomes for monitoring and evaluation.

A number of PCBH program metrics consistent with the Quadruple Aim can be monitored or evaluated by primary care clinic leaders. Some data may be available at local clinics, while other data can be pulled at the service or MHS level. Examples of PCBH metrics consistent with MHS Quadruple Aim include:

• Patient satisfaction with integrated care
• PCM satisfaction with BHC and BHCF services
• Patient access to integrated behavioral health care
• Volume of patients receiving PCBH services. Examples include:
  • Utilization rates of BHC
    • Number of unique patients seen
    • Number of patient encounters
    • Number of same-day patient encounters
    • Average number of visits per patient
    • Utilization rates of BHC for particular conditions or issues
    • Average appointments per workday
  • Utilization rates of BHCF
    • Number of unique patients seen
    • Number of patient encounters
    • Average number of appointments per patient
    • Average appointments per workday
• Breadth of BHC services provided (degree of severity levels and range of problems)
• Quality metrics related to integrated behavioral health services
• Patient outcomes (improvements in symptoms, functioning, and/or military readiness)
  • Percentage of patients satisfied with quality of care, convenience of care
  • Percentage of patients who believe that PCBH helped them deal with their symptom
  • Percentage of patients of patients who felt involved in treatment decision-making
  • Percentage of patients of patients who came back for a second BHC appointment (when one is recommended)
  • Percentage of patients of patients who would recommend seeing a BHC to others
  • Percentage of patients with clinically meaningful improvement. For example:
    • Changes in scores on the BHM-20, PCL, PHQ-9, GAD-7 between visits or between first and last visits
    • Successful tobacco cessation
    • BMI Decrease

Determining the Metrics Most Useful for Your Clinic

1. Define the problem: Identify what needs to be changed in your clinic. Analyze the current state. For example, who is involved? What are the current practices? What structures, policies, practices are currently limiting performance?
2. Define the desired end-state: Envision what change you would like to occur in your clinic after the problem has been resolved. Be concrete and specific. Make your goal a SMART goal (specific, measurable, achievable, realistic, and timely).
3. Define the innovation: Map out the resources required and the steps that need to be taken to move you from your current state to your desired end-state.
4. Define the critical steps to be monitored:Identify critical steps and outcomes of those you’ve mapped out in step #3. These are the things that need to be monitored.
5. Define the measurement strategy: For each critical step and intermediate outcome, determine if information collection mechanisms already exist. If not, determine how the information will be collected (e.g., questionnaires/surveys, counts, interviews, focus groups). Determine who will collect the information and how often the information will be collected and analyzed. For the desired end-state, revisit your SMART goal and determine how you will assess whether or not you’ve met your goal. Define success in terms of degree of change in the measure you will use.
6. Measure the current state before you initiate the change.

Using Results to Improve Service Delivery and Patient Care

It is not enough to simply monitor or evaluate aspects of the PCBH program. Information obtained from program evaluation should be used to shape and inform improvements in the program, leading to improved population health, experience of care, mission readiness, and/or cost management. Here are some examples:

• Identifying problems with access to PCBH services could lead to action plans to increase same-day access to the clinic’s IBHC or to facilitate smoother booking processes for patients
• Identifying patterns in low utilization of PCBH services could lead to implementation of clinical pathways to increase routine inclusion of IBHCs in the care of patients with common medical and behavioral health conditions. Find more information on the PCBH Clinical Pathways
• Determining areas in which patient outcomes are lower than expected could lead to further investigation of contributing factors and plans to further enhance delivery of evidence-based care