Conquering the ARC-PA Pass Rate Report Volume 5: Attrition Criteria and Feedback

scott's thoughts Jan 03, 2023

In today’s issue, we will tackle the remaining subjects for analysis when completing the ARC-PA’s PANCE Required Report for Low First-Time Taker Pass Rates. To summarize the subjects of our last few issues, here again is the list of the details requested by ARC-PA on their form, as these details relate to your PA program’s PANCE outcomes. I will also note the volume in which these subjects were covered in our current series.

Admissions criteria as predictors of success (Volume 2)
Individual course performance (Volume 3)
Course and instructor evaluations (Volume 3)
Program instructional objectives, learning outcomes, and breadth and depth of the curriculum (Volume 4)
Student summative evaluation results (Volume 4)
Remediation practices and results (Volume 4)

Today we’ll cover the final four points:

Attrition criteria and data for cohort being reviewed

Feedback from students who were unsuccessful on PANCE, if available

Preceptor and graduate feedback

Employer feedback (not required, but may be helpful if available to the program)

Attrition Criteria and Data for Cohort Being Reviewed

In this case, your PA program explores the relationship between students who had experienced attrition and deceleration, and their subsequent performance on the PANCE. Your program should thoroughly examine policies and procedures regarding attrition and their impact on specific cohorts that experienced attrition

Our example in this case has a small sample size. They noted that of the four students who experienced deceleration and or leave of absence, three failed the PANCE on the first attempt. The three who failed PANCE had several identifiable risk factors; the one who passed had notably fewer and a higher GPA as well.

The program has incorporated several action plans that directly relate to lowering the level of attrition in the program, including:

Incorporate the first summer GPA as an indicator of inclusion into the academic improvement plan process and student coaching. The threshold will be set at a first Summer GPA below 3.10.
Incorporate a pre-matriculation program for students coming into the academic program in the Summer 2023 semester, designed to provide science foundation knowledge and student skill development.
Automatic placement of decelerated students in a coaching program and academic improvement plan prior to reentering the program.

Feedback from students who were unsuccessful on PANCE

This data, if available, can be analyzed for common themes among students who did not pass. Can these themes provide insights about preemptive intervention in the future? Your program may identify deficits in your preparation process throughout the academic and clinical year, then implement modifications and action plans for subsequent cohorts.

Based on student feedback, we saw one PA program employ the following modifications:

Reset the minimum pass rate for the EOC to 1475 based upon the comparison of students who passed vs fail.
Incorporate the use of the NCCPA practice exams as a component of the student success process.
Increase coaching to better prepare students for the rigor and cognitive fatigue of the PANCE exam.

Preceptor and graduate feedback

This variable requires that we compare students who passed the PANCE versus those that failed by correlating the preceptor ratings and qualitative comments, performance in specific clerkships to determine if program instruction was insufficient, and determining if knowledge deficits convert into clinical performance deficits.

In the analysis we conducted, there were no correlation data that was actionable based upon the preceptor evaluation. However, once more, this does not mean we cannot take lessons from the information gathered. There was a measurable decrease in the overall preceptor ratings when comparing students who failed versus those who passed. This bears watching, as there may be some insights gained from lower preceptor evaluations throughout the clinical year.

Historical Performance on the PANCE

This analysis aims to examine trends regarding organ system performance and task performance to identify the instructional deficits. Subpar performance is correlated with other elements of the pass rate report, thus establishing actionable benchmarks for the future.

Our example program for this analysis established a new benchmark (the national average) related to performance on the PANCE first time pass rate data, starting with the class of 2022. The following are actions that were established based upon this benchmark.

If the program scores below the national average, this will trigger a further investigation in PANCE topics.
If the program finds a three-year period a negative trend, the curriculum committee will review each PANCE topic.
The curriculum committee may or may not recommend an action plan.
If the average scores are -5 or below, this will require action.

Next time…