Conquering the ARC-PA Pass Rate Report Volume 4: Instructional objectives, summative evaluation results, and remediation

Thank you for joining me again as we continue our discussion of meeting the ARC-PA’s requirement of 85% or better first-time PANCE pass rates, and how to complete the necessary report if this benchmark is not met. In our previous issue, we looked at two of the ten ARC-PA points of analysis: individual course performance, and course/instructor evaluations. Today we will cover the next three success predictors as defined by ARC-PA and how to use their analysis to improve your PA program’s PANCE first-time pass rates. 

Program instructional objectives, learning outcomes, and breadth and depth of the curriculum

Your program should measure the adequacy of instructional objectives, learning outcomes and depth and breadth of the curriculum through several different methods. For example:

• Conducting a descriptive and parametric analysis essentially determines the relationship between performance in didactic courses and future success on the PANCE.
• Considering whether the depth and breadth of content is appropriate for PA education
• Considering whether the taxonomy of testing within specific core courses is accurately measuring student achievement
• Asking if the instructional objectives within the curriculum provide a blueprint for the students to effectively learn.

Student Summative Evaluation Results

PA programs can evaluate the summative evaluation as a tool for improving future PANCE performance. We recommend that you determine the correlation strength between the students’ raw scores on the knowledge component of the summative exam, and the students’ PANCE scores. Stratify the results of the PANCE scores, then compare with the summative scores. For example, look for a relationship between students who either failed the PANCE or achieved very low scores and their performance on the summative exam.

Based upon the results of the regression, see if it is possible to prospectively determine students that are, or can become, at-risk. Then, determine whether coaching can be provided through an academic improvement plan to work with such students during the last three to four months of the program.

Here is an example of one such regression we conducted:

EOC Score as a Predictor of PANCE Score

For the program in this analysis, the regression model with just EOC as the predictor variable was statistically significant (p<0.001) and explains about 68% of the variance in PANCE scores. The descriptive analysis demonstrated significant differences in overall performance within the summative exam /EOC among students who scored below 350 on the PANCE versus those that passed. The class average of all students on the EOC was 1479, while the average of the students who failed the exam was 1428. 

This program determined that the minimum passing score for the EOC for the class of 2022 was set at 1475. Students who score below this threshold would henceforth be provided with a supervised study plan before taking the PANCE. The program also embarked upon a process of determining if predicted scores could be generated to identify students at risk in the future. 

The principle behind this process was to triangulate all the data sets together to generate the highest level of statistical correlation. A series of assessment tools including PAEA PACKRAT I, PACKRAT II, EORE, and EOCE) can determine if students require additional success coaching and an academic improvement plan. 

Remediation Practices and Results

This is a tricky one, because while remediation is essential and useful, there does come a point of diminishing returns. There is a statistically significant correlation between a high number of remediations and PANCE failure. But where does the problem lie? Logic tells us that a student who cannot successfully remediate their performance after a certain number of interventions must either be unsuited for a PA education, or perhaps, be the victim of a poor remediation process. 

When responding to ARC-PA, begin by tabulating and analyzing the results of the remediations during the academic and clinical program. For students who failed PANCE who underwent a formal remediation process, provide a brief capsule of subsequent performance. This involves using descriptive analysis and parametric analysis between PANCE and both academic course performance and clinical outcomes, and use regression to determine if PANCE scores are significantly different between students who remediated vs non-remediators.

Example analysis:

For this program, we ran the regression model with just remediations as the predictor variable, which showed statistical significance (p<0.01), explaining about 45% of the variance in PANCE scores. A candidate with a one-unit higher Remediations score is expected to get -18 points lower PANCE score. 

Therefore, the program implemented a system of predictive modeling for the class of 2022 to determine which students were most at risk for failing the PANCE. Those at critical risk and predicted to fail were started on a 10-week study plan monitored by advisors. This system was successful at identifying students with low predicted PANCE based upon PACKRAT I/II /EOR/EOC. 

The program also incorporated a more formalized student coaching process for students who were identified with academic difficulty. This was part of a wider student success process that was implemented for the class of 2022-2023. 

The next issue…

In the next edition of our newsletter, we will cover the final variables that the ARC-PA wishes to see analyzed pertaining to low first-time pass rates for the PANCE, including attrition data, feedback gathered following the PANCE, and historical PANCE performance.