Advanced Assessment Methods: Regression

Little is more important to a PA program than the PANCE pass rates of its graduates. A program that can boast high first-time PANCE pass rates will draw more students. Therefore, it is most desirable to discover variables that can predict passing (or failing) PANCE scores as far ahead of time as possible. With this benefit, students who need assistance can get it, and you may bolster portions of your program to improve overall PANCE scores. 

Correlation and regression are ways to measure variables such as these, as predictors of PANCE scores:

• Admissions criteria
• Course outcomes
• Program instructional objectives, learning outcomes, and breadth and depth of curriculum. 
• Student summative evaluation results
• Remediation practices and results
• Student progress criteria and attrition data

The most common way I see regression performed is through Parametric Analysis to Enhance Assessment Regression.

In this case, we look at each of these specific elements as a predictor of PANCE score. We may at each academic course, each individual and aggregate EORE, and then the PACKRAT exam as a predictor. At the end, after examining all the variables, we have determined which are the most significant variables when it comes to prediction.

Let us look at a chart of information which I discussed in my last blog, the Regression of PACKRAT I as a Predictor of PANCE Scores.

How is regression derived? Here is a breakdown of the components:

• R is the correlation between the regression-predicted value of PANCE scores and the actual PANCE scores. The R is the same as the Pearson Coefficient. 
• R2 represents the proportion of variance in the outcome variable explained by the predictor variable – you can say that a specific course can explain (in this case) 49 percent of the PANCE score – in plain language, it has a 49% chance of predicting the score exactly. It is calculated as the square of the R value.

• Analysis of variance (ANOVA) assesses whether the regression model is statistically significant. When you have multiple regressions taking place, and multiple variables being analyzed, those that are not statistically significant (based on the p value you have set) are eliminated from the model. Those remaining are significant. In most cases, a p value of .05 is the cutoff. 
• The Regression model is statistically significant if the p value associated with ANOVA F statistic is less than the chosen significance level.
• A variable is a statistically significant predictor of the outcome variable if the p value associated with its coefficient is less than the significance level.
• The coefficient value gives you a quantification like this: One point higher in Course X means a proportionally higher score on the PANCE. The coefficient value here is 2.66, which signifies a student who achieves one-unit higher PACKRAT score than another is expected to achieve 2.66 points higher on the PANCE score. The regression equation to predict PANCE score from PACKRAT is as follows:
• PANCE score = (PACKRAT x 2.66) + (66.29) 

Now, I am not suggesting you sit down with a calculator and work these numbers out yourself, unless you too have a strong background in statistics. When I was in my PhD program, I took several semesters of advanced quantitative statistics. We memorized the formulas for regression, and many others, and were able to run the raw data. The equations gave us exactly how to predict a score. When you as a statistician see that, you can say, “Ah, that makes sense. If you plug this into this, that means that is what the PANCE score will be.”  

What I predict for most of you, my readers, is that you will consult with a statistician who can create these regressions for you. I want you not to be able to conduct a regression yourself, but to understand what a regression model means when your statistician hands it over to you. If you have kept your data as required by the ARC-PA 5th Edition Standards, you will have plenty of data available for your statistician to run these regression formulas and produce numbers that you can interpret and understand, and with which you can make some valuable modifications to your program.

We have undertaken the task of interpreting the meaning of advanced assessment methods. Next time, in the final blog for this section, I will demonstrate some instances in which regression modeling provided quite valuable information to PA programs in predicting successful PANCE outcomes, permitting programs to make useful changes to benchmarks and remediation.