Advanced Assessment Methods: How to Apply Advanced Assessment

scott's thoughts Nov 23, 2022

In my last blog, I discussed the use of stepwise regression, in which possible predictor variables are entered into a regression model until the best statistically significant variables are determined. I want to resume this discussion today by looking at some real-life examples of predictor variables and how they were determined.

Example 1

For a 2020 cohort, EORE and the number of remediations were the statistically significant (p < 0.01) predictors remaining in the model. A combination of these variables was the best predictor of PANCE scores. With enough data, you can begin to identify a Multiple Variable Risk Model. The multiple correlation coefficient (correlation between PANCE scores predicted by the model and actual PANCE scores) for this final model was high (R =0.76). The variables (EORE and number of remediations) explained 58.3% of the variance in PANCE scores 

Example 2

For a 2020 cohort, the EORE, History and Physical I, Clinical Medicine I, Clinical Problem...

Continue Reading...

Advanced Assessment Methods: Regression

scott's thoughts Nov 16, 2022

Little is more important to a PA program than the PANCE pass rates of its graduates. A program that can boast high first-time PANCE pass rates will draw more students. Therefore, it is most desirable to discover variables that can predict passing (or failing) PANCE scores as far ahead of time as possible. With this benefit, students who need assistance can get it, and you may bolster portions of your program to improve overall PANCE scores. 

Correlation and regression are ways to measure variables such as these, as predictors of PANCE scores:

  • Admissions criteria
  • Course outcomes
  • Program instructional objectives, learning outcomes, and breadth and depth of curriculum. 
  • Student summative evaluation results
  • Remediation practices and results
  • Student progress criteria and attrition data

The most common way I see regression performed is through Parametric Analysis to Enhance Assessment Regression.

In this case, we look at each of these specific elements as a predictor of PANCE...

Continue Reading...

Advanced Assessment Methods: Performance Variables and PANCE Scores

scott's thoughts Nov 09, 2022

In the past few blogs, we have examined various statistical concepts, so that we can derive interpretations from them. Previously, I discussed statistical significance, used as a pre-designated point at which we can say that the appearance of a correlation is probably not due to random chance. 

The point of looking at correlation coefficients and statistical significance, for the PA program, is eventually to determine what courses, exams or other indicators correlate with passing PANCE scores. We use correlation and statistical significance to find, primarily, which variables can be used to predict PANCE performance.

This chart shows how R values and p values change as we correlate certain variables to PANCE scores. 

  • We can see that “Admissions GPA” did not correlate with PANCE scores – that -.03 is almost a “zero” correlation coefficient, plus an extremely high p value shows high randomness in the numbers. 
  • There is a minor increase...
Continue Reading...

Advanced Assessment Methods: What is Statistical Significance?

scott's thoughts Nov 02, 2022

In my previous blog, we discussed the meaning of correlation and how we can use this measurement to determine if variables move positively or negatively in relation to one another. Today we will expand this discussion by explaining the importance of statistical significance when we determine how strongly two variables correlate.

Descriptive statistics provide a one-dimensional perspective. They can generate correlative relationships, but they cannot determine statistical significance. I think that, for those people who are research-oriented, whenever possible, statistical significance is important, and with educational outcomes, it is certainly appropriate. So, what does it mean to say that two variables correlate with statistical significance? Let us examine this question.

The Usefulness of the Correlation Coefficient

The knowledge of the relationship between two variables is useful in predicting one from the other, especially if one variable is observed in advance of the other....

Continue Reading...

Advanced Assessment Methods: What Does Correlation Really Mean?

scott's thoughts Oct 26, 2022

Thank you for joining me as I continue my blog series on utilizing advanced assessment methods for the vast amounts of data collected by your PA program. We have spent several blogs exploring the benefits of this methodology. Now we will break down the numbers so you can understand what it all means. What do applied statistics tell us in this context about correlations? What constitutes statistical significance? 

Before you race for the door, let me ease your mind. This will not be a statistics course. Statistics may seem so complex that they have no practical application, but that is far from the truth. With just a few definitions and an understanding of how variables move in relation to one another, you can reap the benefits of advanced analysis.

This is a topic near and dear to my heart. I have taught research methodology and stats to graduate health science students for more than fifteen years. I have also been intricately involved with assessment in PA education for many...

Continue Reading...

ARC-PA 5th Edition Standards: Summary

scott's thoughts Oct 19, 2022

Thank you for joining me through this series on responding to the requirements of ARC-PA’s 5th Edition Standards Appendix 14. Before we move on to our next subject, I offer some final insights that I hope will help you in creating a well-rounded and thoroughly supported SSR for your PA program.

Referencing Other Appendices Within Your SSR

The SSR is an interconnected document. Many of the appendices organically connect with each other. In responding to one appendix, you may always incorporate relevant data from other appendices. 14C (effectiveness of a program’s didactic curriculum) connects very well with 14F (presentation of PANCE outcomes).

Is it necessary to include all the same information in Appendix 14F that you have already included in 14C? My recommendation is to include an excerpt from 14C that is appropriate within the answer to 14F.

Be certain that you answer each and all of the questions asked in each appendix template and be cautious about simply...

Continue Reading...

ARC-PA 5th Edition Standards: Appendix 14H

scott's thoughts Oct 12, 2022

ARC-PA’s 5th Edition Standards require that your program’s goals become part of the data you collect and review. Appendix 14H is about your success in meeting program goals. Its instructions are:

  • The program must provide information in the template below for all of its published goals. Space has been provided for six program goals, but if the program has more than six goals use an additional template. Reference other appendices of the SSR as needed.
  • Provide a tabular or graphic display of data collected by the program for each of its goals for the past three academic years. If already provided in another appendix, reference that data in the analysis narrative
  • When creating your data display(s) please keep in mind: 
    • Quantitative data must be reported in aggregate and displayed in tables or graphs that directly support the analysis. 
    • Qualitative data themes used in the analysis must be reported and summarized in the narrative or displayed in an appended...
Continue Reading...

ARC-PA 5th Edition Standards: Appendix 14G, Sufficiency and Effectiveness of Faculty and Staff

scott's thoughts Oct 05, 2022

Appendix 14G looks at the sufficiency and effectiveness of your PA program’s principal and instructional faculty and staff. However, here’s a “warning” note. You still must develop, and explain, your own methodology about how you determine sufficiency. Therefore, the element about the complexity of the program itself must be addressed. 

Appendix 14G expects the following elements on your SSR:

  • Provide narrative describing the factors used to determine effectiveness of principal and instructional faculty in meeting the academic needs of enrolled students and managing the administrative responsibilities of the program
  • Describe how the program collects data related to those factors to determine effectiveness of program faculty in meeting the program’s needs
  • Provide narrative describing the factors used to determine effectiveness of administrative support staff in meeting the administrative responsibilities consistent with the organizational complexity...
Continue Reading...

ARC-PA 5th Edition Standards: Conclusion of Appendix 14F Correlations

scott's thoughts Sep 28, 2022

Today we continue our review of how to meet the requirements ARC-PA’s 5th Edition Standard’s Appendix 14F requirements, which include your presentation of PANCE outcomes in your PA program, looking at how admissions, course grades, test grades and other points of data correlate with these outcomes, and which of these are predictors of success or failure on the test. In this edition of our blog, we will discuss the purpose of correlating PANCE scores with 1) number of C-grades; 2) number of students remediations and 3) preceptor ratings. 

Correlating C-Grades and PANCE scores

When a student receives a near-failing or failing grade in a class, remediation comes swiftly. Something is clearly wrong. But C-Grades, which technically imply “average” performance in a classroom setting, tell their own story. Correlation of C-grades to PANCE scores should indicate whether the number of C grades is statistically significantly correlated with PANCE scores. We have...

Continue Reading...

ARC-PA 5th Edition Standards: Appendix 14F and Presenting Correlations to PANCE Performance

scott's thoughts Sep 21, 2022

Thanks for “clicking in” to my blog once again! As you know, we are deep into a review of the various requirements of Appendix 14 of ARC-PA’s 5th Edition Standards. Today we begin looking at Appendix 14F requirements, and some ideas on how the data may be presented.

In Appendix 14F, the Commission requests your presentation of PANCE outcomes in your PA program. This means looking at how admissions, course grades, test grades and other points of data correlate with PANCE scores, and which of these are predictors of success or failure on the test.

Data analysis related to PANCE outcomes is to include, but is not limited to, correlation of PANCE outcomes and:

  • Admissions criteria as a predictor of success 
  • Course outcomes
  • Course and instructor evaluations by students 
  • Program instructional objectives, learning outcomes, and breadth and depth of the curriculum
  • Student summative evaluation results
  • Remediation practices and results
  • Student progress criteria and...
Continue Reading...
1 2 3 4 5

50% Complete

Two Step

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.