Writing Higher Order Exam Questions

How we assess students matters.  Students will rise to the level of expectation laid before them, and the level at which they are assessed will inform their study strategies.  Consistently presented with lower order exam questions requiring nothing more than rote memorization, students will resort to solely memorizing the information presented in didactics.  If they lack prior experience with high level assessments, students are ill-equipped to transition to assessments requiring complex conceptual integration and practical application. 

As clinicians and PA educators, we know memorization does not translate to ability to apply clinical reasoning to patient care. Yet, all too often, we fail to push our students to develop the synthesis and critical thinking skills needed and are left scratching our heads when they flounder on rotations and underperform on benchmarking exams.

Why do programs fail to assess students at high levels during didactics?  The problem is likely multifactorial.  It is certainly easier to write a straightforward, lower level question.  For newer or inexperienced faculty members, item writing is a skill that takes substantial time to develop and in the absence of intentional mentoring, may never reach the level capable of writing more complex questions. Whatever the barriers within your program, we offer you a few tips to help step-up your question writing complexity.

Use two-step vignettes

A well-crafted vignette providing enough information to recognize the disease state.  Traditional vignette questions require one step application of knowledge to interpret the stem and arrive at an answer, usually the most likely diagnosis.  A two-step vignette requires an additional level of synthesis: describing a disease presentation, requiring the student to make the diagnosis in their head and then apply what they know about the diagnosis to answer a question like “what diagnostic test would confirm the most likely diagnosis?”  Some additional question options to add to vignettes:

• given the most likely diagnosis, what physiologic change has occurred in this patient?
• what information found in this patient’s history would confirm the most likely diagnosis?
• what finding on physical exam would confirm the most likely diagnosis?
• what is the best treatment for the most likely diagnosis?

Using these options, one strong base vignette can result in the addition of five or more questions to your test item bank for use in future years.  Additionally, simple changes to the stem such as altering vital signs to make the patient unstable, adding co-morbidities or allergies, or changing patient age creates another new set of questions with nuanced answers.

Resist the buzzwords

We have all had the experience of a student who uses an eponym or other medical term they have memorized without being able to explain it.  Using this terminology in test items encourages only association of the word with a condition, rather than understanding why they are connected.  To encourage students to move past this type of surface learning, instead of using these terms like “Murphy sign” as an answer, describe it instead as “an abrupt cessation of inspiration with palpation at the right costal margin in the midclavicular line.”

These are just a few examples of ways you can easily increase the complexity of your exam questions and, as a result, challenge your students to move beyond memorization in their study and encourage synthesis and in-depth understanding of concepts better suited for clinical practice.