Volume 7: Common Mistakes: Benchmarks

I am pleased you are joining me as we continue looking at the common sources of observations and citations on the ARC-PA SSR. As we have seen, refiguring the way your program records and structures its data is a great way to begin. Last week, we looked into generally improving a program’s meeting minutes to make sure that they are worded and organized according to SSR language and expectations. Another key to ensuring ARC-PA compliance is the understanding and correct use of benchmarks in your program.

Focusing extensively on program benchmarks may seem random – isn’t a benchmark simply a measuring stick by which the program can see if it is meeting a standard? Perhaps that is part of the problem: benchmarks seem like common sense, and so we assume they are easily grasped. The ARC-PA, however, expects unambiguous language about the reasoning behind both benchmarks and benchmark-based conclusions. I see programs cited repeatedly for poor understanding and explanation of their benchmarks, leading to further problems when those poorly defined benchmarks are used to define areas needing improvement.

Benchmark definitions and categories:

Very well, then. Just how specific and supported does a benchmark need to be? To consistently meet the requirements of ARC-PA, I believe we need to express the following level of granularity with benchmarks.

Define benchmarks for all data sets. Benchmarks must include the definition of what constitutes a strength and what constitutes the area in need of improvement. Here is my example of a benchmark definition:

Remember, SSRs are not entirely “real life” – they do not capture all the strengths or activities of your program. An SSR captures what you define based on your benchmarks.

Elements of the Appendix 14 Template

The Appendix 14 Template, to which each program must respond, requires that programs provide a narrative describing the program’s “approach to the analysis of the qualitative data collected and displayed, including the benchmarks/thresholds (with rationale) used.” Immediately, we see that we must provide rationale for the markers we use to judge the difference between inadequate, adequate, and outstanding performance. Don’t ask the  ARC-PA to guess why your program chose a certain benchmark.

The Template lists the following data for inclusion:

Analysis. Provide narrative describing the analysis of all data collected and displayed. Include resulting conclusion(s) and application of the analysis to the program.

Strengths: List the strengths of the program identified as a result of the data analysis and conclusions documented in this appendix. If none, then leave this section blank.

Modifications: List modifications that occurred (and are now finished) as a result of the program’s ongoing self-assessment process described in this appendix. If none, leave the section blank. Note, that action plans need to be driven by benchmarks. Don’t include anything you came up with without supporting data; this will result in an observation.

Areas in need of improvement: As a result of the data analysis and conclusions identified in this appendix, list the areas in need of improvement and plans for addressing these areas as identified by the program’s process of ongoing self-assessment. If none, then leave this section blank. If completed, list it in the Modifications table. This section, too, must be completely data-driven, and the foundation of your plan should be in that benchmark.

Concerning your program’s strengths and areas in need of improvement, remember:

•    Define areas in need of improvement and strengths with explicit language or definitions that drive your conclusion.
•    Include areas in need of improvement, modifications, and strengths that are explicitly tied to quantitative or qualitative data. The quantitative or qualitative data must be based upon valid benchmarks according to the ARC-PA, derived from the narrative within the Appendix 14 template, and directly related to a data source.
•    Be certain that areas in need of improvement, modifications, and strengths are defined by specific benchmarks within the C 1.01 narrative.
•    Be sure that your SSR does not include any strengths, modifications, or areas that need improvement that are not listed in the Appendix 14 template. 

Insufficient data and benchmarks

Presenting benchmarks that include only one data source is a guarantee for receiving an observation. I have seen this repeatedly. Likewise, drawing conclusions from only one year of data to make modifications or action plans is insufficient.

A program might implement a new assessment tool (a survey, for example) that after one year produces impressive results on which it bases an action plan. But this doesn’t qualify as documented data analysis; one year of data doesn’t cut it with ARC-PA. They want three years of data, at least, before they are willing to concur that your data analysis is verified.

Now, clearly, sometimes one year of data is all you have. Perhaps your program is new, or a certain benchmark has been recently instated. You may present the information with less than three years of data. This in itself is not a problem. However, do not draw conclusions or suggest modifications due to it.

Conclusion

 Commonly in SSR observations, I see the committee using this language: “Benchmarks and rationale were not consistently documented.” Like creating and keeping good meeting minutes, solving this problem is a matter of starting from the ground up and being consistent. Define your program’s benchmarks using adequate data sources, use the ARC-PA’s terminology, and then express consistent usage not only in the benchmark itself, but in determining when that benchmark has been missed, and in the modifications and action plans required to remedy the inadequacy.

In the next blog, we’ll continue examining the common mistakes found in SSRs and what you can do to avoid them. We’ll examine in more depth the requirements for triangulation of data and data analysis, and what exactly that means in the context of your SSR. Please join me then!