Advanced Assessment Methods: What is Statistical Significance?

In my previous blog, we discussed the meaning of correlation and how we can use this measurement to determine if variables move positively or negatively in relation to one another. Today we will expand this discussion by explaining the importance of statistical significance when we determine how strongly two variables correlate.

Descriptive statistics provide a one-dimensional perspective. They can generate correlative relationships, but they cannot determine statistical significance. I think that, for those people who are research-oriented, whenever possible, statistical significance is important, and with educational outcomes, it is certainly appropriate. So, what does it mean to say that two variables correlate with statistical significance? Let us examine this question.

The Usefulness of the Correlation Coefficient

The knowledge of the relationship between two variables is useful in predicting one from the other, especially if one variable is observed in advance of the other. You may recall that in our last blog, we gave the example of precipitation (our independent variable) correlating to tardiness (our dependent variable). In our proposed outcomes, we suggested that the correlation coefficient between these two numbers might be as low as .10 percent or as high as .72 percent. Generally speaking:

• A correlation coefficient (either positive or negative) of:
• ~0.1 is considered small
• ~0.3 is considered medium
• ~0.5 is considered large

So, if our correlation coefficient (or, “R”) were .72 percent, we could say we had an extremely large correlation coefficient. In our PA data, it is common to observe large correlation coefficients (R > 0.5) between performance in PA program variables and PANCE scores.

However, statistical significance can change the dynamic to of your correlation coefficient. This is called the p value, and in plain language, it measures how likely it is that your correlation coefficient is due to random chance.  You may track precipitation and tardiness for a week and find that your correlation coefficient is .80 – extremely high – so that it seems that tardiness moves inexorably in relation to precipitation. But your p value is also extremely high, far more than .05 percent – your small sample size and lack of data mean there is no way to tell if your results are due to random chance. There is no way to determine with any certainty that precipitation and tardiness have anything to do with each other, even though during this week, they certainly seem to go hand in hand. 

The way we solve this problem is to increase sample size. Rather than just one week of data, you want six, or ten – the larger your sample size, the more you will be able to home in on a p value that says your correlation is not random. 

Conventionally, a p value of less than .05 is desirable, meaning there is less than five percent chance that everything you are seeing is due to random chance. More conservative studies (medicine and other hard sciences) might demand closer to a 1% significance level: but a one percent chance of randomness is not going to happen in most cases that you are ever going to see. In most cases of performance data, less than .05 is acceptable, because we can then say, “Look, we have shown statistically that there is less than 5% chance that these results are random.”  

To summarize:

• It is essential to evaluate the statistical significance of the correlation coefficient in addition to its size
• A common conventional alpha value used in educational settings is 0.05 (also referred to as the 5% significance level)
• When more conservative decision-making is desired, we use a lower alpha value of 0.01 (1% significance level) 

When comparing R and p value, there are four scenarios that can happen:

1. Low R value and low p value 

This is an interesting twist – just because your correlation coefficient is not particularly high does not mean there is not statistical significance. It simply means that your model, while producing a certain amount of statistical significance, does not explain much of the variation. This is better than nothing, but you need to better define your variables.

2. Low R value and high p value 

Alas, this is the worst outcome – it means that you have found little correlation, and a high randomness chance as well.

3. High R value and low p value 

This is the best outcome you can have on a model – you have found a high correlation and a low chance of randomness.

4. High R value and high p value 

This model has found a high amount of correlation but no statistical significance – all that correlation could just be random chance. It is time now to better define your variables or get a larger sample size.

In my next blog, we will examine how we can use R and p values when looking for relationships and predictors of PANCE scores. I hope you will join me then!