Covariance & Correlation

PSY 348: Lecture 15

Dave Brocker

Farmingdale State College

Correlation

What do you know about correlations?

  • Correlation ≠ Causation

  • \(r\)

  • Positive correlation: variables move together

    • As one goes up, the other goes up.

    • As one goes down, the other goes down.

Correlation

Assesses how two variables move in relation to one another

Ok, but how?!

Measures of Dispersion

What are measures of dispersion?

How spread out the data are: how much participants differ.

  • How did we measure dispersion when we were looking at one variable?

    • Standard deviation and variance.

Measures of Dispersion

How many variables?

looks_one

Univariate

Uni = one variable

looks_two

Bivariate

Bi = two variables

looks_3

Multivariate

Multi = more than two variables

Measures of Dispersion

Univariate

Standard deviation: how different are participants from the average?

Height: \(M = 5'6"\), \(s = 2\) inches

Bivariate

Covariance: do the variables vary in the same way?

Does the spread-out-ness of height overlap with the spread-out-ness of weight?

Covariance

Professor Brocker wants to know if how many cups of coffee a professor drinks daily impacts their happiness. He asks 50 professors how many cups they have in a typical day and then has them rate their happiness on a scale of 1 to 10.

  • Variable 1: number of cups of coffee (continuous)

  • Variable 2: happiness (continuous)

Covariance

Two things we could find

  • The variance of each variable on its own.

  • The covariance of the two variables together.

Covariance refers to how two continuous variables vary in tandem.

  • Generally, if cups of coffee goes up, what does happiness tend to do?

  • Generally, if happiness goes up, what does cups of coffee tend to do?

Covariance

The formula

Covariance

\(COV_{XY} = \dfrac{\Sigma(X-\bar{X})(Y-\bar{Y})}{N-1}\)

Well, well, well… we meet again

\(s^2 = \dfrac{\Sigma(X-\bar{X})^2}{N-1} = \dfrac{\Sigma(X-\bar{X})(X-\bar{X})}{N-1}\)

  • The variance of a variable is the covariance of that variable with itself.

Covariance

Let’s break it down

Steps

  1. Find the mean of each variable: \(\bar{X} = \dfrac{\Sigma X}{n}\) and \(\bar{Y} = \dfrac{\Sigma Y}{n}\).
  2. Subtract the mean of \(X\) from each \(X\) value, and the mean of \(Y\) from each \(Y\) value: \((X-\bar{X})\) and \((Y-\bar{Y})\).
  3. Multiply the deviation scores of \(X\) with the deviation scores of \(Y\): \((X-\bar{X})(Y-\bar{Y})\).
  4. Sum the products: \(\Sigma(X-\bar{X})(Y-\bar{Y})\).
  5. Divide by \(N-1\): \(\dfrac{\Sigma(X-\bar{X})(Y-\bar{Y})}{N-1}\).

Covariance

A worked example

Professor X (cups) Y (happiness) X - mean(X) Y - mean(Y) Product
1 1 4 −2.00 −2.33 4.67
2 2 5 −1.00 −1.33 1.33
3 2 6 −1.00 −0.33 0.33
4 3 6 0.00 −0.33 0.00
5 4 8 1.00 1.67 1.67
6 6 9 3.00 2.67 8.00
Sum — — — — — 16.00
  • \(\bar{X} = 3.00\), \(\bar{Y} = 6.33\), so \(COV_{XY} = \dfrac{16.00}{6 - 1} = 3.20\).

Covariance

In R

covariance.R

cov(ex$Cups, ex$Happiness)   # covariance
[1] 3.2
var(ex$Cups)                 # the variance is a covariance with itself
[1] 3.2
cov(ex$Cups, ex$Cups)        # same number
[1] 3.2

Covariance

What does it tell us?

Covariance is the bivariate equivalent of variance.

  • The sign tells us the direction: positive means the variables tend to rise together.

  • But the size is hard to interpret. It is in odd units (cups × happiness points) and changes if we change the scale.

  • Variance had the same problem (squared units), so we took the square root to get the standard deviation. For covariance, we standardize it.

Correlation

Standardized covariance

Pearson correlation

\(r = \dfrac{cov_{xy}}{(SD_x)(SD_y)} = \dfrac{\Sigma(X-\bar{X})(Y-\bar{Y})}{(SD_x)(SD_y)(N-1)}\)

  • Correlation is the standardized covariance: we divide the covariance by the standard deviation of \(x\) times the standard deviation of \(y\).

  • Just as a z-score puts a score on a unit-free scale, \(r\) puts the relationship on a unit-free scale.

Correlation

Back to the worked example

Compute r

\(r = \dfrac{cov_{xy}}{(SD_x)(SD_y)} = \dfrac{3.20}{(1.79)(1.86)} = 0.96\)

correlation.R

cor(ex$Cups, ex$Happiness)
[1] 0.9607689
  • A strong positive correlation: professors who drink more coffee tend to report higher happiness.

Correlation

Bounded between -1 and +1

  • Closer to -1 is a negative correlation, closer to +1 is a positive correlation, and closer to 0 means no correlation.

Correlation

The Pearson Product-Moment Correlation (Pearson, 1895)

  • The sign indicates the direction.

  • The number indicates the magnitude (strength) of the relationship:

Rules of thumb

  • 0.1 is a small correlation

  • 0.3 is a moderate correlation

  • 0.5 or higher is a large correlation

Correlation and \(R^2\)

  • Correlation is written \(r\).

  • If we square \(r\), we get \(R^2\), which is always positive.

  • \(R^2\) is the proportion of variance the two variables share:

    • the amount of \(y\) accounted for by \(x\),

    • and the amount of \(x\) accounted for by \(y\).

Correlation and \(R^2\)

Example

Students’ grades in MTH 110 and their grades in this course correlate at \(r = 0.3\).

  • A moderate correlation.

  • How much of your grade in this course is accounted for by your MTH 110 grade?

Compute \(R^2\)

\(R^2 = r^2 = 0.3 \times 0.3 = 0.09\), so 9% of the variance in your grade is accounted for by your MTH 110 grade.

Is the Correlation Significant?

Testing \(H_0: \rho = 0\)

  • Our sample \(r\) estimates the population correlation \(\rho\). Is it different from 0?

The test

\(t = \dfrac{r\sqrt{n-2}}{\sqrt{1-r^2}} \qquad df = n - 2\)

  • Example: \(r = 0.35\) with \(n = 50\) gives \(t(48) = 2.59\), \(p\) = .013: a significant (moderate) positive correlation.

Is the Correlation Significant?

In R

cor_test.R

# cor.test() gives r, t, df, p, and a 95% confidence interval
cor.test(ex1$X, ex1$Y)

    Pearson's product-moment correlation

data:  ex1$X and ex1$Y
t = 11.179, df = 48, p-value = 5.817e-15
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.7488197 0.9124630
sample estimates:
 cor 
0.85

Pitfalls

What can go wrong with \(r\)?

\(r\) measures linear relationships only, is pulled around by outliers, and shrinks when the range of one variable is restricted. Always plot your data.

Interpreting Correlation

  • We don’t use the words caused or resulting from.

  • When reporting \(r\): “As x goes up, y tends to go up.”

  • When reporting \(R^2\): “z% of the variance in y is accounted for by x.”

  • A third variable could explain both, and the direction could run either way.

Reporting Correlation

Correlation matrix

  • In the Method, we report descriptive statistics about our variables (\(M\), \(s\)).

  • It is common to refer readers to a table where they can look at correlations between the variables.

  • Why might this be important?

Correlation Matrix

Table 1 in a paper is typically a combination of descriptive stats and a correlation matrix:

Variable M s 1 2 3
1. Age 20.91 1.76 -- -.03 .65**
2. Song 1 4.21 1.20 -- -.34*
3. Song 2 6.42 2.18 --
* p < .05. ** p < .01.

Reporting Correlation

Example 1

Researchers collected the hours spent studying and the final exam score for 50 students (scatterplot below).

  • Describe the strength and direction of the relationship. Is the correlation weak, moderate, or strong? Positive or negative?

Reporting Correlation

Example 1: the correlation

  • Suppose the computed Pearson correlation coefficient is \(r = 0.85\). Explain what this value indicates about the relationship between the two variables.

Answer

A strong, positive linear relationship: students who studied more hours tended to earn higher exam scores. In APA style: hours studying and exam scores were strongly positively correlated, r(48) = .85, p < .001.

Reporting Correlation

Example 2

Using the same data, the researchers fit a linear regression model to predict exam scores from the number of hours spent studying. The \(R^2\) value for the model is 0.72.

Your questions

  1. What does \(R^2 = 0.72\) tell you? What percentage of the variance is explained, and what percentage remains unexplained?
  2. Would you consider study time a good predictor of exam performance? Why or why not?

Reporting Correlation

Example 2: answers

Answers

  1. 72% of the variance in exam scores is explained by study time, and 28% remains unexplained. (\(R^2 = r^2 = .85^2 \approx .72\))
  2. Study time is a strong predictor. Still, other factors matter, and a strong correlation does not by itself show that studying causes the higher scores.

Summary: Covariance & Correlation

  • Covariance is how two variables vary together: the bivariate equivalent of variance.

  • Correlation (\(r\)) is the standardized covariance: unit-free, bounded between -1 and +1.

  • The sign gives the direction and the size gives the strength; \(r^2\) is the shared variance.

  • Test \(H_0: \rho = 0\) with \(t = r\sqrt{n-2}/\sqrt{1-r^2}\), \(df = n - 2\).

  • Plot your data: outliers, curves and restricted ranges can fool \(r\).

  • Correlation does not imply causation.