Based on Understanding the Covariance Matrix | by Marvin Lanhenke | Towards Data Science See also Covariance and Correlation In Machine Learning | Fireblaze AI School Extremely important linear algebra concept central to many algorithms in SSL and PCA

turns out covariance is closely related to correlation coefficient: correlation coefficient is just the covariance normalized to \([-1, 1]\) Also closely related to variance, but instead of measuring variability in a single variable, we measure variability of two variables

Variance \(\sigma^2 = \frac{1}{(n-1)} \sum_i^n (x_i - \bar{x})(x_i - \bar{x})\) Covariance between two variables \(C_{x,y} = \frac{1}{(n-1)} \sum_i^n (x_i - \bar{x})(y_i - \bar{y})\)

The variance measures the variability in a single feature. But if each sample has two measurable features, then we can measure the covariance, indicating how two features change together (identical to the concept of correlation) Similarly, we can measure the covariance between all pairs of features as a covariance matrix ![[Pasted image 20221102233427.png|500]]

![[Pasted image 20221102233242.png]] High vs Low variance