Principal Component Analysis (PCA) is the classical linear method of dimensionality reduction. It searches for a new orthogonal coordinate system in which the first axis points along the direction of greatest variance in the data, the second along the direction of greatest remaining variance perpendicular to the first, and so on. Keeping the first few components achieves a substantial reduction in the number of variables while preserving most of the information, measured precisely by explained variance. Mathematically it is the decomposition of the covariance matrix into eigenvalues and eigenvectors, or the singular value decomposition of the data matrix, with the eigenvalues giving the magnitude of variance along individual directions. An essential prerequisite is centring and usually standardising the data, because the method is sensitive to scale – a variable measured in units with large numbers would otherwise determine the entire first direction on its own. PCA serves for compression, for removing correlation and noise, for visualisation in two dimensions, and as pre-processing for methods suffering from the curse of dimensionality. Its limitations are linearity and poorer interpretability, since the components are mixtures of the original features; the non-linear alternative is an autoencoder.
Imagine photographing a straggling flock of birds flying distinctly in one direction. You have to record a three-dimensional reality on a flat image, and where you stand matters. Stand perpendicular to the line of flight and you see the flock along its whole length and can tell who is at the front and who at the back. Stand behind it and it merges into a single blob and the information is lost. PCA is exactly this search for the best spot for the photographer: it automatically finds the viewpoint from which the most differences in the data are visible, and throws away all the dull, flattened directions.