An autoencoder is a neural network trained without annotations to reconstruct its own input at the output. It consists of two parts: the encoder compresses the input into a low-dimensional representation in latent space, and the decoder attempts to restore the original data from it. The loss function is the reconstruction error, for example mean squared deviation. The trick lies in the bottleneck in the middle: because the latent vector is much smaller than the input, the network cannot simply copy the data and is forced to discover its internal structure and discard noise and redundancy. An autoencoder is thus a non-linear counterpart to dimensionality reduction – with linear layers and squared loss its solution approaches principal component analysis, and with non-linearities it substantially surpasses it. Practical uses include denoising (a denoising autoencoder learns to restore a clean signal from a corrupted input), anomaly detection based on high reconstruction error, compression and pre-training. Its generative extension, the variational autoencoder, additionally organises the latent space so that it can be sampled from.
Imagine an editor who has to condense every article into a single text message – and a second colleague who has to rewrite the article from that message. They train together until the new version barely differs from the original. To pull it off, the first cannot copy sentences; he has to capture the essence. And the second has to learn how to expand an essence into text. The by-product of this game is that text message – an extremely economical description of what the article is about. And when you suddenly slip them a text of a kind they have never seen, the reconstruction fails – by which you have also detected an anomaly.