Imagine wanting to explain the meaning of words to a computer. Instead of giving it definitions, you assign every word a set of numbers, a kind of coordinates. Words with similar meanings, such as “dog” and “puppy”, get similar coordinates and end up close together in an imaginary space. Words like “dog” and “car”, by contrast, have very different coordinates and lie far apart. This set of numbers, or coordinates, is what is called an embedding. It lets the computer grasp the relationships between words better and work with their meaning, even though it does not understand language as a human does.
An embedding, in machine learning, is a numerical representation of categorical data in the form of a dense floating-point vector. Unlike sparse representations such as one-hot encoding, an embedding captures the semantic relationships between individual categories. Vectors of similar concepts lie closer together in this multi-dimensional space. The words “king” and “queen”, for example, will have similar vector representations. The technique allows neural networks to process and generalise information from discrete inputs – words in text, items in a catalogue – more efficiently. Embeddings are typically learned during the model’s training process on large amounts of data.