{"id":9530,"date":"2026-07-27T12:30:28","date_gmt":"2026-07-27T10:30:28","guid":{"rendered":"https:\/\/www.kubicek.ai\/?post_type=lexicon&p=9530"},"modified":"2026-07-28T09:58:41","modified_gmt":"2026-07-28T07:58:41","slug":"bert","status":"publish","type":"lexicon","link":"https:\/\/www.kubicek.ai\/en\/lexicon\/bert\/","title":{"rendered":"BERT"},"content":{"rendered":"

BERT<\/strong> (Bidirectional Encoder<\/a> Representations from Transformers) is a language model introduced by Google in 2018 that uses only the encoder part of the transformer. Unlike autoregressive models with unidirectional attention, every token sees the context to its left and to its right at the same time, which gives it a deeper grasp of meaning within a sentence but makes fluent text generation impossible. It is trained on two self-supervised tasks: masked language modelling, where about fifteen percent of the tokens in a text are hidden and the model has to fill them in, and next-sentence prediction. The resulting pre-trained model is then fine-tuned for a specific task by adding a small output head \u2013 sentiment classification, named-entity recognition, extracting an answer from a passage. BERT stood at the birth of the “pre-train and fine-tune” paradigm, and its deployment in Google Search in 2019 was among the most significant changes ever made to query understanding. Its lightweight descendants such as DistilBERT, and language-specific variants, are still used today for classification and for producing embeddings.<\/p>\n

\n

Picture the difference as a fill-in-the-blank test. A generative model is like someone writing a story who at any moment sees only what they have already written \u2013 they cannot look ahead, because there is nothing there yet. BERT, by contrast, is like someone handed a finished text with one word erased, asked to supply it. It sees both sides of the gap: “I went to the bank to ___ some money” points unambiguously to “withdraw”. That is exactly why BERT is good at understanding \u2013 working out what a query is about, what mood it carries, which words are company names \u2013 and poor at writing longer texts.<\/p>\n","protected":false},"featured_media":0,"template":"","class_list":["post-9530","lexicon","type-lexicon","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon\/9530","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon"}],"about":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/types\/lexicon"}],"wp:attachment":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/media?parent=9530"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}