{"id":9568,"date":"2026-07-27T12:30:27","date_gmt":"2026-07-27T10:30:27","guid":{"rendered":"https:\/\/www.kubicek.ai\/?post_type=lexicon&p=9568"},"modified":"2026-07-28T09:58:42","modified_gmt":"2026-07-28T07:58:42","slug":"self-supervised-learning","status":"publish","type":"lexicon","link":"https:\/\/www.kubicek.ai\/en\/lexicon\/self-supervised-learning\/","title":{"rendered":"Self-supervised learning"},"content":{"rendered":"

Self-supervised learning<\/strong> is a paradigm in which the model creates its training signal from the structure of the data itself, without human annotations. Formally it is supervised learning on a so-called pretext task whose correct answer can be derived from the data automatically \u2013 which is why it is sometimes a more apt term than unsupervised learning, where no target exists at all. It is precisely this approach that enabled the rise of large language models, because it bypasses the bottleneck of expensive human annotation and opens the way to training on all available text. The best-known pretext task is next-token prediction in autoregressive models and filling in masked words in BERT<\/a>. In computer vision one uses reassembling shuffled patches, colourising a black-and-white image, or contrastive learning, which pulls two differently distorted copies of the same shot together and pushes different shots apart. The learned representations are then exploited by transfer to real tasks. The key idea is that predicting a missing part of the data requires modelling the world that produced it.<\/p>\n

\n

It is a trick for manufacturing an endless supply of exercises complete with worked answers, without paying anyone to mark them. Take any book, cover up a word, and you have both the task and the correct answer \u2013 you are holding it under your finger. You can do this with every word of every book in the world. And here is the crucial part: to fill such a gap reliably, grammar is not enough. In the sentence “After the operation they had to give him a ___ because he had lost a lot of blood”, the student has to understand medicine. So the fill-in-the-blank quietly teaches the whole world.<\/p>\n","protected":false},"featured_media":0,"template":"","class_list":["post-9568","lexicon","type-lexicon","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon\/9568","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon"}],"about":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/types\/lexicon"}],"wp:attachment":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/media?parent=9568"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}