Vector-Field Consistency is a consistency model for replicated data (for example, objects), initially described in a paper which was awarded the best-paper prize in the ACM/IFIP/Usenix Middleware Conference 2007. It has since been enhanced for increased scalability and fault-tolerance in a recent paper. == Description == This consistency model was initially designed for replicated data management in ad hoc gaming in order to minimize bandwidth usage without sacrificing playability. Intuitively, it captures the notion that although players require, wish, and take advantage of information regarding the whole of the game world (as opposed to a restricted view to rooms, arenas, etc. of limited size employed in many multiplayer video games), they need to know information with greater freshness, frequency, and accuracy as other game entities are located closer and closer to the player's position. It prescribes a multidimensional divergence bounding scheme, based on a vector field that employs consistency vectors k=(θ,σ,ν), standing for maximum allowed time - or replica staleness, sequence - or missing updates, and value - or user-defined measured replica divergence, applied to all space coordinates in game scenario or world. The consistency vector-fields emanate from field-generators designated as pivots (for example, players) and field intensity attenuates as distance grows from these pivots in concentric or square-like regions. This consistency model unifies locality-awareness techniques employed in message routing and consistency enforcement for multiplayer games, with divergence bounding techniques traditionally employed in replicated database and web scenarios.
Integreat
Integreat (former project name: Refguide+) is an open source mobile app that provides local information and services tailored to refugees and migrants coming to Germany. The content is maintained by local organizations, such as local governments or integration officers, and made available in locally relevant languages. It was developed by Tür an Tür - Digitalfabrik gGmbH (formerly Tür an Tür - Digital Factory gGmbH) in Augsburg together with a team of researchers and students from the Technical University of Munich. == History == In 1997, the Augsburg association "Tür an Tür", which has been working for refugees since 1992, published the brochure "First Steps", which answers local everyday questions. Since addresses and contact persons change quickly, some information is already outdated after a few weeks. Students of business informatics at the Technical University of Munich therefore developed the app Integreat within eight months together with the association and the social department of the city of Augsburg. The app was then also used by other cities and districts within months. As of February 3, 2022, information is available at 72 locations, including Munich, Dortmund, Nuremberg and Augsburg. == Mode of action == Refugees need information on areas such as registration, contact persons, health care, education, family, work and everyday life. Integreat seeks to provide refugees with this information by allowing them to select their geographic location and receive locally relevant information. This information is available offline once the app is opened so it can be used without an internet connection. In addition, the content is translated into the native languages of refugees and migrants to facilitate access. The content is licensed with a CC BY 4.0 license to facilitate collaboration and translation between content creators and dissemination of the content. Integreat is now being used for a broader migrant audience and says it can also support professionals, volunteers, and counseling centers. == Comparable mobile apps == Other mobile apps that are likewise intended to provide initial orientation for refugees include the app Ankommen, a joint project of the Federal Office for Migration and Refugees, the Goethe-Institut, the Federal Employment Agency and the Bavarian Broadcasting Corporation, which is intended as a companion for the first few weeks in Germany, and the Welcome App, a company-sponsored non-profit initiative for information about Germany and asylum procedures with a regional focus, and a book by the Konrad Adenauer Foundation (KAS) and Verlag Herder with a corresponding app Deutschland - Erste Informationen für Flüchtlinge (Germany - First Information for Refugees) as a companion for Arabic-speaking refugees in Germany.
Object co-segmentation
In computer vision, object co-segmentation is a special case of image segmentation, which is defined as jointly segmenting semantically similar objects in multiple images or video frames. == Challenges == It is often challenging to extract segmentation masks of a target/object from a noisy collection of images or video frames, which involves object discovery coupled with segmentation. A noisy collection implies that the object/target is present sporadically in a set of images or the object/target disappears intermittently throughout the video of interest. Early methods typically involve mid-level representations such as object proposals. == Dynamic Markov networks-based methods == A joint object discover and co-segmentation method based on coupled dynamic Markov networks has been proposed recently, which claims significant improvements in robustness against irrelevant/noisy video frames. Unlike previous efforts which conveniently assumes the consistent presence of the target objects throughout the input video, this coupled dual dynamic Markov network based algorithm simultaneously carries out both the detection and segmentation tasks with two respective Markov networks jointly updated via belief propagation. Specifically, the Markov network responsible for segmentation is initialized with superpixels and provides information for its Markov counterpart responsible for the object detection task. Conversely, the Markov network responsible for detection builds the object proposal graph with inputs including the spatio-temporal segmentation tubes. == Graph cut-based methods == Graph cut optimization is a popular tool in computer vision, especially in earlier image segmentation applications. As an extension of regular graph cuts, multi-level hypergraph cut is proposed to account for more complex high order correspondences among video groups beyond typical pairwise correlations. With such hypergraph extension, multiple modalities of correspondences, including low-level appearance, saliency, coherent motion and high level features such as object regions, could be seamlessly incorporated in the hyperedge computation. In addition, as a core advantage over co-occurrence based approach, hypergraph implicitly retains more complex correspondences among its vertices, with the hyperedge weights conveniently computed by eigenvalue decomposition of Laplacian matrices. == CNN/LSTM-based methods == In action localization applications, object co-segmentation is also implemented as the segment-tube spatio-temporal detector. Inspired by the recent spatio-temporal action localization efforts with tubelets (sequences of bounding boxes), Le et al. present a new spatio-temporal action localization detector Segment-tube, which consists of sequences of per-frame segmentation masks. This Segment-tube detector can temporally pinpoint the starting/ending frame of each action category in the presence of preceding/subsequent interference actions in untrimmed videos. Simultaneously, the Segment-tube detector produces per-frame segmentation masks instead of bounding boxes, offering superior spatial accuracy to tubelets. This is achieved by alternating iterative optimization between temporal action localization and spatial action segmentation. The proposed segment-tube detector is illustrated in the flowchart on the right. The sample input is an untrimmed video containing all frames in a pair figure skating video, with only a portion of these frames belonging to a relevant category (e.g., the DeathSpirals). Initialized with saliency based image segmentation on individual frames, this method first performs temporal action localization step with a cascaded 3D CNN and LSTM, and pinpoints the starting frame and the ending frame of a target action with a coarse-to-fine strategy. Subsequently, the segment-tube detector refines per-frame spatial segmentation with graph cut by focusing on relevant frames identified by the temporal action localization step. The optimization alternates between the temporal action localization and spatial action segmentation in an iterative manner. Upon practical convergence, the final spatio-temporal action localization results are obtained in the format of a sequence of per-frame segmentation masks (bottom row in the flowchart) with precise starting/ending frames.
Nanosemantics
Nanosemantics Lab is a Russian IT company specializing in natural language processing (NLP), computer vision (CV), speech technologies (ASR/TTS) and creation of interactive dialog interfaces, particularly chatbots and virtual assistants, based on artificial intelligence (AI). The company uses neural network platforms, including its own-made platform PuzzleLib which works on Russian-made microprocessor architecture Elbrus and Russia-based Astra Linux operating system. The company was founded in 2005 by Igor Ashmanov and Natalya Kaspersky. == Profile == The company was one of the first on Russian market to develop dialog interfaces for different branches of businesses, as well as to support community of AI developers. The company's most demanded product, as for beginning of the 2020s, is the automated "online advisers", functioning as chat bots, made for helping customers with usage of commercial products. In 2009 the company released an online service called iii.ru, where visitors were able to create their own AI-based virtual personalities entitles "infs" (for free). A visitor was able to train its own "inf" and let them chat to other "live" visitors as well with other "infs". More than 2.3 million of "infs" were created and trained by visitors over several years. Nanosemantics Lab maintains its own linguistic programming language for AI development called Dialog Language (DL). Popular social networks and instant messaging services may be used as base platforms. Nanosemantics' AI bots support different types of businesses: banks and financial services, telecommunications, retail, travel and automobile industry, home appliances production, etc. Among its solutions, Nanosemantics lists projects for various companies and institutions, among them VTB, Beeline, MTS, Sberbank, Higher School of Economics, Webmoney, Gazpromneft, Rostelecom, Ford Motors, Ministry of Health of the Russian Federation and others. The company uses the term "inf" for naming its numerous types of chat bots. The term was coined by co-founder Igor Ashmanov, head of Ashmanov & Partners. A 2014 scholarly research at Higher School of Economics, called "Basics of Business Informatics", states that such "infs", when used at business, may lower load on employees, collect statistics useful for understanding market demand and also may increase customer loyalty by providing fast and informative answers due to usage of large databases. The same research describes Nanosemantics' project for Russian branch of Ford Motors company, when AI capabilities were used for promoting the car model Ford Kuga. The research pointed out that within 2 months since beginning, the promo-website conducted 47774 talks of visitors with the specialized "inf", which indicated several hundred thousand of questions and the longest chat lasted for 3 hours 10 minutes. One-year promo campaign showed that 28.6% of people who made pre-orders talked to an "inf". In 2016 Nanosemantics launched a SaaS platform aimed at creating customized virtual assistants by users. The company's flagship product is considered to be Dialog Operating System (DialogOS), a professional corporate platform for creating intellectual voice and textual bots. It has its own linguistic programming language for creation of flexible scenarios and ready-studied neural natural language processing modules that are able to understand human interlocutors. In 2021 the company presented technology called NLab Speech ASR which contains a set of neural-networking algorithms for processing audio signals and analysis of texts that were trained and calibrated using speech-based big data marked up manually. The technology allows speed of processing of data up to "6 real-time factor" and precision values in noisy audio data may exceed 82%. In March 2022 the technology was included in Russia's Joint Registry for Russian Programs for Computers and Databases. As well, another technology was included: NLab Speech TTS, which is text-to-speech system that produces synthesized speech from printed text. == Joint projects == Nanosemantics participates in Ashmanov & Partners' projects related to AI. Since 2014, it helps in development of hardware "personal assistant" called Lexy, a solution similar to Amazon Alexa and the analogues. In August 2019 it was announced that Nanosemantics is going to participate in creation of open operating system for creating automated voice assistants. The project was called SOVA (Smart Open Virtual Assistant) and received investment of 300 million roubles (~$4,6 million) from Russian state-maintained National Technological Initiative. The company maintains long-term partnerships with Skolkovo Innovation Center (resident of IT cluster), branch association "Neuronet" and Yandex. Together with USA-based startup Remedy Logic, Nanosemantics has developed a medical diagnostic system for finding, using AI, spinal pathologies in tomography images of human bodies. Among them: central, foraminal and lateral lumbar stenosis, hernias, arthrosis. The system offers options of treatment. Since August 2021 the company is the resident of Technology Valley of Moscow State University. Also in 2021, Nanosemantics became a member of Committee on Artificial Intelligence within the Russian Association of Software Developers "Native Soft". The company states as one of its missions support of initiatives aimed at preservation and development of the Russian language. In May 2021, together with Pushkin Institute, the company created a chat bot called Phil, that explains to Russian people meaning of different Russian neologisms, and offers synonyms for them. Bot's vocabulary contains more than 500 neologisms, as well the bot can give advice on jargonisms and other types of specific words. Also in 2021, Nanosemanics Lab has signed the first-ever Russian "Codex of ethics of artificial intelligence". It establishes guidelines for ethical behavior of businesses that implement AI-based solutions. === IT contests === The company regularly organizes All-Russian Turing Test competitions for IT developers. Some of these events are co-organized with Microsoft. During the competitions, judges randomly choose virtual interlocutor and have a short conversation with them. They have to determine if a human or a machine is talking to them. An interlocutor may be either a bot or its human creator or operator. The results are measured in per cent of judges that were successfully convinced by a machine that it was a human. In 2021 Nanosemantics took part in federal project "Artificial Intelligence" by National Technological Initiative. In December 2021 the company together with state enterprise "Resource Center of Universal Design and Rehabilitation Technologies" (RCUD-RT) held an all-Russian hackathon aimed at development of AI solutions for medicine. During 3 days, participants created several training programs for patients with speech disorders. In April 2022, another hackathon by Nanosemantics was held together with MIREA – Russian Technological University. Students were participating and trying to generate algorithms for voice deepfakes. 17 teams contested in creation of software that generated artificial voice of a certain person. == Recognition == Since its foundation, Nanosemantics Lab has received a number of recognitions and awards. Among them are several professional ROTOR awards for the website iii.ru (created in 2009). The website gives the general public the means to create and train virtual assistants, which can then be used on a website or integrated into social networks. In 2013, a virtual assistant called Dana, created for Beeline Kazakhstan, was awarded with professional prize "Crystal Headset" in nomination "the best applying of technology". In 2015, the RBTH international media service included Nanosemantics in its list of "Top 50 Startups" in Russia. In 2016, the company received Russian state-maintained award called Runet Prize in two nominations: "State and Society" and "Technology and Innovation". In 2021, in Velikiy Novgorod, Nanosemantics team has won a hackathon aimed at finding means of discovering corruption schemes in Russian laws. In February 2022 the company won another contest by National Technological Initiative, called "Prochtenie", aimed at creation of AI systems for checking schoolchildren's school essays. The Nanosemantics team was awarded 20 million rubles for "overcoming technological barrier" in contest dedicated to English language, and 12 million for 1st place in special nomination "Structure" in Russian-language essay contest.
Object co-segmentation
In computer vision, object co-segmentation is a special case of image segmentation, which is defined as jointly segmenting semantically similar objects in multiple images or video frames. == Challenges == It is often challenging to extract segmentation masks of a target/object from a noisy collection of images or video frames, which involves object discovery coupled with segmentation. A noisy collection implies that the object/target is present sporadically in a set of images or the object/target disappears intermittently throughout the video of interest. Early methods typically involve mid-level representations such as object proposals. == Dynamic Markov networks-based methods == A joint object discover and co-segmentation method based on coupled dynamic Markov networks has been proposed recently, which claims significant improvements in robustness against irrelevant/noisy video frames. Unlike previous efforts which conveniently assumes the consistent presence of the target objects throughout the input video, this coupled dual dynamic Markov network based algorithm simultaneously carries out both the detection and segmentation tasks with two respective Markov networks jointly updated via belief propagation. Specifically, the Markov network responsible for segmentation is initialized with superpixels and provides information for its Markov counterpart responsible for the object detection task. Conversely, the Markov network responsible for detection builds the object proposal graph with inputs including the spatio-temporal segmentation tubes. == Graph cut-based methods == Graph cut optimization is a popular tool in computer vision, especially in earlier image segmentation applications. As an extension of regular graph cuts, multi-level hypergraph cut is proposed to account for more complex high order correspondences among video groups beyond typical pairwise correlations. With such hypergraph extension, multiple modalities of correspondences, including low-level appearance, saliency, coherent motion and high level features such as object regions, could be seamlessly incorporated in the hyperedge computation. In addition, as a core advantage over co-occurrence based approach, hypergraph implicitly retains more complex correspondences among its vertices, with the hyperedge weights conveniently computed by eigenvalue decomposition of Laplacian matrices. == CNN/LSTM-based methods == In action localization applications, object co-segmentation is also implemented as the segment-tube spatio-temporal detector. Inspired by the recent spatio-temporal action localization efforts with tubelets (sequences of bounding boxes), Le et al. present a new spatio-temporal action localization detector Segment-tube, which consists of sequences of per-frame segmentation masks. This Segment-tube detector can temporally pinpoint the starting/ending frame of each action category in the presence of preceding/subsequent interference actions in untrimmed videos. Simultaneously, the Segment-tube detector produces per-frame segmentation masks instead of bounding boxes, offering superior spatial accuracy to tubelets. This is achieved by alternating iterative optimization between temporal action localization and spatial action segmentation. The proposed segment-tube detector is illustrated in the flowchart on the right. The sample input is an untrimmed video containing all frames in a pair figure skating video, with only a portion of these frames belonging to a relevant category (e.g., the DeathSpirals). Initialized with saliency based image segmentation on individual frames, this method first performs temporal action localization step with a cascaded 3D CNN and LSTM, and pinpoints the starting frame and the ending frame of a target action with a coarse-to-fine strategy. Subsequently, the segment-tube detector refines per-frame spatial segmentation with graph cut by focusing on relevant frames identified by the temporal action localization step. The optimization alternates between the temporal action localization and spatial action segmentation in an iterative manner. Upon practical convergence, the final spatio-temporal action localization results are obtained in the format of a sequence of per-frame segmentation masks (bottom row in the flowchart) with precise starting/ending frames.
NAPLPS
NAPLPS (North American Presentation Layer Protocol Syntax) is a graphics language for use originally with videotex and teletext services. NAPLPS was developed from the Telidon system developed in Canada, with a small number of additions from AT&T Corporation. The basics of NAPLPS were later used as the basis for several other microcomputer-based graphics systems. == History == The Canadian Communications Research Centre (CRC), based in Ottawa, had been working on various graphics systems since the late 1960s, much of it led by Herb Bown. Through the 1970s they turned their attention to building out a system of "picture description instructions", which encoded graphics commands as a text stream. Graphics were encoded as a series of instructions (graphics primitives) each represented by a single ASCII character. Graphic coordinates were encoded in multiple 6-bit strings of XY coordinate data, flagged to place them in the printable ASCII range so that they could be transmitted with conventional text transmission techniques. ASCII SI/SO characters were used to differentiate the text from graphic portions of a transmitted "page". These instructions were decoded by separate programs to produce graphics output, on a plotter for instance. Other work produced a fully interactive version. In 1975, the CRC gave a contract to Norpak to develop an interactive graphics terminal that could decode the instructions and display them on a color display. During this period, a number of companies were developing the first teletext systems, notably the BBC's Ceefax system. Ceefax encoded character data into the lines in the vertical blanking interval of normal television signals where they could not be seen on-screen, and then used a buffer and decoder in the user's television to convert these into "pages" of text on the display. The Independent Broadcasting Authority quickly introduced their own ORACLE system, and the two organizations subsequently agreed to use a single standard, the "Broadcast Teletext Specification". This later became World System Teletext. At about the same time, other organizations were developing videotex systems, similar to teletext except they used modems to transmit their data instead of television signals. This was potentially slower and used up a telephone line, but had the major advantage of allowing the user to transmit data back to the sender. The UK's General Post Office developed a system using the Ceefax/ORACLE standard, launching it as Prestel, while France prepared the first steps for its ultimately very successful Minitel system, using a rival display standard called Antiope. By 1977, the Norpak system was running, and from this work the CRC decided to create their own teletext/videotext system. Unlike the systems being rolled out in Europe, the CRC decided from the start that the system should be able to run on any combination of communications links. For instance, it could use the vertical blanking interval to send data to the user, and a modem to return selections to the servers. It could be used in a one-way or two-way system. In teletext mode, character codes were sent to users' televisions by encoding them as dot patterns in the vertical blanking interval of the video signal. Various technical "tweaks" and details of the NTSC signals used by North American televisions allowed the downstream videotex channel to increase to 600 bit/s, about twice that used in the European systems. In videotext mode, Bell 202 modems were typical, offering a 1,200 bit/s download rate. A set top box attached to the TV decoded these signals back into text and graphics pages, which the user could select among. The system was publicly launched as Telidon on August 15, 1978. Compared to the European standards, the CRC system was faster, bi-directional, and offered real graphics as opposed to simple character graphics. The downside of the system was that it required much more advanced decoders, typically featuring Zilog Z80 or Motorola 6809 processors with RGB and/or RF output. The Innovation, Science and Economic Development Canada (then Department of Communications) launched a four-year plan to fund public roll-outs of the technology in an effort to spur the development of a commercial Telidon system. AT&T Corporation was so impressed by Telidon that they decided to join the project. They added a number of useful extensions, notably the ability to define original graphics commands (macro) and character sets (DRCS). They also tabled algorithms for proportionally spaced text, which greatly improved the quality of the displayed pages. A joint CSA/ANSI working group (X3L2.1) revised the specifications, which were submitted for standardization. In 1983, they became CSA T500 and ANSI X3.110, or NAPLPS. The data encoding system was also standardized as the NABTS (North American Broadcast Teletext Specification) protocol. Business models for Telidon services were poorly developed. Unlike the UK, where teletext was supported by one of only two large companies whose whole revenue model was based on a read-only medium (television), in North America Telidon was being offered by companies who worked on a subscriber basis. == One-way systems == Telidon-based teletext was tested in a few North American trials in the early 1980s — CBC IRIS, TVOntario, MTS-sponsored Project IDA, to name a few. NAPLPS was also part of the NABTS teletext standard, for the encoding and display of teletext pages. In the late 1980s and early 1990s, affiliates of the regional sports network group SportsChannel ran a service called Sports Plus Network, which ran sports news and scores while SportsChannel was not otherwise on the air. The screens, which frequently featured team logos or likenesses of players in addition to text, were drawn entirely with NAPLPS graphics and resembled the loading of Prodigy pages over a modem, though slightly faster. == Two-way systems == Various two-way systems using NAPLPS appeared in North America in the early 1980s. The biggest North American examples were Knight Ridder's Viewtron (based in Miami) and the Los Angeles Times' Gateway service (based in Orange County). Both used the Sceptre NAPLPS terminal from AT&T. The Sceptre contained a slow modem that connected over the consumer's telephone line to host computers. The Sceptre was expensive whether purchased or rented. Despite huge investments by their parent companies, neither Viewtron nor Gateway lasted into the second half of the decade. Another system, Keyfax, was developed by Keycom Electronic Publishing, a joint venture of Honeywell, Centel (since acquired by Sprint) and Field Enterprises, then-owner of the Chicago Sun-Times newspaper. Keyfax had originally been a WST teletext service, broadcast overnights on Field's Chicago television station WFLD-32 and through the VBI of both WFLD and national superstation WTBS; the decision was made to convert Keyfax into a subscription service, using a proprietary NAPLPS terminal device in a last-ditch effort to save the service. It did not work and Keyfax had ceased operations by the end of 1986. Other early-1980s NAPLPS technology was deployed in Canada, both as a way for rural Canadians to get news and weather information and as the platform for touchscreen information kiosks. In Vancouver these were featured at Expo 86. The kiosks became ubiquitous in Toronto under the name Teleguide, and were deployed in many shopping centres and at major tourist attractions. The latter city was the North American nexus of NAPLPS and the home of Norpak, the most successful of NAPLPS-oriented developers. Norpak created and sold hardware and software for NAPLPS development and display. TVOntario also developed NAPLPS content creation software. London, Ontario - based Cableshare used NAPLPS as the basis of touch-screen information kiosks for shopping malls, the flagship of which was deployed at Toronto's Eaton Centre. The system relied on an 8085-based microcomputer which drove several NAPLPS terminals fitted with touch screens, all communicating via Datapac to a back end database. The system offered news, weather and sports information along with shopping mall guides and coupons. Cableshare also developed and sold a leading NAPLPS page creation utility called the "Picture Painter." In the late 1980s, Tribune Media Services (TMS) and the Associated Press operated a cable television channel called AP News Plus that provided NAPLPS-based news screens to cable television subscribers in many U.S. cities. The news pages were created and edited by TMS staffers working on an Atex editing system in Orlando, Florida, and sent by satellite to NAPLPS decoder devices located at the local cable television companies. Among the firms providing technology to TMS and the Associated Press for the AP News Plus channel was Minneapolis-based Electronic Publishers Inc. (1985–1988). In 1981, two amateur radio operators (VE3FTT and VE3GQW) received special permission from the Canad
Large language model
A large language model (LLM) is a neural network trained on a vast amount of text for natural language processing tasks, especially language generation. LLMs can typically generate, summarize, translate and analyze text in many contexts, and are a foundational technology behind modern chatbots. Biased or inaccurate training data can make an LLM's output less reliable. As of 2026, the most capable LLMs are based on transformer architectures, which, according to the 2017 paper "Attention Is All You Need", can be more efficient and parallelizable than earlier statistical and recurrent neural network models. Benchmark evaluations for LLMs attempt to measure model reasoning, factual accuracy, alignment, and safety. == History == Before the emergence of transformer-based models in 2017, some language models were considered large relative to the computational and data constraints of their time. In the early 1990s, IBM's statistical models pioneered word alignment techniques for machine translation, laying the groundwork for corpus-based language modeling. In 2001, a smoothed n-gram model, such as those employing Kneser–Ney smoothing, trained on 300 million words, achieved state-of-the-art perplexity on benchmark tests. During the 2000s, with the rise of widespread internet access, researchers began compiling massive text datasets from the web ("web as corpus") to train statistical language models. Moving beyond n-gram models, researchers started in 2000 to use neural networks as language models. Following the breakthrough of deep neural networks in image classification around 2012, similar architectures were adapted for language tasks. This shift was marked by the development of word embeddings (e.g., Word2Vec by Mikolov in 2013) and sequence-to-sequence (seq2seq) models using LSTM. In 2016, Google transitioned its translation service to neural machine translation (NMT), replacing statistical phrase-based models with deep recurrent neural networks. These early NMT systems used LSTM-based encoder-decoder architectures, as they preceded the invention of transformers. At the 2017 NeurIPS conference, Google researchers introduced the transformer architecture in their landmark paper "Attention Is All You Need". This paper's goal was to improve upon 2014 seq2seq technology, and was based mainly on the attention mechanism developed by Bahdanau et al. in 2014. The following year in 2018, BERT was introduced and quickly became "ubiquitous". Though the original transformer has both encoder and decoder blocks, BERT is an encoder-only model. Academic and research usage of BERT began to decline in 2023, following rapid improvements in the abilities of decoder-only models (such as GPT) to solve tasks via prompting. Although decoder-only GPT-1 was introduced in 2018, it was GPT-2 in 2019 that caught widespread attention because OpenAI claimed to have initially deemed it too powerful to release publicly, out of fear of malicious use. GPT-3 in 2020 went a step further and as of 2025 is available only via API with no offering of downloading the model to execute locally. But it was the consumer-facing chatbot ChatGPT in late 2022 that received extensive media coverage and public attention by 2023. The 2023 GPT-4 was praised for its increased accuracy and as a "holy grail" for its multimodal capabilities. OpenAI did not reveal the high-level architecture and the number of parameters of GPT-4. The release of ChatGPT led to an uptick in LLM usage across several research subfields of computer science, including robotics, software engineering, and societal impact work. In 2024, OpenAI released the reasoning model OpenAI o1, which generates long chains of thought before returning a final answer. Many LLMs with parameter counts comparable to those of OpenAI's GPT series have been developed. Since 2022, weights-available models have been gaining popularity, especially at first with BLOOM and LLaMA, though both have restrictions on usage and deployment. Mistral AI's open-weight models Mistral 7B and Mixtral 8x7B have a more permissive Apache License. In January 2025, DeepSeek released DeepSeek R1, a 671-billion-parameter open-weight model that performs comparably to OpenAI o1 but at a much lower price per token for users. Since 2023, many LLMs have been trained to be multimodal, having the ability to also process or generate other types of data, such as images, audio, or 3D meshes. Open-weight LLMs have become more influential since 2023. Per Vake et al. (2025), community-driven contributions to open-weight models improve their efficiency and performance via collaborative platforms such as Hugging Face. == Dataset preprocessing == === Tokenization === As machine learning algorithms process numbers rather than text, the text must be converted to numbers. In the first step, a vocabulary is decided upon, then integer indices are arbitrarily but uniquely assigned to each vocabulary entry, and finally, an embedding is associated with the integer index. Algorithms include byte-pair encoding (BPE) and WordPiece. There are also special tokens serving as control characters, such as [MASK] for masked-out token (as used in BERT), and [UNK] ("unknown") for characters not appearing in the vocabulary. Also, some special symbols are used to denote special text formatting. For example, "Ġ" denotes a preceding whitespace in RoBERTa and GPT and "##" denotes continuation of a preceding word in BERT. For example, the BPE tokenizer used by the legacy version of GPT-3 would split tokenizer: texts -> series of numerical "tokens" as Tokenization also compresses the datasets. Because LLMs generally require input to be an array that is not jagged, the shorter texts must be "padded" until they match the length of the longest one. ==== Byte-pair encoding ==== As an example, consider a tokenizer based on byte-pair encoding. In the first step, all unique characters (including blanks and punctuation marks) are treated as an initial set of n-grams (i.e. initial set of uni-grams). Successively the most frequent pair of adjacent characters is merged into a bi-gram and all instances of the pair are replaced by it. All occurrences of adjacent pairs of (previously merged) n-grams that most frequently occur together are then again merged into even lengthier n-gram, until a vocabulary of prescribed size is obtained. After a tokenizer is trained, any text can be tokenized by it, as long as it does not contain characters not appearing in the initial-set of uni-grams. === Dataset cleaning === In the context of training LLMs, datasets are typically cleaned by removing low-quality, duplicated, or toxic data. Cleaned datasets can increase training efficiency and lead to improved downstream performance. A trained LLM can be used to clean datasets for training a further LLM. With the increasing proportion of LLM-generated content on the web, data cleaning in the future may include filtering out such content. LLM-generated content can pose a problem if the content is similar to human text (making filtering difficult) but of lower quality (degrading performance of models trained on it). === Synthetic data === Training of largest language models might need more linguistic data than naturally available, or that the naturally occurring data is of insufficient quality. In these cases, synthetic data might be used. == Training == An LLM is a type of foundation model (large X model) trained on language. LLMs can be trained in different ways. In particular, GPT models are first pretrained to predict the next word on a large amount of data, before being fine-tuned. === Cost === Substantial infrastructure is necessary for training the largest models. The tendency towards larger models is visible in the list of large language models. For example, the training of GPT-2 (i.e. a 1.5-billion-parameter model) in 2019 cost $50,000, while training of the PaLM (i.e. a 540-billion-parameter model) in 2022 cost $8 million, and Megatron-Turing NLG 530B (in 2021) cost around $11 million. The qualifier "large" in "large language model" is inherently vague, as there is no definitive threshold for the number of parameters required to qualify as "large". === Fine-tuning === Before being fine-tuned, most LLMs are next-token predictors. The fine-tuning shapes the LLM's behavior via techniques like reinforcement learning from human feedback (RLHF) or constitutional AI. Instruction fine-tuning is a form of supervised learning used to teach LLMs to follow user instructions. In 2022, OpenAI demonstrated InstructGPT, a version of GPT-3 similarly fine-tuned to follow instructions. Reinforcement learning from human feedback (RLHF) involves training a reward model to predict which text humans prefer. Then, the LLM can be fine-tuned through reinforcement learning to better satisfy this reward model. Since humans typically prefer truthful, helpful and harmless answers, RLHF favors such answers. == Architecture == LLMs are generally based on the tra