Leela Chess Zero (abbreviated as LCZero, lc0) is a free, open-source chess engine and volunteer computing project based on Google's AlphaZero engine. It was spearheaded by Gary Linscott, a developer for the Stockfish chess engine, and adapted from the Leela Zero Go engine. Like Leela Zero and AlphaGo Zero, early iterations of Leela Chess Zero started with no intrinsic chess-specific knowledge other than the basic rules of the game. It learned how to play chess through reinforcement learning from repeated self-play, using a distributed computing network coordinated at the Leela Chess Zero website. However, as of November 2024 most models used by the engine are trained through supervised learning on data generated by previous reinforcement learning runs. As of June 2025, Leela Chess Zero has played over 2.5 billion games against itself, playing around 1 million games every day, and is capable of play at a level that is comparable with Stockfish, the leading conventional chess program. == History == The Leela Chess Zero project was first announced on TalkChess.com on January 9, 2018, as an open-source, self-learning chess engine attempting to recreate the success of AlphaZero. Within the first few months of training, Leela Chess Zero had already reached the Grandmaster level, surpassing the strength of early releases of Rybka, Stockfish, and Komodo, despite evaluating orders of magnitude fewer positions due to the size of the deep neural network it uses as its evaluation function. In December 2018, the AlphaZero team published a paper in Science magazine revealing previously undisclosed details of the architecture and training parameters used for AlphaZero. These changes were soon incorporated into Leela Chess Zero and increased both its strength and training efficiency. Work on Leela Chess Zero has informed the AobaZero project for shogi. The engine has been rewritten and carefully iterated upon since its inception, and since 2019 has run on multiple backends, allowing it to run on both CPU and GPU. The engine can be configured to use different weights, including even different architectures. This same mechanism of substitutable weights can also be used for alternative chess rules, such as for the Fischer Random Chess variant, which was done in 2019. == Neural network == Like AlphaZero, Leela Chess Zero employs neural networks which output both a policy vector, a distribution over subsequent moves used to guide search, and a position evaluation. These neural networks are designed to run on GPU, unlike traditional engines. It originally used residual neural networks, but in 2022 switched to using a transformer-based architecture designed by Daniel Monroe and Philip Chalmers. These models represent a chessboard as a sequence of 64 tokens and apply a trunk consisting of a stack of Post-LN encoder layers, outputting a sequence of 64 encoded tokens which is used to generate a position evaluation and a distribution over subsequent moves. They use a custom domain-specific position encoding called smolgen to improve the self-attention layer. As of November 2024, the models used by the engine are significantly larger and more efficient than the residual network used by AlphaZero, reportedly achieving grandmaster-level strength at one position evaluation per move. These models are able to detect and exploit positional features like trapped pieces and fortresses to outmaneuver traditional engines, giving Leela a unique playstyle. There is also evidence that they are able to perform look-ahead. == Program and use == Like AlphaZero, Leela Chess Zero learns through reinforcement learning, continually training on data generated through self-play. However, unlike AlphaZero, Leela Chess Zero decentralizes its data generation through distributed computing, with volunteers generating self-play data on local hardware which is fed to the reinforcement algorithm. In order to contribute training games, volunteers must download the latest non-release candidate (non-rc) version of the engine and the client. The client connects to the Leela Chess Zero server and iteratively receives the latest neural network version and produces self-play games which are sent back to the server and use to train the network. In order to run the Leela Chess Zero engine, two components are needed: the engine binary used to perform search, and a network used to evaluate positions. The client, which is used to contribute training data to the project, is not needed for this purpose. Older networks can also be downloaded and used by placing those networks in the folder with the Lc0 binary. == Spinoffs == In season 15 of the Top Chess Engine Championship, the engine AllieStein competed alongside Leela. AllieStein is a combination of two different spinoffs from Leela: Allie, which uses the same neural network as Leela, but has a unique search algorithm for exploring different lines of play, and Stein, a network which was trained using supervised learning on existing game data from games between other engines. While neither of these projects were admitted to TCEC separately due to their similarity to Leela, the combination of Allie's search algorithm with the Stein network, called AllieStein, was deemed unique enough to warrant its inclusion in the competition. In early 2021, the LcZero blog announced Ceres, a transliteration of the engine to C# which introduced several algorithmic improvements. The engine has performed competitively in tournaments, achieving third place in the TCEC Swiss 7 and fourth place in the TCEC Cup 14. In 2024, the CeresTrain framework was announced to support training deep neural networks for chess in PyTorch. == Competition results == In April 2018, Leela Chess Zero became the first engine using a deep neural network to enter the Top Chess Engine Championship (TCEC), during Season 12 in the lowest division, Division 4. Out of 28 games, it won one, drew two, and lost the remainder; its sole victory came from a position in which its opponent, Scorpio 2.82, crashed in three moves. However, it improved quickly. In July 2018, Leela placed seventh out of eight competitors at the 2018 World Computer Chess Championship. In August 2018, it won division 4 of TCEC season 13 with a record of 14 wins, 12 draws, and 2 losses. In Division 3, Leela scored 16/28 points, finishing third behind Ethereal, which scored 22.5/28 points, and Arasan on tiebreak. By September 2018, Leela had become competitive with the strongest engines in the world. In the 2018 Chess.com Computer Chess Championship (CCCC), Leela placed fifth out of 24 entrants. The top eight engines advanced to round 2, where Leela placed fourth. Leela then won the 30-game match against Komodo to secure third place in the tournament. Leela participated in the "TCEC Cup", an event in which engines from different TCEC divisions can play matches against one another. Leela defeated higher-division engines Laser, Ethereal and Fire before finally being eliminated by Stockfish in the semi-finals. In December 2018, Leela participated in Season 14 of the Top Chess Engine Championship. Leela dominated divisions 3, 2, and 1, easily finishing first in all of them. In the premier division, Stockfish dominated while Houdini, Komodo and Leela competed for second place. It came down to a final-round game where Leela needed to hold Stockfish to a draw with black to finish second ahead of Komodo. Leela managed this and therefore met Stockfish in the superfinal. In a back and forth match, first Stockfish and then Leela took three game leads before Stockfish won by the narrow margin of 50.5–49.5. In February 2019, Leela scored its first major tournament win when it defeated Houdini in the final of the second TCEC cup. Leela did not lose a game the entire tournament. In April 2019, Leela won the Chess.com Computer Chess Championship 7: Blitz Bonanza, becoming the first neural-network project to take the title. In the season 15 of the Top Chess Engine Championship (May 2019), Leela defended its TCEC Cup title, this time defeating Stockfish with a score of 5.5–4.5 (+2 =7 −1) in the final after Stockfish blundered a seven-man tablebase draw. Leela also won the Superfinal for the first time, scoring 53.5–46.5 (+14 −7 =79) versus Stockfish, including winning as both white and black in the same predetermined opening in games 61 and 62. Season 16 of TCEC saw Leela finish in third place in premier division, missing qualification for the Superfinal to Stockfish and the new deep neural network engine AllieStein. Leela was the only engine not to suffer any losses in the Premier division, and defeated Stockfish in one of the six games they played. However, Leela only managed to score nine wins, while AllieStein and Stockfish both scored 14 wins. This inability to defeat weaker engines led to Leela finishing third, half a point behind AllieStein and a point behind Stockfish. In the fourth TCEC Cup, Leela was seeded first as the defending champion,
ConEmu
ConEmu (short for Console emulator) is a free and open-source tabbed terminal emulator for Windows. ConEmu presents multiple consoles and simple GUI applications as one customizable GUI window with tabs and a status bar. It also provides emulation for ANSI escape codes for color, bypassing the capabilities of the standard Windows Console Host to provide 256 and 24-bit color in Windows. The program has a large range of customization, including custom color palettes for the standard 16 colors, hotkeys, transparency, an auto-hideable mode (similar to the way Quake originally displayed its developer console). Initially, the program was created as a companion to Far Manager, bringing some features common for graphical file managers to this console application (thumbnails and tiles, drag and drop with other windows, true color interface, and others). As of 2012, ConEmu could be used with any other Win32 console application or simple GUI tool (such as Notepad, PuTTY or DOSBox). ConEmu doesn't provide any shell itself, but rather allows using any other shell. It does provide a limited macro language, to control the hosted applications startup.
Strategic Air Command Digital Information Network
The Strategic Air Command DIgital Network (SACDIN) was a United States military computer network that provided computerized record communications, replacing the Data Transmission Subsystem and part of the Data Display Subsystem of the SAC Automated Command and Control System. SACDIN enabled a rapid flow of communications from headquarters SAC to its fielded forces, such as B-52 bases and ICBM Launch Control Centers. == Logistics == Major portions of SACDIN were developed, engineered and installed by the International Telephone and Telegraph (ITT) company, under contract to the Electronic Systems Center. == Chronology == 1969 - Headquarters SAC submits a request to the Joint Chiefs of Staff to study an expanded communications system, known as the SAC Total Information Network (SATIN). It would interconnect Air Force Satellite Communications (AFSATCOM), Advanced Airborne Command Post (AABNCP), Airborne Command Post (ABNCP), high frequency/single sideband radio HF/SSB radio, SAC Automated Command and Control System (SACCS), Automatic Digital Information Network (AUTODIN), Survivable Low Frequency Communications System (SLFCS) and Command Data Buffer (CDB) 1977 1 November - SATIN IV was effectively terminated by Congress. The restructured program was renamed SAC Digital Network (SACDIN), and was formulated to meet SAC's minimum essential data communications requirements, but also had the capability to grow in a modular fashion. 1986 ?? ??? - SACDIN replaces much of the SAC Automated Command and Control System (SACCS) and the SAC Automated Total Information Network (SATIN)
Cryptographic bill of materials
Cryptographic bill of materials (CBOM—also cryptography bill of materials) is a structured inventory of all cryptographic assets present in a software, firmware, device, or system. It enumerates algorithms (and parameters such as key sizes and modes), cryptographic libraries or modules, digital certificates, keys and related material, and protocols in use, and maps their relationships to the components that implement or invoke them. CBOMs are used to improve security analysis, compliance, and cryptographic agility, and are increasingly referenced in guidance for post‑quantum cryptography (PQC) migration. == Definition and scope == A CBOM inventories cryptographic primitives and materials—such as encryption and signature algorithms (with specific variants and modes), key sizes, cryptographic libraries/modules, digital certificates (e.g., X.509), keys and other related cryptographic material, and security protocols (e.g., TLS, IPsec). It also documents dependencies (for example, an application uses an algorithm provided by a library; a protocol uses several algorithms) and can capture certificate lifecycles, cryptographic module certifications (e.g., FIPS 140‑3), and policy conformance metadata. In common practice, a CBOM may be embedded within an SBOM format (such as CycloneDX) or exported as a separate, linked artifact. === Typical CBOM fields === The exact schema varies by implementation, but common fields are summarized below (see CycloneDX CBOM guide and NIST SP 1800‑38B). == Relation to SBOM == A CBOM is complementary to, but distinct from, a software bill of materials (SBOM). Whereas an SBOM lists software components and their versions, a CBOM focuses specifically on the cryptography present and how it is configured and used. For example, an SBOM might enumerate inclusion of a library such as OpenSSL, while the CBOM would identify which algorithms and parameters that library enables (e.g., RSA‑2048, ECDH P‑256, AES‑GCM) and list relevant keys and certificates. The pairing enables both supply‑chain transparency and cryptographic transparency. == History == The term and practice emerged in the early–mid 2020s alongside software‑supply‑chain transparency and PQC planning. The OWASP CycloneDX standard introduced native CBOM support (v1.6 and later), modeling algorithms, keys, certificates, and protocols as first‑class “cryptographic assets” and providing dependency semantics (uses/implements) between software and cryptography. Open tooling from industry and researchers (e.g., IBM's CBOMkit and related generators/viewers) appeared to automate discovery and representation of cryptographic use in the CycloneDX CBOM schema. == Regulatory and policy context == In the United States, policy has emphasized cryptographic inventories as a prerequisite to PQC migration. The White House's National Security Memorandum 10 (2022) directed a government‑wide transition to quantum‑resistant cryptography; the Office of Management and Budget's M‑23‑02 (November 2022) operationalized this by requiring agencies to submit a prioritized inventory of cryptographic systems (with algorithm and key details) by 4 May 2023 and annually thereafter, and tasked CISA/NSA/NIST to develop automated discovery and inventory strategies. A 2024 Office of the National Cyber Director report reiterated that a “comprehensive cryptographic inventory” is the baseline for PQC planning and must be maintained iteratively with both automated and manual discovery. NIST's NCCoE practice guide (SP 1800‑38B, preliminary draft) provides concrete methods for cryptographic discovery and documentation across enterprises, aligning with CBOM‑style representations. CISA later published a strategy to migrate federal agencies to automated cryptography discovery and inventory tools to support continuous reporting. Separately, NSA, CISA, and NIST issued joint guidance encouraging all organisations to prepare cryptographic inventories and roadmaps for PQC, beyond government environments. == Role in quantum readiness and cryptographic agility == Because large‑scale quantum computing threatens widely used public‑key algorithms (e.g., RSA, ECC), organisations are planning multi‑year transitions to post-quantum cryptography. CBOMs enable that planning by identifying where quantum‑vulnerable algorithms appear, prioritising high‑impact systems, and tracking replacements over time. A machine‑readable CBOM also supports cryptographic agility and incident response: if an algorithm, library, or certificate lifecycle becomes non‑compliant or vulnerable, the CBOM indicates which products and systems are affected and where mitigations must be applied first. == Standards and tooling == CycloneDX (OWASP): Native CBOM modelling (v1.6+) for algorithms, certificates, keys/related material, and protocols, with dependency semantics and examples. The project publishes a CBOM guide and use‑case profiles (e.g., certificate and algorithm inventories). NIST NCCoE SP 1800‑38 series: Practice guides for PQC migration include enterprise cryptographic discovery methods that produce CBOM‑like inventories and integrate multiple discovery tools. Government automation initiatives: Following M‑23‑02, CISA issued a strategy to migrate to automated cryptography discovery and inventory tools to support agency reporting and continuous inventory management. Open‑source and vendor tools: IBM's CBOMkit and related components generate, analyse, and visualise CBOMs; the IBM CBOM specification work was upstreamed into CycloneDX 1.6. === Data model and interchange (example) === CycloneDX provides machine‑readable encodings (JSON/XML) for CBOM content. The example below (subset) shows an application depending on a crypto library that provides the AES‑256‑GCM algorithm, and the application also depends on a leaf X.509 certificate. See the CycloneDX CBOM guide, JSON reference, and the “Implementation details” use‑case for the semantics of `dependsOn` and `provides`. == Relationship to cybersecurity supply chain initiatives == CBOMs complement SBOM‑focused supply‑chain transparency introduced by U.S. Executive Order 14028 and NTIA/NIST SBOM work. SBOMs document software components; CBOMs add detail on embedded cryptography to support risk management, policy compliance (e.g., disallowing deprecated algorithms), and PQC transition planning.
Ciphertext
In cryptography, ciphertext or cyphertext is the result of encryption performed on plaintext using an algorithm, called a cipher. Ciphertext is also known as encrypted or encoded information because it contains a form of the original plaintext that is unreadable by a human or computer without the proper cipher to decrypt it. This process prevents the loss of sensitive information via hacking. Decryption, the inverse of encryption, is the process of turning ciphertext into readable plaintext. Ciphertext is not to be confused with codetext, because the latter is a result of a code, not a cipher. == Conceptual underpinnings == Let m {\displaystyle m\!} be the plaintext message that Alice wants to secretly transmit to Bob and let E k {\displaystyle E_{k}\!} be the encryption cipher, where k {\displaystyle _{k}\!} is a cryptographic key. Alice must first transform the plaintext into ciphertext, c {\displaystyle c\!} , in order to securely send the message to Bob, as follows: c = E k ( m ) . {\displaystyle c=E_{k}(m).\!} In a symmetric-key system, Bob knows Alice's encryption key. Once the message is encrypted, Alice can safely transmit it to Bob (assuming no one else knows the key). In order to read Alice's message, Bob must decrypt the ciphertext using E k − 1 {\displaystyle {E_{k}}^{-1}\!} which is known as the decryption cipher, D k : {\displaystyle D_{k}:\!} D k ( c ) = D k ( E k ( m ) ) = m . {\displaystyle D_{k}(c)=D_{k}(E_{k}(m))=m.\!} Alternatively, in a non-symmetric key system, everyone, not just Alice and Bob, knows the encryption key; but the decryption key cannot be inferred from the encryption key. Only Bob knows the decryption key D k , {\displaystyle D_{k},} and decryption proceeds as D k ( c ) = m . {\displaystyle D_{k}(c)=m.} == Types of ciphers == The history of cryptography began thousands of years ago. Cryptography uses a variety of different types of encryption. Earlier algorithms were performed by hand and are substantially different from modern algorithms, which are generally executed by a machine. === Historical ciphers === Historical pen and paper ciphers used in the past are sometimes known as classical ciphers. They include: Substitution cipher: the units of plaintext are replaced with ciphertext (e.g., Caesar cipher and one-time pad) Polyalphabetic substitution cipher: a substitution cipher using multiple substitution alphabets (e.g., Vigenère cipher and Enigma machine) Polygraphic substitution cipher: the unit of substitution is a sequence of two or more letters rather than just one (e.g., Playfair cipher) Transposition cipher: the ciphertext is a permutation of the plaintext (e.g., rail fence cipher) Historical ciphers are not generally used as a standalone encryption technique because they are quite easy to crack. Many of the classical ciphers, with the exception of the one-time pad, can be cracked using brute force. === Modern ciphers === Modern ciphers are more secure than classical ciphers and are designed to withstand a wide range of attacks. An attacker should not be able to find the key used in a modern cipher, even if they know any specifics about the plaintext and its corresponding ciphertext. Modern encryption methods can be divided into the following categories: Private-key cryptography (symmetric key algorithm): one shared key is used for encryption and decryption Public-key cryptography (asymmetric key algorithm): two different keys are used for encryption and decryption In a symmetric key algorithm (e.g., DES, AES), the sender and receiver have a shared key established in advance: the sender uses the shared key to perform encryption; the receiver uses the shared key to perform decryption. Symmetric key algorithms can either be block ciphers or stream ciphers. Block ciphers operate on fixed-length groups of bits, called blocks, with an unvarying transformation. Stream ciphers encrypt plaintext digits one at a time on a continuous stream of data, with the transformation of successive digits varying during the encryption process. In an asymmetric key algorithm (e.g., RSA), there are two different keys: a public key and a private key. The public key is published, thereby allowing any sender to perform encryption. The private key is kept secret by the receiver, thereby allowing only the receiver to correctly perform decryption. == Cryptanalysis == Cryptanalysis (also referred to as codebreaking or cracking the code) is the study of applying various methodologies to obtain the meaning of encrypted information, without having access to the cipher required to correctly decrypt the information. This typically involves gaining an understanding of the system design and determining the cipher. Cryptanalysts can follow one or more attack models to crack a cipher, depending upon what information is available and the type of cipher being analyzed. Ciphertext is generally the most easily obtained part of a cryptosystem and therefore is an important part of cryptanalysis. === Attack models === Ciphertext-only: the cryptanalyst has access only to a collection of ciphertexts or code texts. This is the weakest attack model because the cryptanalyst has limited information. Modern ciphers rarely fail under this attack. Known-plaintext: the attacker has a set of ciphertexts to which they know the corresponding plaintext Chosen-plaintext attack: the attacker can obtain the ciphertexts corresponding to an arbitrary set of plaintexts of their own choosing Batch chosen-plaintext attack: where the cryptanalyst chooses all plaintexts before any of them are encrypted. This is often the meaning of an unqualified use of "chosen-plaintext attack". Adaptive chosen-plaintext attack: where the cryptanalyst makes a series of interactive queries, choosing subsequent plaintexts based on the information from the previous encryptions. Chosen-ciphertext attack: the attacker can obtain the plaintexts corresponding to an arbitrary set of ciphertexts of their own choosing Adaptive chosen-ciphertext attack Indifferent chosen-ciphertext attack Related-key attack: similar to a chosen-plaintext attack, except the attacker can obtain ciphertexts encrypted under two different keys. The keys are unknown, but the relationship between them is known (e.g., two keys that differ in the one bit). == Famous ciphertexts == The Babington Plot ciphers The Shugborough inscription The Zimmermann Telegram The Magic Words are Squeamish Ossifrage The cryptogram in "The Gold-Bug" Beale ciphers Kryptos Zodiac Killer ciphers
Contrastive Language-Image Pre-training
Contrastive Language-Image Pre-training (CLIP) is a technique for training a pair of neural network models, one for image understanding and one for text understanding, using a contrastive objective. This method has enabled broad applications across multiple domains, including cross-modal retrieval, text-to-image generation, and aesthetic ranking. == Algorithm == The CLIP method trains a pair of models contrastively. One model takes in a piece of text as input and outputs a single vector representing its semantic content. The other model takes in an image and similarly outputs a single vector representing its visual content. The models are trained so that the vectors corresponding to semantically similar text-image pairs are close together in the shared vector space, while those corresponding to dissimilar pairs are far apart. To train a pair of CLIP models, one would start by preparing a large dataset of image-caption pairs. During training, the models are presented with batches of N {\displaystyle N} image-caption pairs. Let the outputs from the text and image models be respectively v 1 , . . . , v N , w 1 , . . . , w N {\displaystyle v_{1},...,v_{N},w_{1},...,w_{N}} . Two vectors are considered "similar" if their dot product is large. The loss incurred on this batch is the multi-class N-pair loss, which is a symmetric cross-entropy loss over similarity scores: − 1 N ∑ i ln e v i ⋅ w i / T ∑ j e v i ⋅ w j / T − 1 N ∑ j ln e v j ⋅ w j / T ∑ i e v i ⋅ w j / T {\displaystyle -{\frac {1}{N}}\sum _{i}\ln {\frac {e^{v_{i}\cdot w_{i}/T}}{\sum _{j}e^{v_{i}\cdot w_{j}/T}}}-{\frac {1}{N}}\sum _{j}\ln {\frac {e^{v_{j}\cdot w_{j}/T}}{\sum _{i}e^{v_{i}\cdot w_{j}/T}}}} In essence, this loss function encourages the dot product between matching image and text vectors ( v i ⋅ w i {\displaystyle v_{i}\cdot w_{i}} ) to be high, while discouraging high dot products between non-matching pairs. The parameter T > 0 {\displaystyle T>0} is the temperature, which is parameterized in the original CLIP model as T = e − τ {\displaystyle T=e^{-\tau }} where τ ∈ R {\displaystyle \tau \in \mathbb {R} } is a learned parameter. Other loss functions are possible. For example, Sigmoid CLIP (SigLIP) proposes the following loss function: L = 1 N ∑ i , j ∈ 1 : N f ( ( 2 δ i , j − 1 ) ( e τ w i ⋅ v j + b ) ) {\displaystyle L={\frac {1}{N}}\sum _{i,j\in 1:N}f((2\delta _{i,j}-1)(e^{\tau }w_{i}\cdot v_{j}+b))} where f ( x ) = ln ( 1 + e − x ) {\displaystyle f(x)=\ln(1+e^{-x})} is the negative log sigmoid loss, and the Dirac delta symbol δ i , j {\displaystyle \delta _{i,j}} is 1 if i = j {\displaystyle i=j} else 0. == CLIP models == While the original model was developed by OpenAI, subsequent models have been trained by other organizations as well. === Image model === The image encoding models used in CLIP are typically vision transformers (ViT). The naming convention for these models often reflects the specific ViT architecture used. For instance, "ViT-L/14" means a "vision transformer large" (compared to other models in the same series) with a patch size of 14, meaning that the image is divided into 14-by-14 pixel patches before being processed by the transformer. The size indicator ranges from B, L, H, G (base, large, huge, giant), in that order. Other than ViT, the image model is typically a convolutional neural network, such as ResNet (in the original series by OpenAI), or ConvNeXt (in the OpenCLIP model series by LAION). Since the output vectors of the image model and the text model must have exactly the same length, both the image model and the text model have fixed-length vector outputs, which in the original report is called "embedding dimension". For example, in the original OpenAI model, the ResNet models have embedding dimensions ranging from 512 to 1024, and for the ViTs, from 512 to 768. Its implementation of ViT was the same as the original one, with one modification: after position embeddings are added to the initial patch embeddings, there is a LayerNorm. Its implementation of ResNet was the same as the original one, with 3 modifications: In the start of the CNN (the "stem"), they used three stacked 3x3 convolutions instead of a single 7x7 convolution, as suggested by. There is an average pooling of stride 2 at the start of each downsampling convolutional layer (they called it rect-2 blur pooling according to the terminology of ). This has the effect of blurring images before downsampling, for antialiasing. The final convolutional layer is followed by a multiheaded attention pooling. ALIGN a model with similar capabilities, trained by researchers from Google used EfficientNet, a kind of convolutional neural network. === Text model === The text encoding models used in CLIP are typically Transformers. In the original OpenAI report, they reported using a Transformer (63M-parameter, 12-layer, 512-wide, 8 attention heads) with lower-cased byte pair encoding (BPE) with 49152 vocabulary size. Context length was capped at 76 for efficiency. Like GPT, it was decoder-only, with only causally-masked self-attention. Its architecture is the same as GPT-2. Like BERT, the text sequence is bracketed by two special tokens [SOS] and [EOS] ("start of sequence" and "end of sequence"). Take the activations of the highest layer of the transformer on the [EOS], apply LayerNorm, then a final linear map. This is the text encoding of the input sequence. The final linear map has output dimension equal to the embedding dimension of whatever image encoder it is paired with. These models all had context length 77 and vocabulary size 49408. ALIGN used BERT of various sizes. == Dataset == === WebImageText === The CLIP models released by OpenAI were trained on a dataset called "WebImageText" (WIT) containing 400 million pairs of images and their corresponding captions scraped from the internet. The total number of words in this dataset is similar in scale to the WebText dataset used for training GPT-2, which contains about 40 gigabytes of text data. The dataset contains 500,000 text-queries, with up to 20,000 (image, text) pairs per query. The text-queries were generated by starting with all words occurring at least 100 times in English Wikipedia, then extended by bigrams with high mutual information, names of all Wikipedia articles above a certain search volume, and WordNet synsets. The dataset is private and has not been released to the public, and there is no further information on it. ==== Data preprocessing ==== For the CLIP image models, the input images are preprocessed by first dividing each of the R, G, B values of an image by the maximum possible value, so that these values fall between 0 and 1, then subtracting by [0.48145466, 0.4578275, 0.40821073], and dividing by [0.26862954, 0.26130258, 0.27577711]. The rationale was that these are the mean and standard deviations of the images in the WebImageText dataset, so this preprocessing step roughly whitens the image tensor. These numbers slightly differ from the standard preprocessing for ImageNet, which uses [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225]. If the input image does not have the same resolution as the native resolution (224×224 for all except ViT-L/14@336px, which has 336×336 resolution), then the input image is first scaled by bicubic interpolation, so that its shorter side is the same as the native resolution, then the central square of the image is cropped out. === Others === ALIGN used over one billion image-text pairs, obtained by extracting images and their alt-tags from online crawling. The method was described as similar to how the Conceptual Captions dataset was constructed, but instead of complex filtering, they only applied a frequency-based filtering. Later models trained by other organizations had published datasets. For example, LAION trained OpenCLIP with published datasets LAION-400M, LAION-2B, and DataComp-1B. == Training == In the original OpenAI CLIP report, they reported training 5 ResNet and 3 ViT (ViT-B/32, ViT-B/16, ViT-L/14). Each was trained for 32 epochs. The largest ResNet model took 18 days to train on 592 V100 GPUs. The largest ViT model took 12 days on 256 V100 GPUs. All ViT models were trained on 224×224 image resolution. The ViT-L/14 was then boosted to 336×336 resolution by FixRes, resulting in a model. They found this was the best-performing model. In the OpenCLIP series, the ViT-L/14 model was trained on 384 A100 GPUs on the LAION-2B dataset, for 160 epochs for a total of 32B samples seen. == Applications == === Cross-modal retrieval === CLIP's cross-modal retrieval enables the alignment of visual and textual data in a shared latent space, allowing users to retrieve images based on text descriptions and vice versa, without the need for explicit image annotations. In text-to-image retrieval, users input descriptive text, and CLIP retrieves images with matching embeddings. In image-to-text retrieval, images are used to find related text content. CLIP’s ability to connect vis
Consistency (database systems)
In database systems, consistency (or correctness) refers to the requirement that any given database transaction must change affected data only in allowed ways. Any data written to the database must be valid according to all defined rules, including constraints, cascades, triggers, and any combination thereof. This does not guarantee correctness of the transaction in all ways the application programmer might have wanted (that is the responsibility of application-level code) but merely that any programming errors cannot result in the violation of any defined database constraints. In a distributed system, referencing CAP theorem, consistency can also be understood as after a successful write, update or delete of a Record, any read request immediately receives the latest value of the Record. == As an ACID guarantee == Consistency is one of the four guarantees that define ACID transactions; however, significant ambiguity exists about the nature of this guarantee. It is defined variously as: The guarantee that database constraints are not violated, particularly once a transaction commits. The guarantee that any transactions started in the future necessarily see the effects of other transactions committed in the past. As these various definitions are not mutually exclusive, it is possible to design a system that guarantees "consistency" in every sense of the word, as most relational database management systems in common use today arguably do. == As a CAP trade-off == The CAP theorem is based on three trade-offs, one of which is "atomic consistency" (shortened to "consistency" for the acronym), about which the authors note, "Discussing atomic consistency is somewhat different than talking about an ACID database, as database consistency refers to transactions, while atomic consistency refers only to a property of a single request/response operation sequence. And it has a different meaning than the Atomic in ACID, as it subsumes the database notions of both Atomic and Consistent." In the CAP theorem, you can only have two of the following three properties: consistency, availability, or partition tolerance. Therefore, consistency may have to be traded off in some database systems.