Digital data or digital information, in information theory and information systems, is data or information represented as a string of discrete symbols, each of which can take on one of only a finite number of values from some alphabet, such as letters or digits. An example is a text document, which consists of a string of alphanumeric characters. The most common form of digital data in modern information systems is binary data, which is represented by a string of binary digits (bits) each of which can have one of two values, either 0 or 1. Digital data can be contrasted with analog data, which is represented by a value from a continuous range of real numbers. Analog data is transmitted by an analog signal, which not only takes on continuous values but can vary continuously with time, a continuous real-valued function of time. An example is the air pressure variation in a sound wave. Data requires interpretation to become information. In modern (post-1960) computer systems, all data is digital. The word digital comes from the same source as the words digit and digitus (the Latin word for finger), as fingers are often used for counting. Mathematician George Stibitz of Bell Telephone Laboratories used the word digital in reference to the fast electric pulses emitted by a device designed to aim and fire anti-aircraft guns in 1942. The term is most commonly used in computing and electronics, especially where real-world information is converted to binary numeric form as in digital audio and digital photography. == Symbol to digital conversion == Since symbols (for example, alphanumeric characters) are not continuous, representing symbols digitally is rather simpler than conversion of continuous or analog information to digital. Instead of sampling and quantization as in analog-to-digital conversion, such techniques as polling and encoding are used. A symbol input device usually consists of a group of switches that are polled at regular intervals to see which switches are switched. Data will be lost if, within a single polling interval, two switches are pressed, or a switch is pressed, released, and pressed again. This polling can be done by a specialized processor in the device to prevent burdening the main CPU. When a new symbol has been entered, the device typically sends an interrupt, in a specialized format, so that the CPU can read it. For devices with only a few switches (such as the buttons on a joystick), the status of each can be encoded as bits (usually 0 for released and 1 for pressed) in a single word. This is useful when combinations of key presses are meaningful, and is sometimes used for passing the status of modifier keys on a keyboard (such as shift and control). But it does not scale to support more keys than the number of bits in a single byte or word. Devices with many switches (such as a computer keyboard) usually arrange these switches in a scan matrix, with the individual switches on the intersections of x and y lines. When a switch is pressed, it connects the corresponding x and y lines together. Polling (often called scanning in this case) is done by activating each x line in sequence and detecting which y lines then have a signal, thus which keys are pressed. When the keyboard processor detects that a key has changed state, it sends a signal to the CPU indicating the scan code of the key and its new state. The symbol is then encoded or converted into a number based on the status of modifier keys and the desired character encoding. A custom encoding can be used for a specific application with no loss of data. However, using a standard encoding such as ASCII is problematic if a symbol such as 'ß' needs to be converted but is not in the standard. It is estimated that in the year 1986, less than 1% of the world's technological capacity to store information was digital and in 2007 it was already 94%. The year 2002 is assumed to be the year when humankind was able to store more information in digital than in analog format (the "beginning of the digital age"). == States == Digital data come in these three states: data at rest, data in transit, and data in use. The confidentiality, integrity, and availability have to be managed during the entire lifecycle from 'birth' to the destruction of the data. === Data at rest === Data at rest in information technology means data that is housed physically on computer data storage in any digital form (e.g. cloud storage, file hosting services, databases, data warehouses, spreadsheets, archives, tapes, off-site or cloud backups, mobile devices etc.). Data at rest includes both structured and unstructured data. This type of data is subject to threats from hackers and other malicious threats to gain access to the data digitally or physical theft of the data storage media. To prevent this data from being accessed, modified or stolen, organizations will often employ security protection measures such as password protection, data encryption, or a combination of both. The security options used for this type of data are broadly referred to as data-at-rest protection (DARP). Definitions include: "...all data in computer storage while excluding data that is traversing a network or temporarily residing in computer memory to be read or updated." "...all data in storage but excludes any data that frequently traverses the network or that which resides in temporary memory. Data at rest includes but is not limited to archived data, data which is not accessed or changed frequently, files stored on hard drives, USB thumb drives, files stored on backup tape and disks, and also files stored off-site or on a storage area network (SAN)." While it is generally accepted that archive data (i.e. which never changes), regardless of its storage medium, is data at rest and active data subject to constant or frequent change is data in use. “Inactive data” could be taken to mean data which may change, but infrequently. The imprecise nature of terms such as “constant” and “frequent” means that some stored data cannot be comprehensively defined as either data at rest or in use. These definitions could be taken to assume that Data at Rest is a superset of data in use; however, data in use, subject to frequent change, has distinct processing requirements from data at rest, whether completely static or subject to occasional change. ==== Security ==== Because of its nature data at rest is of increasing concern to businesses, government agencies and other institutions. Mobile devices are often subject to specific security protocols to protect data at rest from unauthorized access when lost or stolen and there is an increasing recognition that database management systems and file servers should also be considered as at risk; the longer data is left unused in storage, the more likely it might be retrieved by unauthorized individuals outside the network. Data encryption, which prevents data visibility in the event of its unauthorized access or theft, is commonly used to protect data in motion and increasingly promoted for protecting data at rest. The encryption of data at rest should only include strong encryption methods such as AES or RSA. Encrypted data should remain encrypted when access controls such as usernames and password fail. Increasing encryption on multiple levels is recommended. Cryptography can be implemented on the database housing the data and on the physical storage where the databases are stored. Data encryption keys should be updated on a regular basis. Encryption keys should be stored separately from the data. Encryption also enables crypto-shredding at the end of the data or hardware lifecycle. Periodic auditing of sensitive data should be part of policy and should occur on scheduled occurrences. Finally, only store the minimum possible amount of sensitive data. Tokenization is a non-mathematical approach to protecting data at rest that replaces sensitive data with non-sensitive substitutes, referred to as tokens, which have no extrinsic or exploitable meaning or value. This process does not alter the type or length of data, which means it can be processed by legacy systems such as databases that may be sensitive to data length and type. Tokens require significantly less computational resources to process and less storage space in databases than traditionally encrypted data. This is achieved by keeping specific data fully or partially visible for processing and analytics while sensitive information is kept hidden. Lower processing and storage requirements makes tokenization an ideal method of securing data at rest in systems that manage large volumes of data. A further method of preventing unwanted access to data at rest is the use of data federation especially when data is distributed globally (e.g. in off-shore archives). An example of this would be a European organisation which stores its archived data off-site in the US. Under the terms of the USA PATRIOT Act the American authorities can demand
Tertiary review
In software engineering, a tertiary review is a systematic review of systematic reviews. It is also referred to as a tertiary study in the software engineering literature. However, Umbrella review is the term more commonly used in medicine. Kitchenham et al. suggest that methodologically there is no difference between a systematic review and a tertiary review. However, as the software engineering community has started performing tertiary reviews new concerns unique to tertiary reviews have surfaced. These include the challenge of quality assessment of systematic reviews, search validation and the additional risk of double counting. == Examples of Tertiary reviews in software engineering literature == Test quality Machine Learning Test-driven development
The Best Free AI Paraphrasing Tool for Beginners
Trying to pick the best AI paraphrasing tool? An AI paraphrasing tool is software that uses machine learning to help you get more done — it scales effortlessly from a single task to thousands. The best picks balance beginner-friendly simplicity with the depth power users need, and they ship updates often. Whether you are a beginner or a pro, the right AI paraphrasing tool slots into your workflow and pays for itself fast. Read on for hands-on impressions, pricing tiers, and the standout features that matter.
Ashutosh Saxena
Ashutosh Saxena is an Indian-American computer scientist, researcher, and entrepreneur known for his contributions to the field of artificial intelligence and large-scale robot learning. His interests include building enterprise AI agents and embodied AI. Saxena is the co-founder and CEO of Caspar.AI, where generative AI parses data from ambient 3D radar sensors to predict 20+ health & wellness markers for pro-active patient care. Prior to Caspar.AI, Ashutosh co-founded Cognical Katapult (NSDQ: KPLT), which provides a no credit required alternative to traditional financing for online and omni-channel retail. Before Katapult, Saxena was an assistant professor in the Computer Science Department and faculty director of the RoboBrain Project (a large-scale AI model for robotics) at Cornell University. == Education == In 2009, with artificial intelligence pioneer Andrew Ng as his advisor, Saxena received both his M.S. and Ph.D. in computer science with an emphasis on artificial intelligence from Stanford University. Saxena received his bachelor's degree in electrical engineering from the Indian Institute of Technology, Kanpur in 2004. == Career == Saxena was the chief scientist of New York-based Holopad, where he worked with Steven Spielberg's team to create walkthroughs and 3D experiences for his movie TinTin. His past experiences include building acoustic AI models at Bose Corporation. Once Ashutosh completed his undergraduate degree, he became a researcher at the Commonwealth Scientific and Industrial Research Organization, where he developed AI models for medical devices. Before Caspar, Saxena pursued other entrepreneurial ventures, such as ZunaVision, an artificial intelligence startup he co-founded with Andrew Ng that uses AI to embed advertising space within videos. Ashutosh served as the CTO of ZunaVision from 2008 to 2010. After ZunaVision, Saxena co-founded Cognical Katapult, which provided financing solutions to nonprime and underbanked consumers powered by artificial intelligence. From 2014 to 2016, Saxena served as the faculty director of the RoboBrain project, which was a joint venture that he started between Stanford University, Cornell University, Brown University, and the University of California, Berkeley that made a knowledge engine for robots. Saxena co-founded Brain of Things in 2015 with David Cheriton, who serves as chief scientist, and was listed as the fastest growing private company reaching an annual recurring revenue of $8 million in three years. It has been widely covered in several outlets including Forbes Japan, and MIT Technology Review. Saxena's work on deep learning won test of time award in 2023 by Robotics Science and Systems. Ashutosh has been recognized for his work by receiving the Alfred P. Sloan Fellow in 2011, Google Faculty Research Award in 2012, Microsoft Faculty Fellowship in 2012, NSF Career award in 2013, One of the Eight Innovators to Watch by the Smithsonian Institution in 2015, and received TR35 Innovator Award by MIT Technology Review in 2018. He was named by San Francisco Business Times as a 40 under 40 young business leader. == Research == Saxena has authored over 100 published papers in the areas of large-scale robot learning and artificial intelligence, with 20,000+ citations. His work in the fields of computer vision and deep learning have been featured in press releases and academic journal reviews. Ashutosh's early work includes the Stanford Artificial Intelligence Robot (STAIR), an AI models that enables to perform tasks such as unload items from a dishwasher, which was covered on the front-page of New York Times. His work on Make3D, was the first work that estimated 3D depth from a single still image. At Cornell University, Ashutosh led the Robot Learning Lab, which used a machine learning approach to train robots to perform tasks in human environments such as generalizing manipulation in 3D point-clouds where robots learn to transfer manipulation trajectories to novel objects utilizing a large sample of demonstrations from crowdsourcing.
RAMnets
RAMnets is one of the oldest practical neurally inspired classification algorithms. The RAMnets is also known as a type of "n-tuple recognition method" or "weightless neural network". == Algorithm == Consider (let us say N) sets of n distinct bit locations are selected randomly. These are the n-tuples. The restriction of a pattern to an n-tuple can be regarded as an n-bit number which, together with the identity of the n-tuple, constitutes a `feature' of the pattern. The standard n-tuple recognizer operates simply as follows: A pattern is classified as belonging to the class for which it has the most features in common with at least one training pattern of that class. This is the Θ {\displaystyle \Theta } = 0 case of a more general rule whereby the class assigned to unclassified pattern u is a c r g m a x ( ∑ i = 1 N Θ ( ∑ v ∈ D c δ ( α i ( u ) , α i ( v ) ) ) ) {\displaystyle {\begin{aligned}{\underset {c}{a}}rgmax(\sum _{i=1}^{N}\Theta (\sum _{v\in D_{c}}\delta (\alpha _{i}(u),\alpha _{i}(v))))\end{aligned}}} where Dc is the set of training patterns in class c, Θ ( x ) {\displaystyle \Theta (x)} = x for 0 ≤ x ≤ θ {\displaystyle 0\leq x\leq \theta } , Θ ( x ) = θ {\displaystyle \Theta (x)=\theta } for x ≥ θ {\displaystyle x\geq \theta } , δ i , j {\displaystyle \delta _{i,j}} is the Kronecker delta( δ i , j {\displaystyle \delta _{i,j}} =1 if i=j and 0 otherwise.)and ( α i ( u ) ) {\displaystyle (\alpha _{i}(u))} is the ith feature of the pattern u: ∑ j = 0 n − 1 u η i ( j ) 2 j {\displaystyle \sum _{j=0}^{n-1}u_{\eta }i(j)2^{j}} Here uk is the kth bit of u and u η i ( j ) {\displaystyle u_{\eta }i(j)} is the jth bit location of the ith n-tuple. With C classes to distinguish, the system can be implemented as a network of NC nodes, each of which is a random access memory (RAM); hence the term RAMnet. The memory content m c i α {\displaystyle m_{ci\alpha }} at address α {\displaystyle \alpha } of the ith node allocated to class c is set to m c i α {\displaystyle m_{ci\alpha }} = Θ ( ∑ v ∈ D c δ ( α , α i ( v ) ) ) {\displaystyle \Theta (\sum _{v\in D_{c}}\delta (\alpha ,\alpha _{i}(v)))} In the usual θ {\displaystyle \theta } = 1 case, the 1-bit content of m c i α {\displaystyle m_{ci\alpha }} is set if any pattern of Dc has feature α {\displaystyle \alpha } and unset otherwise. Recognition is accomplished by summing the contents of the nodes of each class at the addresses given by the features of the unclassified pattern. That is, pattern u is assigned to class a c r g m a x ( ∑ i = 1 N m c i α ( u ) ) {\displaystyle {\begin{aligned}{\underset {c}{a}}rgmax(\sum _{i=1}^{N}m_{ci\alpha }(u))\end{aligned}}} == RAM-discriminators and WiSARD == The RAMnets formed the basis of a commercial product known as WiSARD (Wilkie, Stonham and Aleksander Recognition Device) was the first artificial neural network machine to be patented. A RAM-discriminator consists of a set of X one-bit word RAMs with n inputs and a summing device (Σ). Any such RAM-discriminator can receive a binary pattern of X⋅n bits as input. The RAM input lines are connected to the input pattern by means of a biunivocal pseudo-random mapping. The summing device enables this network of RAMs to exhibit – just like other ANN models based on synaptic weights – generalization and noise tolerance. In order to train the discriminator one has to set all RAM memory locations to 0 and choose a training set formed by binary patterns of X⋅n bits. For each training pattern, a 1 is stored in the memory location of each RAM addressed by this input pattern. Once the training of patterns is completed, RAM memory contents will be set to a certain number of 0's and 1's. The information stored by the RAM during the training phase is used to deal with previous unseen patterns. When one of these is given as input, the RAM memory contents addressed by the input pattern are read and summed by Σ. The number r thus obtained, which is called the discriminator response, is equal to the number of RAMs that output 1. r reaches the maximum X if the input belongs to the training set. r is equal to 0 if no n-bit component of the input pattern appears in the training set (not a single RAM outputs 1). Intermediate values of r express a kind of “similarity measure” of the input pattern with respect to the patterns in the training set. A system formed by various RAM-discriminators is called WiSARD. Each RAM-discriminator is trained on a particular class of patterns, and classification by the multi-discriminator system is performed in the following way. When a pattern is given as input, each RAM-discriminator gives a response to that input. The various responses are evaluated by an algorithm which compares them and computes the relative confidence c of the highest response (e.g., the difference d between the highest response and the second highest response, divided by the highest response). A schematic representation of a RAM-discriminator and a 10 RAM-discriminator WiSARD is shown in Figure 1.
Concordancer
A concordancer is a computer program that automatically constructs a concordance—an alphabetised index of every occurrence of a word or phrase in a body of text, each entry displayed with its surrounding context. Concordancers are primary tools in corpus linguistics, lexicography, computer-assisted translation, and language teaching. The most common display format is the key word in context (KWIC) layout, in which each hit appears centred on a line with a fixed span of words to its left and right, enabling rapid scanning of usage patterns across many occurrences. == History == === Pre-computational concordances === The compilation of concordances predates computers by many centuries. Around 1230, the French Dominican cardinal Hugh of Saint-Cher directed a team of friars in assembling a concordance of the Latin Vulgate Bible, generally regarded as the first systematic concordance of any text. To help readers locate passages, Hugh divided each biblical chapter into lettered sections. Later milestones include a Hebrew Old Testament concordance compiled by Rabbi Mordecai Nathan (1448), Alexander Cruden's Complete Concordance to the Holy Scriptures (1737), and the manuscript Asaf ha-Mazkir, an unfinished concordance to the Babylonian Talmud compiled by Moses Rigotz around the turn of the 19th century. === First computer concordance === The first concordance produced with computing assistance was the Index Thomisticus, a comprehensive lexical index of the writings of and around Thomas Aquinas, totalling approximately 10.6 million Latin words. The Italian Jesuit priest Roberto Busa conceived the project in 1946 and secured the sponsorship of IBM in 1949 after a meeting with chairman Thomas J. Watson. Keypunch operators in Gallarate, Italy, encoded the texts onto punched cards from around 1950. IBM executive Paul Tasman developed the processing methods. The full 56-volume printed edition was completed around 1980, followed by a CD-ROM edition in 1989 and a web-accessible version in 2005. === The KWIC format === The key word in context (KWIC) display was formalised as a computational technique by Hans Peter Luhn, a researcher at IBM, in a 1960 paper in American Documentation. In KWIC output, each instance of the search term (the node word) is centred on a line with a fixed window of words to each side; sorting the resulting lines alphabetically by the immediately adjacent word reveals collocational and phraseological patterns at a glance. === COCOA === One of the first dedicated concordancing programs was COCOA (COunt and COncordance Generation on Atlas), created in 1965 by D. B. Russell at University College London and the Atlas Computer Laboratory in Harwell, Oxfordshire. Written in approximately 4,000 cards of FORTRAN, it processed text annotated with flat, non-hierarchical markup tags and could produce word counts and concordances in multiple languages. Within its first six months COCOA had been applied to texts in at least six languages. A second version designed for multiple mainframe platforms was distributed to British computing centres in the mid-1970s. Growing dissatisfaction with its interface and the eventual withdrawal of Atlas Laboratory support prompted British funding bodies to commission a successor program. === Oxford Concordance Program === The Oxford Concordance Program (OCP) was designed and written in FORTRAN by Susan Hockey and Ian Marriott at Oxford University Computing Services (OUCS) between 1979 and 1980 and first released in 1981. Hockey and Marriott acknowledged that OCP owed much to COCOA and the CLOC system at the University of Birmingham. OCP accepted COCOA-format markup to encode metadata such as author, act, scene, and line number, and was described by its authors as "a machine-independent text analysis program for producing word lists, indices and concordances in a variety of languages and alphabets." By the mid-1980s it had been licensed to approximately 240 institutions in 23 countries. A personal computer version, Micro-OCP, was developed for the IBM PC and sold by Oxford University Press from the late 1980s. Version 2 was rewritten in 1985–86 and documented in the same 1987 article by Hockey and co-author John Martin. === Personal computer era === The availability of affordable personal computers in the 1980s and 1990s enabled standalone concordancing applications that analysts could run locally without specialist computing facilities. MicroConcord, developed by Mike Scott and Tim Johns and published by Oxford University Press in 1993 for MS-DOS, was among the first concordancers designed specifically for classroom language teaching. WordSmith Tools, also developed by Mike Scott, was first released in 1996 and became one of the most widely used corpus analysis suites in academic linguistics research. Other tools from this era include TACT (University of Toronto, 1989), a suite of MS-DOS freeware programs for literary text analysis, and MonoConc, a Windows concordancer created by Michael Barlow. === Web-based concordancers === From the late 1990s onwards, web-based concordancers hosted on remote servers gave researchers browser access to large preloaded corpora without requiring local storage or processing. The Sketch Engine, developed by Adam Kilgarriff and Pavel Rychlý (Masaryk University), was launched commercially in July 2003 by Lexical Computing Limited and introduced word sketches—automatically generated one-page profiles of a word's typical grammatical relations and collocations. AntConc, created by Laurence Anthony at Waseda University, Tokyo, was first released in 2002 as freeware for Windows, macOS, and Linux. == Features == Modern concordancers typically offer a range of analytical functions beyond basic KWIC display. These commonly include: KWIC display with the node word centred and context words in aligned columns, sortable by the word one, two, or three positions to the left or right of the node (L1–L3 and R1–R3) Concordance plots, visualising the distribution of hits as marks along a scaled bar representing each text in the corpus Frequency and word lists, both alphabetical and ranked by frequency Collocation statistics, identifying words that co-occur with the search term more often than chance, quantified by measures such as mutual information, the t-score, or log-likelihood Keyword analysis, comparing word frequencies between a study corpus and a reference corpus to identify statistically distinctive items N-gram analysis, finding frequently recurring word sequences of a specified length Part-of-speech tagging integration, allowing searches filtered to particular grammatical categories Unicode support for multilingual text Bilingual and parallel concordancers additionally display aligned text in two or more languages side by side, enabling comparison of translation equivalents across language pairs. == Notable concordancers == === WordSmith Tools === Created by Mike Scott and first released in 1996, WordSmith Tools is a Windows corpus analysis suite that evolved from MicroConcord. Its three core modules are Concord (KWIC concordances), WordList (frequency and alphabetical word lists), and Keywords (statistical keyword identification relative to a reference corpus). Oxford University Press used WordSmith Tools for dictionary preparation work. Version 4.0 is freely available; later versions are sold by Lexical Analysis Software Limited. === AntConc === AntConc is a freeware, multiplatform concordancing toolkit created by Laurence Anthony, Professor of Applied Linguistics at Waseda University, Tokyo. First released in 2002 and formally described in a 2005 academic paper, it runs on Windows, macOS, and Linux. Its tools include a KWIC concordancer, a concordance plot for visualising distribution across texts, a collocates tool, a keyword list, and an n-gram analysis module. Because it is free and requires only plain text files, AntConc is widely used in linguistics courses and independent research worldwide. === Sketch Engine === The Sketch Engine is a corpus management and query system co-created by Adam Kilgarriff and Pavel Rychlý and launched in 2003 by Lexical Computing Limited. It provides browser-based access to over 800 corpora in more than 100 languages. Beyond concordance searching, it offers word sketches, collocation analysis, distributional thesaurus construction, keyword and terminology extraction, and diachronic analysis. It is used by major publishers including Macmillan and Oxford University Press for lexicographic research. A subset tool, SKELL (Sketch Engine for Language Learning), is freely accessible to individual learners. === Wmatrix === Wmatrix is a web-based corpus processing environment developed by Paul Rayson at the University Centre for Computer Corpus Research on Language (UCREL), Lancaster University. Alongside concordances and frequency lists, Wmatrix integrates CLAWS part-of-speech tagging and the USAS semantic tagger, enabling keyword analysis simultane
Tagged Deterministic Finite Automaton
In the automata theory, a tagged deterministic finite automaton (TDFA) is an extension of deterministic finite automaton (DFA). In addition to solving the recognition problem for regular languages, TDFA is also capable of submatch extraction and parsing. While canonical DFA can find out if a string belongs to the language defined by a regular expression, TDFA can also extract substrings that match specific subexpressions. More generally, TDFA can identify positions in the input string that match tagged positions in a regular expression (tags are meta-symbols similar to capturing parentheses, but without the pairing requirement). == History == TDFA were first described by Ville Laurikari in 2000. Prior to that it was unknown whether it is possible to perform submatch extraction in one pass on a deterministic finite-state automaton, so this paper was an important advancement. Laurikari described TDFA construction and gave a proof that the determinization process terminates, however the algorithm did not handle disambiguation correctly. In 2007 Chris Kuklewicz implemented TDFA in a Haskell library Regex-TDFA with POSIX longest-match semantics. Kuklewicz gave an informal description of the algorithm and answered the principal question whether TDFA are capable of POSIX longest-match disambiguation, which was doubted by other researchers. In 2017 Ulya Trafimovich described TDFA with one-symbol lookahead. The use of a lookahead symbol reduces the number of registers and register operations in a TDFA, which makes it faster and often smaller than Laurikari TDFA. Trafimovich called TDFA variants with and without lookahead TDFA(1) and TDFA(0) by analogy with LR parsers LR(1) and LR(0). The algorithm was implemented in the open-source lexer generator RE2C. Trafimovich formalized Kuklewicz disambiguation algorithm. In 2018 Angelo Borsotti worked on an experimental Java implementation of TDFA; it was published later in 2021. In 2019 Borsotti and Trafimovich adapted POSIX disambiguation algorithm by Okui and Suzuki to TDFA. They gave a formal proof of correctness of the new algorithm and showed that it is faster than Kuklewicz algorithm in practice. In 2020 Trafimovich published an article about TDFA implementation in RE2C. In 2022 Borsotti and Trafimovich published a paper with a detailed description of TDFA construction. The paper incorporated their past research and presented multi-pass TDFA that are better suited to just-in-time determinization. They also compared TDFA against other algorithms and provided benchmarks. == Formal definition == TDFA have the same basic structure as ordinary DFA: a finite set of states linked by transitions. In addition to that, TDFA have a fixed set of registers that hold tag values, and register operations on transitions that set or copy register values. The values may be scalar offsets, or offset lists for tags that match repeatedly (the latter can be represented efficiently using a trie structure). There is no one-to-one mapping between tags in a regular expression and registers in a TDFA: a single tag may need many registers, and the same register may hold values of different tags. The following definition is according to Trafimovich and Borsotti. The original definition by Laurikari is slightly different. A tagged deterministic finite automaton F {\displaystyle F} is a tuple ( Σ , T , S , S f , s 0 , R , R f , δ , φ ) {\displaystyle (\Sigma ,T,S,S_{f},s_{0},R,R_{f},\delta ,\varphi )} , where: Σ {\displaystyle \Sigma } is a finite set of symbols (alphabet) T {\displaystyle T} is a finite set of tags S {\displaystyle S} is a finite set of states with initial state s 0 {\displaystyle s_{0}} and a subset of final states S f ⊆ S {\displaystyle S_{f}\subseteq S} R {\displaystyle R} is a finite set of registers with a subset of final registers R f {\displaystyle R_{f}} (one per tag) δ : S × Σ → S × O ∗ {\displaystyle \delta :S\times \Sigma \rightarrow S\times O^{}} is a transition function φ : S f → O ∗ {\displaystyle \varphi :S_{f}\rightarrow O^{}} is a final function, where O {\displaystyle O} is a set of register operations of the following types: set register i {\displaystyle i} to nil or to the current position: i ← v {\displaystyle i\leftarrow v} , where v ∈ { n , p } {\displaystyle v\in \{\mathbf {n} ,\mathbf {p} \}} copy register j {\displaystyle j} to register i {\displaystyle i} : i ← j {\displaystyle i\leftarrow j} copy register j {\displaystyle j} to register i {\displaystyle i} and append history: i ← j ⋅ h {\displaystyle i\leftarrow j\cdot h} , where h {\displaystyle h} is a string over { n , p } {\displaystyle \{\mathbf {n} ,\mathbf {p} \}} === Example === Figure 0 shows an example TDFA for regular expression ( 1 a 2 ) ∗ 3 ( a | 4 b ) 5 b ∗ {\displaystyle (1a2)^{}3(a|4b)5b^{}} with alphabet Σ = { a , b } {\displaystyle \Sigma =\{a,b\}} and a set of tags T = { 1 , 2 , 3 , 4 , 5 } {\displaystyle T=\{1,2,3,4,5\}} that matches strings of the form a … a b … b {\displaystyle a\dots ab\dots b} with at least one symbol. TDFA has four states S = { 0 , 1 , 2 , 3 } {\displaystyle S=\{0,1,2,3\}} three of which are final S f = { 1 , 2 , 3 } {\displaystyle S_{f}=\{1,2,3\}} . The set of registers is R = { r 1 , r 2 , r 3 , r 4 , r 5 } {\displaystyle R=\{r_{1},r_{2},r_{3},r_{4},r_{5}\}} with a subset of final registers R f = { r 1 , r 2 , r 3 , r 4 , r 5 } {\displaystyle R_{f}=\{r_{1},r_{2},r_{3},r_{4},r_{5}\}} where register r i {\displaystyle r_{i}} corresponds to i {\displaystyle i} -th tag. Transitions have operations defined by the δ {\displaystyle \delta } function, and final states have operations defined by the φ {\displaystyle \varphi } function (marked with wide-tipped arrow). For example, to match string a a b {\displaystyle aab} , one starts in state 0, matches the first a {\displaystyle a} and moves to state 1 (setting registers r 1 , r 2 {\displaystyle r_{1},r_{2}} to undefined and r 3 {\displaystyle r_{3}} to the current position 0), matches the second a {\displaystyle a} and loops to state 1 (register values are now r 1 = 0 , r 2 = r 3 = 1 {\displaystyle r_{1}=0,r_{2}=r_{3}=1} ), matches b {\displaystyle b} and moves to state 2 (register values are now r 1 = 1 , r 2 = r 3 = r 4 = 2 {\displaystyle r_{1}=1,r_{2}=r_{3}=r_{4}=2} ), executes the final operations in state 2 (register values are now r 1 = 1 , r 2 = r 3 = r 4 = 2 , r 5 = 3 {\displaystyle r_{1}=1,r_{2}=r_{3}=r_{4}=2,r_{5}=3} ) and finally exits TDFA. == Complexity == Canonical DFA solve the recognition problem in linear time. The same holds for TDFA, since the number of registers and register operations is fixed and depends only on the regular expression, but not on the length of input. The overhead on submatch extraction depends on tag density in a regular expression and nondeterminism degree of each tag (the maximum number of registers needed to track all possible values of the tag in a single TDFA state). On one extreme, if there are no tags, a TDFA is identical to a canonical DFA. On the other extreme, if every subexpression is tagged, a TDFA effectively performs full parsing and has many operations on every transition. In practice for real-world regular expressions with a few submatch groups the overhead is negligible compared to matching with canonical DFA. == TDFA construction == TDFA construction is performed in a few steps. First, a regular expression is converted to a tagged nondeterministic finite automaton (TNFA). Second, a TNFA is converted to a TDFA using a determinization procedure; this step also includes disambiguation that resolves conflicts between ambiguous TNFA paths. After that, a TDFA can optionally go through a number of optimizations that reduce the number of registers and operations, including minimization that reduces the number of states. Algorithms for all steps of TDFA construction with pseudocode are given in the paper by Borsotti and Trafimovich. This section explains TDFA construction on the example of a regular expression a ∗ t b ∗ | a b {\displaystyle a^{}tb^{}|ab} , where t {\displaystyle t} is a tag and { a , b } {\displaystyle \{a,b\}} are alphabet symbols. === Tagged NFA === TNFA is a nondeterministic finite automaton with tagged ε-transitions. It was first described by Laurikari, although similar constructions were known much earlier as Mealy machines and nondeterministic finite-state transducers. TNFA construction is very similar to Thompson's construction: it mirrors the structure of a regular expression. Importantly, TNFA preserves ambiguity in a regular expression: if it is possible to match a string in two different ways, then TNFA for this regular expression has two different accepting paths for this string. TNFA definition by Borsotti and Trafimovich differs from the original one by Laurikari in that TNFA can have negative tags on transitions: they are needed to make the absence of match explicit in cases when there is a bypass for a tagged transition. Figure 1 shows TNFA for the example regu