Eimear E. Kenny is a researcher in population genetics and translation genomics, and is the Founding Director of the Institute for Genomic Health, and Endowed Chair and Professor of Genomic Health at the Icahn School of Medicine at Mount Sinai. She is known for novel approaches in computational genomics, advancing the study of human genetic variation and its connection to disease risk and diagnosis. Her research has laid the foundation for integrating artificial intelligence (AI) and genomics into precision medicine and routine clinical care. By combining genomics, computer science, and medicine, her work leverages genomic sequencing technologies and machine learning algorithms to uncover insights that improve patient care, accelerate genomic data analysis, and enable the future of AI-driven healthcare. She has led multiple genomics-based clinical trials, applying computational biology and AI in clinical settings to advance genomic medicine and precision healthcare. == Research == A recipient of the Early-Career Award from the American Society of Human Genetics (USA), Kenny, as of 2024, leads a team in genetics, computer science, and medicine, focusing on genetic ancestry, large-scale genomics, clinical trials, and genomic medicine at the Institute for Genomic Health. The lab works to advance understanding of genetic ancestry and its impact on health in order to inform better clinical medicine models. She is recognized for her work to leverage biobanks for translational genomics and her development of new genetic tests an strategies for health care management. In one study, she and her colleagues investigated genetic disorders that might be under-diagnosed due to insufficient data, and found a variant in a collagen gene associated with Steel syndrome. This syndrome caused short stature and bone and joint issues and was thought to be rare. However, the study revealed it is common in individuals with Puerto Rican ancestry. Three of Kenny's genomic medicine clinical trials assessed how to bring new technology, such as digital apps, or information, such as polygenic risk scores, into routine clinical care. In the 2010s, Kenny was instrumental in several large-scale sequencing studies, including the 1000 Genomes Project, the Exome Sequencing Project, the Genome Sequencing Project, and the Trans-Omics for Precision Medicine. In 2012, she led work that discovered the variant responsible for blond hair in Melanesia, work that was featured in the Smithsonian NHGRI Human Genome Exhibit in Washington, D.C. In 2017, her group was one of the first to demonstrate that polygenic risk scores derived in predominantly European populations have reduced accuracy when applied in populations now widely acknowledged as a major challenge in the field of genomic risk prediction. As of 2024, she is Principal Investigator in many NIH-funded international consortium focused on computational genomics and genomic medicine, including Electronic Medical Records and Genomics, Polygenic Risk Methods in Diverse Populations, and the Human Pangenome Reference Consortium. In 2023, Kenny played a key role in a groundbreaking advancement in genomics research by helping to map a diverse human pangenome—a major shift from reliance on a single reference genome. Unlike the earlier genetic map, based on one man of mixed European and African ancestry in Buffalo, this new pangenome project captures far greater human genetic diversity. As reported by The Washington Post, Kenny's work demonstrates how a more inclusive human genome can drive discoveries in rare genetic diseases, improve genomic medicine, and accelerate the future of precision healthcare. Kenny was co-developer and current license holder for Random Forest adMIXture (RFMix), a patented software for inferring continental and sub-continental ancestry at genomic loci. == Education and career == Kenny graduated from Trinity College Dublin with a BA in Biochemistry in 1999 and did a masters in Bioinformatics at Leeds University. She received her PhD in Computational Genomics at Rockefeller University, and did her post-doctoral work in the lab of Dr. Carlos D. Bustamante at Stanford University. === Academic appointments === As of 2024, at Mount Sinai, she serves as the Endowed Chair and Professor of Genomic Health, Professor at the Department of Medicine and Professor at the Department of Genetics and Genomic Sciences. Since 2018 she has served as the Founding Director of the Institute for Genomic Health, and since 2022, she also serves as the Founding Director of the Center for Translational Genomics. She is also the Director of Translational Research, Division for Genomic Medicine. Former appointments include Assistant Professor at the Department of Genetics and Genomic Sciences and Member at The Charles Bronfman Institute of Personalized Medicine, both at Mount Sinai. She was also Bioinformatics Programmer at the California Institute of Technology, and research assistant at the Massachusetts Institute of Technology. == Publications == As of 2024, Kenny is an advisor to Cell Genomics. Google Scholar reports 50,623 citations, an h-index of 66 and an i10-index of 130. The five most-cited articles she contributed to are: Auton, A; Brooks, LD; Durbin, RM; Garrison, EP; Kang, HM; Korbel, JO; Marchini, JL; McCarthy, S; McVean, GA; Abecasis, GR (2015). "A global reference for human genetic variation". Nature. 526 (7571): 68–74. Bibcode:2015Natur.526...68T. doi:10.1038/nature15393. PMC 4750478. PMID 26432245.. Cited by 14847 Abecasis, GR; Auton, A; Brooks, LD; DePristo, MA; Durbin, RM; Handsaker, RE; Kang, HM; Marth, GT; McVean, GA (2012). "An integrated map of genetic variation from 1,092 human genomes". Nature. 491 (7422): 56–65. Bibcode:2012Natur.491...56T. doi:10.1038/nature11632. PMC 3498066. PMID 23128226.. Cited by 8287 Jacob A. Tennessen et al. Evolution and Functional Impact of Rare Coding Variation from Deep Sequencing of Human Exomes.Science337,64–69(2012).DOI:10.1126/science.1219240 Cited by 1886 Taliun, D.; Harris, D.N.; Kessler, M.D.; et al. (2021). "Sequencing of 53,831 diverse genomes from the NHLBI TOPMed Program". Nature. 590 (7845): 290–299. Bibcode:2021Natur.590..290T. doi:10.1038/s41586-021-03205-y. PMC 7875770. PMID 33568819.. Cited by 1369 Vilhjálmsson, BJ; et al. (2015). "Modeling Linkage Disequilibrium Increases Accuracy of Polygenic Risk Scores". Am J Hum Genet. 97 (4): 576–92. doi:10.1016/j.ajhg.2015.09.001. PMC 4596916. PMID 26430803.. Cited by 1327
Situated approach (artificial intelligence)
In artificial intelligence research, the situated approach builds agents that are designed to behave effectively successfully in their environment. This requires designing AI "from the bottom-up" by focussing on the basic perceptual and motor skills required to survive. The situated approach gives a much lower priority to abstract reasoning or problem-solving skills. The approach was originally proposed as an alternative to traditional approaches (that is, approaches popular before 1985 or so). After several decades, classical AI technologies started to face intractable issues (e.g. combinatorial explosion) when confronted with real-world modeling problems. All approaches to address these issues focus on modeling intelligences situated in an environment. They have become known as the situated approach to AI. == Emergence of a concept == === From traditional AI to Nouvelle AI === During the late 1980s, the approach now known as Nouvelle AI (Nouvelle means new in French) was pioneered at the MIT Artificial Intelligence Laboratory by Rodney Brooks. As opposed to classical or traditional artificial intelligence, Nouvelle AI purposely avoided the traditional goal of modeling human-level performance, but rather tries to create systems with intelligence at the level of insects, closer to real-world robots. But eventually, at least at MIT new AI did lead to an attempt for humanoid AI in the Cog Project. === From Nouvelle AI to behavior-based and situated AI === The conceptual shift introduced by nouvelle AI flourished in the robotics area, given way to behavior-based robotics (BBR), a methodology for developing AI based on a modular decomposition of intelligence. It was made famous by Rodney Brooks: his subsumption architecture was one of the earliest attempts to describe a mechanism for developing BBAI. It is extremely popular in robotics and to a lesser extent to implement intelligent virtual agents because it allows the successful creation of real-time dynamic systems that can run in complex environments. For example, it underlies the intelligence of the Sony Aibo and many RoboCup robot teams. Realizing that in fact all these approaches were aiming at building not an abstract intelligence, but rather an intelligence situated in a given environment, they have come to be known as the situated approach. In fact, this approach stems out from early insights of Alan Turing, describing the need to build machines equipped with sense organs to learn directly from the real-world instead of focusing on abstract activities, such as playing chess. == Definitions == Classically, a software entity is defined as a simulated element, able to act on itself and on its environment, and which has an internal representation of itself and of the outside world. An entity can communicate with other entities, and its behavior is the consequence of its perceptions, its representations, and its interactions with the other entities. === AI loop === Simulating entities in a virtual environment requires simulating the entire process that goes from a perception of the environment, or more generally from a stimulus, to an action on the environment. This process is called the AI loop and technology used to simulate it can be subdivided in two categories. Sensorimotor or low-level AI deals with either the perception problem (what is perceived?) or the animation problem (how are actions executed?). Decisional or high-level AI deals with the action selection problem (what is the most appropriate action in response to a given perception, i.e. what is the most appropriate behavior?). === Traditional or symbolic AI === There are two main approaches in decisional AI. The vast majority of the technologies available on the market, such as planning algorithms, finite-state machines (FSA), or expert systems, are based on the traditional or symbolic AI approach. Its main characteristics are: It is top-down: it subdivides, in a recursive manner, a given problem into a series of sub-problems that are supposedly easier to solve. It is knowledge-based: it relies on a symbolic description of the world, such as a set of rules. However, the limits of traditional AI, which goal is to build systems that mimic human intelligence, are well-known: inevitably, a combinatorial explosion of the number of rules occurs due to the complexity of the environment. In fact, it is impossible to predict all the situations that will be encountered by an autonomous entity. === Situated or behavioral AI === In order to address these issues, another approach to decisional AI, also known as situated or behavioral AI, has been proposed. It does not attempt to model systems that produce deductive reasoning processes, but rather systems that behave realistically in their environment. The main characteristics of this approach are the following: It is bottom-up: it relies on elementary behaviors, which can be combined to implement more complex behaviors. It is behavior-based: it does not rely on a symbolic description of the environment, but rather on a model of the interactions of the entities with their environment. The goal of situated AI is to model entities that are autonomous in their environment. This is achieved thanks to both the intrinsic robustness of the control architecture, and its adaptation capabilities to unforeseen situations. === Situated agents === In artificial intelligence and cognitive science, the term situated refers to an agent which is embedded in an environment. The term situated is commonly used to refer to robots, but some researchers argue that software agents can also be situated if: they exist in a dynamic (rapidly changing) environment, which they can manipulate or change through their actions, and which they can sense or perceive. Examples might include web-based agents, which can alter data or trigger processes (such as purchases) over the Internet, or virtual-reality bots which inhabit and change virtual worlds, such as Second Life. Being situated is generally considered to be part of being embodied, but it is useful to consider each perspective individually. The situated perspective emphasizes that intelligent behavior derives from the environment and the agent's interactions with it. The nature of these interactions are defined by an agent's embodiment. == Implementation principles == === Modular decomposition === The most important attribute of a system driven by situated AI is that the intelligence is controlled by a set of independent semi-autonomous modules. In the original systems, each module was actually a separate device or was at least conceived of as running on its own processing thread. Generally, though, the modules are just abstractions. In this respect, situated AI may be seen as a software engineering approach to AI, perhaps akin to object oriented design. Situated AI is often associated with reactive planning, but the two are not synonymous. Brooks advocated an extreme version of cognitive minimalism which required initially that the behavior modules were finite-state machines and thus contained no conventional memory or learning. This is associated with reactive AI because reactive AI requires reacting to the current state of the world, not to an agent's memory or preconception of that world. However, learning is obviously key to realistic strong AI, so this constraint has been relaxed, though not entirely abandoned. === Action selection mechanism === The situated AI community has presented several solutions to modeling decision-making processes, also known as action selection mechanisms. The first attempt to solve this problem goes back to subsumption architectures, which were in fact more an implementation technique than an algorithm. However, this attempt paved the way to several others, in particular the free-flow hierarchies and activation networks. A comparison of the structure and performances of these two mechanisms demonstrated the advantage of using free-flow hierarchies in solving the action selection problem. However, motor schemas and process description languages are two other approaches that have been used with success for autonomous robots. == Notes and references == Arsenio, Artur M. (2004) Towards an embodied and situated AI, In: Proceedings of the International FLAIRS conference, 2004. (online) The Artificial Life Route To Artificial Intelligence: Building Embodied, Situated Agents, Luc Steels and Rodney Brooks Eds., Lawrence Erlbaum Publishing, 1995. (ISBN 978-0805815184) Rodney A. Brooks Cambrian Intelligence (MIT Press, 1999) ISBN 0-262-52263-2; collection of early papers including "Intelligence without representation" and "Intelligence without reason", from 1986 & 1991 respectively. Ronald C. Arkin Behavior-Based Robotics (MIT Press, 1998) ISBN 0-262-01165-4 Hendriks-Jansen, Horst (1996) Catching Ourselves in the Act: Situated Activity, Interactive Emergence, Evolution, and Human Thought. Cambridge, Mass.: MIT Press.
Ciphertext
In cryptography, ciphertext or cyphertext is the result of encryption performed on plaintext using an algorithm, called a cipher. Ciphertext is also known as encrypted or encoded information because it contains a form of the original plaintext that is unreadable by a human or computer without the proper cipher to decrypt it. This process prevents the loss of sensitive information via hacking. Decryption, the inverse of encryption, is the process of turning ciphertext into readable plaintext. Ciphertext is not to be confused with codetext, because the latter is a result of a code, not a cipher. == Conceptual underpinnings == Let m {\displaystyle m\!} be the plaintext message that Alice wants to secretly transmit to Bob and let E k {\displaystyle E_{k}\!} be the encryption cipher, where k {\displaystyle _{k}\!} is a cryptographic key. Alice must first transform the plaintext into ciphertext, c {\displaystyle c\!} , in order to securely send the message to Bob, as follows: c = E k ( m ) . {\displaystyle c=E_{k}(m).\!} In a symmetric-key system, Bob knows Alice's encryption key. Once the message is encrypted, Alice can safely transmit it to Bob (assuming no one else knows the key). In order to read Alice's message, Bob must decrypt the ciphertext using E k − 1 {\displaystyle {E_{k}}^{-1}\!} which is known as the decryption cipher, D k : {\displaystyle D_{k}:\!} D k ( c ) = D k ( E k ( m ) ) = m . {\displaystyle D_{k}(c)=D_{k}(E_{k}(m))=m.\!} Alternatively, in a non-symmetric key system, everyone, not just Alice and Bob, knows the encryption key; but the decryption key cannot be inferred from the encryption key. Only Bob knows the decryption key D k , {\displaystyle D_{k},} and decryption proceeds as D k ( c ) = m . {\displaystyle D_{k}(c)=m.} == Types of ciphers == The history of cryptography began thousands of years ago. Cryptography uses a variety of different types of encryption. Earlier algorithms were performed by hand and are substantially different from modern algorithms, which are generally executed by a machine. === Historical ciphers === Historical pen and paper ciphers used in the past are sometimes known as classical ciphers. They include: Substitution cipher: the units of plaintext are replaced with ciphertext (e.g., Caesar cipher and one-time pad) Polyalphabetic substitution cipher: a substitution cipher using multiple substitution alphabets (e.g., Vigenère cipher and Enigma machine) Polygraphic substitution cipher: the unit of substitution is a sequence of two or more letters rather than just one (e.g., Playfair cipher) Transposition cipher: the ciphertext is a permutation of the plaintext (e.g., rail fence cipher) Historical ciphers are not generally used as a standalone encryption technique because they are quite easy to crack. Many of the classical ciphers, with the exception of the one-time pad, can be cracked using brute force. === Modern ciphers === Modern ciphers are more secure than classical ciphers and are designed to withstand a wide range of attacks. An attacker should not be able to find the key used in a modern cipher, even if they know any specifics about the plaintext and its corresponding ciphertext. Modern encryption methods can be divided into the following categories: Private-key cryptography (symmetric key algorithm): one shared key is used for encryption and decryption Public-key cryptography (asymmetric key algorithm): two different keys are used for encryption and decryption In a symmetric key algorithm (e.g., DES, AES), the sender and receiver have a shared key established in advance: the sender uses the shared key to perform encryption; the receiver uses the shared key to perform decryption. Symmetric key algorithms can either be block ciphers or stream ciphers. Block ciphers operate on fixed-length groups of bits, called blocks, with an unvarying transformation. Stream ciphers encrypt plaintext digits one at a time on a continuous stream of data, with the transformation of successive digits varying during the encryption process. In an asymmetric key algorithm (e.g., RSA), there are two different keys: a public key and a private key. The public key is published, thereby allowing any sender to perform encryption. The private key is kept secret by the receiver, thereby allowing only the receiver to correctly perform decryption. == Cryptanalysis == Cryptanalysis (also referred to as codebreaking or cracking the code) is the study of applying various methodologies to obtain the meaning of encrypted information, without having access to the cipher required to correctly decrypt the information. This typically involves gaining an understanding of the system design and determining the cipher. Cryptanalysts can follow one or more attack models to crack a cipher, depending upon what information is available and the type of cipher being analyzed. Ciphertext is generally the most easily obtained part of a cryptosystem and therefore is an important part of cryptanalysis. === Attack models === Ciphertext-only: the cryptanalyst has access only to a collection of ciphertexts or code texts. This is the weakest attack model because the cryptanalyst has limited information. Modern ciphers rarely fail under this attack. Known-plaintext: the attacker has a set of ciphertexts to which they know the corresponding plaintext Chosen-plaintext attack: the attacker can obtain the ciphertexts corresponding to an arbitrary set of plaintexts of their own choosing Batch chosen-plaintext attack: where the cryptanalyst chooses all plaintexts before any of them are encrypted. This is often the meaning of an unqualified use of "chosen-plaintext attack". Adaptive chosen-plaintext attack: where the cryptanalyst makes a series of interactive queries, choosing subsequent plaintexts based on the information from the previous encryptions. Chosen-ciphertext attack: the attacker can obtain the plaintexts corresponding to an arbitrary set of ciphertexts of their own choosing Adaptive chosen-ciphertext attack Indifferent chosen-ciphertext attack Related-key attack: similar to a chosen-plaintext attack, except the attacker can obtain ciphertexts encrypted under two different keys. The keys are unknown, but the relationship between them is known (e.g., two keys that differ in the one bit). == Famous ciphertexts == The Babington Plot ciphers The Shugborough inscription The Zimmermann Telegram The Magic Words are Squeamish Ossifrage The cryptogram in "The Gold-Bug" Beale ciphers Kryptos Zodiac Killer ciphers
Dynamic knowledge repository
The dynamic knowledge repository (DKR) is a concept developed by Douglas C. Engelbart as a primary strategic focus for allowing humans to address complex problems. He has proposed that a DKR will enable us to develop a collective IQ greater than any individual's IQ. References and discussion of Engelbart's DKR concept are available at the Doug Engelbart Institute. == Definition == A knowledge repository is a computerized system that systematically captures, organizes and categorizes an organization's knowledge. The repository can be searched and data can be quickly retrieved. The effective knowledge repositories include factual, conceptual, procedural and meta-cognitive techniques. The key features of knowledge repositories include communication forums. A knowledge repository can take many forms to "contain" the knowledge it holds. A customer database is a knowledge repository of customer information and insights – or electronic explicit knowledge. A Library is a knowledge repository of books – physical explicit knowledge. A community of experts is a knowledge repository of tacit knowledge or experience. The nature of the repository only changes to contain/manage the type of knowledge it holds. A repository (as opposed to an archive) is designed to get knowledge out. It should therefore have some rules of structure, classification, taxonomy, record management, etc., to facilitate user engagement.
Format-preserving encryption
In cryptography, format-preserving encryption (FPE), refers to encrypting in such a way that the output (the ciphertext) is in the same format as the input (the plaintext). The meaning of "format" varies. Typically only finite sets of characters are used; numeric, alphabetic or alphanumeric. For example: Encrypting a 16-digit credit card number so that the ciphertext is another 16-digit number. Encrypting an English word so that the ciphertext is another English word. Encrypting an n-bit number so that the ciphertext is another n-bit number (this is the definition of an n-bit block cipher). For such finite domains, and for the purposes of the discussion below, the cipher is equivalent to a permutation of N integers {0, ... , N−1} where N is the size of the domain. == Motivation == === Restricted field lengths or formats === One motivation for using FPE comes from the problems associated with integrating encryption into existing applications, with well-defined data models. A typical example would be a credit card number, such as 1234567812345670 (16 bytes long, digits only). Adding encryption to such applications might be challenging if data models are to be changed, as it usually involves changing field length limits or data types. For example, output from a typical block cipher would turn credit card number into a hexadecimal (e.g.0x96a45cbcf9c2a9425cde9e274948cb67, 34 bytes, hexadecimal digits) or Base64 value (e.g. lqRcvPnCqUJc3p4nSUjLZw==, 24 bytes, alphanumeric and special characters), which will break any existing applications expecting the credit card number to be a 16-digit number. Apart from simple formatting problems, using AES-128-CBC, this credit card number might get encrypted to the hexadecimal value 0xde015724b081ea7003de4593d792fd8b695b39e095c98f3a220ff43522a2df02. In addition to the problems caused by creating invalid characters and increasing the size of the data, data encrypted using the CBC mode of an encryption algorithm also changes its value when it is decrypted and encrypted again. This happens because the random seed value that is used to initialize the encryption algorithm and is included as part of the encrypted value is different for each encryption operation. Because of this, it is impossible to use data that has been encrypted with the CBC mode as a unique key to identify a row in a database. FPE attempts to simplify the transition process by preserving the formatting and length of the original data, allowing a drop-in replacement of plaintext values with their ciphertexts in legacy applications. == Comparison to truly random permutations == Although a truly random permutation is the ideal FPE cipher, for large domains it is infeasible to pre-generate and remember a truly random permutation. So the problem of FPE is to generate a pseudorandom permutation from a secret key, in such a way that the computation time for a single value is small (ideally constant, but most importantly smaller than O(N)). == Comparison to block ciphers == An n-bit block cipher technically is a FPE on the set {0, ..., 2n-1}. If an FPE is needed on one of these standard sized sets (for example, n = 64 for DES and n = 128 for AES) a block cipher of the right size can be used. However, in typical usage, a block cipher is used in a mode of operation that allows it to encrypt arbitrarily long messages, and with an initialization vector as discussed above. In this mode, a block cipher is not an FPE. == Definition of security == In cryptographic literature (see most of the references below), the measure of a "good" FPE is whether an attacker can distinguish the FPE from a truly random permutation. Various types of attackers are postulated, depending on whether they have access to oracles or known ciphertext/plaintext pairs. == Algorithms == In most of the approaches listed here, a well-understood block cipher (such as AES) is used as a primitive to take the place of an ideal random function. This has the advantage that incorporation of a secret key into the algorithm is easy. Where AES is mentioned in the following discussion, any other good block cipher would work as well. === The FPE constructions of Black and Rogaway === Implementing FPE with security provably related to that of the underlying block cipher was first undertaken in a paper by cryptographers John Black and Phillip Rogaway, which described three ways to do this. They proved that each of these techniques is as secure as the block cipher that is used to construct it. This means that if the AES algorithm is used to create an FPE algorithm, then the resulting FPE algorithm is as secure as AES because an adversary capable of defeating the FPE algorithm can also defeat the AES algorithm. Therefore, if AES is secure, then the FPE algorithms constructed from it are also secure. In all of the following, E denotes the AES encryption operation that is used to construct an FPE algorithm and F denotes the FPE encryption operation. ==== FPE from a prefix cipher ==== One simple way to create an FPE algorithm on {0, ..., N-1} is to assign a pseudorandom weight to each integer, then sort by weight. The weights are defined by applying an existing block cipher to each integer. Black and Rogaway call this technique a "prefix cipher" and showed it was provably as good as the block cipher used. Thus, to create an FPE on the domain {0,1,2,3}, given a key K apply AES(K) to each integer, giving, for example, weight(0) = 0x56c644080098fc5570f2b329323dbf62 weight(1) = 0x08ee98c0d05e3dad3eb3d6236f23e7b7 weight(2) = 0x47d2e1bf72264fa01fb274465e56ba20 weight(3) = 0x077de40941c93774857961a8a772650d Sorting [0,1,2,3] by weight gives [3,1,2,0], so the cipher is F(0) = 3 F(1) = 1 F(2) = 2 F(3) = 0 This method is only useful for small values of N. For larger values, the size of the lookup table and the required number of encryptions to initialize the table gets too big to be practical. ==== FPE from cycle walking ==== If there is a set M of allowed values within the domain of a pseudorandom permutation P (for example P can be a block cipher like AES), an FPE algorithm can be created from the block cipher by repeatedly applying the block cipher until the result is one of the allowed values (within M). CycleWalkingFPE(x) { if P(x) is an element of M then return P(x) else return CycleWalkingFPE(P(x)) } The recursion is guaranteed to terminate. (Because P is one-to-one and the domain is finite, repeated application of P forms a cycle, so starting with a point in M the cycle will eventually terminate in M.) This has the advantage that the elements of M do not have to be mapped to a consecutive sequence {0,...,N-1} of integers. It has the disadvantage, when M is much smaller than P's domain, that too many iterations might be required for each operation. If P is a block cipher of a fixed size, such as AES, this is a severe restriction on the sizes of M for which this method is efficient. For example, an application may want to encrypt 100-bit values with AES in a way that creates another 100-bit value. With this technique, AES-128-ECB encryption can be applied until it reaches a value which has all of its 28 highest bits set to 0, which will take an average of 228 iterations to happen. ==== FPE from a Feistel network ==== It is also possible to make a FPE algorithm using a Feistel network. A Feistel network needs a source of pseudo-random values for the sub-keys for each round, and the output of the AES algorithm can be used as these pseudo-random values. When this is done, the resulting Feistel construction is good if enough rounds are used. One way to implement an FPE algorithm using AES and a Feistel network is to use as many bits of AES output as are needed to equal the length of the left or right halves of the Feistel network. If a 24-bit value is needed as a sub-key, for example, it is possible to use the lowest 24 bits of the output of AES for this value. This may not result in the output of the Feistel network preserving the format of the input, but it is possible to iterate the Feistel network in the same way that the cycle-walking technique does to ensure that format can be preserved. Because it is possible to adjust the size of the inputs to a Feistel network, it is possible to make it very likely that this iteration ends very quickly on average. In the case of credit card numbers, for example, there are 1015 possible 16-digit credit card numbers (accounting for the redundant check digit), and because the 1015 ≈ 249.8, using a 50-bit wide Feistel network along with cycle walking will create an FPE algorithm that encrypts fairly quickly on average. === The Thorp shuffle === A Thorp shuffle is like an idealized card-shuffle, or equivalently a maximally-unbalanced Feistel cipher where one side is a single bit. It is easier to prove security for unbalanced Feistel ciphers than for balanced ones. === VIL mode === For domain sizes that are a power of two, and an existing block cipher with a smaller bl
Elasticity (data store)
The elasticity of a data store relates to the flexibility of its data model and clustering capabilities. The greater the number of data model changes that can be tolerated, and the more easily the clustering can be managed, the more elastic the data store is considered to be. == Types == === Clustering elasticity === Clustering elasticity is the ease of adding or removing nodes from the distributed data store. Usually, this is a difficult and delicate task to be done by an expert in a relational database system. Some NoSQL data stores, like Apache Cassandra have an easy solution, and a node can be added/removed with a few changes in the properties and by adding specifying at least one seed. === Data-modelling elasticity === Relational databases are most often very inelastic, as they have a predefined data model that can only be adapted through redesign. Most NoSQL data stores, however, do not have a fixed schema. Each row can have a different number and even different type of columns. Concerning the data store, modifications in the schema are no problem. This makes this kind of data stores more elastic concerning the data model. The drawback is that the programmer has to take into account that the data model may change over time.
Data marketplace
Data marketplace is an online platform for sharing and consuming data in the form of data assets or data products. Part of the data management stack, it aims to bring together data producers and data consumers (including business users and AI) in a single space, with the objective of increasing access to understandable, high-quality data. Included within its Data Marketplaces and Exchange (DME) category by Gartner, data marketplaces can provide data internally within an organization, externally with partners, or as open data. == Concept == Digitization has dramatically increased data volumes within organizations, with IDC predicting that by 2025 the world will contain 175 zettabytes of data. This has created a need to both manage this data and provide access to it to enable business intelligence and data analysis. However, data is often scattered within multiple systems (such as data warehouses and data lakes), and is in formats that are only understandable by technical experts, such as data scientists. According to IDC, 81% of IT leaders cite data silos as a major barrier to digital transformation. This means that data is not freely available to business users or external audiences such as partners or citizens, limiting its value, and holding back AI deployments. Data marketplaces solve this issue, providing seamless, self-service access to high-quality data in an understandable, secure and auditable manner. They break down data silos, reduce friction in data access, and enable a broader range of users, including non-technical profiles, to find, understand, and consume data autonomously. Data assets on the marketplace can be raw data, data visualizations or data products. Data marketplaces combine data management functions such as data governance with the user-friendly experience offered by e-commerce marketplaces in order to increase the usage of data. These include features such as powerful search engines, feedback, ratings, subscriptions and product description sheets. According to Gartner, data marketplaces provide infrastructure, transactional capabilities, and services for both consumers and providers of data assets. == History and timeline == Data marketplaces have evolved since they first emerged in terms of both their scope and usage. === 2000s === With the rise of the internet, data brokers began collecting, aggregating, distributing and selling personal, financial and marketing data to third parties online. Data marketplaces were deployed to monetize this data, making it discoverable and accessible to users, either through subscriptions or one-off purchases. At the same time, regulations, such as the US Open Government Initiative of 2009 and others around the world mandated greater transparency and data sharing with the public. Data sharing portals were created by public and government bodies to make this information available through self-service to all users. === 2010s === Due to the growth of big data and cloud platforms, cloud-based data exchange platforms emerged. These were offered by major infrastructure providers, and included Amazon Web Services (AWS) Data Exchange, Snowflake Data Marketplace, and the Google Cloud Platform. These platforms moved beyond simple data brokerage or open data by providing structured, catalogued data sharing between organizations. === 2020s === Driven by a need to increase internal data sharing with both business users and AI, organizations are now looking to adopt internal data marketplaces. These aim to democratize data consumption by providing seamless access for all employees and AI to trusted data, including data products, through an intuitive, e-commerce style experience. According to Gartner analyst Richa Jha, "by providing a single, governed platform for discovering, sharing, and scaling data products, data marketplaces drive productivity, collaboration, and ROI across the enterprise." == Data marketplaces within the overall data architecture == Data marketplaces provide a consumption and collaboration layer for data. That means they complement and integrate with other parts of the overall data architecture, including: === Data warehouses and data lakes === Data marketplaces connect to data sources, such as data warehouses or data lakes, to provide intuitive access to the data stored within them, enabling data to be shared and distributed to non-technical audiences. Access can be direct, with data and data products stored within the data marketplace or virtualized. === Data catalog === A data catalog provides a technical inventory of an organization's data estate. It collects technical information on all available data assets within an organization, based on metadata descriptions. This ensures traceability, and supports compliance and governance requirements. Unlike a data marketplace, a data catalog does not provide access to data, and is designed to be used by data professionals, rather than the business. This means it lacks an intuitive, understandable interface and is consequently not easily accessible by business users. === Data mesh === Data mesh is an architecture and framework for data management, first defined by Zhamak Dehghani in 2019. It aims to decentralize data ownership to delegate responsibility, empowering teams and focusing on delivering data to users in the form of self-service data products. The data marketplace is a central pillar of data mesh, providing intuitive access to these data products, and creating a collaboration space for data owners and data consumers. === Data product === Data products are high-value, consumable data assets that package high-quality data and associated tools to enable seamless usage by business users at scale. First defined by McKinsey in 2022, they have an identified owner, a service level agreement (SLA), and a reusability logic. == Core components of a data marketplace == A data marketplace typically includes specific core components: === E-commerce style interface === An e-commerce style experience that engages non-technical users, minimizes the need for training and builds confidence and trust in data. Look and feel should be customizable to incorporate corporate design guidelines to ensure consistency with other organizational applications. === Built-in data catalog === As in a standalone data catalog, this indexes all available data, based on metadata that includes type, source, owner, freshness, and quality level. === Discovery and search engine === This enables users to search, filter, explore and discover available data intuitively. As in an e-commerce marketplace, it should be intelligent, and provide relevant results based on natural language queries. === Access control and security management === Data marketplaces will contain data that needs to be protected under regulations such as the General Data Protection Regulation (GDPR) in Europe, the California Consumer Privacy Act (CCPA) in the United States, and sector-specific frameworks in industries such as finance and healthcare. To ensure both security and compliance while maximizing data consumption, the data marketplace should include granular access management and a full audit trail. === Semantic layer and business glossary === Different parts of the business are likely to use different terms to describe data. This leads to inconsistencies and an inability to share data across systems and teams. The semantic layer and business glossary standardize a shared vocabulary and common definitions of business indicators and concepts, providing a single language for data across the business and for AI agents. === Data governance mechanisms === These enforce corporate data governance policies, ensuring data traceability through data lineage, quality certification, usage monitoring, and continuous improvement through user feedback loops. === Collaboration features === As on an e-commerce website, a data marketplace should provide collaboration features that bring together data users and data owners. This includes the ability to rate data products, share use cases, and provide feedback to data owners, creating a community around data and supporting a data-driven culture. == Types of data marketplace == While they share the same underlying technology, data marketplaces can be deployed in three broad ways: === Internal data marketplaces === These bring together data from across an organization and make it available via self-service to employees from across the business. They aim to widen access to data and consequently to improve decision-making and reporting, increase performance and maximize efficiency. === Ecosystem data marketplaces === These extend sharing beyond a single organization, enabling multiple partners (public institutions, industry players, research bodies) to share and consume data within a governed framework. Data can be provided by all parties or simply by one organization and consumed by others. Ecosystem data marketplaces are particularly relevant in