Nuance Communications, Inc. was an American multinational computer software technology corporation, headquartered in Burlington, Massachusetts, that markets speech recognition and artificial intelligence software. Nuance merged with its competitor in the commercial large-scale speech application business, ScanSoft, in October 2005. ScanSoft was a Xerox spin-off that was bought in 1999 by Visioneer, a hardware and software scanner company, which adopted ScanSoft as the new merged company name. The original ScanSoft had its roots in Kurzweil Computer Products. In April 2021, Microsoft announced it would buy Nuance Communications. The deal is an all-cash transaction of $19.7 billion, including company debt, or $56 per share. The acquisition was completed in March 2022. == History == The Speech Technology and Research (STAR) Laboratory at SRI International began the journey that, in 1994, resulted in a spin-off company; Corona Corporation (later renamed to Nuance Communications ). Nuance Communications (NUAN) went public on the Nasdaq Stock Market in 1995. Nuance focused on commercializing advanced speech recognition technologies. Nuance was an early spinoff of SRI's Speech Technology and Research (STAR) Laboratory, a world leader in audio processing, speech and speaker analytics and spoken language research. The technology that served as the foundation of Nuance's speech recognition solution started at the STAR Lab and helped launch Nuance more than 20 years ago. In 1995, The SRI Language Modeling Toolkit (SRILM) was developed. This provides the tools to build and apply statistical language models (LMs), primarily for use in speech recognition, statistical tagging and segmentation, and machine translation. In terms of commercialization of natural automated speech recognition, SRI's natural language speech recognition software was the first to be deployed by a major corporation. In 1996, Charles Schwab & Co., Inc., used Nuance's speech recognition technology to allow customers to receive stock quotes over the telephone. One of the key features of the ‘Schwab Discount Brokerage system’, was the ability to recognize English words even when spoken by customers with accents. In 1997, Nuance Communications developed the first large scale commercial dialog system for United Parcel Services (UPS). UPS used the voice recognition platform to handle very large numbers of inquiries about package status. The company that would later merge with Nuance Communications started life as Visioneer, incorporated in 1992. In 1999, Visioneer acquired ScanSoft, Inc. (SSFT), and the combined company became known as ScanSoft. In September 2005, ScanSoft Inc. acquired and merged with Nuance Communications (NUAN), a natural language DOD-project spinoff from SRI International. The resulting company adopted the Nuance name. During the prior decade, the two companies competed in the commercial large-scale speech application business. === Data breach === Between 2014 and 2017, Nuance exposed over 45,000 patient records. == Solutions == Customer service virtual assistants Speech recognition — for people Speech recognition — for business Speech recognition — for physicians Accessibility Power PDF Managed Print Services Transcription === ScanSoft origins === In 1974, Raymond Kurzweil founded Kurzweil Computer Products, Inc. to develop the first omni-font optical character-recognition system – a computer program capable of recognizing text written in any normal font. In 1980, Kurzweil sold his company to Xerox. The company became known as Xerox Imaging Systems (XIS), and later ScanSoft. In March 1992, a new company called Visioneer, Inc. was founded to develop scanner hardware and software products, such as a sheetfed scanner called PaperMax and the document management software PaperPort. Visioneer eventually sold its hardware division to Primax Electronics, Ltd. in January 1999. Two months later, in March, Visioneer acquired ScanSoft from Xerox to form a new public company with ScanSoft as the new company-wide name. Prior to 2001, ScanSoft focused primarily on desktop imaging software such as TextBridge, PaperPort and OmniPage. Beginning with the December 2001 acquisition of Lernout & Hauspie assets, the company moved into the speech recognition business and began to compete with Nuance. Lernout & Hauspie had acquired speech recognition company Dragon Systems in June 2001, shortly before becoming bankrupt in October. Scansoft acquired speech recognition company SpeechWorks in 2003. === Partnership with Siri and Apple Inc. === In 2013, Nuance confirmed that its natural language processing algorithms supported Apple's Siri voice assistant. === Focus on health care === In 2019, Nuance spun off its automotive division as the company Cerence, allowing it to focus on health care applications. === Acquisition by Microsoft === On April 12, 2021, Microsoft announced that it would buy Nuance Communications for $19.7 billion, or $56 a share, a 22% increase over the previous closing price. Nuance's CEO, Mark Benjamin, stayed with the company. This was Microsoft's second-biggest acquisition up to that point, after its purchase of LinkedIn for $24 billion (~$30.7 billion in 2024) in 2016. Shortly after the deal, the Competition and Markets Authority, a UK regulatory body, stated it was looking into the deal on the basis of antitrust concerns. In December 2021, it was reported that the deal would be approved by the European Union. The acquisition was completed on March 4, 2022. In May 2023, Nuance announced an unspecified number of layoffs.
Autonomic networking
Autonomic networking follows the concept of Autonomic Computing, an initiative started by IBM in 2001. Its ultimate aim is to create self-managing networks to overcome the rapidly growing complexity of the Internet and other networks and to enable their further growth, far beyond the size of today. == Increasing size and complexity == The ever-growing management complexity of the Internet caused by its rapid growth is seen by some experts as a major problem that limits its usability in the future. What's more, increasingly popular smartphones, PDAs, networked audio and video equipment, and game consoles need to be interconnected. Pervasive Computing not only adds features, but also burdens existing networking infrastructure with more and more tasks that sooner or later will not be manageable by human intervention alone. Another important aspect is the price of manually controlling huge numbers of vitally important devices of current network infrastructures. == Autonomic nervous system == The autonomic nervous system (ANS) is the part of complex biological nervous systems that is not consciously controlled. It regulates bodily functions and the activity of specific organs. As proposed by IBM, future communication systems might be designed in a similar way to the ANS. == Components of autonomic networking == As autonomics conceptually derives from biological entities such as the human autonomic nervous system, each of the areas can be metaphorically related to functional and structural aspects of a living being. In the human body, the autonomic system facilitates and regulates a variety of functions including respiration, blood pressure and circulation, and emotive response. The autonomic nervous system is the interconnecting fabric that supports feedback loops between internal states and various sources by which internal and external conditions are monitored. === Autognostics === Autognostics includes a range of self-discovery, awareness, and analysis capabilities that provide the autonomic system with a view on high-level state. In metaphor, this represents the perceptual sub-systems that gather, analyze, and report on internal and external states and conditions – for example, this might be viewed as the eyes, visual cortex and perceptual organs of the system. Autognostics, or literally "self-knowledge", provides the autonomic system with a basis for response and validation. A rich autognostic capability may include many different "perceptual senses". For example, the human body gathers information via the usual five senses, the so-called sixth sense of proprioception (sense of body position and orientation), and through emotive states that represent the gross wellness of the body. As conditions and states change, they are detected by the sensory monitors and provide the basis for adaptation of related systems. Implicit in such a system are imbedded models of both internal and external environments such that relative value can be assigned to any perceived state - perceived physical threat (e.g. a snake) can result in rapid shallow breathing related to fight-flight response, a phylogenetically effective model of interaction with recognizable threats. In the case of autonomic networking, the state of the network may be defined by inputs from: individual network elements such as switches and network interfaces including specification and configuration historical records and current state traffic flows end-hosts application performance data logical diagrams and design specifications Most of these sources represent relatively raw and unprocessed views that have limited relevance. Post-processing and various forms of analysis must be applied to generate meaningful measurements and assessments against which current state can be derived. The autognostic system interoperates with: configuration management - to control network elements and interfaces policy management - to define performance objectives and constraints autodefense - to identify attacks and accommodate the impact of defensive responses === Configuration management === Configuration management is responsible for the interaction with network elements and interfaces. It includes an accounting capability with historical perspective that provides for the tracking of configurations over time, with respect to various circumstances. In the biological metaphor, these are the hands and, to some degree, the memory of the autonomic system. On a network, remediation and provisioning are applied via configuration setting of specific devices. Implementation affecting access and selective performance with respect to role and relationship are also applied. Almost all the "actions" that are currently taken by human engineers fall under this area. With only a few exceptions, interfaces are set by hand, or by extension of the hand, through automated scripts. Implicit in the configuration process is the maintenance of a dynamic population of devices under management, a historical record of changes and the directives which invoked change. Typical to many accounting functions, configuration management should be capable of operating on devices and then rolling back changes to recover previous configurations. Where change may lead to unrecoverable states, the sub-system should be able to qualify the consequences of changes prior to issuing them. As directives for change must originate from other sub-systems, the shared language for such directives must be abstracted from the details of the devices involved. The configuration management sub-system must be able to translate unambiguously between directives and hard actions or to be able to signal the need for further detail on a directive. An inferential capacity may be appropriate to support sufficient flexibility (i.e. configuration never takes place because there is no unique one-to-one mapping between directive and configuration settings). Where standards are not sufficient, a learning capacity may also be required to acquire new knowledge of devices and their configuration. Configuration management interoperates with all of the other sub-systems including: autognostics - receives direction for and validation of changes policy management - implements policy models through mapping to underlying resources security - applies access and authorization constraints for particular policy targets autodefense - receives direction for changes === Policy management === Policy management includes policy specification, deployment, reasoning over policies, updating and maintaining policies, and enforcement. Policy-based management is required for: constraining different kinds of behavior including security, privacy, resource access, and collaboration configuration management describing business processes and defining performance defining role and relationship, and establishing trust and reputation It provides the models of environment and behavior that represent effective interaction according to specific goals. In the human nervous system metaphor, these models are implicit in the evolutionary "design" of biological entities and specific to the goals of survival and procreation. Definition of what constitutes a policy is necessary to consider what is involved in managing it. A relatively flexible and abstract framework of values, relationships, roles, interactions, resources, and other components of the network environment is required. This sub-system extends far beyond the physical network to the applications in use and the processes and end-users that employ the network to achieve specific goals. It must express the relative values of various resources, outcomes, and processes and include a basis for assessing states and conditions. Unless embodied in some system outside the autonomic network or implicit to the specific policy implementation, the framework must also accommodate the definition of process, objectives and goals. Business process definitions and descriptions are then an integral part of the policy implementation. Further, as policy management represents the ultimate basis for the operation of the autonomic system, it must be able to report on its operation with respect to the details of its implementation. The policy management sub-system interoperates (at least) indirectly with all other sub-systems but primarily interacts with: autognostics - providing the definition of performance and accepting reports on conditions configuration management - providing constraints on device configuration security - providing definitions of roles, access and permissions === Autodefense === Autodefense represents a dynamic and adaptive mechanism that responds to malicious and intentional attacks on the network infrastructure, or use of the network infrastructure to attack IT resources. As defensive measures tend to impede the operation of IT, it is optimally capable of balancing performance objectives with typically over-riding threat management actions. In the
Knowledge space
In mathematical psychology and education theory, a knowledge space is a combinatorial structure used to formulate mathematical models describing the progression of a human learner. Knowledge spaces were introduced in 1985 by Jean-Paul Doignon and Jean-Claude Falmagne, and remain in extensive use in the education theory. Modern applications include two computerized tutoring systems, ALEKS and the defunct RATH. Formally, a knowledge space assumes that a domain of knowledge is a collection of concepts or skills, each of which must be eventually mastered. Not all concepts are interchangeable; some require other concepts as prerequisites. Conversely, competency at one skill may ease the acquisition of another through similarity. A knowledge space marks out which collections of skills are feasible: they can be learned without mastering any other skills. Under reasonable assumptions, the collection of feasible competencies forms the mathematical structure known as an antimatroid. Researchers and educators usually explore the structure of a discipline's knowledge space as a latent class model. == Motivation == Knowledge Space Theory attempts to address shortcomings of standardized testing when used in educational psychometry. Common tests, such as the SAT and ACT, compress a student's knowledge into a very small range of ordinal ranks, in the process effacing the conceptual dependencies between questions. Consequently, the tests cannot distinguish between true understanding and guesses, nor can they identify a student's particular weaknesses, only the general proportion of skills mastered. The goal of knowledge space theory is to provide a language by which exams can communicate What the student can do and What the student is ready to learn. == Model structure == Knowledge Space Theory-based models presume that an educational subject S can be modeled as a finite set Q of concepts, skills, or topics. Each feasible state of knowledge about S is then a subset of Q; the set of all such feasible states is K. The precise term for the information (Q, K) depends on the extent to which K satisfies certain axioms: A knowledge structure assumes that K contains the empty set (a student may know nothing about S) and Q itself (a student may have fully mastered S). A knowledge space is a knowledge structure that is closed under set union: if, for each topic, there is an expert in a class on that topic, then it is possible, with enough time and effort, for each student in the class to become an expert on all those topics simultaneously. A quasi-ordinal knowledge space is a knowledge space that is also closed under set intersection: if student a knows topics A and B; and student c knows topics B and C; then it is possible for another student b to know only topic B. A well-graded knowledge space or learning space is a knowledge space satisfying the following axiom: If S∈K, then there exists x∈S such that S\{x}∈K In educational terms, any feasible body of knowledge can be learned one concept at a time. === Prerequisite partial order === The more contentful axioms associated with quasi-ordinal and well-graded knowledge spaces each imply that the knowledge space forms a well-understood (and heavily studied) mathematical structure: A quasi-ordinal knowledge space can be associated with a distributive lattice under set union and set intersection. The name "quasi-ordinal" arises from Birkhoff's representation theorem, which explains that distributive lattices uniquely correspond to partial orders. A well-graded knowledge space is an antimatroid, a type of mathematical structure that describes certain problems solvable with a greedy algorithm. In either case, the mathematical structure implies that set inclusion defines partial order on K, interpretable as an educational prerequirement: if a(⪯)b in this partial order, then a must be learned before b. === Inner and outer fringe === The prerequisite partial order does not uniquely identify a curriculum; some concepts may lead to a variety of other possible topics. But the covering relation associated with the prerequisite partial does control curricular structure: if students know a before a lesson and b immediately after, then b must cover a in the partial order. In such a circumstance, the new topics covered between a and b constitute the outer fringe of a ("what the student was ready to learn") and the inner fringe of b ("what the student just learned"). == Construction of knowledge spaces == In practice, there exist several methods to construct knowledge spaces. The most frequently used method is querying experts. There exist several querying algorithms that allow one or several experts to construct a knowledge space by answering a sequence of simple questions. Another method is to construct the knowledge space by explorative data analysis (for example by item tree analysis) from data. A third method is to derive the knowledge space from an analysis of the problem solving processes in the corresponding domain.
Ilya Sutskever
Ilya Sutskever (Hebrew: איליה סוצקבר; born 1986) is a computer scientist who specializes in machine learning. He has made several major contributions to the field of deep learning, including sequence-to-sequence learning, reasoning models, GPT models, and contributions to CLIP, DALL-E, and AlphaGo. With Alex Krizhevsky and Geoffrey Hinton, he co-created AlexNet, a convolutional neural network. One of the most highly cited computer scientists in history, he has won the NeurIPS Test of Time Award for his lasting impact on AI research three times in a row (2022–2024) and received the National Academy of Sciences Award for the Industrial Application of Science in 2026. Sutskever co-founded and was chief scientist at OpenAI, where he oversaw the research breakthroughs that led to large language models and to the launch of ChatGPT. He also led the research that led to reasoning models such as o1. In 2023, he was one of the members of OpenAI's board that ousted Sam Altman as its CEO; Altman was reinstated a week later, and Sutskever stepped down from the board. In June 2024, Sutskever co-founded the company Safe Superintelligence Inc., alongside Daniel Gross and Daniel Levy. Within a year, the company was valued at more than $30 billion. == Early life and education == Sutskever was born in 1986 into a Jewish family in Nizhny Novgorod, Russia (then Gorky, Russian SFSR, Soviet Union). At the age of 5, he immigrated to Israel with his family and grew up in Jerusalem. Sutskever proved to be a good student in school, and in eighth grade started taking classes at the Open University of Israel. At 16, he moved with his family to Canada, where he attended high school for a month before being admitted to the University of Toronto in Ontario as a third-year undergraduate student. At the University of Toronto, Sutskever received a bachelor's degree in mathematics in 2005, a master's degree in computer science in 2007, and a PhD in computer science in 2013. His doctoral advisor was Geoffrey Hinton. In 2012, Sutskever built AlexNet in collaboration with Geoffrey Hinton and Alex Krizhevsky. == Career and research == In 2012, Sutskever spent about two months as a postdoc with Andrew Ng at Stanford University. He then returned to the University of Toronto and joined Hinton's new research company DNNResearch, a spinoff of Hinton's research group. In 2013, Google acquired DNNResearch and hired Sutskever as a research scientist at Google Brain. At Google Brain, Sutskever worked with Oriol Vinyals and Quoc Viet Le to create the sequence-to-sequence learning algorithm, and worked on TensorFlow. He is also one of the AlphaGo paper's many co-authors. At the end of 2015, Sutskever left Google to become cofounder and chief scientist of the newly founded organization OpenAI. In 2022, Sutskever tweeted, "it may be that today's large neural networks are slightly conscious", which triggered debates about AI consciousness. He is considered to have played a key role in the development of ChatGPT, and later in leading the research that led to reasoning models. He is credited with establishing OpenAI’s scaling ethos. In 2023, he announced that he would co-lead OpenAI's new "Superalignment" project, which was trying to solve the alignment of superintelligences within four years. He wrote that even if superintelligence seems far off, it could happen this decade. Sutskever was formerly one of the six board members of the nonprofit entity that controlled OpenAI. In November 2023, the board fired Sam Altman, saying that "he was not consistently candid in his communications with the board". He authored a 52-page memo that relied heavily on information from Mira Murati, accusing Altman of lying, manipulating executives, and fostering internal division. Sutskever submitted the memo to the board after months of tension and dissatisfaction with Altman's leadership style, and ultimately joined the board in voting for Altman's termination. In an all-hands company meeting shortly after the board meeting, Sutskever said that firing Altman was "the board doing its duty", but the next week, he expressed regret at having participated in Altman's ouster. Altman's firing and OpenAI's co-founder Greg Brockman's resignation led three senior researchers to resign from OpenAI. After that, Sutskever stepped down from the OpenAI board and was absent from OpenAI's office. Some sources suggested he was leading the team remotely, while others said he no longer had access to the team's work. In May 2024, Sutskever announced his departure from OpenAI to focus on a new project that was "very personally meaningful" to him. His decision followed a turbulent period at OpenAI marked by leadership crises and internal debates about the direction of AI development and alignment protocols. Jan Leike, the other leader of the superalignment project, announced his departure hours later, citing an erosion of safety and trust in OpenAI's leadership. In June 2024, Sutskever announced Safe Superintelligence Inc., a new company he founded with Daniel Gross and Daniel Levy with offices in Palo Alto and Tel Aviv. In contrast to OpenAI, which releases revenue-generating products, Sutskever said the new company's "first product will be the safe superintelligence, and it will not do anything else up until then". In September 2024, the company announced that it had raised $1 billion from venture capital firms including Andreessen Horowitz, Sequoia Capital, DST Global, and SV Angel. In March 2025, Safe Superintelligence Inc. raised $2 billion more and reportedly reached a $32 billion valuation, notably due to Sutskever's reputation. In June 2025, SSI rejected an offer from Meta Platforms to buy the company. Sutskever became CEO of SSI shortly thereafter, after co-founder and CEO Gross left for Meta. In an October 2024 interview after winning the Nobel Prize in Physics, Geoffrey Hinton expressed support for Sutskever's decision to fire Altman, emphasizing concerns about AI safety. During the Musk v. Altman trial in 2026, Sutskever confirmed he had a $7 billion stake in OpenAI. === Awards and honors === In 2015, Sutskever was named in MIT Technology Review's 35 Innovators Under 35. In 2018, he was the keynote speaker at Nvidia Ntech 2018 and AI Frontiers Conference 2018. In 2022, he was elected a Fellow of the Royal Society (FRS). In 2023 and 2024, included in Time's list of the 100 most influential people in AI In 2022, 2023, and 2024, he won Neural Information Processing Systems’ Test of Time award, which recognizes papers that significantly shaped the AI field over at least ten years. In 2025, he received an honorary doctorate from his alma mater, the University of Toronto In 2026, he received the National Academy of Sciences Award for the Industrial Application of Science, presented for the first time in artificial intelligence.
Eimear Kenny
Eimear E. Kenny is a researcher in population genetics and translation genomics, and is the Founding Director of the Institute for Genomic Health, and Endowed Chair and Professor of Genomic Health at the Icahn School of Medicine at Mount Sinai. She is known for novel approaches in computational genomics, advancing the study of human genetic variation and its connection to disease risk and diagnosis. Her research has laid the foundation for integrating artificial intelligence (AI) and genomics into precision medicine and routine clinical care. By combining genomics, computer science, and medicine, her work leverages genomic sequencing technologies and machine learning algorithms to uncover insights that improve patient care, accelerate genomic data analysis, and enable the future of AI-driven healthcare. She has led multiple genomics-based clinical trials, applying computational biology and AI in clinical settings to advance genomic medicine and precision healthcare. == Research == A recipient of the Early-Career Award from the American Society of Human Genetics (USA), Kenny, as of 2024, leads a team in genetics, computer science, and medicine, focusing on genetic ancestry, large-scale genomics, clinical trials, and genomic medicine at the Institute for Genomic Health. The lab works to advance understanding of genetic ancestry and its impact on health in order to inform better clinical medicine models. She is recognized for her work to leverage biobanks for translational genomics and her development of new genetic tests an strategies for health care management. In one study, she and her colleagues investigated genetic disorders that might be under-diagnosed due to insufficient data, and found a variant in a collagen gene associated with Steel syndrome. This syndrome caused short stature and bone and joint issues and was thought to be rare. However, the study revealed it is common in individuals with Puerto Rican ancestry. Three of Kenny's genomic medicine clinical trials assessed how to bring new technology, such as digital apps, or information, such as polygenic risk scores, into routine clinical care. In the 2010s, Kenny was instrumental in several large-scale sequencing studies, including the 1000 Genomes Project, the Exome Sequencing Project, the Genome Sequencing Project, and the Trans-Omics for Precision Medicine. In 2012, she led work that discovered the variant responsible for blond hair in Melanesia, work that was featured in the Smithsonian NHGRI Human Genome Exhibit in Washington, D.C. In 2017, her group was one of the first to demonstrate that polygenic risk scores derived in predominantly European populations have reduced accuracy when applied in populations now widely acknowledged as a major challenge in the field of genomic risk prediction. As of 2024, she is Principal Investigator in many NIH-funded international consortium focused on computational genomics and genomic medicine, including Electronic Medical Records and Genomics, Polygenic Risk Methods in Diverse Populations, and the Human Pangenome Reference Consortium. In 2023, Kenny played a key role in a groundbreaking advancement in genomics research by helping to map a diverse human pangenome—a major shift from reliance on a single reference genome. Unlike the earlier genetic map, based on one man of mixed European and African ancestry in Buffalo, this new pangenome project captures far greater human genetic diversity. As reported by The Washington Post, Kenny's work demonstrates how a more inclusive human genome can drive discoveries in rare genetic diseases, improve genomic medicine, and accelerate the future of precision healthcare. Kenny was co-developer and current license holder for Random Forest adMIXture (RFMix), a patented software for inferring continental and sub-continental ancestry at genomic loci. == Education and career == Kenny graduated from Trinity College Dublin with a BA in Biochemistry in 1999 and did a masters in Bioinformatics at Leeds University. She received her PhD in Computational Genomics at Rockefeller University, and did her post-doctoral work in the lab of Dr. Carlos D. Bustamante at Stanford University. === Academic appointments === As of 2024, at Mount Sinai, she serves as the Endowed Chair and Professor of Genomic Health, Professor at the Department of Medicine and Professor at the Department of Genetics and Genomic Sciences. Since 2018 she has served as the Founding Director of the Institute for Genomic Health, and since 2022, she also serves as the Founding Director of the Center for Translational Genomics. She is also the Director of Translational Research, Division for Genomic Medicine. Former appointments include Assistant Professor at the Department of Genetics and Genomic Sciences and Member at The Charles Bronfman Institute of Personalized Medicine, both at Mount Sinai. She was also Bioinformatics Programmer at the California Institute of Technology, and research assistant at the Massachusetts Institute of Technology. == Publications == As of 2024, Kenny is an advisor to Cell Genomics. Google Scholar reports 50,623 citations, an h-index of 66 and an i10-index of 130. The five most-cited articles she contributed to are: Auton, A; Brooks, LD; Durbin, RM; Garrison, EP; Kang, HM; Korbel, JO; Marchini, JL; McCarthy, S; McVean, GA; Abecasis, GR (2015). "A global reference for human genetic variation". Nature. 526 (7571): 68–74. Bibcode:2015Natur.526...68T. doi:10.1038/nature15393. PMC 4750478. PMID 26432245.. Cited by 14847 Abecasis, GR; Auton, A; Brooks, LD; DePristo, MA; Durbin, RM; Handsaker, RE; Kang, HM; Marth, GT; McVean, GA (2012). "An integrated map of genetic variation from 1,092 human genomes". Nature. 491 (7422): 56–65. Bibcode:2012Natur.491...56T. doi:10.1038/nature11632. PMC 3498066. PMID 23128226.. Cited by 8287 Jacob A. Tennessen et al. Evolution and Functional Impact of Rare Coding Variation from Deep Sequencing of Human Exomes.Science337,64–69(2012).DOI:10.1126/science.1219240 Cited by 1886 Taliun, D.; Harris, D.N.; Kessler, M.D.; et al. (2021). "Sequencing of 53,831 diverse genomes from the NHLBI TOPMed Program". Nature. 590 (7845): 290–299. Bibcode:2021Natur.590..290T. doi:10.1038/s41586-021-03205-y. PMC 7875770. PMID 33568819.. Cited by 1369 Vilhjálmsson, BJ; et al. (2015). "Modeling Linkage Disequilibrium Increases Accuracy of Polygenic Risk Scores". Am J Hum Genet. 97 (4): 576–92. doi:10.1016/j.ajhg.2015.09.001. PMC 4596916. PMID 26430803.. Cited by 1327
Manifold regularization
In machine learning, manifold regularization is a technique for using the shape of a dataset to constrain the functions that should be learned on that dataset. In many machine learning problems, the data to be learned do not cover the entire input space. For example, a facial recognition system may not need to classify any possible image, but only the subset of images that contain faces. The technique of manifold learning assumes that the relevant subset of data comes from a manifold, a mathematical structure with useful properties. The technique also assumes that the function to be learned is smooth: data with different labels are not likely to be close together, and so the labeling function should not change quickly in areas where there are likely to be many data points. Because of this assumption, a manifold regularization algorithm can use unlabeled data to inform where the learned function is allowed to change quickly and where it is not, using an extension of the technique of Tikhonov regularization. Manifold regularization algorithms can extend supervised learning algorithms in semi-supervised learning and transductive learning settings, where unlabeled data are available. The technique has been used for applications including medical imaging, geographical imaging, and object recognition. == Manifold regularizer == === Motivation === Manifold regularization is a type of regularization, a family of techniques that reduces overfitting and ensures that a problem is well-posed by penalizing complex solutions. In particular, manifold regularization extends the technique of Tikhonov regularization as applied to Reproducing kernel Hilbert spaces (RKHSs). Under standard Tikhonov regularization on RKHSs, a learning algorithm attempts to learn a function f {\displaystyle f} from among a hypothesis space of functions H {\displaystyle {\mathcal {H}}} . The hypothesis space is an RKHS, meaning that it is associated with a kernel K {\displaystyle K} , and so every candidate function f {\displaystyle f} has a norm ‖ f ‖ K {\displaystyle \left\|f\right\|_{K}} , which represents the complexity of the candidate function in the hypothesis space. When the algorithm considers a candidate function, it takes its norm into account in order to penalize complex functions. Formally, given a set of labeled training data ( x 1 , y 1 ) , … , ( x ℓ , y ℓ ) {\displaystyle (x_{1},y_{1}),\ldots ,(x_{\ell },y_{\ell })} with x i ∈ X , y i ∈ Y {\displaystyle x_{i}\in X,y_{i}\in Y} and a loss function V {\displaystyle V} , a learning algorithm using Tikhonov regularization will attempt to solve the expression arg min f ∈ H 1 ℓ ∑ i = 1 ℓ V ( f ( x i ) , y i ) + γ ‖ f ‖ K 2 {\displaystyle {\underset {f\in {\mathcal {H}}}{\arg \!\min }}{\frac {1}{\ell }}\sum _{i=1}^{\ell }V(f(x_{i}),y_{i})+\gamma \left\|f\right\|_{K}^{2}} where γ {\displaystyle \gamma } is a hyperparameter that controls how much the algorithm will prefer simpler functions over functions that fit the data better. Manifold regularization adds a second regularization term, the intrinsic regularizer, to the ambient regularizer used in standard Tikhonov regularization. Under the manifold assumption in machine learning, the data in question do not come from the entire input space X {\displaystyle X} , but instead from a nonlinear manifold M ⊂ X {\displaystyle M\subset X} . The geometry of this manifold, the intrinsic space, is used to determine the regularization norm. === Laplacian norm === There are many possible choices for the intrinsic regularizer ‖ f ‖ I {\displaystyle \left\|f\right\|_{I}} . Many natural choices involve the gradient on the manifold ∇ M {\displaystyle \nabla _{M}} , which can provide a measure of how smooth a target function is. A smooth function should change slowly where the input data are dense; that is, the gradient ∇ M f ( x ) {\displaystyle \nabla _{M}f(x)} should be small where the marginal probability density P X ( x ) {\displaystyle {\mathcal {P}}_{X}(x)} , the probability density of a randomly drawn data point appearing at x {\displaystyle x} , is large. This gives one appropriate choice for the intrinsic regularizer: ‖ f ‖ I 2 = ∫ x ∈ M ‖ ∇ M f ( x ) ‖ 2 d P X ( x ) {\displaystyle \left\|f\right\|_{I}^{2}=\int _{x\in M}\left\|\nabla _{M}f(x)\right\|^{2}\,d{\mathcal {P}}_{X}(x)} In practice, this norm cannot be computed directly because the marginal distribution P X {\displaystyle {\mathcal {P}}_{X}} is unknown, but it can be estimated from the provided data. === Graph-based approach of the Laplacian norm === When the distances between input points are interpreted as a graph, then the Laplacian matrix of the graph can help to estimate the marginal distribution. Suppose that the input data include ℓ {\displaystyle \ell } labeled examples (pairs of an input x {\displaystyle x} and a label y {\displaystyle y} ) and u {\displaystyle u} unlabeled examples (inputs without associated labels). Define W {\displaystyle W} to be a matrix of edge weights for a graph, where W i j {\displaystyle W_{ij}} is a similarity built from distance measure between the data points x i {\displaystyle x_{i}} and x j {\displaystyle x_{j}} (so that more close implies higher W i j {\displaystyle W_{ij}} ). Define D {\displaystyle D} to be a diagonal matrix with D i i = ∑ j = 1 ℓ + u W i j {\displaystyle D_{ii}=\sum _{j=1}^{\ell +u}W_{ij}} and L {\displaystyle L} to be the Laplacian matrix D − W {\displaystyle D-W} . Then, as the number of data points ℓ + u {\displaystyle \ell +u} increases, L {\displaystyle L} converges to the Laplace–Beltrami operator Δ M {\displaystyle \Delta _{M}} , which is the divergence of the gradient ∇ M {\displaystyle \nabla _{M}} . Then, if f {\displaystyle \mathbf {f} } is a vector of the values of f {\displaystyle f} at the data, f = [ f ( x 1 ) , … , f ( x l + u ) ] T {\displaystyle \mathbf {f} =[f(x_{1}),\ldots ,f(x_{l+u})]^{\mathrm {T} }} , the intrinsic norm can be estimated: ‖ f ‖ I 2 = 1 ( ℓ + u ) 2 f T L f {\displaystyle \left\|f\right\|_{I}^{2}={\frac {1}{(\ell +u)^{2}}}\mathbf {f} ^{\mathrm {T} }L\mathbf {f} } As the number of data points ℓ + u {\displaystyle \ell +u} increases, this empirical definition of ‖ f ‖ I 2 {\displaystyle \left\|f\right\|_{I}^{2}} converges to the definition when P X {\displaystyle {\mathcal {P}}_{X}} is known. === Solving the regularization problem with graph-based approach === Using the weights γ A {\displaystyle \gamma _{A}} and γ I {\displaystyle \gamma _{I}} for the ambient and intrinsic regularizers, the final expression to be solved becomes: arg min f ∈ H 1 ℓ ∑ i = 1 ℓ V ( f ( x i ) , y i ) + γ A ‖ f ‖ K 2 + γ I ( ℓ + u ) 2 f T L f {\displaystyle {\underset {f\in {\mathcal {H}}}{\arg \!\min }}{\frac {1}{\ell }}\sum _{i=1}^{\ell }V(f(x_{i}),y_{i})+\gamma _{A}\left\|f\right\|_{K}^{2}+{\frac {\gamma _{I}}{(\ell +u)^{2}}}\mathbf {f} ^{\mathrm {T} }L\mathbf {f} } As with other kernel methods, H {\displaystyle {\mathcal {H}}} may be an infinite-dimensional space, so if the regularization expression cannot be solved explicitly, it is impossible to search the entire space for a solution. Instead, a representer theorem shows that under certain conditions on the choice of the norm ‖ f ‖ I {\displaystyle \left\|f\right\|_{I}} , the optimal solution f ∗ {\displaystyle f^{}} must be a linear combination of the kernel centered at each of the input points: for some weights α i {\displaystyle \alpha _{i}} , f ∗ ( x ) = ∑ i = 1 ℓ + u α i K ( x i , x ) {\displaystyle f^{}(x)=\sum _{i=1}^{\ell +u}\alpha _{i}K(x_{i},x)} Using this result, it is possible to search for the optimal solution f ∗ {\displaystyle f^{}} by searching the finite-dimensional space defined by the possible choices of α i {\displaystyle \alpha _{i}} . === Functional approach of the Laplacian norm === The idea beyond the graph-Laplacian is to use neighbors to estimate the Laplacian. This method is akin to local averaging methods, that are known to scale poorly in high-dimensional problems. Indeed, the graph Laplacian is known to suffer from the curse of dimensionality. Luckily, it is possible to leverage expected smoothness of the function to estimate thanks to more advanced functional analysis. This method consists of estimating the Laplacian operator using derivatives of the kernel reading ∂ 1 , j K ( x i , x ) {\displaystyle \partial _{1,j}K(x_{i},x)} where ∂ 1 , j {\displaystyle \partial _{1,j}} denotes the partial derivatives according to the j-th coordinate of the first variable. This second approach to the Laplacian norm is to put in relation with meshfree methods, that contrast with the finite difference method in PDE. == Applications == Manifold regularization can extend a variety of algorithms that can be expressed using Tikhonov regularization, by choosing an appropriate loss function V {\displaystyle V} and hypothesis space H {\displaystyle {\mathcal {H}}} . Two commonly used examples are the families of support vector machines and regularized least squares algorithm
Predictive Model Markup Language
The Predictive Model Markup Language (PMML) is an XML-based predictive model interchange format conceived by Robert Lee Grossman, then the director of the National Center for Data Mining at the University of Illinois at Chicago. PMML provides a way for analytic applications to describe and exchange predictive models produced by data mining and machine learning algorithms. It supports common models such as logistic regression and other feedforward neural networks. Version 0.9 was published in 1998. Subsequent versions have been developed by the Data Mining Group. Since PMML is an XML-based standard, the specification comes in the form of an XML schema. PMML itself is a mature standard with over 30 organizations having announced products supporting PMML. == PMML components == A PMML file can be described by the following components: Header: contains general information about the PMML document, such as copyright information for the model, its description, and information about the application used to generate the model such as name and version. It also contains an attribute for a timestamp which can be used to specify the date of model creation. Data Dictionary: contains definitions for all the possible fields used by the model. It is here that a field is defined as continuous, categorical, or ordinal (attribute optype). Depending on this definition, the appropriate value ranges are then defined as well as the data type (such as, string or double). Data Transformations: transformations allow for the mapping of user data into a more desirable form to be used by the mining model. PMML defines several kinds of simple data transformations. Normalization: map values to numbers, the input can be continuous or discrete. Discretization: map continuous values to discrete values. Value mapping: map discrete values to discrete values. Functions (custom and built-in): derive a value by applying a function to one or more parameters. Aggregation: used to summarize or collect groups of values. Model: contains the definition of the data mining model. E.g., A multi-layered feedforward neural network is represented in PMML by a "NeuralNetwork" element which contains attributes such as: Model Name (attribute modelName) Function Name (attribute functionName) Algorithm Name (attribute algorithmName) Activation Function (attribute activationFunction) Number of Layers (attribute numberOfLayers) This information is then followed by three kinds of neural layers which specify the architecture of the neural network model being represented in the PMML document. These attributes are NeuralInputs, NeuralLayer, and NeuralOutputs. Besides neural networks, PMML allows for the representation of many other types of models including support vector machines, association rules, Naive Bayes classifier, clustering models, text models, decision trees, and different regression models. Mining Schema: a list of all fields used in the model. This can be a subset of the fields as defined in the data dictionary. It contains specific information about each field, such as: Name (attribute name): must refer to a field in the data dictionary Usage type (attribute usageType): defines the way a field is to be used in the model. Typical values are: active, predicted, and supplementary. Predicted fields are those whose values are predicted by the model. Outlier Treatment (attribute outliers): defines the outlier treatment to be use. In PMML, outliers can be treated as missing values, as extreme values (based on the definition of high and low values for a particular field), or as is. Missing Value Replacement Policy (attribute missingValueReplacement): if this attribute is specified then a missing value is automatically replaced by the given values. Missing Value Treatment (attribute missingValueTreatment): indicates how the missing value replacement was derived (e.g. as value, mean or median). Targets: allows for post-processing of the predicted value in the format of scaling if the output of the model is continuous. Targets can also be used for classification tasks. In this case, the attribute priorProbability specifies a default probability for the corresponding target category. It is used if the prediction logic itself did not produce a result. This can happen, e.g., if an input value is missing and there is no other method for treating missing values. Output: this element can be used to name all the desired output fields expected from the model. These are features of the predicted field and so are typically the predicted value itself, the probability, cluster affinity (for clustering models), standard error, etc. The latest release of PMML, PMML 4.1, extended Output to allow for generic post-processing of model outputs. In PMML 4.1, all the built-in and custom functions that were originally available only for pre-processing became available for post-processing too. == PMML 4.0, 4.1, 4.2 and 4.3 == PMML 4.0 was released on June 16, 2009. Examples of new features included: Improved Pre-Processing Capabilities: Additions to built-in functions include a range of Boolean operations and an If-Then-Else function. Time Series Models: New exponential Smoothing models; also place holders for ARIMA, Seasonal Trend Decomposition, and Spectral density estimation, which are to be supported in the near future. Model Explanation: Saving of evaluation and model performance measures to the PMML file itself. Multiple Models: Capabilities for model composition, ensembles, and segmentation (e.g., combining of regression and decision trees). Extensions of Existing Elements: Addition of multi-class classification for Support Vector Machines, improved representation for Association Rules, and the addition of Cox Regression Models. PMML 4.1 was released on December 31, 2011. New features included: New model elements for representing Scorecards, k-Nearest Neighbors (KNN) and Baseline Models. Simplification of multiple models. In PMML 4.1, the same element is used to represent model segmentation, ensemble, and chaining. Overall definition of field scope and field names. A new attribute that identifies for each model element if the model is ready or not for production deployment. Enhanced post-processing capabilities (via the Output element). PMML 4.2 was released on February 28, 2014. New features include: Transformations: New elements for implementing text mining New built-in functions for implementing regular expressions: matches, concat, and replace Simplified outputs for post-processing Enhancements to Scorecard and Naive Bayes model elements PMML 4.3 was released on August 23, 2016. New features include: New Model Types: Gaussian Process Bayesian Network New built-in functions Usage clarifications Documentation improvements Version 4.4 was released in November 2019. == Release history == == Data Mining Group == The Data Mining Group is a consortium managed by the Center for Computational Science Research, Inc., a nonprofit founded in 2008. The Data Mining Group also developed a standard called Portable Format for Analytics, or PFA, which is complementary to PMML.