Random forests or random decision forests is an ensemble learning method for classification, regression and other tasks that works by creating a multitude of decision trees during training. For classification tasks, the output of the random forest is the class selected by most trees. For regression tasks, the output is the average of the predictions of the trees. Random forests correct for decision trees' habit of overfitting to their training set. The first algorithm for random decision forests was created in 1995 by Tin Kam Ho using the random subspace method, which, in Ho's formulation, is a way to implement the "stochastic discrimination" approach to classification proposed by Eugene Kleinberg. An extension of the algorithm was developed by Leo Breiman and Adele Cutler, who registered "Random Forests" as a trademark in 2006 (as of 2019, owned by Minitab, Inc.). The extension combines Breiman's "bagging" idea and random selection of features, introduced first by Ho and later independently by Amit and Geman in order to construct a collection of decision trees with controlled variance. == History == The general method of random decision forests was first proposed by Salzberg and Heath in 1993, with a method that used a randomized decision tree algorithm to create multiple trees and then combine them using majority voting. This idea was developed further by Ho in 1995. Ho established that forests of trees splitting with oblique hyperplanes can gain accuracy as they grow without suffering from overtraining, as long as the forests are randomly restricted to be sensitive to only selected feature dimensions. A subsequent work along the same lines concluded that other splitting methods behave similarly, as long as they are randomly forced to be insensitive to some feature dimensions. This observation that a more complex classifier (a larger forest) gets more accurate nearly monotonically is in sharp contrast to the common belief that the complexity of a classifier can only grow to a certain level of accuracy before being hurt by overfitting. The explanation of the forest method's resistance to overtraining can be found in Kleinberg's theory of stochastic discrimination. The early development of Breiman's notion of random forests was influenced by the work of Amit and Geman who introduced the idea of searching over a random subset of the available decisions when splitting a node, in the context of growing a single tree. The idea of random subspace selection from Ho was also influential in the design of random forests. This method grows a forest of trees, and introduces variation among the trees by projecting the training data into a randomly chosen subspace before fitting each tree or each node. Finally, the idea of randomized node optimization, where the decision at each node is selected by a randomized procedure, rather than a deterministic optimization was first introduced by Thomas G. Dietterich. The proper introduction of random forests was made in a paper by Leo Breiman, that has become one of the world's most cited papers. This paper describes a method of building a forest of uncorrelated trees using a CART like procedure, combined with randomized node optimization and bagging. In addition, this paper combines several ingredients, some previously known and some novel, which form the basis of the modern practice of random forests, in particular: Using out-of-bag error as an estimate of the generalization error. Measuring variable importance through permutation. The report also offers the first theoretical result for random forests in the form of a bound on the generalization error which depends on the strength of the trees in the forest and their correlation. == Algorithm == === Preliminaries: decision tree learning === Decision trees are a popular method for various machine learning tasks. Tree learning is almost "an off-the-shelf procedure for data mining", say Hastie et al., "because it is invariant under scaling and various other transformations of feature values, is robust to inclusion of irrelevant features, and produces inspectable models. However, they are seldom accurate". In particular, trees that are grown very deep tend to learn highly irregular patterns: they overfit their training sets, i.e. have low bias, but very high variance. Random forests are a way of averaging multiple deep decision trees, trained on different parts of the same training set, with the goal of reducing the variance. This comes at the expense of a small increase in the bias and some loss of interpretability, but generally greatly boosts the performance in the final model. === Bagging === The training algorithm for random forests applies the general technique of bootstrap aggregating, or bagging, to tree learners. Given a training set X = x1, ..., xn with responses Y = y1, ..., yn, bagging repeatedly (B times) selects a random sample with replacement of the training set and fits trees to these samples: After training, predictions for unseen samples x' can be made by averaging the predictions from all the individual regression trees on x': f ^ = 1 B ∑ b = 1 B f b ( x ′ ) {\displaystyle {\hat {f}}={\frac {1}{B}}\sum _{b=1}^{B}f_{b}(x')} or by taking the plurality vote in the case of classification trees. This bootstrapping procedure leads to better model performance because it decreases the variance of the model, without increasing the bias. This means that while the predictions of a single tree are highly sensitive to noise in its training set, the average of many trees is not, as long as the trees are not correlated. Simply training many trees on a single training set would give strongly correlated trees (or even the same tree many times, if the training algorithm is deterministic); bootstrap sampling is a way of de-correlating the trees by showing them different training sets. Additionally, an estimate of the uncertainty of the prediction can be made as the standard deviation of the predictions from all the individual regression trees on x′: σ = ∑ b = 1 B ( f b ( x ′ ) − f ^ ) 2 B − 1 . {\displaystyle \sigma ={\sqrt {\frac {\sum _{b=1}^{B}(f_{b}(x')-{\hat {f}})^{2}}{B-1}}}.} The number B of samples (equivalently, of trees) is a free parameter. Typically, a few hundred to several thousand trees are used, depending on the size and nature of the training set. B can be optimized using cross-validation, or by observing the out-of-bag error: the mean prediction error on each training sample xi, using only the trees that did not have xi in their bootstrap sample. The training and test error tend to level off after some number of trees have been fit. === From bagging to random forests === The above procedure describes the original bagging algorithm for trees. Random forests also include another type of bagging scheme: they use a modified tree learning algorithm that selects, at each candidate split in the learning process, a random subset of the features. This process is sometimes called "feature bagging". The reason for doing this is the correlation of the trees in an ordinary bootstrap sample: if one or a few features are very strong predictors for the response variable (target output), these features will be selected in many of the B trees, causing them to become correlated. An analysis of how bagging and random subspace projection contribute to accuracy gains under different conditions is given by Ho. Typically, for a classification problem with p {\displaystyle p} features, p {\displaystyle {\sqrt {p}}} (rounded down) features are used in each split. For regression problems the inventors recommend p / 3 {\displaystyle p/3} (rounded down) with a minimum node size of 5 as the default. In practice, the best values for these parameters should be tuned on a case-to-case basis for every problem. === ExtraTrees === Adding one further step of randomization yields extremely randomized trees, or ExtraTrees. As with ordinary random forests, they are an ensemble of individual trees, but there are two main differences: (1) each tree is trained using the whole learning sample (rather than a bootstrap sample), and (2) the top-down splitting is randomized: for each feature under consideration, a number of random cut-points are selected, instead of computing the locally optimal cut-point (based on, e.g., information gain or the Gini impurity). The values are chosen from a uniform distribution within the feature's empirical range (in the tree's training set). Then, of all the randomly chosen splits, the split that yields the highest score is chosen to split the node. Similar to ordinary random forests, the number of randomly selected features to be considered at each node can be specified. Default values for this parameter are p {\displaystyle {\sqrt {p}}} for classification and p {\displaystyle p} for regression, where p {\displaystyle p} is the number of features in the model. === Random forests for high-dimensional data === The basic random forest procedure may
Attack path management
Attack path management is a cybersecurity technique that involves the continuous discovery, mapping, and risk assessment of identity-based attack paths. Attack path management is distinct from other computer security mitigation strategies in that it does not rely on finding individual attack paths through vulnerabilities, exploits, or offensive testing. Rather, attack path management techniques analyze all attack paths present in an environment based on active identity management policies, authentication configurations, and active authenticated "sessions" between objects. == Overview == Attack path management relies on concepts such as mapping and removing attack paths, identifying attack path choke points, and remediation of attack paths. Identity-based attacks are present in most publicly disclosed breaches, whether through social engineering to gain initial access to Active Directories or lateral movement for privilege escalation. Attackers require privileges to attack an environment’s most sensitive segments. Attack path management often involves removing out-of-date privileges and privilege assignments given to overly large groups. In attack path management, attack graphs are used to represent how a network of machines’ security is vulnerable to attack. The nodes in an attack graph represent principals and other objects such as machines, accounts, and security groups. The edges in an attack graph represent the links and relationships between nodes. Some nodes are easy to penetrate due to short paths from regular users to domain admins, resulting in focal points of concentrated network traffic, which are known as attack path choke points. Attack graphs are often analyzed using algorithms and visualization. Attack path management also identifies tier 0 assets, which are considered the most vulnerable because they have direct or indirect control of an Active Directory or Microsoft Entra ID environment.
Lethal autonomous weapon
A lethal autonomous weapon (LAW), also known as a lethal autonomous weapon system (LAWS), autonomous weapon system (AWS), robotic weapon, or killer robot, is a type of military drone or military robot, which is autonomous in that it can independently search for and engage targets based on programmed constraints and descriptions. As of 2025, most military drones (including unmanned aerial vehicles and unmanned combat aerial vehicles) and military robots are not truly autonomous. LAWs may engage in drone warfare in the air, on land, on water, underwater, or in space. == Definitions == In weapons development, the term "autonomous" is somewhat ambiguous and can vary hugely between different scholars, nations and organizations. There is no definition of lethal autonomous weapon systems that is generally agreed upon among different countries. The official United States Department of Defense Policy on Autonomy in Weapon Systems (Department of Defense Directive 3000.09) defines an Autonomous Weapon System as one that "...once activated, can select and engage targets without further intervention by a human operator." Heather Roff, a writer for Case Western Reserve University School of Law, describes autonomous weapon systems as "... capable of learning and adapting their 'functioning in response to changing circumstances in the environment in which [they are] deployed,' as well as capable of making firing decisions on their own." The British Ministry of Defence states "Whilst definitions can vary, the key difference is that an automated system is capable of carrying out complicated tasks but is incapable of complex decision-making, whereas an autonomous system is capable of deciding a course of action without depending on human oversight and control." Scholars such as Peter Asaro and Mark Gubrud believe that any weapon system that is capable of releasing a lethal force without the operation, decision, or confirmation of a human supervisor can be deemed autonomous. == Automatic defensive systems == Some definitions of autonomous weapon systems are broad enough to include land mines and naval mines, simple automatically-triggered lethal weapons that have been in use for centuries. Some current examples of LAWs are automated "hardkill" active protection systems, such as a radar-guided close-in weapon systems (CIWS) used to defend ships that have been in use since the 1970s (e.g., the US Phalanx CIWS). Such systems can autonomously identify and attack oncoming missiles, rockets, artillery fire, aircraft, and surface vessels according to criteria set by the human operator. Similar systems exist for tanks, such as the Russian Arena, the Israeli Trophy, and the German AMAP-ADS. Several types of stationary sentry guns, which can fire at humans and vehicles, are used in South Korea and Israel. Many missile defence systems, such as Iron Dome, also have autonomous targeting capabilities. The main reason for not having a "human in the loop" in these systems is the need for rapid response. They have generally been used to protect personnel and installations against incoming projectiles. == Autonomous offensive systems == According to The Economist in 2018, as technology advances, applications of uncrewed undersea vehicles could include mine clearance, mine-laying, anti-submarine sensor networking in contested waters, patrolling with active sonar, resupplying manned submarines, and becoming low-cost missile platforms. In 2017 the Russian Federation was developing artificially intelligent missiles, drones, unmanned vehicles, military robots and medic robots. In 2018, the U.S. Nuclear Posture Review alleged that Russia was developing a "new intercontinental, nuclear-armed, nuclear-powered, undersea autonomous torpedo" named "Status 6". Israeli Minister Ayoob Kara stated in 2017 that Israel is developing military robots, including ones as small as flies. In October 2018, Zeng Yi, a senior executive at the Chinese defense firm Norinco, gave a speech in which he said that "In future battlegrounds, there will be no people fighting", and that the use of lethal autonomous weapons in warfare is "inevitable". In 2019, US Defense Secretary Mark Esper lashed out at China for selling drones capable of taking life with no human oversight. As of 2020, DARPA was working on making swarms of 250 autonomous lethal drones available to the American military. The US Navy is developing unmanned surface vehicles, also called sea drones, including Ghost Fleet Overlord, with plans to equip them with weapons and with the potential to use them semi-autonomously. In 2020 a Kargu 2 drone hunted down and attacked a human target in Libya, according to a report from the UN Security Council's Panel of Experts on Libya, published in March 2021. This may have been the first time an autonomous killer robot armed with lethal weaponry attacked human beings. In May 2021 Israel conducted an AI-guided combat drone swarm attack in Gaza. In the Russo-Ukrainian war, Ukraine has developed advanced drones with integrated artificial intelligence for a range of drone warfare purposes, including to attack infrastructure in Russia, although as of May 2026, Al Jazeera reported that humans remain in control of operation. == Ethical and legal issues == === Degree of human control === Three classifications of the degree of human control of autonomous weapon systems were laid out by Bonnie Docherty in a 2012 Human Rights Watch report. human-in-the-loop: a human must instigate the action of the weapon (in other words not fully autonomous). human-on-the-loop: a human may abort an action. human-out-of-the-loop: no human action is involved. === Standard used in US policy === Department of Defense Directive 3000.09 states that "Autonomous … weapons systems shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force." However, as noted in the Bulletin of the Atomic Scientists, the policy requires that autonomous weapon systems that kill people or use kinetic force, selecting and engaging targets without further human intervention, be certified as compliant with "appropriate levels" and other standards, not that such weapon systems cannot meet these standards and are therefore forbidden. "Semi-autonomous" hunter-killers that autonomously identify and attack targets do not even require certification. Deputy Defense Secretary Robert O. Work said in 2016 that the Defense Department would "not delegate lethal authority to a machine to make a decision", but might need to reconsider this since "authoritarian regimes" may do so. In October 2016 President Barack Obama stated that early in his career he was wary of a future in which a US president making use of drone warfare could "carry on perpetual wars all over the world, and a lot of them covert, without any accountability or democratic debate". In the US, security-related AI has fallen under the purview of the National Security Commission on Artificial Intelligence since 2018. On October 31, 2019, the United States Department of Defense's Defense Innovation Board published the draft of a report outlining five principles for weaponized AI and making 12 recommendations for the ethical use of artificial intelligence by the Department of Defense that would ensure a human operator would always be able to look into the 'black box' and understand the kill-chain process. A major concern is how the report will be implemented. === Possible violations of ethics and international acts === Stuart Russell, professor of computer science from University of California, Berkeley stated the concern he has with LAWs is that his view is that it is unethical and inhumane. The main issue with this system is it is hard to distinguish between combatants and non-combatants. There is concern by some economists and legal scholars about whether LAWs would violate International Humanitarian Law, especially the principle of distinction, which requires the ability to discriminate combatants from non-combatants, and the principle of proportionality, which requires that damage to civilians be proportional to the military aim. This concern is often invoked as a reason to ban "killer robots" altogether - but it is doubtful that this concern can be an argument against LAWs that do not violate International Humanitarian Law. A 2021 report by the American Congressional Research Service states that "there are no domestic or international legal prohibitions on the development of use of LAWs," although it acknowledges ongoing talks at the UN Convention on Certain Conventional Weapons (CCW). LAWs are said by some to blur the boundaries of who is responsible for a particular killing. Philosopher Robert Sparrow argues that autonomous weapons are causally but not morally responsible, similar to child soldiers. In each case, he argues there is a risk of atrocities occurring without an appropriate subject to hold responsible, which violates jus in bell
Computational creativity
Computational creativity (also known as artificial creativity, mechanical creativity, creative computing or creative computation) is a multidisciplinary endeavour that is located at the intersection of the fields of artificial intelligence, cognitive psychology, philosophy, and the arts (e.g., computational art as part of computational culture). Is the application of computer systems to emulate human-like creative processes, facilitating the generation of artistic and design outputs that mimic innovation and originality. The goal of computational creativity is to model, simulate or replicate creativity using a computer, to achieve one of several ends: To construct a program or computer capable of human-level creativity. To better understand human creativity and to formulate an algorithmic perspective on creative behavior in humans. To design programs that can enhance human creativity without necessarily being creative themselves. The field of computational creativity concerns itself with theoretical and practical issues in the study of creativity. Theoretical work on the nature and proper definition of creativity is performed in parallel with practical work on the implementation of systems that exhibit creativity, with one strand of work informing the other. The applied form of computational creativity is known as media synthesis. == Theoretical issues == Theoretical approaches concern the essence of creativity. Especially, under what circumstances it is possible to call the model a "creative" if eminent creativity is about rule-breaking or the disavowal of convention. This is a variant of Ada Lovelace's objection to machine intelligence, as recapitulated by modern theorists such as Teresa Amabile. If a machine can do only what it was programmed to do, how can its behavior ever be called creative? Indeed, not all computer theorists would agree with the premise that computers can only do what they are programmed to do—a key point in favor of computational creativity. == Defining creativity in computational terms == Because no single perspective or definition seems to offer a complete picture of creativity, the AI researchers Newell, Shaw and Simon developed the combination of novelty and usefulness into the cornerstone of a multi-pronged view of creativity, one that uses the following four criteria to categorize a given answer or solution as creative: The answer is novel and useful (either for the individual or for society) The answer demands that we reject ideas we had previously accepted The answer results from intense motivation and persistence The answer comes from clarifying a problem that was originally vague Margaret Boden focused on the first two of these criteria, arguing instead that creativity (at least when asking whether computers could be creative) should be defined as "the ability to come up with ideas or artifacts that are new, surprising, and valuable". Mihaly Csikszentmihalyi argued that creativity had to be considered instead in a social context, and his DIFI (Domain-Individual-Field Interaction) framework has since strongly influenced the field. In DIFI, an individual produces works whose novelty and value are assessed by the field—other people in society—providing feedback and ultimately adding the work, now deemed creative, to the domain of societal works from which an individual might be later influenced. Whereas the above reflects a top-down approach to computational creativity, an alternative thread has developed among bottom-up computational psychologists involved in artificial neural network research. During the late 1980s and early 1990s, for example, such generative neural systems were driven by genetic algorithms. Experiments involving recurrent nets were successful in hybridizing simple musical melodies and predicting listener expectations. == Historical evolution of computational creativity == The use computational processes to generate creative artifacts has been present from early times in history. During the late 1800's, methods for composing music combinatorily were explored, involving prominent figures like Mozart, Bach, Haydn, and Kiernberger. This approach extended to analytical endeavors as early as 1934, where simple mechanical models were built to explore mathematical problem solving. Professional interest in the creative aspect of computation also was commonly addressed in early discussions of artificial intelligence. The 1956 Dartmouth Conference, listed creativity, invention, and discovery as key goals for artificial intelligence. As the development of computers allowed systems of greater complexity, the 1970's and 1980's saw invention of early systems that modelled creativity using symbolic or rule-based approaches. The field of creative storytelling investigated several such models. Meehan's TALE-SPIN (1977) generated narratives through simulation of character goals and decision trees. Dehn's AUTHOR (1981) approached generation by simulating an author's process for crafting a story. Beyond narrative generation, computational creativity expanded into artistic and scientific domains. Artistic image generation was one of the disciplines that saw early potential in generated artifacts through computational creativity. One of the most prominent examples was Harold Cohen's AARON, which produced art through composition and adaptation of figures based on a large set of symbolic rules and heuristics for visual composition. Some systems also tackled creativity in scientific endeavors. BACON was said to rediscover natural laws like Boyle's Law and Kepler's law through hypothesis testing in constrained spaces. By the 1990's the modeling techniques became more adaptive, attempting to implement cognitive creative rules for generation. Turner's MINSTREL (1993) introduced TRAMs (Transform Recall Adapt Methods) to simulate creative re-use of prior material for generative storytelling. Meanwhile, Pérez y Pérez's MEXICA (1999) modeled the creative writing process using cycles of engagement and reflection. As systems increasingly incorporated models of internal evaluation, another approach that emerged was that of combining symbolic generation with domain-specific evaluation metrics, modeling generative and selective steps to creativity In the field of generational humor, the JAPE system (1994) generated pun-based riddles using Prolog and WordNet, applying symbolic pattern-matching rules and a large lexical database (WordNet) to compose riddles involving wordplay. WordNet is a system developed by George Miller and his team at Princeton, its platform and inspired word-mapping structures have been used as the backbone of several syntactic and semantic AI programs. A notable system for music generation was David Cope's EMI (Experiments in Musical Intelligence) or Emmy, which was trained in the styles of artists like Bach, Beethoven, or Chopin and generated novel pieces in their style through pattern abstraction and recomposition. In the 2000s and beyond, machine learning began influencing creative system design. Researchers such as Mihalcea and Strapparava trained classifiers to distinguish humorous from non-humorous text, using stylistic and semantic features. Meanwhile custom computational approaches led to chess systems like Deep Blue generating quasi-creative gameplay strategies through search algorithms and parallel processing constrained by specific rules and patterns for evaluation. The institutional development of computational creativity grew along its technical advances. Dedicated workshops such as the IJWCC emerged in the 1990s, growing out of interdisciplinary conferences focused on AI and creativity. By the early 2000s, the field coalesced around annual conferences like the International Conference on Computational Creativity (ICCC). Recently, with the advent of Deep Learning, Transformers, and further refinement in Machine Learning structures, computational creativity's implementation space has new tools for development. == Machine learning for computational creativity == While traditional computational approaches to creativity rely on the explicit formulation of prescriptions by developers and a certain degree of randomness in computer programs, machine learning methods allow computer programs to learn on heuristics from input data enabling creative capacities within the computer programs. Especially, deep artificial neural networks allow to learn patterns from input data that allow for the non-linear generation of creative artefacts. Before 1989, artificial neural networks have been used to model certain aspects of creativity. Peter Todd (1989) first trained a neural network to reproduce musical melodies from a training set of musical pieces. Then he used a change algorithm to modify the network's input parameters. The network was able to randomly generate new music in a highly uncontrolled manner. In 1992, Todd extended this work, using the so-called distal teacher approach that had been d
Spark NLP
Spark NLP is an open-source text processing library for advanced natural language processing for the Python, Java and Scala programming languages. The library is built on top of Apache Spark and its Spark ML library. Its purpose is to provide an API for natural language processing pipelines that implement recent academic research results as production-grade, scalable, and trainable software. The library offers pre-trained neural network models, pipelines, and embeddings, as well as support for training custom models. == Features == The design of the library makes use of the concept of a pipeline which is an ordered set of text annotators. Out of the box annotators include, tokenizer, normalizer, stemming, lemmatizer, regular expression, TextMatcher, chunker, DateMatcher, SentenceDetector, DeepSentenceDetector, POS tagger, ViveknSentimentDetector, sentiment analysis, named entity recognition, conditional random field annotator, deep learning annotator, spell checking and correction, dependency parser, typed dependency parser, document classification, and language detection. The Models Hub is a platform for sharing open-source as well as licensed pre-trained models and pipelines. It includes pre-trained pipelines with tokenization, lemmatization, part-of-speech tagging, and named entity recognition that exist for more than thirteen languages; word embeddings including GloVe, ELMo, BERT, ALBERT, XLNet, Small BERT, and ELECTRA; sentence embeddings including Universal Sentence Embeddings (USE) and Language Agnostic BERT Sentence Embeddings (LaBSE). It also includes resources and pre-trained models for more than two hundred languages. Spark NLP base code includes support for East Asian languages such as tokenizers for Chinese, Japanese, Korean; for right-to-left languages such as Urdu, Farsi, Arabic, Hebrew and pre-trained multilingual word and sentence embeddings such as LaUSE and a translation annotator. == Usage in healthcare == Spark NLP for Healthcare is a commercial extension of Spark NLP for clinical and biomedical text mining. It provides healthcare-specific annotators, pipelines, models, and embeddings for clinical entity recognition, clinical entity linking, entity normalization, assertion status detection, de-identification, relation extraction, and spell checking and correction. The library offers access to several clinical and biomedical transformers: JSL-BERT-Clinical, BioBERT, ClinicalBERT, GloVe-Med, GloVe-ICD-O. It also includes over 50 pre-trained healthcare models, that can recognize the entities such as clinical, drugs, risk factors, anatomy, demographics, and sensitive data. == Spark OCR == Spark OCR is another commercial extension of Spark NLP for optical character recognition (OCR) from images, scanned PDF documents, and DICOM files. It is a software library built on top of Apache Spark. It provides several image pre-processing features for improving text recognition results such as adaptive thresholding and denoising, skew detection & correction, adaptive scaling, layout analysis and region detection, image cropping, removing background objects. Due to the tight coupling between Spark OCR and Spark NLP, users can combine NLP and OCR pipelines for tasks such as extracting text from images, extracting data from tables, recognizing and highlighting named entities in PDF documents or masking sensitive text in order to de-identify images. Several output formats are supported by Spark OCR such as PDF, images, or DICOM files with annotated or masked entities, digital text for downstream processing in Spark NLP or other libraries, structured data formats (JSON and CSV), as files or Spark data frames. Users can also distribute the OCR jobs across multiple nodes in a Spark cluster. == License and availability == Spark NLP is licensed under the Apache 2.0 license. The source code is publicly available on GitHub as well as documentation and a tutorial. Prebuilt versions of Spark NLP are available in PyPi and Anaconda Repository for Python development, in Maven Central for Java & Scala development, and in Spark Packages for Spark development. == Award == In March 2019, Spark NLP received Open Source Award for its contributions in natural language processing in Python, Java, and Scala.
AI data center
An AI data center is a specialized data center facility designed for the computationally intensive tasks of training and running inference for artificial intelligence (AI) and machine learning models. Unlike general-purpose data centers, they are optimized for the parallel processing demands of AI workloads, typically using hardware such as AI accelerators (e.g., GPUs, TPUs) and high-speed interconnects. The global push to construct these specialized facilities accelerated dramatically during the AI boom of the 2020s. Memory manufacturers prioritized production of High Bandwidth Memory (HBM) essential for AI servers, which led to a global memory supply shortage amid a broader competition for advanced chips, power, and infrastructure. Major tech companies are estimated to spend $650 billion on AI data centers in 2026. == Architecture == Data centers for building and running large machine learning models contain specialized computer chips, GPUs, that use 2 to 4 times as much energy as their regular CPU counterparts (250-500 watts). AI data centers use 60 or more kilowatts per server rack, whereas more standard data centers typically use 5 to 10 kilowatts per rack. == Operators == As of August 2025, The Information tracked 18 planned or existing AI data centers in the United States, operated by Amazon Web Services, CoreWeave, Crusoe, Meta, Microsoft/OpenAI, Oracle, Tesla, and xAI. Other AI data center operators include Digital Realty and Alibaba. Data centers are also being built in China, India, Europe, Saudi Arabia, and Canada. The New Yorker described CoreWeave as the most prominent AI data center operator in the United States. Two types of data center providers for machine learning have been noted: hyperscalers and neoclouds. The Verge listed large technology companies such as Google, Meta, Microsoft, Oracle and Amazon as hyperscalers. The New York Times described neoclouds as "a new generation of data center providers". CoreWeave, Nebius, Nscale, and Lambda have been described as examples of neoclouds. In January 2025, OpenAI, in partnership with Oracle and Softbank, announced the Stargate project, which as of September 2025 is composed of six built or proposed AI data centers in the United States. In response to the Stargate project, Amazon launched in October 2025 an AI data center on 1,200 acres of farmland in Indiana. This data center, known as Project Rainier, is one of the largest AI data centers in the world, with Amazon spending $11 billion on the project. Rainier is specifically intended for training and running machine learning models from Anthropic. As of that time, this facility contains seven data centers (out of an estimated 30 planned) and will use 2.2 gigawatts of electricity (equivalent to 1 million households) and millions of gallons of water per year. Computer chips from Annapurna Labs and Anthropic, Trainium 2, were designed for use in such facilities. Amazon pumped millions of gallons of water out of the ground to construct the data center, and as of June 2025, Indiana state officials are investigating whether this dewatering process led to dry wells for local residents. In November 2025, Anthropic announced a plan in partnership with Fluidstack to develop artificial intelligence infrastructure in the United States, including data centers in New York and Texas, worth $50 billion. Other AI data center projects include the Colossus supercomputer from xAI, a Louisiana-based project from Meta, Hyperion, expected to use 5 GW of power, and a second Ohio-based Meta project, Prometheus, with a capacity of 1 GW. A 3,200-acre AI data center, capable of 4.4-4.5 GW of power and located on the decommissioned Homer City Generating Station, is under construction as of 2025, and will use seven 30-acre gas generating stations supplied by EQT. As of December 2025, CRH is working on over 100 data centers in the United States. In 2025, ExxonMobil and NextEra announced plans to build a data center powered by natural gas and using carbon capture technology, with 1.2 GW of power capacity. They previously purchased 2,500 acres of land in the Southeastern United States and plan to market the data center to an artificial intelligence company. The increased interest in AI data centers has led to several executives from companies in that space becoming billionaires, including CoreWeave, QTS, Nebius, Astera Labs, Groq, Fermi (which is connected to former United States Secretary of Energy Rick Perry), Snowflake and Cipher Mining. Several companies involved in cryptocurrency mining, such as Bitdeer, CoreWeave, Cipher Mining, TeraWulf, IREN, Core Scientific, and CleanSpark have also been involved with AI data centers. == Finances == Between January and August 2024, Microsoft, Meta, Google and Amazon collectively spent $125 billion on AI data centers. Citigroup forecasted that $2.8 trillion would be spent on AI data centers by 2030, while McKinsey and Company estimated that almost $7 trillion would be spent globally by that time. According to S&P Global, $61 billion has been spent on the data center market as a whole in 2025, while debt issuance for data centers was $182 billion during the same year. Large technology companies have offloaded the financial risks of building AI data centers by setting up special purpose vehicles or by contracting with neoclouds. For example, Meta's Hyperion was mostly funded by Blue Owl Capital, which did so using a bond offering from PIMCO. Those bonds were sold to a number of clients, including BlackRock. Meta did not borrow money itself and instead established a special purpose vehicle from which it would rent the data center. This deal was structured by Morgan Stanley for $30 billion, the largest known private capital transaction as of 2025. Neoclouds such as CoreWeave have gone into debt to buy computer chips from Nvidia for their data centers, and the chips themselves have been used for loan collateral. As of December 2025, CoreWeave took out three GPU-backed loans, collectively worth $12.4 billion, from private credit firms (Blackstone, Coatue, BlackRock, PIMCO) and from banks (Goldman Sachs, JPMorgan Chase, Wells Fargo). Thus, these companies provide an indirect connection between private credit and established banks. Data centers have also established asset-backed securities, and debt for data centers has its own derivative financial products. The real estate industry, including asset managers, public companies and private investors, has also invested in data centers. == Energy sourcing == == Environmental footprint == Average AI data centers have an electricity footprint equivalent to 100,000 households, and use billions of gallons of water for cooling their hardware. In 2025, the International Energy Agency estimated that the larger AI data centers currently under construction could consume as much electricity as 2 million households. A 2024 report from the United States Department of Energy stated that data centers overall used 17 billion gallons of water per year in the United States, primarily due to "rapid proliferation of AI servers", and that this usage was forecasted to grow to nearly 80 billion gallons by 2028. Researchers estimated that AI data centers in the United States would emit 24-44 million metric tons of carbon dioxide and use 731–1,125 million cubic meters of water per year between 2024 and 2030. Peaking power plants, which have been proposed as a power source for AI data centers, emit sulfur dioxide and have historically been located disproportionately near communities of color in the United States. Reciprocating internal combustion engines, proposed as another power source for a data center, emit PM 2.5, nitrogen oxides, and volatile organic compounds. == AI data centers in the United States == In the United States, both the Biden administration and second Trump administration supported the construction of AI data centers. In January 2025, then-president Joe Biden signed an executive order for federal government agencies to support AI data centers on federal sites built by private companies, study their effect on energy prices, and encourage their use of renewable energy. In April 2025, the United States Department of Energy suggested 16 possible sites, including Los Alamos National Laboratory, Sandia National Laboratories and Oak Ridge National Laboratory. In its July 2025 AI Action Plan, the second Trump administration supported increased production of AI data centers. Several US states have incentivized local data center construction. For example, in 2024, lawmakers in Michigan approved tax breaks for data center equipment and construction material. Some data center companies have also invested or promised to invest in the infrastructure of local communities. In December 2025, Democratic senators Elizabeth Warren, Chris Van Hollen, and Richard Blumenthal wrote to seven technology companies (Google, Microsoft, Amazon, Meta, CoreWeave, Digital Realty, and Equinix) that they w
Copyright and artificial intelligence in the United Kingdom
The interaction of artificial intelligence and copyright law has become one of the most contentious tech policy debates in the United Kingdom, centering on whether AI developers should be permitted to train their models on copyrighted material without explicit consent or remuneration. This debate has exposed a deep fracture between the creative industries, which seek to protect their intellectual property from unauthorised commercial exploitation, and tech companies. The academic and library sectors are also impacted, and argue that overly restrictive copyright laws hinder scientific research and the UK's sovereign AI capabilities. In 2024, the UK government proposed a broad text and data mining (TDM) exception to copyright that would have allowed AI companies to use publicly available copyrighted material for training, offering creators only an "opt-out" mechanism, similar to the exception introduced in Europe. This proposal faced intense opposition from across the creative sector. Trade unions representing writers, musicians, performers, and journalists argued that such an exception would effectively expropriate their members' work for the commercial benefit of tech giants. A report from the House of Lords Communications and Digital Committee, warned that generative AI posed a "clear and present danger" to the £124 billion creative economy. The government abandoned the opt-out model in March 2026, opting instead to build a stronger evidence base before pursuing any copyright reform. Conversely, the academic and library sectors have raised significant concerns that the UK's current TDM exception, which is strictly limited to non-commercial research, is too narrow. Universities and research libraries occupy a dual role as both creators of vast datasets and beneficiaries of TDM exceptions. They argue that the current legal framework restricts their ability to computationally analyse the very research they produce, thereby hobbling the UK's "AI for Science" strategy. Advocacy groups have highlighted a "triple payment" problem, wherein publicly funded research is handed over to publishers, who then charge universities substantial subscription fees and demand additional payments for specific TDM licences. This tension is further complicated by the commercial practices of major academic publishers. While publishers often restrict universities from using subscribed databases for AI training, they have simultaneously entered into lucrative, multi-million-dollar licensing agreements to sell access to this academic content to commercial AI developers. Furthermore, academics have accused publishers of actively steering authors away from permissive open-access licences towards more restrictive variants. By doing so, publishers retain the exclusive commercial rights necessary to strike these AI training deals, often without consulting the original authors or offering them any additional remuneration. This dynamic has not only reopened debates within the Open Access movement but has also created complex legal scenarios where publishers, rather than authors, control the terms of copyright litigation against major tech companies. == Training on copyrighted material == The question of whether AI developers should be permitted to train their models on copyrighted material without payment or consent has been one of the most contentious policy debates in the UK AI landscape. In 2024, the then-Conservative government proposed a broad text and data mining (TDM) exception that would have allowed AI companies to use any publicly available copyrighted material for training purposes, with creators able only to "opt out" of having their work used. This proposal provoked intense opposition from writers, musicians, visual artists, publishers, and broadcasters, who argued it would effectively expropriate their intellectual property for the commercial benefit of AI companies. The debate over text and data mining exceptions extends significantly beyond generative AI and the creative industries, implicating a wide range of scientific, industrial, and academic research applications. TDM is a foundational process for analysing large datasets to identify patterns, trends, and correlations, which is heavily utilised in fields such as medical research, climate modelling, and financial services. In the scientific and academic sectors, researchers rely on TDM to process vast amounts of published literature. For example, in biomedical research, TDM is used to accelerate drug discovery, identify new uses for existing medicines, and extract insights from clinical notes and genomic datasets. However, the application of traditional copyright frameworks to scientific literature has been criticised by academics. Researchers argue that scientific writing is intended to convey factual, verifiable information rather than creative originality, and that copyright restrictions on TDM hinder reproducibility, validation, and the advancement of science. The current UK copyright exception for TDM (Section 29A of the Copyright, Designs and Patents Act 1988) is limited strictly to non-commercial research, which creates barriers for public-private research partnerships and commercial scientific development. Beyond academia, non-generative AI and TDM are critical to various industrial and commercial operations. In the financial services sector, TDM is employed to monitor transactions, detect fraud, and analyse market feeds. Other non-generative applications include search engine indexing, plagiarism detection software, and media monitoring. A 2026 report by Public First estimated that 19% of UK businesses use specialised TDM tools, and that a restrictive copyright regime requiring licenses for all copyrighted content could cost the UK economy £220 billion in lost AI-driven GDP growth by 2035 compared to a broad commercial TDM exemption. Industry advocates argue that the lack of a commercial TDM exception in the UK creates legal uncertainty that stifles innovation across these broader, non-generative applications of data analysis. === Tech and AI industry positions === The technology and artificial intelligence industries lobbied for a broad text and data mining (TDM) exception to UK copyright law, arguing that such an exception is essential for the UK to remain globally competitive in AI development. Industry bodies such as techUK have argued that without a TDM exception, the UK risks becoming an "AI taker rather than an AI maker," as developers will relocate training operations to jurisdictions with more permissive copyright regimes, such as the United States, Japan, Singapore, and the European Union. During the UK government's 2024–2025 consultation on copyright and AI, major AI developers and trade associations strongly supported "Option 2" (a broad TDM exception) or "Option 3" (a TDM exception with an opt-out mechanism). OpenAI stated in its consultation response that a broad TDM exception is "necessary to drive AI innovation and investment in the UK," arguing that developers should be permitted to train models on lawfully accessed copies without further distribution. The Computer and Communications Industry Association (CCIA) similarly argued that restricting TDM to non-commercial development would undermine the government's ambitions for the UK tech sector and frustrate partnerships between commercial entities and research institutions. Tech industry advocates have also highlighted the economic implications of copyright policy. According to analysis by the think tank UK Day One, adopting an overly restrictive licensing-only approach could result in the UK economy losing up to £182 billion over 20 years, whereas a broad TDM exception could generate a positive impact of £131.61 billion over the same period. Following the government's March 2026 decision to drop plans for a TDM exception in favour of a market-led licensing approach, techUK's Deputy CEO Antony Walker criticised the move, stating that "copyright material cannot be used for AI development and training without permission" under the current framework, which he argued would push AI model training to the US. === Creative sector and political opposition to text and data mining === In March 2026, the House of Lords Communications and Digital Committee published a report, AI, Copyright and the Creative Industries, which concluded that the creative industries face "a clear and present danger from generative AI" and that it would be "a very poor bet" for the government to weaken copyright protections to attract AI investment. The Committee noted that the creative industries contributed £124 billion to the UK economy in 2023 and employed 2.4 million people, compared to the AI sector's £12 billion GVA and 86,000 employees in 2024. The Committee called on the government to develop a "licensing-first" regime underpinned by mandatory transparency requirements, and to rule out any new commercial TDM exception with an opt-out model. Tra