Pruning is a data compression technique in machine learning and search algorithms that reduces the size of decision trees by removing sections of the tree that are non-critical and redundant to classify instances. Pruning reduces the complexity of the final classifier, and hence improves predictive accuracy by the reduction of overfitting. One of the questions that arises in a decision tree algorithm is the optimal size of the final tree. A tree that is too large risks overfitting the training data and poorly generalizing to new samples. A small tree might not capture important structural information about the sample space. However, it is hard to tell when a tree algorithm should stop because it is impossible to tell if the addition of a single extra node will dramatically decrease error. This problem is known as the horizon effect. A common strategy is to grow the tree until each node contains a small number of instances then use pruning to remove nodes that do not provide additional information. Pruning should reduce the size of a learning tree without reducing predictive accuracy as measured by a cross-validation set. There are many techniques for tree pruning that differ in the measurement that is used to optimize performance. == Techniques == Pruning processes can be divided into two types (pre- and post-pruning). Pre-pruning procedures prevent a complete induction of the training set by replacing a stop () criterion in the induction algorithm (e.g. max. Tree depth or information gain (Attr)> minGain). Pre-pruning methods are considered to be more efficient because they do not induce an entire set, but rather trees remain small from the start. Prepruning methods share a common problem, the horizon effect. This is to be understood as the undesired premature termination of the induction by the stop () criterion. Post-pruning (or just pruning) is the most common way of simplifying trees. Here, nodes and subtrees are replaced with leaves to reduce complexity. Pruning can not only significantly reduce the size but also improve the classification accuracy of unseen objects. It may be the case that the accuracy of the assignment on the train set deteriorates, but the accuracy of the classification properties of the tree increases overall. The procedures are differentiated on the basis of their approach in the tree (top-down or bottom-up). === Bottom-up pruning === These procedures start at the last node in the tree (the lowest point). Following recursively upwards, they determine the relevance of each individual node. If the relevance for the classification is not given, the node is dropped or replaced by a leaf. The advantage is that no relevant sub-trees can be lost with this method. These methods include Reduced Error Pruning (REP), Minimum Cost Complexity Pruning (MCCP), or Minimum Error Pruning (MEP). === Top-down pruning === In contrast to the bottom-up method, this method starts at the root of the tree. Following the structure below, a relevance check is carried out which decides whether a node is relevant for the classification of all n items or not. By pruning the tree at an inner node, it can happen that an entire sub-tree (regardless of its relevance) is dropped. One of these representatives is pessimistic error pruning (PEP), which brings quite good results with unseen items. == Pruning algorithms == === Reduced error pruning === One of the simplest forms of pruning is reduced error pruning. Starting at the leaves, each node is replaced with its most popular class. If the prediction accuracy is not affected then the change is kept. While somewhat naive, reduced error pruning has the advantage of simplicity and speed. === Cost complexity pruning === Cost complexity pruning generates a series of trees T 0 … T m {\displaystyle T_{0}\dots T_{m}} where T 0 {\displaystyle T_{0}} is the initial tree and T m {\displaystyle T_{m}} is the root alone. At step i {\displaystyle i} , the tree is created by removing a subtree from tree i − 1 {\displaystyle i-1} and replacing it with a leaf node with value chosen as in the tree building algorithm. The subtree that is removed is chosen as follows: Define the error rate of tree T {\displaystyle T} over data set S {\displaystyle S} as err ( T , S ) {\displaystyle \operatorname {err} (T,S)} . The subtree t {\displaystyle t} that minimizes err ( prune ( T , t ) , S ) − err ( T , S ) | leaves ( T ) | − | leaves ( prune ( T , t ) ) | {\displaystyle {\frac {\operatorname {err} (\operatorname {prune} (T,t),S)-\operatorname {err} (T,S)}{\left\vert \operatorname {leaves} (T)\right\vert -\left\vert \operatorname {leaves} (\operatorname {prune} (T,t))\right\vert }}} is chosen for removal. The function prune ( T , t ) {\displaystyle \operatorname {prune} (T,t)} defines the tree obtained by pruning the subtrees t {\displaystyle t} from the tree T {\displaystyle T} . Once the series of trees has been created, the best tree is chosen by generalized accuracy as measured by a training set or cross-validation. == Examples == Pruning could be applied in a compression scheme of a learning algorithm to remove the redundant details without compromising the model's performances. In neural networks, pruning removes entire neurons or layers of neurons.
Nouvelle AI
Nouvelle artificial intelligence (Nouvelle AI) is an approach to artificial intelligence pioneered in the 1980s by Rodney Brooks, who was then part of MIT artificial intelligence laboratory. Nouvelle AI differs from classical AI by aiming to produce robots with intelligence levels similar to insects. Researchers believe that intelligence can emerge organically from simple behaviors as these intelligences interacted with the "real world", instead of using the constructed worlds which symbolic AIs typically needed to have programmed into them. == Motivation == The differences between nouvelle AI and symbolic AI are apparent in early robots Shakey and Freddy. These robots contained an internal model (or "representation") of their micro-worlds consisting of symbolic descriptions. As a result, this structure of symbols had to be renewed as the robot moved or the world changed. Shakey's planning programs assessed the program structure and broke it down into the necessary steps to complete the desired action. This level of computation required a large amount time to process, so Shakey typically performed its tasks very slowly. Symbolic AI researchers had long been plagued by the problem of updating, searching, and otherwise manipulating the symbolic worlds inside their AIs. A nouvelle system refers continuously to its sensors rather than to an internal model of the world. It processes the external world information it needs from the senses when it is required. As Brooks puts it, "the world is its own best model--always exactly up to date and complete in every detail." A central idea of nouvelle AI is that simple behaviors combine to form more complex behaviors over time. For example, simple behaviors can include elements like "move forward" and "avoid obstacles." A robot using nouvelle AI with simple behaviors like collision avoidance and moving toward a moving object could possibly come together to produce a more complex behavior like chasing a moving object. === The frame problem === The frame problem describes an issue with using first-order logic (FOL) to express facts about a robot in the world. Representing the state of a robot with traditional FOL requires the use of many axioms (symbolic language) to imply that things about an environment do not change arbitrarily. Nouvelle AI seeks to sidestep the frame problem by dispensing with filling the AI or robot with volumes of symbolic language and instead letting more complex behaviors emerge by combining simpler behavioral elements. === Embodiment === The goal of traditional AI was to build intelligences without bodies, which would only have been able to interact with the world via keyboard, screen, or printer. However, nouvelle AI attempts to build embodied intelligence situated in the real world. Brooks quotes approvingly from the brief sketches that Turing gave in 1948 and 1950 of the "situated" approach. Turing wrote of equipping a machine "with the best sense organs that money can buy" and teaching it "to understand and speak English" by a process that would "follow the normal teaching of a child." This approach was contrasted to the others where they focused on abstract activities such as playing chess. == Brooks' robots == === Insectoid robots === Brooks focused on building robots that acted like simple insects while simultaneously working to remove some traditional AI characteristics. He created insect-like robots, named Allen and Herbert after cognitive science and AI pioneers Allen Newell and Herbert A. Simon. Brooks's insectoid robots contained no internal models of the world. Herbert, for example, discarded a high volume of the information received from its sensors and never stored information for more than two seconds. ==== Allen ==== Allen had a ring of twelve ultrasonic sonars as its primary sensors and three independent behavior-producing modules. These modules were programmed to avoid both stationary and moving objects. With only this module activated, Allen stayed in the middle of a room until an object approached and then it ran away while avoiding obstacles in its way. ==== Herbert ==== Herbert used infrared sensors to avoid obstacles and a laser system to collect 3D data over a distance of about 12 feet. Herbert also carried a number of simple sensors in its "hand." The robot's testing ground was the real world environment of the busy offices and workspaces of the MIT AI lab where it searched for empty soda cans and carried them away, a seemingly goal-oriented activity that emerged as a result of 15 simple behavior units combining. As a parallel, Simon noted that an ant's complicated path is due to the structure of its environment rather than the depth of its thought processes. ==== Other insectoid robots ==== Other robots by Brooks' team were Genghis and Squirt. Genghis had six legs and was able to walk over rough terrain and follow a human. Squirt's behavior modules had it stay in dark corners until it heard a noise, then it would begin to follow the source of the noise. Brooks agreed that the level of nouvelle AI had come near the complexity of a real insect, which raised a question about whether or not insect level-behavior was and is a reasonable goal for nouvelle AI. === Humanoid robots === Brooks' own recent work has taken the opposite direction to that proposed by Von Neumann in the quotations "theorists who select the human nervous system as their model are unrealistically picking 'the most complicated object under the sun,' and that there is little advantage in selecting instead the ant, since any nervous system at all exhibits exceptional complexity." ==== Cog ==== In the 1990s, Brooks decided to pursue the goal of human-level intelligence and, with Lynn Andrea Stein, built a humanoid robot called Cog. Cog is a robot with an extensive collection of sensors, a face, and arms (among other features) that allow it to interact with the world and gather information and experience so as to assemble intelligence organically in the manner described above by Turing. The team believed that Cog would be able to learn and able to find a correlation between the sensory information it received and its actions, and to learn common sense knowledge on its own. As of 2003, all development of the project had ceased.
Mating pool
Mating pool is a concept used in evolutionary algorithms and means a population of parents for the next population. The mating pool is formed by candidate solutions that the selection operators deem to have the highest fitness in the current population. Solutions that are included in the mating pool are referred to as parents. Individual solutions can be repeatedly included in the mating pool, with individuals of higher fitness values having a higher chance of being included multiple times. Crossover operators are then applied to the parents, resulting in recombination of genes recognized as superior. Lastly, random changes in the genes are introduced through mutation operators, increasing the genetic variation in the gene pool. Those two operators improve the chance of creating new, superior solutions. A new generation of solutions is thereby created, the children, who will constitute the next population. Depending on the selection method, the total number of parents in the mating pool can be different to the size of the initial population, resulting in a new population that’s smaller. To continue the algorithm with an equally sized population, random individuals from the old populations can be chosen and added to the new population. At this point, the fitness value of the new solutions is evaluated. If the termination conditions are fulfilled, processes come to an end. Otherwise, they are repeated. The repetition of the steps result in candidate solutions that evolve towards the most optimal solution over time. The genes will become increasingly uniform towards the most optimal gene, a process called convergence. If 95% of the population share the same version of a gene, the gene has converged. When all the individual fitness values have reached the value of the best individual, i.e. all the genes have converged, population convergence is achieved. == Mating pool creation == Several methods can be applied to create a mating pool. All of these processes involve the selective breeding of a particular number of individuals within a population. There are multiple criteria that can be employed to determine which individuals make it into the mating pool and which are left behind. The selection methods can be split into three general types: fitness proportionate selection, ordinal based selection and threshold based selection. === Fitness proportionate selection === In the case of fitness proportionate selection, random individuals are selected to enter the pool. However, the ones with a higher level of fitness are more likely to be picked and therefore have a greater chance of passing on their features to the next generation. One of the techniques used in this type of parental selection is the roulette wheel selection. This approach divides a hypothetical circular wheel into different slots, the size of which is equal to the fitness values of each potential candidate. Afterwards, the wheel is rotated and a fixed point determines which individual gets picked. The greater the fitness value of an individual, the higher the probability of being chosen as a parent by the random spin of the wheel. Alternatively, stochastic universal sampling can be implemented. This selection method is also based on the rotation of a spinning wheel. However, in this case there is more than one fixed point and as a result all of the mating pool members will be selected simultaneously. === Ordinal based selection === The ordinal based selection methods include the tournament and ranking selection. Tournament selection involves the random selection of individuals of a population and the subsequent comparison of their fitness levels. The winners of these “tournaments” are the ones with the highest values and will be put into the mating pool as parents. In ranking selection all the individuals are sorted based on their fitness values. Then, the selection of the parents is made according to the rank of the candidates. Every individual has a chance of being chosen, but higher ranked ones are favored === Threshold based selection === The last type of selection method is referred to as the threshold based method. This includes the truncation selection method, which sorts individuals based on their phenotypic values on a specific trait and later selects the proportion of them that are within a certain threshold as parents.
Cellular evolutionary algorithm
A cellular evolutionary algorithm (cEA) is a kind of evolutionary algorithm (EA) in which individuals cannot mate arbitrarily, but every one interacts with its closer neighbors on which a basic EA is applied (selection, variation, replacement). The cellular model simulates natural evolution from the point of view of the individual, which encodes a tentative optimization, learning, or search problem solution. The essential idea of this model is to provide the EA population with a special structure defined as a connected graph, in which each vertex is an individual who communicates with his nearest neighbors. Particularly, individuals are conceptually set in a toroidal mesh, and are only allowed to recombine with close individuals. This leads to a kind of locality known as "isolation by distance". The set of potential mates of an individual is called its "neighborhood". It is known that, in this kind of algorithm, similar individuals tend to cluster creating niches, and these groups operate as if they were separate sub-populations (islands). There is no clear borderline between adjacent groups, and close niches could be easily colonized by competitive niches and potentially merge solution contents during the process. Simultaneously, farther niches can be affected more slowly. == Introduction == A cellular evolutionary algorithm (cEA) usually evolves a structured bidimensional grid of individuals, although other topologies are also possible. In this grid, clusters of similar individuals are naturally created during evolution, promoting exploration in their boundaries, while exploitation is mainly performed by direct competition and merging inside them. The grid is usually 2D toroidal structure, although the number of dimensions can be easily extended (to 3D) or reduced (to 1D, e.g. a ring). The neighborhood of a particular point of the grid (where an individual is placed) is defined in terms of the Manhattan distance from it to others in the population. Each point of the grid has a neighborhood that overlaps the neighborhoods of nearby individuals. In the basic algorithm, all the neighborhoods have the same size and identical shapes. The two most commonly used neighborhoods are L5, also called the Von Neumann or NEWS (North, East, West and South) neighborhood, and C9, also known as the Moore neighborhood. Here, L stands for "linear" while C stands for "compact". In cEAs, the individuals can only interact with their neighbors in the reproductive cycle where the variation operators are applied. This reproductive cycle is executed inside the neighborhood of each individual and, generally, consists in selecting two parents among its neighbors according to a certain criterion, applying the variation operators to them (recombination and mutation for example), and replacing the considered individual by the recently created offspring following a given criterion, for instance, replace if the offspring represents a better solution than the considered individual. == Synchronous versus asynchronous == In a regular synchronous cEA, the algorithm proceeds from the very first top left individual to the right and then to the several rows by using the information in the population to create a new temporary population. After finishing with the bottom-right last individual the temporary population is full with the newly computed individuals, and the replacement step starts. In it, the old population is completely and synchronously replaced with the newly computed one according to some criterion. Usually, the replacement keeps the best individual in the same position of both populations, that is, elitism is used. According to the update policy of the population used, an asynchronous cEA may also be defined and is a well-known issue in cellular automata. In asynchronous cEAs the order in which the individuals in the grid are update changes depending on the choice of criterion: line sweep, fixed random sweep, new random sweep, and uniform choice. All four proceed using the newly computed individual (or the original if better) for the computations of its neighbors. The overlap of the neighborhoods provides an implicit mechanism of solution migration to the cEA. Since the best solutions spread smoothly through the whole population, genetic diversity in the population is preserved longer than in non structured EAs. This soft dispersion of the best solutions through the population is one of the main issues of the good tradeoff between exploration and exploitation that cEAs perform during the search. This tradeoff can be tuned (and by extension the genetic diversity level along the evolution) by modifying (for instance) the size of the neighborhood used, as the overlap degree between the neighborhoods grows according to the size of the neighborhood. A cEA can be seen as a cellular automaton (CA) with probabilistic rewritable rules, where the alphabet of the CA is equivalent to the potential number of solutions of the problem. Hence, knowledge from research in CAs can be applied to cEAs. == Parallelism == Cellular EAs are very amenable to parallelism, thus usually found in the literature of parallel metaheuristics. In particular, fine grain parallelism can be used to assign independent threads of execution to every individual, thus allowing the whole cEA to run on a concurrent or actually parallel hardware platform. In this way, large time reductions can be obtained when running cEAs on FPGAs or GPUs. However, it is important to stress that cEAs are a model of search, in many senses different from traditional EAs. Also, they can be run in sequential and parallel platforms, reinforcing the fact that the model and the implementation are two different concepts. See here for a complete description on the fundamentals for the understanding, design, and application of cEAs.
Correspondence analysis
Correspondence analysis (CA) is a multivariate statistical technique proposed by Herman Otto Hartley (Hirschfeld) and later developed by Jean-Paul Benzécri. It is conceptually similar to principal component analysis, but applies to categorical rather than continuous data. In a manner similar to principal component analysis, it provides a means of displaying or summarising a set of data in two-dimensional graphical form. Its aim is to display in a biplot any structure hidden in the multivariate setting of the data table. As such it is a technique from the field of multivariate ordination. Since the variant of CA described here can be applied either with a focus on the rows or on the columns it should in fact be called simple (symmetric) correspondence analysis. It is traditionally applied to the contingency table of a pair of nominal variables where each cell contains either a count or a zero value. If more than two categorical variables are to be summarized, a variant called multiple correspondence analysis should be chosen instead. CA may also be applied to binary data given the presence/absence coding represents simplified count data i.e. a 1 describes a positive count and 0 stands for a count of zero. Depending on the scores used CA preserves the chi-square distance between either the rows or the columns of the table. Because CA is a descriptive technique, it can be applied to tables regardless of a significant chi-squared test. Although the χ 2 {\displaystyle \chi ^{2}} statistic used in inferential statistics and the chi-square distance are computationally related they should not be confused since the latter works as a multivariate statistical distance measure in CA while the χ 2 {\displaystyle \chi ^{2}} statistic is in fact a scalar not a metric. == Details == Like principal components analysis, correspondence analysis creates orthogonal components (or axes) and, for each item in a table i.e. for each row, a set of scores (sometimes called factor scores, see Factor analysis). Correspondence analysis is performed on the data table, conceived as matrix C of size m × n where m is the number of rows and n is the number of columns. In the following mathematical description of the method capital letters in italics refer to a matrix while letters in italics refer to vectors. Understanding the following computations requires knowledge of matrix algebra. === Preprocessing === Before proceeding to the central computational step of the algorithm, the values in matrix C have to be transformed. First compute a set of weights for the columns and the rows (sometimes called masses), where row and column weights are given by the row and column vectors, respectively: w m = 1 n C C 1 , w n = 1 n C 1 T C . {\displaystyle w_{m}={\frac {1}{n_{C}}}C\mathbf {1} ,\quad w_{n}={\frac {1}{n_{C}}}\mathbf {1} ^{T}C.} Here n C = ∑ i = 1 n ∑ j = 1 m C i j {\displaystyle n_{C}=\sum _{i=1}^{n}\sum _{j=1}^{m}C_{ij}} is the sum of all cell values in matrix C, or short the sum of C, and 1 {\displaystyle \mathbf {1} } is a column vector of ones with the appropriate dimension. Put in simple words, w m {\displaystyle w_{m}} is just a vector whose elements are the row sums of C divided by the sum of C, and w n {\displaystyle w_{n}} is a vector whose elements are the column sums of C divided by the sum of C. The weights are transformed into diagonal matrices W m = diag ( 1 / w m ) {\displaystyle W_{m}=\operatorname {diag} (1/{\sqrt {w_{m}}})} and W n = diag ( 1 / w n ) {\displaystyle W_{n}=\operatorname {diag} (1/{\sqrt {w_{n}}})} where the diagonal elements of W n {\displaystyle W_{n}} are 1 / w n {\displaystyle 1/{\sqrt {w_{n}}}} and those of W m {\displaystyle W_{m}} are 1 / w m {\displaystyle 1/{\sqrt {w_{m}}}} respectively i.e. the vector elements are the inverses of the square roots of the masses. The off-diagonal elements are all 0. Next, compute matrix P {\displaystyle P} by dividing C {\displaystyle C} by its sum P = 1 n C C . {\displaystyle P={\frac {1}{n_{C}}}C.} In simple words, Matrix P {\displaystyle P} is just the data matrix (contingency table or binary table) transformed into portions i.e. each cell value is just the cell portion of the sum of the whole table. Finally, compute matrix S {\displaystyle S} , sometimes called the matrix of standardized residuals, by matrix multiplication as S = W m ( P − w m w n ) W n {\displaystyle S=W_{m}(P-w_{m}w_{n})W_{n}} Note, the vectors w m {\displaystyle w_{m}} and w n {\displaystyle w_{n}} are combined in an outer product resulting in a matrix of the same dimensions as P {\displaystyle P} . In words the formula reads: matrix outer ( w m , w n ) {\displaystyle \operatorname {outer} (w_{m},w_{n})} is subtracted from matrix P {\displaystyle P} and the resulting matrix is scaled (weighted) by the diagonal matrices W m {\displaystyle W_{m}} and W n {\displaystyle W_{n}} . Multiplying the resulting matrix by the diagonal matrices is equivalent to multiply the i-th row (or column) of it by the i-th element of the diagonal of W m {\displaystyle W_{m}} or W n {\displaystyle W_{n}} , respectively. === Interpretation of preprocessing === The vectors w m {\displaystyle w_{m}} and w n {\displaystyle w_{n}} are the row and column masses or the marginal probabilities for the rows and columns, respectively. Subtracting matrix outer ( w m , w n ) {\displaystyle \operatorname {outer} (w_{m},w_{n})} from matrix P {\displaystyle P} is the matrix algebra version of double centering the data. Multiplying this difference by the diagonal weighting matrices results in a matrix containing weighted deviations from the origin of a vector space. This origin is defined by matrix outer ( w m , w n ) {\displaystyle \operatorname {outer} (w_{m},w_{n})} . In fact matrix outer ( w m , w n ) {\displaystyle \operatorname {outer} (w_{m},w_{n})} is identical with the matrix of expected frequencies in the chi-squared test. Therefore S {\displaystyle S} is computationally related to the independence model used in that test. But since CA is not an inferential method the term independence model is inappropriate here. === Orthogonal components === The table S {\displaystyle S} is then decomposed by a singular value decomposition as S = U Σ V ∗ {\displaystyle S=U\Sigma V^{}\,} where U {\displaystyle U} and V {\displaystyle V} are the left and right singular vectors of S {\displaystyle S} and Σ {\displaystyle \Sigma } is a square diagonal matrix with the singular values σ i {\displaystyle \sigma _{i}} of S {\displaystyle S} on the diagonal. Σ {\displaystyle \Sigma } is of dimension p ≤ ( min ( m , n ) − 1 ) {\displaystyle p\leq (\min(m,n)-1)} hence U {\displaystyle U} is of dimension m×p and V {\displaystyle V} is of n×p. As orthonormal vectors U {\displaystyle U} and V {\displaystyle V} fulfill U ∗ U = V ∗ V = I {\displaystyle U^{}U=V^{}V=I} . In other words, the multivariate information that is contained in C {\displaystyle C} as well as in S {\displaystyle S} is now distributed across two (coordinate) matrices U {\displaystyle U} and V {\displaystyle V} and a diagonal (scaling) matrix Σ {\displaystyle \Sigma } . The vector space defined by them has as number of dimensions p, that is the smaller of the two values, number of rows and number of columns, minus 1. === Inertia === While a principal component analysis may be said to decompose the (co)variance, and hence its measure of success is the amount of (co-)variance covered by the first few PCA axes - measured in eigenvalue -, a CA works with a weighted (co-)variance which is called inertia. The sum of the squared singular values is the total inertia I {\displaystyle \mathrm {I} } of the data table, computed as I = ∑ i = 1 p σ i 2 . {\displaystyle \mathrm {I} =\sum _{i=1}^{p}\sigma _{i}^{2}.} The total inertia I {\displaystyle \mathrm {I} } of the data table can also computed directly from S {\displaystyle S} as I = ∑ i = 1 n ∑ j = 1 m s i j 2 . {\displaystyle \mathrm {I} =\sum _{i=1}^{n}\sum _{j=1}^{m}s_{ij}^{2}.} The amount of inertia covered by the i-th set of singular vectors is ι i {\displaystyle \iota _{i}} , the principal inertia. The higher the portion of inertia covered by the first few singular vectors i.e. the larger the sum of the principal inertiae in comparison to the total inertia, the more successful a CA is. Therefore, all principal inertia values are expressed as portion ϵ i {\displaystyle \epsilon _{i}} of the total inertia ϵ i = σ i 2 / ∑ i = 1 p σ i 2 {\displaystyle \epsilon _{i}=\sigma _{i}^{2}/\sum _{i=1}^{p}\sigma _{i}^{2}} and are presented in the form of a scree plot. In fact a scree plot is just a bar plot of all principal inertia portions ϵ i {\displaystyle \epsilon _{i}} . === Coordinates === To transform the singular vectors to coordinates which preserve the chi-square distances between rows or columns an additional weighting step is necessary. The resulting coordinates are called principal coordinates in CA text books. If principal coordinates are used for
SCADA Strangelove
SCADA Strangelove is an independent group of information security researchers founded in 2012, focused on security assessment of industrial control systems (ICS) and SCADA. == Activities == Main fields of research include: Discovery of 0-day vulnerabilities in cyber physical systems and coordinated vulnerability disclosure; Security assessment of ICS protocols and development suites; Identification of publicly Internet-connected ICS components and secure it with help of proper authorities; Development of security hardening guides for ICS software; Mapping cybersecurity on to functional safety; Awareness control and delivery of information regarding the actual security state of ICS systems. SCADA Strangelove's interests expand further than classic ICS components and covers various embedded systems, however, and encompass smart home components, solar panels, wind turbines, SmartGrid as well as other areas. == Projects == Group members have and continue to develop and publish numerous open source tools for scanning, fingerprinting, security evaluation and password bruteforcing for ICS devices. These devices work over industrial protocols such as modbus, Siemens S7, MMS, ISO EC 60870, ProfiNet. In 2014 Shodan used some of the published tools for building a map of ICS devices which is publicly available on the Internet. Open source security assessment frameworks, such as THC Hydra, Metasploit, and DigitalBond Redpoint have used Shodan-developed tools and techniques. The group has published security-hardening guidelines for industrial solutions based on Siemens SIMATIC WinCC and WinCC Flexible. The guidelines contain detailed security configuration walk-throughs, descriptions of internal security features and appropriate best practices. Among the group’s more noticeable projects is Choo Choo PWN (CCP) also named the Critical Infrastructure Attack (CIA). This is an interactive laboratory built upon ICS software and hardware used in real world. Every system is connected to a toy city infrastructure, which includes factories, railroads and other facilities. The laboratory has been demonstrated at various conferences including PHDays, Power of Community, and 30C3. Primarily the laboratory is used for the discovery of new vulnerabilities and for evaluation of security mechanisms, however it is also used for workshops and other educational activities. At Positive Hack Days IV, contestants found several 0-day vulnerabilities in Indusoft Web Studio 7.1 by Schneider Electric, and in specific ICS hardware RTU PET-7000 during the ICS vulnerability discovery challenge. The group supports Secure Open SmartGrid (SCADASOS) project to find and fix vulnerabilities in intellectual power grid components such as photovoltaic power station, wind turbine, power inverter. More than 80 000 industrial devices were discovered and isolated from the Internet in 2015. == Appearances == Group members are frequently seen presenting at conferences like CCC, SCADA Security Scientific Symposium, Positive Hack Days. Most notable talks are: === 29C3 === An overview of vulnerabilities discovered in the widely distributed Siemens SIMATIC WinCC software and tools that are implemented for searching ICS on the Internet. === PHDays === This talk consisted of an overview of vulnerabilities discovered in various systems produced by ABB, Emerson, Honeywell and Siemens and was presented at PHDays III and PHDays IV. === Confidence 2014 === Implications of security research aimed at realization of various industrial network protocols Profinet, Modbus, DNP3, IEC 61850-8-1 (MMS), IEC (International Electrotechnical Commission) 61870-5-101/104, FTE (Fault Tolerant Ethernet), Siemens S7. === PacSec 2014 === Presentations of security research showing the impact of radio and 3G/4G networks on the security of mobile devices as well as on industrial equipment. === 31C3 === Analysis of security architecture and implementation of the most wide spread platforms for wind and solar energy generation which produce many gigawatts of it. === 32C3 === Cybersecurity assessment of railway signaling systems such as Automatic Train Control (ATC), Computer-based interlocking (CBI) and European Train Control System (ETCS). === China Internet Security Conference 2016 === In "Greater China Cyber Threat Landscape" keynote by Sergey Gordeychik an overview of vulnerabilities, attacks and cyber-security incidents in Greater China region was presented. === Recon 2017 === In talk "Hopeless: Relay Protection for Substation Automation" by Kirill Nesterov and Alexander Tlyapov security analysis results of key Digital Substation component - Relay Protection Terminals was presented. Vulnerabilities, including remote code execution in Siemens SIPROTEC, General Electric Line Distance Relay, NARI and ABB protective relays was presented. == Philosophy == All names, catchwords and graphical elements refer to Stanley Kubrick’s film, Dr. Strangelove. In their talks, group members often refer to Cold War events such as the Caribbean Crisis, and draw parallels between nuclear arms race and the current escalation of cyberwar. Group members follow the approach of “responsible disclosure” and “ready to wait for years, while vendor is patching the vulnerability”. Public exploits for discovered vulnerabilities are not published. This is on account of the longevity of ICS and by implication the long process of patching ICS. However, conflicts still happen, notably in 2012 when the talk at DEF CON was called off due to a dispute of persistent weaknesses in Siemens industrial software.
Computational learning theory
In computer science, computational learning theory (or just learning theory) is a subfield of artificial intelligence devoted to studying the design and analysis of machine learning algorithms. == Overview == Theoretical results in machine learning often focus on a type of inductive learning known as supervised learning. In supervised learning, an algorithm is provided with labeled samples. For instance, the samples might be descriptions of mushrooms, with labels indicating whether they are edible or not. The algorithm uses these labeled samples to create a classifier. This classifier assigns labels to new samples, including those it has not previously encountered. The goal of the supervised learning algorithm is to optimize performance metrics, such as minimizing errors on new samples. In addition to performance bounds, computational learning theory studies the time complexity and feasibility of learning . In computational learning theory, a computation is considered feasible if it can be done in polynomial time . There are two kinds of time complexity results: Positive results – Showing that a certain class of functions is learnable in polynomial time. Negative results – Showing that certain classes cannot be learned in polynomial time. Negative results often rely on commonly believed, but yet unproven assumptions, such as: Computational complexity – P ≠ NP (the P versus NP problem); Cryptographic – One-way functions exist. There are several different approaches to computational learning theory based on making different assumptions about the inference principles used to generalise from limited data. This includes different definitions of probability (see frequency probability, Bayesian probability) and different assumptions on the generation of samples. The different approaches include: Exact learning, proposed by Dana Angluin; Probably approximately correct learning (PAC learning), proposed by Leslie Valiant; VC theory, proposed by Vladimir Vapnik and Alexey Chervonenkis; Inductive inference as developed by Ray Solomonoff; Algorithmic learning theory, from the work of E. Mark Gold; Online machine learning, from the work of Nick Littlestone. While its primary goal is to understand learning abstractly, computational learning theory has led to the development of practical algorithms. For example, PAC theory inspired boosting, VC theory led to support vector machines, and Bayesian inference led to belief networks.