Rprop, short for resilient backpropagation, is a learning heuristic for supervised learning in feedforward artificial neural networks. This is a first-order optimization algorithm. This algorithm was created by Martin Riedmiller and Heinrich Braun in 1992. Similarly to the Manhattan update rule, Rprop takes into account only the sign of the partial derivative over all patterns (not the magnitude), and acts independently on each "weight". For each weight, if there was a sign change of the partial derivative of the total error function compared to the last iteration, the update value for that weight is multiplied by a factor η−, where η− < 1. If the last iteration produced the same sign, the update value is multiplied by a factor of η+, where η+ > 1. The update values are calculated for each weight in the above manner, and finally each weight is changed by its own update value, in the opposite direction of that weight's partial derivative, so as to minimise the total error function. η+ is empirically set to 1.2 and η− to 0.5. Rprop can result in very large weight increments or decrements if the gradients are large, which is a problem when using mini-batches as opposed to full batches. RMSprop addresses this problem by keeping the moving average of the squared gradients for each weight and dividing the gradient by the square root of the mean square. RPROP is a batch update algorithm. Next to the cascade correlation algorithm and the Levenberg–Marquardt algorithm, Rprop is one of the fastest weight update mechanisms. == Variations == Martin Riedmiller developed three algorithms, all named RPROP. Igel and Hüsken assigned names to them and added a new variant: RPROP+ is defined at A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm. RPROP− is defined at Advanced Supervised Learning in Multi-layer Perceptrons – From Backpropagation to Adaptive Learning Algorithms. Backtracking is removed from RPROP+. iRPROP− is defined in Rprop – Description and Implementation Details and was reinvented by Igel and Hüsken. This variant is very popular and most simple. iRPROP+ is defined at Improving the Rprop Learning Algorithm and is very robust and typically faster than the other three variants.
CPT Corporation
CPT Corporation was founded in 1971 by Dean Scheff in Minneapolis, Minnesota, with co-founders James Wienhold and Richard Eichhorn. CPT first designed, manufactured, and marketed the CPT 4200, a dual-cassette-tape machine that controlled a modified IBM Selectric typewriter to support text editing and word processing. The CPT 4200 was followed in 1976 by the CPT VM (Visual Memory), a partial-page display-screen dual-cassette-tape unit, and shortly thereafter by the CPT 8000, a full-page display dual-diskette desktop microcomputer that drove stand-alone daisy wheel printers. Subsequent products included (1) variants on the 8000 series; (2) the CPT 6000 series, which had a lower capacity, smaller screen, and was less expensive; (3) the CPT 9000 series, which had a larger capacity and could run IBM personal computer software; (4) the CPT Phoenix series, which had a graphical capabilities; (5) CPT PT, a software-only reduced version that ran on IBM personal computers and clones; and (6) other related products. The CPT logo—originally three letters chosen to sound well together—began to be taken as an acronym for "cassette powered typewriting," and subsequently for "computer processed text," and numerous other variants. Major competition was IBM, Wang, Lanier, Xerox, and other word processing vendors. CPT Corporation was fifth in size among Minnesota-based top high-tech companies, after 3M, Honeywell, Control Data, and Medtronic. Corporate revenues grew to approximately a quarter-billion dollars per year in the mid-1980s, then declined with the proliferation of personal computers. CPT ultimately ceased major manufacturing late in the 20th century. == Selected products == === Cassette based === The CPT 4200 was a dual-cassette-tape unit with a small built-in keyboard that controlled a modified IBM Selectric typewriter. Keystrokes entered on the typewriter appeared on the paper as they were recorded on the output cassette, which formed a magnetic replica of the characters printed on the page. That output cassette could later be used as an input cassette, where it would be played back to the typewriter along with new keystrokes to accomplish text editing. The keyboard of the CPT 4200 had action keys for "skip", "read" and "stop", mode keys for "word", "line", "paragraph," and "page." Pressing "read" transferred a word, line, paragraph, or page (depending on which mode key had been selected) from the input tape to both the typewriter and the output tape. Line boundaries (aka printer margins) recorded on the input tape were ignored or retained depending on whether or not the "adjust" key had been selected. Alternatively, pressing "skip" moved past the corresponding amount of text on the input tape without sending it to the typewriter or to the output tape. The Selectric's keyboard was active for any new typing, which would appear on the paper and transferred to the output tape. Thus a document was edited by reading back those parts of the text to be retained and skipping those parts to be discarded, with new typing added from the Selectric's keyboard. Price: approx. $5000, 1980-era values. The CPT Communicator was an add-on to the CPT 4200 that allowed data to be transferred from one text-editing machine to another, or between a text-editing machine and a remote computer, via phone lines. Price: not available. === Microprocessor based === ==== CPT 8000 series ==== The CPT 8000 was the company's first microcomputer product, exhibited in spring of 1976. It was a self-contained desktop machine with two 8-inch floppy diskette drives, a movable keyboard, and a full-page vertically oriented CRT display simulating paper with black characters on a white background, for a wysiwyg view of text on paper. It was promoted as familiar and easy to use for those experienced with typewriters. A keyboard with a large set of extra keys made operating the 8000 quite easy even for people without any computer skills or background. IN, OUT, PRINT, OOPS OOPS was changed thinking it was insulting to the buyer to assume they would ever make an error. The CPT 8000 was designed to show a full page of text with a static line showing the margin and tab stops. An additional line would display status or error messages with a times square like display. The times square error and status messages were very well done, "The printer needs a new ribbon" rather than "ERROR 034892". The text page could both smooth pan and scroll by the hardware in the display board and nothing quite like it existed for a very long time. The 8000 ran its own multitasking hardware interrupt-driven operating system but it also ran CP/M quite well. So unlike other companies that sold Wordprocessor only systems, CPT had a system that could run any of the many popular CP/M applications. Using the CP/M OS users could develop Fortran, CBasic, Cobol and other language's programs. The 8000 used Intel's 8080 microprocessor. The display board was bleeding-edge, high-speed logic. The parts available at this time were pushed to their limits to provide the speed needed to display this much text. There were times that batches of parts from one manufacturer simply could not be clocked as fast as the 8000 display required. Memory was initially 64K, but larger boards of 128K were most common then later 256K were offered. The 8080 accessed this additional RAM by running a custom page flipping circuit. The 8000 was originally priced at $8000 and its daisy wheel printer an additional $8000. The model number having been confused with the price at its first appearance at the Hanover fair. An RS-232 serial communication option was available for the 8000 series that allowed the electronic transfer of documents. One very popular use of this was to access the Westlaw system. A tempest approved version of the 8000 was developed that was RF tight with nothing being emitted that could be monitored or spied on. === Storage Systems === ==== CPT WordPak ==== The CPT WordPak series was CPT's first external document storage system that enabled multiple 8000 series workstations to store documents in an electronic filing cabinet. Prior to WordPak, all documents were stored on removable 8-inch floppy diskettes. Sharing documents involved handing off the original disk, or copying the document to a second disk and 'sneaker-net-ing' (walking it over) to the second 8000. But this resulted in two copies of the document, one at each workstation. A circuit board with a proprietary cable connector was installed in the 8000/6000 family of "workstations" and connected to the WordPak by a multi-conductor cable. WordPak 1 consisted of a single Shugart Associates SA4000 14"-diameter hard disk with a capacity of 30 megabytes. WordPak 2 added a 2nd drive for a total of 60 megabytes. ==== CPT SRS 45 ==== The CPT SRS 45 was what would now be called a server (quite likely the first of its kind) but in practice was much more. It was maybe the worlds easiest networking shared resource system. It combined a ZIP drive for backup and hard disk(s) that would be shared simultaneously by up to eight CPT machines that had the PC AT bus. The primary person responsible for its development was Bill Davidson whose wife Cheryl was responsible for bringing up CP/M, MP/M and other Digital Research products running on the Phoenix. The brilliance of the system were the networking cards that plugged into the individual machines. These used the 55AA installable driver of the IBM BIOS to simply add the zip and hard disk drives to each computers drives list. So a system that started with floppy drives A and B and a C hard disk on the machine would have the SRS 45 drives added as drives D (E, F depending on the number of hard disk) and Z for the zip drive. Sharing (avoiding writing to the same file at the same time) was handled by simply assigning parts of the drives for individuals and other directories for shared use. No "driver" software was needed. You simply plugged in the networking card and your machine had additional drives that were internal to the SRS45. This approach was far ahead of its time and sadly never recognized for its brilliance. The SRS45 as were all CPT machines not just dedicated Word Processors. === Personal-computer based === ==== CPT PT software ==== CPT PT was a reduced a version of the software that ran under MS-DOS as an application on IBM PC compatible computers. The corporation intended it as a bridge to allow data to flow in and out of personal computer packages, as well as providing a personal-computer word processing application for those familiar with standalone CPT equipment or who preferred the CPT style of dual-window text editing. Price: approx. $200, 1980-era values. ==== CPT Genius Display ==== The Genius display was a stand-alone, vertically-oriented (portrait) configuration monochrome grey-scale CRT monitor unit and an IBM PC form factor display card to allow high-resolution, full-page text & graphics on IBM PC compatible computers.
Information Harvesting
Information Harvesting (IH) was an early data mining product from the 1990s. It was invented by Ralphe Wiggins and produced by the Ryan Corp, later Information Harvesting Inc., of Cambridge, Massachusetts. Wiggins had a background in genetic algorithms and fuzzy logic. IH sought to infer rules from sets of data. It did this first by classifying various input variables into one of a number of bins, thereby putting some structure on the continuous variables in the input. IH then proceeds to generate rules, trading off generalization against memorization, that will infer the value of the prediction variable, possibly creating many levels of rules in the process. It included strategies for checking if overfitting took place and, if so, correcting for it. Because of its strategies for correcting for overfitting by considering more data, and refining the rules based on that data, IH might also be considered to be a form of machine learning. The advantage of IH, as compared with other data mining products of its time and even later, was that it provided a mechanism for finding multiple rules that would classify the data and determining, according to set criteria, the best rules to use.
Probit model
In statistics, a probit model is a type of regression where the dependent variable can take only two values, for example married or not married. The word is a portmanteau, coming from probability + unit. The purpose of the model is to estimate the probability that an observation with particular characteristics will fall into a specific one of the categories; moreover, classifying observations based on their predicted probabilities is a type of binary classification model. A probit model is a popular specification for a binary response model. As such it treats the same set of problems as does logistic regression using similar techniques. When viewed in the generalized linear model framework, the probit model employs a probit link function. It is most often estimated using the maximum likelihood procedure, such an estimation being called a probit regression. == Conceptual framework == Suppose a response variable Y is binary, that is it can have only two possible outcomes which we will denote as 1 and 0. For example, Y may represent presence/absence of a certain condition, success/failure of some device, answer yes/no on a survey, etc. We also have a vector of regressors X, which are assumed to influence the outcome Y. Specifically, we assume that the model takes the form P ( Y = 1 ∣ X ) = Φ ( X T β ) , {\displaystyle P(Y=1\mid X)=\Phi (X^{\operatorname {T} }\beta ),} where P is the probability and Φ {\displaystyle \Phi } is the cumulative distribution function (CDF) of the standard normal distribution. The parameters β are typically estimated by maximum likelihood. It is possible to motivate the probit model as a latent variable model. Suppose there exists an auxiliary random variable Y ∗ = X T β + ε , {\displaystyle Y^{\ast }=X^{T}\beta +\varepsilon ,} where ε ~ N(0, 1). Then Y can be viewed as an indicator for whether this latent variable is positive: Y = { 1 Y ∗ > 0 0 otherwise } = { 1 X T β + ε > 0 0 otherwise } {\displaystyle Y=\left.{\begin{cases}1&Y^{}>0\\0&{\text{otherwise}}\end{cases}}\right\}=\left.{\begin{cases}1&X^{\operatorname {T} }\beta +\varepsilon >0\\0&{\text{otherwise}}\end{cases}}\right\}} The use of the standard normal distribution causes no loss of generality compared with the use of a normal distribution with an arbitrary mean and standard deviation, because adding a fixed amount to the mean can be compensated by subtracting the same amount from the intercept, and multiplying the standard deviation by a fixed amount can be compensated by multiplying the weights by the same amount. To see that the two models are equivalent, note that P ( Y = 1 ∣ X ) = P ( Y ∗ > 0 ) = P ( X T β + ε > 0 ) = P ( ε > − X T β ) = P ( ε < X T β ) by symmetry of the normal distribution = Φ ( X T β ) {\displaystyle {\begin{aligned}P(Y=1\mid X)&=P(Y^{\ast }>0)\\&=P(X^{\operatorname {T} }\beta +\varepsilon >0)\\&=P(\varepsilon >-X^{\operatorname {T} }\beta )\\&=P(\varepsilon
Multidimensional analysis
In statistics, econometrics and related fields, multidimensional analysis (MDA) is a data analysis process that groups data into two categories: data dimensions and measurements. For example, a data set consisting of the number of wins for a single football team at each of several years is a single-dimensional (in this case, longitudinal) data set. A data set consisting of the number of wins for several football teams in a single year is also a single-dimensional (in this case, cross-sectional) data set. A data set consisting of the number of wins for several football teams over several years is a two-dimensional data set. == Higher dimensions == In many disciplines, two-dimensional data sets are also called panel data. While, strictly speaking, two- and higher-dimensional data sets are "multi-dimensional", the term "multidimensional" tends to be applied only to data sets with three or more dimensions. For example, some forecast data sets provide forecasts for multiple target periods, conducted by multiple forecasters, and made at multiple horizons. The three dimensions provide more information than can be gleaned from two-dimensional panel data sets. == Software == Computer software for MDA include Online analytical processing (OLAP) for data in relational databases, pivot tables for data in spreadsheets, and Array DBMSs for general multi-dimensional data (such as raster data) in science, engineering, and business.
Semantic space
Semantic spaces in the natural language domain aim to create representations of natural language that are capable of capturing meaning. The original motivation for semantic spaces stems from two core challenges of natural language: Vocabulary mismatch (the fact that the same meaning can be expressed in many ways) and ambiguity of natural language (the fact that the same term can have several meanings). The application of semantic spaces in natural language processing (NLP) aims at overcoming limitations of rule-based or model-based approaches operating on the keyword level. The main drawback with these approaches is their brittleness, and the large manual effort required to create either rule-based NLP systems or training corpora for model learning. Rule-based and machine learning based models are fixed on the keyword level and break down if the vocabulary differs from that defined in the rules or from the training material used for the statistical models. Research in semantic spaces dates back more than 20 years. In 1996, two papers were published that raised a lot of attention around the general idea of creating semantic spaces: latent semantic analysis and Hyperspace Analogue to Language. However, their adoption was limited by the large computational effort required to construct and use those semantic spaces. A breakthrough with regard to the accuracy of modelling associative relations between words (e.g. "spider-web", "lighter-cigarette", as opposed to synonymous relations such as "whale-dolphin", "astronaut-driver") was achieved by explicit semantic analysis (ESA) in 2007. ESA was a novel (non-machine learning) based approach that represented words in the form of vectors with 100,000 dimensions (where each dimension represents an Article in Wikipedia). However practical applications of the approach are limited due to the large number of required dimensions in the vectors. More recently, advances in neural network techniques in combination with other new approaches (tensors) led to a host of new recent developments: Word2vec from Google, GloVe from Stanford University, and fastText from Facebook AI Research (FAIR) labs.
Concept class
In computational learning theory in mathematics, a concept over a domain X is a total Boolean function over X. A concept class is a class of concepts. Concept classes are a subject of computational learning theory. Concept class terminology frequently appears in model theory associated with probably approximately correct (PAC) learning. In this setting, if one takes a set Y as a set of (classifier output) labels, and X is a set of examples, the map c : X → Y {\displaystyle c:X\to Y} , i.e. from examples to classifier labels (where Y = { 0 , 1 } {\displaystyle Y=\{0,1\}} and where c is a subset of X), c is then said to be a concept. A concept class C {\displaystyle C} is then a collection of such concepts. Given a class of concepts C, a subclass D is reachable if there exists a sample s such that D contains exactly those concepts in C that are extensions to s. Not every subclass is reachable. == Background == A sample s {\displaystyle s} is a partial function from X {\displaystyle X} to { 0 , 1 } {\displaystyle \{0,1\}} . Identifying a concept with its characteristic function mapping X {\displaystyle X} to { 0 , 1 } {\displaystyle \{0,1\}} , it is a special case of a sample. Two samples are consistent if they agree on the intersection of their domains. A sample s ′ {\displaystyle s'} extends another sample s {\displaystyle s} if the two are consistent and the domain of s {\displaystyle s} is contained in the domain of s ′ {\displaystyle s'} . == Examples == Suppose that C = S + ( X ) {\displaystyle C=S^{+}(X)} . Then: the subclass { { x } } {\displaystyle \{\{x\}\}} is reachable with the sample s = { ( x , 1 ) } {\displaystyle s=\{(x,1)\}} ; the subclass S + ( Y ) {\displaystyle S^{+}(Y)} for Y ⊆ X {\displaystyle Y\subseteq X} are reachable with a sample that maps the elements of X − Y {\displaystyle X-Y} to zero; the subclass S ( X ) {\displaystyle S(X)} , which consists of the singleton sets, is not reachable. == Applications == Let C {\displaystyle C} be some concept class. For any concept c ∈ C {\displaystyle c\in C} , we call this concept 1 / d {\displaystyle 1/d} -good for a positive integer d {\displaystyle d} if, for all x ∈ X {\displaystyle x\in X} , at least 1 / d {\displaystyle 1/d} of the concepts in C {\displaystyle C} agree with c {\displaystyle c} on the classification of x {\displaystyle x} . The fingerprint dimension F D ( C ) {\displaystyle FD(C)} of the entire concept class C {\displaystyle C} is the least positive integer d {\displaystyle d} such that every reachable subclass C ′ ⊆ C {\displaystyle C'\subseteq C} contains a concept that is 1 / d {\displaystyle 1/d} -good for it. This quantity can be used to bound the minimum number of equivalence queries needed to learn a class of concepts according to the following inequality: F D ( C ) − 1 ≤ # E Q ( C ) ≤ ⌈ F D ( C ) ln ( | C | ) ⌉ {\textstyle FD(C)-1\leq \#EQ(C)\leq \lceil FD(C)\ln(|C|)\rceil } .