Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems this reduces the very high computational burden, achieving faster iterations in exchange for a lower convergence rate. The basic idea behind stochastic approximation can be traced back to the Robbins–Monro algorithm of the 1950s. Today, stochastic gradient descent has become an important optimization method in machine learning. == Background == Both statistical estimation and machine learning consider the problem of minimizing an objective function that has the form of a sum: Q ( w ) = 1 n ∑ i = 1 n Q i ( w ) , {\displaystyle Q(w)={\frac {1}{n}}\sum _{i=1}^{n}Q_{i}(w),} where the parameter w {\displaystyle w} that minimizes Q ( w ) {\displaystyle Q(w)} is to be estimated. Each summand function Q i {\displaystyle Q_{i}} is typically associated with the i {\displaystyle i} -th observation in the data set (used for training). In classical statistics, sum-minimization problems arise in least squares and in maximum-likelihood estimation (for independent observations). The general class of estimators that arise as minimizers of sums are called M-estimators. However, in statistics, it has been long recognized that requiring even local minimization is too restrictive for some problems of maximum-likelihood estimation. Therefore, contemporary statistical theorists often consider stationary points of the likelihood function (or zeros of its derivative, the score function, and other estimating equations). The sum-minimization problem also arises for empirical risk minimization. There, Q i ( w ) {\displaystyle Q_{i}(w)} is the value of the loss function at i {\displaystyle i} -th example, and Q ( w ) {\displaystyle Q(w)} is the empirical risk. When used to minimize the above function, a standard (or "batch") gradient descent method would perform the following iterations: w := w − η ∇ Q ( w ) = w − η n ∑ i = 1 n ∇ Q i ( w ) . {\displaystyle w:=w-\eta \,\nabla Q(w)=w-{\frac {\eta }{n}}\sum _{i=1}^{n}\nabla Q_{i}(w).} The step size is denoted by η {\displaystyle \eta } (sometimes called the learning rate in machine learning) and here " := {\displaystyle :=} " denotes the update of a variable in the algorithm. In many cases, the summand functions have a simple form that enables inexpensive evaluations of the sum-function and the sum gradient. For example, in statistics, one-parameter exponential families allow economical function-evaluations and gradient-evaluations. However, in other cases, evaluating the sum-gradient may require expensive evaluations of the gradients from all summand functions. When the training set is enormous and no simple formulas exist, evaluating the sums of gradients becomes very expensive, because evaluating the gradient requires evaluating all the summand functions' gradients. To economize on the computational cost at every iteration, stochastic gradient descent samples a subset of summand functions at every step. This is very effective in the case of large-scale machine learning problems. == Iterative method == In stochastic (or "on-line") gradient descent, the true gradient of Q ( w ) {\displaystyle Q(w)} is approximated by a gradient at a single sample: w := w − η ∇ Q i ( w ) . {\displaystyle w:=w-\eta \,\nabla Q_{i}(w).} As the algorithm sweeps through the training set, it performs the above update for each training sample. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles. Typical implementations may use an adaptive learning rate so that the algorithm converges. In pseudocode, stochastic gradient descent can be presented as : A compromise between computing the true gradient and the gradient at a single sample is to compute the gradient against more than one training sample (called a "mini-batch") at each step. This can perform significantly better than "true" stochastic gradient descent described, because the code can make use of vectorization libraries rather than computing each step separately as was first shown in where it was called "the bunch-mode back-propagation algorithm". It may also result in smoother convergence, as the gradient computed at each step is averaged over more training samples. The convergence of stochastic gradient descent has been analyzed using the theories of convex minimization and of stochastic approximation. Briefly, when the learning rates η {\displaystyle \eta } decrease with an appropriate rate, and subject to relatively mild assumptions, stochastic gradient descent converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum. This is in fact a consequence of the Robbins–Siegmund theorem. == Linear regression == Suppose we want to fit a straight line y ^ = w 1 + w 2 x {\displaystyle {\hat {y}}=w_{1}+w_{2}x} to a training set with observations ( ( x 1 , y 1 ) , ( x 2 , y 2 ) … , ( x n , y n ) ) {\displaystyle ((x_{1},y_{1}),(x_{2},y_{2})\ldots ,(x_{n},y_{n}))} and corresponding estimated responses ( y ^ 1 , y ^ 2 , … , y ^ n ) {\displaystyle ({\hat {y}}_{1},{\hat {y}}_{2},\ldots ,{\hat {y}}_{n})} using least squares. The objective function to be minimized is Q ( w ) = ∑ i = 1 n Q i ( w ) = ∑ i = 1 n ( y ^ i − y i ) 2 = ∑ i = 1 n ( w 1 + w 2 x i − y i ) 2 . {\displaystyle Q(w)=\sum _{i=1}^{n}Q_{i}(w)=\sum _{i=1}^{n}\left({\hat {y}}_{i}-y_{i}\right)^{2}=\sum _{i=1}^{n}\left(w_{1}+w_{2}x_{i}-y_{i}\right)^{2}.} The last line in the above pseudocode for this specific problem will become: [ w 1 w 2 ] ← [ w 1 w 2 ] − η [ ∂ ∂ w 1 ( w 1 + w 2 x i − y i ) 2 ∂ ∂ w 2 ( w 1 + w 2 x i − y i ) 2 ] = [ w 1 w 2 ] − η [ 2 ( w 1 + w 2 x i − y i ) 2 x i ( w 1 + w 2 x i − y i ) ] . {\displaystyle {\begin{bmatrix}w_{1}\\w_{2}\end{bmatrix}}\leftarrow {\begin{bmatrix}w_{1}\\w_{2}\end{bmatrix}}-\eta {\begin{bmatrix}{\frac {\partial }{\partial w_{1}}}(w_{1}+w_{2}x_{i}-y_{i})^{2}\\{\frac {\partial }{\partial w_{2}}}(w_{1}+w_{2}x_{i}-y_{i})^{2}\end{bmatrix}}={\begin{bmatrix}w_{1}\\w_{2}\end{bmatrix}}-\eta {\begin{bmatrix}2(w_{1}+w_{2}x_{i}-y_{i})\\2x_{i}(w_{1}+w_{2}x_{i}-y_{i})\end{bmatrix}}.} Note that in each iteration or update step, the gradient is only evaluated at a single x i {\displaystyle x_{i}} . This is the key difference between stochastic gradient descent and batched gradient descent. In general, given a linear regression y ^ = ∑ k ∈ 1 : m w k x k {\displaystyle {\hat {y}}=\sum _{k\in 1:m}w_{k}x_{k}} problem, stochastic gradient descent behaves differently when m < n {\displaystyle m ArcObjects is a development environment of the ArcGIS family of applications. Using Visual Basic for Applications, C# or Java SDK for ArcGIS, it allows developers to extend these applications.ArcObjects is a library of Component Object Model (COM) components that build up the foundation of Esri's ArcGIS platform. ArcObjects is written primarily in the C++ programming language. Since ArcGIS is completely built on top of ArcObjects, the ArcGIS platform can be fully customized and extended by making use of its COM services and capabilities. This allows for easy extension of the ArcObjects data model with any programming language that is compatible with COM, such as Visual Basic, C#, Visual Basic.NET, Java and Python. COM enables components to be reused at a binary level, meaning developers do not require access to the source code of ArcObjects in order to extend the ArcGIS platform. For this reason, an ArcObjects programmer can make use of any type inside the ArcObjects system without knowing the implementation details of the type, only needing to know what the type is able to do. The ArcObjects data model is based on the COM standard, which makes it compatible with other COM objects and applications. This allows for easy integration and collaboration with other systems that are also based on the COM standard. The ArcGIS platform was built using ArcObjects types, such as classes, interfaces, and enumerations. ArcObjects use COM interfaces to organize and communicate properties and methods of its classes, ensuring compatibility with other COM-based objects and systems. When working with an ArcObjects COM class, its properties and methods are accessed solely through one of its implemented interfaces via the process of Query Interface (QI). Multiple interfaces are commonly available for classes in ArcObjects. For example, it is possible to query for additional interfaces implemented by an object after instantiation via the process of QI. Although only one interface can be used when instantiating an object, multiple interfaces are often available for classes in ArcObjects, allowing for greater flexibility and compatibility with other systems based on the COM standard. Marcus Hutter (born 14 April 1967 in Munich) is a German computer scientist, professor and artificial intelligence researcher. As a senior researcher at DeepMind, he studies the mathematical foundations of artificial general intelligence. Hutter studied physics and computer science at the Technical University of Munich. In 2000, he joined Jürgen Schmidhuber's group at the Dalle Molle Institute for Artificial Intelligence Research in Manno, Switzerland. He developed a mathematical formalism of artificial general intelligence named AIXI. He has served as a professor at the College of Engineering, Computing and Cybernetics of the Australian National University in Canberra, Australia. == Research == Starting in 2000, Hutter developed and published a mathematical theory of artificial general intelligence, AIXI, based on idealised intelligent agents and reward-motivated reinforcement learning. His first book Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability was published in 2005 by Springer. Also in 2005, Hutter published with his doctoral student Shane Legg an intelligence test for artificial intelligence devices. In 2009, Hutter developed and published the theory of feature reinforcement learning. In 2014, Lattimore and Hutter published an asymptotically optimal extension of the AIXI agent. An accessible podcast with Lex Fridman about his theory of Universal AI appeared in 2021 and a more technical follow-up with Tim Nguyen in 2024 in the Cartesian Cafe. His new (2024) book also gives a more accessible introduction to Universal AI and progress in the 20 years since his first book, including a chapter on ASI safety, which featured as a keynote at the inaugural workshop on AI safety in Sydney. == Hutter Prize == In 2006, Hutter announced the Hutter Prize for Lossless Compression of Human Knowledge, with a total of €50,000 in prize money. In 2020, Hutter raised the prize money for the Hutter Prize to €500,000. In search of the best AI analytics tool? An AI analytics tool is software that uses machine learning to help you get more done — it turns a rough idea into a polished result in seconds. When choosing one, weigh output quality, pricing, export formats, and how well it fits the tools you already use. Whether you are a beginner or a pro, the right AI analytics tool slots into your workflow and pays for itself fast. We tested the leading options and ranked them by quality, value, and ease of use. Helsinki Finite-State Technology (HFST) is a computer programming library and set of utilities for natural language processing with finite-state automata and finite-state transducers. It is free and open-source software, released under a mix of the GNU General Public License version 3 (GPLv3) and the Apache License. == Features == The library functions as an interchanging interface to multiple backends, such as OpenFST, foma and SFST. The utilities comprise various compilers, such as hfst-twolc (a compiler for morphological two-level rules), hfst-lexc (a compiler for lexicon definitions) and hfst-regexp2fst (a regular expression compiler). Functions from Xerox's proprietary scripting language xfst is duplicated in hfst-xfst, and the pattern matching utility pmatch in hfst-pmatch, which goes beyond the finite-state formalism in having recursive transition networks (RTNs). The library and utilities are written in C++, with an interface to the library in Python and a utility for looking up results from transducers ported to Java and Python. Transducers in HFST may incorporate weights depending on the backend. For performing FST operations, this is currently only possible via the OpenFST backend. HFST provides two native backends, one designed for fast lookup (hfst-optimized-lookup), the other for format interchange. Both of them can be weighted. == Uses == HFST has been used for writing various linguistic tools, such as spell-checkers, hyphenators, and morphologies. Morphological dictionaries written in other formalisms have also been converted to HFST's formats. A number of vector graphics editors exist for various platforms. Potential users of these editors will make comparisons based on factors such as the availability for the user's platform, the software license, the feature set, the merits of the user interface (UI) and the focus of the program. Some programs are more suitable for artistic work while others are better for technical drawings. Another important factor is the application's support of various vector and bitmap image formats for import and export. The tables in this article compare general and technical information for a number of vector graphics editors. See the article on each editor for further information. This article is neither all-inclusive nor necessarily up-to-date. == Some editors in detail == Adobe Fireworks (formerly Macromedia Fireworks) is a vector editor with bitmap editing capabilities with its main purpose being the creation of graphics for Web and screen. Fireworks supports RGB color scheme and has no CMYK support. This means it is mostly used for screen design. The native Fireworks file format is editable PNG (FWPNG or PNG). Adobe Fireworks has a competitive price, but its features can seem limited in comparison with other products. It is easier to learn than other products and can produce complex vector artwork. The Fireworks editable PNG file format is not supported by other Adobe products. Fireworks can manage the PSD and AI file formats which enables it to be integrated with other Adobe apps. Fireworks can also open FWPNG/PNG, PSD, AI, EPS, JPG, GIF, BMP, TIFF file formats, and save/export to FWPNG/PNG, PSD, AI (v.8), FXG (v.2.0), JPG, GIF, PDF, SWF and some others. Some support for exporting to SVG is available via a free Export extension. On May 6, 2013, Adobe announced that Fireworks would be phased out. Adobe Flash (formerly a Macromedia product) has straightforward vector editing tools that make it easier for designers and illustrators to use. The most important of these tools are vector lines and fills with bitmap-like selectable areas, simple modification of curves via the "selection" or the control points/handles through "direct selection" tools. Flash uses Actionscript for OOP, and has full XML functionality through E4X support. Adobe FreeHand (formerly Macromedia Freehand and Aldus Freehand) is mainly used by professional graphic designers. The functionality of FreeHand includes the flexibility of the application in the wide design environment, catering to the output needs of both traditional image reproduction methods and to contemporary print and digital media with its page-layout capabilities and text attribute controls. Specific functions of FreeHand include a superior image-tracing operation for vector editing, page layout features within multiple-page documents, and embedding custom print-settings (such as variable halftone-screen specifications within a single graphic, etc.) to each document independent of auxiliary printer-drivers. User-operation is considered to be more suited for designers with an artistic background compared to designers with a technical background. When being marketed, FreeHand lacked the promotional backing, development and PR support in comparison to other similar products. FreeHand was transferred to the classic print group after Macromedia was purchased by Adobe in 2005. On May 16, 2007, Adobe announced that no further updates to Freehand would be developed but continues to sell FreeHand MX as a Macromedia product. FreeHand continues to run on Mac OS X Snow Leopard (using an Adobe fix) and on Windows 7. For macOS, Affinity Designer is able to open version 10 & MX Freehand files. Adobe Illustrator is a commonly used editor because of Adobe's market dominance, but is more expensive than other similar products. It is primarily developed consistently in line with other Adobe products and is best integrated with Adobe's Creative Suite packages. The ai file format is proprietary, but some vector editors can open and save in that format. Illustrator imports over two dozen formats, including PSD, PDF and SVG, and exports AI, PDF, SVG, SVGZ, GIF, JPG, PNG, WBMP, and SWF. However, the user must be aware of unchecking the "Preserve Illustrator Editing Capabilities" option if generating interoperable SVG files is desired. Affinity Designer by Serif Europe (the successor to their previous product, DrawPlus) is non-subscription-based software that is often described as an alternative to Adobe Illustrator. The application can open Portable Document Format (PDF), Adobe Photoshop, and Adobe Illustrator files, as well as export to those formats and to the Scalable Vector Graphics (SVG) and Encapsulated PostScript (EPS) formats. It also supports import from some Adobe Freehand files (specifically versions 10 & MX). Apache OpenOffice Draw is the vector graphics editor of the Apache OpenOffice open source office suite. It supports many import and export file formats and is available for multiple desktop operating systems. Boxy SVG is a chromium-based vector graphics editor for creating illustrations, as well as logos, icons, and other elements of graphic design. It is primarily focused on editing drawings in the SVG file format. The program is available as both a web app and a desktop application for Windows, macOS, ChromeOS, and Linux-based operating systems. Collabora Online Draw is the vector graphics editor of the Collabora Online open source office suite. It supports many import and export file formats and is accessible via any modern web browser, it also supports desktop editing features, Collabora Office is available for desktop and mobile operating systems, it is the enterprise ready version of LibreOffice. ConceptDraw PRO is a business diagramming tool and vector graphics editor available for both Windows and macOS. It supports multi-page documents, and includes an integrated presentation mode. ConceptDraw PRO supports imports and exports several formats, including Microsoft Visio and Microsoft PowerPoint. Corel Designer (originally Micrografx Designer) is one of the earliest vector-based graphics editors for the Microsoft Windows platform. The product is mainly used for the creation of engineering drawings and is shipped with extensive libraries for the needs of engineers. It is also flexible enough for most vector graphics design applications. CorelDRAW is an editor used in the graphic design, sign making and fashion design industries. CorelDRAW is capable of limited interoperation by reading file formats from Adobe Illustrator. CorelDRAW has over 50 import and export filters, on-screen and dialog box editing and the ability to create multi-page documents. It can also generate TrueType and Type 1 fonts, although refined typographic control is better suited to a more specific application. Some other features of CorelDRAW include the creation and execution of VBA macros, viewing of colour separations in print preview mode and integrated professional imposing options. Dia is a free and open-source diagramming and vector graphics editor available for Windows, Linux and other Unix-based computer operating systems. Dia has a modular design and several shape packages for flowcharting, network diagrams and circuit diagrams. Its design was inspired by Microsoft Visio, although it uses a Single Document Interface similar to other GNOME software (such as GIMP). DrawPlus, first built for the Windows platform in 1993, has matured into a full featured vector graphics editor for home and professional users. Also available as a feature-limited free 'starter edition': DrawPlus SE. DrawPlus developers, Serif Europe, have now ceased its development in order to focus on its successor, Affinity Designer. Edraw Max is a cross-platform diagram software and vector graphics editor available for Windows, Mac and Linux. It supports kinds of diagram types. It supports imports and exports SVG, PDF, HTML, Multiple page TIFF, Microsoft Visio and Microsoft PowerPoint. Embroidermodder is a free machine embroidery software tool that supports a variety of formats and allows the user to add custom modifications to their embroidery designs. Fatpaint is a free, light-weight, browser-based graphic design application with built-in vector drawing tools. It can be accessed through any browser with Flash 9 installed. Its integration with Zazzle makes it particularly suitable for people who want to create graphics for custom printed products such as T-shirts, mugs, iPhone cases, flyers and other promotional products. Figma is a collaborative web-based online vector graphics editor, used primarily for UX design and prototyping. GIMP, which works mainly with raster images, offers a limited set of features to create and record SVG files. It can also load and handle SVG files created with other software like Inkscape. Inkscape is a free and open-source vector editor with the primary native format being SVG. Inkscape is available for Linux, Windows, Mac OS X, and Judea Pearl (Hebrew: יהודה פרל; born September 4, 1936) is an Israeli-American electrical engineer, computer scientist and philosopher, best known for championing the probabilistic approach to artificial intelligence and the development of Bayesian networks (see the article on belief propagation). He is also credited for developing a theory of causal and counterfactual inference based on structural models (see article on causality). In 2011, the Association for Computing Machinery (ACM) awarded Pearl with the Turing Award, the highest distinction in computer science, "for fundamental contributions to artificial intelligence through the development of a calculus for probabilistic and causal reasoning". He is the author of several books, including the technical Causality: Models, Reasoning and Inference, and The Book of Why, a book on causality aimed at the general public. Judea Pearl is the father of journalist Daniel Pearl, who was kidnapped and murdered by terrorists in Pakistan connected with Al-Qaeda and the International Islamic Front in 2002. == Biography == Judea Pearl was born in Tel Aviv, British Mandate for Palestine, in 1936 to Eliezer and Tova Pearl, who were Polish Jewish immigrants, grew up in Bnei Brak. His grandfather Chaim Pearl was one of Bnei Brak's founders. He is a descendant of Menachem Mendel of Kotzk on his mother's side. After serving in the Israel Defense Forces and joining a kibbutz, Pearl decided to study engineering in 1956. He received a B.S. in electrical engineering from the Technion 1960. That same year, he emigrated to the United States and pursued graduate studies. He received an M.S. in electrical engineering from the Newark College of Engineering (now New Jersey Institute of Technology) in 1961, and went on to receive an M.S. in physics from Rutgers University and a PhD in electrical engineering from the Polytechnic Institute of Brooklyn (now the New York University Tandon School of Engineering) in 1965. He worked at RCA Research Laboratories (now SRI International) in Princeton, New Jersey on superconductive parametric amplifiers and storage devices and at Electronic Memories, Inc., on advanced memory systems. When semiconductors "wiped out" Pearl's work, as he later expressed it, he joined UCLA's School of Engineering in 1970 and started work on probabilistic artificial intelligence. He is one of the founding editors of the Journal of Causal Inference. Pearl is currently a professor of computer science and statistics and director of the Cognitive Systems Laboratory at UCLA. He and his wife, Ruth, had three children. In addition, as of 2011, he is a member of the International Advisory Board of NGO Monitor. Former Israeli Chief Rabbi, Rabbi Yisrael Meir Lau, partnered with Judea Pearl in the documentary With My Whole Broken Heart. == Murder of Daniel Pearl == In 2002, his son, Daniel Pearl, a journalist working for the Wall Street Journal was kidnapped and murdered in Pakistan, leading Judea and the other members of the family and friends to create the Daniel Pearl Foundation. On the seventh anniversary of Daniel's death, Judea wrote an article in the Wall Street Journal titled Daniel Pearl and the Normalization of Evil: When will our luminaries stop making excuses for terror?. Emeritus Chief Rabbi Jonathan Sacks quoted Judea Pearl's beliefs in a lesson on Judaism: "I asked Judea Pearl, father of the murdered journalist Daniel Pearl, why he was working for reconciliation between Jews and Muslims...he replied with heartbreaking lucidity, 'Hate killed my son. Therefore I am determined to fight hate.'" == Views == On his religious views, Pearl states that he is a "practicing disbeliever." He is very connected to Jewish traditions such as holidays and kiddush on Friday night. Pearl sits on the NGO Monitor international advisory board, a right-wing organization based in Jerusalem that reports on non-governmental organization activity from a pro-Israel perspective. == Research == Pearl is credited for "laying the foundations of modern artificial intelligence, so computer systems can process uncertainty and relate causes to effects." He is one of the pioneers of Bayesian networks and the probabilistic approach to artificial intelligence, and one of the first to mathematize causal modeling in the empirical sciences. His work is also intended as a high-level cognitive model. He is interested in the philosophy of science, knowledge representation, nonstandard logics, and learning. Pearl is described as "one of the giants in the field of artificial intelligence" by UCLA computer science professor Richard E. Korf. His work on causality has "revolutionized the understanding of causality in statistics, psychology, medicine and the social sciences" according to the Association for Computing Machinery. === Notable contributions === A summary of Pearl's scientific contributions is available in a chronological account authored by Stuart J. Russell (2012). An annotated bibliography of Pearl's contributions was compiled by the ACM in 2012. A video describing Pearl's major contributions to AI is available here. Pearl's opinion pieces, touching on Jewish identity, the war on terrorism, and the Middle East conflict can be accessed here. === Books === Heuristics, Addison-Wesley, 1984 Probabilistic Reasoning in Intelligent Systems, Morgan-Kaufmann, 1988 Pearl, Judea (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press. I Am Jewish: Personal Reflections Inspired by the Last Words of Daniel Pearl, Jewish Lights, 2004. (Winner of a 2004 National Jewish Book Award) Causal Inference in Statistics: A Primer, (with Madelyn Glymour and Nicholas Jewell), Wiley, 2016. ISBN 978-1-119-18684-7 A previous survey: Causal inference in statistics: An overview, Statistics Surveys, 3:96–146, 2009. Pearl, Judea; Dana Mackenzie (2018). "The Book of Why: The New Science of Cause and Effect". Science. 361 (6405): 855. Bibcode:2018Sci...361..855.. doi:10.1126/science.aau9731. === Awards ===ArcObjects
Marcus Hutter
AI Analytics Tools: Free vs Paid (2026)
HFST
Comparison of vector graphics editors
Judea Pearl