AI Assistant Maker

AI Assistant Maker — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

IPUMS

IPUMS, originally the Integrated Public Use Microdata Series, is the world's largest individual-level population database. IPUMS consists of microdata samples from United States (IPUMS-USA) and international (IPUMS-International) census records, as well as data from U.S. and international surveys. The records are converted into a consistent format and made available to researchers through a web-based data dissemination and analysis system. IPUMS is housed at the Institute for Social Research and Data Innovation (ISRDI), an interdisciplinary research center at the University of Minnesota, under the direction of Professor Steven Ruggles. == Description == IPUMS includes all persons enumerated in the United States censuses from 1850 to 1950 (though, the 1890 census is missing because it was destroyed in a fire) and from the American Community Survey since 2000 and the Current Population Survey since 1962. IPUMS includes household-level data for United States Censuses from 1790 to 1840, due to the first six censuses only including the name of the head of household, with tallied household totals following. IPUMS provides consistent variable names, coding schemes, and documentation across all the samples, facilitating the analysis of long-term change. IPUMS-International includes countries from Africa, Asia, Europe, and Latin America for 1960 forward. The database currently includes more than a billion individuals enumerated in 365 censuses from 94 countries around the world. IPUMS-International converts census microdata for multiple countries into a consistent format, allowing for comparisons across countries and time periods. Special efforts are made to simplify use of the data while losing no meaningful information. Comprehensive documentation is provided in a coherent form to facilitate comparative analyses of social and economic change. Additional databases in the IPUMS family include the: North Atlantic Population Project (NAPP) IPUMS National Historical Geographic Information System (NHGIS) IPUMS Health Surveys IPUMS Global Health IPUMS Time Use The Journal of American History described the effort as "One of the great archival projects of the past two decades." Liens Socio, the French portal for the social sciences, gave IPUMS the only “best site” designation that has gone to any non-French website, writing “IPUMS est un projet absolument extraordinaire...époustouflante [mind-blowing]!” The official motto of IPUMS is "use it for good, never for evil." All public IPUMS data and documentation are available online free of charge.
Read more →
Physical schema

A physical data model (or database design) is a representation of a data design as implemented, or intended to be implemented, in a database management system. In the lifecycle of a project it typically derives from a logical data model, though it may be reverse-engineered from a given database implementation. A complete physical data model will include all the database artifacts required to create relationships between tables or to achieve performance goals, such as indexes, constraint definitions, linking tables, partitioned tables or clusters. Analysts can usually use a physical data model to calculate storage estimates; it may include specific storage allocation details for a given database system. As of 2012 seven main databases dominate the commercial marketplace: Informix, Oracle, Postgres, SQL Server, Sybase, IBM Db2 and MySQL. Other RDBMS systems tend either to be legacy databases or used within academia such as universities or further education colleges. Physical data models for each implementation would differ significantly, not least due to underlying operating-system requirements that may sit underneath them. For example: SQL Server runs only on Microsoft Windows operating-systems (Starting with SQL Server 2017, SQL Server runs on Linux. It's the same SQL Server database engine, with many similar features and services regardless of your operating system), while Oracle and MySQL can run on Solaris, Linux and other UNIX-based operating-systems as well as on Windows. This means that the disk requirements, security requirements and many other aspects of a physical data model will be influenced by the RDBMS that a database administrator (or an organization) chooses to use. == Physical schema == Physical schema is a term used in data management to describe how data is to be represented and stored (files, indices, etc.) in secondary storage using a particular database management system (DBMS) (e.g., Oracle RDBMS, Sybase SQL Server, etc.). In the ANSI/SPARC Architecture three schema approach, the internal schema is the view of data that involved data management technology. This is as opposed to an external schema that reflects an individual's view of the data, or the conceptual schema that is the integration of a set of external schemas. The logical schema was the way data were represented to conform to the constraints of a particular approach to database management. At that time the choices were hierarchical and network. Describing the logical schema, however, still did not describe how physically data would be stored on disk drives. That is the domain of the physical schema. Now logical schemas describe data in terms of relational tables and columns, object-oriented classes, and XML tags. A single set of tables, for example, can be implemented in numerous ways, up to and including an architecture where table rows are maintained on computers in different countries.
Read more →
PL/Perl

PL/Perl (Procedural Language/Perl) is a procedural language supported by the PostgreSQL RDBMS. PL/Perl, as an imperative programming language, allows more control than the relational algebra of SQL. Programs created in the PL/Perl language are called functions and can use most of the features that the Perl programming language provides, including common flow control structures and syntax that has incorporated regular expressions directly. These functions can be evaluated as part of a SQL statement, or in response to a trigger or rule. The design goals of PL/Perl were to create a loadable procedural language that: can be used to create functions and trigger procedures, adds control structures to the SQL language, can perform complex computations, can be defined to be either trusted or untrusted by the server, is easy to use. PL/Perl is one of many "PL" languages available for PostgreSQL PL/pgSQL PL/Java, plPHP, PL/Python, PL/R, PL/Ruby, PL/sh, and PL/Tcl.
Read more →
Apple Intelligence

Apple Intelligence is a generative artificial intelligence system developed by Apple Inc. Relying on a combination of on-device and server processing, it was announced on June 10, 2024, at the 2024 Worldwide Developers Conference, as a built-in feature of Apple's iOS 18, iPadOS 18, and macOS Sequoia, which were announced alongside Apple Intelligence. Apple Intelligence is free for all users with supported devices. On macOS, Apple Intelligence is available only on Apple silicon Mac computers; Intel-based Mac computers are not supported. Features include writing tools that assist users with grammar and proofreading, image generation, summaries of system notifications, AI-assisted image retouching in the Photos app, and integration with ChatGPT, the popular chatbot by OpenAI. As of March 2026, Apple Intelligence is not available yet on devices purchased in mainland China or on any device using an Apple Account set to mainland China, even if the device was bought elsewhere. == History == === Background === Apple first implemented artificial intelligence features in its products with the release of Siri in the iPhone 4S in 2011. In the years after its release, Apple engaged in efforts to ensure its artificial intelligence operations remained covert; according to University of California, Berkeley professor Trevor Darrell, the company's secrecy deterred graduate students. The company started expanding its artificial intelligence team in 2015, opening up its operations by publishing more scientific papers and joining AI industry research groups. Apple reportedly acquired more AI companies from 2016 to 2020. In 2017, Apple released the iPhone 8 and the iPhone X with the A11 Bionic processor, which featured its first dedicated Neural Engine for accelerating common machine learning tasks. Despite its investments in artificial intelligence, Siri was criticized both by reviewers and internally at Apple for lagging behind other AI assistants. The rapid development of generative artificial intelligence and the release of ChatGPT in late 2022 reportedly blindsided Apple executives and forced the company to refocus its efforts on AI. In an interview with Good Morning America, Apple CEO Tim Cook stated that generative AI had "great promise" but had some potential dangers, and that it was "looking closely" at ChatGPT. It was first reported in July 2023 that Apple was creating its own internal large language model, codenamed "Ajax". In October 2023, Apple was reportedly on track to release new generative AI features into its operating systems by 2024, including a significantly redeveloped Siri. In an earnings call in February 2024, Cook stated that the company was spending a "tremendous amount of time and effort" into AI features that would be shared "later that year". === Google deal === In January 2026, Apple and Google announced a multi-year partnership under which Apple’s next-generation foundation models are expected to incorporate Google’s Gemini models and cloud infrastructure. According to the companies, the collaboration is intended to support future Apple Intelligence features, including enhancements to Siri, while Apple Intelligence will continue to operate on Apple devices and through Apple’s Private Cloud Compute system, which Apple states is designed to preserve user privacy. On an earnings call, Apple reported to investors that they were integrating an on-device model of the Google Gemini AI to Siri, as the development of their model was beset with setbacks. Apple has previously tested and used other third-party AI models like ChatGPT, but according to a Bloomberg article by Mark Gurman, Apple pushed forward the proposed Google deal; by using Google's Gemini model possessing 1.2 trillion parameters, Apple would integrate a much larger and more complex model than those it previously developed and used. Of note, comparable AI models from other major companies (including OpenAI and Meta) have also been reported to operate at a similar “trillion-parameter” scale and to compete against Gemini-class systems on benchmarks. == Models == Apple Intelligence consists of an on-device model as well as a cloud model running on servers primarily using Apple silicon. Both models consist of a generic foundation model, as well as multiple adapter models that are more specialized to particular tasks like text summarization and tone adjustment. It was launched for developers and testers on July 29, 2024, in U.S. English, with the developer betas of iOS 18.1, macOS 15.1, and iPadOS 18.1, released partially on October 28, 2024, and will fully launch by 2026. According to a human evaluation done by Apple's machine learning division, the on-device foundation model beat or tied equivalent small models by Mistral AI, Microsoft, and Google, while the server foundation models beat the performance of OpenAI's GPT-3, while roughly matching the performance of GPT-4. Apple's cloud models are built on a Private Cloud Compute platform which is allegedly designed with user privacy and end-to-end encryption in mind. Unlike other generative AI services like ChatGPT which use servers from third parties, Apple Intelligence's cloud models are run entirely on Apple servers with custom Apple silicon hardware built for end-to-end encryption. It was also designed to make sure that the software running on said servers matches the independently verifiable software accessible to researchers. In case of a software mismatch, Apple devices will refuse to connect to the servers. On June 10, 2025, Apple announced that Apple's on-device foundation models will be available to third-party applications as part of the Foundation Models API, with support for structured data response and tool calling. == Features == === Writing tools === Apple Intelligence features writing tools that are powered by LLMs. Selected text can be proofread, rewritten, made more friendly, concise or professional, similar to the AI writing features of the popular online English-language writing assistant tool Grammarly. It can also be used to generate summaries, key points, tables, and lists from an article or piece of writing. In iOS 18.2 and macOS 15.2, a ChatGPT integration was added to Writing Tools through "Compose" and "Describe your change" features. === Real-time Translation === Apple Intelligence enables the real-time translation of messages, photos and videos, and phone calls, through Apple's hardware. For communicating with foreigners, using the Translate app on iPhone to show subtitles in their language or to play back the translated audio naturally in their language, and also by wearing AirPods with Live Translation can now help to understand what someone is saying in users' preferred language in conversation. If both have headphones, simultaneous interpretation can be achieved. === Image Playground === Apple Intelligence can be used to generate images on-device with the Image Playground app. Similarly to OpenAI's DALL-E, it can be used to generate images using AI, using phrases and descriptions to output an image with customizable styles such as Animation and Sketch. In Notes, users can access Image Playground on iPad through the Image Wand tool in the Apple Pencil palette without having to open the Image Playground app. Rough sketches made with Apple Pencil can be transformed into images. As part of iOS, iPadOS, and macOS 26, Image Playground now integrates with the image generation models built into ChatGPT. === Genmoji === Using Apple Intelligence text-to-image models, users can generate unique "Genmoji" images by typing descriptions (prompting). Users can pick people in photos to have Genmoji generate images that resemble them. Similarly to emoji, Genmoji can be added inline to text messages, tapbacks, stickers and can be shared in Messages as well in third-party applications as inline messages or as stickers. === Siri overhaul === Siri, which used to be Apple's virtual assistant, has been updated to be an LLM chatbot, with enhanced capabilities made possible by Apple Intelligence. The latest iteration features an updated user interface, improved natural language processing, and the option to interact via text by double tapping the home bar without enabling the feature in the Accessibility menu, or double-clicking the command key on macOS. In a later update, Apple Intelligence will add the ability for Siri to use personal context from device activities to answer queries. === Mail === Apple Intelligence adds a feature called Priority Messages to the Mail app, which shows urgent emails such as same-day invitations or boarding passes, with AI generated summaries of the email. The Mail app also gains the ability to categorize incoming mail into Primary, Transactions, Updates, and Promotions based on what the email contains, which Apple claims is done all on-device. === Photos === Apple's Photos app includes a feature to create custom memory movies and enhanced search capabilities. Users can describe
Read more →
Path tracing

Path tracing is a rendering algorithm in computer graphics that simulates how light interacts with objects and participating media to generate realistic (physically plausible) images. It is based on earlier, more limited, ray tracing algorithms. Path tracing is used to create photorealistic images for artistic purposes, and for applications such as architectural rendering and product design. It is also used to render frames for animated films, and visual effects for film and television. Because it can be very accurate and unbiased, it is commonly used to generate reference images when testing the quality of other rendering algorithms. The technique uses the Monte Carlo method to compute estimates of global illumination and simulate the ways different materials reflect (or scatter), transmit, absorb, and emit light. It can incorporate simple modeling of the effects of aperture and lens (depth of field, and bokeh) and shutter speed (motion blur), or more realistic simulation of the optical components in a camera. The algorithm works by describing illumination in a scene using the rendering equation, or light transport equation, and finding an approximate solution using Monte Carlo integration. An inefficient (but accurate) version of the algorithm can be very simple, and involves tracing a ray from the camera, allowing this ray to bounce in random directions as it hits different objects in the scene, and computing the amount of light transmitted along the path to the camera whenever the path encounters a light source. This process is repeated many times for each pixel (each repetition, with generated path and transmitted light, is called a sample), and the results are averaged. One main difference between this algorithm and standard ray tracing is that a single unbranching path is traced each time, while "Whitted-style" or "Cook-style" ray tracing recursively samples branching paths (e.g. when light is both reflected and refracted by a glass object). More practical versions incorporate improvements such as quasi-Monte Carlo methods (techniques that distribute samples more evenly), importance sampling (take more samples of paths that are likely to transport more light), and next event estimation (allow a very limited form of branching, and sample additional paths that connect to the lights more directly). Because path tracing uses random samples there is noise in the final image, which decreases as more samples are taken. Images commonly require many thousands of samples per pixel (spp) to reduce noise to an acceptable level, and denoising techniques (e.g. based on neural networks) are often used. Denoising is usually necessary when path tracing is used for real-time rendering in video games, because relatively few samples can be taken. Many alternative algorithms for path tracing have been developed, although they do not always outperform more straightforward implementations. These include bidirectional path tracing (which traces paths forwards from the light source as well as backwards from the camera), Metropolis light transport, and ways of combining path tracing with photon mapping. Video games often use biased versions of path tracing to improve performance (e.g. limiting the number of bounces in each path). A family of techniques called ReSTIR has been developed that can help real-time path tracing by sharing data between nearby pixels and consecutive frames. == History == Like all ray tracing methods, path tracing is based on ray casting, which Arthur Appel used for computer graphics rendering in the late 1960s. In 1980, John Turner Whitted published a recursive ray tracing algorithm that allows rendering images of scenes containing mirrored surfaces and refractive transparent objects. In 1984, Cook et al. described a form of ray tracing called distributed ray tracing, which uses Monte Carlo integration to render effects such as depth of field, motion blur, reflection from rough surfaces, and area lights. The same year, the radiosity method (not a ray tracing method) was published, which was the first physically based method for rendering diffuse global illumination. In 1986, Jim Kajiya published a paper exploring how to use distributed ray tracing to render physically-based global illumination, and this paper also introduced and named the method called "path tracing". Path tracing and other distributed ray tracing techniques were further refined in the late 1980s and early 1990s by researchers such as James Arvo and Peter Shirley, and by Greg Ward in the open source Radiance software. Despite being theoretically able to render any lighting, the original form of path tracing can sometimes be very inefficient (or noisy) for rendering light that is reflected or refracted before illuminating a visible surface, including diffuse global illumination where light enters an area through narrow gaps, because it traces paths only from the camera. To address this, variations of path tracing that trace paths from both the camera and from light sources, called bidirectional path tracing, were published in 1993 by Eric Lafortune and Yves Willems, and in 1997 by Eric Veach and Leonidas Guibas. In 1997 Veach and Guibas also published an alternative method called Metropolis light transport, which combines bidirectional path tracing with the Metropolis method. Veach's lengthy Ph.D. dissertation described both techniques, along with the theoretical background of path tracing; later, the book Physically Based Rendering (which won an Academy Award for Technical Achievement in 2014) helped to make information about path tracing more widely available. Path tracing requires tracing a large number of paths of light in order to produce an image with a visually acceptable amount of noise. This made path tracing very slow on computers available in the 1980s and 1990s, and noise remained a problem when trying to reproduce the style of earlier computer graphics animated films. Most animated films produced until around 2010, by studios such as Pixar, used rasterization-based rendering, with ray tracing used selectively for reflections (and later for precomputed or cached global illumination). However the speed of computers rapidly increased during the 1990s. Blue Sky Studios pioneered using Monte Carlo ray tracing for global illumination in animation, including in the 1998 short film "Bunny", but they did not disclose the precise techniques used. Path tracing gradually become more practical for film production in the early 2000s. The Arnold renderer, developed by Marcos Fajardo, was used by Sony Pictures Imageworks to produce the feature-length film Monster House, released in 2006. Pixar rewrote their RenderMan software to use path tracing, and released their first feature-length path-traced film Finding Dory in 2016. Although path tracing still had a large computational cost, animation studios discovered that less human labor was required when using it, for example because global illumination no longer needed to be faked by manually placing lights. The amount of noise present in path traced images still caused difficulties, particularly when rendering motion blur (which was used extensively by earlier animated films) but denoising techniques were developed to address this. New techniques were also needed for rendering hair and fur, and to handle the extremely large scenes sometimes required by films. Renderers such as Arnold, and Disney's Hyperion, originally only used CPUs for rendering, but as GPUs became more capable (and APIs such as CUDA, OpenCL, and OptiX were released) researchers and developers began adapting algorithms and implementations to use GPUs. GPUs can dramatically reduce rendering time: for example using a high-end GPU to accelerate portions of the rendering code can make it over 30 times faster than using only a high-end CPU. == Description == Kajiya's 1986 paper defined a recursive integral equation called the rendering equation, which describes a simplified form of light transport. Using Monte Carlo integration for the integral on the right side of the equation leads fairly directly to the path tracing algorithm: I ( x , x ′ ) = g ( x , x ′ ) [ ϵ ( x , x ′ ) + ∫ S ρ ( x , x ′ , x ″ ) I ( x ′ , x ″ ) d x ″ ] {\displaystyle I(x,x')=g(x,x')\left[\epsilon (x,x')+\int _{S}\rho (x,x',x'')I(x',x'')dx''\right]} This expresses I(x,x'), the light arriving at point x from point x', as the product of a geometry term, g(x,x'), which is 0 if there is something blocking the light between the two points and 1 otherwise, and the amount of light leaving point x' and traveling towards x. The light leaving point x' is the sum of the light emitted by the surface at x', and the integral of the light arriving at x' from all other points in the scene (the integration domain S) and being reflected towards x. The factor ρ(x,x',x''), which calculates how much light is reflected, must take into account the angles at which the light is arriving and leaving, and
Read more →
AI notetaker

An AI notetaker is a tool using artificial intelligence to take notes during meetings. They are created by tech companies such as Microsoft and Google; by AI transcription services such Otter.ai, and by smaller firms such as Cluely and Krisp. Some business executives send AI notetakers to attend meetings not only to take notes, but also to answer questions on their behalf. The use of AI notetakers raises ethical questions, including recording meetings without the consent of all participants and the possibility that the notetaker will hallucinate and misrepresent what was said during meetings. There are also concerns when it comes to the privacy and security of meeting data and the sensitive information that lives inside meetings. Further controversies have developed from the use of AI notetakers such as Cluely to cheat in technical job interviews. == Technology == Large technology companies have integrated transcription capabilities into broader productivity and accessibility tools, including real-time captioning, dictation, and meeting documentation features embedded in operating systems and office platforms. Standalone transcription platforms, such as Transkriptor, focus specifically on automated transcription workflows and apply AI-based speech recognition to convert audio and video recordings into text. The software supports transcription in multiple languages and processes recordings uploaded via a web interface as well as through mobile and browser extensions. Tools of this type typically provide editable, time-aligned transcripts and export options for text and subtitle formats, cloud-based processing, multilingual support, and automation in transcription technology.
Read more →
Relational data stream management system

A relational data stream management system (RDSMS) is a distributed, in-memory data stream management system (DSMS) that is designed to use standards-compliant SQL queries to process unstructured and structured data streams in real-time. Unlike SQL queries executed in a traditional RDBMS, which return a result and exit, SQL queries executed in a RDSMS do not exit, generating results continuously as new data become available. Continuous SQL queries in a RDSMS use the SQL Window function to analyze, join and aggregate data streams over fixed or sliding windows. Windows can be specified as time-based or row-based. == RDSMS SQL Query Examples == Continuous SQL queries in a RDSMS conform to the ANSI SQL standards. The most common RDSMS SQL query is performed with the declarative SELECT statement. A continuous SQL SELECT operates on data across one or more data streams, with optional keywords and clauses that include FROM with an optional JOIN subclause to specify the rules for joining multiple data streams, the WHERE clause and comparison predicate to restrict the records returned by the query, GROUP BY to project streams with common values into a smaller set, HAVING to filter records resulting from a GROUP BY, and ORDER BY to sort the results. The following is an example of a continuous data stream aggregation using a SELECT query that aggregates a sensor stream from a weather monitoring station. The SELECTquery aggregates the minimum, maximum and average temperature values over a one-second time period, returning a continuous stream of aggregated results at one second intervals. RDSMS SQL queries also operate on data streams over time or row-based windows. The following example shows a second continuous SQL query using the WINDOW clause with a one-second duration. The WINDOW clause changes the behavior of the query, to output a result for each new record as it arrives. Hence the output is a stream of incrementally updated results with zero result latency.
Read more →
Magic Quadrant

Magic Quadrant (MQ) is a series of market research reports published by research and advisory firm Gartner that rely on proprietary qualitative data analysis methods to demonstrate market trends, such as direction, maturity, and participants. Their analyses are conducted for several specific technology industries and are updated every 1–2 years: once an updated report has been published, its predecessor is "retired". == Rating == Gartner rates vendors upon two criteria: completeness of vision and ability to execute. Completeness of vision – Reflects the vendor's innovation, and whether the vendor drives or follows the market. Ability to execute – Summarizes factors such as the vendor's financial viability, market responsiveness, product development, sales channels and customer base. The two component scores lead to a vendor position in one of four quadrants: === Leaders === Vendors in the "Leaders" quadrant have the highest composite scores for their completeness of vision and ability to execute. A vendor in the Leaders quadrant has the market share, credibility, and marketing & sales capabilities needed to drive the acceptance of new technologies. These vendors demonstrate a clear understanding of market needs, they are innovators and thought leaders, and they have well-articulated plans that customers and prospects can use when designing their infrastructures and strategies. In addition, they have a presence in the five major geographical regions, consistent financial performance, and broad platform support. === Challengers === Vendors in the "Challengers" quadrant have high scores mainly for their ability to execute. They both participate in the market and execute well enough to be a serious threat to vendors in the "Leaders" quadrant. They have strong products, as well as sufficiently credible market position and resources to sustain continued growth. Financial viability is not an issue for vendors in the "Challengers" quadrant, but they lack the size and influence of vendors in the "Leaders" quadrant due to their relative lack of vision. === Visionaries === Vendors in the "Visionaries" quadrant have high scores mainly for their completeness of vision. They deliver innovative products that address operationally or financially important end-user problems at a broad scale, but have not yet demonstrated the ability to capture market share or maintain sustainable levels of profitability. Visionary vendors are frequently privately held companies and acquisition targets for larger, established companies. The likelihood of acquisition often reduces the risks associated with installing their systems. === Niche Players === Vendors in the "Niche Players" quadrant have relatively low scores for both their ability to execute and their completeness of vision. They are often narrowly focused on specific market or vertical segments. This quadrant often also includes vendors that are adapting their existing products to enter the market under consideration, or larger vendors having difficulty developing and executing on their vision. == Gartner Critical Capabilities == Gartner Critical Capabilities complement Magic Quadrant analysis to offer deeper insight into the products and services offered by multiple vendors by a comparative analysis that scores competing products or services against a set of critical differentiators identified by Gartner. Gartner has periodically ended Magic Quadrant listings for IT Service Management, Web Content Management, and other industries as those markets have fully matured or other factors rendered the analytic framework inapplicable. == Criticism == The Magic Quadrant, and analysts in general, skew the market: according to research, by applying their methodologies to describe a market, they change that marketplace to fit their tools. Another criticism is that open source vendors are not considered sufficiently by analysts like Gartner, as has been published in an online discussion between a VP from Talend and a German Research VP from Gartner. On May 29, 2009 (2009-05-29), software vendor ZL Technologies filed a federal lawsuit against Gartner that challenged the "legitimacy" of Gartner's Magic Quadrant rating system. Gartner filed a motion to dismiss by claiming First Amendment protection since it contends that its MQ reports contain "pure opinion", which legally means opinions that are not based on fact. The court threw out the ZL case because it lacked a specific complaint. The decision was upheld on appeal.
Read more →
Big data

Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing software. Data with many entries (rows) offers greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate. Big data analysis challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy, and data sources. Big data was originally associated with three key concepts: volume, variety, and velocity. The analysis of big data that have only volume, velocity, and variety can pose challenges in sampling. A fourth concept, veracity, which refers to the level of reliability of data, was thus added. Without sufficient investment in expertise to ensure big data veracity, the volume and variety of data can produce costs and risks that exceed an organization's capacity to create and capture value from big data. Current usage of the term big data tends to refer to the use of predictive analytics, user behavior analytics, or certain other advanced data analytics methods that extract value from big data, and seldom to a particular size of data set. "There is little doubt that the quantities of data now available are indeed large, but that's not the most relevant characteristic of this new data ecosystem." Analysis of data sets can find new correlations to "spot business trends, prevent diseases, combat crime and so on". Scientists, business executives, medical practitioners, advertising and governments alike regularly meet difficulties with large datasets in areas including Internet searches, fintech, healthcare analytics, geographic information systems, urban informatics, and business informatics. Scientists encounter limitations in e-Science work, including meteorology, genomics, connectomics, complex physics simulations, biology, and environmental research. The size and number of available data sets have grown rapidly as data is collected by devices such as mobile devices, cheap and numerous information-sensing Internet of things devices, aerial (remote sensing) equipment, software logs, cameras, microphones, radio-frequency identification (RFID) readers and wireless sensor networks. The world's technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s; as of 2012, every day 2.5 exabytes (2.17×260 bytes) of data are generated. Based on an IDC report prediction, the global data volume was predicted to grow exponentially from 4.4 zettabytes to 44 zettabytes between 2013 and 2020. By 2025, IDC predicts there will be 163 zettabytes of data. According to IDC, global spending on big data and business analytics (BDA) solutions is estimated to reach $215.7 billion in 2021. Statista reported that the global big data market is forecasted to grow to $103 billion by 2027. In 2011 McKinsey & Company reported, if US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year. In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data. And users of services enabled by personal-location data could capture $600 billion in consumer surplus. One question for large enterprises is determining who should own big-data initiatives that affect the entire organization. Relational database management systems and desktop statistical software packages used to visualize data often have difficulty processing and analyzing big data. The processing and analysis of big data may require "massively parallel software running on tens, hundreds, or even thousands of servers". What qualifies as "big data" varies depending on the capabilities of those analyzing it and their tools. Furthermore, expanding capabilities make big data a moving target. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration." == Definition == The term big data has been in use since the 1990s, with some giving credit to John Mashey for popularizing the term. Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time. Big data philosophy encompasses unstructured, semi-structured and structured data; however, the main focus is on unstructured data. Big data "size" is a constantly moving target; as of 2012 ranging from a few dozen terabytes to many zettabytes of data. Big data requires a set of techniques and technologies with new forms of integration to reveal insights from datasets that are diverse, complex, and of a massive scale. Variability is often included as an additional quality of big data. A 2018 definition states "Big data is where parallel computing tools are needed to handle data", and notes, "This represents a distinct and clearly defined change in the computer science used, via parallel programming theories, and losses of some of the guarantees and capabilities made by Codd's relational model." In a comparative study of big datasets, Kitchin and McArdle found that none of the commonly considered characteristics of big data appear consistently across all of the analyzed cases. For this reason, other studies identified the redefinition of power dynamics in knowledge discovery as the defining trait. Instead of focusing on the intrinsic characteristics of big data, this alternative perspective pushes forward a relational understanding of the object claiming that what matters is the way in which data is collected, stored, made available and analyzed. === Big data vs. business intelligence === The growing maturity of the concept more starkly delineates the difference between "big data" and "business intelligence": Business intelligence uses applied mathematics tools and descriptive statistics with data with high information density to measure things, detect trends, etc. Big data uses mathematical analysis, optimization, inductive statistics, and concepts from nonlinear system identification to infer laws (regressions, nonlinear relationships, and causal effects) from large sets of data with low information density to reveal relationships and dependencies, or to perform predictions of outcomes and behaviors. == Characteristics == Big data can be described by the following characteristics: Volume The quantity of generated and stored data. The size of the data determines the value and potential insight, and whether it can be considered big data or not. The size of big data is usually larger than terabytes and petabytes. Variety The type and nature of the data. Earlier technologies like RDBMSs were capable to handle structured data efficiently and effectively. However, the change in type and nature from structured to semi-structured or unstructured challenged the existing tools and technologies. Big data technologies evolved with the prime intention to capture, store, and process the semi-structured and unstructured (variety) data generated with high speed (velocity), and huge in size (volume). Later, these tools and technologies were explored and used for handling structured data also but preferable for storage. Eventually, the processing of structured data was still kept as optional, either using big data or traditional RDBMSs. This helps in analyzing data towards effective usage of the hidden insights exposed from the data collected via social media, log files, sensors, etc. Big data draws from text, images, audio, video; plus it completes missing pieces through data fusion. Velocity The speed at which the data is generated and processed to meet the demands and challenges that lie in the path of growth and development. Big data is often available in real-time. Compared to small data, big data is produced more continually. Two kinds of velocity related to big data are the frequency of generation and the frequency of handling, recording, and publishing. Veracity The truthfulness or reliability of the data, which refers to the data quality and the data value. Big data must not only be large in size, but also must be reliable in order to achieve value in the analysis of it. The data quality of captured data can vary greatly, affecting an accurate analysis. Value The worth in information that can be achieved by the processing and analysis of large datasets. Value also can be measured by an assessment of the other qualities of big data. Value may also represent the profitability of information that is retrieved from the analysis of big data. Variability The characteristic of the changing formats, structure, or sources of big data. Big data can include structured, unstructured,
Read more →
Library history

Library history is a subdiscipline within library science and library and information science focusing on the history of libraries and their role in societies and cultures. Some see the field as a subset of information history. Library history is an academic discipline and should not be confused with its object of study (history of libraries): the discipline is much younger than the libraries it studies. Library history begins in ancient societies through contemporary issues facing libraries today. Topics include recording mediums, cataloguing systems, scholars, scribes, library supporters and librarians. == Earliest libraries == The earliest records of a library institution as it is presently understood can be dated back to around 5,000 years ago in the Southwest Asian regions of the world. One of the oldest libraries found is that of the ancient library at Ebla (circa 2500 BCE) in present-day Syria. In the 1970s, the excavation at Ebla's library unearthed over 20,000 clay tablets written in cuneiform script. === Library in Mesopotamia === The Assyrian King Assurbanipal created one of the greatest libraries in Nineveh in the seventh century BCE. The collection consisted of over 30,000 tablets written in a variety of languages. The collection was cataloged both by the shape of the tablet and by the subject of the content. The library had separate rooms for the different topics: government, history, law, astronomy, geography, and so on. The tablets also contained myths, hymns, and even jokes. Assurbanipal would send scribes to visit every corner of his kingdom to copy the content of other libraries. His library contained many of the most important literary works of the day, including the epic of Gilgamesh. Assurbanipal's Royal Library also had one of the first library catalogs. Unfortunately, Nineveh was eventually destroyed, and the library was lost in a fire. === Libraries in Ancient Greece === The Greek government was the first to sponsor public libraries. By 500 BCE both Athens and Samos had begun creating libraries for the public, though as most of the population was illiterate these spaces were serving a small, educated portion of the community. Athens developed a city archive at the Metroon in 405 BCE, where documents were stored in sealed jars. These would have saved the documents, but they would have been difficult to consult regularly. In Paros, around the same time, contracts were placed in the temple for safe keeping, and a book curse was placed for extra protection. === Library of Alexandria === The Library at Alexandria, Egypt, was renowned in the third century BCE while kings Ptolemy I Soter and Ptolemy II Philadelphus reigned. The library included a museum, garden, meeting areas and of course reading rooms. The Great Library, as it is known, was one of many in Alexandria. From its inception around the second century BCE, Alexandria was a well-known center for learning. It earned renown as the intellectual capital of the Western world up through the third century CE. The librarians at Alexandria collected, copied, and organized scrolls from across the known world. According to a primary source, every ship that came to Alexandria was required to hand over their books to be copied, and the copies would be returned to the owner, the library keeping the original. The Library of Alexandria was damaged by various disasters over time, including fire, invasion, and earthquake. Scholars believe the collection slowly diminished over time due to theft and efforts to remove it ahead of invading armies. While there are popular stories about how the library was ultimately destroyed, most of these are more myth than fact. === Libraries in Rome === Julius Caesar and his successor Augustus were the first to establish public libraries in ancient Rome, including the library of Apollo on the Palatine Hill. Several emperors followed suit over the next four centuries, including Hadrian, Tiberius, and Vespasian. Roman aristocrats also had personal libraries, which usually contained works in both Greek and Latin. A valuable example of this has been found at Herculaneum near Pompeii. Papyrus manuscripts in Herculaneum's Villa of the Papyri were encased in ash after the eruption of Vesuvius in 79 CE. Modern archaeology is now able to scan these artifacts and discern their contents, including many writings from Philodemus. The average Roman would not have been familiar with books beyond what they might hear read aloud in the forum. Public figures would pay for particular passages to be read aloud to the public from the steps of a public library. === Libraries in the Middle Ages === In the European Middle Ages, libraries began to become more prevalent, despite a widespread reduction in new writing beyond religious themes. Most libraries were initially connected to monasteries or religious institutions. Scriptoriums copied Christian religious texts to share with other religious centers or to be read aloud to their own parishioners. The Holy Roman Emperor Charlemagne (r. 786-814) had a large impact on the advancement of written culture in the Medieval Christian world, acquiring as many written works as he could, and employing many scribes to copy and recirculate vernacular versions of religious works. Most of the text held in small personal libraries was still religious in nature. == Early modern libraries == === Libraries of the Renaissance === During the Renaissance era the merchant middle class grew, and more people found benefits in education. They relied on libraries as a place to study and gain knowledge. Libraries provided a valuable resource, enriching the culture of those who were educated. Universities that had been started in the Middle Ages, founded their own libraries. Books in these libraries could not be borrowed from these libraries and were generally chained to the shelves to prevent theft. As more of the population became literate, new ideas like Humanism and Natural Law spawned an increase personal libraries, although they remained small. Gutenberg's invention of the printing press in 1456 opened the door to the modern era for libraries. == Oldest working libraries == According to the German librarian Michael Knoche, it is not possible to determine which library is the “oldest”: "Precise year dates are a construct, especially in the case of very old libraries. When a collection of books deserves to be called a library depends very much on the point of view of the observer." Various libraries are referred to as the “oldest”: The library founded in the 6th century of the Saint Catherine's Monastery in Sinai is "reputedly the oldest continuously run library in existence today", according to the Library of Congress. Its collection of religious and secular manuscripts is ranging from Bibles, liturgies and prayer books to legal documents such as deeds, court cases and fatwahs (legal opinions). The Al Qarawiyyin Library was founded in 859 by Fatima al-Fihri and is often regarded as the oldest working library in the world. It is in Fez, Morocco and is part of the oldest continually operating university in the world, the University of al-Qarawiyyin. The library houses approximately 4,000 ancient Islamic manuscripts. These manuscripts include 9th century Qurans and the oldest known accounts of the Islamic prophet Muhammed. The Malatestiana Library (Italian: Biblioteca Malatestiana) is a public library in the city of Cesena in northern Italy. Opened in 1454 it is significant for being the first civic library in Europe open to the general public. == Library history reports and writings of the early 19th and 20th century == In the early 19th and 20th century, representative titles were created reporting library history in the United States and the United Kingdom. American titles include Public Libraries in the United States of America, Their History, Condition, and Management (1876), Memorial History of Boston (1881) by Justin Winsor, Public Libraries in America (1894) by William I. Fletcher, and History of the New York Public Library (1923) by Henry M. Lydenberg. British titles include Old English Libraries (1911) by Earnest A. Savage and The Chained Library: A Survey of Four Centuries in the Evolution of the English Library by Burnett Hillman Streeter. In the beginning of the 20th century, library historians began applying scientific research methodologies to examine the library as a social agency. Two works that demonstrate this argument are Geschichte der Bibliotheken (1925) by Alfred Hessel and the Library Quarterly article from 1931, “The Sociological Beginnings of the Library Movement in America” by Arnold Borden. With the establishment of library schools, master's theses and doctoral dissertations represented the shift in serious research regarding libraries and library history. Two published doctoral dissertations that mark this trend are Foundations of the Public Library: The Origins of the American Public Library Movement in Ne
Read more →
Informedia Digital Library

The Informedia Digital Library is an ongoing research program at Carnegie Mellon University to build search engines and information visualization technology for many types of media. The program has carried out research on spoken document retrieval, video information retrieval, video segmentation, face recognition, and cross-language information retrieval. The Lycos search engine was an early product of the Informedia Digital Library Project. The project is led by Howard Wactlar. Researchers on the project have included: Michael Mauldin, Alex Hauptmann, Michael Christel, Michael Witbrock, Raj Reddy, Takeo Kanade and Scott Stevens.
Read more →
Car–Parrinello molecular dynamics

Car–Parrinello molecular dynamics (CPMD) refers to either a method used in molecular dynamics (also known as the Car–Parrinello method) or the computational chemistry software package used to implement this method. The CPMD method is one of the major methods for calculating ab initio molecular dynamics (ab initio MD or AIMD). Ab initio molecular dynamics (AIMD) is a computational method that uses first principles through quantum mechanics to simulate the motion of atoms in a system. It is a type of molecular dynamics (MD) simulation that does not rely on empirical potentials or force fields to describe the interactions between atoms, but rather calculates these interactions entirely from the electronic structure of the system using quantum mechanics. In an ab initio MD simulation, the total energy of the system is calculated at each time step using density functional theory (DFT), Hartree-Fock (HF), or other electronic structure calculation methods. The forces acting on each atom are then determined from the gradient of the energy with respect to the atomic coordinates, and the equations of motion are solved to predict the trajectory of the atoms. AIMD permits chemical bond breaking and forming events to occur and accounts for electronic polarization effect. Therefore, Ab initio MD simulations can be used to study a wide range of phenomena, including the structural, thermodynamic, and dynamic properties of materials and chemical reactions. They are particularly useful for systems that are not well described by empirical potentials or force fields, such as systems with strong electronic correlation or systems with many degrees of freedom. However, ab initio MD simulations are computationally demanding and require significant computational resources. The CPMD method is related to the more common Born–Oppenheimer molecular dynamics (BOMD) method in that the quantum mechanical effect of the electrons is included in the calculation of energy and forces for the classical motion of the nuclei. CPMD and BOMD are different types of AIMD. However, whereas BOMD treats the electronic structure problem within the time-independent Schrödinger equation, CPMD explicitly includes the electrons as active degrees of freedom, via (fictitious) dynamical variables. The software is a parallelized plane wave / pseudopotential implementation of density functional theory, particularly designed for ab initio molecular dynamics. == Car–Parrinello method == The Car–Parrinello method is a type of molecular dynamics, usually employing periodic boundary conditions, planewave basis sets, and density functional theory, proposed by Roberto Car and Michele Parrinello in 1985 while working at SISSA, who were subsequently awarded the Dirac Medal by ICTP in 2009. In contrast to Born–Oppenheimer molecular dynamics wherein the nuclear (ions) degree of freedom are propagated using ionic forces which are calculated at each iteration by approximately solving the electronic problem with conventional matrix diagonalization methods, the Car–Parrinello method explicitly introduces the electronic degrees of freedom as (fictitious) dynamical variables, writing an extended Lagrangian for the system which leads to a system of coupled equations of motion for both ions and electrons. In this way, an explicit electronic minimization at each time step, as done in Born–Oppenheimer MD, is not needed: after an initial standard electronic minimization, the fictitious dynamics of the electrons keeps them on the electronic ground state corresponding to each new ionic configuration visited along the dynamics, thus yielding accurate ionic forces. In order to maintain this adiabaticity condition, it is necessary that the fictitious mass of the electrons is chosen small enough to avoid a significant energy transfer from the ionic to the electronic degrees of freedom. This small fictitious mass in turn requires that the equations of motion are integrated using a smaller time step than the one (1–10 fs) commonly used in Born–Oppenheimer molecular dynamics. Currently, the CPMD method can be applied to systems that consist of a few tens or hundreds of atoms and access timescales on the order of tens of picoseconds. == General approach == In CPMD the core electrons are usually described by a pseudopotential and the wavefunction of the valence electrons are approximated by a plane wave basis set. The ground state electronic density (for fixed nuclei) is calculated self-consistently, usually using the density functional theory method. Kohn-Sham equations are often used to calculate the electronic structure, where electronic orbitals are expanded in a plane-wave basis set. Then, using that density, forces on the nuclei can be computed, to update the trajectories (using, e.g. the Verlet integration algorithm). In addition, however, the coefficients used to obtain the electronic orbital functions can be treated as a set of extra spatial dimensions, and trajectories for the orbitals can be calculated in this context. == Fictitious dynamics == CPMD is an approximation of the Born–Oppenheimer MD (BOMD) method. In BOMD, the electrons' wave function must be minimized via matrix diagonalization at every step in the trajectory. CPMD uses fictitious dynamics to keep the electrons close to the ground state, preventing the need for a costly self-consistent iterative minimization at each time step. The fictitious dynamics relies on the use of a fictitious electron mass (usually in the range of 400 – 800 a.u.) to ensure that there is very little energy transfer from nuclei to electrons, i.e. to ensure adiabaticity. Any increase in the fictitious electron mass resulting in energy transfer would cause the system to leave the ground-state BOMD surface. === Lagrangian === L = 1 2 ( ∑ I n u c l e i M I R ˙ I 2 + μ ∑ i o r b i t a l s ∫ d r | ψ ˙ i ( r , t ) | 2 ) − E [ { ψ i } , { R I } ] + ∑ i j Λ i j ( ∫ d r ψ i ψ j − δ i j ) , {\displaystyle {\mathcal {L}}={\frac {1}{2}}\left(\sum _{I}^{\mathrm {nuclei} }\ M_{I}{\dot {\mathbf {R} }}_{I}^{2}+\mu \sum _{i}^{\mathrm {orbitals} }\int d\mathbf {r} \ |{\dot {\psi }}_{i}(\mathbf {r} ,t)|^{2}\right)-E\left[\{\psi _{i}\},\{\mathbf {R} _{I}\}\right]+\sum _{ij}\Lambda _{ij}\left(\int d\mathbf {r} \ \psi _{i}\psi _{j}-\delta _{ij}\right),} where μ {\displaystyle \mu } is the fictitious mass parameter; E[{ψi},{RI}] is the Kohn–Sham energy density functional, which outputs energy values when given Kohn–Sham orbitals and nuclear positions. === Orthogonality constraint === ∫ d r ψ i ∗ ( r , t ) ψ j ( r , t ) = δ i j , {\displaystyle \int d\mathbf {r} \ \psi _{i}^{}(\mathbf {r} ,t)\psi _{j}(\mathbf {r} ,t)=\delta _{ij},} where δij is the Kronecker delta. === Equations of motion === The equations of motion are obtained by finding the stationary point of the Lagrangian under variations of ψi and RI, with the orthogonality constraint. M I R ¨ I = − ∇ I E [ { ψ i } , { R I } ] {\displaystyle M_{I}{\ddot {\mathbf {R} }}_{I}=-\nabla _{I}\,E\left[\{\psi _{i}\},\{\mathbf {R} _{I}\}\right]} μ ψ ¨ i ( r , t ) = − δ E δ ψ i ∗ ( r , t ) + ∑ j Λ i j ψ j ( r , t ) , {\displaystyle \mu {\ddot {\psi }}_{i}(\mathbf {r} ,t)=-{\frac {\delta E}{\delta \psi _{i}^{}(\mathbf {r} ,t)}}+\sum _{j}\Lambda _{ij}\psi _{j}(\mathbf {r} ,t),} where Λij is a Lagrangian multiplier matrix to comply with the orthonormality constraint. === Born–Oppenheimer limit === In the formal limit where μ → 0, the equations of motion approach Born–Oppenheimer molecular dynamics. == Software packages == There are a number of software packages available for performing AIMD simulations. Some of the most widely used packages include: CP2K: an open-source software package for AIMD. Quantum Espresso: an open-source package for performing DFT calculations. It includes a module for AIMD. VASP: a commercial software package for performing DFT calculations. It includes a module for AIMD. Gaussian: a commercial software package that can perform AIMD. NWChem: an open-source software package for AIMD. LAMMPS: an open-source software package for performing classical and ab initio MD simulations. SIESTA: an open-source software package for AIMD. ORCA: a general-purpose quantum chemistry package. == Applications == Studying the behavior of water across different environments, such as near a hydrophobic graphene sheet. Investigating the structure and dynamics of liquid water at ambient temperature. Solving the heat transfer problems (heat conduction and thermal radiation), such as in Si/Ge superlattices. Probing the proton transfer along hydrogen-bonds in different environments, such as in 1D water chains inside carbon nanotubes. Evaluating the critical point of crystals, composites, and solid-state materials, such as aluminum. Predicting and modelling different phases and phase transitions, such as in the amorphous phase of the phase-change memory material GeSbTe. Studying the combustion of combustibles, such as lignite-water systems. Measuring th
Read more →
Lossless join decomposition

In database design, a lossless join decomposition is a decomposition of a relation r {\displaystyle r} into relations r 1 , r 2 {\displaystyle r_{1},r_{2}} such that a natural join of the two smaller relations yields back the original relation. This is central in removing redundancy safely from databases while preserving the original data. Lossless join can also be called non-additive. == Definition == A relation r {\displaystyle r} on schema R {\displaystyle R} decomposes losslessly onto schemas R 1 {\displaystyle R_{1}} and R 2 {\displaystyle R_{2}} if π R 1 ( r ) ⋈ π R 2 ( r ) = r {\displaystyle \pi _{R_{1}}(r)\bowtie \pi _{R_{2}}(r)=r} , that is r {\displaystyle r} is the natural join of its projections onto the smaller schemas. A pair ( R 1 , R 2 ) {\displaystyle (R_{1},R_{2})} is a lossless-join decomposition of R {\displaystyle R} or said to have a lossless join with respect to a set of functional dependencies F {\displaystyle F} if any relation r ( R ) {\displaystyle r(R)} that satisfies F {\displaystyle F} decomposes losslessly onto R 1 {\displaystyle R_{1}} and R 2 {\displaystyle R_{2}} . Decompositions into more than two schemas can be defined in the same way. == Criteria == A decomposition R = R 1 ∪ R 2 {\displaystyle R=R_{1}\cup R_{2}} has a lossless join with respect to F {\displaystyle F} if and only if the closure of R 1 ∩ R 2 {\displaystyle R_{1}\cap R_{2}} includes R 1 ∖ R 2 {\displaystyle R_{1}\setminus R_{2}} or R 2 ∖ R 1 {\displaystyle R_{2}\setminus R_{1}} . In other words, one of the following must hold: ( R 1 ∩ R 2 ) → ( R 1 ∖ R 2 ) ∈ F + {\displaystyle (R_{1}\cap R_{2})\to (R_{1}\setminus R_{2})\in F^{+}} ( R 1 ∩ R 2 ) → ( R 2 ∖ R 1 ) ∈ F + {\displaystyle (R_{1}\cap R_{2})\to (R_{2}\setminus R_{1})\in F^{+}} === Criteria for multiple sub-schemas === Multiple sub-schemas R 1 , R 2 , . . . , R n {\displaystyle R_{1},R_{2},...,R_{n}} have a lossless join if there is some way in which we can repeatedly perform lossless joins until all the schemas have been joined into a single schema. Once we have a new sub-schema made from a lossless join, we are not allowed to use any of its isolated sub-schema to join with any of the other schemas. For example, if we can do a lossless join on a pair of schemas R i , R j {\displaystyle R_{i},R_{j}} to form a new schema R i , j {\displaystyle R_{i,j}} , we use this new schema (rather than R i {\displaystyle R_{i}} or R j {\displaystyle R_{j}} ) to form a lossless join with another schema R k {\displaystyle R_{k}} (which may already be joined (e.g., R k , l {\displaystyle R_{k,l}} )). == Example == Let R = { A , B , C , D } {\displaystyle R=\{A,B,C,D\}} be the relation schema, with attributes A, B, C and D. Let F = { A → B C } {\displaystyle F=\{A\rightarrow BC\}} be the set of functional dependencies. Decomposition into R 1 = { A , B , C } {\displaystyle R_{1}=\{A,B,C\}} and R 2 = { A , D } {\displaystyle R_{2}=\{A,D\}} is lossless under F because R 1 ∩ R 2 = A {\displaystyle R_{1}\cap R_{2}=A} and we have a functional dependency A → B C {\displaystyle A\rightarrow BC} . In other words, we have proven that ( R 1 ∩ R 2 → R 1 ∖ R 2 ) ∈ F + {\displaystyle (R_{1}\cap R_{2}\rightarrow R_{1}\setminus R_{2})\in F^{+}} .
Read more →
Bartels–Stewart algorithm

In numerical linear algebra, the Bartels–Stewart algorithm is used to numerically solve the Sylvester matrix equation A X − X B = C {\displaystyle AX-XB=C} . Developed by R.H. Bartels and G.W. Stewart in 1971, it was the first numerically stable method that could be systematically applied to solve such equations. The algorithm works by using the real Schur decompositions of A {\displaystyle A} and B {\displaystyle B} to transform A X − X B = C {\displaystyle AX-XB=C} into a triangular system that can then be solved using forward or backward substitution. In 1979, G. Golub, C. Van Loan and S. Nash introduced an improved version of the algorithm, known as the Hessenberg–Schur algorithm. It remains a standard approach for solving Sylvester equations when X {\displaystyle X} is of small to moderate size. == The algorithm == Let X , C ∈ R m × n {\displaystyle X,C\in \mathbb {R} ^{m\times n}} , and assume that the eigenvalues of A {\displaystyle A} are distinct from the eigenvalues of B {\displaystyle B} . Then, the matrix equation A X − X B = C {\displaystyle AX-XB=C} has a unique solution. The Bartels–Stewart algorithm computes X {\displaystyle X} by applying the following steps: 1.Compute the real Schur decompositions R = U T A U , {\displaystyle R=U^{T}AU,} S = V T B T V . {\displaystyle S=V^{T}B^{T}V.} The matrices R {\displaystyle R} and S {\displaystyle S} are block-upper triangular matrices, with diagonal blocks of size 1 × 1 {\displaystyle 1\times 1} or 2 × 2 {\displaystyle 2\times 2} . 2. Set F = U T C V . {\displaystyle F=U^{T}CV.} 3. Solve the simplified system R Y − Y S T = F {\displaystyle RY-YS^{T}=F} , where Y = U T X V {\displaystyle Y=U^{T}XV} . This can be done using forward substitution on the blocks. Specifically, if s k − 1 , k = 0 {\displaystyle s_{k-1,k}=0} , then ( R − s k k I ) y k = f k + ∑ j = k + 1 n s k j y j , {\displaystyle (R-s_{kk}I)y_{k}=f_{k}+\sum _{j=k+1}^{n}s_{kj}y_{j},} where y k {\displaystyle y_{k}} is the k {\displaystyle k} th column of Y {\displaystyle Y} . When s k − 1 , k ≠ 0 {\displaystyle s_{k-1,k}\neq 0} , columns [ y k − 1 ∣ y k ] {\displaystyle [y_{k-1}\mid y_{k}]} should be concatenated and solved for simultaneously. 4. Set X = U Y V T . {\displaystyle X=UYV^{T}.} === Computational cost === Using the QR algorithm, the real Schur decompositions in step 1 require approximately 10 ( m 3 + n 3 ) {\displaystyle 10(m^{3}+n^{3})} flops, so that the overall computational cost is 10 ( m 3 + n 3 ) + 2.5 ( m n 2 + n m 2 ) {\displaystyle 10(m^{3}+n^{3})+2.5(mn^{2}+nm^{2})} . === Simplifications and special cases === In the special case where B = − A T {\displaystyle B=-A^{T}} and C {\displaystyle C} is symmetric, the solution X {\displaystyle X} will also be symmetric. This symmetry can be exploited so that Y {\displaystyle Y} is found more efficiently in step 3 of the algorithm. == The Hessenberg–Schur algorithm == The Hessenberg–Schur algorithm replaces the decomposition R = U T A U {\displaystyle R=U^{T}AU} in step 1 with the decomposition H = Q T A Q {\displaystyle H=Q^{T}AQ} , where H {\displaystyle H} is an upper-Hessenberg matrix. This leads to a system of the form H Y − Y S T = F {\displaystyle HY-YS^{T}=F} that can be solved using forward substitution. The advantage of this approach is that H = Q T A Q {\displaystyle H=Q^{T}AQ} can be found using Householder reflections at a cost of ( 5 / 3 ) m 3 {\displaystyle (5/3)m^{3}} flops, compared to the 10 m 3 {\displaystyle 10m^{3}} flops required to compute the real Schur decomposition of A {\displaystyle A} . == Software and implementation == The subroutines required for the Hessenberg-Schur variant of the Bartels–Stewart algorithm are implemented in the SLICOT library. These are used in the MATLAB control system toolbox. == Alternative approaches == For large systems, the O ( m 3 + n 3 ) {\displaystyle {\mathcal {O}}(m^{3}+n^{3})} cost of the Bartels–Stewart algorithm can be prohibitive. When A {\displaystyle A} and B {\displaystyle B} are sparse or structured, so that linear solves and matrix vector multiplies involving them are efficient, iterative algorithms can potentially perform better. These include projection-based methods, which use Krylov subspace iterations, methods based on the alternating direction implicit (ADI) iteration, and hybridizations that involve both projection and ADI. Iterative methods can also be used to directly construct low rank approximations to X {\displaystyle X} when solving A X − X B = C {\displaystyle AX-XB=C} .
Read more →
Bartels–Stewart algorithm

In numerical linear algebra, the Bartels–Stewart algorithm is used to numerically solve the Sylvester matrix equation A X − X B = C {\displaystyle AX-XB=C} . Developed by R.H. Bartels and G.W. Stewart in 1971, it was the first numerically stable method that could be systematically applied to solve such equations. The algorithm works by using the real Schur decompositions of A {\displaystyle A} and B {\displaystyle B} to transform A X − X B = C {\displaystyle AX-XB=C} into a triangular system that can then be solved using forward or backward substitution. In 1979, G. Golub, C. Van Loan and S. Nash introduced an improved version of the algorithm, known as the Hessenberg–Schur algorithm. It remains a standard approach for solving Sylvester equations when X {\displaystyle X} is of small to moderate size. == The algorithm == Let X , C ∈ R m × n {\displaystyle X,C\in \mathbb {R} ^{m\times n}} , and assume that the eigenvalues of A {\displaystyle A} are distinct from the eigenvalues of B {\displaystyle B} . Then, the matrix equation A X − X B = C {\displaystyle AX-XB=C} has a unique solution. The Bartels–Stewart algorithm computes X {\displaystyle X} by applying the following steps: 1.Compute the real Schur decompositions R = U T A U , {\displaystyle R=U^{T}AU,} S = V T B T V . {\displaystyle S=V^{T}B^{T}V.} The matrices R {\displaystyle R} and S {\displaystyle S} are block-upper triangular matrices, with diagonal blocks of size 1 × 1 {\displaystyle 1\times 1} or 2 × 2 {\displaystyle 2\times 2} . 2. Set F = U T C V . {\displaystyle F=U^{T}CV.} 3. Solve the simplified system R Y − Y S T = F {\displaystyle RY-YS^{T}=F} , where Y = U T X V {\displaystyle Y=U^{T}XV} . This can be done using forward substitution on the blocks. Specifically, if s k − 1 , k = 0 {\displaystyle s_{k-1,k}=0} , then ( R − s k k I ) y k = f k + ∑ j = k + 1 n s k j y j , {\displaystyle (R-s_{kk}I)y_{k}=f_{k}+\sum _{j=k+1}^{n}s_{kj}y_{j},} where y k {\displaystyle y_{k}} is the k {\displaystyle k} th column of Y {\displaystyle Y} . When s k − 1 , k ≠ 0 {\displaystyle s_{k-1,k}\neq 0} , columns [ y k − 1 ∣ y k ] {\displaystyle [y_{k-1}\mid y_{k}]} should be concatenated and solved for simultaneously. 4. Set X = U Y V T . {\displaystyle X=UYV^{T}.} === Computational cost === Using the QR algorithm, the real Schur decompositions in step 1 require approximately 10 ( m 3 + n 3 ) {\displaystyle 10(m^{3}+n^{3})} flops, so that the overall computational cost is 10 ( m 3 + n 3 ) + 2.5 ( m n 2 + n m 2 ) {\displaystyle 10(m^{3}+n^{3})+2.5(mn^{2}+nm^{2})} . === Simplifications and special cases === In the special case where B = − A T {\displaystyle B=-A^{T}} and C {\displaystyle C} is symmetric, the solution X {\displaystyle X} will also be symmetric. This symmetry can be exploited so that Y {\displaystyle Y} is found more efficiently in step 3 of the algorithm. == The Hessenberg–Schur algorithm == The Hessenberg–Schur algorithm replaces the decomposition R = U T A U {\displaystyle R=U^{T}AU} in step 1 with the decomposition H = Q T A Q {\displaystyle H=Q^{T}AQ} , where H {\displaystyle H} is an upper-Hessenberg matrix. This leads to a system of the form H Y − Y S T = F {\displaystyle HY-YS^{T}=F} that can be solved using forward substitution. The advantage of this approach is that H = Q T A Q {\displaystyle H=Q^{T}AQ} can be found using Householder reflections at a cost of ( 5 / 3 ) m 3 {\displaystyle (5/3)m^{3}} flops, compared to the 10 m 3 {\displaystyle 10m^{3}} flops required to compute the real Schur decomposition of A {\displaystyle A} . == Software and implementation == The subroutines required for the Hessenberg-Schur variant of the Bartels–Stewart algorithm are implemented in the SLICOT library. These are used in the MATLAB control system toolbox. == Alternative approaches == For large systems, the O ( m 3 + n 3 ) {\displaystyle {\mathcal {O}}(m^{3}+n^{3})} cost of the Bartels–Stewart algorithm can be prohibitive. When A {\displaystyle A} and B {\displaystyle B} are sparse or structured, so that linear solves and matrix vector multiplies involving them are efficient, iterative algorithms can potentially perform better. These include projection-based methods, which use Krylov subspace iterations, methods based on the alternating direction implicit (ADI) iteration, and hybridizations that involve both projection and ADI. Iterative methods can also be used to directly construct low rank approximations to X {\displaystyle X} when solving A X − X B = C {\displaystyle AX-XB=C} .
Read more →