Semantic integration is the process of interrelating information from diverse sources, for example calendars and to do lists, email archives, presence information (physical, psychological, and social), documents of all sorts, contacts (including social graphs), search results, and advertising and marketing relevance derived from them. In this regard, semantics focuses on the organization of and action upon information by acting as an intermediary between heterogeneous data sources, which may conflict not only by structure but also context or value. == Applications and methods == In enterprise application integration (EAI), semantic integration can facilitate or even automate the communication between computer systems using metadata publishing. Metadata publishing potentially offers the ability to automatically link ontologies. One approach to (semi-)automated ontology mapping requires the definition of a semantic distance or its inverse, semantic similarity and appropriate rules. Other approaches include so-called lexical methods, as well as methodologies that rely on exploiting the structures of the ontologies. For explicitly stating similarity/equality, there exist special properties or relationships in most ontology languages. OWL, for example has "owl:equivalentClass", "owl:equivalentProperty" and "owl:sameAs". Eventually system designs may see the advent of composable architectures where published semantic-based interfaces are joined together to enable new and meaningful capabilities. These could predominately be described by means of design-time declarative specifications, that could ultimately be rendered and executed at run-time. Semantic integration can also be used to facilitate design-time activities of interface design and mapping. In this model, semantics are only explicitly applied to design and the run-time systems work at the syntax level. This "early semantic binding" approach can improve overall system performance while retaining the benefits of semantic driven design. == Semantic integration situations == From the industry use case, it has been observed that the semantic mappings were performed only within the scope of the ontology class or the datatype property. These identified semantic integrations are (1) integration of ontology class instances into another ontology class without any constraint, (2) integration of selected instances in one ontology class into another ontology class by the range constraint of the property value and (3) integration of ontology class instances into another ontology class with the value transformation of the instance property. Each of them requires a particular mapping relationship, which is respectively: (1) equivalent or subsumption mapping relationship, (2) conditional mapping relationship that constraints the value of property (data range) and (3) transformation mapping relationship that transforms the value of property (unit transformation). Each identified mapping relationship can be defined as either (1) direct mapping type, (2) data range mapping type or (3) unit transformation mapping type. == KG vs. RDB approaches == In the case of integrating supplemental data source, KG(Knowledge graph) formally represents the meaning involved in information by describing concepts, relationships between things, and categories of things. These embedded semantics with the data offer significant advantages such as reasoning over data and dealing with heterogeneous data sources. The rules can be applied on KG more efficiently using graph query. For example, the graph query does the data inference through the connected relations, instead of repeated full search of the tables in relational database. KG facilitates the integration of new heterogeneous data by just adding new relationships between existing information and new entities. This facilitation is emphasized for the integration with existing popular linked open data source such as Wikidata.org. SQL query is tightly coupled and rigidly constrained by datatype within the specific database and can join tables and extract data from tables, and the result is generally a table, and a query can join tables by any columns which match by datatype. SPARQL query is the standard query language and protocol for Linked Open Data on the web and loosely coupled with the database so that it facilitates the reusability and can extract data through the relations free from the datatype, and not only extract but also generate additional knowledge graph with more sophisticated operations(logic: transitive/symmetric/inverseOf/functional). The inference based query (query on the existing asserted facts without the generation of new facts by logic) can be fast comparing to the reasoning based query (query on the existing plus the generated/discovered facts based on logic). The information integration of heterogeneous data sources in traditional database is intricate, which requires the redesign of the database table such as changing the structure and/or addition of new data. In the case of semantic query, SPARQL query reflects the relationships between entities in a way that aligned with human's understanding of the domain, so the semantic intention of the query can be seen on the query itself. Unlike SPARQL, SQL query, which reflects the specific structure of the database and derived from matching the relevant primary and foreign keys of tables, loses the semantics of the query by missing the relationships between entities. Below is the example that compares SPARQL and SQL queries for medications that treats "TB of vertebra". SELECT ?medication WHERE { ?diagnosis a example:Diagnosis . ?diagnosis example:name “TB of vertebra” . ?medication example:canTreat ?diagnosis . } SELECT DRUG.medID FROM DIAGNOSIS, DRUG, DRUG_DIAGNOSIS WHERE DIAGNOSIS.diagnosisID=DRUG_DIAGNOSIS.diagnosisID AND DRUG.medID=DRUG_DIAGNOSIS.medID AND DIAGNOSIS.name=”TB of vertebra” == Examples == The Pacific Symposium on Biocomputing has been a venue for the popularization of the ontology mapping task in the biomedical domain, and a number of papers on the subject can be found in its proceedings.
Maximum inner-product search
Maximum inner-product search (MIPS) is a search problem, with a corresponding class of search algorithms which attempt to maximise the inner product between a query and the data items to be retrieved. MIPS algorithms are used in a wide variety of big data applications, including recommendation algorithms and machine learning. Formally, for a database of vectors x i {\displaystyle x_{i}} defined over a set of labels S {\displaystyle S} in an inner product space with an inner product ⟨ ⋅ , ⋅ ⟩ {\displaystyle \langle \cdot ,\cdot \rangle } defined on it, MIPS search can be defined as the problem of determining a r g m a x i ∈ S ⟨ x i , q ⟩ {\displaystyle {\underset {i\in S}{\operatorname {arg\,max} }}\ \langle x_{i},q\rangle } for a given query q {\displaystyle q} . Although there is an obvious linear-time implementation, it is generally too slow to be used on practical problems. However, efficient algorithms exist to speed up MIPS search. Under the assumption of all vectors in the set having constant norm, MIPS can be viewed as equivalent to a nearest neighbor search (NNS) problem in which maximizing the inner product is equivalent to minimizing the corresponding distance metric in the NNS problem. Like other forms of NNS, MIPS algorithms may be approximate or exact. MIPS search is used as part of DeepMind's RETRO algorithm.
Online Safety Amendment (Social Media Minimum Age) Act 2024
The Online Safety Amendment (Social Media Minimum Age) Act 2024 is an Australian act of parliament that prohibits minors under the age of 16 from holding an account on certain social media platforms. It is an amendment to the Online Safety Act 2021 and was passed by the Parliament of Australia on 29 November 2024. It imposes monetary penalties on social media companies that fail to take reasonable steps to prevent minors under 16 that are located in Australia from having accounts on their services. The legislation allows the government to determine which social media platforms must ban age‑restricted users and proclaim a date for the commencement of the ban, with those provisions taking effect on 10 December 2025. Facebook, Instagram, Reddit, Snapchat, TikTok, Twitter, Threads, Twitch, Kick, and YouTube were age‑restricted on 10 December 2025, with the possibility that more platforms may be added. The act is being challenged in the High Court by the Digital Freedom Project. == Background == The ban on access to social media by young people by the federal government originated in November 2023, when shadow communications minister David Coleman introduced a private member's bill requiring the government to conduct a trial for age-verification technology on pornography and social media platforms. While the bill did not succeed, the Albanese government funded the trial in the 2024 Australian federal budget. In June 2024, opposition leader Peter Dutton pledged that a Coalition government would implement a ban on social media for under-16s within 100 days of taking office. The following month, prime minister Anthony Albanese announced the government would introduce legislation banning under-16s from social media. The Online Safety Amendment (Social Media Minimum Age) Bill 2024 was introduced into parliament by minister for communications Michelle Rowland on 21 November 2024, passing both houses on 28 November 2024. The ban on access to social media by young people by the federal government also gained momentum following an entreaty by the wife of the premier of South Australia, Peter Malinauskas, to her husband. She requested that he read The Anxious Generation by Jonathan Haidt and take action to address the impact of social media on the mental health of children. The couple have four young children, and, thinking of them, the premier thought that government should play a part in helping parents to regulate use of social media by their children at home. Malinauskas contacted former High Court chief justice Robert French, who agreed to look at the issue, and in September 2024 handed the premier a 267 page proposal, which he dubbed a "Swiss Army knife" rather than a machete, to adjust to social media's "changing landscape and its complexity". The leaders of other states and territories gave their support to Malinauskas's idea, and he took the French report to National Cabinet to collaborate with chief ministers, premiers, and the prime minister. Community support swelled after stories of parents who had lost their children to suicide after being bullied on social media were published. Albanese himself was moved by a personal letter received from Kelly O'Brien, whose 12-year-old daughter Charlotte had taken her own life due to bullying at school. An event took place at the sidelines of the United Nations General Assembly session in September 2025 at which a mother spoke of her daughter's suicide as "death by bullying ... enabled by social media". The speech won support from world leaders in Greece, Fiji, Tonga and the president of the European Commission Ursula von der Leyen. In early September 2024, South Australia proposed legislation similar to the federal law now in place. The state-based version was intended to ban users under the age of 14, unlike the federal law, which bans those under 16. The state-based law also proposed to require parental consent for 14 and 15‑year‑olds. Later in September, prime minister Anthony Albanese announced that his government intended to introduce legislation to set a minimum age requirement for social media. In November 2024, the federal government indicated their intention to engage the Age Check Certification Scheme following a tender process for an age assurance technology trial. The Albanese government's proposed ban was supported by the governments of every state and territory. Albanese described social media as a "scourge", and said "I want people to spend more time on the footy field or the netball court than they're spending on their phones", that family members are "worried sick about the safety of our kids online", and that social media "is having a negative impact on young people's mental health and on anxiety". Albanese's statements followed an earlier pledge by Liberal opposition leader Peter Dutton who was pushed by the early advocacy of shadow communications minister David Coleman to implement a ban on social media for under 16s within 100 days of being elected. The opposition organised an open letter signed by 140 experts who specialise in child welfare and technology. The opposition was concerned about the invasion of privacy that will occur with the introduction of identification-based age checks. An advocacy group for digital companies in Australia called the plans a "20th Century response to 21st Century challenges". A director of a mental health service voiced concerns, stating that "73% of young people across Australia who accessed mental health support did so through social media". == Implementation == Social media companies will receive a transition period of one year after the legislation is enacted to introduce reasonable controls preventing minors under the age of 16 from holding accounts on their services while physically located in Australia. Enforcement will involve fines of up to A$49.5 million for companies failing to take such steps, with no consequences for parents and children who violate the restrictions. There are no parental consent exceptions to the ban, and while the use of virtual private networks (VPNs) to access these services remains legal in Australia, the services are expected to try to stop under 16s from using VPNs to pretend to be outside Australia. The expectation is to make best-efforts to implement the ban on platforms including Facebook, Instagram, Reddit, Snapchat, TikTok, Twitter, Threads, Twitch, Kick and YouTube. Some social media companies are now obligated to become good enough at profiling Australian children under 16 to satisfy the Australian government they tried to implement the ban to avoid being fined. Consequently, social media companies said they will try to identify restricted users using various methods including behavioural inferencing. On 5 November 2025, it was announced that online gaming platform Roblox will not be banned, but Reddit and live-streaming platform Kick will be added to the list of platforms to be banned. A report by Age Check Certification Scheme, a UK company recruited by the government to consult on the technology used to implement the restrictions, was issued in June 2025, ahead of the December deadline to implement the ban. In June 2025, the preliminary report was released, which stated that "there are no significant technological barriers" to implementing the ban. In late July 2025, Google warned that it would sue the Australian government if YouTube was included in the ban. On 30 July, the government announced that it would extend its social media age limit to include YouTube, following advice from Grant. On 30 July 2025, the minister for communications, Anika Wells, published the Online Safety (Age-Restricted Social Media Platforms) Rules 2025, which specify exactly which types of social media platforms will be banned for certain users. On 31 August 2025, the full report was released, which stated that it would technically be possible to implement the ban; however, coordination among different services is required to successfully implement it. It also highlighted the benefits and flaws of different methods of age verification. On 16 September 2025, it was announced that the eSafety Commissioner will be able to take legal action against social media companies that have not pursued reasonable steps to bar users under the age of 16, and that fines can range up to A$49.5 million against these companies in court. On 19 November 2025, Meta announced that from 4 December their platforms (Instagram, Facebook, and Threads) would be removing users under the age of 16 ahead of the 10 December deadline. Users will be able to scan a face or provide an identity document to prove their age. On 21 November 2025, the eSafety Commissioner announced that the live-streaming platform Twitch will be included in the ban, but that Pinterest would not be. In December 2025, eSafety Commissioner Julie Inman Grant suggested efforts to block users include use by social media companies of various "signals" to identify children that are
Web presence
A web presence is a location on the World Wide Web where a person, business, or some other entity is represented (see also web property and point of presence). Examples of a web presence for a person could be a personal website, a blog, a profile page, a wiki page, or a social media point of presence (e.g. a LinkedIn profile, a Facebook account, or a Twitter account). Examples of a web presence for a business or some other entity could be a corporate website, a microsite, a page on a review site, a wiki page, or a social media point of presence (e.g., a LinkedIn company page and/or group, a Facebook business/brand/product page, or a Twitter account). Every web presence is associated with a unique web address to distinguish one point of presence from another. == Owned vs. unowned == Web presence can either be owned or unowned. Owned media exists when a single person or group can control the content that is published on its web presence (e.g. a corporate website or a personal Twitter account). However, when a single person or group cannot solely control the content, the creator is different from the owner. This is considered unowned media (see earned media). A Wikipedia page or a Yelp page about a person, company, or product would be an example of a known (or "earned") web presence. Occasionally, a first form of media known as "paid media" is often included in the discussion of media types: "earned vs. owned vs. paid". Paid media is commonly found in the form of advertisements, but it is not considered a form of web presence. == Management == Web presence management is the process of establishing and maintaining a digital footprint on the web. The three factors that are considered include the following: where a person or business has web presence; how each web presence represents its enterprise; and what is published at a point of presence. Web presence management is the discipline of determining and governing: the distribution of policy documents which platforms are most appropriate (e.g. internal vs. external blog, YouTube vs. Vimeo) the single inventory of personal or corporate web presence (e.g. partners or advocates) where on the web a business and any relatable assets are represented where on the web a business and any relatable assets are impersonated or pirated web properties with the particular entities they represent who has control over which web properties new web properties which are not in the personal or corporate inventory (e.g. someone creates a new presence) authorized and unauthorized changes to the creation (e.g. branding) of a web presence a workflow for creating a web property that follows its corporate standards === Management system === The purpose of a web presence management system is to manage the web presence of a person or business. This includes the collection of domain names, websites, social media, and other web pages where he, she, or it is being represented. The tool generally offers the following key functions: new presence discovery, inventory management, change detection, access control, stakeholder coordination, and compliance workflow. A web presence management system is meant to have a broader reach so that it emphasizes where a presence has been established, will be established, must be maintained, or must be remediated. An example of a web presence management system is the Brandle Presence Manager. In order to publish content to the various points of web presence, multiple content management systems and sometimes even social media management systems are often used. The primary focus of most content and social media management systems is limited to their specific web platforms. === Domain names === Another aspect of web presence management is managing the collection of domain names registered to the person or business. Any entity may register multiple domain names for the same property. As a result, they can link alternative spellings, different top-level domains, aliases, brands, or products to the same website. Similarly, negative or derogatory domain names may also be registered. This is done to prevent certain domain names from being used against the person or business. It is common for a larger business to have domain names registered by multiple employees at multiple domain name registrars, possibly a result of organizational or geographical requirements. Consequently, a web presence management system can be used to monitor all domain names registered by the business, regardless of the registrars used. == Discovery == Web presence discovery is the process of monitoring the web for a new point of presence about a person or business. Web presence discovery is often included in a web presence management system. Whether a new domain is registered, a new website is published, or a new social media account is established, it occurs outside of the person's or business’ control. As a result, its purpose is to assess a new point of presence and appropriately handle any violations. Web presence discovery differs from content listening. The former involves looking for new properties on the web, whereas the latter refers to analyzing content that already exists to hear how a person or business is seen often in near real time. Examples of content listening systems include Sysomos and Radian6, which is now a subsidiary of Salesforce.com. === Brand protection === A person or business may choose to watch for a new web presence that might appear to misrepresent or mislead an audience, such as counterfeiters, spoofers, or malicious hackers. One of the early software in the online brand protection marketplace was MarkMonitor, now part of Thomson Reuters. This software helped detect rogue domain names and websites. However, the modern day growth of social media has seen a rise in the number of fraudulent brand impersonations. It has become much easier for a new web presence to be created on those platforms, which results in a greater frequency of them today. As a preventive measure, online brand protection providers are now adding social media to their domain and website discovery options. === Security === The widespread growth of social media has also made it easier for unauthorized individuals to impersonate an employee. Consequently, social media has now become a recognized threat vector in that it can be used to socially engineer an attack on a business. To counter this, companies are able to use web presence monitoring tools to detect new points of presence on the web and thereby defend against socially engineered attacks. === Distributed inventory management === A web presence monitoring system can be used by a business to associate a new web property with its corporate inventory. It is designed to address autonomous, distributed behaviors. This usually applies to larger businesses whose geographically diverse employees are more prone to creating new points of presence on the web. For example, a retail chain may allow each local store to create and manage their web presence to market to and communicate with their local customer base. Similarly, a global business may have teams in each country or region who create and manage a web presence to adapt to local languages or cultures. == Monitoring == Web presence monitoring is the process of monitoring a known inventory of web presence to detect any changes that are made. Web presence monitoring is often included in a web presence management system and can serve multiple purposes for both larger corporations and certain individuals, such as celebrities. It is important to note that presence monitoring differs from content listening. The former involves monitoring the properties (e.g. branding) of a web property in an established inventory, whereas the latter refers to analyzing content that already exists to hear how a person or business is seen often in near real time. Additionally, presence monitoring focuses on owned media and content listening on earned media. === Corporate, brand, and regulatory compliance === Many companies ensure that certain standards are met for a property on the web that represents their business. For companies in regulated industries, such as finance and healthcare, the company may be required by law to ensure that all publicized content, regardless of platform or technology, follow specific requirements. The widespread growth of social media has seen a rise in the number of fraudulent corporate impersonations. It has become much easier for a new web presence to be created on these platforms, and so these are much more prevalent than they used to be. As a preventive measure, a web presence monitoring system alerts the company when a known property is changed, allowing for the property to be reviewed and amended so that it follows the proper standards. . A web presence monitoring system helps alert the company when a known property is changed, so it can be reviewed and brought back, if necessary, into compliance with the appro
Data integration
Data integration is the process of combining, sharing, or synchronizing data from multiple sources to provide users with a unified view. There are a wide range of possible applications for data integration, from commercial (such as when a business merges multiple databases) to scientific (combining research data from different bioinformatics repositories). The decision to integrate data tends to arise when the volume, complexity (that is, big data) and need to share existing data explodes. It has become the focus of extensive theoretical work, and numerous open problems remain unsolved. Data integration encourages collaboration between internal as well as external users. The data being integrated must be received from a heterogeneous database system and transformed to a single coherent data store that provides synchronous data across a network of files for clients. A common use of data integration is in data mining when analyzing and extracting information from existing databases that can be useful for Business information. == History == Issues with combining heterogeneous data sources, often referred to as information silos, under a single query interface have existed for some time. In the early 1980s, computer scientists began designing systems for interoperability of heterogeneous databases. The first data integration system driven by structured metadata was designed in 1991 at the University of Minnesota for the Integrated Public Use Microdata Series (IPUMS). IPUMS used a data warehousing approach, which extracts, transforms, and loads data from heterogeneous sources into a unique view schema so data from different sources become compatible. By making thousands of population databases interoperable, IPUMS demonstrated the feasibility of large-scale data integration. The data warehouse approach offers a tightly coupled architecture because the data are already physically reconciled in a single queryable repository, so it usually takes little time to resolve queries. The data warehouse approach is less feasible for data sets that are frequently updated, requiring the extract, transform, load (ETL) process to be continuously re-executed for synchronization. Difficulties also arise in constructing data warehouses when one has only a query interface to summary data sources and no access to the full data. This problem frequently emerges when integrating several commercial query services like travel or classified advertisement web applications. A trend began in 2009 favoring the loose coupling of data and providing a unified query-interface to access real time data over a mediated schema (see Figure 2), which allows information to be retrieved directly from original databases. This is consistent with the SOA approach popular in that era. This approach relies on mappings between the mediated schema and the schema of original sources, and translating a query into decomposed queries to match the schema of the original databases. Such mappings can be specified in two ways: as a mapping from entities in the mediated schema to entities in the original sources (the "Global-as-View" (GAV) approach), or as a mapping from entities in the original sources to the mediated schema (the "Local-as-View" (LAV) approach). The latter approach requires more sophisticated inferences to resolve a query on the mediated schema, but makes it easier to add new data sources to a (stable) mediated schema. As of 2010, some of the work in data integration research concerns the semantic integration problem. This problem addresses not the structuring of the architecture of the integration, but how to resolve semantic conflicts between heterogeneous data sources. For example, if two companies merge their databases, certain concepts and definitions in their respective schemas like "earnings" inevitably have different meanings. In one database it may mean profits in dollars (a floating-point number), while in the other it might represent the number of sales (an integer). A common strategy for the resolution of such problems involves the use of ontologies which explicitly define schema terms and thus help to resolve semantic conflicts. This approach represents ontology-based data integration. On the other hand, the problem of combining research results from different bioinformatics repositories requires bench-marking of the similarities, computed from different data sources, on a single criterion such as positive predictive value. This enables the data sources to be directly comparable and can be integrated even when the natures of experiments are distinct. As of 2011, it was determined that current data modeling methods were imparting data isolation into every data architecture in the form of islands of disparate data and information silos. This data isolation is an unintended artifact of the data modeling methodology that results in the development of disparate data models. Disparate data models, when instantiated as databases, form disparate databases. Enhanced data model methodologies have been developed to eliminate the data isolation artifact and to promote the development of integrated data models. One enhanced data modeling method recasts data models by augmenting them with structural metadata in the form of standardized data entities. As a result of recasting multiple data models, the set of recast data models will now share one or more commonality relationships that relate the structural metadata now common to these data models. Commonality relationships are a peer-to-peer type of entity relationships that relate the standardized data entities of multiple data models. Multiple data models that contain the same standard data entity may participate in the same commonality relationship. When integrated data models are instantiated as databases and are properly populated from a common set of master data, then these databases are integrated. Since 2011, data hub approaches have been of greater interest than fully structured (typically relational) Enterprise Data Warehouses. Since 2013, data lake approaches have risen to the level of Data Hubs. (See all three search terms popularity on Google Trends.) These approaches combine unstructured or varied data into one location, but do not necessarily require an (often complex) master relational schema to structure and define all data in the Hub. In recent times, as the number of applications being used have increased many fold and application to application integration have become critical and this has given rise to [Unified APIs] that help application developers integrate their apps with other apps and more recently with [MCP - Model Context Protocol] taking it a step further for AI Agents. Data integration plays a big role in business regarding data collection used for studying the market. Converting the raw data retrieved from consumers into coherent data is something businesses try to do when considering what steps they should take next. Organizations are more frequently using data mining for collecting information and patterns from their databases, and this process helps them develop new business strategies to increase business performance and perform economic analyses more efficiently. Compiling the large amount of data they collect to be stored in their system is a form of data integration adapted for Business intelligence to improve their chances of success. == Example == Consider a web application where a user can query a variety of information about cities (such as crime statistics, weather, hotels, demographics, etc.). Traditionally, the information must be stored in a single database with a single schema. But any single enterprise would find information of this breadth somewhat difficult and expensive to collect. Even if the resources exist to gather the data, it would likely duplicate data in existing crime databases, weather websites, and census data. A data-integration solution may address this problem by considering these external resources as materialized views over a virtual mediated schema, resulting in "virtual data integration". This means application-developers construct a virtual schema—the mediated schema—to best model the kinds of answers their users want. Next, they design "wrappers" or adapters for each data source, such as the crime database and weather website. These adapters simply transform the local query results (those returned by the respective websites or databases) into an easily processed form for the data integration solution (see figure 2). When an application-user queries the mediated schema, the data-integration solution transforms this query into appropriate queries over the respective data sources. Finally, the virtual database combines the results of these queries into the answer to the user's query. This solution offers the convenience of adding new sources by simply constructing an adapter or an application software blade for them. It contrasts with ETL systems or with a si
Security and Privacy in Computer Systems
Security and Privacy in Computer Systems is a paper by Willis Ware that was first presented to the public at the 1967 Spring Joint Computer Conference. == Significance == Ware's presentation was the first public conference session about information security and privacy in respect of computer systems, especially networked or remotely-accessed ones. The IEEE Annals of the History of Computing said that Ware's 1967 Spring Joint Computer Conference session, together with 1970's Ware report, marked the start of the field of computer security.
Data profiling
Data profiling is the process of examining the data available from an existing information source (e.g. a database or a file) and collecting statistics or informative summaries about that data. The purpose of these statistics may be to: Find out whether existing data can be easily used for other purposes Improve the ability to search data by tagging it with keywords, descriptions, or assigning it to a category Assess data quality, including whether the data conforms to particular standards or patterns Assess the risk involved in integrating data in new applications, including the challenges of joins Discover metadata of the source database, including value patterns and distributions, key candidates, foreign-key candidates, and functional dependencies Assess whether known metadata accurately describes the actual values in the source database Understanding data challenges early in any data intensive project, so that late project surprises are avoided. Finding data problems late in the project can lead to delays and cost overruns. Have an enterprise view of all data, for uses such as master data management, where key data is needed, or data governance for improving data quality. == Introduction == Data profiling refers to the analysis of information for use in a data warehouse in order to clarify the structure, content, relationships, and derivation rules of the data. Profiling helps to not only understand anomalies and assess data quality, but also to discover, register, and assess enterprise metadata. The result of the analysis is used to determine the suitability of the candidate source systems, usually giving the basis for an early go/no-go decision, and also to identify problems for later solution design. == How data profiling is conducted == Data profiling utilizes methods of descriptive statistics such as minimum, maximum, mean, mode, percentile, standard deviation, frequency, variation, aggregates such as count and sum, and additional metadata information obtained during data profiling such as data type, length, discrete values, uniqueness, occurrence of null values, typical string patterns, and abstract type recognition. The metadata can then be used to discover problems such as illegal values, misspellings, missing values, varying value representation, and duplicates. Different analyses are performed for different structural levels. E.g. single columns could be profiled individually to get an understanding of frequency distribution of different values, type, and use of each column. Embedded value dependencies can be exposed in a cross-columns analysis. Finally, overlapping value sets possibly representing foreign key relationships between entities can be explored in an inter-table analysis. Normally, purpose-built tools are used for data profiling to ease the process. The computational complexity increases when going from single column, to single table, to cross-table structural profiling. Therefore, performance is an evaluation criterion for profiling tools. == When is data profiling conducted? == According to Kimball, data profiling is performed several times and with varying intensity throughout the data warehouse developing process. A light profiling assessment should be undertaken immediately after candidate source systems have been identified and DW/BI business requirements have been satisfied. The purpose of this initial analysis is to clarify at an early stage if the correct data is available at the appropriate detail level and that anomalies can be handled subsequently. If this is not the case the project may be terminated. Additionally, more in-depth profiling is done prior to the dimensional modeling process in order assess what is required to convert data into a dimensional model. Detailed profiling extends into the ETL system design process in order to determine the appropriate data to extract and which filters to apply to the data set. Additionally, data profiling may be conducted in the data warehouse development process after data has been loaded into staging, the data marts, etc. Conducting data at these stages helps ensure that data cleaning and transformations have been done correctly and in compliance of requirements. == Benefits and examples == Data profiling can improve data quality, shorten the implementation cycle of major projects, and improve users' understanding of data. Discovering business knowledge embedded in data itself is one of the significant benefits derived from data profiling. It can improve data accuracy in corporate databases.