AI Face Swap App

AI Face Swap App — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

• 

  PropBank

  PropBank is a corpus that is annotated with verbal propositions and their arguments—a "proposition bank". Although "PropBank" refers to a specific corpus produced by Martha Palmer et al., the term propbank is also coming to be used as a common noun referring to any corpus that has been annotated with propositions and their arguments. The PropBank project has played a role in research in natural language processing, and has been used in semantic role labelling. == Comparison == PropBank differs from FrameNet, the resource to which it is most frequently compared, in several ways. PropBank is a verb-oriented resource, while FrameNet is centered on the more abstract notion of frames, which generalizes descriptions across similar verbs (e.g. "describe" and "characterize") as well as nouns and other words (e.g. "description"). PropBank does not annotate events or states of affairs described using nouns. PropBank commits to annotating all verbs in a corpus, whereas the FrameNet project chooses sets of example sentences from a large corpus and only in a few cases has annotated longer continuous stretches of text. PropBank-style annotations often remain close to the syntactic level, while FrameNet-style annotations are sometimes more semantically motivated. From the start, PropBank was developed with the idea of serving as training data for machine learning-based semantic role labeling systems in mind. It requires that all arguments to a verb be syntactic constituents and different senses of a word are only distinguished if the differences bear on the arguments. Due to such differences, semantic role labeling with respect to PropBank is often a somewhat easier task than producing FrameNet-style annotations.

  Read more →
• 

  Cambridge Semantics

  Cambridge Semantics is a privately held company headquartered in Boston, Massachusetts with an office in San Diego, California. The company is an enterprise big data management and exploratory analytics software company. == History == Cambridge Semantics was founded in 2007 by Sean Martin, Lee Feigenbaum, Simon Martin, Rouben Meschian, Ben Szekely and Emmett Eldred who all previously worked at IBM's Advanced Technology Internet Group. In 2012, Cambridge Semantics appointed Chuck Pieper as chief executive. Pieper was previously at Credit Suisse. In January 2016, Cambridge Semantics acquired SPARQL City and its graph database intellectual property. On April 18, 2024, Altair Engineering acquired Cambridge Semantics. On 26 March 2025, Siemens announced the acquisition of Altair. == Products == Anzo Smart Data Lake uses Semantic Web Technologies. It allows IT departments and their business users to access data. AnzoGraph DB Graph database. AnzoGraph DB is a massively parallel processing (MPP) native graph database built for diverse data harmonization and analytics at scale (trillions of triples and more), speed and deep link insights. It is used for embedded analytics that require graph algorithms, graph views, named queries, aggregates, geospatial, built-in data science functions, data warehouse-style BI and reporting functions. It allows users to load and query RDF data using SPARQL or Cypher for OLAP-style analytics. == Marketing == Cambridge Semantics named SIIA Codie award 2018 finalist. Cambridge Semantics named 2018 Gold Stevie Award Winner for 'Big Data Solutions'. Cambridge Semantics named KMWorld’s 2018 ‘100 Companies That Matter in Knowledge Management’. Cambridge Semantics named to Database Trends and Applications' 'Trend-Setting Products in Data and Information Management for 2018'. Cambridge Semantics named to KMWorld Trend-Setting Products of 2017. Cambridge Semantics named to Database Trends and Applications 'DBTA 100: The Companies That Matter Most in Data'. Cambridge Semantics named SIIA Codie award 2017 winner for ‘Best Text Analytics and Semantic Technology Solution’. Cambridge Semantics named 2017 Silver Stevie Award Winner for 'Big Data Solutions'. Cambridge Semantics named KMWorld’s 2017 ‘100 Companies That Matter in Knowledge Management’. Cambridge Semantics named SIIA Codie award 2016 finalist. Cambridge Semantics named KMWorld’s 2016 ‘100 Companies That Matter in Knowledge Management’ and KMWorld Trend-Setting Products of 2015. Cambridge Semantics named 2016 Bio-IT World Best of Show People's Choice Award Contenders and 2015 Bio-IT best of show finalist. Anzo Insider Trading Investigation and Surveillance named 2015 CODiE Award finalist. Cambridge Semantics Selected as Finalist for 2014 MIT Sloan CIO Symposium's Innovation Showcase. Cambridge Semantics named SIIA CODiE Award 2014 finalist. Cambridge Semantics Win 2013 SIIA CODiE Award for best business intelligence and analytics solution. Cambridge Semantics wins KMWorld 2012 Promise Award. Cambridge Semantics wins Best of Show at 2012 Bio-IT World Conference.

  Read more →
• 

  Secret London

  Secret London is a Facebook group started by 21-year-old Bristol University graduate, Tiffany Philippou, on 19 January 2010 in response to a Saatchi & Saatchi competition. The group grew rapidly (180,000 members as of 8 February 2010) and is composed mostly of Londoners who use the site to share suggestions and photos of London. After the group's early success, the founder announced her intention to launch a website of the same name by crowdsourcing the design and development. The website was launched on 16 February 2010. == Other secret cities == Following the initial success of Secret London, a number of other secret groups were independently started around the world, some of which already have over 100,000 users. As of 19 February 2010, the list of other groups includes: Secret Frankfurt, Secret Tel Aviv, Secret Paris, Secret New York, Secret Tokyo, Secret Toronto, Secret Los Angeles, Secret Exeter, Secret Boston, Secret Norwich, Secret Singapore, Secret Brighton, Secret Minneapolis, Secret Sydney, Secret Canberra, Secret Brisbane, Secret Wellington, Secret Christchurch, Secret Madeira, Secret Funchal, Secret Bristol and Secret Cardiff. == Controversy == Some commentators have questioned whether it possible to share secrets without compromising them, and whether sharing tips publicly will lead to over-exposure of the businesses who are recommended.

  Read more →
• 

  Bus encryption

  Bus encryption is the use of encrypted program instructions on a data bus in a computer that includes a secure cryptoprocessor for executing the encrypted instructions. Bus encryption is used primarily in electronic systems that require high security, such as automated teller machines, TV set-top boxes, and secure data communication devices such as two-way digital radios. Bus encryption can also mean encrypted data transmission on a data bus from one processor to another processor. For example, from the CPU to a GPU which does not require input of encrypted instructions. Such bus encryption is used by Windows Vista and newer Microsoft operating systems to protect certificates, BIOS, passwords, and program authenticity. PVP-UAB (Protected Video Path) provides bus encryption of premium video content in PCs as it passes over the PCIe bus to graphics cards to enforce digital rights management. The need for bus encryption arises when multiple people have access to the internal circuitry of an electronic system, either because they service and repair such systems, stock spare components for the systems, own the system, steal the system, or find a lost or abandoned system. Bus encryption is necessary not only to prevent tampering of encrypted instructions that may be easily discovered on a data bus or during data transmission, but also to prevent discovery of decrypted instructions that may reveal security weaknesses that an intruder can exploit. In TV set-top boxes, it is necessary to download program instructions periodically to customer's units to provide new features and to fix bugs. These new instructions are encrypted before transmission, but must also remain secure on data buses and during execution to prevent the manufacture of unauthorized cable TV boxes. This can be accomplished by secure crypto-processors that read encrypted instructions on the data bus from external data memory, decrypt the instructions in the cryptoprocessor, and execute the instructions in the same cryptoprocessor.

  Read more →
• 

  Continuous Function Chart

  A Continuous Function Chart (CFC) is a graphic editor that can be used in conjunction with the STEP 7 software package or with other tools, such as CODESYS. It is used to create the entire software structure of the CPU from ready-made blocks. When working with the editor, you place blocks on function charts, assign parameters to them, and interconnect them. Interconnecting means, for example, that values are transferred from one output to one or more inputs during communication between the blocks. Continuous function charts are basically used for controlling continuous processes, where all the logic is executed and outputs are calculated in each PLC scan. Whereas in SFC, execution will be sequential as done is batch processes.

  Read more →
• 

  Multiple encryption

  Multiple encryption is the process of encrypting an already encrypted message one or more times, either using the same or a different algorithm. It is also known as cascade encryption, cascade ciphering, cipher stacking, multiple encryption, and superencipherment. Superencryption refers to the outer-level encryption of a multiple encryption. Some cryptographers, like Matthew Green of Johns Hopkins University, say multiple encryption addresses a problem that mostly doesn't exist: Modern ciphers rarely get broken... You’re far more likely to get hit by malware or an implementation bug than you are to suffer a catastrophic attack on Advanced Encryption Standard (AES). However, from the previous quote an argument for multiple encryption can be made, namely poor implementation. Using two different cryptomodules and keying processes from two different vendors requires both vendors' wares to be compromised for security to fail completely. == Independent keys == Picking any two ciphers, if the key used is the same for both, the second cipher could possibly undo the first cipher, partly or entirely. This is true of ciphers where the decryption process is exactly the same as the encryption process (a reciprocal cipher) – the second cipher would completely undo the first. If an attacker were to recover the key through cryptanalysis of the first encryption layer, the attacker could possibly decrypt all the remaining layers, assuming the same key is used for all layers. To prevent that risk, one can use keys that are statistically independent for each layer (e.g. independent RNGs). Ideally each key should have separate and different generation, sharing, and management processes. == Independent Initialization Vectors == For en/decryption processes that require sharing an Initialization Vector (IV) / nonce these are typically, openly shared or made known to the recipient (and everyone else). Its good security policy never to provide the same data in both plaintext and ciphertext when using the same key and IV. Therefore, its recommended (although at this moment without specific evidence) to use separate IVs for each layer of encryption. == Importance of the first layer == With the exception of the one-time pad, no cipher has been theoretically proven to be unbreakable. Furthermore, some recurring properties may be found in the ciphertexts generated by the first cipher. Since those ciphertexts are the plaintexts used by the second cipher, the second cipher may be rendered vulnerable to attacks based on known plaintext properties (see references below). This is the case when the first layer is a program P that always adds the same string S of characters at the beginning (or end) of all ciphertexts (commonly known as a magic number). When found in a file, the string S allows an operating system to know that the program P has to be launched in order to decrypt the file. This string should be removed before adding a second layer. To prevent this kind of attack, one can use the method provided by Bruce Schneier: Generate a random pad R of the same size as the plaintext. Encrypt R using the first cipher and key. XOR the plaintext with the pad, then encrypt the result using the second cipher and a different (!) key. Concatenate both ciphertexts in order to build the final ciphertext. A cryptanalyst must break both ciphers to get any information. This will, however, have the drawback of making the ciphertext twice as long as the original plaintext. Note, however, that a weak first cipher may merely make a second cipher that is vulnerable to a chosen plaintext attack also vulnerable to a known plaintext attack. However, a block cipher must not be vulnerable to a chosen plaintext attack to be considered secure. Therefore, the second cipher described above is not secure under that definition, either. Consequently, both ciphers still need to be broken. The attack illustrates why strong assumptions are made about secure block ciphers and ciphers that are even partially broken should never be used. == The Rule of Two == The Rule of Two is a data security principle from the NSA's Commercial Solutions for Classified Program (CSfC). It specifies two completely independent layers of cryptography to protect data. For example, data could be protected by both hardware encryption at its lowest level and software encryption at the application layer. It could mean using two FIPS-validated software cryptomodules from different vendors to en/decrypt data. The importance of vendor and/or model diversity between the layers of components centers around removing the possibility that the manufacturers or models will share a vulnerability. This way if one components is compromised there is still an entire layer of encryption protecting the information at rest or in transit. The CSfC Program offers solutions to achieve diversity in two ways. "The first is to implement each layer using components produced by different manufacturers. The second is to use components from the same manufacturer, where that manufacturer has provided NSA with sufficient evidence that the implementations of the two components are independent of one another." The principle is practiced in the NSA's secure mobile phone called Fishbowl. The phones use two layers of encryption protocols, IPsec and Secure Real-time Transport Protocol (SRTP), to protect voice communications. The Samsung Galaxy S9 Tactical Edition is also an approved CSfC Component.

  Read more →
• 

  Manufacturing Automation Protocol

  Manufacturing Automation Protocol (MAP) was a computer network standard released in 1982 for interconnection of devices from multiple manufacturers. It was developed by General Motors to combat the proliferation of incompatible communications standards used by suppliers of automation products such as programmable controllers. By 1985 demonstrations of interoperability were carried out and 21 vendors offered MAP products. In 1986 the Boeing corporation merged its Technical Office Protocol with the MAP standard, and the combined standard was referred to as "MAP/TOP". The standard was revised several times between the first issue in 1982 and MAP 3.0 in 1987, with significant technical changes that made interoperation between different revisions of the standard difficult. Although promoted and used by manufacturers such as General Motors, Boeing, and others, it lost market share to the contemporary Ethernet standard and was not widely adopted. Difficulties included changing protocol specifications, the expense of MAP interface links, and the speed penalty of a token-passing network. The token bus network protocol used by MAP became standardized as IEEE standard 802.4 but this committee disbanded in 2004 due to lack of industry attention.

  Read more →
• 

  Eduroam

  eduroam (a portmanteau of education and roaming) is an international Wi-Fi internet access roaming service for users in research, higher education and further education. It provides researchers, teachers, and students network access when visiting an institution other than their own. Users are authenticated with credentials from their home institution, regardless of the location of the eduroam access point. Authorization to access the Internet and other resources are handled by the visited institution. Users do not have to pay to use eduroam. In some countries, Internet access via eduroam is also available at other locations than the participating institutions, e.g. in libraries, public buildings, railway stations, city centres and airports. It is also available at many primary and secondary education institutions in Brazil and the US. == History == The eduroam initiative started in 2002 when during the preparations for the creation of TERENA's task force TF-Mobility, Klaas Wierenga of SURFnet shared the idea of combining a RADIUS-based infrastructure with IEEE 802.1X technology to provide roaming network access across research and education networks. Initially, the service was joined by institutions in the Netherlands, Germany, Finland, Portugal, Croatia and the United Kingdom. Later, other NRENs in Europe embraced the idea and started joining the infrastructure, which was then called eduroam. Since 2004, the European Union co-funded further research and development work related to the eduroam service through the GN2 and GN3 projects. From September 2007, the European Union also funded through these projects the continued operation and maintenance of the eduroam service at the European level. The first non-European country to join eduroam was Australia, in December 2004. In Canada, eduroam started as an initiative of the University of British Columbia, which was later taken over by CANARIE as a service of its Canadian Access Federation. In the United States, eduroam was initially a pilot project between the National Science Foundation and the University of Tennessee (UTK). In 2012, Internet2 announced the addition of eduroam to its NET+ service offerings. AnyRoam LLC, a private company, was formed by former UTK staff to serve as an Internet2 active corporate member administering the US top-level servers. In 2021, Internet2 assumed direct management of the eduroam service for US-based organizations. == Technology == The eduroam service uses IEEE 802.1X as the authentication method and a hierarchical system of RADIUS servers. The hierarchy typically consists of RADIUS servers at the participating institutions, national RADIUS servers run by the National Roaming Operators, and regional top-level RADIUS servers for individual world regions. In some cases, institutions contact each other directly via DNS lookups () When a user visits a remote institution, the user's device presents their credentials to the local RADIUS server. That RADIUS server discovers that it is not responsible for the realm of the user's home institution and proxies the access request to another RADIUS server, typically the national RADIUS server. If the visited institution is in a different country than the home institution, the request is in turn proxied to the regional top-level RADIUS server, and then to the national RADIUS server of the user's home country. That national server forwards the credentials to the home institution, where they are verified. The RADIUS response travels back over the proxy-hierarchy to the visited institution and the user is granted access. In eduroam, the user credentials are always presented in the form of an EAP method (). The EAP method is responsible for ensuring that the users credentials are secure, and private. The users credentials can then travel via a number of intermediate servers, not under the control of the home institution of the user. This requirement limits the types of EAP methods that can be used. EAP methods which do not provide for security or privacy of user credentials cannot be used in eduroam. The most commonly used EAP methods in eduroam are EAP-TLS, PEAP, and EAP-TTLS. The methods used generally fall into two broad categories: those that use credentials in the form of some public-key mechanism with certificates and those that use so-called tunnelled authentication with "inner" passwords or other credentials. Most institutions use a tunnelled authentication method that requires a server certificate. These server certificates are used to set up a secure tunnel between the mobile device and the authentication server, through which the user credentials (e.g. name and password) are securely transported. A complication arises if the user's home institution does not use a two-letter country-code top-level domain as part of its realm, but a generic top-level domain such as .edu or .org. By inspection of such realms, it is not possible to determine which national RADIUS server the request should be routed to. Such domains will thus, by default, fail to work in international roaming. The workaround for this problem involves the creation of exceptions in the international RADIUS request routing tables; however, this workaround does not scale as the number of exception entries grows. Several solutions have been proposed to eliminate this workaround in the future, the most promising of which is RADIUS over TLS with Dynamic Discovery, which does not rely on static routing tables inside a RADIUS server configuration to route requests to their proper destination. Instead, the participating institution adds one NAPTR DNS resource record to its own domain's DNS zone, which states by which server eduroam authentication for the domain is handled. == Governance == GÉANT has established a lightweight global governance structure. Recognising the large variety in the organisation and funding of research and education (networking) in different countries and regions, rules imposed on the operations of eduroam are limited to technical and administrative requirements that are necessary to ensure the smooth and secure operations of eduroam worldwide. Moreover, the eduroam operators have the leading role in creating and maintaining the rules of the global eduroam governance. The Global eduroam Governance Committee (GeGC) has the central role in the global eduroam governance structure. While its structure has evolved over time, it presently has three representatives from each of five regions — mirroring those used by the Regional Internet registries — serving a two-year term. In addition, GÉANT may appoint one or more experts as non-voting members of the GeGC. == Geographical deployment == eduroam is available at selected locations in countries with a National Roaming Operator that has signed the eduroam Compliance Statement. Those sixty-seven countries are listed below. In addition, there may be pilot deployments in countries that are in the process of joining eduroam. === Middle East === eduroam is deployed in: === Europe === The NRENs that are members of the consortium of the GN3 project have joined the European eduroam confederation by signing the confederation's policy that requires its members to comply with a set of technical and organisational requirements, which are more specific than those in the global eduroam Compliance Statement. As a consequence, eduroam is deployed in the following countries: In addition, three NRENs that are associate members of the consortium of the GN3 project without voting rights joined the European eduroam confederation; they represent Belarus (UIIP), Moldova (RENAM) and Russia (Joint Supercomputer Center of the Russian Academy of Sciences). Finally, five NRENs not involved in the GN3 project joined the European eduroam confederation on a voluntary basis, enabling the deployment of the service in: The European top-level RADIUS servers are operated by SURFnet and Forskningsnettet. === Asia-Pacific === eduroam is deployed in the following countries and economies: The Asia-Pacific top-level RADIUS servers are operated by AARNet and by the University of Hong Kong. === North America === eduroam is deployed in: === Latin America === eduroam is deployed in: === Africa === eduroam is deployed in: The inter-African RADIUS servers are operated by West-African research and education network WACREN, the UbuntuNet Alliance and TENET.

  Read more →
• 

  IT8

  IT8 is a set of American National Standards Institute (ANSI) standards for color communications and control specifications. Formerly governed by the IT8 Committee, IT8 activities were merged with those of the Committee for Graphics Arts Technologies Standards (CGATS Archived November 9, 2018, at the Wayback Machine) in 1994. == Standards list == The following is a list of the IT8 standards, according to the NPES Standards Blue Book Archived July 19, 2011, at the Wayback Machine: === IT8.6 - 2002 - Graphic technology - Prepress digital data exchange - Diecutting data (DDES3) === This standard establishes a data exchange format to enable transfer of numerical control information between diecutting systems and electronic prepress systems. The information will typically consist of numerical control information used in the manufacture of dies. 37 pp. === IT8.7/1 - 1993 (R2003) - Graphic technology - Color transmission target for input scanner calibration === This standard defines an input test target that will allow any color input scanner to be calibrated with any film dye set used to create the target. It is intended to address the color transparency products that are generally used for input to the preparatory process for printing and publishing. This standard defines the layout and colorimetric values of a target that can be manufactured on any positive color transparency film and that is intended for use in the calibration of a photographic film/scanner combination. 32 pp. === IT8.7/2 - 1993 (R2003) Graphic technology - Color reflection target for input scanner calibration === This standard defines an input test target that will allow any color input scanner to be calibrated with any film dye set used to create the target. It is intended to address the color photographic paper products that are generally used for input to the preparatory process for printing and publishing. It defines the layout and colorimetric values of the target that can be manufactured on any color photographic paper and is intended for use in the calibration of a photographic paper/scanner combination. 29 pp. === IT8.7/3 - 1993 (R2003) Graphic technology - Input data for characterization of 4-color process printing === The purpose of this standard is to specify an input data file, a measurement procedure and an output data format to characterize any four-color printing process. The output data (characterization) file should be transferred with any four-color (cyan, magenta, yellow and black) halftone image files to enable a color transformation to be undertaken when required. 29 pp. == Targets == Calibrating all devices involved in the process chain (original, scanner/digital camera, monitor/printer) is required for an authentic color reproduction, because their actual color spaces differ device-specifically from the reference color spaces. An IT8 calibration is done with what are called IT8 targets, which are defined by the IT8 standards. Example Special targets, implementing the IT8.7/1 (transparent target) or IT8.7/2 (reflective target) standards, are needed for calibrating scanners. These targets consists of 24 grey fields and 264 color fields in 22 columns: Column 01 to 12: HCL color model, which differ in Hue, Chroma, and Lightness Column 13 to 16: CMYK-Colors Cyan, Magenta, Yellow, and Key (black) in different steps of brightness Column 17 to 19: RGB-Colors Red, Green, and Blue in different steps of brightness Column 20 to 22: undefined, producers' choice After scanning such a target, an ICC profile gets calculated on the basis of reference values. This profile is used for all subsequent scans and assures color fidelity.

  Read more →
• 

  Change data capture

  In databases, change data capture (CDC) is a set of software design patterns used to determine and track the data that has changed (the "deltas") so that action can be taken using the changed data. The result is a delta-driven dataset. CDC is an approach to data integration that is based on the identification, capture and delivery of the changes made to enterprise data sources. For instance it can be used for incremental update of data loading. CDC occurs often in data warehouse environments since capturing and preserving the state of data across time is one of the core functions of a data warehouse, but CDC can be utilized in any database or data repository system. == Methodology == System developers can set up CDC mechanisms in a number of ways and in any one or a combination of system layers from application logic down to physical storage. In a simplified CDC context, one computer system has data believed to have changed from a previous point in time, and a second computer system needs to take action based on that changed data. The former is the source, the latter is the target. It is possible that the source and target are the same system physically, but that would not change the design pattern logically. Multiple CDC solutions can exist in a single system. === Timestamps on rows === Tables whose changes must be captured may have a column that represents the time of last change. Names such as LAST_UPDATE, LAST_MODIFIED, etc. are common. Any row in any table that has a timestamp in that column that is more recent than the last time data was captured is considered to have changed. Timestamps on rows are also frequently used for optimistic locking so this column is often available. === Version numbers on rows === Database designers give tables whose changes must be captured a column that contains a version number. Names such as VERSION_NUMBER, etc. are common. One technique is to mark each changed row with a version number. A current version is maintained for the table, or possibly a group of tables. This is stored in a supporting construct such as a reference table. When a change capture occurs, all data with the latest version number is considered to have changed. Once the change capture is complete, the reference table is updated with a new version number. (Do not confuse this technique with row-level versioning used for optimistic locking. For optimistic locking each row has an independent version number, typically a sequential counter. This allows a process to atomically update a row and increment its counter only if another process has not incremented the counter. But CDC cannot use row-level versions to find all changes unless it knows the original "starting" version of every row. This is impractical to maintain.) === Status indicators on rows === This technique can either supplement or complement timestamps and versioning. It can configure an alternative if, for example, a status column is set up on a table row indicating that the row has changed (e.g., a boolean column that, when set to true, indicates that the row has changed). Otherwise, it can act as a complement to the previous methods, indicating that a row, despite having a new version number or a later date, still shouldn't be updated on the target (for example, the data may require human validation). === Time/version/status on rows === This approach combines the three previously discussed methods. As noted, it is not uncommon to see multiple CDC solutions at work in a single system, however, the combination of time, version, and status provides a particularly powerful mechanism and programmers should utilize them as a trio where possible. The three elements are not redundant or superfluous. Using them together allows for such logic as, "Capture all data for version 2.1 that changed between 2005-06-01 00:00 and 2005-07-01 00:00 where the status code indicates it is ready for production." === Triggers on tables === May include a publish/subscribe pattern to communicate the changed data to multiple targets. In this approach, triggers log events that happen to the transactional table into another queue table that can later be "played back". For example, imagine an Accounts table, when transactions are taken against this table, triggers would fire that would then store a history of the event or even the deltas into a separate queue table. The queue table might have schema with the following fields: Id, TableName, RowId, Timestamp, Operation. The data inserted for our Account sample might be: 1, Accounts, 76, 2008-11-02 00:15, Update. More complicated designs might log the actual data that changed. This queue table could then be "played back" to replicate the data from the source system to a target. Data capture offers a challenge in that the structure, contents and use of a transaction log is specific to a database management system. Unlike data access, no standard exists for transaction logs. Most database management systems do not document the internal format of their transaction logs, although some provide programmatic interfaces to their transaction logs (for example: Oracle, DB2, SQL/MP, SQL/MX and SQL Server 2008). Other challenges in using transaction logs for change data capture include: Coordinating the reading of the transaction logs and the archiving of log files (database management software typically archives log files off-line on a regular basis). Translation between physical storage formats that are recorded in the transaction logs and the logical formats typically expected by database users (e.g., some transaction logs save only minimal buffer differences that are not directly useful for change consumers). Dealing with changes to the format of the transaction logs between versions of the database management system. Eliminating uncommitted changes that the database wrote to the transaction log and later rolled back. Dealing with changes to the metadata of tables in the database. CDC solutions based on transaction log files have distinct advantages that include: minimal impact on the database (even more so if one uses log shipping to process the logs on a dedicated host). no need for programmatic changes to the applications that use the database. low latency in acquiring changes. transactional integrity: log scanning can produce a change stream that replays the original transactions in the order they were committed. Such a change stream include changes made to all tables participating in the captured transaction. no need to change the database schema == Confounding factors == As often occurs in complex domains, the final solution to a CDC problem may have to balance many competing concerns. === Unsuitable source systems === Change data capture both increases in complexity and reduces in value if the source system saves metadata changes when the data itself is not modified. For example, some Data models track the user who last looked at but did not change the data in the same structure as the data. This results in noise in the Change Data Capture. === Tracking the capture === Actually tracking the changes depends on the data source. If the data is being persisted in a modern database then Change Data Capture is a simple matter of permissions. Two techniques are in common use: Tracking changes using database triggers Reading the transaction log as, or shortly after, it is written. If the data is not in a modern database, CDC becomes a programming challenge. === Push versus pull === Push: the source process creates a snapshot of changes within its own process and delivers rows downstream. The downstream process uses the snapshot, creates its own subset and delivers them to the next process. Pull: the target that is immediately downstream from the source, prepares a request for data from the source. The downstream target delivers the snapshot to the next target, as in the push model. === Alternatives === Sometimes the slowly changing dimension is used as an alternative method. CDC and SCD are similar in that both methods can detect changes in a data set. The most common forms of SCD are type 1 (overwrite), type 2 (maintain history) or 3 (only previous and current value). SCD 2 can be useful if history is needed in the target system. CDC overwrites in the target system (akin to SCD1), and is ideal when only the changed data needs to arrive at the target, i.e. a delta-driven dataset.

  Read more →
• 

  Social media use in education

  Social media in education is the use of social media to enhance education. Social media are "a group of Internet-based applications...that allow the creation and exchange of user-generated content". It is also known as the read/write web. As time went on and technology evolved, social media has been an integral part of people's lives, including students, scholars, and teachers. However, social media are controversial because, in addition to providing new means of connection, critics claim that they damage self-esteem, shorten attention spans, and increase mental health issues. A 2016 dissertation presented surveys that focused on the impact of social media. It reported that 54.6% of students believed that social media affected their studies positively (38% agree, 16.6% strongly agree). About 40% disagreed, and 4.7% of students strongly disagreed. 53% of female students reported that social media negatively impacted their studies. Among male students, 40% agreed that social media had a negative impact on studies, while 59% disagreed. A 2023 article dives deep into the rewards system of the brain in response to social media. This study compares the social rewards system in our brain to those from social media. From ages 10-12, most are receiving a cell phone, social rewards in the brain start to feel more satisfying. Leading to adulthood, the effects of social rewards are less likely to feel reliant on feedback from peers. Equivalent to a more mature prefrontal cortex, this enables a better management of their emotional reaction to these social rewards, meaning a more balanced and controlled reaction. == History == A survey from Cambridge International of nearly 20,000 teachers and students (ages 12–19) from 100 countries found that 48% of students use a desktop computer in class, 42% uses phones, 33% use interactive whiteboards and 20% use tablets. Desktop computers are more used than tablets. Teachers were abandoning the "no phones at school" rule. A 2024 research survey through Common Sense Education reported 54% of age 8-12 and 69% of ages 13-18 social media is an extensive distraction from homework. === United States === The long-running technology boom accelerated after the millennium. As of 2018, 95% of US teenage students had access to a smartphone and 45% said they were online almost constantly. In the early days of social media, access to technology was a significant issue as many students did not own not compatible devices and school budgets were often insufficient to purchase devices for student use. Despite backlash, Missouri passed a law that prohibited teachers from communicating privately with students over social media in 2011. Supporters were concerned that online communication between underage students and faculty could lead to inappropriate relationships. Some schools adopted a "Bring Your Own Device" (BYOD) policy, allowing students to bring Internet-accessing devices, such as phones or tablets to class. During the pandemic, the federal government offered funds that allowed more schools to purchase devices. Over time, more students acquired phones with social media access. Personal devices increased student satisfaction, but reduced teachers' ability to control device use in their classrooms. A 2018 Pew Research study reported that 95% of teenagers had a phone and used social media consistently. === Canada === The Peel District School Board (PDSB) in Ontario accepted the use of social media in the classroom. In 2013, the PDSB introduced BYOD and unblocked many social media sites. That was later replaced by a policy that dealt specifically with social media. == Uses == === Classroom === In the classroom, social media offers a way to systematically distribute and gather information from students. Teachers can supply documents, and audio/video media to students for immediate or later use. One study on higher education reported that devices and social media: created opportunities for interaction provided occasions for collaboration sped up information access offered more ways to learn situated learning. Frustrations included anti-technology instructors, device challenges, and devices as a distraction. Social media in classrooms can have a negative effect. A Yale University publication reported that students who used laptops in class for non-academic reasons had poorer performance. Students spent most of their time on social media, shopping, and other personal activities. Social media has helped many educators mentor their students more effectively. === Outside of class === Social media offer a venue for video calls, stories, feeds, and game playing that can enhance the learning process. Teachers can utilize social media to communicate with their students. Social media can provide students with resources that they can utilize in essays, projects, and presentations. Students can easily access comments made by teachers and peers and offer feedback to teachers. Social media can offer students the opportunity to collaborate by sharing information without requiring face to face meetings. Social media can allow students to more easily connect with experts, to go beyond course materials. Instructors in a 2010 study reported that online technologies (social media) can help students become comfortable having discussions outside the classroom better than traditional means. Teachers may face some risk when using social media outside the classroom, without appropriate work rules. Studies explores how college students' engagement with social media platforms influences their communication preferences and habits, particularly in relation to using school email for academic purposes. === Professional development === Social media can aid professional development, as teachers become students, enhancing knowledge transfer, skill master, and collaboration. === Non-academic uses === Schools can use social media to make public announcements. Teachers and administrators can communicate other important information to parents and students and to receive feedback from them. Families can keep up with school events and policies. === Ecology education === The potential of using social media in ecological, nature and forest education include: virtual nature groups can help promote good habits in forest tourism and recreation (nature ethics), by entering general rules in the regulations by administrators, e.g. "DO NOT PICK UP PLANTS UNKNOWN TO US", which is to protects rare species from pointless picking. social media activity motivates people to learn about nature in the field, allows them to gain knowledge, dispels popular myths, enables contact with scientists and practitioners, promotes valuable literature, websites, and at the same time reveals distortions and substantive errors in popular news services. contact is not only virtual. Despite financial barriers and distance, Internet users organize nature conventions. Such meetings are an opportunity not only to make friends, but also to learn about nature together and have fun. the possibility of contact between scientists and nature lovers via Facebook has become a source of cooperation in species inventory, e.g. the online campaign of the NATRIX Herpetological Society, which consists not only of collecting reports of observations of the smooth snake by Internet users, but also of drawing attention to the biology and threats to this species. Social media has become a place where ecology education quickly reaches people of different ages and social statuses. The nature groups that have been created, in which nature lovers, biologists, foresters and scientists participate, can have a real impact on the state of knowledge and data collection through citizen science. == Apps and services == Social media can allow students to participate in their field by working with organizations outside the classroom. By offering easier access to peers outside the classroom, students can broaden their perspectives and find support resources. Social media aided learning outside of the classroom through collaboration and innovation. One specific study, "Exploring education-related use of social media," called this "audience connectors". Audience connectors bring students together while studying with WhatsApp and Facebook. This study reported that "60 percent [of students in the study] agreed that technology changes education for the better." While social media can promote a beneficial education platform, downsides exist. Students may become skilled at "lifting material from the internet" rather than enhancing their personal understanding. Another downside is student attention spans decline. A concern raised by the students of this study showed how many use spell-check as a crutch and will see a trend of points taken off when spell-check is not an option. Apps like X allowed teachers to make classroom accounts where students can learn about social media in a controlled context. Teachers can post assignments on th

  Read more →
• 

  Cloud Data Management Interface

  ISO/IEC 17826 Information technology — Cloud Data Management Interface (CDMI) Version 2.0.0 is an international standard that specifies a protocol for self-provisioning, administering and managing access to data stored in cloud storage, object storage, storage area network and network attached storage systems. The CDMI standard is developed and maintained by the Storage Networking Industry Association, who makes a publicly accessible version of the specification available. CDMI defines new resource representations to enable standardized management of any URI-accessible data, and defines RESTful HTTP operations using these representations to discover the capabilities of the storage system, discover stored data, access and update management metadata, specify data storage protocols (such as iSCSI and NFS) through which the stored data is accessed, and provide cross-system and cross-cloud import and export in order to enable data portability. Management functions enabled by CDMI include managing data ownership, identity mapping, access controls, user-specified metadata, and to declaratively specify desired data protection, data retention, constraints on geographic placement, desired quality of service, data versioning and security requirements. CDMI also defines utility services to facilitate data management, such the ability to query data matching specific criteria, and includes extensions to perform bulk updates using CDMI Jobs. == Capabilities == Compliant implementations must provide access to a set of configuration parameters known as capabilities. These are either boolean values that represent whether or not a system supports things such as queues, export via other protocols, path-based storage and so on, or numeric values expressing system limits, such as how much metadata may be placed on an object. As a minimal compliant implementation can be quite small, with few features, clients need to check the cloud storage system for a capability before attempting to use the functionality it represents. Resource allocation assignments limited to the data management interface protocols must possess access bypass capabilities which extend beyond the layered framework. This integral function is vital to the prevention of transport layer session hijacking by unauthorized entities which may circumvent standard interfacing security parameters. == Containers == A CDMI client may access objects, including containers, by either name or object id (OID), assuming the CDMI server supports both methods. When storing objects by name, it is natural to use nested named containers; the resulting structure corresponds exactly to a traditional filesystem directory structure. == Objects == Objects are similar to files in a traditional file system, but are enhanced with an increased amount and capacity for metadata. As with containers, they may be accessed by either name or OID. When accessed by name, clients use URLs that contain the full pathname of objects to create, read, update and delete them. When accessed by OID, the URL specifies an OID string in the cdmi-objectid container; this container presents a flat name space conformant with standard object storage system semantics. Subject to system limits, objects may be of any size or type and have arbitrary user-supplied metadata attached to them. Systems that support query allow arbitrary queries to be run against the metadata. == Domains, Users and Groups == CDMI supports the concept of a domain, similar in concept to a domain in the Windows Active Directory model. Users and groups created in a domain share a common administrative database and are known to each other on a "first name" basis, i.e. without reference to any other domain or system. Domains also function as containers for usage and billing summary data. == Access Control == CDMI exactly follows the ACL and ACE model used for file authorization operations by NFSv4. This makes it also compatible with Microsoft Windows systems. == Metadata == CDMI draws much of its metadata model from the XAM specification. Objects and containers have "storage system metadata", "data system metadata" and arbitrary user specified metadata, in addition to the metadata maintained by an ordinary filesystem (atime etc.). == Queries == CDMI specifies a way for systems to support arbitrary queries against CDMI containers, with a rich set of comparison operators, including support for regular expressions. == Queues == CDMI supports the concept of persistent FIFO (first-in, first-out) queues. These are useful for job scheduling, order processing and other tasks in which lists of things must be processed in order. == Compliance == Both retention intervals and retention holds are supported by CDMI. A retention interval consists of a start time and a retention period. During this time interval, objects are preserved as immutable and may not be deleted. A retention hold is usually placed on an object because of judicial action and has the same effect: objects may not be changed nor deleted until all holds placed on them are removed. == Billing == Summary information suitable for billing clients for on-demand services can be obtained by authorized users from systems that support it. == Serialization == Serialization of objects and containers allows export of all data and metadata on a system and importation of that data into another cloud system. == Foreign protocols == CDMI supports export of containers as NFS or CIFS shares. Clients that mount these shares see the container hierarchy as an ordinary filesystem directory hierarchy, and the objects in the containers as normal files. Metadata outside of ordinary filesystem metadata may or may not be exposed. Provisioning of iSCSI LUNs is also supported. == Client SDKs == CDMI Reference Implementation Droplet libcdmi-java libcdmi-python .NET SDK

  Read more →
• 

  Reflection lines

  Engineers use reflection lines to judge a surface's quality. Reflection lines reveal surface flaws, particularly discontinuities in normals indicating that the surface is not C 2 {\displaystyle C^{2}} . Reflection lines may be created and examined on physical surfaces or virtual surfaces with the help of computer graphics. For example, the shiny surface of an automobile body is illuminated with reflection lines by surrounding the car with parallel light sources. Virtually, a surface can be rendered with reflection lines by modulating the surfaces point-wise color according to a simple calculation involving the surface normal, viewing direction and a square wave environment map. == Mathematical definition == Consider a point p {\displaystyle p} on a surface M {\displaystyle M} with (normalized) normal n {\displaystyle n} . If an observer views this point from infinity at view direction v {\displaystyle v} then the reflected view direction r {\displaystyle r} is: r = v − 2 ( n ⋅ v ) n . {\displaystyle r=v-2(n\cdot v)n.} (The vector v {\displaystyle v} is decomposed into its normal part v n = ( n ⋅ v ) v {\displaystyle v_{n}=(n\cdot v)v} and tangential part v t = v − v n {\displaystyle v_{t}=v-v_{n}} . Upon reflection, the tangential part is kept and the normal part is negated.) For reflection lines we consider the surface M {\displaystyle M} surrounded by parallel lines with direction a {\displaystyle a} , representing infinite, non-dispersive light sources. For each point p {\displaystyle p} on M {\displaystyle M} we determine which line is seen from direction v {\displaystyle v} . The position on each line is of no interest. Define the vector r p {\displaystyle r_{p}} to be the reflection direction r {\displaystyle r} projected onto a plane P {\displaystyle P} that is orthogonal to a {\displaystyle a} : r p = r − ( r ⋅ a ) a {\displaystyle r_{p}=r-(r\cdot a)a} and similarly let v p {\displaystyle v_{p}} be the viewing direction projected onto P {\displaystyle P} : v p = v − ( v ⋅ a ) a {\displaystyle v_{p}=v-(v\cdot a)a} Finally, define v o {\displaystyle v_{o}} to be the direction lying in P {\displaystyle P} perpendicular to a {\displaystyle a} and v p {\displaystyle v_{p}} : v o = a × v p {\displaystyle v_{o}=a\times v_{p}} Using these vectors, the reflection line function θ ( p ) : M → ( − π , π ] {\displaystyle \theta (p):M\rightarrow (-\pi ,\pi ]} is a scalar function mapping points p {\displaystyle p} on the surface to angles between v p {\displaystyle v_{p}} and r p {\displaystyle r_{p}} : θ = arctan ⁡ ( r p ⋅ v o , r p ⋅ v p ) {\displaystyle \theta =\arctan {(r_{p}\cdot v_{o},r_{p}\cdot v_{p})}} where a r c t a n ( y , x ) {\displaystyle arctan(y,x)} is the atan2 function producing a number in the range ( − π , π ] {\displaystyle (-\pi ,\pi ]} . ( v p {\displaystyle v_{p}} and v o {\displaystyle v_{o}} can be viewed as a local coordinate system in P {\displaystyle P} with x {\displaystyle x} -axis in direction v p {\displaystyle v_{p}} and y {\displaystyle y} -axis in direction v o {\displaystyle v_{o}} .) Finally, to render the reflection lines positive values θ > 0 {\displaystyle \theta >0} are mapped to a light color and non-positive values to a dark color. == Highlight lines == Highlight lines are a view-independent alternative to reflection lines. Here the projected normal is directly compared against some arbitrary vector x {\displaystyle x} perpendicular to the light source: θ = arctan ⁡ ( n a ⋅ a ⊥ , n a ⋅ x ) {\displaystyle \theta =\arctan {(n_{a}\cdot a^{\perp },n_{a}\cdot x)}} where n a {\displaystyle n_{a}} is the surface normal projected on the light source plane P {\displaystyle P} : n a ^ / | n a ^ | , n a ^ = n − ( n ⋅ a ) a {\displaystyle {\hat {n_{a}}}/|{\hat {n_{a}}}|,{\hat {n_{a}}}=n-(n\cdot a)a} The relationship between reflection lines and highlight lines is likened to that between specular and diffuse shading.

  Read more →
• 

  Key-agreement protocol

  In cryptography, a key-agreement protocol is a protocol whereby two (or more) parties generate a cryptographic key as a function of information provided by each honest party so that no party can predetermine the resulting value. In particular, all honest participants influence the outcome. A key-agreement protocol is a specialisation of a key-exchange protocol. At the completion of the protocol, all parties share the same key. A key-agreement protocol precludes undesired third parties from forcing a key choice on the agreeing parties. A secure key agreement can ensure confidentiality and data integrity in communications systems, ranging from simple messaging applications to complex banking transactions. Secure agreement is defined relative to a security model, for example the Universal Model. More generally, when evaluating protocols, it is important to state security goals and the security model. For example, it may be required for the session key to be authenticated. A protocol can be evaluated for success only in the context of its goals and attack model. An example of an adversarial model is the Dolev–Yao model. In many key exchange systems, one party generates the key, and sends that key to the other party; the other party has no influence on the key. == Exponential key exchange == The first publicly known public-key agreement protocol that meets the above criteria was the Diffie–Hellman key exchange, in which two parties jointly exponentiate a generator with random numbers, in such a way that an eavesdropper cannot feasibly determine what the resultant shared key is. Exponential key agreement in and of itself does not specify any prior agreement or subsequent authentication between the participants. It has thus been described as an anonymous key agreement protocol. == Symmetric key agreement == Symmetric key agreement (SKA) is a method of key agreement that uses solely symmetric cryptography and cryptographic hash functions as cryptographic primitives. It is related to symmetric authenticated key exchange. SKA may assume the use of initial shared secrets or a trusted third party with whom the agreeing parties share a secret is assumed. If no third party is present, then achieving SKA can be trivial: we tautologically assume that two parties that share an initial secret and have achieved SKA. SKA contrasts with key-agreement protocols that include techniques from asymmetric cryptography, such as key encapsulation mechanisms. The initial exchange of a shared key must be done in a manner that is private and integrity-assured. Historically, this was achieved by physical means, such as by using a trusted courier. An example of a SKA protocol is the Needham–Schroeder protocol. It establishes a session key between two parties on the same network, using a server as a trusted third party. The original Needham–Schroeder protocol is vulnerable to a replay attack. Timestamps and nonces are included to fix this attack. It forms the basis for the Kerberos protocol. === Types of key agreement === Boyd et al. classify two-party key agreement protocols according to two criteria as follows: whether a pre-shared key already exists or not the method of generating the session key. The pre-shared key may be shared between the two parties, or each party may share a key with a trusted third party. If there is no secure channel (as may be established via a pre-shared key), it is impossible to create an authenticated session key. The session key may be generated via: key transport, key agreement and hybrid. If there is no trusted third party, then the cases of key transport and hybrid session key generation are indistinguishable. SKA is concerned with protocols in which the session key is established using only symmetric primitives. == Authentication == Anonymous key exchange, like Diffie–Hellman, does not provide authentication of the parties, and is thus vulnerable to man-in-the-middle attacks. A wide variety of cryptographic authentication schemes and protocols have been developed to provide authenticated key agreement to prevent man-in-the-middle and related attacks. These methods generally mathematically bind the agreed key to other agreed-upon data, such as the following: public–private key pairs shared secret keys passwords === Public keys === A widely used mechanism for defeating such attacks is the use of digitally signed keys that must be integrity-assured: if Bob's key is signed by a trusted third party vouching for his identity, Alice can have considerable confidence that a signed key she receives is not an attempt to intercept by Eve. When Alice and Bob have a public-key infrastructure, they may digitally sign an agreed Diffie–Hellman key, or exchanged Diffie–Hellman public keys. Such signed keys, sometimes signed by a certificate authority, are one of the primary mechanisms used for secure web traffic (including HTTPS, SSL or TLS protocols). Other specific examples are MQV, YAK and the ISAKMP component of the IPsec protocol suite for securing Internet Protocol communications. However, these systems require care in endorsing the match between identity information and public keys by certificate authorities in order to work properly. === Hybrid systems === Hybrid systems use public-key cryptography to exchange secret keys, which are then used in a symmetric-key cryptography systems. Most practical applications of cryptography use a combination of cryptographic functions to implement an overall system that provides all of the four desirable features of secure communications (confidentiality, integrity, authentication, and non-repudiation). === Passwords === Password-authenticated key agreement protocols require the separate establishment of a password (which may be smaller than a key) in a manner that is both private and integrity-assured. These are designed to resist man-in-the-middle and other active attacks on the password and the established keys. For example, DH-EKE, SPEKE, and SRP are password-authenticated variations of Diffie–Hellman. === Other tricks === If one has an integrity-assured way to verify a shared key over a public channel, one may engage in a Diffie–Hellman key exchange to derive a short-term shared key, and then subsequently authenticate that the keys match. One way is to use a voice-authenticated read-out of the key, as in PGPfone. Voice authentication, however, presumes that it is infeasible for a man-in-the-middle to spoof one participant's voice to the other in real-time, which may be an undesirable assumption. Such protocols may be designed to work with even a small public value, such as a password. Variations on this theme have been proposed for Bluetooth pairing protocols. In an attempt to avoid using any additional out-of-band authentication factors, Davies and Price proposed the use of the interlock protocol of Ron Rivest and Adi Shamir, which has been subject to both attack and subsequent refinement.

  Read more →
• 

  Data deduplication

  In computing, data deduplication is a technique for eliminating duplicate copies of repeating data. Successful implementation of the technique can improve storage utilization, which may in turn lower capital expenditure by reducing the overall amount of storage media required to meet storage capacity needs. It can also be applied to network data transfers to reduce the number of bytes that must be sent. The deduplication process requires comparison of data 'chunks' (also known as 'byte patterns') which are unique, contiguous blocks of data. These chunks are identified and stored during a process of analysis, and compared to other chunks within existing data. Whenever a match occurs, the redundant chunk is replaced with a small reference that points to the stored chunk. Given that the same byte pattern may occur dozens, hundreds, or even thousands of times (the match frequency is dependent on the chunk size), the amount of data that must be stored or transferred can be greatly reduced. A related technique is single-instance (data) storage, which replaces multiple copies of content at the whole-file level with a single shared copy. While possible to combine this with other forms of data compression and deduplication, it is distinct from newer approaches to data deduplication (which can operate at the segment or sub-block level). Deduplication is different from data compression algorithms, such as LZ77 and LZ78. Whereas compression algorithms identify redundant data inside individual files and encodes this redundant data more efficiently, the intent of deduplication is to inspect large volumes of data and identify large sections – such as entire files or large sections of files – that are identical, and replace them with a shared copy. == Functioning principle == For example, a typical email system might contain 100 instances of the same 1 MB (megabyte) file attachment. Each time the email platform is backed up, all 100 instances of the attachment are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; the subsequent instances are referenced back to the saved copy for deduplication ratio of roughly 100 to 1. Deduplication is often paired with data compression for additional storage saving: Deduplication is first used to eliminate large chunks of repetitive data, and compression is then used to efficiently encode each of the stored chunks. In computer code, deduplication is done by, for example, storing information in variables so that they don't have to be written out individually but can be changed all at once at a central referenced location. Examples are CSS classes and named references in MediaWiki. == Benefits == Storage-based data deduplication reduces the amount of storage needed for a given set of files. It is most effective in applications where many copies of very similar or even identical data are stored on a single disk. In the case of data backups, which routinely are performed to protect against data loss, most data in a given backup remain unchanged from the previous backup. Common backup systems try to exploit this by omitting (or hard linking) files that haven't changed or storing differences between files. Neither approach captures all redundancies, however. Hard-linking does not help with large files that have only changed in small ways, such as an email database; differences only find redundancies in adjacent versions of a single file (consider a section that was deleted and later added in again, or a logo image included in many documents). In-line network data deduplication is used to reduce the number of bytes that must be transferred between endpoints, which can reduce the amount of bandwidth required. See WAN optimization for more information. Virtual servers and virtual desktops benefit from deduplication because it allows nominally separate system files for each virtual machine to be coalesced into a single storage space. At the same time, if a given virtual machine customizes a file, deduplication will not change the files on the other virtual machines—something that alternatives like hard links or shared disks do not offer. Backing up or making duplicate copies of virtual environments is similarly improved. == Classification == === Post-process versus in-line deduplication === Deduplication may occur "in-line", as data is flowing, or "post-process" after it has been written. With post-process deduplication, new data is first stored on the storage device and then a process at a later time will analyze the data looking for duplication. The benefit is that there is no need to wait for the hash calculations and lookup to be completed before storing the data, thereby ensuring that store performance is not degraded. Implementations offering policy-based operation can give users the ability to defer optimization on "active" files, or to process files based on type and location. One potential drawback is that duplicate data may be unnecessarily stored for a short time, which can be problematic if the system is nearing full capacity. Alternatively, deduplication hash calculations can be done in-line: synchronized as data enters the target device. If the storage system identifies a block which it has already stored, only a reference to the existing block is stored, rather than the whole new block. The advantage of in-line deduplication over post-process deduplication is that it requires less storage and network traffic, since duplicate data is never stored or transferred. On the negative side, hash calculations may be computationally expensive, thereby reducing the storage throughput. However, certain vendors with in-line deduplication have demonstrated equipment which performs in-line deduplication at high rates. Post-process and in-line deduplication methods are often heavily debated. === Data formats === The SNIA Dictionary identifies two methods: Content-agnostic data deduplication – a data deduplication method that does not require awareness of specific application data formats. Content-aware data deduplication – a data deduplication method that leverages knowledge of specific application data formats. === Source versus target deduplication === Another way to classify data deduplication methods is according to where they occur. Deduplication occurring close to where data is created, is referred to as "source deduplication". When it occurs near where the data is stored, it is called "target deduplication". Source deduplication ensures that data on the data source is deduplicated. This generally takes place directly within a file system. The file system will periodically scan new files creating hashes and compare them to hashes of existing files. When files with same hashes are found then the file copy is removed and the new file points to the old file. Unlike hard links however, duplicated files are considered to be separate entities and if one of the duplicated files is later modified, then using a system called copy-on-write a copy of that changed file or block is created. The deduplication process is transparent to the users and backup applications. Backing up a deduplicated file system will often cause duplication to occur resulting in the backups being bigger than the source data. Source deduplication can be declared explicitly for copying operations, as no calculation is needed to know that the copied data is in need of deduplication. This leads to a new form of link on file systems, called a reference-counted link, or reflink, in some systems (e.g. Linux), or a cloned file on macOS, where one or more inodes (file information entries) are made to share some or all of their data. It is named analogously to hard links, which work at the inode level, and symbolic links, which work at the filename level.The individual entries have a copy-on-write behavior that is non-aliasing, i.e. changing one copy afterwards will not affect other copies. Microsoft's ReFS also supports this operation. Target deduplication is the process of removing duplicates when the data was not generated at that location. Example of this would be a server connected to a SAN/NAS, The SAN/NAS would be a target for the server (target deduplication). The server is not aware of any deduplication, the server is also the point of data generation. A second example would be backup. Generally this will be a backup store such as a data repository or a virtual tape library. === Deduplication methods === One of the most common forms of data deduplication implementations works by comparing chunks of data to detect duplicates. For that to happen, each chunk of data is assigned an identification, calculated by the software, typically using cryptographic hash functions. In many implementations, the assumption is made that if the identification is identical, the data is identical, even though this cannot be true in all cases due to the pigeonhole principle; other implementations do not as

  Read more →