MovieRide FX is a patented automated special visual effects video compositing engine used in the MovieRide FX mobile application for Android (requires Android 2.3 or later) and iOS (compatible with iPhone 4 and up, iPad, and iPod Touch (new generation), requires iOS 7 or later). MovieRide FX allows the user to personalize a "Hollywood-style" movie clip by inserting themself into the clip as the "actor". == Features == The MovieRide FX app uses the relevant mobile device's camera to record a video of the user and insert it into a pre-packaged "Hollywood style" movie clip. The "actor" is extracted from their recorded video clip through various known effects such as masking, keying, and motion tracking. The "actor" is then inserted into one of the pre-packaged movie clips created by the MovieRide FX visual effects artists. This is done through an automated process requiring little or no artistic or technical skill from the user. The custom movie clips pre-packaged with MovieRide FX offer the user a variety of movie scenarios. Additional clips based on popular television and movie themes are continually being developed and are available on a freemium basis. == Sharing == Once the user's footage has automatically been composited into a movie clip and rendered as an .mp4 file, it can be shared via social media, such as Facebook, YouTube, and Twitter, and by e-mail. == History == === 2012 === MovieRide FX was created by Grant Waterston and Johann Mynhardt, who started development in 2012. === 2013 === The beta version was released on Google Play in July 2013. In August 2013 MovieRide FX was a New Media Award winner in the "New Media" category of the Accolade International Awards in Los Angeles. In October 2013 MovieRide FX was awarded exhibitor space in the ‘start-up village’ at the Apps-World Expo in London. === 2014 === MovieRide FX reached the 100 000 – 500 000 downloads category on the Google Play Store in June 2014. The official Android version was launched in July 2014. iOS version released in August 2014. MovieRide FX was selected as one of the "Top 150" startups at the Pioneer Festival in Vienna in September 2014. In November 2014 MovieRide FX was shortlisted for the Appster Awards in the "Best Entertainment App" and "Most Innovative App" categories and was awarded exhibitor space at the ‘start-up village’ at the Apps-World Expo in London. Patent applications were filed in South Africa, the EU and USA in April 2014. === 2015 === In September 2015 MovieRide FX was shortlisted for "Best Software innovation" at The Technology Expo Awards in London. === 2016 === In April 2016 MovieRide FX was nominated for a National Science and Technology Forum (NSTF) award for 'Research leading to Innovation by a corporate organization' In August 2016 Movie Ride FX won two Gold Awards at the 2016 Mobile Marketing Awards (MMA Smarties SA). These two Gold awards were for the 'Innovation' and 'Best in Show’ categories. In December 2016 FlicJam Inc. was formed in the US to access the larger global market. EU patent application was published in March 2016. === 2017 === South African patent was granted in February 2017. === 2018 === US patent was granted in March 2018.
Automation in construction
Automation in construction is the combination of methods, processes, and systems that allow for greater machine autonomy in construction activities. Construction automation may have multiple goals, including but not limited to, reducing jobsite injuries, decreasing activity completion times, and assisting with quality control and quality assurance. Some systems may be fielded as a direct response to increasing skilled labor shortages in some countries. Opponents claim that increased automation may lead to less construction jobs and that software leaves heavy equipment vulnerable to hackers. Research insights on this subject are today published in several journals such as Automation in Construction by Elsevier. == Uses of automation in construction == Equipment control and management: Automation can be used to control and monitor construction equipment, such as cranes, excavators, and bulldozers. Material handling: Automated systems can be used to handle, transport, and place materials such as concrete, bricks, and stones. Surveying: Automated survey equipment and drones can be used to collect and analyze data on construction sites. Quality control: Automated systems can be used to monitor and control the quality of materials and construction processes. Safety management: Automated systems can be used to monitor and control safety conditions on construction sites. Scheduling and planning: Automated systems can be used to manage schedules, resources, and costs. Waste management: Automated systems can be used to manage and dispose of waste materials generated during construction. 3D printing: Automated 3D printing can be used to create prototypes, models, and even full-scale building components. == Autonomous heavy equipment == Advances in sensors, machine learning, and autonomous vehicle technology have led to the development of self-operating construction equipment and retrofit systems designed to automate excavators, bulldozers, tracked loaders, skid steer loaders, and haul trucks, allowing them to perform tasks with limited human supervision. Since 2017, tech companies have developed autonomous or semi-autonomous retrofit kits that can be installed on existing construction machinery. Examples include Bedrock Robotics, Built Robotics, and SafeAI, which develop sensor and software systems that enable excavators and other earthmoving machines to operate with varying degrees of autonomy. Major equipment manufacturers have also introduced autonomous capabilities: Caterpillar and John Deere have developed autonomous or semi-autonomous systems for construction and mining equipment, including haul trucks and earthmoving machines. == Transportation сonstruction == Kratos Defense & Security Solutions fielded the world’s first Autonomous Truck-Mounted Attenuator (ATMA) in 2017, in conjunction with Royal Truck & Equipment. == Benefits of automation in construction == The use of automation in construction has become increasingly prevalent in recent years due to its numerous benefits. Automation in construction refers to the use of machinery, software, and other technologies to perform tasks that were previously done manually by workers. One of the most significant benefits of automation in construction is increased productivity. Automation can help speed up construction processes, reduce project completion times, and improve overall efficiency. For example, using automated machinery for tasks such as concrete pouring, bricklaying, and welding can significantly increase the speed and accuracy of these tasks, allowing for more work to be completed in a shorter amount of time. Another benefit of automation in construction is improved safety. By automating tasks that are hazardous to workers, such as demolition or working at height, companies can reduce the risk of accidents and injuries on site. Automation can also help to reduce worker fatigue, which can be a significant factor in accidents and mistakes. Overall, the use of automation in construction can improve productivity, reduce costs, increase safety, and improve the quality of construction projects. As technology continues to advance, the use of automation is likely to become even more prevalent in the construction industry.
Business continuity and disaster recovery auditing
Given organizations' increasing dependency on information technology (IT) to run their operations, business continuity planning (and its subset IT service continuity planning) covers the entire organization, while disaster recovery focuses on IT. Auditing documents covering an organization's business continuity and disaster recovery (BCDR) plans provides a third-party validation to stakeholders that the documentation is complete and does not contain material misrepresentations. == Overview == Often used together, the terms business continuity (BC) and disaster recovery (DR) are very different. BC refers to the ability of a business to continue critical functions and business processes after the occurrence of a disaster, whereas DR refers specifically to the IT functions of the business, albeit a subset of BC. == Metrics == The primary objective is to protect the organization in the event that all or part of its operations and/or computer services are rendered partially or completely unusable. === DR metrics === Minimizing downtime and data loss during disaster recovery is typically measured in terms of two key concepts: Recovery time objective (RTO), time until a system is completely up and running Recovery point objective (RPO), a measure of the ability to recover files by specifying a point in time the backup copy will restore to. == The auditor's role == Role of the Internal Auditor in Auditing a Disaster Recovery Plan (DRP): 1. Governance & Oversight - Confirm roles, responsibilities, and oversight are defined, and DRP aligns with risk appetite and continuity strategy. 2. Risk Assessment & BIA - Verify risk and impact assessments identify critical systems and define RTO/RPO. 3. Plan Design & Documentation - Ensure the DRP is current, complete, and includes key recovery procedures. 4. Testing & Validation - Confirm regular DRP testing occurs and results are used to improve the plan. 5. Backup & Recovery - Assess backup frequency and recovery capabilities against RTO/RPO targets. 6. Communication & Training - Verify staff are trained and communication protocols are in place for crises. 7. Maintenance & Improvement - Ensure the DRP is regularly updated and lessons learned are integrated. == Documentation == === Disaster recovery plan === A disaster recovery plan (DRP) is a documented process or set of procedures to execute an organization's disaster recovery processes and recover and protect a business IT infrastructure in the event of a disaster. It is "a comprehensive statement of consistent actions to be taken before, during and after a disaster". The disaster could be natural, environmental or man-made. Man-made disasters could be intentional (for example, an act of a terrorist) or unintentional (that is, accidental, such as the breakage of a man-made dam or even "fat fingers" - or errant commands entered - on a computer system). ==== Types of plans ==== Although there is no one-size-fits-all plan, there are three basic strategies: prevention, including proper backups, having surge protectors and generators detection, a byproduct of routine inspections, which may discover new (potential) threats correction The latter may include securing proper insurance policies, and holding a "lessons learned" brainstorming session. ==== Best practices ==== To maximize their effectiveness, DRPs are most effective when updated frequently, and should: be an integral part of all business analysis processes, be revisited at every major corporate acquisition, at every new product launch and at every new system development milestone. be thoroughly tested, not just unpracticed bureaucratic documentation Adequate records need to be retained by the organization. The auditor examines records, billings, and contracts to verify that records are being kept. One such record is a current list of the organization's hardware and software vendors. Such list is made and periodically updated to reflect changing business practices and as part of an IT asset management system. Copies of it are stored on and off site and are made available or accessible to those who require them. An auditor tests the procedures used to meet this objective and determine their effectiveness. === Relationship to BCPs === Disaster recovery is a subset of business continuity. Where DRP encompasses the policies, tools and procedures to enable recovery of data following a catastrophic event, BCP involves keeping all aspects of a business functioning regardless of potential disruptive events. As such, a business continuity plan is a comprehensive organizational strategy that includes the DRP as well as threat prevention, detection, recovery, and resumption of operations should a data breach or other disaster event occur. Therefore, BCP consists of five component plans: Business resumption plan Occupant emergency plan Continuity of operations plan Incident management plan Disaster recovery plan The first three components (business resumption, occupant emergency, and continuity of operations plans) do not deal with the IT infrastructure. The incident management plan (IMP) does deal with the IT infrastructure, but since it establishes structure and procedures to address cyber attacks against an organization's IT systems, it generally does not represent an agent for activating the DRP; thus DRP is the only BCP component of active interest to IT. == Testing == The overall categorization of tests are functional- and discussion-based. Types of tests include: tabletop exercises, checklists, simulations, parallel processing (testing recovery site while primary site is in operation), and full interruption (fail over) tests. These apply to both BC and DR. == Benefits == Like every insurance plan, there are benefits that can be obtained from proper business continuity planning, including: Studies have shown a correlation between higher spending on auditing fees and lower rates of Incidents. Minimizing risk of delays Guaranteeing the reliability of standby systems (even automating the failure detection and recovery in certain scenarios) Providing a standard for testing the plan Minimizing decision-making during a disaster Reducing potential legal liabilities Lowering unnecessarily stressful work environment === Planning and testing methodology === According to Geoffrey H. Wold of the Disaster Recovery Journal, the entire process involved in developing a Disaster Recovery Plan consists of 10 steps: Performing a risk assessment: The planning committee prepares a risk analysis and a business impact analysis (BIA) that includes a range of possible disasters. Each functional area of the organization is analyzed to determine potential consequences. Traditionally, fire has posed the greatest threat. A thorough plan provides for "worst case" situations, such as destruction of the main building. Establishing priorities for processing and operations: Critical needs of each department are evaluated and prioritized. Written agreements for alternatives selected are prepared, with details specifying duration, termination conditions, system testing, cost, any special security procedures, procedure for the notification of system changes, hours of operation, the specific hardware and other equipment required for processing, personnel requirements, definition of the circumstances constituting an emergency, process to negotiate service extensions, guarantee of compatibility, availability, non-mainframe resource requirements, priorities, and other contractual issues. Collecting data: This includes various lists (employee backup position listing, critical telephone numbers list, master call list, master vendor list, notification checklist), inventories (communications equipment, documentation, office equipment, forms, insurance policies, workgroup and data center computer hardware, microcomputer hardware and software, office supply, off-site storage location equipment, telephones, etc.), distribution register, software and data files backup/retention schedules, temporary location specifications, any other such lists, materials, inventories, and documentation. Pre-formatted forms are often used to facilitate the data gathering process. Organizing and documenting a written plan Developing testing criteria and procedures: reasons for testing include Determining the feasibility and compatibility of backup facilities and procedures. Identifying areas in the plan that need modification. Providing training to the team managers and team members. Demonstrating the ability of the organization to recover. Providing motivation for maintaining and updating the disaster recovery plan. Testing the plan: An initial "dry run" of the plan is performed by conducting a structured walk-through test. An actual test-run must be performed. Problems are corrected. Initial testing can be plan is done in sections and after normal business hours to minimize disruptions. Subsequent tests occur during normal business hours. === Caveats/controversie
Memory-hard function
In cryptography, a memory-hard function (MHF) is a function that costs a significant amount of memory to efficiently evaluate. It differs from a memory-bound function, which incurs cost by slowing down computation through memory latency. MHFs have found use in key stretching and proof of work as their increased memory requirements significantly reduce the computational efficiency advantage of custom hardware over general-purpose hardware compared to non-MHFs. == Introduction == MHFs are designed to consume large amounts of memory on a computer in order to reduce the effectiveness of parallel computing. In order to evaluate the function using less memory, a significant time penalty is incurred. As each MHF computation requires a large amount of memory, the number of function computations that can occur simultaneously is limited by the amount of available memory. This reduces the efficiency of specialised hardware, such as application-specific integrated circuits and graphics processing units, which utilise parallelisation, in computing a MHF for a large number of inputs, such as when brute-forcing password hashes or mining cryptocurrency. == Motivation and examples == Bitcoin's proof-of-work uses repeated evaluation of the SHA-256 function, but modern general-purpose processors, such as off-the-shelf CPUs, are inefficient when computing a fixed function many times over. Specialized hardware, such as application-specific integrated circuits (ASICs) designed for Bitcoin mining, can use 30,000 times less energy per hash than x86 CPUs whilst having much greater hash rates. This led to concerns about the centralization of mining for Bitcoin and other cryptocurrencies. Because of this inequality between miners using ASICs and miners using CPUs or off-the shelf hardware, designers of later proof-of-work systems utilised hash functions for which it was difficult to construct ASICs that could evaluate the hash function significantly faster than a CPU. As memory cost is platform-independent, MHFs have found use in cryptocurrency mining, such as for Litecoin, which uses scrypt as its hash function. They are also useful in password hashing because they significantly increase the cost of trying many possible passwords against a leaked database of hashed passwords without significantly increasing the computation time for legitimate users. == Measuring memory hardness == There are various ways to measure the memory hardness of a function. One commonly seen measure is cumulative memory complexity (CMC). In a parallel model, CMC is the sum of the memory required to compute a function over every time step of the computation. Other viable measures include integrating memory usage against time and measuring memory bandwidth consumption on a memory bus. Functions requiring high memory bandwidth are sometimes referred to as "bandwidth-hard functions". == Variants == MHFs can be categorized into two different groups based on their evaluation patterns: data-dependent memory-hard functions (dMHF) and data-independent memory-hard functions (iMHF). As opposed to iMHFs, the memory access pattern of a dMHF depends on the function input, such as the password provided to a key derivation function. Examples of dMHFs are scrypt and Argon2d, while examples of iMHFs are Argon2i and catena. Many of these MHFs have been designed to be used as password hashing functions because of their memory hardness. A notable problem with dMHFs is that they are prone to side-channel attacks such as cache timing. This has resulted in a preference for using iMHFs when hashing passwords. However, iMHFs have been mathematically proven to have weaker memory hardness properties than dMHFs.
Dashboard (computing)
In computer information systems, a dashboard is a type of graphical user interface which often provides at-a-glance views of data relevant to a particular objective or process through a combination of visualizations and summary information. In other usage, "dashboard" is another name for "progress report" or "report" and is considered a form of data visualization. The dashboard is often accessible by a web browser and is typically linked to regularly updating data sources. Dashboards are often interactive and facilitate users to explore the data themselves, usually by clicking into elements to view more detailed information. The term dashboard originates from the automobile dashboard where drivers monitor the major functions at a glance via the instrument panel. == History == The idea of digital dashboards followed the study of decision support systems in the 1970s. Early predecessors of the modern business dashboard were first developed in the 1980s in the form of Executive Information Systems (EISs). Due to problems primarily with data refreshing and handling, it was soon realized that the approach wasn't practical as information was often incomplete, unreliable, and spread across too many disparate sources. Thus, EISs hibernated until the 1990s when the information age quickened pace and data warehousing, and online analytical processing (OLAP) allowed dashboards to function adequately. Despite the availability of enabling technologies, the dashboard use didn't become popular until later in that decade, with the rise of key performance indicators (KPIs), and the introduction of Robert S. Kaplan and David P. Norton's balanced scorecard. In the late 1990s, Microsoft promoted a concept known as the Digital Nervous System and "digital dashboards" were described as being one leg of that concept. Today, the use of dashboards forms an important part of Business Performance Management (BPM). Initially dashboards were used for monitoring purposes, now with the advancement of technology, dashboards are being used for more analytical purposes. The use of dashboards has now been incorporating; scenario analysis, drill down capabilities, and presentation format flexibility. == Benefits == Digital dashboards allow managers to monitor the contribution of the various departments in their organization. In addition, they enable “rolling up” of information to present a consolidated view across an organization. To gauge exactly how well an organization is performing overall, digital dashboards allow you to capture and report specific data points from each department within the organization, thus providing a "snapshot" of performance. Benefits of using digital dashboards include: Visual presentation of performance measures Ability to identify and correct negative trends Measure efficiencies/inefficiencies Ability to generate detailed reports showing new trends Ability to make more informed decisions based on collected business intelligence Dashboards offers a holistic view of the entire business as it gives the manager a bird's eye view into the performance of sales, data inventory, web traffic, social media analytics and other associated data that is visually presented on a single dashboard. Dashboards lead to better management of marketing/financial strategies as a dashboard for the display of marketing data makes the process of marketing easier and more reliable as compared to doing it manually. Web analytics play a crucial role in shaping the marketing strategy of many businesses. Dashboards also facilitate for better tracking of sales and financial reporting as the data is more precise and in one area. Lastly, dashboards offer for better customer service through monitoring because they keep both the managers and the clients updated on the project progress through automated emails and notifications. == Align strategies and organizational goals == Gain total visibility of all systems instantly Quick identification of data outliers and correlations Consolidated reporting into one location Available on mobile devices to quickly access metrics == Classification == Dashboards can be broken down according to role and are either strategic, analytical, operational, or informational. Dashboards are the 3rd step on the information ladder, demonstrating the conversion of data to increasingly valuable insights. Strategic dashboards support managers at any level in an organization and provide the quick overview that decision-makers need to monitor the health and opportunities of the business. Dashboards of this type focus on high-level measures of performance and forecasts. Strategic dashboards benefit from static snapshots of data (daily, weekly, monthly, and quarterly) that are not constantly changing from one moment to the next. Dashboards for analytical purposes often include more context, comparisons, and history, along with subtler performance evaluators. In addition, analytical dashboards typically support interactions with the data, such as drilling down into the underlying details. Dashboards for monitoring operations are often designed differently from those that support strategic decision making or data analysis and often require monitoring of activities and events that are constantly changing and might require attention and response at a moment's notice. == Types of dashboards == Digital dashboards may be laid out to track the flows inherent in the business processes that they monitor. Graphically, users may see the high-level processes and then drill down into low-level data. This level of detail is often buried deep within the corporate enterprise and otherwise unavailable to the senior executives. Three main types of digital dashboards dominate the market today: desktop software applications, web-browser-based applications, and desktop applications are also known as desktop widgets. The last are driven by a widget engine. Both Desktop and Browser-based providers enable the distribution of dashboards via a web browser. An example of the latter is web-based-browser Asana, which helps teams orchestrate their work, from daily tasks to strategic cross-functional initiatives. With it, teams can manage everything from company objectives to digital transformation to product launches and marketing campaigns. Specialized dashboards may track all corporate functions. Examples include human resources, recruiting, sales, operations, security, information technology, project management, customer relationship management, digital marketing and many more departmental dashboards. For a smaller organization like a startup a compact startup scorecard dashboard tracks important activities across lot of domains ranging from social media to sales. Digital dashboard projects involve business units as the driver and the information technology department as the enabler. Therefore, the success of dashboard projects depends on the relevancy/importance of information provided within the dashboard. This includes the metrics chosen to monitor and the timeliness of the data forming those metrics; data must be up to date and accurate. Key performance indicators, balanced scorecards, and sales performance figures are some of the content appropriate on business dashboards. === Performance Dashboards === Dashboards involve the combination of visual and functional features. This combination of features helps improve cognition and interpretation. A performance dashboard sits at the intersection of two powerful disciplines: business intelligence and performance management. Therefore, there are different users who could use these dashboards for different reasons. For example, a level of workers could look at monitoring inventory while those in more managerial roles can look at lagging measure. Then executives could utilize the dashboard to evaluate strategic performance against objectives. == Dashboards and scorecards == Balanced scorecards and dashboards have been linked together as if they were interchangeable. However, although both visually display critical information, the difference is in the format: Scorecards can open the quality of an operation while dashboards provide calculated direction. A balanced scorecard has what they called a "prescriptive" format. It should always contain these components: Perspectives – group Objectives – verb-noun phrases pulled from a strategy plan Measures – also called metric or key performance indicators (KPIs) Spotlight indicators – red, yellow, or green symbols that provide an at-a-glance view of a measure's performance. Each of these sections ensures that a Balanced Scorecard is essentially connected to the businesses critical strategic needs. The design of a dashboard is more loosely defined. Dashboards are usually a series of graphics, charts, gauges and other visual indicators that can be monitored and interpreted. Even when there is a strategic link, on a dashboard, it may not be noticed as such since objectives are not normally pre
TSheets
TSheets was a web-based and mobile time tracking and employee scheduling app. The service was accessed via a web browser or a mobile app. TSheets was an alternative to a paper timesheet or punch cards. == History == Based in Eagle, Idaho, TSheets was co-founded in 2006 by CEO Matt Rissell and CTO Brandon Zehm. In 2008, TSheets released a native employee time tracking app for the iPhone. In 2012, TSheets released an integration with accounting and payroll software QuickBooks. In 2015, TSheets accepted $15 million in growth equity funding from Summit Partners, bought a building in Eagle, Idaho, and opened a second location in Sydney, Australia. On 5 December 2017, Intuit announced an agreement to acquire TSheets. The transaction was valued at approximately $340 million of cash and other consideration and closed on 11 January 2018. After the transaction closed, Time Capture became a new business unit within Intuit's Small Business and Self-Employed Group with Matt Rissell assuming the leader role reporting to Alex Chriss. TSheets's Eagle, Idaho site became an Intuit location.
Data deduplication
In computing, data deduplication is a technique for eliminating duplicate copies of repeating data. Successful implementation of the technique can improve storage utilization, which may in turn lower capital expenditure by reducing the overall amount of storage media required to meet storage capacity needs. It can also be applied to network data transfers to reduce the number of bytes that must be sent. The deduplication process requires comparison of data 'chunks' (also known as 'byte patterns') which are unique, contiguous blocks of data. These chunks are identified and stored during a process of analysis, and compared to other chunks within existing data. Whenever a match occurs, the redundant chunk is replaced with a small reference that points to the stored chunk. Given that the same byte pattern may occur dozens, hundreds, or even thousands of times (the match frequency is dependent on the chunk size), the amount of data that must be stored or transferred can be greatly reduced. A related technique is single-instance (data) storage, which replaces multiple copies of content at the whole-file level with a single shared copy. While possible to combine this with other forms of data compression and deduplication, it is distinct from newer approaches to data deduplication (which can operate at the segment or sub-block level). Deduplication is different from data compression algorithms, such as LZ77 and LZ78. Whereas compression algorithms identify redundant data inside individual files and encodes this redundant data more efficiently, the intent of deduplication is to inspect large volumes of data and identify large sections – such as entire files or large sections of files – that are identical, and replace them with a shared copy. == Functioning principle == For example, a typical email system might contain 100 instances of the same 1 MB (megabyte) file attachment. Each time the email platform is backed up, all 100 instances of the attachment are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; the subsequent instances are referenced back to the saved copy for deduplication ratio of roughly 100 to 1. Deduplication is often paired with data compression for additional storage saving: Deduplication is first used to eliminate large chunks of repetitive data, and compression is then used to efficiently encode each of the stored chunks. In computer code, deduplication is done by, for example, storing information in variables so that they don't have to be written out individually but can be changed all at once at a central referenced location. Examples are CSS classes and named references in MediaWiki. == Benefits == Storage-based data deduplication reduces the amount of storage needed for a given set of files. It is most effective in applications where many copies of very similar or even identical data are stored on a single disk. In the case of data backups, which routinely are performed to protect against data loss, most data in a given backup remain unchanged from the previous backup. Common backup systems try to exploit this by omitting (or hard linking) files that haven't changed or storing differences between files. Neither approach captures all redundancies, however. Hard-linking does not help with large files that have only changed in small ways, such as an email database; differences only find redundancies in adjacent versions of a single file (consider a section that was deleted and later added in again, or a logo image included in many documents). In-line network data deduplication is used to reduce the number of bytes that must be transferred between endpoints, which can reduce the amount of bandwidth required. See WAN optimization for more information. Virtual servers and virtual desktops benefit from deduplication because it allows nominally separate system files for each virtual machine to be coalesced into a single storage space. At the same time, if a given virtual machine customizes a file, deduplication will not change the files on the other virtual machines—something that alternatives like hard links or shared disks do not offer. Backing up or making duplicate copies of virtual environments is similarly improved. == Classification == === Post-process versus in-line deduplication === Deduplication may occur "in-line", as data is flowing, or "post-process" after it has been written. With post-process deduplication, new data is first stored on the storage device and then a process at a later time will analyze the data looking for duplication. The benefit is that there is no need to wait for the hash calculations and lookup to be completed before storing the data, thereby ensuring that store performance is not degraded. Implementations offering policy-based operation can give users the ability to defer optimization on "active" files, or to process files based on type and location. One potential drawback is that duplicate data may be unnecessarily stored for a short time, which can be problematic if the system is nearing full capacity. Alternatively, deduplication hash calculations can be done in-line: synchronized as data enters the target device. If the storage system identifies a block which it has already stored, only a reference to the existing block is stored, rather than the whole new block. The advantage of in-line deduplication over post-process deduplication is that it requires less storage and network traffic, since duplicate data is never stored or transferred. On the negative side, hash calculations may be computationally expensive, thereby reducing the storage throughput. However, certain vendors with in-line deduplication have demonstrated equipment which performs in-line deduplication at high rates. Post-process and in-line deduplication methods are often heavily debated. === Data formats === The SNIA Dictionary identifies two methods: Content-agnostic data deduplication – a data deduplication method that does not require awareness of specific application data formats. Content-aware data deduplication – a data deduplication method that leverages knowledge of specific application data formats. === Source versus target deduplication === Another way to classify data deduplication methods is according to where they occur. Deduplication occurring close to where data is created, is referred to as "source deduplication". When it occurs near where the data is stored, it is called "target deduplication". Source deduplication ensures that data on the data source is deduplicated. This generally takes place directly within a file system. The file system will periodically scan new files creating hashes and compare them to hashes of existing files. When files with same hashes are found then the file copy is removed and the new file points to the old file. Unlike hard links however, duplicated files are considered to be separate entities and if one of the duplicated files is later modified, then using a system called copy-on-write a copy of that changed file or block is created. The deduplication process is transparent to the users and backup applications. Backing up a deduplicated file system will often cause duplication to occur resulting in the backups being bigger than the source data. Source deduplication can be declared explicitly for copying operations, as no calculation is needed to know that the copied data is in need of deduplication. This leads to a new form of link on file systems, called a reference-counted link, or reflink, in some systems (e.g. Linux), or a cloned file on macOS, where one or more inodes (file information entries) are made to share some or all of their data. It is named analogously to hard links, which work at the inode level, and symbolic links, which work at the filename level.The individual entries have a copy-on-write behavior that is non-aliasing, i.e. changing one copy afterwards will not affect other copies. Microsoft's ReFS also supports this operation. Target deduplication is the process of removing duplicates when the data was not generated at that location. Example of this would be a server connected to a SAN/NAS, The SAN/NAS would be a target for the server (target deduplication). The server is not aware of any deduplication, the server is also the point of data generation. A second example would be backup. Generally this will be a backup store such as a data repository or a virtual tape library. === Deduplication methods === One of the most common forms of data deduplication implementations works by comparing chunks of data to detect duplicates. For that to happen, each chunk of data is assigned an identification, calculated by the software, typically using cryptographic hash functions. In many implementations, the assumption is made that if the identification is identical, the data is identical, even though this cannot be true in all cases due to the pigeonhole principle; other implementations do not as