Effective accelerationism (e/acc) is a 21st-century ideological movement that advocates for an explicitly pro-technology stance. Its proponents believe that unrestricted technological progress, especially driven by artificial intelligence, is a solution to universal human problems, such as poverty, war, and climate change. They perceive themselves as a counterweight to more cautious views on technological innovation and often label their opponents derogatorily as "doomers" or "decels" (short for decelerationists). The movement carries utopian undertones and advocates for faster AI progress to ensure human survival and propagate consciousness throughout the universe. Although effective accelerationism has been described as a fringe movement and as cult-like, it has gained mainstream visibility in 2023. A number of high-profile Silicon Valley figures, including investors Marc Andreessen and Garry Tan, explicitly endorsed it by adding "e/acc" to their public social media profiles. == Etymology and central beliefs == Effective accelerationism, a portmanteau of "effective altruism" and "accelerationism", is a fundamentally techno-optimist movement. According to Guillaume Verdon, one of the movement's founders, its aim is for human civilization to "clim[b] the Kardashev gradient", meaning its purpose is for human civilization to rise to next levels on the Kardashev scale by maximizing energy usage. To achieve this goal, effective accelerationism wants to accelerate technological progress. It is strongly focused on artificial general intelligence (AGI), because it sees AGI as fundamental for climbing the Kardashev scale. The movement therefore advocates for unrestricted development and deployment of artificial intelligence. Regulation of artificial intelligence and government intervention in markets more generally is met with opposition. Many of its proponents have libertarian views and think that AGI will be most aligned if many AGIs compete against each other on the marketplace. The founders of the movement see it as rooted in Jeremy England's theory on the origin of life, which is focused on entropy and thermodynamics. According to them, the universe aims to increase entropy, and life is a way of increasing it. By spreading life throughout the universe and making life use up ever increasing amounts of energy, the universe's purpose would thus be fulfilled. == History == === Intellectual origins === While Nick Land is seen as the intellectual originator of contemporary accelerationism in general, the precise origins of effective accelerationism remain unclear. The earliest known reference to the movement can be traced back to a May 2022 newsletter published by four pseudonymous authors known by their X (formerly Twitter) usernames @BasedBeffJezos, @bayeslord, @zestular and @creatine_cycle. Effective accelerationism is an extension of the TESCREAL movement, being etymologically derived from Effective Altruism and heavily rooted in the older Silicon Valley subcultures of transhumanism and extropianism (which similarly emphasized the value of progress and resisted efforts to restrain the development of technology), alongside elements of singularitarianism, cosmism, and longtermism. It is also often considered to have emerged at least in part from the work of the Cybernetic Culture Research Unit (of which Nick Land was a leading member, alongside writers such as Mark Fisher and Sadie Plant). It is sometimes compared and contrasted with the work of philosopher Benjamin Bratton on planetary computation. === Disclosure of the identity of BasedBeffJezos === Forbes disclosed in December 2023 that the @BasedBeffJezos persona is maintained by Guillaume Verdon, a Canadian former Google quantum computing engineer and theoretical physicist. The revelation was supported by a voice analysis conducted by the National Center for Media Forensics of the University of Colorado Denver, which further confirmed the match between Jezos and Verdon. The magazine justified its decision to disclose Verdon's identity on the grounds of it being "in the public interest". On 29 December 2023 Guillaume Verdon was interviewed by Lex Fridman on the Lex Fridman Podcast and introduced as the "creator of the effective accelerationism movement". === Second Trump presidency === Following Donald Trump's victory in the 2024 U.S. presidential election, several prominent tech industry figures expressed support for positions aligned with effective accelerationism, particularly regarding deregulation and technological advancement. The potential appointment of Elon Musk to government roles focused on auditing federal programs drew support from venture capitalists who anticipated reduced regulatory oversight of the technology sector. Notable tech figures publicly connected these developments to the movement's principles. Aaron Levie, CEO of Box, expressed support for "removing unnecessary red tape and over-regulation", while Mark Pincus, early Facebook investor and Zynga founder, explicitly referenced "effective accelerationism" in his post-election commentary. Venture capitalists viewed the incoming administration as an opportunity to ease regulations that had affected technology mergers and acquisitions during the previous years. == Relation to other movements == === Traditional accelerationism === Traditional accelerationism, as developed by the British philosopher Nick Land, sees the acceleration of technological change as a way to bring about a fundamental transformation of current culture, society, and the political economy. This is done through capitalism, which Land views as "an autonomous force that’s reconfiguring society" that can overcome its limits if intensified. Land's work has also been characterized as concerning "the supposedly inevitable 'disintegration of the human species' when artificial intelligence improves sufficiently." While both concern ideas like a technocapital singularity and AGI progress, effective accelerationism focuses on using AGI for the greatest ethical good for conscious life and civilization (whether human or machine), as well as expanding civilization and maximizing energy usage in order to align with the "will of the universe". Land focuses on capitalist self-optimization as the driver of modernity, progress, and the eroding of existing social orders. Land has expressed support for effective accelerationism, while Thomas Murphy referred to the movement as "Nick Land diluted for LinkedIn". === Effective altruism === Effective accelerationism diverges from the principles of effective altruism, which prioritizes using evidence and reasoning to identify the most effective ways to altruistically improve the world. This divergence comes primarily from one of the causes effective altruists focus on – AI existential risk. Effective altruists (particularly longtermists) argue that AI companies should be cautious and strive to develop safe AI systems, as they fear that any misaligned AGI could eventually lead to human extinction. Proponents of effective accelerationism generally consider existential risks from AGI to be negligible, and claim that even if they were not, decentralized free markets would much better mitigate this risk than centralized governmental regulation. === Degrowth === Effective accelerationism stands in stark contrast with the degrowth movement, sometimes described by it as "decelerationism" or "decels". The degrowth movement advocates for reducing economic activity and consumption to address ecological and social issues. Effective accelerationism on the contrary embraces technological progress, energy consumption and the dynamics of capitalism, rather than advocating for a reduction in economic activity. == Reception == The "Techno-Optimist Manifesto", a 2023 essay by Marc Andreessen, has been described by the Financial Times and the German Süddeutsche Zeitung as espousing the views of effective accelerationism. Mother Jones also characterized it as expressing effective accelerationism and reported that Andressen cited Land's work. David Swan of The Sydney Morning Herald has criticized effective accelerationism due to its opposition to government and industry self-regulation. He argues that "innovations like AI needs thoughtful regulations and guardrails ... to avoid the myriad mistakes Silicon Valley has already made." During the 2023 Reagan National Defense Forum, U.S. Secretary of Commerce Gina Raimondo cautioned against embracing the "move fast and break things" mentality associated with "effective acceleration [sic]". She emphasized the need to exercise caution in dealing with AI, stating "that's too dangerous. You can't break things when you are talking about AI." In a similar vein, Ellen Huet argued on Bloomberg News that some of the ideas of the movement were "deeply unsettling", focusing especially on Guillaume Verdon's "post-humanism" and the view that "natural selection could lead AI to replace us as the dominant spe
Data-driven model
Data-driven models are a class of computational models that primarily rely on historical data collected throughout a system's or process' lifetime to establish relationships between input, internal, and output variables. Commonly found in numerous articles and publications, data-driven models have evolved from earlier statistical models, overcoming limitations posed by strict assumptions about probability distributions. These models have gained prominence across various fields, particularly in the era of big data, artificial intelligence, and machine learning, where they offer valuable insights and predictions based on the available data. == Background == These models have evolved from earlier statistical models, which were based on certain assumptions about probability distributions that often proved to be overly restrictive. The emergence of data-driven models in the 1950s and 1960s coincided with the development of digital computers, advancements in artificial intelligence research, and the introduction of new approaches in non-behavioural modelling, such as pattern recognition and automatic classification. == Key Concepts == Data-driven models encompass a wide range of techniques and methodologies that aim to intelligently process and analyse large datasets. Examples include fuzzy logic, fuzzy and rough sets for handling uncertainty, neural networks for approximating functions, global optimization and evolutionary computing, statistical learning theory, and Bayesian methods. These models have found applications in various fields, including economics, customer relations management, financial services, medicine, and the military, among others. Machine learning, a subfield of artificial intelligence, is closely related to data-driven modelling as it also focuses on using historical data to create models that can make predictions and identify patterns. In fact, many data-driven models incorporate machine learning techniques, such as regression, classification, and clustering algorithms, to process and analyse data. In recent years, the concept of data-driven models has gained considerable attention in the field of water resources, with numerous applications, academic courses, and scientific publications using the term as a generalization for models that rely on data rather than physics. This classification has been featured in various publications and has even spurred the development of hybrid models in the past decade. Hybrid models attempt to quantify the degree of physically based information used in hydrological models and determine whether the process of building the model is primarily driven by physics or purely data-based. As a result, data-driven models have become an essential topic of discussion and exploration within water resources management and research. The term "data-driven modelling" (DDM) refers to the overarching paradigm of using historical data in conjunction with advanced computational techniques, including machine learning and artificial intelligence, to create models that can reveal underlying trends, patterns, and, in some cases, make predictions Data-driven models can be built with or without detailed knowledge of the underlying processes governing the system behavior, which makes them particularly useful when such knowledge is missing or fragmented.
SIGINT Activity Designator
A SIGINT Activity Designator (or SIGAD) identifies a signals intelligence (SIGINT) line of collection activity associated with a signals collection station, such as a base or a ship. For example, the SIGAD for Menwith Hill in the UK is USD1000. SIGADs are used by the signals intelligence agencies of Australia, Canada, New Zealand, the United Kingdom, and the United States (the Five Eyes). There are several thousand SIGADs including the substation SIGADs denoted with a trailing alpha character. Several dozen of these are significant. The leaked Boundless Informant reporting screenshot showed that it summarized 504 active SIGADs during a 30-day period in March 2013. == General format == A SIGAD consists of five to eight case insensitive alphanumeric characters. It takes the general form of an alphanumeric designator normally composed of a two- or three-letter prefix followed by one to three numbers. Often a dash is used to separate the alphabetic and numeric characters in the primary part of the designator, but less frequently a space is used as a separator or the alphabetic and numeric characters are concatenated together. An additional alphabetic character can be added to denote a sub-designator for a subset of the primary unit, such as a detachment. Lastly, a numeric character can be added after the aforementioned alphabetic to provide for a sub-sub-designator. In the examples below an X represents an alphabetic character and an N represents a numeric character that are part of the primary designator. Likewise, an x represents an alphabetic character and an n represents a numeric character that are part of a sub-designator. Here are valid generalized examples of SIGADs: The first two characters show which country operates the particular SIGINT facility, which can be US for the United States, UK for the United Kingdom, CA for Canada, AU for Australia and NZ for New Zealand. A third letter shows what sort of staff runs the station. SIGADs beginning with US without a third letter are used for intercept facilities run by the NSA. == PRISM SIGAD == One prominent SIGAD as of April 2013 is US-984XN, with an unclassified codename of PRISM. It is "the number one source of raw intelligence used for NSA analytic reports" according to National Security Agency sources in a document leaked by Edward Snowden. The President's Daily Brief, an all-source intelligence product, cited SIGAD US-984XN as a source in 1,477 items in 2012. The U.S. government operates the PRISM electronic surveillance collection program through NSA's Special Source Operations, an alliance with trusted telecommunications providers. == SIGADs for spy ships == The declassified SIGAD for the USS Liberty (AGTR-5) was USN-855. The USS Liberty incident occurred on 8 June 1967, during the Six-Day War, when Israeli Air Force jet fighter aircraft and Israeli Navy motor torpedo boats attacked the USS Liberty in international waters. The USS Pueblo (AGER-2) was a technical research ship, which was boarded and captured by North Korean forces on 23 January 1968, in what is known as the Pueblo incident. The declassified SIGAD for the NSA Direct Support Unit (DSU) from the Naval Security Group (NSG) on the USS Pueblo patrol involved in the incident was USN-467Y. The USS Pueblo, which officially remains a commissioned vessel of the United States Navy, is the only ship of the U.S. Navy currently being held captive. == Vietnam War SIGADs == The following are the Vietnam War-era declassified SIGADs from inside South Vietnam during the period of 1969 to 1975: Some locations have multiple SIGADs due to different types of collection activities and/or collection at different times during the period. The SIGADs beginning with USA were operated by the United States Air Force's United States Air Force Security Service (USAFSS). The SIGADs beginning with USM were operated by the United States Army's Army Security Agency (ASA). Lastly, the SIGADs beginning with USN were operated by the United States Navy's Naval Security Group (NAVSECGRU). All three of these units have been merged into other units or inactivated. The above list consists of the higher-echelon SIGADs. It does not include the numerous miscellaneous and temporary detachments, or direction finding stations belonging to major units or sites unless that detachment or site was the only one stationed in South Vietnam. Many of the "dets" were short-lived, often formed to support ongoing MACV operations or forward deployments of combat operational or maneuver units. These detachments usually were designated by a letter suffix attached to the higher-echelon SIGAD such as "USM-633J," which was a detachment of the 372d Radio Research Company, USM-633, supporting the United States Army's 25th Infantry Division. === Supporting Southeast Asia SIGADs === The following declassified SIGADs were highly relevant to the Vietnam Campaign, but were located in areas outside of South Vietnam in Southeast Asia. Again, detachments are not listed separately. In the case of the USS Maddox, naval Direct Support Units (DSUs) used the SIGAD USN-467 as a generic designator for their missions. Each specific patrol received a letter suffix for its duration. The subsequent mission would receive the next letter in an alphabetic sequence. Thus, SIGAD USN-467N specifically designates the USS Maddox patrol involved with the Gulf of Tonkin incident. == Joint Base SIGADs == In November 2005, the US Congress performed a fifth round of Base Realignment and Closure. This 2005 law also created twelve joint bases by merging adjacent installations belonging to different services in an effort to reduce costs and improve efficiencies. Joint bases with a primarily SIGINT mission have SIGADs that begin with USJ. A joint base would have a primary SIGAD in the general form of USJ-NNN, where NNN are numeric characters. An actual example is not given, since these units are currently active.
Peñabot
Peñabot is the nickname for automated social media accounts allegedly used by the Mexican government of Enrique Peña Nieto and the PRI political party to keep unfavorable news from reaching the Mexican public. Peñabot accusations are related to the broader issue of fake news in the 21st century. == History of disinformation in Mexican politics == The PRI political party has been reported to use fake news since before Peña Nieto. The main tactic originally was to spread such propaganda through open radio and television networks. Such tactic was effective in Mexico, because newspaper readership is low and cable TV is largely limited to the middle classes; consequently, the country's two major television networks – Televisa and TV Azteca – exert a significant influence in national politics. Televisa itself, not only owns around two-thirds of the programming on Mexico's TV channels, making it not only Mexico's largest television network, but also is the largest media network in the Spanish-speaking world. == Peñabots == Analysts have given the name Peñabots to a suspected network of automated accounts on social media used by the Mexican government to spread pro-government propaganda and to marginalize dissenting opinions in social media. The bots were first noticed in the 2012 elections when they were used to disseminate opinions in support of Enrique Peña Nieto on social networks such as Twitter and Facebook. According to Aristegui Noticias, their usage went against articles 6 and 134 of the Mexican Constitution. Those used by Peña Nieto's government cost an estimated 80 million pesos monthly, which news outlets argued only helped the government spread fake support towards the president, but did not have a benefit towards Mexican people (with whom EPN was highly unpopular). Facebook held approximately 640,321 Peñabots, while Twitter had less. As of July 2017, Oxford Internet Institute's Computational Propaganda Research Project claimed many western democracies, Mexico included, perform social media manipulation, thus saying the manipulation comes directly from the Mexican government itself. During Peña Nieto's subsequent presidency, analysts noted that Peñabots were used to overpower trending topics that critiqued government, to flood trending government critical hashtags with spam, to create fake trends by pushing alternative hashtags, and to push smear campaigns and threats against government-critical activists and journalists. Peñabots were distinguished as their pattern of activity was distinct from that of ordinary interaction on social networks. === Meadebots === On Twitter it was reported that about 94% of the followers of 2018 presidential candidate from the PRI Jose Antonio Meade were bots. When Antonio Meade presented himself as a candidate for the 2018 presidential election, his social media accounts such as "@MovimientoMEADE" (created by the PRI's official account @PRI_Nacional), obtained a huge quantity of followers in a short span of time. Some users noticed and brought it to attention, and after investigation it was reported 94% of such followers were bots (702,000 out of 747,000), and the account was eliminated from Twitter after 20 hours. The fake accounts used the hashtags #YoConMeade and #Meade18. It was further revealed was that Meade's official account on Twitter, @JoseAMeadeK had 25% bots (216,000 fake followers out of the 981,000). == Manipulation of news media in Mexico, through television == The Mexican government of Peña Nieto has been accused of using various means to keep unfavorable news from reaching the Mexican people. Many Mexicans have protested this practice as it clearly goes against the freedom of speech. The PRI has been reported to use fake news since before Peña Nieto. The main tactic has been to spread such propaganda through radio and television. This tactic is perceived as effective in Mexico, because newspaper readership is low and research on the Internet and cable TV is largely limited to the middle classes; consequently, the country's two major television networks – Televisa and TV Azteca – exert a significant influence in national politics. Televisa itself, owns around two-thirds of the programming on Mexico's TV channels, making it not only Mexico's largest television network, but also is the largest media network in the Spanish-speaking world. In June 2012, before the 2012 Mexican presidential elections, the British newspaper The Guardian published a series of allegations claiming Televisa, sold favorable coverage to top politicians in its news and entertainment shows, this scandal became known as the Televisa controversy. The documents published by 'The Guardian alleged that a secretive circle within Televisa manipulated news coverage to favor PRI presidential candidate Enrique Peña Nieto, who was poised as favorite to win. Televisa's secret circle supposedly commissioned videos to promote Peña Nieto and lash out his political rivals in 2009. The Guardian documents suggest that Televisa's secret team distributed such videos through e-mail, posting them posted them on Facebook and YouTube, some can still be seen there. Another document was a PowerPoint presentation, with a slide explicitly aimed at rival leftist candidate of the Party of the Democratic Revolution (PRD), Andrés Manuel López Obrador. Supposedly given to The Guardian by a Televisa employee. The document's authenticity was never possible to confirm– however dates, names, and events largely coincide. Televisa refused to talk the documents, and denied a relationship with the PRI or its presidential candidate, saying that they had provided equal media coverage to all parties. Televisa published an article supposedly showing discrepancies in The Guardian documents and denying accusations. Mexican citizens complained about the perceived favoritism towards Enrique Peña Nieto and the PRI, protesting through the Yo Soy 132 movement which Televisa covered in detail. However, Televisa's news media coverage is perceived to have been biased, by using a media coverage tactic Mexican citizens call cortinas de humo (smoke screens). These introduce a news scandal giving extensive coverage to distract citizens from a potential conflict-of-interest or controversy that could damage the image of the politician favored by the network. An example of a perceived smoke screen would be the news media coverage of "Caso Michoacán" and "Caso Paolette" distracting all the attention from the parallel "Yo soy 132" movement. A few years later, on the day of September 11, 2016; factual evidence of Televisa's performing media manipulation emerged, when a Televisa news anchor while live-on air reading a teleprompter, mistakenly read out loud that "try that Jaime "Ël Bronco" Rodríguez Calderón (Nuevo Leon's governor) is mentioned as little as possible". Newspaper El Universal caught it on video and published it social media. Televisa didn't mention the story and declined to comment. Lack of news coverage concerning Nuevo León's Governor Jaime Rodriguez, is perceived due to him being the first elected governor to not be part of any political party (Independent Governor), and because unlike the governors from the PRI preceding him, the independent governor "El Bronco" doesn't spend money on publicity at all, preferring to communicate all news by using social media such as Twitter and Facebook. While the incident may have proven Televisa's bias, there wasn't anything to incriminate the PRI political party or Enrique Peña Nieto, though it did further suspicion of Televisa manipulating news media. In contrast, a December 2017 article of The New York Times, reported Enrique Peña Nieto spending about 2000 million dollars on publicity, during his first 5 years as president, the largest publicity budget ever spent by a Mexican President. Additionally, 68 percent of news journalists admitted to not believe to have enough freedom of speech, and award-winning news reporter Carmen Aristegui was controversially fired shortly after revealing the Mexican White House scandals. == Violence and spying towards news journalists and civil rights activists == Far for only being receiving accusations of spreading fake news, the Mexican government of EPN (Enrique Peña Nieto) has also been accused of violence towards news journalists, and of spying on them, and also towards civil right leaders and their families. During his tenure as president, Peña Nieto has been accused of failing to protect news journalists, whose deaths are speculated to be politically triggered, by politicians attempting to prevent them from covering political scandals. The New York Times published a news report on the matter titled, "In Mexico it's easy to kill a journalist", on it mentioning how during EPN's government, Mexico became one of the worst countries on which to be a journalist. The assassination of journalist Javier Valdez on May 23, 2017, received national coverage, with multiple news journalists
Cypherpunks (book)
Cypherpunks: Freedom and the Future of the Internet is a 2012 book by Julian Assange, in discussion with Internet activists and cypherpunks Jacob Appelbaum, Andy Müller-Maguhn and Jérémie Zimmermann. Its primary topic is society's relationship with information security. In the book, the authors warn that the Internet has become a tool of the police state, and that the world is inadvertently heading toward a form of totalitarianism. They promote the use of cryptography to protect against state surveillance. In the introduction, Assange says that the book is "not a manifesto [...] [but] a warning". He told Guardian journalist Decca Aitkenhead: A well-defined mathematical algorithm can encrypt something quickly, but to decrypt it would take billions of years – or trillions of dollars' worth of electricity to drive the computer. So cryptography is the essential building block of independence for organisations on the Internet, just like armies are the essential building blocks of states, because otherwise one state just takes over another. There is no other way for our intellectual life to gain proper independence from the security guards of the world, the people who control physical reality. Assange later wrote in The Guardian: "Strong cryptography is a vital tool in fighting state oppression." saying that was the message of his book, Cypherpunks. Cypherpunks is published by OR Books. It is primarily a transcript of World Tomorrow episode eight, a two-part interview between Assange, Jacob Appelbaum, Andy Müller-Maguhn, and Jérémie Zimmermann. In the foreword, Assange said, "the Internet, our greatest tool for emancipation, has been transformed into the most dangerous facilitator of totalitarianism we have ever seen".
Brain technology
Brain technology, or self-learning know-how systems, defines a technology that employs latest findings in neuroscience. [see also neuro implants] The term was first introduced by the Artificial Intelligence Laboratory in Zurich, Switzerland, in the context of the Roboy project. Brain Technology can be employed in robots, know-how management systems and any other application with self-learning capabilities. In particular, Brain Technology applications allow the visualization of the underlying learning architecture often coined as "know-how maps". == Research and applications == The first demonstrations of BC in humans and animals took place in the 1960s when Grey Walter demonstrated use of non-invasively recorded encephalogram (EEG) signals from a human subject to control a slide projector (Graimann et al., 2010). Soon after Jacques J. Vidal coined the term brain–computer interface (BCI) in 1971, the Defense Advanced Research Projects Agency (DARPA) first starting funding brain–computer interface research and has since funded several brain–computer interface projects. That market is expected to reach a value of $1.72 billion by 2022. Brain–computer interfaces record brain activity, transmit the information out of the body, signal-process the data via algorithms, and convert them into command control signals. In 2012, a landmark study in Nature, led by pioneer Leigh Hochberg, MD, PhD, demonstrated that two people with tetraplegia were able to control robotic arms through thought when connected to the BrainGate neural interface system. The two participants were able to reach for and grasp objects in three-dimensional space, and one participant used the system to serve herself coffee for the first time since becoming paralyzed nearly 15 years prior. And in October 2020, two patients were able to wirelessly control an operating system to text, email, shop and bank using direct thought through the Stentrode brain computer interface (Journal of NeuroInterventional Surgery) in a study led by Thomas Oxley. This was the first time a brain–computer interface was implanted via the patient's blood vessels, eliminating the need for open brain surgery. Currently a number of groups are exploring a range of experimental devices using brain–computer interfaces, which have the potential to fundamentally change the way of life for patients with paralysis and a wide range of neurological disorders. These include: as Elon Musk, Facebook, and the University of California in San Francisco. The systems. This technology is also being explored as a neuromodulation device and may ultimately help diagnose and treat a range of brain pathologies, such as epilepsy and Parkinson's disease.
Change data capture
In databases, change data capture (CDC) is a set of software design patterns used to determine and track the data that has changed (the "deltas") so that action can be taken using the changed data. The result is a delta-driven dataset. CDC is an approach to data integration that is based on the identification, capture and delivery of the changes made to enterprise data sources. For instance it can be used for incremental update of data loading. CDC occurs often in data warehouse environments since capturing and preserving the state of data across time is one of the core functions of a data warehouse, but CDC can be utilized in any database or data repository system. == Methodology == System developers can set up CDC mechanisms in a number of ways and in any one or a combination of system layers from application logic down to physical storage. In a simplified CDC context, one computer system has data believed to have changed from a previous point in time, and a second computer system needs to take action based on that changed data. The former is the source, the latter is the target. It is possible that the source and target are the same system physically, but that would not change the design pattern logically. Multiple CDC solutions can exist in a single system. === Timestamps on rows === Tables whose changes must be captured may have a column that represents the time of last change. Names such as LAST_UPDATE, LAST_MODIFIED, etc. are common. Any row in any table that has a timestamp in that column that is more recent than the last time data was captured is considered to have changed. Timestamps on rows are also frequently used for optimistic locking so this column is often available. === Version numbers on rows === Database designers give tables whose changes must be captured a column that contains a version number. Names such as VERSION_NUMBER, etc. are common. One technique is to mark each changed row with a version number. A current version is maintained for the table, or possibly a group of tables. This is stored in a supporting construct such as a reference table. When a change capture occurs, all data with the latest version number is considered to have changed. Once the change capture is complete, the reference table is updated with a new version number. (Do not confuse this technique with row-level versioning used for optimistic locking. For optimistic locking each row has an independent version number, typically a sequential counter. This allows a process to atomically update a row and increment its counter only if another process has not incremented the counter. But CDC cannot use row-level versions to find all changes unless it knows the original "starting" version of every row. This is impractical to maintain.) === Status indicators on rows === This technique can either supplement or complement timestamps and versioning. It can configure an alternative if, for example, a status column is set up on a table row indicating that the row has changed (e.g., a boolean column that, when set to true, indicates that the row has changed). Otherwise, it can act as a complement to the previous methods, indicating that a row, despite having a new version number or a later date, still shouldn't be updated on the target (for example, the data may require human validation). === Time/version/status on rows === This approach combines the three previously discussed methods. As noted, it is not uncommon to see multiple CDC solutions at work in a single system, however, the combination of time, version, and status provides a particularly powerful mechanism and programmers should utilize them as a trio where possible. The three elements are not redundant or superfluous. Using them together allows for such logic as, "Capture all data for version 2.1 that changed between 2005-06-01 00:00 and 2005-07-01 00:00 where the status code indicates it is ready for production." === Triggers on tables === May include a publish/subscribe pattern to communicate the changed data to multiple targets. In this approach, triggers log events that happen to the transactional table into another queue table that can later be "played back". For example, imagine an Accounts table, when transactions are taken against this table, triggers would fire that would then store a history of the event or even the deltas into a separate queue table. The queue table might have schema with the following fields: Id, TableName, RowId, Timestamp, Operation. The data inserted for our Account sample might be: 1, Accounts, 76, 2008-11-02 00:15, Update. More complicated designs might log the actual data that changed. This queue table could then be "played back" to replicate the data from the source system to a target. Data capture offers a challenge in that the structure, contents and use of a transaction log is specific to a database management system. Unlike data access, no standard exists for transaction logs. Most database management systems do not document the internal format of their transaction logs, although some provide programmatic interfaces to their transaction logs (for example: Oracle, DB2, SQL/MP, SQL/MX and SQL Server 2008). Other challenges in using transaction logs for change data capture include: Coordinating the reading of the transaction logs and the archiving of log files (database management software typically archives log files off-line on a regular basis). Translation between physical storage formats that are recorded in the transaction logs and the logical formats typically expected by database users (e.g., some transaction logs save only minimal buffer differences that are not directly useful for change consumers). Dealing with changes to the format of the transaction logs between versions of the database management system. Eliminating uncommitted changes that the database wrote to the transaction log and later rolled back. Dealing with changes to the metadata of tables in the database. CDC solutions based on transaction log files have distinct advantages that include: minimal impact on the database (even more so if one uses log shipping to process the logs on a dedicated host). no need for programmatic changes to the applications that use the database. low latency in acquiring changes. transactional integrity: log scanning can produce a change stream that replays the original transactions in the order they were committed. Such a change stream include changes made to all tables participating in the captured transaction. no need to change the database schema == Confounding factors == As often occurs in complex domains, the final solution to a CDC problem may have to balance many competing concerns. === Unsuitable source systems === Change data capture both increases in complexity and reduces in value if the source system saves metadata changes when the data itself is not modified. For example, some Data models track the user who last looked at but did not change the data in the same structure as the data. This results in noise in the Change Data Capture. === Tracking the capture === Actually tracking the changes depends on the data source. If the data is being persisted in a modern database then Change Data Capture is a simple matter of permissions. Two techniques are in common use: Tracking changes using database triggers Reading the transaction log as, or shortly after, it is written. If the data is not in a modern database, CDC becomes a programming challenge. === Push versus pull === Push: the source process creates a snapshot of changes within its own process and delivers rows downstream. The downstream process uses the snapshot, creates its own subset and delivers them to the next process. Pull: the target that is immediately downstream from the source, prepares a request for data from the source. The downstream target delivers the snapshot to the next target, as in the push model. === Alternatives === Sometimes the slowly changing dimension is used as an alternative method. CDC and SCD are similar in that both methods can detect changes in a data set. The most common forms of SCD are type 1 (overwrite), type 2 (maintain history) or 3 (only previous and current value). SCD 2 can be useful if history is needed in the target system. CDC overwrites in the target system (akin to SCD1), and is ideal when only the changed data needs to arrive at the target, i.e. a delta-driven dataset.