AI Coding Claude

AI Coding Claude — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

• 

  Leakage (machine learning)

  In statistics and machine learning, leakage (also known as data leakage or target leakage) refers to the use of information during model training that would not be available at prediction time. This results in overly optimistic performance estimates, as the model appears to perform better during evaluation than it actually would in a production environment. Leakage is often subtle and indirect, making it difficult to detect and eliminate. It can lead a statistician or modeler to select a suboptimal model, which may be outperformed by a leakage-free alternative. == Leakage modes == Leakage can occur at multiple stages of the machine learning workflow. Broadly, its sources can be divided into two categories: those arising from features and those arising from training examples. === Feature leakage === Feature or column-wise leakage is caused by the inclusion of columns which are one of the following: a duplicate label, a proxy for the label, or the label itself. These features, known as anachronisms, will not be available when the model is used for predictions, and result in leakage if included when the model is trained. For example, including a "MonthlySalary" column when predicting "YearlySalary"; or "MinutesLate" when predicting "IsLate". === Training example leakage === Row-wise leakage is caused by improper sharing of information between rows of data. Types of row-wise leakage include: Premature featurization; leaking from premature featurization before Cross-validation/Train/Test split (must fit MinMax/ngrams/etc on only the train split, then transform the test set) Duplicate rows between train/validation/test (for example, oversampling a dataset to pad its size before splitting; or, different rotations/augmentations of a single image; bootstrap sampling before splitting; or duplicating rows to up sample the minority class) Non-independent and identically distributed random (non-IID) data Time leakage (for example, splitting a time-series dataset randomly instead of newer data in test set using a train/test split or rolling-origin cross-validation) Group leakage—not including a grouping split column (for example, Andrew Ng's group had 100k x-rays of 30k patients, meaning ~3 images per patient. The paper used random splitting instead of ensuring that all images of a patient were in the same split. Hence the model partially memorized the patients instead of learning to recognize pneumonia in chest x-rays.) A 2023 review found data leakage to be "a widespread failure mode in machine-learning (ML)-based science", having affected at least 294 academic publications across 17 disciplines, and causing a potential reproducibility crisis. == Detection == Data leakage in machine learning can be detected through various methods, focusing on performance analysis, feature examination, data auditing, and model behavior analysis. Performance-wise, unusually high accuracy or significant discrepancies between training and test results often indicate leakage. Inconsistent cross-validation outcomes may also signal issues. Feature examination involves scrutinizing feature importance rankings and ensuring temporal integrity in time series data. A thorough audit of the data pipeline is crucial, reviewing pre-processing steps, feature engineering, and data splitting processes. Detecting duplicate entries across dataset splits is also important. For language models, the Min-K% method can detect the presence of data in a pretraining dataset. It presents a sentence suspected to be present in the pretraining dataset, and computes the log-likelihood of each token, then compute the average of the lowest K of these. If this exceeds a threshold, then the sentence is likely present. This method is improved by comparing against a baseline of the mean and variance. Analyzing model behavior can reveal leakage. Models relying heavily on counter-intuitive features or showing unexpected prediction patterns warrant investigation. Performance degradation over time when tested on new data may suggest earlier inflated metrics due to leakage. Advanced techniques include backward feature elimination, where suspicious features are temporarily removed to observe performance changes. Using a separate hold-out dataset for final validation before deployment is advisable.

  Read more →
• 

  Randonautica

  Randonautica (a portmanteau of "random" + "nautica") is an app launched on February 22, 2020 founded by Auburn Salcedo and Joshua Lengfelder. It randomly generates coordinates that encourages the user to explore their local area and report what is found. According to its creators, the app is "an attractor of strange things," letting one choose specific coordinates based on a specific theme. It gained controversy after a report of two teenagers coincidentally finding a corpse while using the application. == Overview == The app, which creators claim to be inspired by chaos theory and Guy Debord's Theory of the Dérive, offers its users three types of coordinates to choose from: an attractor, a void, or an anomaly. The app has a cult following on YouTube and TikTok and there is a subreddit made by the creators for users of the app. == History == 29-year-old circus performer Joshua Lengfelder discovered a bot called Fatum Project in a fringe science chat group on Telegram in January 2019. According to The New York Times, "He absorbed the project’s theories about how random exploration could break people out of their predetermined realities, and how people could influence random outcomes with their minds." Lengfelder then created a Telegram bot using Fatum Project's code, generating coordinates. He then created the subreddit r/randonauts in March. In October, developer Simon Nishi McCorkindale made the bot's webpage. With the help of Auburn Salcedo, chief executive of a TV agency, both created Randonauts LLC. Salcedo became the chief operating officer while Lengfelder was the CEO. The app, called Randonautica, was launched on February 22, 2020. Later the same year the app and back-end got completely overhauled by a new team of developers and got a more visual and friendlier design and logo. In April 2022 Lengfelder exited Randonauts LLC and Auburn Salcedo became CEO. == Reception == The app has as many as 10.8 million users as of July 2020, gaining popularity amid the COVID-19 pandemic in the United States as restrictions have been lightened. Emma Chamberlain made a YouTube video about the app that helped increase its following. i-D reported that the hashtag #randonautica has gained 176.5 million views on TikTok, although it has not marketed itself yet. === Controversy === With the app's popularity, users started reporting coincidences which many find unsettling. The majority of reports were from TikTok and Reddit, as well as Telegram. The most notable controversy involved a group of people heading to a beach in Duwamish Head, Puget Sound, West Seattle per the app, where they found a bag with two dead bodies, a 27-year-old male and a 36-year-old female, as reported by the Seattle Police homicide detectives. In August 2020, police arrested and charged their landlord, Michael Lee Dudley, in connection with the murders. In March 2021, Dudley was denied bail while other people were under suspicion of aiding Dudley in the dismemberment and disposal of the bodies, but no one else had been charged. This has caused speculation that the app has an intended, puzzle-like theme. However, Lengfelder stated that it is "a shocking coincidence." Salcedo called the videos fake, and that "It’s so hard to manage, because people are really taking creative liberties after seeing how much traction the app is getting in that fear factor." In 2022, Michael Dudley was convicted of second degree murder for killing both victims, who were identified as Jessica Lewis and Austin Wenner. He was sentenced to 46 years in prison the following year. In their questions page, Randonautica's creators have said that if the app generates coordinates inside a private property, it is a violation of their terms and conditions to trespass. In addition, Randonautica has also received allegations that the app is used for human trafficking, which its creators have denied, saying that data collected by the app are anonymous. It also ensured that the app is not designed to violate religious customs, saying that "the app is simply a tool. Just as a knife can be used either to prepare dinner or to cut somebody."

  Read more →
• 

  Scenery generator

  A scenery generator (or terrain generator) is a software used to create landscape images, 3D models, and animations. These programs often use procedural generation to generate the landscapes, or sometimes created and rendered by a 3D artist. These programs are often used in video games or movies. Basic elements of landscapes created by scenery generators include terrain, water, foliage, and clouds. The process for basic random generation uses a diamond square algorithm. == Common features == Most scenery generators can create basic heightmaps to simulate the variation of elevation in basic terrain. Common techniques include Simplex noise, fractals, or the diamond-square algorithm, which can generate 2-dimensional heightmaps. A version of scenery generator can be very simplistic. Using a diamond-square algorithm with some extra steps involving fractals, an algorithm for random generation of terrain can be made with only 120 lines of code. The program in example takes a grid and then divides the grid repeatedly. Each smaller grid is then split into squares and diamonds and the algorithm then makes the randomized terrain for each square and diamond. Most programs for creating landscapes also allow for adjustment and editing of the landscape. For example, World Creator allows for terrain sculpting, which uses a similar brush system as Photoshop, and allows for additional terrain enhancement with its procedural techniques such as erosion, sediments, and more. Other tools in the World Creator program include terrain stamping, which allows you to import elevation maps and use them as a base. The programs tend to also allow for additional placement of rocks, trees, etc. These can be done procedurally or by hand depending on the program. Typically the models used for the placement objects are the same as to lessen the amount of work that would be done if the user was to create a multitude of different trees. The terrain generated the computer does a generation of multifractals then integrates them until finally rendering them onto the screen. These techniques are typically done “on-the-fly” which typically for a 128 × 128 resolution terrain would mean 1.5 seconds on a CPU from the early 1990s. == Applications == Scenery generators are commonly used in movies, animations, 3D rendering, and video games. For example, Industrial Light & Magic used E-on Vue to create the fictional environments for Pirates of the Caribbean: Dead Man's Chest. In such live-action cases, a 3D model of the generated environment is rendered and blended with live-action footage. Scenery generated by the software may also be used to create completely computer-generated scenes. In the case of animated movies such as Kung Fu Panda, the raw generation is assisted by hand-painting to accentuate subtle details. Environmental elements not commonly associated with landscapes, such as ocean waves, have also been handled by the software. Scenery generation is used in most 3D based video-games. These typically use either custom or purchased engines that contain their own scenery generators. For some games they tend to use a procedurally generated terrain. These typically use a form of height mapping and use of Perlin noise. This will create a grid that with one point in a 2D coordinate will create the same heightmap as it is pseudorandom, meaning it will result in the same output with the same input. This can then easily be translated into the product 3D image. These can then be changed from the editor tools in most engines if the terrain will be custom built. With recent developments neural networks can be built to create or texture the terrain based on previously suggested artwork or heightmap data. These would be generated using algorithms that have been able to identify images and similarities between them. With the info the machine can take other heightmaps and render a very similar looking image to the style image. This can be used to create similar images in example a Studio Ghibli or Van Gogh art-style. == Software == Most game engines, whether custom or proprietary, will have terrain generation built in. Some terrain generator programs include, Terragen, which can create terrain, water, atmosphere and lighting; L3DT, which provides similar functions to Terragen, and has a 2048 × 2048 resolution limit; and World Creator, which can create terrain, and is fully GPU powered. === List of 3D terrain generation software ===

  Read more →
• 

  Secure state

  A secure state is an information systems security term to describe where entities in a computer system are divided into subjects and objects, and it can be formally proven that each state transition preserves security by moving from one secure state to another secure state. Thereby it can be inductively proven that the system is secure. As defined in the Bell–LaPadula model, the secure state is built on the concept of a state machine with a set of allowable states in a system. The transition from one state to another state is defined by transition functions. A system state is defined to be "secure" if the only permitted access modes of subjects to objects are in accordance with a security policy.

  Read more →
• 

  AsoSoft text corpus

  The AsoSoft text corpus is the first large-scale Kurdish text corpus, collected and processed by the AsoSoft research and development group. It contains 458,000 documents (188 million tokens) that are collected from sources such as websites, news agencies, books, and magazines. The corpus is partially tagged by topic, so it can be used for topic identification tasks. Also, it is applicable for extracting language model and computational lexicon information. Part of the corpus (75 million tokens) is available online for non-commercial use. The corpus uses the TEI format.

  Read more →
• 

  Groover

  Groover is an online platform, record label and distributor, connecting artists and musicians with music professionals and media outlets. The service was founded in 2018 in France and operates from offices in Paris and New York. The platform has over 3,000 active contacts, including SPIN Magazine and Sofar Sounds. Groover uses a micro-payment model. Among the platform's over 500,000 regular users are record labels such as Ninja Tune, Ba Da Bing Records, Dance To The Radio, Roche Musique, Wagram Music, Secret City Records, and artists including Bonobo, Michael Bolton, Aloe Blacc, Haddaway, Passenger, La Femme and Chinese Man. == History == Groover was launched at the MaMA Music Convention in October 2018. It was co-founded by Dorian Perron, Romain Palmieri, and Rafaël Cohen while they were students at UC Berkeley. Initially growing in France, the company has expanded to the United States, Canada, the United Kingdom, Brazil, Italy, and elsewhere in Europe. In March 2019, Groover was part of the Business France delegation at the South by Southwest (SXSW) festival. In June 2019, Groover raised €1.3 million from various angel investors. In April 2021, Groover acquired the platform Soonvibes, which had 70,000 users at the time, in order to strengthen its community in the electronic music space. In November 2021, Groover announced a €6 million funding round from Bpifrance Creative Industries and Partech. Between 2023 and 2025, Groover entered strategic partnerships with major artist service providers, including CD Baby, TuneCore, SoundCloud, UnitedMasters, Symphonic Distribution, Audiomack and SACEM. In February 2024, Groover announced a Series A funding round of $8 million from OneRagTime, Trind, Techmind, and Mozza Angels. == Function == Using a micro-payment system, professionals listen to tracks and provide written feedback. These professionals retain full editorial independence and are under no obligation to share the track or contact the artist. == Awards == 2nd Prize for Music Innovation 2023 from the Centre national de la musique (France) "Future Creator" Award at the Petit Poucet Competition 2019 Jury's Special Mention at the MaMA Invent 2019 competition 1st Prize for Digital Initiative in Culture, Communication & Media 2019 awarded by Audiens "Start-up of the Year" at the Social Music Awards 2020 French American Entrepreneurship Award 2022 at the French Consulate in New York

  Read more →
• 

  Screenless video

  Screenless video is any system for transmitting visual information from a video source without the use of a screen. Screenless computing systems can be divided into three groups: Visual Image, Retinal Direct, and Synaptic Interface. == Visual image == Visual Image screenless display includes any image that the eye can perceive. The most common example of Visual Image screenless display is a hologram. In these cases, light is reflected off some intermediate object (hologram, LCD panel, or cockpit window) before it reaches the retina. In the case of LCD panels the light is refracted from the back of the panel, but is nonetheless a reflected source. Google has proposed a similar system to replace the screens of tablet computers and smartphones. == Retinal display == Virtual retinal display systems are a class of screenless displays in which images are projected directly onto the retina. They are distinguished from visual image systems because light is not reflected from some intermediate object onto the retina, it is instead projected directly onto the retina. Retinal Direct systems, once marketed, hold out the promise of extreme privacy when computing work is done in public places because most snooping relies on viewing the same light as the person who is legitimately viewing the screen, and retinal direct systems send light only into the pupils of their intended viewer. == Synaptic interface == Synaptic Interface screenless video does not use light at all. Visual information completely bypasses the eye and is transmitted directly to the brain. While such systems have only been implemented in humans in rudimentary form - for example, displaying single Braille characters to blind people – success has been achieved in sampling usable video signals from the biological eyes of a living horseshoe crab through their optic nerves, and in sending video signals from electronic cameras into the creatures' brains using the same method.

  Read more →
• 

  Spatiotemporal reservoir resampling

  Spatiotemporal reservoir resampling, commonly known as ReSTIR (from "Reservoir-based SpatioTemporal Importance Resampling"), is a collection of computer graphics techniques for reusing samples during rendering. It was developed primarily to allow more realistic lighting in real-time rendering, because relatively few rays can be traced per pixel while maintaining an acceptable frame rate. It can also be used to speed up off-line path tracing. The first ReSTIR paper, published in 2020, provided algorithms for direct lighting, allowing scenes containing thousands of lights to be rendered in real time on a high-end GPU. Researchers later proposed versions for rendering indirect lighting (and more recently, motion blur and depth of field) and built up a framework of mathematical concepts and notation conventions that help analyze such algorithms. A major focus of this work is removing or reducing the bias that could be introduced when samples from other pixels or frames are reused—or selectively allowing some bias in order to speed up rendering and reduce variance (visible as "noise" in the image). Versions for path tracing apply transformations called shift mappings to samples, typically reusing parts of paths closer to the light and modifying the portion closer to the camera. ReSTIR-related papers and talks have been presented every year at the SIGGRAPH conference since 2020. One of the first games to incorporate ReSTIR into its rendering was Cyberpunk 2077. == Overview and motivation == According to Chris Wyman, one of the co-authors of the original paper, although developers commonly thought that bias was acceptable for real-time rendering, end users (e.g. gamers) are well-aware of the artifacts caused by bias and many have a negative opinion of common sample-reuse techniques such as temporal anti-aliasing (TAA), which may cause "ghosting" when the camera moves, and denoising, which causes blurring and other artifacts. ReSTIR techniques can reduce or avoid these types of bias by reusing samples of the set of possible paths taken by light to reach the camera, instead of reusing rendered pixel color values (which are typically the average of multiple samples, discarding information such as the direction of the light). While other techniques reuse samples in a generic post-processing step, ReSTIR passes can test for shadowing, and reused samples are converted into pixel color values by rendering code that takes the characteristics of different materials into account (e.g. by implementing BRDFs). However the output of ReSTIR is noisy, and a denoising pass is typically still used. Stochastic ray tracing techniques such as path tracing need to average multiple samples (produced by tracing individual rays) in order to render a visually acceptable image. When using a simple unbiased renderer based on Monte Carlo integration, halving the deviation of the result (apparent as "noise" in the image) requires multiplying the number of samples by four, meaning that a rapidly increasingly number of samples is needed to improve quality, Standard ways to mitigate this problem include importance sampling (which requires finding improved sampling distributions for specific situations), and quasi-Monte Carlo integration (which usually still requires tracing a large number of rays). ReSTIR offers a solution that multiplies the effective number of samples while tracing a fixed number of additional rays per frame. Temporal reuse multiplies the effective sample count by the number of frames rendered. Spatial reuse multiplies the effective count by the number of neighboring pixels examined. These two types of reuse can be combined, allowing spatial reuse to be applied recursively, which appears to offer an exponentially increasing effective sample count, however this is quickly limited by the size of the neighborhood used for spatial reuse. Spatial reuse is also potentially less effective near shadow and object edges, especially for objects with fine geometric detail, and temporal reuse is limited by movement of the camera and scene elements. == Variations == Many variations of ReSTIR have been proposed that generalize or improve the original technique (which builds on an earlier method called RIS), specialize it for particular types of illumination or other visual effects, or allow incorporation into rendering algorithms other than standard path tracing. Some published versions are listed below. == Algorithms == === Basic algorithm === ReSTIR uses a combination of resampled importance sampling (RIS) and weighted reservoir sampling (WRS) which the authors call streaming RIS. RIS processes samples from an initial probability distribution (e.g. a probability distribution for which a cheap sampling method exists) and generates samples in a new probability distribution (e.g. a sampling distribution that is optimal for rendering but is impractical to draw samples from directly). WRS allows this to be done while storing only a small number of samples in memory, which is especially helpful on a GPU. Information about the samples is stored in a data structure called a reservoir. WRS also allows samples from multiple reservoirs to be combined ("merged") into a single reservoir; this is crucial for sample reuse. Each pixel has a reservoir, typically containing only a single sample when ReSTIR is used for real-time rendering (some implementations use a larger number, e.g. four samples). The reservoir is typically initialized to a sample drawn using a simple method and is then updated by RIS steps and by reservoir merging, so that the pixel value produced by shading using the sample(s) currently in the reservoir, times the weight for the sample, is always an unbiased estimate of the correct pixel value. If appropriate resampling steps are used, the variance of this estimate (or some function of it, typically the luminance of the RGB color value) decreases with each step. A possible sequence of steps performed for each frame, suitable for computing unbiased direct illumination (DI) is: Perform reservoir resampling by drawing multiple light samples and using streaming RIS to choose one, using probabilities based on a target function, e.g. the luminance of the sample's contribution to the pixel. A weight is also computed for the sample. Typically, a single visibility check is performed here, after choosing a sample, setting the weight to 0 if the light is shadowed. Resampling (combined with the visibility check) ensures that the expected value of the weight times the sample brightness is the correct (unbiased) value for the pixel. (temporal reuse) For each pixel, merge the sample(s) from the previous frame into the current reservoir. Multiple importance sampling (MIS) weights are used to avoid bias due to the fact that the samples in the previous frame's reservoirs may have a different target probability distribution if the objects, lights, or camera have moved. (spatial reuse) For each pixel, choose one or more neighboring pixels and merge their samples into the current pixel's reservoir. Multiple importance sampling (MIS) weights are used to avoid bias due to the fact that the samples in each pixel's reservoir have a different target probability distribution. Because computing unbiased MIS weights requires tracing additional rays (along with other work such as evaluating BRDFs), real-time rendering often uses only a single neighboring pixel. Use the sample in each pixel's reservoir, along with its weight, to determine the color of the pixel for the current frame. Alternatively, multiple samples examined during the preceding steps may be averaged and used to shade the pixel instead (decoupled shading and sampling). For direct lighting, the initial samples used in step 1 are typically drawn by importance sampling from the set of lights in a scene. The algorithm above (from the original ReSTIR paper) draws many lower-quality light samples (e.g. 32) using a fast method, without considering visibility, and chooses one using streaming RIS. Visibility is then tested for the final chosen sample. Considering visibility for each sample drawn would require tracing 32 rays, which would make it much more expensive. The intent is to reduce the number of rays traced, relying on the sample reuse in steps 2 and 3 to make up for the loss of quality caused by rejecting many of the rays due to shadowing. A large part of the initial efforts to optimize ReSTIR (to make it run in real-time on available hardware) went into reducing the cost of randomly sampling the lights. Glossy surfaces may require a larger number of samples, and combining light sampling with BRDF sampling (using MIS) may increase quality. Step 2 (temporal reuse) is sometimes skipped for off-line rendering, and the output of multiple repetitions of initial sampling and spatial reuse is averaged instead; this helps avoids artifacts due to correlations. Step 3 (spatial reuse) may be repeated multiple times in a single frame.

  Read more →
• 

  Embodied agent

  In artificial intelligence, an embodied agent, also sometimes referred to as an interface agent, is an intelligent agent that interacts with the environment through a physical body within that environment. Agents that are represented graphically with a body, for example a human or a cartoon animal, are also called embodied agents, although they have only virtual, not physical, embodiment. A branch of artificial intelligence focuses on empowering such agents to interact autonomously with human beings and the environment. Mobile robots are one example of physically embodied agents; Ananova and Microsoft Agent are examples of graphically embodied agents. Embodied conversational agents are embodied agents (usually with a graphical front-end as opposed to a robotic body) that are capable of engaging in conversation with one another and with humans employing the same verbal and nonverbal means that humans do (such as gesture, facial expression, and so forth). == Embodied conversational agents == Embodied conversational agents are a form of intelligent user interface. Graphically embodied agents aim to unite gesture, facial expression and speech to enable face-to-face communication with users, providing a powerful means of human-computer interaction. == Advantages == Face-to-face communication allows communication protocols that give a much richer communication channel than other means of communicating. It enables pragmatic communication acts such as conversational turn-taking, facial expression of emotions, information structure and emphasis, visualization and iconic gestures, and orientation in a three-dimensional environment. This communication takes place through both verbal and non-verbal channels such as gaze, gesture, spoken intonation and body posture. Research has found that users prefer a non-verbal visual indication of an embodied system's internal state to a verbal indication, demonstrating the value of additional non-verbal communication channels. As well as this, the face-to-face communication involved in interacting with an embodied agent can be conducted alongside another task without distracting the human participants, instead improving the enjoyment of such an interaction. Furthermore, the use of an embodied presentation agent results in improved recall of the presented information. Embodied agents also provide a social dimension to the interaction. Humans willingly ascribe social awareness to computers, and thus interaction with embodied agents follows social conventions, similar to human to human interactions. This social interaction both raises the believably and perceived trustworthiness of agents, and increases the user's engagement with the system. Rickenberg and Reeves found that the presence of an embodied agent on a website increased the level of user trust in that website, but also increased users' anxiety and affected their performance, as if they were being watched by a real human. Another effect of the social aspect of agents is that presentations given by an embodied agent are perceived as being more entertaining and less difficult than similar presentations given without an agent. Research shows that perceived enjoyment, followed by perceived usefulness and ease of use, is the major factor influencing user adoption of embodied agents. A study in January 2004 by Byron Reeves at Stanford demonstrated how digital characters could "enhance online experiences" through explaining how virtual characters essentially add a sense of familiarity to the user experience and make it more approachable. This increase in likability in turn helps make the products better, which benefits both the end users and those creating the product. === Applications === The rich style of communication that characterizes human conversation makes conversational interaction with embodied conversational agents ideal for many non-traditional interaction tasks. A familiar application of graphically embodied agents is computer games; embodied agents are ideal for this setting because the richer communication style makes interacting with the agent enjoyable. Embodied conversational agents have also been used in virtual training environments, portable personal navigation guides, interactive fiction and storytelling systems, interactive online characters and automated presenters and commentators. Major virtual assistants like Siri, Amazon Alexa and Google Assistant do not come with any visual embodied representation, which is believed to limit the sense of human presence by users. The U.S. Department of Defense utilizes a software agent called SGT STAR on U.S. Army-run Web sites and Web applications for site navigation, recruitment and propaganda purposes. Sgt. Star is run by the Army Marketing and Research Group, a division operated directly from The Pentagon. Sgt. Star is based upon the ActiveSentry technology developed by Next IT, a Washington-based information technology services company. Other such bots in the Sgt. Star "family" are utilized by the Federal Bureau of Investigation and the Central Intelligence Agency for intelligence gathering purposes.

  Read more →
• 

  Dark data

  Dark data is data which is acquired through various computer network operations but not used in any manner to derive insights or for decision making. The ability of an organisation to collect data can exceed the throughput at which it can analyse the data. In some cases the organisation may not even be aware that the data is being collected. IBM estimate that roughly 90 percent of data generated by sensors and analog-to-digital conversions never get used. In an industrial context, dark data can include information gathered by sensors and telematics. Organizations retain dark data for a multitude of reasons, and it is estimated that most companies are only analyzing 1% of their data. Often it is stored for regulatory compliance and record keeping. Some organizations believe that dark data could be useful to them in the future, once they have acquired better analytic and business intelligence technology to process the information. Because storage is inexpensive, storing data is easy. However, storing and securing the data usually entails greater expenses (or even risk) than the potential return profit. In academic discourse, the term dark data was essentially coined by Bryan P. Heidorn. He uses it to describe research data, especially from the long tail of science (the many, small research projects), which are not or no longer available for research because they disappear in a drawer without adequate data management. Without this, the data become dark, and further reasons for this are e.g. missing metadata annotation, missing data management plans and data curators. == Analysis == The term "dark data" very often refers to data that is not amenable to computer processing. For example, a company might have a great deal of data that exists only as scanned page-images. Even the bare text in such documents is not available without something like Optical character recognition, which can vary greatly in accuracy. Even with OCR, the significance of each part of the data is unavailable. An obvious examples is whether a capitalized word is a name or not, and if so, whether it represents a person, place, organization, or even a work of art. Bibliographic and other references, data within tables (that may be labeled quite adequately for humans, but not for processing), and countless assertions represented with the full complexity and ambiguity of human language. A lot of unused data is very valuable, and would be used if it could be; but is blocked because it is in formats that are difficult to process, categorise, identify, and analyse. Often the reason that business does not use their dark data is because of the amount of resources it would take and the difficulty of having that data analysed. In other words, the data is "dark" not because it is not used, but because it cannot (feasibly or affordably) be used, given its poor representation. There are many data representations that can make data much more accessible for automation. However, a great deal of information lacks any such identification of information items or relationships; and much more loses it during "downhill" conversion such as saving to page-oriented representations, printing, scanning, or faxing. The journey back "uphill" can be costly. According to Computer Weekly, 60% of organisations believe that their own business intelligence reporting capability is "inadequate" and 65% say that they have "somewhat disorganised content management approaches". == Relevance == Useful data may become dark data after it becomes irrelevant, as it is not processed fast enough. This is called "perishable insights" in "live flowing data". For example, if the geolocation of a customer is known to a business, the business can make offer based on the location, however if this data is not processed immediately, it may be irrelevant in the future. According to IBM, about 60 percent of data loses its value immediately. == Storage == According to the New York Times, 90% of energy used by data centres is wasted. If data was not stored, energy costs could be saved. Furthermore, there are costs associated with the underutilisation of information and thus missed opportunities. According to Datamation, "the storage environments of EMEA organizations consist of 54 percent dark data, 32 percent redundant, obsolete and trivial data and 14 percent business-critical data. By 2020, this can add up to $891 billion in storage and management costs that can otherwise be avoided." The continuous storage of dark data can put an organisation at risk, especially if this data is sensitive. In the case of a breach, this can result in serious repercussions. These can be financial, legal and can seriously hurt an organisation's reputation. For example, a breach of private records of customers could result in the stealing of sensitive information, which could result in identity theft. Another example could be the breach of the company's own sensitive information, for example relating to research and development. These risks can be mitigated by assessing and auditing whether this data is useful to the organisation, employing strong encryption and security and finally, if it is determined to be discarded, then it should be discarded in a way that it becomes unretrievable. == Future == It is generally considered that as more advanced computing systems for analysis of data are built, the higher the value of dark data will be. It has been noted that "data and analytics will be the foundation of the modern industrial revolution". Of course, this includes data that is currently considered "dark data" since there are not enough resources to process it. All this data that is being collected can be used in the future to bring maximum productivity and an ability for organisations to meet consumers' demand. Technology advancements are helping to leverage this dark data affordably. Furthermore, many organisations do not realise the value of dark data right now, for example in healthcare and education organisations deal with large amounts of data that could create a significant "potential to service students and patients in the manner in which the consumer and financial services pursue their target population".

  Read more →
• 

  NHS COVID-19

  NHS COVID-19 was a voluntary contact tracing app for monitoring the spread of the COVID-19 pandemic in England and Wales, in use from 24 September 2020 until 27 April 2023. It was available for Android and iOS smartphones, and could be used by anyone aged 16 or over. Two versions of the app were created. The first was commissioned by NHSX and developed by the Pivotal division of American software company VMware. A pilot deployment began in May 2020, but on 18 June development of the app was abandoned in favour of a second design using the Apple/Google Exposure Notification system. Scotland and Northern Ireland had separate contact tracing apps. A 2023 study estimated that in its first year of use, the app's contact tracing function prevented an estimated 1 million cases, and 9,600 deaths. == Description == The app allowed users to: See the alert level of their local authority area (in Wales) or information about restrictions (in England); to enable this, the user must enter the first half of their postcode "Check in" at places displaying an NHS QR code poster (no longer required by legislation after 26 January 2022, removed from the app the next month) Be notified when they have been in close contact with someone who has tested positive for the virus Be notified when local health protection teams determine that people with the virus had attended a business or other venue around the same time as the user Check their symptoms, and book a coronavirus test if necessary If asked to self-isolate, receive information and a daily "countdown". At first, "close contact" was defined as being within 2 metres for 15 minutes, or within 4 metres for a longer time. These time durations were reduced from 29 October 2020, to as little as three minutes when the other person is at their most infectious, i.e. soon after they begin showing symptoms. === Implementation === The Android app was coded in Kotlin, and the iOS app in Swift. The backend used Java and is deployed to Amazon Web Services using Terraform. The code of the app and back-end is open-source and available on GitHub. == Context == The app was part of the UK's test and trace programme which was chaired by Dido Harding; from 12 May 2020 Tom Riordan, chief executive of Leeds City Council, led the tracing effort. == First phase and cancellation == === Description === In March 2020, NHSX commissioned a contact tracing app to monitor the spread in the United Kingdom of the coronavirus disease 2019 (COVID-19) in the 2020 pandemic, developed by the Pivotal division of American software company VMware. The app used a centralised approach, in contrast to the Google / Apple contact tracing project. NHSX consulted ethicists and GCHQ's National Cyber Security Centre (NCSC) about the privacy aspects. The app recorded the make and model of the phone and asked the user for their postcode area. It generated a unique installation identification number and also a daily identification number. It then used Bluetooth Low Energy (BLE) to record the daily identification number of other users nearby. If a user was unwell, they could tell the app about symptoms which are characteristic of COVID-19, such as a fever and cough. These details were then passed to a central NHS server. This would assess the information and notify other users that have been in contact, giving them appropriate advice such as physical distancing. The NHS would also arrange for a swab test of the unwell user and the outcome would determine further notifications to contacts: if the test confirmed infection with COVID-19, the contacts would be asked to isolate. By June 2020, £11.8 million had been spent on the app; in 2020–21, £35 million was spent on the app. === Deployment === The first public trial of the app began on the Isle of Wight on 5 May 2020 and by 11 May it had been downloaded 55,000 times. When the first national contact tracing schemes were launched – Test, Trace, Protect in Wales on 13 May, then on 28 May NHS Test and Trace in England, and Test and Protect in Scotland – the app was not ready to be included. Replying to a question at the government's daily briefing on 8 June, Hancock was unable to give a date for rollout of the app in England, saying it would be brought in "when it's right to do so". On 17 June, Lord Bethell, junior minister for Innovation at the Department of Health and Social Care, said "we're seeking to get something going before the winter ... it isn't a priority for us at the moment". On 18 June, Health Secretary Matt Hancock announced development would switch to the Apple/Google system after admitting that Apple's restrictions on usage of Bluetooth prevented the app from working effectively. At the same press briefing Dido Harding, leader of the UK's test and trace programme, said "What we've done in really rigorously testing both our own Covid-19 app and the Google-Apple version is demonstrate that none of them are working sufficiently well enough to be actually reliable to determine whether any of us should self-isolate for two weeks [and] that's true across the world". === Concerns === The first, ultimately rejected, version of the app was subject to privacy concerns, the government backtracking on initial statements that the data collected from the app would not be shared outside the NHS. Matthew Gould, CEO of NHSX, the government department responsible for the app, said the data would be accessible to other organisations, but did not disclose which. Data collected would not necessarily be anonymised and would be held in a centralised repository. Over 150 of the UK's security and privacy experts warned the app's data could be used by 'a bad actor (state, private sector, or hacker)' to spy on citizens. Fears were discussed by the House of Commons' Human Rights Select Committee about plans for the app to record user location data. Parliament's Joint Committee on Human Rights said this version of the app should not be released without proper privacy protections. The second version of the app, released nationwide, addressed these concerns by employing a decentralised framework, the Apple/Google Exposure Notification system. Under this system, users remain pseudonymous: a person diagnosed with COVID-19 does not know which people are informed about an encounter, and contacted persons do not receive any information about the person diagnosed with COVID-19. The functionality of the app was also questioned in late April and early May 2020, as the software's use of Bluetooth required the app to be constantly running, meaning users could not use other apps or lock their device if the app was to function properly. The developers of the app were said to have found a way of working around this restriction. === Related contracts === Faculty – a company linked to Cambridge Analytica – provided research and modelling to NHSX in support of the response to the pandemic. Palantir, also linked to Cambridge Analytica, provided their data management platform. These contracts began in February and March respectively. == Second phase == As outlined on cancellation of the first app on 18 June 2020, the Department of Health and Social Care published on 30 July a brief description of the "next phase" app. Users would be able to scan a QR code at venues they visit, and later be notified if they had visited a place which was the source of a number of infections; the app would also assist with identifying symptoms and ordering a test. By using the Exposure Notification system from Apple and Google, personal data would be decentralised. Zuhlke Engineering Ltd, the UK branch of Swiss-based Zühlke Group, used 70 staff to complete the development of the app in 12 weeks. Zuhlke Engineering was awarded "Development Team of the Year" title at UK IT Industry awards in November 2021 for development of NHS COVID-19 application. === Timeline === Testing of the app by NHS volunteer responders, and selected residents of the Isle of Wight and the London Borough of Newham, began around 13 August. The app was made available to the public (aged 16 or over) in England and Wales on 24 September. An updated app released on 29 October, in part from collaboration with the Alan Turing Institute, improved the accuracy of measurements of the distance between the user's phone and other phones. At the same time, the duration threshold for determining exposure was reduced; this was expected to lead to an increase in the number of users told to self-isolate. An update to the app in April 2021, timed to coincide with easing of restrictions on hospitality businesses, was blocked by Apple and Google. It was intended that users who tested positive would be asked to share their history of visited venues, to assist in warning others, but this would have contravened assurances by Apple and Google that location data from devices would not be shared. === Statistics and effectiveness === The app was downloaded six million times on the first day it was generally availa

  Read more →
• 

  TeaOnHer

  TeaOnHer is a male-oriented dating surveillance mobile app that allows men to anonymously rate and comment on women they are dating. It was set up in response to the existence of Tea, a female-oriented dating app that allowed women to rate and comment on men. In 2025, Cosmopolitian magazine described it as America's second most popular mobile app, with it being the second most popular app in the lifestyle section of Apple's App Store. The TeaOnHer app has fewer features than the rival Tea app, focusing instead on anonymous commenting. It is listed as having been developed by a company called Newville Media Corporation. TechCrunch reported in 2025 that TeaOnHer had leaked credentials of some of its users.

  Read more →
• 

  Yahoo Groups

  Yahoo! Groups was a free-to-use system of electronic mailing lists offered by Yahoo!. Prior to February 2020, Yahoo! Groups was one of the world's largest collections of online discussion boards. It allowed members to subscribe to various groups, read subscribed discussions online, view and share photos, files and bookmarks within a group, access a group calendar, create polls for group members, and receive email notifications of new discussion topics. Some groups were simply announcement boards, to which only the group moderators could post, while others were discussion forums. Depending on each group's settings, membership could be open to everyone or only to invited or approved people. On February 1, 2020, Yahoo! removed online access to discussions and all other features except simple membership management, essentially turning all groups into mailing lists, and on October 13, 2020, it announced that Yahoo Groups would shut down completely on December 15, 2020. == History == In 1998 Yahoo! Clubs was launched as an extension of services developed by Yahoo! Messenger. In August 2000 Yahoo acquired eGroups.com. Yahoo! Groups was launched in early 2001 as an integration of technology from eGroups.com and community groups from both eGroups.com and Yahoo! Clubs. In 2001 Yahoo! deleted adult groups from its search directory, making it very difficult to locate Yahoo! groups with adult content. The Groups Updates Email feature was introduced in 2010. It summarized, in a single email, all the updates that occurred every twenty-four hours in all groups. In September 2010, a major facelift was rolled out, making Yahoo! Groups look very similar to Facebook. In December, Yahoo! Groups Japan emailed its users and posted a notice on its homepage, to announce that its service, which commenced in February 2004, would be closing on May 28, 2014. In October 2019, Yahoo! announced that all content that had been posted to Yahoo! Groups will be deleted on December 14, 2019; that date was later amended to January 31, 2020. Yahoo! announced that adding new content would be blocked on October 28, 2019. Once the content was deleted, users of Yahoo! Groups were only able to browse the group directory, request invitations and, if members of a group, send messages to that group. On October 13, 2020, Yahoo! announced they would be shutting down Yahoo! Groups on December 15, 2020. The site was closed down a few days after the advertised date, displaying a message that the service was officially shut down. This message stopped appearing at the end of January 2021 and the Yahoo! Groups web address began redirecting to the main Yahoo! page. === Criticism and controversy === On August 31, 2010, Yahoo! Groups started rolling out a major software change, which was denounced by a large number of users. The re-model was completely abandoned on January 12, 2011. == Site statistics == In August 2008, Yahoo! Group staff reported that there were 113 million users, and nine million Groups using 22 languages. In July 2010, the web analytics website Quantcast reported around 915 thousand unique visitors daily to the Yahoo! Groups website (US). In January 2011, that number had increased to 933 thousand unique visitors daily. The number did not include Yahoo! Group members who accessed the Groups site via email. In September 2010, at its "Product Runway" event, Yahoo! told reporters that Yahoo! Groups had 115 million group members and that there were 10 million Yahoo! groups. == Archives ==

  Read more →
• 

  CrocBITE

  CrocBITE (currently CrocAttack) was an online database of wild crocodilian attacks reported on humans in the world. The non-profit online research tool helped to scientifically analyze crocodilian behavior via complex models. Users were encouraged to feed information in a crowdsourcing manner. This website excludes captive crocodilian attacks, as well as non-fatal bites on professional handlers, rangers, staff, or researchers, and crocodilian attacks on pets and livestock, because its primary goal is to analyze natural human-crocodilian conflict in the wild for conservation and management purposes, and that these incidents do are not considered indicative of natural species behavior or typical human-wildlife conflict, as well as not providing enough useful data and helping researchers understand wild population behavior or typical human-wildlife conflict dynamics and helps create safety strategies for people living or working near wild crocodilians, rather than tracking workplace accidents in zoos or farms. While fatal incidents involving handlers are sometimes included on the website, typical captive incidents (such as handlers being bitten by them in zoos) are excluded because they are considered manageable professional risks rather than general public safety threats. == About == The online database was established in 2013 (2013) by Dr Adam Britton, a researcher at Charles Darwin University, his student Brandon Sideleau and Erin Britton. It was a compilation of government records, individual reports, registered contributors and historical data. Dr Simon Pooley, Junior Research fellow, Imperial College London joined hands to further the studies. The collaboration culminated when Dr Pooley met Dr Britton at the IUCN Crocodile Specialist Group, in Louisiana in 2014. The program received funds from Economic and Social Research Council, United Kingdom to the tune of A$30,000 and unspecified resourced plus amount from Big Gecko Crocodilian Research, Crocodillian.com and Charles Darwin University. The research yielded pertinent observations that provide inside into crocodile attacks. It was observed that most attacks on humans occur from bites of Saltwater crocodile as against the popular understanding of Nile crocodiles taking the top spot. This is not, however, believed to be the actual case, as most attacks by the Nile crocodile are believed to go unreported or only reported on a local level. The broad category of Nile crocodile attacks were segmented into West African crocodile and Crocodylus niloticus (the Nile Crocodile) species to get a clear understanding of their respective attack zones. The objective was that the information would be used by communities and conservation managers to help inform and educate people about how to keep safe. The information was vital for Australia and Africa where such attacks are more likely than in other parts of the world. This was the only database of its kind with such comprehensive collection of information made available online. The database is no longer online, and its founder Adam Britton is in custody having pleaded guilty to charges of bestiality on September 25, 2023. It has been rebranded and renamed CrocAttack, and serves as a updated database focusing on human-crocodilian conflict and records over 8,500 incidents from the past decades.

  Read more →
• 

  Israeli cybersecurity industry

  The Israeli cybersecurity industry is a rapidly growing sector within Israel's technology and innovation ecosystem. Israel is internationally recognized as a powerhouse in the cybersecurity domain, with numerous cybersecurity startups, established companies, research institutions, and government initiatives. Tel Aviv itself is being ranked 7th in annual list of best global tech ecosystems, as reported by the Jerusalem Post. == History == The roots of Israel's cybersecurity industry can be traced back to the country's strong focus on national security and intelligence. The establishment of elite military units such as Unit 8200, the Israeli Intelligence Corps unit responsible for signals intelligence and code decryption, played a significant role in the development of cybersecurity expertise in the country. Many former members of Unit 8200 have gone on to establish successful cybersecurity companies or join existing organizations, bringing their unique skill sets and experience to the private sector. == Market overview == As of 2024, Israel housed more than 450 cybersecurity startups and companies. In 2023, the value of exits by Israeli tech companies reached $7.5 billion. Israel's cybersecurity industry is characterized by a high concentration of startups develop new technologies in areas such as network security, endpoint protection, data security, cloud security, and threat intelligence. In recent years, the sector has attracted significant investment from both local and international venture capital firms, as well as major technology companies such as Microsoft, Google, and IBM. Several Israeli cybersecurity companies have gained global recognition and success, with some being acquired by major corporations or conducting successful initial public offerings (IPOs). === Key Israeli cybersecurity companies === Some key Israeli cybersecurity companies include: Check Point Software Technologies CyberArk Cato Networks Radware Wiz === Financial activity === Israel’s cybersecurity sector has seen significant financial activity. As of 2023, mergers and acquisitions in the cybersecurity sector totaled $2.8 billion. In the first quarter of 2024, the sector secured $846 million in private funding. == Background == The military experience helped much. Israel's mandatory military service, combined with the expertise developed within elite units such as Unit 8200, has fostered a strong talent pool with practical experience in cybersecurity. Israel's thriving startup ecosystem, often referred to as the "Startup Nation," has fostered an environment of innovation and collaboration that has contributed to the growth of the cybersecurity industry. Israeli cybersecurity companies often collaborate with international partners, both in the private and public sectors, to share knowledge and develop joint solutions. === Government Initiatives and Support === The government also supported well through various initiatives, such as the Israel National Cyber Directorate (INCD), which works to strengthen cybersecurity defenses and promote the development of the sector. === Academic institutions === Israeli universities and research centers are involved in cybersecurity research and education, contributing to the development of new technologies and training the next generation of cybersecurity professionals. Academic Tech transfer offices in Israel also facilitate the commercialization of cybersecurity technologies. Some academic institutions with cybersecurity laboratories include: Tel Aviv University Technion Ben-Gurion University

  Read more →