AI For Business Ualbany

AI For Business Ualbany — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

• 

  T-vertices

  T-vertices is a term used in computer graphics to describe a problem that can occur during mesh refinement or mesh simplification. The most common case occurs in naive implementations of continuous level of detail, where a finer-level mesh is "sewn" together with a coarser-level mesh by simply aligning the finer vertices on the edges of the coarse polygons. The result is a continuous mesh, however due to the nature of the z-buffer and certain lighting algorithms such as Gouraud shading, visual artifacts can often be detected. Some modeling algorithms such as subdivision surfaces will fail when a model contains T-vertices.

  Read more →
• 

  Honey encryption

  Honey encryption is a type of data encryption that "produces a ciphertext, which, when decrypted with an incorrect key as guessed by the attacker, presents a plausible-looking yet incorrect plaintext." == Creators == Ari Juels and Thomas Ristenpart of the University of Wisconsin, the developers of the encryption system, presented a paper on honey encryption at the 2014 Eurocrypt cryptography conference. == Method of protection == A brute-force attack involves repeated decryption with random keys; this is equivalent to picking random plaintexts from the space of all possible plaintexts with a uniform distribution. This is effective because even though the attacker is equally likely to see any given plaintext, most plaintexts are extremely unlikely to be legitimate i.e. the distribution of legitimate plaintexts is non-uniform. Honey encryption defeats such attacks by first transforming the plaintext into a space such that the distribution of legitimate plaintexts is uniform. Thus an attacker guessing keys will see legitimate-looking plaintexts frequently and random-looking plaintexts infrequently. This makes it difficult to determine when the correct key has been guessed. In effect, honey encryption "[serves] up fake data in response to every incorrect guess of the password or encryption key." The security of honey encryption relies on the fact that the probability of an attacker judging a plaintext to be legitimate can be calculated (by the encrypting party) at the time of encryption. This makes honey encryption difficult to apply in certain applications e.g. where the space of plaintexts is very large or the distribution of plaintexts is unknown. It also means that honey encryption can be vulnerable to brute-force attacks if this probability is miscalculated. For example, it is vulnerable to known-plaintext attacks: if the attacker has a crib that a plaintext must match to be legitimate, they will be able to brute-force even Honey Encrypted data if the encryption did not take the crib into account. == Example == An encrypted credit card number is susceptible to brute-force attacks because not every string of digits is equally likely. The number of digits can range from 13 to 19, though 16 is the most common. Additionally, it must have a valid IIN and the last digit must match the checksum. An attacker can also take into account the popularity of various services: an IIN from MasterCard is probably more likely than an IIN from Diners Club Carte Blanche. Honey encryption can protect against these attacks by first mapping credit card numbers to a larger space where they match their likelihood of legitimacy. Numbers with invalid IINs and checksums are not mapped at all (i.e. have probability 0 of legitimacy). Numbers from large brands like MasterCard and Visa map to large regions of this space, while less popular brands map to smaller regions, etc. An attacker brute-forcing such an encryption scheme would only see legitimate-looking credit card numbers when they brute-force, and the numbers would appear with the frequency the attacker would expect from the real world. == Application == Juels and Ristenpart aim to use honey encryption to protect data stored on password manager services. Juels stated that "password managers are a tasty target for criminals," and worries that "if criminals get a hold of a large collection of encrypted password vaults they could probably unlock many of them without too much trouble." Hristo Bojinov, CEO and founder of Anfacto, noted that "Honey Encryption could help reduce their vulnerability. But he notes that not every type of data will be easy to protect this way. … Not all authentication or encryption system yield themselves to being honeyed."

  Read more →
• 

  Interplanetary Internet

  The interplanetary Internet is a conceived computer network in space, consisting of a set of network nodes that can communicate with each other. These nodes are the planet's orbiters and landers, and the Earth ground stations. For example, the orbiters collect the scientific data from the Curiosity rover on Mars through near-Mars communication links, transmit the data to Earth through direct links from the Mars orbiters to the Earth ground stations via the NASA Deep Space Network, and finally the data routed through Earth's internal internet. Interplanetary communication is greatly delayed by interplanetary distances, as data transmission can only go as fast as the speed of light, so a new set of protocols and technologies that are tolerant to large delays and errors are required. The interplanetary Internet has been envisioned as a store and forward network of internets that is often disconnected, has a wireless backbone fraught with error-prone links and delays ranging from tens of minutes to even hours, even when there is a connection. As of 2024 agencies and companies working towards bringing the network to fruition include NASA, ESA, SpaceX and Blue Origin. == Challenges and reasons == In the core implementation of Interplanetary Internet, satellites orbiting a planet communicate to other planet's satellites. Simultaneously, these planets revolve around the Sun with long distances, and thus many challenges face the communications. The reasons and the resultant challenges are: The motion and long distances between planets: The interplanetary communication is greatly delayed due to the interplanetary distances and the motion of the planets. The delay is variable and long, ranging from a couple of minutes (Earth-to-Mars), to a couple of hours (Pluto-to-Earth), depending on their relative positions. The interplanetary communication also suspends due to the solar conjunction, when the sun's radiation hinders the direct communication between the planets. As such, the communication characterizes lossy links and intermittent link connectivity. Low embeddable payload: Satellites can only carry a small payload, which poses challenges to the power, mass, size, and cost for communication hardware design. An asymmetric bandwidth would be the result of this limitation. This asymmetry reaches ratios up to 1000:1 as downlink:uplink bandwidth portion. Absence of fixed infrastructure: The graph of participating nodes in a specific planet-to-planet communication keeps changing over time, due to the constant motion. The routes of the planet-to-planet communication are planned and scheduled rather than being opportunistic. The Interplanetary Internet design must address these challenges to operate successfully and achieve good communication with other planets. It also must use the few available resources efficiently in the system. == Development == Space communication technology has steadily evolved from expensive, one-of-a-kind point-to-point architectures, to the re-use of technology on successive missions, to the development of standard protocols agreed upon by space agencies of many countries. This last phase has gone on since 1982 through the efforts of the Consultative Committee for Space Data Systems (CCSDS), a body composed of the major space agencies of the world. It has 11 member agencies, 32 observer agencies, and over 119 industrial associates. The evolution of space data system standards has gone on in parallel with the evolution of the Internet, with conceptual cross-pollination where fruitful, but largely as a separate evolution. Since the late 1990s, familiar Internet protocols and CCSDS space link protocols have integrated and converged in several ways; for example, the successful FTP file transfer to Earth-orbiting STRV 1B on January 2, 1996, which ran FTP over the CCSDS IPv4-like Space Communications Protocol Specifications (SCPS) protocols. Internet Protocol use without CCSDS has taken place on spacecraft, e.g., demonstrations on the UoSAT-12 satellite, and operationally on the Disaster Monitoring Constellation. Having reached the era where networking and IP on board spacecraft have been shown to be feasible and reliable, a forward-looking study of the bigger picture was the next phase. The Interplanetary Internet study at NASA's Jet Propulsion Laboratory (JPL) was started by a team of scientists at JPL led by internet pioneer Vinton Cerf and the late Adrian Hooke. Cerf was appointed as a distinguished visiting scientist at JPL in 1998, while Hooke was one of the founders and directors of CCSDS. While IP-like SCPS protocols are feasible for short hops, such as ground station to orbiter, rover to lander, lander to orbiter, probe to flyby, and so on, delay-tolerant networking is needed to get information from one region of the Solar System to another. It becomes apparent that the concept of a region is a natural architectural factoring of the Interplanetary Internet. A region is an area where the characteristics of communication are the same. Region characteristics include communications, security, the maintenance of resources, perhaps ownership, and other factors. The Interplanetary Internet is a "network of regional internets". What is needed then, is a standard way to achieve end-to-end communication through multiple regions in a disconnected, variable-delay environment using a generalized suite of protocols. Examples of regions might include the terrestrial Internet as a region, a region on the surface of the Moon or Mars, or a ground-to-orbit region. The recognition of this requirement led to the concept of a "bundle" as a high-level way to address the generalized Store-and-Forward problem. Bundles are an area of new protocol development in the upper layers of the OSI model, above the Transport Layer with the goal of addressing the issue of bundling store-and-forward information so that it can reliably traverse radically dissimilar environments constituting a "network of regional internets". Delay-tolerant networking (DTN) was designed to enable standardized communications over long distances and through time delays. At its core is the Bundle Protocol (BP), which is similar to the Internet Protocol, or IP, that serves as the heart of the Internet here on Earth. The big difference between the regular Internet Protocol (IP) and the Bundle Protocol is that IP assumes a seamless end-to-end data path, while BP is built to account for errors and disconnections — glitches that commonly plague deep-space communications. Bundle Service Layering, implemented as the Bundling protocol suite for delay-tolerant networking, will provide general-purpose delay-tolerant protocol services in support of a range of applications: custody transfer, segmentation and reassembly, end-to-end reliability, end-to-end security, and end-to-end routing among them. The Bundle Protocol was first tested in space on the UK-DMC satellite in 2008. An example of one of these end-to-end applications flown on a space mission is the CCSDS File Delivery Protocol (CFDP), used on the Deep Impact comet mission. CFDP is an international standard for automatic, reliable file transfer in both directions. CFDP should not be confused with Coherent File Distribution Protocol, which has the same acronym and is an IETF-documented experimental protocol for rapidly deploying files to multiple targets in a highly networked environment. In addition to reliably copying a file from one entity (such as a spacecraft or ground station) to another entity, CFDP has the capability to reliably transmit arbitrarily small messages defined by the user, in the metadata accompanying the file, and to reliably transmit commands relating to file system management that are to be executed automatically on the remote end-point entity (such as a spacecraft) upon successful reception of a file. == Protocol == The Consultative Committee for Space Data Systems (CCSDS) packet telemetry standard defines the protocol used for the transmission of spacecraft instrument data over the deep-space channel. Under this standard, an image or other data sent from a spacecraft instrument is transmitted using one or more packets. === CCSDS packet definition === A packet is a block of data with length that can vary between successive packets, ranging from 7 to 65,542 bytes, including the packet header. Packetized data is transmitted via frames, which are fixed-length data blocks. The size of a frame, including frame header and control information, can range up to 2048 bytes. Packet sizes are fixed during the development phase. Because packet lengths are variable but frame lengths are fixed, packet boundaries usually do not coincide with frame boundaries. === Telecom processing notes === Data in a frame is typically protected from channel errors by error-correcting codes. Even when the channel errors exceed the correction capability of the error-correcting code, the presence of errors is nearly always detected by the e

  Read more →
• 

  Data transformation (computing)

  In computing, data transformation is the process of converting data from one format or structure into another format or structure. It is a fundamental aspect of most data integration and data management tasks such as data wrangling, data warehousing, data integration and application integration. Data transformation can be simple or complex based on the required changes to the data between the source (initial) data and the target (final) data. Data transformation is typically performed via a mixture of manual and automated steps. Tools and technologies used for data transformation can vary widely based on the format, structure, complexity, and volume of the data being transformed. A master data recast is another form of data transformation where the entire database of data values is transformed or recast without extracting the data from the database. All data in a well-designed database is directly or indirectly related to a limited set of master database tables by a network of foreign key constraints. Each foreign key constraint is dependent upon a unique database index from the parent database table. Therefore, when the proper master database table is recast with a different unique index, the directly and indirectly related data are also recast or restated. The directly and indirectly related data may also still be viewed in the original form since the original unique index still exists with the master data. Also, the database recast must be done in such a way as to not impact the applications architecture software. When the data mapping is indirect via a mediating data model, the process is also called data mediation. == Data transformation process == Data transformation can be divided into the following steps, each applicable as needed based on the complexity of the transformation required. Data discovery Data mapping Code generation Code execution Data review These steps are often the focus of developers or technical data analysts who may use multiple specialized tools to perform their tasks. The steps can be described as follows: Data discovery is the first step in the data transformation process. Typically the data is profiled using profiling tools or sometimes using manually written profiling scripts to better understand the structure and characteristics of the data and decide how it needs to be transformed. Data mapping is the process of defining how individual fields are mapped, modified, joined, filtered, aggregated etc. to produce the final desired output. Developers or technical data analysts traditionally perform data mapping since they work in the specific technologies to define the transformation rules (e.g. visual ETL tools, transformation languages). Code generation is the process of generating executable code (e.g. SQL, Python, R, or other executable instructions) that will transform the data based on the desired and defined data mapping rules. Typically, the data transformation technologies generate this code based on the definitions or metadata defined by the developers. Code execution is the step whereby the generated code is executed against the data to create the desired output. The executed code may be tightly integrated into the transformation tool, or it may require separate steps by the developer to manually execute the generated code. Data review is the final step in the process, which focuses on ensuring the output data meets the transformation requirements. It is typically the business user or final end-user of the data that performs this step. Any anomalies or errors in the data that are found and communicated back to the developer or data analyst as new requirements to be implemented in the transformation process. == Types of data transformation == === Batch data transformation === Traditionally, data transformation has been a bulk or batch process, whereby developers write code or implement transformation rules in a data integration tool, and then execute that code or those rules on large volumes of data. This process can follow the linear set of steps as described in the data transformation process above. Batch data transformation is the cornerstone of virtually all data integration technologies such as data warehousing, data migration and application integration. When data must be transformed and delivered with low latency, the term "microbatch" is often used. This refers to small batches of data (e.g. a small number of rows or a small set of data objects) that can be processed very quickly and delivered to the target system when needed. === Benefits of batch data transformation === Traditional data transformation processes have served companies well for decades. The various tools and technologies (data profiling, data visualization, data cleansing, data integration etc.) have matured and most (if not all) enterprises transform enormous volumes of data that feed internal and external applications, data warehouses and other data stores. === Limitations of traditional data transformation === This traditional process also has limitations that hamper its overall efficiency and effectiveness. The people who need to use the data (e.g. business users) do not play a direct role in the data transformation process. Typically, users hand over the data transformation task to developers who have the necessary coding or technical skills to define the transformations and execute them on the data. This process leaves the bulk of the work of defining the required transformations to the developer, which often in turn do not have the same domain knowledge as the business user. The developer interprets the business user requirements and implements the related code/logic. This has the potential of introducing errors into the process (through misinterpreted requirements), and also increases the time to arrive at a solution. This problem has given rise to the need for agility and self-service in data integration (i.e. empowering the user of the data and enabling them to transform the data themselves interactively). There are companies that provide self-service data transformation tools. They are aiming to efficiently analyze, map and transform large volumes of data without the technical knowledge and process complexity that currently exists. While these companies use traditional batch transformation, their tools enable more interactivity for users through visual platforms and easily repeated scripts. Still, there might be some compatibility issues (e.g. new data sources like IoT may not work correctly with older tools) and compliance limitations due to the difference in data governance, preparation and audit practices. === Interactive data transformation === Interactive data transformation (IDT) is an emerging capability that allows business analysts and business users the ability to directly interact with large datasets through a visual interface, understand the characteristics of the data (via automated data profiling or visualization), and change or correct the data through simple interactions such as clicking or selecting certain elements of the data. Although interactive data transformation follows the same data integration process steps as batch data integration, the key difference is that the steps are not necessarily followed in a linear fashion and typically don't require significant technical skills for completion. There are a number of companies that provide interactive data transformation tools, including Trifacta, Alteryx and Paxata. They are aiming to efficiently analyze, map and transform large volumes of data while at the same time abstracting away some of the technical complexity and processes which take place under the hood. Interactive data transformation solutions provide an integrated visual interface that combines the previously disparate steps of data analysis, data mapping and code generation/execution and data inspection. That is, if changes are made at one step (like for example renaming), the software automatically updates the preceding or following steps accordingly. Interfaces for interactive data transformation incorporate visualizations to show the user patterns and anomalies in the data so they can identify erroneous or outlying values. Once they've finished transforming the data, the system can generate executable code/logic, which can be executed or applied to subsequent similar data sets. By removing the developer from the process, interactive data transformation systems shorten the time needed to prepare and transform the data, eliminate costly errors in the interpretation of user requirements and empower business users and analysts to control their data and interact with it as needed. == Transformational languages == There are numerous languages available for performing data transformation. Many transformation languages require a grammar to be provided. In many cases, the grammar is structured using something closely resembling Backus–Naur form (BNF). There are numerous languages

  Read more →
• 

  XLeratorDB

  XLeratorDB is a suite of database function libraries that enable Microsoft SQL Server to perform a wide range of additional (non-native) business intelligence and ad hoc analytics. The libraries, which are embedded and run centrally on the database, include more than 450 individual functions similar to those found in Microsoft Excel spreadsheets. The individual functions are grouped and sold as six separate libraries based on usage: finance, statistics, math, engineering, unit conversions and strings. WestClinTech, the company that developed XLeratorDB, claims it is "the first commercial function package add-in for Microsoft SQL Server." == Company history == WestClinTech (LLC), founded by software industry veterans Charles Flock and Joe Stampf in 2008, is located in Irvington, New York, United States. Flock was a co-founder of The Frustum Group, developer of the OPICS enterprise banking and trading platform, which was acquired by London-based Misys, PLC in 1996. Stampf joined Frustum in 1994 and with Flock remained active with the company after acquisition, helping to develop successive generations of OPICS now employed by over 150 leading financial institutions worldwide. Following a full year of research, development and testing, WestClinTech introduced and recorded its first commercial sale of XLeratorDB in April 2009. In September 2009, XLeratorDB became available to all Federal agencies through NASA's Strategic Enterprise-Wide Procurement (SEWP-IV) program, a government-wide acquisition contract. == Technology == XLeratorDB uses Microsoft SQL CLR(Common Language Runtime) technology. SQL CLR allows managed code to be hosted by, and run in, the Microsoft SQL Server environment. SQL CLR relies on the creation, deployment and registration of .NET Framework assemblies that are physically stored in managed code dynamic-link libraries (DLL). The assemblies may contain .NET namespaces, classes, functions, and properties. Because managed code compiles to native code prior to execution, functions using SQL CLR can achieve significant performance increases versus the equivalent functions written in T-SQL in some scenarios. XLeratorDB requires Microsoft SQL Server 2005 or SQL Server 2005 Express editions, or later (compatibility mode 90 or higher). The product installs with PERMISSION_SET=SAFE. SAFE mode, the most restrictive permission set, is accessible by all users. Code executed by an assembly with SAFE permissions cannot access external system resources such as files, the network, the internet, environment variables, or the registry. == Functions == In computer science, a function is a portion of code within a larger program which performs a specific task and is relatively independent of the remaining code. As used in database and spreadsheet applications these functions generally represent mathematical formulas widely used across a variety of fields. While this code may be user-generated, it is also embedded as a pre-written sub-routine in applications. These functions are typically identified by common nomenclature which corresponds to their underlying operations: e.g. IRR identifies the function which calculates Internal Rate of Return on a series of periodic cash flows. === Function uses === As subroutines, functions can be integrated and used in a variety of ways, and as part of larger, more complicated applications. Within large enterprise applications they may, for example, play an important role in defining business rules or risk management parameters, while remaining virtually invisible to end users. Within database management systems and spreadsheets, however, these kinds of functions also represent discrete sets of tools; they can be accessed directly and utilized on a stand-alone basis, or in more complex, user-defined configurations. In this context, functions can be used for business intelligence and ad hoc analysis of data in fields such as finance, statistics, engineering, math, etc. === Function types === XLeratorDB uses three kinds of functions to perform analytic operations: scalar, aggregate, and a hybrid form which WestClinTech calls Range Queries. Scalar functions take a single value, perform an operation and return a single value. An example of this type of function is LOG, which returns the logarithm of a number to a specified base. Aggregate functions operate on a series of values but return a single, summarizing value. An example of this type of function is AVG, which returns the average of values in a specified group. In XLeratorDB there are some functions which have characteristics of aggregate functions (operating on multiple series of values) but cannot be processed in SQL CLR using single column inputs, such as AVG does. For example, irregular internal rate of return (XIRR), a financial function, operates on a collection of cash flow values from one column, but must also apply variable period lengths from another column and an initial iterative assumption from a third, in order to return a single, summarizing value. WestClinTech documentation notes that Range Queries specify the data to be included in the result set of the function independently of the WHERE clause associated with the T-SQL statement, by incorporating a SELECT statement into the function as a string argument; the function then traps that SELECT statement, executes it internally and processes the result. Some XLeratorDB functions that employ Range Queries are: NPV, XNPV, IRR, XIRR, MIRR, MULTINOMIAL, and SERIESSUM. Within the application these functions are identified by a "_q" naming convention: e.g. NPV_q, IRR_q, etc. == Analytic functions == === SQL Server functions === Microsoft SQL Server is the #3 selling database management system (DBMS), behind Oracle and IBM. (While versions of SQL Server have been on the market since 1987, XLeratorDB is compatible with only the 2005 edition and later.) Like all major DBMS, SQL Server performs a variety of data mining operations by returning or arraying data in different views (also known as drill-down). In addition, SQL Server uses Transact-SQL (T-SQL) to execute four major classes of pre-defined functions in native mode. Functions operating on the DBMS offer several advantages over client layer applications like Excel: they utilize the most up-to-date data available; they can process far larger quantities of data; and, the data is not subject to exporting and transcription errors. SQL Server 2008 includes a total of 58 functions that perform relatively basic aggregation (12), math (23) and string manipulation (23) operations useful for analytics; it includes no native functions that perform more complex operations directly related to finance, statistics or engineering. === Excel functions === Microsoft Excel, a component of Microsoft Office suite, is one of the most widely used spreadsheet applications on the market today. In addition to its inherent utility as a stand-alone desktop application, Excel overlaps and complements the functionality of DBMS in several ways: storing and arraying data in rows and columns; performing certain basic tasks such as pivot table and aggregating values; and facilitating sharing, importing and exporting of database data. Excel's chief limitation relative to a true database is capacity; Excel 2003 is limited to some 65k rows and 256 columns; Excel 2007 extends this capacity to roughly 1million rows and 16k columns. By comparison, SQL Server is able to manage over 500k terabytes of memory. Excel offers, however, an extensive library of specialized pre-written functions which are useful for performing ad hoc analysis on database data. Excel 2007 includes over 300 of these pre-defined functions, although customized functions can also be created by users, or imported from third party developers as add-ons. Excel functions are grouped by type: === Excel business intelligence functions === Operating on the client computing layer Excel plays an important role as a business intelligence tool because it: performs a wide array of complex analytic functions not native to most DBMS software offers far greater ad hoc reporting and analytic flexibility than most enterprise software provides a medium for sharing and collaborating because of its ubiquity throughout the enterprise Microsoft reinforces this positioning with Business Intelligence documentation that positions Excel in a clearly pivotal role. === XLeratorDB vs. Excel functions === While operating within the database environment, XLeratorDB functions utilize the same naming conventions and input formats, and in most cases, return the same calculation results as Excel functions. XLeratorDB, coupled with SQL Server's native capabilities, compares to Excel's function sets as follows:

  Read more →
• 

  Comparison of OLAP servers

  The following tables compare general and technical information for a number of online analytical processing (OLAP) servers. Please see the individual products articles for further information. == General information == == Data storage modes == == APIs and query languages == APIs and query languages OLAP servers support. == OLAP distinctive features == A list of OLAP features that are not supported by all vendors. All vendors support features such as parent-child, multilevel hierarchy, drilldown. == System limits == == Security == == Operating systems == The OLAP servers can run on the following operating systems: Note (1):The server availability depends on Java Virtual Machine not on the operating system == Support information ==

  Read more →
• 

  Electronic lab notebook

  An electronic lab notebook or electronic laboratory notebook (ELN) is a computer program designed to replace paper laboratory notebooks. Lab notebooks in general are used by scientists, engineers, and technicians to document research, experiments, and procedures performed in a laboratory. A lab notebook is often maintained to be a legal document and may be used in a court of law as evidence. Similar to an inventor's notebook, the lab notebook is also often referred to in patent prosecution and intellectual property litigation. Electronic lab notebooks offer many benefits to the user as well as organizations; they are easier to search upon, simplify data copying and backups, and support collaboration amongst many users. ELNs can have fine-grained access controls, and can be more secure than their paper counterparts. They also allow the direct incorporation of data from instruments, replacing the practice of printing out data to be stapled into a paper notebook. == Types == ELNs can be divided into two categories: "Specific ELNs" contain features designed to work with specific applications, scientific instrumentation or data types. "Cross-disciplinary ELNs" or "Generic ELNs" are designed to support access to all data and information that needs to be recorded in a lab notebook. Lab Platforms that combine an ELN, LIMS, and scientific data management together, all-in-one configurable software environment. Solutions range from specialized programs designed from the ground up for use as an ELN, to modifications or direct use of more general programs. Examples of using more general software as an ELN include using OpenWetWare, a MediaWiki install (running the same software that Wikipedia uses), WordPress, or the use of general note taking software such as OneNote as an ELN. ELN's come in many different forms. They can be standalone programs, use a client-server model, or be entirely web-based. Some use a lab-notebook approach, others resemble a blog. ELNs are embracing artificial intelligence and LLM technology to provide scientific AI chat assistants. A good many variations on the "ELN" acronym have appeared. Differences between systems with different names are often subtle, with considerable functional overlap between them. Examples include "ERN" (Electronic Research Notebook), "ERMS" (Electronic Resource (or Research or Records) Management System (or Software) and SDMS (Scientific Data (or Document) Management System (or Software). Ultimately, these types of systems all strive to do the same thing: Capture, record, centralize and protect scientific data in a way that is highly searchable, historically accurate, and legally stringent, and which also promotes secure collaboration, greater efficiency, reduced mistakes and lowered total research costs. == Objectives == A good electronic laboratory notebook should offer a secure environment to protect the integrity of both data and process, whilst also affording the flexibility to adopt new processes or changes to existing processes without recourse to further software development. The package architecture should be a modular design, so as to offer the benefit of minimizing validation costs of any subsequent changes that you may wish to make in the future as your needs change. A good electronic laboratory notebook should be an "out of the box" solution that, as standard, has fully configurable forms to comply with the requirements of regulated analytical groups through to a sophisticated ELN for inclusion of structures, spectra, chromatograms, pictures, text, etc. where a preconfigured form is less appropriate. All data within the system may be stored in a database (e.g. MySQL, MS-SQL, Oracle) and be fully searchable. The system should enable data to be collected, stored and retrieved through any combination of forms or ELN that best meets the requirements of the user. The application should enable secure forms to be generated that accept laboratory data input via PCs and/or laptops / palmtops, and should be directly linked to electronic devices such as laboratory balances, pH meters, etc. Networked or wireless communications should be accommodated for by the package which will allow data to be interrogated, tabulated, checked, approved, stored and archived to comply with the latest regulatory guidance and legislation. A system should also include a scheduling option for routine procedures such as equipment qualification and study related timelines. It should include configurable qualification requirements to automatically verify that instruments have been cleaned and calibrated within a specified time period, that reagents have been quality-checked and have not expired, and that workers are trained and authorized to use the equipment and perform the procedures. == Regulatory and legal aspects == The laboratory accreditation criteria found in the ISO 17025 standard needs to be considered for the protection and computer backup of electronic records. These criteria can be found specifically in clause 4.13.1.4 of the standard. Electronic lab notebooks used for development or research in regulated industries, such as medical devices or pharmaceuticals, are expected to comply with FDA regulations related to software validation. The purpose of the regulations is to ensure the integrity of the entries in terms of time, authorship, and content. Unlike ELNs for patent protection, FDA is not concerned with patent interference proceedings, but is concerned with avoidance of falsification. Typical provisions related to software validation are included in the medical device regulations at 21 CFR 820 (et seq.) and Title 21 CFR Part 11. Essentially, the requirements are that the software has been designed and implemented to be suitable for its intended purposes. Evidence to show that this is the case is often provided by a Software Requirements Specification (SRS) setting forth the intended uses and the needs that the ELN will meet; one or more testing protocols that, when followed, demonstrate that the ELN meets the requirements of the specification and that the requirements are satisfied under worst-case conditions. Security, audit trails, prevention of unauthorized changes without substantial collusion of otherwise independent personnel (i.e., those having no interest in the content of the ELN such as independent quality unit personnel) and similar tests are fundamental. Finally, one or more reports demonstrating the results of the testing in accordance with the predefined protocols are required prior to release of the ELN software for use. If the reports show that the software failed to satisfy any of the SRS requirements, then corrective and preventive action ("CAPA") must be undertaken and documented. Such CAPA may extend to minor software revisions, or changes in architecture or major revisions. CAPA activities need to be documented as well. Aside from the requirements to follow such steps for regulated industry, such an approach is generally a good practice in terms of development and release of any software to assure its quality and fitness for use. There are standards related to software development and testing that can be applied (see ref.).

  Read more →
• 

  TikTokification

  TikTokification (also written TikTok-ification) is a term used to describe the widespread adoption of TikTok's short-form, vertical video format and its algorithmic content-delivery model across the broader social media landscape. The phenomenon encompasses the strategic and cultural changes made by competing platforms such as Instagram, YouTube, Facebook, Snapchat, and LinkedIn in response to TikTok's global dominance. Beyond platform design, the term is also used more broadly to describe shifts in media consumption habits, advertising strategies, and, more critically, the potential cognitive and psychological effects associated with constant short-form video consumption. == Background == === Origins of short-form video === The short-form video format predates TikTok. Vine, launched in 2013, popularised six-second looping videos before shutting down in 2017. TikTok itself, known as Douyin in the Chinese market, was created by the Chinese technology company ByteDance in September 2016. Following its international expansion and its 2018 merger with Musical.ly, TikTok grew rapidly. By 2020, the application had surpassed two billion total downloads worldwide, with over 800 million monthly active users. A key driver of TikTok's success was its recommendation algorithm. The platform's "For You Page" (FYP) serves content to users based on behaviour rather than follower count, making it possible for unknown creators to achieve widespread reach organically. Analysts noted that TikTok serves "fast, visually engaging, and authentic videos that feel more like entertainment than advertising," fundamentally reshaping consumer expectations of digital content. TikTok has been described as "the center of the internet for young people," where users go for entertainment, news, trends, and shopping. As of the mid-2020s, TikTok had approximately 1.12 billion monthly active users. == Platform responses == TikTok's success compelled nearly every major social media platform to restructure its product around short-form video. In 2020, Instagram launched Reels and YouTube launched Shorts, both directly in response to TikTok's growth. Platforms like Meta's Instagram Reels and Google's YouTube Shorts subsequently expanded aggressively, launching new features, creator tools, and even considering separate standalone applications to compete. LinkedIn, traditionally a professional networking site, began experimenting with TikTok-style short-form vertical video feeds. Facebook launched a singular unified video feed combining Reels, long videos, and live videos, similar in structure to TikTok's feed. Snapchat redesigned its application to combine Stories and Spotlight into a unified entertainment feed. YouTube extended its Shorts format to allow videos up to three minutes in length, up from the previous limit of sixty seconds, as of October 2024. Despite these adaptations, experts noted that none of TikTok's rivals had matched its algorithmic precision as of mid-2025. == Societal and cultural impact == === Media and journalism === News organisations have also been affected by TikTokification. Short-form video grew rapidly as a format for news content, driven in large part by TikTok's popularity. According to Pew Research Center, 17% of adults in the United States reported regularly getting news from TikTok in 2024, with 63% of teenagers saying they used the platform as a news source. In response, major publishers began creating bespoke short-form content for TikTok's audience, with organisations such as the BBC building dedicated internal TikTok teams. === Advertising and commerce === TikTokification has had significant effects on the advertising industry. US social video advertising spending was projected to surpass linear television advertising spending for the first time in 2025. Global social commerce sales were projected to reach approximately $900 billion in 2025, with platforms like Douyin and TikTok driving a large share of that growth. TikTok itself generated an estimated $23.6 billion in advertising revenue in 2024. Short-form video has been described as bridging the gap between brand awareness and direct conversion. Surveys have found that consumers trust user-generated content 8.7 times more than influencer content and 6.6 times more than branded content, prompting brands to favour creator-led video formats. === Attention spans and cognitive effects === A growing body of research has examined the cognitive consequences of heavy short-form video consumption, a set of effects sometimes referred to as "TikTok Brain." A large systematic review and meta-analysis published in Psychological Bulletin, analysing data from 98,299 participants across 71 studies, found that the more short-form video content a person watches, the poorer their cognitive performance in attention and inhibitory control. The review also found that greater engagement with short-form video was associated with higher levels of anxiety, depression, and stress, as well as sleep disturbances. The platform's inherent demand for engaging content has resulted in the proliferation of sludge content, a genre of split screen video with the main video on the top and an unrelated attention-grabbing video on the bottom, typically repetitive gameplay (notably of the endless runner mobile game Subway Surfers) or oddly satisfying videos, designed to maximize viewer retention in cases where the main video may appear uninteresting and would normally cause the viewer to skip it. Sludge content is often described as overstimulating, reflecting and contributing to declining attention spans, though the scholarly evidence supporting such claims is not conclusive. Dr. Yann Poncin, associate professor at the Child Study Center at Yale University, noted that "infinite scrolling and short-form video are designed to capture your attention in short bursts," contrasting this with earlier entertainment formats that guided audiences through longer narratives. Research suggests that children and teenagers may be particularly vulnerable, with early exposure to rapid frame changes potentially conditioning the brain's neural pathways to require constant stimulation, making it more challenging to engage with slower-paced activities. A separate study published in Nature Communications by researchers at the Technical University of Denmark documented a notable decrease in collective attention span over time, attributing it in part to the increasing volume and pace of content production and consumption online. Researchers caution, however, that the majority of relevant studies are cross-sectional, meaning they capture data at a single point in time and cannot establish causality. It remains possible that individuals with pre-existing conditions such as anxiety or attention deficits may be more likely to engage heavily with these platforms as a coping mechanism. === Academic and sociological analysis === Scholars have framed TikTokification within the context of the attention economy. A 2024 academic analysis described TikTok as representing "a new paradigm of social media communication" shaped by youth culture, mobile technology, and the economics of attention, in which spectators become active contributors to a shared content pipeline. The same analysis noted that TikTok "reflects a new mode of communication influenced by avant-garde cinema, the use of mobile technology, and the social habits of particular social groups." US social media users were projected to spend 61.1% of their time on social networks watching videos in 2025, up from 33.3% in 2019, before TikTok became widely popular, underscoring the scale of the behavioural shift. == Monetisation challenges == Despite high engagement levels, monetising short-form video has remained difficult for platforms and creators alike. Unlike long-form YouTube content, short clips offer limited space for advertisers to insert advertisements. YouTube Shorts pays approximately four cents per 1,000 views, considerably less than its long-form counterpart. From 2025 onward, platforms began introducing creator funds, advertisements, and AI-driven content recommendations as part of broader efforts to make short-form video economically sustainable for creators.

  Read more →
• 

  Circular thresholding

  Circular thresholding is an algorithm for automatic image threshold selection in image processing. Most threshold selection algorithms assume that the values (e.g. intensities) lie on a linear scale. However, some quantities such as hue and orientation are a circular quantity, and therefore require circular thresholding algorithms. The example shows that the standard linear version of Otsu's method when applied to the hue channel of an image of blood cells fails to correctly segment the large white blood cells (leukocytes). In contrast the white blood cells are correctly segmented by the circular version of Otsu's method. == Methods == There are a relatively small number of circular image threshold selection algorithms. The following examples are all based on Otsu's method for linear histograms: (Tseng, Li and Tung 1995) smooth the circular histogram, and apply Otsu's method. The histogram is cyclically rotated so that the selected threshold is shifted to zero. Otsu's method and histogram rotation are applied iteratively until several heuristics involving class size, threshold location, and class variance are satisfied. (Wu et al. 2006) smooth the circular histogram until it contains only two peaks. The histogram is cyclically rotated so that the midpoint between the peaks is shifted to zero. Otsu's method and histogram rotation are applied iteratively until convergence of the threshold. (Lai and Rosin 2014) applied Otsu's method to the circular histogram. For the two class circular thresholding task they showed that, for a histogram with an even number of bins, the optimal solution for Otsu's criterion of within-class variance is obtained when the histogram is split into two halves. Therefore the optimal solution can be efficiently obtained in linear rather than quadratic time. == References and further reading == D.-C. Tseng, Y.-F. Li, and C.-T. Tung, Circular histogram thresholding for color image segmentation in Proc. Int. Conf. Document Anal. Recognit., 1995, pp. 673–676. J. Wu, P. Zeng, Y. Zhou, and C. Olivier, A novel color image segmentation method and its application to white blood cell image analysis in Proc. Int. Conf. Signal Process., vol. 2. 2006, pp. 16–20. Y.K. Lai, P.L. Rosin, Efficient Circular Thresholding, IEEE Trans. on Image Processing 23(3), 992–1001 (2014). doi:10.1109/TIP.2013.2297014

  Read more →
• 

  Information leakage

  Information leakage happens whenever a system that is designed to be closed to an eavesdropper reveals some information to unauthorized parties nonetheless. In other words: Information leakage occurs when secret information correlates with, or can be correlated with, observable information. For example, when designing an encrypted instant messaging network, a network engineer without the capacity to crack encryption codes could see when messages are transmitted, even if he could not read them. == Risk vectors == A modern example of information leakage is the leakage of secret information via data compression, by using variations in data compression ratio to reveal correlations between known (or deliberately injected) plaintext and secret data combined in a single compressed stream. Another example is the key leakage that can occur when using some public-key systems when cryptographic nonce values used in signing operations are insufficiently random. Bad randomness cannot protect proper functioning of a cryptographic system, even in a benign circumstance, it can easily produce crackable keys that cause key leakage. Information leakage can sometimes be deliberate: for example, an algorithmic converter may be shipped that intentionally leaks small amounts of information, in order to provide its creator with the ability to intercept the users' messages, while still allowing the user to maintain an illusion that the system is secure. This sort of deliberate leakage is sometimes known as a subliminal channel. Generally, only very advanced systems employ defenses against information leakage. Following are the commonly implemented countermeasures : Use steganography to hide the fact that a message is transmitted at all. Use chaffing to make it unclear to whom messages are transmitted (but this does not hide from others the fact that messages are transmitted). For busy re-transmitting proxies, such as a Mixmaster node: randomly delay and shuffle the order of outbound packets - this will assist in disguising a given message's path, especially if there are multiple, popular forwarding nodes, such as are employed with Mixmaster mail forwarding. When a data value is no longer going to be used, erase it from the memory.

  Read more →
• 

  Sex differences in social media use

  Men and women use social media in different ways and with different frequencies. In general, several researchers have found that women tend to use social network services (SNSs) more than men and primiarly to socialize. == Differences == === Predilection for usage === Many studies have found that women are more likely to use either specific SNSs such as Facebook or MySpace or SNSs in general. In 2015, 73% of online men and 80% of online women used social networking sites. The gap in gender differences has become less apparent in LinkedIn. In 2015 about 26 percent of online men and 25% of online women used the business-and employee-oriented networking site. Researchers who have examined the gender of users of multiple SNSs have found contradictory results. Hargittai's groundbreaking 2007 study examining race, gender, and other differences between undergraduate college student users of SNSs found that women were not only more likely to have used SNSes than men but that they were also more likely to have used many different services, including Facebook, MySpace, and Friendster; these differences persisted in several models and analyses. Although she only surveyed students at one institution – the University of Illinois at Chicago – Hargittai selected that institution intentionally as "an ideal location for studies of how different kinds of people use online sites and services." In contrast, data collected by the Pew Internet & American Life Project found that men were more likely to have multiple SNS profiles. Although the sample sizes of the two surveys are comparable – 1,650 Internet users in the Pew survey compared with 1,060 in Hargittai's survey – the data from the Pew survey are newer and arguably more representative of the entire adult United States population. Pinterest, Facebook, and Instagram attract more females. Picture sharing sites overall are very popular among women. Pinterest alone attracts three times as many female users than male. However, use of Pinterest by men has increased from 5% in 2012. Facebook attracts about 77% of women online. Instagram is also more likely to attract women. Men are more likely to participate in online forums like Reddit, Digg or Slashdot. One in five men claim to be a part of an online forum. === Uses === In general, women seem to use SNSs more to explicitly foster social connections. A study conducted by Pew research centers found that women were more avid users of social media. In November 2010, the gap between men and women was as high as 15%. Female participants in a multi-stage study conducted in 2007 to discover the motivations of Facebook users scored higher on scales for social connection and posting of photographs. Studies have also been conducted on the differences between females and males with regards to blogging. The Pew Research Center found that younger females are more likely to blog than males their own age, even males that are older than them. Similarly, in a study of blogs maintained in MySpace, women were found to be more likely to not only write blogs but also write about family, romantic relationships, friendships, and health in those blogs. A study of Swedish SNS users found that women were more likely to have expressions of friendship, specifically in the areas of (a) publishing photos of their friends, (b) specifically naming their best friends, and (c) writing poems to and about their friends. Women were also more likely to have expressions related to family relationships and romantic relationships. One of the key findings of this research is that those men who do have expressions of romantic relationships in their profile had expressions just as strong as the women. However, the researcher speculated that this may be in part due to a desire to publicly express heterosexual behaviors and mannerisms instead of merely expressing romantic feelings. A large-scale study of gender differences in MySpace found that both men and women tended to have a majority of female Friends, and both men and women tended to have a majority of female "Top" Friends in the site. A later study found women to author disproportionately many (public) comments in MySpace, but an investigation into the role of emotion in public MySpace comments found that women both give and receive stronger positive emotion. It was hypothesised that women are simply more effective at using social networking sites because they are better able to harness positive emotion. A study focused on the influence of gender and personality on individuals' use of online social networking websites such as Facebook, reported that men use social networking sites with the intention of forming new relationships, whereas, women use them more for relationship maintenance. In addition to this, women are more likely to use Facebook or MySpace to compare themselves to others and also to search for information. Men, however, are more likely to look at other people's profiles with in the intention to find friends. Women were less successful at actually finding new friends, but more successful at "maintaining existing relationships, making new relationships, using for academic purposes and following specific agenda". Similarly, men also self-reported this motivation "while women reported using them more for relationship maintenance". === Personality === OCEAN personality traits are known to systematically vary between human males and females. In one study, the same women were more extraverted and agreeable, such as less neurotic while on social media than offline. Other studies associated neuroticism with female use of social media. === Privacy === Privacy has been the primary topic of many studies of SNS users, and many of these studies have found differences between male and female SNS users, although some studies have found results contradictory to those found in other studies. Some researchers have found that women are more protective of their personal information and more likely to have private profiles. Other researchers have found that women are less likely to post some types of information. Acquisti and Gross found that women in their sample were less likely to reveal their sexual orientation, personal address, or cell phone number. This is similar to Pew Internet & American Life research of children users of SNSs that found that boys and girls presented different views of privacy and behaviors, with girls being more concerned about and restrictive of information such as city, town, last name, and cell phone number that could be used to locate them. At least one group of researchers has found that women are less likely to share information that "identifies them directly – last name, cell phone number, and address or home phone number," linking that resistance to women's greater concerns about "cyberstalking", "cyberbullying", and security problems. Despite these concerns about privacy, researchers have found that women are more likely to maintain up-to-date photos of themselves. Further, Kolek and Saunders found in their sample of college student Facebook users that women were more likely to not only post a photograph of themselves in their profile but that they were more likely to have a publicly viewable Facebook account (a contradictory finding compared to many other studies), post photos, and post photo albums. Women were more likely to have: (a) a publicly viewable Facebook account, (b) more photo albums, (c) more photos, (d) a photo of themselves as their profile picture, (e) positive references to alcohol, partying, or drugs, and (f) more positive references to or about the institution or institution-related activities. In general, women were more likely to disclose information about themselves in their Facebook profile, with the primary exception of sharing their telephone number. Similarly, female respondents to Strano's study were more likely to keep their profile photo recent and choose a photo that made them appear attractive, happy, and fun-loving. Citing several examples, Strano opined that there may also be a difference in how men and women Facebook users display and interpret profile photos depicting relationships. Privacy has also been a concern for the SnapChat app, which allows you to send messages either text or photo or video which then disappear. One study has shown that security is not a major concern for the majority of users and that most do not use Snapchat to send sensitive content (although up to 25% may do so experimentally). As part of their research almost no statistically significant gender differences were found. === Cyberbullying === Past research carried out to investigate if there are any gender differences in cyber-bullying has found that boys commit more cyber verbal bullying, cyber forgery and more violence based on hidden identity or presenting themselves as other person. === Mansplaining === A 2021 article found that mansplaining could be seen more prominent online rather than offl

  Read more →
• 

  Reverse proxy

  In computer networks, a reverse proxy or surrogate server is a proxy server that appears to any client to be an ordinary web server, but in reality merely acts as an intermediary that forwards the client's requests to one or more ordinary web servers. Reverse proxies help increase scalability, performance, resilience, and security, but they also carry a number of risks. Companies that run web servers often set up reverse proxies to facilitate the communication between an Internet user's browser and the web servers. An important advantage of doing so is that the web servers can be hidden behind a firewall on a company-internal network, and only the reverse proxy needs to be directly exposed to the Internet. Reverse proxy servers are implemented in popular open-source web servers. Dedicated reverse proxy servers are used by some of the biggest websites on the Internet. A reverse proxy is capable of tracking IP addresses of requests that are relayed through it as well as reading and/or modifying any non-encrypted traffic. However, this implies that anyone who has compromised the server could do so as well. Reverse proxies differ from forward proxies, which are used when the client is restricted to a private, internal network and asks a forward proxy to retrieve resources from the public Internet. == Uses == Large websites and content delivery networks use reverse proxies, together with other techniques, to balance the load between internal servers. Reverse proxies can keep a cache of static content, which further reduces the load on these internal servers and the internal network. It is also common for reverse proxies to add features such as compression or TLS encryption to the communication channel between the client and the reverse proxy. Reverse proxies can inspect HTTP headers, which, for example, allows them to present a single IP address to the Internet while relaying requests to different internal servers based on the URL of the HTTP request. Reverse proxies can hide the existence and characteristics of origin servers. This can make it more difficult to determine the actual location of the origin server / website and, for instance, more challenging to initiate legal action such as takedowns or block access to the website, as the IP address of the website may not be immediately apparent. Additionally, the reverse proxy may be located in a different jurisdiction with different legal requirements, further complicating the takedown process. Application firewall features can protect against common web-based attacks, like a denial-of-service attack (DoS) or distributed denial-of-service attacks (DDoS). Without a reverse proxy, removing malware or initiating takedowns (while simultaneously dealing with the attack) on one's own site, for example, can be difficult. In the case of secure websites, a web server may not perform TLS encryption itself, but instead offload the task to a reverse proxy that may be equipped with TLS acceleration hardware. (See TLS termination proxy.) A reverse proxy can distribute the load from incoming requests to several servers, with each server supporting its own application area. In the case of reverse proxying web servers, the reverse proxy may have to rewrite the URL in each incoming request in order to match the relevant internal location of the requested resource. A reverse proxy can reduce load on its origin servers by caching static content and dynamic content, known as web acceleration. Proxy caches of this sort can often satisfy a considerable number of website requests, greatly reducing the load on the origin server(s). A reverse proxy can optimize content by compressing it in order to speed up loading times. In a technique named "spoon-feeding", a dynamically generated page can be produced in its entirety and served to the reverse proxy, which can feed the page to the client as the connection allows. The program that generates the page need not remain open, thus releasing server resources during the possibly extended time the client requires to complete the transfer. Reverse proxies can operate wherever multiple web-servers must be accessible via a single public IP address. The web servers listen on different ports in the same machine, with the same local IP address or, possibly, on different machines with different local IP addresses. The reverse proxy analyzes each incoming request and delivers it to the right server within the local area network. Reverse proxies can perform A/B testing and multivariate testing without requiring application code to handle the logic of which version is served to a client. A reverse proxy can add access authentication to a web server that does not have any authentication. == Risks == When the transit traffic is encrypted and the reverse proxy needs to filter/cache/compress or otherwise modify or improve the traffic, the proxy first must decrypt and re-encrypt communications. This requires the proxy to possess the TLS certificate and its corresponding private key, extending the number of systems that can have access to non-encrypted data and making it a more valuable target for attackers. The vast majority of external data breaches happen either when hackers succeed in abusing an existing reverse proxy that was intentionally deployed by an organization, or when hackers succeed in converting an existing Internet-facing server into a reverse proxy server. Compromised or converted systems allow external attackers to specify where they want their attacks proxied to, enabling their access to internal networks and systems. Applications that were developed for the internal use of a company are not typically hardened to public standards and are not necessarily designed to withstand all hacking attempts. When an organization allows external access to such internal applications via a reverse proxy, they might unintentionally increase their own attack surface and invite hackers. If a reverse proxy is not configured to filter attacks or it does not receive daily updates to keep its attack signature database up to date, a zero-day vulnerability can pass through unfiltered, enabling attackers to gain control of the system(s) that are behind the reverse proxy server. Giving the reverse proxy of a third party access to private keys (for caching or optimizing content) places the entire triad of confidentiality, integrity and availability in the hands of the third party who operates the proxy. A reverse proxy is a single point of failure for the back-end services it fronts: an outage caused by misconfiguration, a denial-of-service attack, or a software fault can make every fronted service unreachable to outside clients, even when the back-end services themselves remain healthy. For example, a 2020 outage at Cloudflare briefly took down major sites and services that relied on its reverse-proxy edge, including Discord.

  Read more →
• 

  Leakage (machine learning)

  In statistics and machine learning, leakage (also known as data leakage or target leakage) refers to the use of information during model training that would not be available at prediction time. This results in overly optimistic performance estimates, as the model appears to perform better during evaluation than it actually would in a production environment. Leakage is often subtle and indirect, making it difficult to detect and eliminate. It can lead a statistician or modeler to select a suboptimal model, which may be outperformed by a leakage-free alternative. == Leakage modes == Leakage can occur at multiple stages of the machine learning workflow. Broadly, its sources can be divided into two categories: those arising from features and those arising from training examples. === Feature leakage === Feature or column-wise leakage is caused by the inclusion of columns which are one of the following: a duplicate label, a proxy for the label, or the label itself. These features, known as anachronisms, will not be available when the model is used for predictions, and result in leakage if included when the model is trained. For example, including a "MonthlySalary" column when predicting "YearlySalary"; or "MinutesLate" when predicting "IsLate". === Training example leakage === Row-wise leakage is caused by improper sharing of information between rows of data. Types of row-wise leakage include: Premature featurization; leaking from premature featurization before Cross-validation/Train/Test split (must fit MinMax/ngrams/etc on only the train split, then transform the test set) Duplicate rows between train/validation/test (for example, oversampling a dataset to pad its size before splitting; or, different rotations/augmentations of a single image; bootstrap sampling before splitting; or duplicating rows to up sample the minority class) Non-independent and identically distributed random (non-IID) data Time leakage (for example, splitting a time-series dataset randomly instead of newer data in test set using a train/test split or rolling-origin cross-validation) Group leakage—not including a grouping split column (for example, Andrew Ng's group had 100k x-rays of 30k patients, meaning ~3 images per patient. The paper used random splitting instead of ensuring that all images of a patient were in the same split. Hence the model partially memorized the patients instead of learning to recognize pneumonia in chest x-rays.) A 2023 review found data leakage to be "a widespread failure mode in machine-learning (ML)-based science", having affected at least 294 academic publications across 17 disciplines, and causing a potential reproducibility crisis. == Detection == Data leakage in machine learning can be detected through various methods, focusing on performance analysis, feature examination, data auditing, and model behavior analysis. Performance-wise, unusually high accuracy or significant discrepancies between training and test results often indicate leakage. Inconsistent cross-validation outcomes may also signal issues. Feature examination involves scrutinizing feature importance rankings and ensuring temporal integrity in time series data. A thorough audit of the data pipeline is crucial, reviewing pre-processing steps, feature engineering, and data splitting processes. Detecting duplicate entries across dataset splits is also important. For language models, the Min-K% method can detect the presence of data in a pretraining dataset. It presents a sentence suspected to be present in the pretraining dataset, and computes the log-likelihood of each token, then compute the average of the lowest K of these. If this exceeds a threshold, then the sentence is likely present. This method is improved by comparing against a baseline of the mean and variance. Analyzing model behavior can reveal leakage. Models relying heavily on counter-intuitive features or showing unexpected prediction patterns warrant investigation. Performance degradation over time when tested on new data may suggest earlier inflated metrics due to leakage. Advanced techniques include backward feature elimination, where suspicious features are temporarily removed to observe performance changes. Using a separate hold-out dataset for final validation before deployment is advisable.

  Read more →
• 

  NATGRID

  The National Intelligence Grid or NATGRID is an integrated intelligence master database structure for counter-terrorism purposes which connects databases of various core security agencies under the Government of India. It collects and analyses comprehensive patterns procured from 21 different organizations that can be readily accessed by security agencies round the clock. As of September 2025 its CEO is Hirdesh Kumar. NATGRID came into existence after the 2008 Mumbai attacks. The Government of India in July 2016 appointed Ashok Patnaik as the Chief Executive Officer (CEO) of NATGRID. The appointment is being seen as the government's effort to revive the project. Patnaik's appointment was valid till 31 December 2018. As of 2019, NATGRID is headed by an Indian Police Service (IPS) officer Ashish Gupta. The Ministry of Home Affairs on 5 February 2020 announced in Parliament that Project NATGRID with all its required physical infrastructures been completed as of 31 March 2020 and the NATGRID solution went live as of 31 December 2020. == Reason for establishment == The landscape of Terrorism in India and the subsequent response by Law enforcement in India have necessitated a sophisticated data-integration framework, positioning NATGRID as a vital tool for national security agencies. This shift towards Mass surveillance in India is rooted in a broader policy evolution of state monitoring, which is technologically enabled by the India Stack—the foundational digital infrastructure providing the API-based backbone for government service delivery and identity verification. This ecosystem is further bolstered by advanced Signal intelligence capabilities and the implementation of SIM binding, a security protocol that anchors a user’s digital identity to a specific mobile device and verified SIM card to prevent identity fraud and unauthorized access. Collectively, these elements form a 360-degree surveillance and authentication grid designed to preemptively identify threats by synthesizing historical, financial, and real-time communication data across disparate platforms. === Terror attacks in India === The 2008 Mumbai attacks led to the exposure of several weaknesses in India's intelligence gathering and action networks. NATGRID is part of the radical overhaul of the security and intelligence apparatuses of India that was mooted by the then Home Minister P. Chidambaram in 2009. The National Investigation Agency (NIA) and the National Counter Terrorism Centre (NCTC) are two organisations established in the aftermath of the Mumbai attacks of 2008. Before the Mumbai attacks, a Pakistani origin American Lashkar-e-Taiba (LeT) operative David Coleman Headley had visited India several times and done a recce of the places that came under attack on 26/11. Despite having travelled to India several times and having returned to the US through Pakistan or West Asia, his trips failed to raise the suspicion of Indian agencies as they lacked a system that could reveal a pattern in his unusual travel itineraries and trips to the country. It was argued that if they had a system like the NATGRID in place, Headley would have been apprehended well before the attacks. === Need for the integrated intelligence system === During the inauguration of NATGRID campus in Bengaluru, the Minister of Home Affairs, Amit Shah stated that a new national database is in the process of being made which will bring a change in the current ways of functioning of agencies once it's ready also adding that the government has entrusted the task of developing and operating a state-of-the-art and innovative technology system. It is accessible to 11 central agencies in the first phase and in later phases will be made accessible to police of all States and Union Territories and only authorized personnel are allowed access to the platform on a case-to-case basis for investigations into suspected cases of terrorism. NATGRID has a total fund allocation of ₹3,400 crore (US$355 million). d == Legal framework == Relevant legal framework: Digital Personal Data Protection Act, 2023 – The legislative framework governing how digital data is handled. Information Technology Act - Interception Rules, 2002 – The specific regulations under the Information Technology Act that govern these agencies. National Security Act of 1980, evidence-based preventative detention of suspects Right to Information Act, 2005, for obtaining information from the government and used by activists and whistleblowers == Structure and functions == === Multi-agency integrated intelligence database === NATGRID is an intelligence sharing network that collates data from the standalone databases of the various agencies and ministries of the Indian government. It is a counter terrorism measure that collects and collates a host of information from government databases including tax and bank account details, credit/debit card transactions, visa and immigration records and itineraries of rail and air travel. It also has access to the Crime and Criminal Tracking Network and Systems, a database that links crime information, including First Information Reports, across 14,000 police stations in India. This combined data will be made available to 11 central agencies, which are: the Research and Analysis Wing (R&AW), Intelligence Bureau (IB), National Investigation Agency (NIA), Central Bureau of Investigation (CBI), Narcotics Control Bureau (NCB), Financial Intelligence Unit (India) (FIU), Enforcement Directorate (ED), Central Board of Direct Taxes (CBDT), Central Board of Indirect Taxes and Customs (CBIC), Directorate of Revenue Intelligence (DRI) and Directorate General of GST Intelligence. Also as stated by the MHA, NATGRID will have an in-built mechanism for continuous upgradation. In the later phases of NATGRID integration, the central government further plans to integrate 950 additional organizations into it. === Key components and users === ==== Some important backend data feeds to the NATGRID (middleware) ==== National Crime Records Bureau's Crime and Criminal Tracking Network and Systems (CCTNS) national-integrated law-and-order database for the state-level police forces: CCTNS is a mission-mode project under the National e-Governance Plan that interconnects over 15,000 police stations across India. It serves as the primary source for NATGRID to access digitized FIR (First Information Report) data and criminal history records from state-level law enforcement. NSA's National Technical Research Organisation (NTRO) national security-based database feed to NATGRID: NTRO serves as a primary technical data provider to NATGRID, offering specialized intercepts and satellite imagery. While NATGRID functions as a centralized data-integration middleware under the Ministry of Home Affairs, NTRO reports to the National Security Advisor within the Prime Minister's Office. DRDO's NETRA (Network Traffic Analysis) ELINT-based mass surveillance system for monitor internal internet traffic for keywords related to terrorism and criminal activity within Indian borders: Developed by the Centre for Artificial Intelligence and Robotics (CAIR), NETRA is an internet monitoring system capable of scanning traffic for specific trigger words. It provides digital behavioral triggers that NATGRID can cross-reference against structural data like financial or travel records. NETRA is a massive software network used to intercept and analyze internet traffic (emails, social media, blogs) for keywords like "bomb," "attack," or "kill." The intelligence gathered by NETRA regarding suspicious digital patterns or "keyword hits" can be fed into NATGRID. This allows an investigator to see if a person flagged by NETRA also has suspicious travel (from airline databases) or financial records (from bank databases) linked within NATGRID. Department of Telecommunications (DoT's Central Monitoring System (CMS) for lawfully intercepting national and international telecomm data: CMS is the centralized system for lawful interception of all telecommunications (phone calls, SMS, and data) in India, managed by the Department of Telecommunications (DoT). While CMS focuses on the content and metadata of real-time communication, NATGRID focuses on historical/structural data (tax, travel, identity). They represent two halves of a 360-degree surveillance profile: CMS listens to what a suspect says, while NATGRID tracks where they go and what they own. The CMS allows for the lawful interception of telecommunications metadata and content in real-time. In the broader surveillance architecture, CMS provides the "active" communication profile while NATGRID provides the "static" historical profile. Telecom Enforcement Resource and Monitoring (TERM) - Telecomm Regulatory & Verification Node for telecomm KYC: TERM cells verify subscriber identity (KYC) and maintain the integrity of telecom databases. NATGRID relies on these audited records to ensure the accuracy of telephone-to-identity mapping. TERM

  Read more →
• 

  HTTP Strict Transport Security

  HTTP Strict Transport Security (HSTS) is a policy mechanism that helps to protect websites against man-in-the-middle attacks such as protocol downgrade attacks and cookie hijacking. It allows web servers to declare that web browsers (or other complying user agents) should automatically interact with it using only HTTPS connections, which provide Transport Layer Security (TLS/SSL), unlike the insecure HTTP used alone. HSTS is an IETF standards track protocol and is specified in RFC 6797. The HSTS Policy is communicated by the server to the user agent via an HTTP response header field named Strict-Transport-Security. HSTS Policy specifies a period of time during which the user agent should only access the server in a secure fashion. Websites using HSTS often do not accept clear text HTTP, either by rejecting connections over HTTP or systematically redirecting users to HTTPS (though this is not required by the specification). The consequence of this is that a user-agent not capable of doing TLS will not be able to connect to the site. The protection normally only applies after a user has visited the site at least once, relying on the principle of "trust on first use". The way this protection works is that when a user entering or selecting an HTTP (not HTTPS) URL to the site, the client, such as a Web browser, will automatically upgrade to HTTPS without making an HTTP request, thereby preventing any HTTP man-in-the-middle attack from occurring. To counteract this problem, an HSTS preload list maintained by Google Chrome and used by other major web browsers is maintained. If a domain is on this list, the browser skips the initial request and encrypts all communication immediately. Additional domains can be registered at no cost. == Specification history == The HSTS specification was published as RFC 6797 on 19 November 2012 after being approved on 2 October 2012 by the IESG for publication as a Proposed Standard RFC. The authors originally submitted it as an Internet Draft on 17 June 2010. With the conversion to an Internet Draft, the specification name was altered from "Strict Transport Security" (STS) to "HTTP Strict Transport Security", because the specification applies only to HTTP. The HTTP response header field defined in the HSTS specification however remains named "Strict-Transport-Security". The last so-called "community version" of the then-named "STS" specification was published on 18 December 2009, with revisions based on community feedback. The original draft specification by Jeff Hodges from PayPal, Collin Jackson, and Adam Barth was published on 18 September 2009. The HSTS specification is based on original work by Jackson and Barth as described in their paper "ForceHTTPS: Protecting High-Security Web Sites from Network Attacks". Additionally, HSTS is the realization of one facet of an overall vision for improving web security, put forward by Jeff Hodges and Andy Steingruebl in their 2010 paper The Need for Coherent Web Security Policy Framework(s). == HSTS mechanism overview == A server implements an HSTS policy by supplying a header over an HTTPS connection (HSTS headers over HTTP are ignored). For example, a server could send a header such that future requests to the domain for the next year (max-age is specified in seconds; 31,536,000 is equal to one non-leap year) use only HTTPS: Strict-Transport-Security: max-age=31536000. When a web application issues HSTS Policy to user agents, conformant user agents behave as follows: Automatically turn any insecure links referencing the web application into secure links (e.g. http://example.com/some/page/ will be modified to https://example.com/some/page/ before accessing the server). If the security of the connection cannot be ensured (e.g. the server's TLS certificate is not trusted), the user agent must terminate the connection and should not allow the user to access the web application. This helps protect web application users against some passive (eavesdropping) and active network attacks. A man-in-the-middle attacker has a greatly reduced ability to intercept requests and responses between a user and a web application server while the user's browser has HSTS Policy in effect for that web application. == Applicability == The most important security vulnerability that HSTS can fix is SSL-stripping man-in-the-middle attacks, first publicly introduced by Moxie Marlinspike in his 2009 BlackHat Federal talk "New Tricks For Defeating SSL In Practice". The SSL (and TLS) stripping attack works by transparently converting a secure HTTPS connection into a plain HTTP connection. The user can see that the connection is insecure, but crucially there is no way of knowing whether the connection should be secure. At the time of Marlinspike's talk, many websites did not use TLS/SSL, therefore there was no way of knowing (without prior knowledge) whether the use of plain HTTP was due to an attack, or simply because the website had not implemented TLS/SSL. Additionally, no warnings are presented to the user during the downgrade process, making the attack fairly subtle to all but the most vigilant. Marlinspike's sslstrip tool, presented at Black Hat DC 2009, fully automates the attack. HSTS addresses this problem by informing the browser that connections to the site should always use TLS/SSL. The HSTS header can be stripped by the attacker if this is the user's first visit. Google Chrome, Mozilla Firefox, Internet Explorer, and Microsoft Edge attempt to limit this problem by including a "pre-loaded" list of HSTS sites. Unfortunately this solution cannot scale to include all websites on the internet. See limitations, below. HSTS can also help to prevent having one's cookie-based website login credentials stolen by widely available tools such as Firesheep. Because HSTS is time limited, it is sensitive to attacks involving shifting the victim's computer time e.g. using false NTP packets. == Limitations == The initial request remains unprotected from active attacks if it uses an insecure protocol such as plain HTTP or if the URI for the initial request was obtained over an insecure channel. The same applies to the first request after the activity period specified in the advertised HSTS Policy max-age (sites should set a period of several days or months depending on user activity and behavior). === Solutions with preload list === Google Chrome, Mozilla Firefox, and Internet Explorer/Microsoft Edge address this limitation by implementing a "HSTS preloaded list", which is a list that contains known sites supporting HSTS. This list is distributed with the browser so that it uses HTTPS for the initial request to the listed sites as well. As previously mentioned, these pre-loaded lists cannot scale to cover the entire Web. A potential solution might be achieved by using DNS records to declare HSTS Policy, and accessing them securely via DNSSEC, optionally with certificate fingerprints to ensure validity (which requires running a validating resolver to avoid last mile issues). Junade Ali has noted that HSTS is ineffective against the use of false domains; by using DNS-based attacks, it is possible for a man-in-the-middle interceptor to serve traffic from an artificial domain which is not on the HSTS Preload list, this can be made possible by DNS Spoofing Attacks, or simply a domain name that misleadingly resembles the real domain name such as www.example.org instead of www.example.com. Even with an HSTS preloaded list, HSTS cannot prevent advanced attacks against TLS itself, such as the BEAST or CRIME attacks introduced by Juliano Rizzo and Thai Duong. Attacks against TLS itself are orthogonal to HSTS policy enforcement. Neither can it protect against attacks on the server - if someone compromises it, it will happily serve any content over TLS. === Privacy issues === HSTS can be used to near-indelibly tag visiting browsers with recoverable identifying data (supercookies) which can persist in and out of browser "incognito" privacy modes. By creating a web page that makes multiple HTTP requests to selected domains, for example, if twenty browser requests to twenty different domains are used, theoretically over one million visitors can be distinguished (220) due to the resulting requests arriving via HTTP vs. HTTPS; the latter being the previously recorded binary "bits" established earlier via HSTS headers. == Browser support == Chromium and Google Chrome since version 4.0.211.0 Firefox since version 4; with Firefox 17, Mozilla integrates a list of websites supporting HSTS. Opera since version 12 Safari since OS X Mavericks (version 10.9, late 2013) Internet Explorer 11 on Windows 8.1 and Windows 7 with KB3058515 installed (Released as a Windows Update in June 2015) Microsoft Edge and Internet Explorer 11 on Windows 10 BlackBerry 10 Browser and WebView since BlackBerry OS 10.3.3. == Deployment best practices == Depending on the actual deployment there are certain threats (e.g. cookie injection attacks) t

  Read more →