AI Avatar Kiosk

AI Avatar Kiosk — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

Isotropic position

In the fields of machine learning, the theory of computation, and random matrix theory, a probability distribution over vectors is said to be in isotropic position if its covariance matrix is proportional to the identity matrix. == Formal definitions == Let D {\textstyle D} be a distribution over vectors in the vector space R n {\textstyle \mathbb {R} ^{n}} . Then D {\textstyle D} is in isotropic position if, for vector v {\textstyle v} sampled from the distribution, E v v T = I d . {\displaystyle \mathbb {E} \,vv^{\mathsf {T}}=\mathrm {Id} .} A set of vectors is said to be in isotropic position if the uniform distribution over that set is in isotropic position. In particular, every orthonormal set of vectors is isotropic. As a related definition, a convex body K {\textstyle K} in R n {\textstyle \mathbb {R} ^{n}} is called isotropic if it has volume | K | = 1 {\textstyle |K|=1} , center of mass at the origin, and there is a constant α > 0 {\textstyle \alpha >0} such that ∫ K ⟨ x , y ⟩ 2 d x = α 2 | y | 2 , {\displaystyle \int _{K}\langle x,y\rangle ^{2}dx=\alpha ^{2}|y|^{2},} for all vectors y {\textstyle y} in R n {\textstyle \mathbb {R} ^{n}} ; here | ⋅ | {\textstyle |\cdot |} stands for the standard Euclidean norm.
Read more →
Semi-Automatic Ground Environment

The Semi-Automated Ground Environment (SAGE) was a system of large computers and associated networking equipment that coordinated data from many radar sites and processed it to produce a single unified image of the airspace over a wide area. SAGE directed and controlled the NORAD response to a possible Soviet air attack, operating in this role from the late 1950s into the 1980s. The processing power behind SAGE was supplied by the largest discrete component-based computer ever built, the AN/FSQ-7, manufactured by IBM. Each SAGE Direction Center (DC) housed an FSQ-7 which occupied an entire floor, approximately 22,000 square feet (2,000 m2) not including supporting equipment. The FSQ-7 was actually two computers, "A" side and "B" side. Computer processing was switched from "A" side to "B" side on a regular basis, allowing maintenance on the unused side. Information was fed to the DCs from a network of radar stations as well as readiness information from various defense sites. The computers, based on the raw radar data, developed "tracks" for the reported targets, and automatically calculated which defenses were within range. Operators used light guns to select targets on-screen for further information, select one of the available defenses, and issue commands to attack. These commands would then be automatically sent to the defense site via teleprinter. Connecting the various sites was an enormous network of telephones, modems and teleprinters. Later additions to the system allowed SAGE's tracking data to be sent directly to CIM-10 Bomarc missiles and some of the US Air Force's interceptor aircraft in-flight, directly updating their autopilots to maintain an intercept course without operator intervention. Each DC also forwarded data to a Combat Center (CC) for "supervision of the several sectors within the division" ("each combat center [had] the capability to coordinate defense for the whole nation"). SAGE became operational in the late 1950s and early 1960s at an estimated total cost between 8 and 12 billion dollars, four times the cost of the Manhattan Project. Throughout its development, there were continual concerns about its real ability to deal with large attacks, and the Operation Sky Shield tests showed that only about one-fourth of enemy bombers would have been intercepted. Nevertheless, SAGE was the backbone of NORAD's air defense system into the 1980s, by which time the tube-based FSQ-7s were increasingly costly to maintain and completely outdated. Today the same command and control task is carried out by microcomputers, based on the same basic underlying data. == Background == === Earlier systems === Just prior to World War II, Royal Air Force (RAF) tests with the new Chain Home (CH) radars had demonstrated that relaying information to the fighter aircraft directly from the radar sites was not feasible. The radars determined the map coordinates of the enemy, but could generally not see the fighters at the same time. This meant the fighters had to be able to determine where to fly to perform an interception but were often unaware of their own exact location and unable to calculate an interception while also flying their aircraft. The solution was to send all of the radar information to a central control station where operators collated the reports into single tracks, and then reported these tracks to the airbases, or sectors. The sectors used additional systems to track their own aircraft, plotting both on a single large map. Operators viewing the map could then see what direction their fighters would have to fly to approach their targets and relay that simply by telling them to fly along a certain heading or vector. This Dowding system was the first ground-controlled interception (GCI) system of large scale, covering the entirety of the UK. It proved enormously successful during the Battle of Britain, and is credited as being a key part of the RAF's success. The system was slow, often providing information that was up to five minutes out of date. Against propeller driven bombers flying at perhaps 225 miles per hour (362 km/h) this was not a serious concern, but it was clear the system would be of little use against jet-powered bombers flying at perhaps 600 miles per hour (970 km/h). The system was extremely expensive in manpower terms, requiring hundreds of telephone operators, plotters and trackers in addition to the radar operators. This was a serious drain on manpower, making it difficult to expand the network. The idea of using a computer to handle the task of taking reports and developing tracks had been explored beginning late in the war. By 1944, analog computers had been installed at the CH stations to automatically convert radar readings into map locations, eliminating two people. Meanwhile, the Royal Navy began experimenting with the Comprehensive Display System (CDS), another analog computer that took X and Y locations from a map and automatically generated tracks from repeated inputs. Similar systems began development with the Royal Canadian Navy, DATAR, and the US Navy, the Naval Tactical Data System (NTDS). A similar system was also specified for the Nike SAM project, specifically referring to a US version of CDS, coordinating the defense over a battle area so that multiple batteries did not fire on a single target. All of these systems were relatively small in geographic scale, generally tracking within a city-sized area. === Valley Committee === When the Soviet Union tested its first atomic bomb in August 1949, the topic of air defense of the US became important for the first time. A study group, the "Air Defense Systems Engineering Committee", was set up under the direction of Dr. George Valley to consider the problem and is known to history as the "Valley Committee". Their December report noted a key problem in air defense using ground-based radars. A bomber approaching a radar station would detect the signals from the radar long before the reflection off the bomber was strong enough to be detected by the station. The committee suggested that when this occurred, the bomber would descend to low altitude, thereby greatly limiting the radar horizon, allowing the bomber to fly past the station undetected. Although flying at low altitude greatly increased fuel consumption, the team calculated that the bomber would only need to do this for about 10% of its flight, making the fuel penalty acceptable. The only solution to this problem was to build a huge number of stations with overlapping coverage. At that point the problem became one of managing the information. Manual plotting was ruled out as too slow, and a computerized solution was the only possibility. To handle this task, the computer would need to be fed information directly, eliminating any manual translation by phone operators, and it would have to be able to analyze that information and automatically develop tracks. A system tasked with defending cities against the predicted future Soviet bomber fleet would have to be dramatically more powerful than the models used in the NTDS or DATAR. The Committee then had to consider whether or not such a computer was possible. The Valley Committee was introduced to Jerome Wiesner, associate director of the Research Laboratory of Electronics at MIT. Wiesner noted that the Servomechanisms Laboratory had already begun development of a machine that might be fast enough. This was the Whirlwind I, originally developed for the Office of Naval Research as a general purpose flight simulator that could simulate any current or future aircraft by changing its software. Wiesner introduced the Valley Committee to Whirlwind's project lead, Jay Forrester, who convinced him that Whirlwind was sufficiently capable. In September 1950, an early microwave early-warning radar system at Hanscom Field was connected to Whirlwind using a custom interface developed by Forrester's team. An aircraft was flown past the site, and the system digitized the radar information and successfully sent it to Whirlwind. With this demonstration, the technical concept was proven. Forrester was invited to join the committee. === Project Charles === With this successful demonstration, Louis Ridenour, chief scientist of the Air Force, wrote a memo stating "It is now apparent that the experimental work necessary to develop, test, and evaluate the systems proposals made by ADSEC will require a substantial amount of laboratory and field effort." Ridenour approached MIT President James Killian with the aim of beginning a development lab similar to the war-era Radiation Laboratory that made enormous progress in radar technology. Killian was initially uninterested, desiring to return the school to its peacetime civilian charter. Ridenour eventually convinced Killian the idea was sound by describing the way the lab would lead to the development of a local electronics industry based on the needs of the lab and the students who would leave the lab to start their
Read more →
Initialization vector

In cryptography, an initialization vector (IV) or starting variable is an input to a cryptographic primitive being used to provide the initial state. The IV is typically required to be random or pseudorandom, but sometimes an IV only needs to be unpredictable or unique. Randomization is crucial for some encryption schemes to achieve semantic security, a property whereby repeated usage of the scheme under the same key does not allow an attacker to infer relationships between (potentially similar) segments of the encrypted message. For block ciphers, the use of an IV is described by the modes of operation. Some cryptographic primitives require the IV only to be non-repeating, and the required randomness is derived internally. In this case, the IV is commonly called a nonce (a number used only once), and the primitives (e.g. CBC) are considered stateful rather than randomized. This is because an IV need not be explicitly forwarded to a recipient but may be derived from a common state updated at both sender and receiver side. (In practice, a short nonce is still transmitted along with the message to consider message loss.) An example of stateful encryption schemes is the counter mode of operation, which has a sequence number for a nonce. The IV size depends on the cryptographic primitive used; for block ciphers it is generally the cipher's block-size. In encryption schemes, the unpredictable part of the IV has at best the same size as the key to compensate for time/memory/data tradeoff attacks. When the IV is chosen at random, the probability of collisions due to the birthday problem must be taken into account. Traditional stream ciphers such as RC4 do not support an explicit IV as input, and a custom solution for incorporating an IV into the cipher's key or internal state is needed. Some designs realized in practice are known to be insecure; the WEP protocol is a notable example, and is prone to related-IV attacks. == Motivation == A block cipher is one of the most basic primitives in cryptography, and frequently used for data encryption. However, by itself, it can only be used to encode a data block of a predefined size, called the block size. For example, a single invocation of the AES algorithm transforms a 128-bit plaintext block into a ciphertext block of 128 bits in size. The key, which is given as one input to the cipher, defines the mapping between plaintext and ciphertext. If data of arbitrary length is to be encrypted, a simple strategy is to split the data into blocks each matching the cipher's block size, and encrypt each block separately using the same key. This method is not secure as equal plaintext blocks get transformed into equal ciphertexts, and a third party observing the encrypted data may easily determine its content even when not knowing the encryption key. To hide patterns in encrypted data while avoiding the re-issuing of a new key after each block cipher invocation, a method is needed to randomize the input data. In 1980, the NIST published a national standard document designated Federal Information Processing Standard (FIPS) PUB 81, which specified four so-called block cipher modes of operation, each describing a different solution for encrypting a set of input blocks. The first mode implements the simple strategy described above, and was specified as the electronic codebook (ECB) mode. In contrast, each of the other modes describe a process where ciphertext from one block encryption step gets intermixed with the data from the next encryption step. To initiate this process, an additional input value is required to be mixed with the first block, and which is referred to as an initialization vector. For example, the cipher-block chaining (CBC) mode requires an unpredictable value, of size equal to the cipher's block size, as additional input. This unpredictable value is added to the first plaintext block before subsequent encryption. In turn, the ciphertext produced in the first encryption step is added to the second plaintext block, and so on. The ultimate goal for encryption schemes is to provide semantic security: by this property, it is practically impossible for an attacker to draw any knowledge from observed ciphertext. It can be shown that each of the three additional modes specified by the NIST are semantically secure under so-called chosen-plaintext attacks. == Properties == Properties of an IV depend on the cryptographic scheme used. A basic requirement is uniqueness, which means that no IV may be reused under the same key. For block ciphers, repeated IV values devolve the encryption scheme into electronic codebook mode: equal IV and equal plaintext result in equal ciphertext. In stream cipher encryption uniqueness is crucially important as plaintext may be trivially recovered otherwise. Example: Stream ciphers encrypt plaintext P to ciphertext C by deriving a key stream K from a given key and IV and computing C as C = P xor K. Assume that an attacker has observed two messages C1 and C2 both encrypted with the same key and IV. Then knowledge of either P1 or P2 reveals the other plaintext since C1 xor C2 = (P1 xor K) xor (P2 xor K) = P1 xor P2. Many schemes require the IV to be unpredictable by an adversary. This is effected by selecting the IV at random or pseudo-randomly. In such schemes, the chance of a duplicate IV is negligible, but the effect of the birthday problem must be considered. As for the uniqueness requirement, a predictable IV may allow recovery of (partial) plaintext. Example: Consider a scenario where a legitimate party called Alice encrypts messages using the cipher-block chaining mode. Consider further that there is an adversary called Eve that can observe these encryptions and is able to forward plaintext messages to Alice for encryption (in other words, Eve is capable of a chosen-plaintext attack). Now assume that Alice has sent a message consisting of an initialization vector IV1 and starting with a ciphertext block CAlice. Let further PAlice denote the first plaintext block of Alice's message, let E denote encryption, and let PEve be Eve's guess for the first plaintext block. Now, if Eve can determine the initialization vector IV2 of the next message she will be able to test her guess by forwarding a plaintext message to Alice starting with (IV2 xor IV1 xor PEve); if her guess was correct this plaintext block will get encrypted to CAlice by Alice. This is because of the following simple observation: CAlice = E(IV1 xor PAlice) = E(IV2 xor (IV2 xor IV1 xor PAlice)). Depending on whether the IV for a cryptographic scheme must be random or only unique the scheme is either called randomized or stateful. While randomized schemes always require the IV chosen by a sender to be forwarded to receivers, stateful schemes allow sender and receiver to share a common IV state, which is updated in a predefined way at both sides. == Block ciphers == Block cipher processing of data is usually described as a mode of operation. Modes are primarily defined for encryption as well as authentication, though newer designs exist that combine both security solutions in so-called authenticated encryption modes. While encryption and authenticated encryption modes usually take an IV matching the cipher's block size, authentication modes are commonly realized as deterministic algorithms, and the IV is set to zero or some other fixed value. == Stream ciphers == In stream ciphers, IVs are loaded into the keyed internal secret state of the cipher, after which a number of cipher rounds are executed prior to releasing the first bit of output. For performance reasons, designers of stream ciphers try to keep that number of rounds as small as possible, but because determining the minimal secure number of rounds for stream ciphers is not a trivial task, and considering other issues such as entropy loss, unique to each cipher construction, related-IVs and other IV-related attacks are a known security issue for stream ciphers, which makes IV loading in stream ciphers a serious concern and a subject of ongoing research. == WEP IV == The 802.11 encryption algorithm called WEP (short for Wired Equivalent Privacy) used a short, 24-bit IV, leading to reused IVs with the same key, which led to it being easily cracked. Packet injection allowed for WEP to be cracked in times as short as several seconds. This ultimately led to the deprecation of WEP. == SSL 2.0 IV == In cipher-block chaining mode (CBC mode), the IV need not be secret, but must be unpredictable (In particular, for any given plaintext, it must not be possible to predict the IV that will be associated to the plaintext in advance of the generation of the IV.) at encryption time. Additionally for the output feedback mode (OFB mode), the IV must be unique. In particular, the (previously) common practice of re-using the last ciphertext block of a message as the IV for the next message is insecure (for example, this method was used by SSL 2.0). If an attacker knows
Read more →
WYSIWYS

In cryptography, What You See Is What You Sign (WYSIWYS) is a property of digital signature systems that ensures the semantic content of signed messages can not be changed, either by accident or intent. == Mechanism of WYSIWYS == When digitally signing a document, the integrity of the signature relies not just on the soundness of the digital signature algorithms that are used, but also on the security of the computing platform used to sign the document. The WYSIWYS property of digital signature systems aims to tackle this problem by defining a desirable property that the visual representation of a digital document should be consistent across computing systems, particularly at the points of digital signature and digital signature verification. It is relatively easy to change the interpretation of a digital document by implementing changes on the computer system where the document is being processed, and the greater the semantic distance, the easier it gets. From a semantic perspective this creates uncertainty about what exactly has been signed. WYSIWYS is a property of a digital signature system that ensures that the semantic interpretation of a digitally signed message cannot be changed, either by accident or by intent. This property also ensures that a digital document to be signed can not contain hidden semantic content that can be revealed after the signature has been applied. Though a WYSIWYS implementation is only as secure as the computing platform it is running on, various methods have been proposed to make WYSIWYS more robust. The term WYSIWYS was coined by Peter Landrock and Torben Pedersen to describe some of the principles in delivering secure and legally binding digital signatures for Pan-European projects.
Read more →
Image stitching

Image stitching or photo stitching is the process of combining multiple photographic images with overlapping fields of view to produce a segmented panorama or high-resolution image. Commonly performed through the use of computer software, most approaches to image stitching require nearly exact overlaps between images and identical exposures to produce seamless results, although some stitching algorithms actually benefit from differently exposed images by doing high-dynamic-range imaging in regions of overlap. Some digital cameras can stitch their photos internally. == Applications == Image stitching is widely used in modern applications, such as the following: Document mosaicing Image stabilization feature in camcorders that use frame-rate image alignment High-resolution image mosaics in digital maps and satellite imagery Medical imaging Multiple-image super-resolution imaging Video stitching Object insertion == Process == The image stitching process can be divided into three main components: image registration, calibration, and blending. === Image stitching algorithms === In order to estimate image alignment, algorithms are needed to determine the appropriate mathematical model relating pixel coordinates in one image to pixel coordinates in another. Algorithms that combine direct pixel-to-pixel comparisons with gradient descent (and other optimization techniques) can be used to estimate these parameters. Distinctive features can be found in each image and then efficiently matched to rapidly establish correspondences between pairs of images. When multiple images exist in a panorama, techniques have been developed to compute a globally consistent set of alignments and to efficiently discover which images overlap one another. A final compositing surface onto which to warp or projectively transform and place all of the aligned images is needed, as are algorithms to seamlessly blend the overlapping images, even in the presence of parallax, lens distortion, scene motion, and exposure differences. === Image stitching issues === Since the illumination in two views cannot be guaranteed to be identical, stitching two images could create a visible seam. Other reasons for seams could be the background changing between two images for the same continuous foreground. Other major issues to deal with are the presence of parallax, lens distortion, scene motion, and exposure differences. In a non-ideal real-life case, the intensity varies across the whole scene, and so does the contrast and intensity across frames. Additionally, the aspect ratio of a panorama image needs to be taken into account to create a visually pleasing composite. For panoramic stitching, the ideal set of images will have a reasonable amount of overlap (at least 15–30%) to overcome lens distortion and have enough detectable features. The set of images will have consistent exposure between frames to minimize the probability of seams occurring. === Keypoint detection === Feature detection is necessary to automatically find correspondences between images. Robust correspondences are required in order to estimate the necessary transformation to align an image with the image it is being composited on. Corners, blobs, Harris corners, and differences of Gaussians of Harris corners are good features since they are repeatable and distinct. One of the first operators for interest point detection was developed by Hans Moravec in 1977 for his research involving the automatic navigation of a robot through a clustered environment. Moravec also defined the concept of "points of interest" in an image and concluded these interest points could be used to find matching regions in different images. The Moravec operator is considered to be a corner detector because it defines interest points as points where there are large intensity variations in all directions. This often is the case at corners. However, Moravec was not specifically interested in finding corners, just distinct regions in an image that could be used to register consecutive image frames. Harris and Stephens improved upon Moravec's corner detector by considering the differential of the corner score with respect to direction directly. They needed it as a processing step to build interpretations of a robot's environment based on image sequences. Like Moravec, they needed a method to match corresponding points in consecutive image frames, but were interested in tracking both corners and edges between frames. SIFT and SURF are recent key-point or interest point detector algorithms but a point to note is that SURF is patented and its commercial usage restricted. Once a feature has been detected, a descriptor method like SIFT descriptor can be applied to later match them. === Registration === Image registration involves matching features in a set of images or using direct alignment methods to search for image alignments that minimize the sum of absolute differences between overlapping pixels. When using direct alignment methods one might first calibrate one's images to get better results. Additionally, users may input a rough model of the panorama to help the feature matching stage, so that e.g. only neighboring images are searched for matching features. Since there are smaller group of features for matching, the result of the search is more accurate and execution of the comparison is faster. To estimate a robust model from the data, a common method used is known as RANSAC. The name RANSAC is an abbreviation for "RANdom SAmple Consensus". It is an iterative method for robust parameter estimation to fit mathematical models from sets of observed data points which may contain outliers. The algorithm is non-deterministic in the sense that it produces a reasonable result only with a certain probability, with this probability increasing as more iterations are performed. It being a probabilistic method means that different results will be obtained for every time the algorithm is run. The RANSAC algorithm has found many applications in computer vision, including the simultaneous solving of the correspondence problem and the estimation of the fundamental matrix related to a pair of stereo cameras. The basic assumption of the method is that the data consists of "inliers", i.e., data whose distribution can be explained by some mathematical model, and "outliers" which are data that do not fit the model. Outliers are considered points which come from noise, erroneous measurements, or simply incorrect data. For the problem of homography estimation, RANSAC works by trying to fit several models using some of the point pairs and then checking if the models were able to relate most of the points. The best model – the homography, which produces the highest number of correct matches – is then chosen as the answer for the problem; thus, if the ratio of number of outliers to data points is very low, the RANSAC outputs a decent model fitting the data. === Calibration === Image calibration aims to minimize differences between an ideal lens models and the camera-lens combination that was used, optical defects such as distortions, exposure differences between images, vignetting, camera response and chromatic aberrations. If feature detection methods were used to register images and absolute positions of the features were recorded and saved, stitching software may use the data for geometric optimization of the images in addition to placing the images on the panosphere. Panotools and its various derivative programs use this method. ==== Alignment ==== Alignment may be necessary to transform an image to match the view point of the image it is being composited with. Alignment, in simple terms, is a change in the coordinates system so that it adopts a new coordinate system which outputs image matching the required viewpoint. The types of transformations an image may go through are pure translation, pure rotation, a similarity transform which includes translation, rotation and scaling of the image which needs to be transformed, Affine or projective transform. Projective transformation is the farthest an image can transform (in the set of two dimensional planar transformations), where only visible features that are preserved in the transformed image are straight lines whereas parallelism is maintained in an affine transform. Projective transformation can be mathematically described as x ′ = H ⋅ x , {\displaystyle x'=H\cdot x,} where x {\displaystyle x} is points in the old coordinate system, x ′ {\displaystyle x'} is the corresponding points in the transformed image and H {\displaystyle H} is the homography matrix. Expressing the points x {\displaystyle x} and x ′ {\displaystyle x'} using the camera intrinsics ( K {\displaystyle K} and K ′ {\displaystyle K'} ) and its rotation and translation [ R t ] {\displaystyle [R\,t]} to the real-world coordinates X {\displaystyle X} and < m a t h > x {\displaystyle $x} and x ′ {\displaystyle x'} '$ , we get Using the abo
Read more →
Cloud Data Management Interface

ISO/IEC 17826 Information technology — Cloud Data Management Interface (CDMI) Version 2.0.0 is an international standard that specifies a protocol for self-provisioning, administering and managing access to data stored in cloud storage, object storage, storage area network and network attached storage systems. The CDMI standard is developed and maintained by the Storage Networking Industry Association, who makes a publicly accessible version of the specification available. CDMI defines new resource representations to enable standardized management of any URI-accessible data, and defines RESTful HTTP operations using these representations to discover the capabilities of the storage system, discover stored data, access and update management metadata, specify data storage protocols (such as iSCSI and NFS) through which the stored data is accessed, and provide cross-system and cross-cloud import and export in order to enable data portability. Management functions enabled by CDMI include managing data ownership, identity mapping, access controls, user-specified metadata, and to declaratively specify desired data protection, data retention, constraints on geographic placement, desired quality of service, data versioning and security requirements. CDMI also defines utility services to facilitate data management, such the ability to query data matching specific criteria, and includes extensions to perform bulk updates using CDMI Jobs. == Capabilities == Compliant implementations must provide access to a set of configuration parameters known as capabilities. These are either boolean values that represent whether or not a system supports things such as queues, export via other protocols, path-based storage and so on, or numeric values expressing system limits, such as how much metadata may be placed on an object. As a minimal compliant implementation can be quite small, with few features, clients need to check the cloud storage system for a capability before attempting to use the functionality it represents. Resource allocation assignments limited to the data management interface protocols must possess access bypass capabilities which extend beyond the layered framework. This integral function is vital to the prevention of transport layer session hijacking by unauthorized entities which may circumvent standard interfacing security parameters. == Containers == A CDMI client may access objects, including containers, by either name or object id (OID), assuming the CDMI server supports both methods. When storing objects by name, it is natural to use nested named containers; the resulting structure corresponds exactly to a traditional filesystem directory structure. == Objects == Objects are similar to files in a traditional file system, but are enhanced with an increased amount and capacity for metadata. As with containers, they may be accessed by either name or OID. When accessed by name, clients use URLs that contain the full pathname of objects to create, read, update and delete them. When accessed by OID, the URL specifies an OID string in the cdmi-objectid container; this container presents a flat name space conformant with standard object storage system semantics. Subject to system limits, objects may be of any size or type and have arbitrary user-supplied metadata attached to them. Systems that support query allow arbitrary queries to be run against the metadata. == Domains, Users and Groups == CDMI supports the concept of a domain, similar in concept to a domain in the Windows Active Directory model. Users and groups created in a domain share a common administrative database and are known to each other on a "first name" basis, i.e. without reference to any other domain or system. Domains also function as containers for usage and billing summary data. == Access Control == CDMI exactly follows the ACL and ACE model used for file authorization operations by NFSv4. This makes it also compatible with Microsoft Windows systems. == Metadata == CDMI draws much of its metadata model from the XAM specification. Objects and containers have "storage system metadata", "data system metadata" and arbitrary user specified metadata, in addition to the metadata maintained by an ordinary filesystem (atime etc.). == Queries == CDMI specifies a way for systems to support arbitrary queries against CDMI containers, with a rich set of comparison operators, including support for regular expressions. == Queues == CDMI supports the concept of persistent FIFO (first-in, first-out) queues. These are useful for job scheduling, order processing and other tasks in which lists of things must be processed in order. == Compliance == Both retention intervals and retention holds are supported by CDMI. A retention interval consists of a start time and a retention period. During this time interval, objects are preserved as immutable and may not be deleted. A retention hold is usually placed on an object because of judicial action and has the same effect: objects may not be changed nor deleted until all holds placed on them are removed. == Billing == Summary information suitable for billing clients for on-demand services can be obtained by authorized users from systems that support it. == Serialization == Serialization of objects and containers allows export of all data and metadata on a system and importation of that data into another cloud system. == Foreign protocols == CDMI supports export of containers as NFS or CIFS shares. Clients that mount these shares see the container hierarchy as an ordinary filesystem directory hierarchy, and the objects in the containers as normal files. Metadata outside of ordinary filesystem metadata may or may not be exposed. Provisioning of iSCSI LUNs is also supported. == Client SDKs == CDMI Reference Implementation Droplet libcdmi-java libcdmi-python .NET SDK
Read more →
Visual networking

Visual networking refers to an emerging class of user applications that combine digital video and social networking capabilities. It is based upon the premise that visual literacy, "the ability to interpret, negotiate and make meaning from information presented in the form of a moving image", is a powerful force in how humans communicate, entertain and learn. The duality of visual networking—subsuming entertainment and communications, professional and personal content, video and other digital media, data networks and social networks to create immersive experiences, when, where and how the user wants it. These applications have changed video content from long-form movies and broadcast television programming to a database of segments or "clips", and social network annotations. And the generation and distribution of content takes on a new dimension with Web 2.0 applications—participatory social-networks or communities that facilitate interactive creativity, collaboration and sharing between users. == History == The rise of visual networking is relatively recent phenomenon driven by the emergence of social networking capabilities and the ability to deliver interactive video over a broadband network. It is a natural evolution of the current social networking phenomena whereby social networking annotations are layered over broadband video to create highly interactive and immersive experiences between individuals and their content. Until early 2005 this was not considered viable due to the lack of web and broadband infrastructure designed to support the transmission of web video and the still nascent stage of social networks like MySpace and Facebook. The introduction of YouTube in February 2005 marked the first significant combination of broadband video and social network systems designed to allow users to share, rate and tag user generated and premium content. From 2006 to 2008 this trend continued to gain steam as individuals and businesses pursued new combinations of video and social networking across a wide range of entertainment, communication and learning applications. == Broadband video takes off == Video has largely been defined by its use as an entertainment medium. Since the commercial availability of the television in the late '30s video has become the dominant entertainment medium far eclipsing audio and text based entertainment both in terms of time and dollars spent. Within the past decade, video use has rapidly evolved across a broader range of devices, multiple locations and user applications. The popularization of the long-tail and user-generated video has further challenged people's ideas of what's possible with video. A key advantage of video relative to other media is its superior ability to communicate ideas and emotions economically. If a picture is worth a thousand words, then a video may be worth a thousand pictures. Video by its very nature is highly experiential, making communications more compelling, informative and memorable. == Social networking meets video == At the core of visual networking is the concept that people can participate in communities of content and communities of interest. A community of interest is defined as a community of people who share a common interest or passion. These people exchange ideas and thoughts about the given passion, but may know (or care) little about each other outside of this area. Participation in a community of interest can be compelling, entertaining and create a ‘sticky’ community where people return frequently and remain for extended periods. The unparalleled potential of the Internet to promote such connections is only now being fully recognized and exploited, through Web-based groups established for that purpose. Based on the six degrees of separation concept (the idea that any two people on the planet could make contact through a chain of no more than five intermediaries), social networking establishes interconnected Internet communities (sometimes known as personal networks) that help people make contacts that would be good for them to know, but that they would be unlikely to have met otherwise. == Transition from search to discovery == The phrase The Long Tail was, according to Chris Anderson, first coined by himself in October 2004. Anderson argued that products that are in low demand or have low sales volume can collectively make up a market share that rivals or exceeds the relatively few current bestsellers and blockbusters, if the store or distribution channel is large enough. The Long Tail also has implications for the producers of content; especially those whose products could not—for economic reasons—find a place in pre-Internet information distribution channels controlled by book publishers, record companies, movie studios, and television networks. Looked at from the producers' side, the Long Tail has made possible a flowering of creativity across all fields of human endeavor. One example of this is YouTube, where thousands of diverse videos—whose content, production value or lack of popularity make them inappropriate for traditional television—are easily accessible to a wide range of viewers. The benefit to the consumer is that they know have an almost infinite choice of content to select from able to create their own specific channels based upon their unique needs. A potential negative side effect of the long tail is the rapidly growing inventory of text, audio and video content. The storage and distribution systems of the past restricted the number of songs, video, and books making it easier to search for what was relevant to the individual. As the long-tail has grown, more and more relevant and irrelevant content passes an individual by without their knowledge. This is especially true for video because unlike text-based files which can searched and indexed for easy finding, video typically has only its title as a clue to what's in it. This lack of comprehensive meta-data has limited the applicability of traditional search models. Augmenting traditional search has been the emergence of content based discovery tools that make people aware of relevant content based upon their participation in communities of interest and/or communities of content. The idea is that users may or may not start out searching for something, but they soon begin reacting to things they find, exploring links on pages they stumble upon and taking cues from fellow surfers about where to go. Instead of the old, passive, lean-back style of watching video, viewers are actively seeking content through discovery. People interact with each other, posting comments on what they just saw. Many sites now allow people to vote on videos, ranking and rating them. Ranking is the result of one of a number of algorithms that measure how many people have watched something or how many sites link to it. == Early examples == YouTube is the best early example of a visual networking experience. YouTube is a video sharing website where users can upload, view and share video clips. Unregistered users can watch most videos on the site, while registered users are permitted to upload an unlimited number of videos. Few statistics are publicly available regarding the number of videos on YouTube. However, in July 2006, the company revealed that more than 100 million videos were being watched every day, and 2.5 billion videos were watched in June 2006. 50,000 videos were being added per day in May 2006, and this increased to 65,000 by July. In January 2008 alone, nearly 79 million users watched over 3 billion videos on YouTube. Telepresence refers to a set of technologies which allow a person to feel as if they were present, to give the appearance that they were present, or to have an effect, at a location other than their true location. Telepresence requires that the senses of the user, or users, are provided with such stimuli as to give the feeling of being in that other location. Additionally, the user(s) may be given the ability to affect the remote location. In this case, the user's position, movements, actions, voice, etc. may be sensed, transmitted and duplicated in the remote location to bring about this effect. Therefore, information may be traveling in both directions between the user and the remote location. Critical the creating an in-person experience is the presence of high-definition video perfectly synchronized with stereophonic sound. A minimum system usually includes visual feedback. Ideally, the entire field of view of the user is filled with a view of the remote location, and the viewpoint corresponds to the movement and orientation of the user's head. In this way, it differs from television or cinema, where the viewpoint is out of the control of the viewer. == Other applications == While still in its infancy, visual networking applications are beginning to emerge that span both consumer and business markets. === Mobile video === Proliferation of multi-function mobile devices, particularl
Read more →
Data set (IBM mainframe)

In the context of IBM mainframe computers in the IBM System/360 line and its successors, a data set (IBM preferred) or dataset is a computer file having a record organization. Use of this term began with, e.g., DOS/360 and OS/360, and is still used by their successors, including the current VSE and z/OS. Documentation for these systems historically preferred this term rather than file. A data set is typically stored on a direct access storage device (DASD) or magnetic tape, however unit record devices, such as punch card readers, card punches, line printers and page printers can provide input/output (I/O) for a data set (file). Data sets are not unstructured streams of bytes, but rather are organized in various logical record and block structures determined by the DSORG (data set organization), RECFM (record format), and other parameters. These parameters are specified at the time of the data set allocation (creation), for example with Job Control Language DD statements. Within a running program they are stored in the Data Control Block (DCB) or Access Control Block (ACB), which are data structures used to access data sets using access methods. Records in a data set may be fixed, variable, or “undefined” length. == Data set organization == For OS/360, the DCB's DSORG parameter specifies how the data set is organized. It may be CQ Queued Telecommunications Access Method (QTAM) in Message Control Program (MCP) CX Communications line group DA Basic Direct Access Method (BDAM) GS Graphics device for Graphics Access Method(GAM) IS Indexed Sequential Access Method (ISAM) MQ QTAM message queue in application PO Partitioned Organization PS Physical Sequential among others. Data sets on tape may only be DSORG=PS. The choice of organization depends on how the data is to be accessed, and in particular, how it is to be updated. Programmers utilize various access methods (such as QSAM or VSAM) in programs for reading and writing data sets. Access method depends on the given data set organization. == Record format (RECFM) == Regardless of organization, the physical structure of each record is essentially the same, and is uniform throughout the data set. This is specified in the DCB RECFM parameter. RECFM=F means that the records are of fixed length, specified via the LRECL parameter. RECFM=V specifies a variable-length record. V records when stored on media are prefixed by a Record Descriptor Word (RDW) containing the integer length of the record in bytes and flag bits. With RECFM=FB and RECFM=VB, multiple logical records are grouped together into a single physical block on tape or DASD. FB and VB are fixed-blocked, and variable-blocked, respectively. RECFM=U (undefined) is also variable length, but the length of the record is determined by the length of the block rather than by a control field. The BLKSIZE parameter specifies the maximum length of the block. RECFM=FBS could be also specified, meaning fixed-blocked standard, meaning all the blocks except the last one were required to be in full BLKSIZE length. RECFM=VBS, or variable-blocked spanned, means a logical record could be spanned across two or more blocks, with flags in the RDW indicating whether a record segment is continued into the next block and/or was continued from the previous one. This mechanism eliminates the need for using any "delimiter" byte value to separate records. Thus data can be of any type, including binary integers, floating-point, or characters, without introducing a false end-of-record condition. The data set is an abstraction of a collection of records, in contrast to files as unstructured streams of bytes. == Partitioned data set == A partitioned data set (PDS) is a data set containing multiple members, each of which holds a separate sub-data set, similar to a directory in other types of file systems. This type of data set is often used to hold load modules (old format bound executable programs), source program libraries (especially Assembler macro definitions), ISPF screen definitions, and Job Control Language. A PDS may be compared to a Zip file or COM Structured Storage. A Partitioned Data Set can only be allocated on a single volume and have a maximum size of 65,535 tracks. Besides members, a PDS contains also a directory. Each member can be accessed indirectly via the directory structure. Once a member is located, the data stored in that member are handled in the same manner as a PS (sequential) data set. Whenever a member is deleted, the space it occupied is unusable for storing other data. Likewise, if a member is re-written, it is stored in a new spot at the back of the PDS and leaves wasted “dead” space in the middle. The only way to recover “dead” space is to perform file compression. Compression, which is done using the IEBCOPY utility, moves all members to the front of the data space and leaves free usable space at the back. (Note that in modern parlance, this kind of operation might be called defragmentation or garbage collection; data compression nowadays refers to a different, more complicated concept.) PDS files can only reside on DASD, not on magnetic tape, in order to use the directory structure to access individual members. Partitioned data sets are most often used for storing multiple job control language files, utility control statements, and executable modules. An improvement of this scheme is a Partitioned Data Set Extended (PDSE or PDS/E, sometimes just libraries) introduced with DFSMSdfp for MVS/XA and MVS/ESA systems. A PDS/E library can store program objects or other types of members, but not both. BPAM cannot process a PDS/E containing program objects. PDS/E structure is similar to PDS and is used to store the same types of data. However, PDS/E files have a better directory structure which does not require pre-allocation of directory blocks when the PDS/E is defined (and therefore does not run out of directory blocks if not enough were specified). Also, PDS/E automatically stores members in such a way that compression operation is not needed to reclaim "dead" space. PDS/E files can only reside on DASD in order to use the directory structure to access individual members. == Generation Data Group == A Generation Data Group (GDG) is a group of non-VSAM data sets that are successive generations of historically-related data stored on an IBM mainframe (running OS/360 and its successors or DOS/360 and its successors). A GDG is usually cataloged. An individual member of the GDG collection is called a "Generation Data Set." The latter may be identified by an absolute number, ACCTG.OURGDG(1234), or a relative number: (-1) for the previous generation, (0) for the current one, and (+1) the next generation. A GDG specifies how many generations of a data set are to be kept and at what age a generation will be deleted. Whenever a new generation is created, the system checks whether one or more obsolete generations are to be deleted. The purpose of GDGs is to automate archival, using the command language JCL, the data set name given is generic. When DSN appears, the GDG data set appears along with the history number, where (0) is the most recent version (-1), (-2), ... are previous generations (+1) a new generation (see DD) Another use of GDGs is to be able to address all generations simultaneously within a JCL script without having to know the number of currently available generations. To do this, you have to omit the parentheses and the generation number in the JCL when specifying the dataset. === GDG JCL & features === Generation Data Groups are defined using either the BLDG statement of the IEHPROGM utility or the DEFINE GENERATIONGROUP statement of the newer IDCAMS utility, which allows setting various parameters. LIMIT(10) would limit the number of generations limit to 10. SCRATCH FOR (91) would retain each member, up to the limited#generations, at least 91 days. IDCAMS can also delete (and optionally uncatalog) a GDG. ==== Example ==== Creation of a standard GDG for five safety scopes, each at least 35 days old: Delete a standard GDG:
Read more →
Plumbr

Plumbr was an Estonian software product company founded in late 2011 that developed performance monitoring software. The Plumbr product was built on top of a proprietary algorithm that automatically detected the root causes of performance issues by interpreting application performance data. In October 2020, Plumbr was acquired by Splunk. == Products == Plumbr monitored customers' JVM applications for memory leaks, garbage collection pauses and locked threads. Plumbr problem detection algorithms were based on analysis of performance data of thousands of applications. Plumbr consisted of an agent and a portal. Plumbr Agent was attached to application runtime and sent memory usage and garbage collection information to Plumbr Portal. On Plumbr Portal one could see information such as heap and permgen memory usage, garbage collection pauses' and lock contention duration. Clients that were not able to send data to third parties could order a self-hosted portal and have a full solution in-house. In case of performance incidents Plumbr provided its users with information on problem severity and problem's root cause location in source code or runtime configuration, and listed the steps needed to take to remediate the problem. Clients included NASA, NATO, Dell, HBO, Experian, EMC Corporation.
Read more →
Social news website

A social news website is a website that features user-posted stories. Such stories are ranked based on popularity, as voted on by other users of the site or by website administrators. Users typically comment online on the news posts and these comments may also be ranked in popularity. Since their emergence with the birth of Web 2.0, social news sites have been used to link many types of information, including news, humor, support, and discussion. All such websites allow the users to submit content and each site differs in how the content is moderated. On the Slashdot and Fark websites, administrators decide which articles are selected for the front page. On Reddit and Digg, the articles that get the most votes from the community of users will make it to the front page. Many social news websites also feature an online comment system, where users discuss the issues raised in an article. Some of these sites have also applied their voting system to the comments, so that the most popular comments are displayed first. Some social news websites also have a social networking service, in that users can set up a user profile and follow other users' online activity on the website. Like many other Web 2.0 tools, social news websites use the collective intelligence of all of the users to operate. Social news websites also "impl[y] the technical, economic, legal, and human enhancement of a universally distributed intelligence that will unleash a positive dynamic of recognition and skills mobilization". Social news websites help participants to share a collective vision and awareness of how their actions are integrated with those of other individuals. Social news websites provide a new and innovative way to participate in a community that is constantly being flooded with new information. These social news websites "include opportunities for peer-to-peer learning, a changed attitude toward intellectual property, the diversification of cultural expression, the development of skills valued in the modern workplace, and a more empowered conception of citizenship". These websites can help to shape and reshape democratic opinions and perspectives. Social news sites may mitigate the gatekeeping of mainstream news sources and allow the public to decide what counts as "news", which may facilitate a more participatory culture. Social news sites may also support democratic participation by allowing users from across geographic and national boundaries to access the same information, respond to fellow users' views and beliefs, and create a virtual sphere for users to contribute within. == Websites == === Active === ==== Fark ==== Fark, which started in 1997, features news on any topic. On Fark, users can submit articles to the administrators of the site. Each day, these administrators pick out 50 articles to display on the front page. ==== Slashdot ==== Slashdot, started in 1997, was one of the first social news websites. It focuses mainly on science and technology-related news. Users can submit stories and the editors pick out the best stories each day for the front page. Users can then post comments on the stories. The influx of web traffic that resulted from Slashdot linking to external websites led to the effect being called the Slashdot effect ==== Digg ==== Digg, started in December 2004, introduced the voting system. This system allows users to "digg" or "bury" articles. "Digging" is the equivalent of voting positively, so that popular articles are displayed first. "Burying" does not lower an article's score. However, if an article is buried enough times, it will be automatically deleted from the site. Digg offers a social networking service, as members can follow other members and build personal profiles with information about their interests. ==== Reddit ==== Reddit, started in June 2005, is a social news website where users can submit articles and comments and vote on these submissions. The submissions are organized into categories called "subreddits". Unlike Digg, with Reddit, users can directly affect an article's score. An "upvote" will increase the score and a "downvote" will decrease it. Articles with the highest scores are displayed on the front page. There is also a page for "controversial" articles, that have an almost equal number of upvotes and downvotes. Free speech debates have arisen due to the shutting down of obscene or potentially illegal "subreddits" (including /r/jailbait, a collection of sexually suggestive underage pictures.) Reddit introduced a system of user-created communities called "subreddits", which are essentially categories for a specific type of news. Comments on the featured posts are shown in a hierarchical fashion also based on votes. Users have the ability to earn "karma" for their participation and time on the website. ==== Hacker News ==== Hacker News, started in February 2007, is a social news site focusing on computer science and entrepreneurship, created by Paul Graham and run by his startup incubator, Y Combinator. === Defunct === ==== Newsvine ==== Newsvine, started in March 2006, was a social news website mostly focused on politics, both international and domestic. The Newsvine home page allowed users to customize "seeds" and story feeds. Users received articles via "The Wire" from sources including The Associated Press or The Huffington Post, and from "The Vine" a stream of content from other Newsvine users. The "Top of the Vine" displayed the most voted and commented on articles of the day, week, month, or year. Additionally, Newsvine allowed members to create their own "Customizable Column", which could highlight a user's content posted, recent comments, and information about the specific Newsvine member. ==== feedalizr ==== feedalizr was a cross-platform, desktop social media aggregator built using Adobe Integrated Runtime that consolidates the updates from social media and social networking websites. Users can then use this application to update those sites from their desktop and view a consolidated stream of information. ==== Voat ==== Voat, launched in April 2014 and discontinued in December of 2020, was also a social news website and is very similar to Reddit visually and functionally. The site's userbase included a large number of alt right users, many of whom migrated to Voat after being banned on Reddit. ==== Prismatic ==== Prismatic combined machine learning, user experience design, and interaction design to create a new way to discover, consume, and share media. Prismatic software used social network aggregation and machine learning algorithms to filter the content that aligns with the interests of a specific user. Prismatic integrated with Facebook, Twitter, and Pocket to gather information about user's interests and suggest the most relevant stories to read. ==== Artifact ==== Artifact was an iOS and Android app that used machine learning to personalize news recommendations to readers, and also had social features such as liking articles, commenting, and reputation scores for users.
Read more →
Social network game

A social network game (sometimes simply referred to as a social media game, social gaming, or online social game) is a type of online game that is played through social networks or social media. They typically feature gamification systems with multiplayer gameplay mechanics. Social network games were originally implemented as browser games. As mobile gaming took off, the games moved to mobile as well. While they share many aspects of traditional video games, social network games often employ additional ones that make them distinct. Traditionally they are oriented to be social games and casual games. The first cross-platform "Facebook-to-Mobile" social network game was developed in 2011 by a Finnish company Star Arcade. Social network games are amongst the most popular games played in the world, with several products with tens of millions of players. (Lil) Green Patch, Happy Farm, and Mob Wars were some of the first successful games of this genre. FarmVille, Mafia Wars, Kantai Collection, and The Sims Social are more recent examples of popular social network game. Major companies that made or published social network games include Zynga, Wooga and Bigpoint Games. == Demographics == As of 2010, it was reported that 55 percent of the social network gaming demographic in the United States consisted of women while in the United Kingdom, women made up nearly 60 percent of the demographic. In addition, most social gamers were around the 30 to 59 age range, with the average social gamer being 43 years old. Social gaming may appeal more to the older demographic because it is free, easier to advance through in a short period, does not involve as much violence as traditional video games, and is easier to grasp. Other games target certain demographics that use social media, such as Pot Farm creating a community by involving elements of cannabis subculture in its gameplay. == Technology and platforms == A social network video game is a client-server application. The client in the web era was implemented with a mix of web technologies like Flash, HTML5, PHP and JavaScript. When mobile games moved to mobile, social game front ends were developed using mobile platform technologies like Java, Objective-C, Swift and C++. The back end was a mix of programming languages and systems, including PHP, Ruby, C++ and go. Where social network video games diverged from traditional game development was the combination of real-time analytics to continuously optimize game mechanics to drive growth, revenue, and engagement. == Distinct features == The following table outlines common characteristics of social games, mentioned by Björk at the 2010 GCO Games Convention Online: A social network game may employ any of the following features: asynchronous gameplay, which allows rules to be resolved without needing players to play at the same time. gamification, which video game mechanics such as achievements and points are applied to those experienced when playing games in order to motivate and engage users. community, as one of the most distinct features of social video games is in leveraging the player's social network. Quests or game goals may only be possible if a player "shares" with friends connected by the social network hosting the game or gets them to play, as well as "neighbors" or "allies". a lack of victory conditions: there are generally no victory conditions since most developers count on users playing their games often. The game never ends and no one is ever declared winner. Instead, many casual games have "quests" or "missions" for players to complete. This is not true for board game-like social games, such as Scrabble. a virtual currency which players usually must purchase with real-world money. With the in-game currency, players can buy upgrades that would otherwise take much longer to earn through in-game achievements. In many cases, some upgrades are only available with the virtual currency. == Engagement strategies == Since social network games are often less challenging than console games and they have relatively shorter game play, they use different techniques to stretch game play and tools to retain users. Continuous goals: The games assign specific goals for users to achieve. As they advance in the game, the goals become more challenging and time-consuming. They also provide frequent feedback with their performance. Every action will translate towards a certain goal that will be used to attain higher gaming capitals. Gaming capitals: Players are encouraged to earn different badges, trophies, and accolades that indicate their progress and accomplishments. Some achievements are unlocked just by advancing in the game while others may significantly alter the rationale behind the game and require extensive investment from players. The ways of gaining gaming capital are not limited to playing games but the games-related productive activities that are appreciated in the player's social circle too. By accumulating gaming capitals, they provide an intrinsic benefit to gamers as there is an avenue to boost their accomplishment and showcase their expertise of the game. The achievements are visible to their network of friends. Gaming capitals are a way for developers to increase replay value provides extended play time, and players get more value from the game. Motivation for collecting gaming capitals: 1. Legitimization: refers to society's willingness to approve or condone certain behavior. Collecting is about channeling one's materialistic desires into more meaningful pursuits. Game achievements serve a similar purpose, allowing players to justify the hours spent playing the game. 2. Self-extension: Gathering and controlling meaningful objects or experiences can work to gain one an improved sense of self. The collector's goal to complete a collection is symbolically about completing the self too. Events timed to real world: Popular games such as Dragon City and Wild Ones require users to wait a certain time period before their "energy bars" replenish. Without energy, they are unable to conduct any form of action. Gamers are forced to wait and return after their energy replenishes to continue playing. == Monetization == Social network games frequently monetize based on virtual good transactions, but other games are emerging that utilize newer economic models. === Virtual goods === Gamers will be able to purchase in game items like power-ups, avatar accessories, or decorative items users purchase within the game itself. This is realized by monetize products that do not technically exist. Virtual goods account for over 90% of all revenue generated by the world's top social game developers. Designers optimize user experience through additional gameplay, missions, and quests, without having to worry about overhead or unused stock. == Advertising == The following are common ways of advertising in social network games: === Banner advertisements === As banner ads within social networks tend to be where ad response is low, they tend to be priced at bottom-of-the-barrel CPMs of around $2. However, because social games generate so many page views, they are the biggest part of advertising revenue for the social gaming industry. === Video ads === Videos are the ad format with the most revenue per view. They tend to be higher-priced, either by CPMs ($35+ CPM in social games) or cost-per-completed-view. According to studies, video ads result in highest brand recall thus a good return on investment for advertisers. Video ads are shown either in in-game interstitials (e.g. when the game is loading a new screen) or through incentive-based advertising, i.e. you will get either an in-game reward or Facebook credits for watching an advertisement. === Product placement === A brand or product will be injected in a game in some way. Due to the variety of ways in which product placement can be accomplished in any media, and because the category is nascent, this category is not standardized at all, but some examples include branded in-game goods or even in-game quests. For example, in a game where you run a restaurant, you might be asked to collect ingredients to make a Starbucks Frappuccino, and receive in-game rewards for doing so. As these product placement deals are non-standard, they are largely charged with a production fee, which can be $350,000 to $750,000 depending on the type of placement and the popularity of the game. === Lead generation offers === Another form of advertising that is prevalent in many social games are lead generation offers. In this form of advertising, companies, usually from different industries, aim to convince players to sign up for their goods or services and in exchange, players will receive virtual gifts or advance in the game as a reward. === Sponsorship === ==== White label games ==== Applications that are built once, then individualized and licensed again and again. Developer can create a quality app focused on fun while leaving the edge
Read more →
Hike Messenger

Hike Messenger, aka Hike Sticker Chat, is a multifunctional Indian social media and social networking service offering instant messaging (IM) and Voice over IP (VoIP) services that was launched on December 11, 2012, by Kavin Bharti Mittal. Hike functioned through SMS. The app registration used a s‍tandard, one-time password (OTP) based authentication process. It was estimated to be worth $1.4 billion and had more than 100 million registered users. It went defunct on January 6, 2021, as they were unable to compete with global messaging platforms. The app re-appeared on google play store and apple app store on 19 September 2025. == History == Hike Messenger was launched on December 12, 2012, by its founder, Kavin Bharti Mittal. The majority of users were from India, with 80% under the age of 25. The company purchased startups like TinyMogul and Hoppr in 2015. After buying US-based free voice calling company Zip Phones, Hike provided VoIP calling services. On March 5, 2015, Hike launched the 'Great Indian Sticker Challenge' to create more stickers. In February 2017, Hike acquired the social networking app Pulse. From version 5.0, it became the first social messaging app to start a mobile payment service in India. The timeline feature came back after multiple user requests and the introduction of a personalized digital envelope called Blue Packets for sending monetary gifts through a built-in wallet. In 2017, the acquisition of Bengaluru-based startup Creo was announced to enable third-party developers to build services on top of the Hike platform. In 2018, Hike provided 1 billion users with internet access by targeting smaller cities. In January 2019, the company discarded the previous super-app approach, and began launching specialized apps for specific use-cases. In May 2019, Hike announced a collaboration with Indraprastha Institute of Information Technology, Delhi (IIIT-D) to develop a variety of machine learning models. In April 2019, the company launched its first standalone app, Hike Sticker Chat. A separate content app Hike News & Content was also launched. In 2021, Hike shut down its messaging service and shifted focus to gaming and community platforms. It launched Rush, a real-money gaming app featuring casual titles like ludo and carrom, which scaled to over 10 million users and generated more than US$500 million in gross revenue over four years. The company also introduced Vibe, an approval-only community app, as part of its pivot away from the super-app and messaging model. In September 2025, following the passage of the Promotion and Regulation of Online Gaming Act, which banned real-money gaming in India, Hike announced its complete closure. Founder Kavin Bharti Mittal stated that while the company had begun international expansion, scaling globally under the new regulatory regime would require a full reset that was not a viable use of capital or resources. On 19 September 2025, hike was relaunched on play store and app store by the name hike messenger. == Application == === Timeline of Features === On 15 April 2014, Hike introduced unlimited free SMS via a service called Hike Offline, through credits earned by users from regular chatting, as connectivity is still a major issue in many parts of India. In an attempt to appeal to its younger users, Hike introduced features that find resonance with the local market, such as Last Seen Privacy and localized sticker packs. It also introduced a two-way chat theme, allowing users to change the chat background for themselves and for their friends simultaneously. The app also started showing live Cricket scores in collaboration with Cricbuzz, as well as news, casual games, and social media feeds. Hike also added a file transfer service, allowing files less than 100MB of all formats, with a view on further increasing the size limit to 1 GB. With the launch of version 2.9.2.0 in January 2015, Hike implemented support for sending uncompressed images and a "quick upload" feature optimized for 2G speed. Later that month, Hike introduced a voice calling feature for its users. In September 2015, Hike launched free group call support with up to 100 people in a simultaneous conference call environment. In November 2016, Hike announced the launch of a feature called Stories that allows people to share real-life moments using fun live filters which automatically get deleted after 48 hours, and a new camera design with localized filters. Hike 4.0 launched on 26 August 2015 with the tagline 'Got a Gang? Get on Hike'. Hike 4.0 was an optimization-focused update, increasing the performance of the app on poor networks. It supported photo filters, doodles, and bite-sized news updates in under 100 characters. Hike launched News Feed with Hindi language support on 29 September 2015 to cater for the needs of the non-English population. Hike launched version 3.5 as the biggest update for Windows Phone 8.1 during December 2015 which changed the user interface for more simpler navigation, supported sending unlimited non-media files and documents of any format and better group admin settings. It also included ten brand new chat themes. Hike launched a microapp feature which was live for two days on 8 May 2016, as a Mother's Day special in which users could add images, quotes or messages as a token of love with customized e-cards and stickers on their timeline not only on Hike, but also on other platforms. On 26 October 2016, Hike Messenger rolled out the beta version of a video calling feature ahead of WhatsApp starting with the Android users which also lets recipients preview a video call before deciding to take it and is optimized to even work under 2G conditions. On 24 December 2016, Hike rolled out a short 20-second Video Stories feature that can be directly shared with friends or posted on a public timeline with different filters in collaboration with content creators with the same 48-hour time limit before being automatically deleted. The Stories feature continues to receive constant future updates to include and enable content, public story option, private user messaging and geo-tagging. In September 2017, Hike launched personalized sticker packs with 20,000+ graphical stickers for over 500 colleges that covered around 1,000 colleges by December 2018 across India which can be used across different geographies, and are highly customized for users with availability in 40+ local languages that support automatic sticker suggestions where the application suggests the best reply for any sticker message and also allows users to "nudge", a feature used to ping the receiver. Hike started supporting user comments on friend's posts, added a specific message reply function, a redesigned camera interface to support front flash and user mentions with the help of the @ symbol. In December, 2017, Hike launched group voting, bill splitting, checklists and event reminders for group chat that supports up to 1,000 users both on iOS and Android platform. Hike launched another feature called Hike Land, which is a virtual world with beta trial to start from March 2020, that will use Hike Moji where online users with their digital avatar can hang out with other users and will be built inside the Hike Sticker Chat application. It is mainly targeted but not restricted towards 16 to 21 years age group of people. Without unveiling much about Hike Land, a separate website has been created with option to reserve spots by giving details like name, gender and phone number that will link the user profile from the Hike Sticker Chat account though it is not a necessity. ==== Hike Direct ==== The Hike Direct feature is based on the technology known as WiFi Direct, which initially was also called WiFi P2P and got introduced to users by October 2015, which enables sharing of files such as music, apps, videos without a live internet connection within a 100-meter radius by creating a wireless network between two or more devices with a transfer speed of 100MB per minute. For privacy and security reasons, Hike didn't show the recipient's location or proximity and works only when two users are connected in the same room by adding one another into the contact list. ==== Hike Wallet ==== In June 2017, Hike announced the launch of version 5.0 with multiple new features like User Chat Themes, Night Mode and Magic Selfie. along with a built-in Wallet partnered with Yes Bank. This feature was first rolled out to Android users followed by iOS users at a later stage. Hike collaborated with Airtel Payment Bank to power its digital payment wallet by November 2017 where Hike users have access to Airtel Payments Bank's merchant & utility payment services and know your customer (KYC) infrastructure with 5 million transactions happening from services like recharge and P2P. Hike formed a partnership with Ola Cabs to bring a taxi and auto-rickshaw booking facility from 14 February 2018. With Hike Wallet facility users could now book bus tickets with 3
Read more →
Photometric stereo

Photometric stereo is a technique in computer vision for estimating the surface normals of objects by observing that object under different lighting conditions (photometry). It is based on the fact that the amount of light reflected by a surface is dependent on the orientation of the surface in relation to the light source and the observer. By measuring the amount of light reflected into a camera, the space of possible surface orientations is limited. Given enough light sources from different angles, the surface orientation may be constrained to a single orientation or even overconstrained. The technique was originally introduced by Woodham in 1980. The special case where the data is a single image is known as shape from shading, and was analyzed by B. K. P. Horn in 1989. Photometric stereo has since been generalized to many other situations, including extended light sources and non-Lambertian surface finishes. Current research aims to make the method work in the presence of projected shadows, highlights, and non-uniform lighting. Photometric stereo is widely used in various fields, including archaeology, cultural heritage conservation, and quality control. It is now integrated into widely used open-source software, such as Meshroom. == Basic method == Under Woodham's original assumptions — Lambertian reflectance, known point-like distant light sources, and uniform albedo — the problem can be solved by inverting the linear equation I = L ⋅ n {\displaystyle I=L\cdot n} , where I {\displaystyle I} is a (known) vector of m {\displaystyle m} observed intensities, n {\displaystyle n} is the (unknown) surface normal, and L {\displaystyle L} is a (known) 3 × m {\displaystyle 3\times m} matrix of normalized light directions. This model can easily be extended to surfaces with non-uniform albedo, while keeping the problem linear. Taking an albedo reflectivity of k {\displaystyle k} , the formula for the reflected light intensity becomes I = k ( L ⋅ n ) . {\displaystyle I=k(L\cdot n).} If L {\displaystyle L} is square (there are exactly 3 lights) and non-singular, it can be inverted, giving L − 1 I = k n . {\displaystyle L^{-1}I=kn.} Since the normal vector is known to have length 1, k {\displaystyle k} must be the length of the vector k n {\displaystyle kn} , and n {\displaystyle n} is the normalised direction of that vector. If L {\displaystyle L} is not square (there are more than 3 lights), a generalisation of the inverse can be obtained using the Moore–Penrose pseudoinverse, by simply multiplying both sides with L T {\displaystyle L^{T}} , giving L T I = L T k ( L ⋅ n ) , {\displaystyle L^{T}I=L^{T}k(L\cdot n),} ( L T L ) − 1 L T I = k n , {\displaystyle (L^{T}L)^{-1}L^{T}I=kn,} after which the normal vector and albedo can be solved as described above. == Non-Lambertian surfaces == The classical photometric stereo problem concerns itself only with Lambertian surfaces, with perfectly diffuse reflection. This is unrealistic for many types of materials, especially metals, glass and smooth plastics, and will lead to aberrations in the resulting normal vectors. Many methods have been developed to lift this assumption. In this section, a few of these are listed. === Specular reflections === Historically, in computer graphics, the commonly used model to render surfaces started with Lambertian surfaces and progressed first to include simple specular reflections. Computer vision followed a similar course with photometric stereo. Specular reflections were among the first deviations from the Lambertian model. These are a few adaptations that have been developed. Many techniques ultimately rely on modelling the reflectance function of the surface, that is, how much light is reflected in each direction. This reflectance function has to be invertible. The reflected light intensities towards the camera is measured, and the inverse reflectance function is fit onto the measured intensities, resulting in a unique solution for the normal vector. === General BRDFs and beyond === According to the Bidirectional reflectance distribution function (BRDF) model, a surface may distribute the amount of light it receives in any outward direction. This is the most general known model for opaque surfaces. Some techniques have been developed to model (almost) general BRDFs. In practice, all of these require many light sources to obtain reliable data. These are methods in which surfaces with general BRDFs can be measured. Determine the explicit BRDF prior to scanning. To do this, a different surface is required that has the same or a very similar BRDF, of which the actual geometry (or at least the normal vectors for many points on the surface) is already known. The lights are then individually shone upon the known surface, and the amount of reflection into the camera is measured. Using this information, a look-up table can be created that maps reflected intensities for each light source to a list of possible normal vectors. This puts constraints on the possible normal vectors the surface may have, and reduces the photometric stereo problem to an interpolation between measurements. Typical known surfaces to calibrate the look-up table with are spheres for their wide variety of surface orientations. Restricting the BRDF to be symmetrical. If the BRDF is symmetrical, the direction of the light can be restricted to a cone about the direction to the camera. Which cone this is depends on the BRDF itself, the normal vector of the surface, and the measured intensity. Given enough measured intensities and the resulting light directions, these cones can be approximated and therefore the normal vectors of the surface. Some progress has been made towards modelling an even more general surfaces, such as Spatially Varying Bidirectional Distribution Functions (SVBRDF), Bidirectional surface scattering reflectance distribution functions (BSSRDF), and accounting for interreflections. However, such methods are still fairly restrictive in photometric stereo. Better results have been achieved with structured light. == Uncalibrated photometric stereo == Uncalibrated Photometric Stereo is an approach in photometric stereo that aims to reconstruct the 3D shape of an object from images captured under unknown lighting conditions. Unlike classical methods, which often assume controlled or known lighting setups, this approach removes these constraints, making it adaptable to diverse and real-world environments. The advent of deep learning has revolutionized universal PS by replacing handcrafted assumptions with data-driven models. Recent approaches leverage Transformer-based architectures and multi-scale encoder–decoder networks to directly estimate surface normals from input images. Uncalibrated Photometric Stereo is inherently an ill-posed problem, as it attempts to recover 3D shape and lighting conditions simultaneously from images alone. This leads to fundamental ambiguities in the reconstruction process, which manifest as systematic errors in the recovered geometry, including global distortions in the object's overall shape, and misinterpretation of surface orientation, where concave regions may appear convex and vice versa. To address the challenges of uncalibrated photometric stereo, hybrid methods have emerged that combine multi-view stereo and photometric stereo. These approaches leverage the strengths of both techniques, including geometric reliability and resolution.
Read more →
Merit Network

Merit Network, Inc., is a nonprofit member-governed organization providing high-performance computer networking and related services to educational, government, health care, and nonprofit organizations, primarily in Michigan. Created in 1966, Merit operates the longest running regional computer network in the United States. == Organization == Created in 1966 as the Michigan Educational Research Information Triad by Michigan State University (MSU), the University of Michigan (U-M), and Wayne State University (WSU), Merit was created to investigate resource sharing by connecting the mainframe computers at these three Michigan public research universities. Merit's initial three node packet-switched computer network was operational in October 1972 using custom hardware based on DEC PDP-11 minicomputers and software developed by the Merit staff and the staffs at the three universities. Over the next dozen years the initial network grew as new services such as dial-in terminal support, remote job submission, remote printing, and file transfer were added; as gateways to the national and international Tymnet, Telenet, and Datapac networks were established, as support for the X.25 and TCP/IP protocols was added; as additional computers such as WSU's MVS system and the UM's electrical engineering's VAX running UNIX were attached; and as new universities became Merit members. Merit's involvement in national networking activities started in the mid-1980s with connections to the national supercomputing centers and work on the 56 kbit/s National Science Foundation Network (NSFNET), the forerunner of today's Internet. From 1987 until April 1995, Merit re-engineered and managed the NSFNET backbone service. MichNet, Merit's regional network in Michigan was attached to NSFNET and in the early 1990s Merit began extending "the Internet" throughout Michigan, offering both direct connect and dial-in services, and upgrading the statewide network from 56 kbit/s to 1.5 Mbit/s, and on to 45, 155, 622 Mbit/s, and eventually 1 and 10 Gbit/s. In 2003 Merit began its transition to a facilities based network, using fiber optic facilities that it shares with its members, that it purchases or leases under long-term agreements, or that it builds. In addition to network connectivity services, Merit offers a number of related services within Michigan and beyond, including: Internet2 connectivity, VPN, Network monitoring, Voice over IP (VOIP), Cloud storage, E-mail, Domain Name, Network Time, VMware and Zimbra software licensing, Colocation, and professional development seminars, workshops, classes, conferences, and meetings. == History == === Creating the network: 1966 to 1973 === The Michigan Educational Research Information Triad (MERIT) was formed in the fall of 1966 by Michigan State University (MSU), University of Michigan (U-M), and Wayne State University (WSU). More often known as the Merit Computer Network or simply Merit, it was created to design and implement a computer network connecting the mainframe computers at the universities. In the fall of 1969, after funding for the initial development of the network had been secured, Bertram Herzog was named director for MERIT. Eric Aupperle was hired as senior engineer, and was charged with finding hardware to make the network operational. The National Science Foundation (NSF) and the State of Michigan provided the initial funding for the network. In June 1970, the Applied Dynamics Division of Reliance Electric in Saline, Michigan was contracted to build three Communication Computers or CCs. Each would consist of a Digital Equipment Corporation (DEC) PDP-11 computer, dataphone interfaces, and interfaces that would attach them directly to the mainframe computers. The cost was to be slightly less than the $300,000 ($2,487,100, adjusted for inflation) originally budgeted. Merit staff wrote the software that ran on the CCs, while staff at each of the universities wrote the mainframe software to interface to the CCs. The first completed connection linked the IBM S/360-67 mainframe computers running the Michigan Terminal System at WSU and U-M, and was publicly demonstrated on December 14, 1971. The MSU node was completed in October 1972, adding a CDC 6500 mainframe running Scope/Hustler. The network was officially dedicated on May 15, 1973. === Expanding the network: 1974 to 1985 === In 1974, Herzog returned to teaching in the University of Michigan's Industrial Engineering Department, and Aupperle was appointed as director. Use of the all uppercase name "MERIT" was abandoned in favor of the mixed case "Merit". The first network connections were host to host interactive connections which allowed person to remote computer or local computer to remote computer interactions. To this, terminal to host connections, batch connections (remote job submission, remote printing, batch file transfer), and interactive file copy were added. And, in addition to connecting to host computers over custom hardware interfaces, the ability to connect to hosts or other networks over groups of asynchronous ports and via X.25 were added. Merit interconnected with Telenet (later SprintNet) in 1976 to give Merit users dial-in access from locations around the United States. Dial-in access within the U.S. and internationally was further expanded via Merit's interconnections to Tymnet, ADP's Autonet, and later still the IBM Global Network as well as Merit's own expanding network of dial-in sites in Michigan, New York City, and Washington, D.C. In 1978, Western Michigan University (WMU) became the fourth member of Merit (prompting a name change, as the acronym Merit no longer made sense as the group was no longer a triad). To expand the network, the Merit staff developed new hardware interfaces for the Digital PDP-11 based on printed circuit technology. The new system became known as the Primary Communications Processor (PCP), with the earliest PCPs connecting a PDP-10 located at WMU and a DEC VAX running UNIX at U-M's Electrical Engineering department. A second hardware technology initiative in 1983 produced the smaller Secondary Communication Processors (SCP) based on DEC LSI-11 processors. The first SCP was installed at the Michigan Union in Ann Arbor, creating UMnet, which extended Merit's network connectivity deeply into the U-M campus. In 1983 Merit's PCP and SCP software was enhanced to support TCP/IP and Merit interconnected with the ARPANET. === National networking, NSFNET, and the Internet: 1986 to 1995 === In 1986 Merit engineered and operated leased lines and satellite links that allowed the University of Michigan to access the supercomputing facilities at Pittsburgh, San Diego, and NCAR. In 1987, Merit, IBM and MCI submitted a winning proposal to NSF to implement a new NSFNET backbone network. The new NSFNET backbone network service began July 1, 1988. It interconnected supercomputing centers around the country at 1.5 megabits per second (T1), 24 times faster than the 56 kilobits-per-second speed of the previous network. The NSFNET backbone grew to link scientists and educators on university campuses nationwide and connect them to their counterparts around the world. The NSFNET project caused substantial growth at Merit, nearly tripling the staff and leading to the establishment of a new 24-hour Network Operations Center at the U-M Computer Center. In September 1990 in anticipation of the NSFNET T3 upgrade and the approaching end of the 5-year NSFNET cooperative agreement, Merit, IBM, and MCI formed Advanced Network and Services (ANS), a new non-profit corporation with a more broadly based Board of Directors than the Michigan-based Merit Network. Under its cooperative agreement with NSF, Merit remained ultimately responsible for the operation of NSFNET, but subcontracted much of the engineering and operations work to ANS. In 1991 the NSFNET backbone service was expanded to additional sites and upgraded to a more robust 45 Mbit/s (T3) based network. The new T3 backbone was named ANSNet and provided the physical infrastructure used by Merit to deliver the NSFNET Backbone Service. On April 30, 1995, the NSFNET project came to an end, when the NSFNET backbone service was decommissioned and replaced by a new Internet architecture with commercial Internet service providers (ISPs) interconnected at Network Access Points provided by multiple providers across the country. === Bringing the Internet to Michigan: 1985 to 2001 === During the 1980s, Merit Network grew to serve eight member universities, with Oakland University joining in 1985 and Central Michigan University, Eastern Michigan University, and Michigan Technological University joining in 1987. In 1990, Merit's board of directors formally changed the organization's name to Merit Network, Inc., and created the name MichNet to refer to Merit's statewide network. The board also approved a staff proposal to allow organizations other than publicly supported universities, referred to as aff
Read more →
Data lineage

Data lineage refers to the process of tracking how data is generated, transformed, transmitted and used across systems over time. It documents data's origins, transformations and movements, providing detailed visibility into its life cycle. This process simplifies the identification of errors in data analytics workflows, by enabling users to trace issues back to their root causes. Data lineage facilitates the ability to replay specific segments or inputs of the dataflow. This can be used in debugging or regenerating lost outputs. In database systems, this concept is closely related to data provenance, which involves maintaining records of inputs, entities, systems and processes that influence data. Data provenance provides a historical record of data origins and transformations. It supports forensic activities such as data-dependency analysis, error/compromise detection, recovery, auditing and compliance analysis: "Lineage is a simple type of why provenance." Data governance plays a critical role in managing metadata by establishing guidelines, strategies and policies. Enhancing data lineage with data quality measures and master data management adds business value. Although data lineage is typically represented through a graphical user interface (GUI), the methods for gathering and exposing metadata to this interface can vary. Based on the metadata collection approach, data lineage can be categorized into three types: Those involving software packages for structured data, programming languages and Big data systems. Data lineage information includes technical metadata about data transformations. Enriched data lineage may include additional elements such as data quality test results, reference data, data models, business terminology, data stewardship information, program management details and enterprise systems associated with data points and transformations. Data lineage visualization tools often include masking features that allow users to focus on information relevant to specific use cases. To unify representations across disparate systems, metadata normalization or standardization may be required. == Representation of data lineage == Representation broadly depends on the scope of the metadata management and reference point of interest. Data lineage provides sources of the data and intermediate data flow hops from the reference point with backward data lineage, leading to the final destination's data points and its intermediate data flows with forward data lineage. These views can be combined with end-to-end lineage for a reference point that provides a complete audit trail of that data point of interest from sources to their final destinations. As the data points or hops increase, the complexity of such representation becomes incomprehensible. Thus, the best feature of the data lineage view is the ability to simplify the view by temporarily masking unwanted peripheral data points. Tools with the masking feature enable scalability of the view and enhance analysis with the best user experience for both technical and business users. Data lineage also enables companies to trace sources of specific business data to track errors, implement changes in processes and implementing system migrations to save significant amounts of time and resources. Data lineage can improve efficiency in business intelligence BI processes. Data lineage can be represented visually to discover the data flow and movement from its source to destination via various changes and hops on its way in the enterprise environment. This includes how the data is transformed along the way, how the representation and parameters change and how the data splits or converges after each hop. A simple representation of the Data Lineage can be shown with dots and lines, where dots represent data containers for data points, and lines connecting them represent transformations the data undergoes between the data containers. Data lineage can be visualized at various levels based on the granularity of the view. At a very high-level, data lineage is visualized as systems that the data interacts with before it reaches its destination. At its most granular, visualizations at the data point level can provide the details of the data point and its historical behavior, attribute properties and trends and data quality of the data passed through that specific data point in the data lineage. The scope of the data lineage determines the volume of metadata required to represent its data lineage. Usually, data governance and data management of an organization determine the scope of the data lineage based on their regulations, enterprise data management strategy, data impact, reporting attributes and critical data elements of the organization. == Rationale == Distributed systems like Google Map Reduce, Microsoft Dryad, Apache Hadoop (an open-source project) and Google Pregel provide such platforms for businesses and users. However, even with these systems, Big Data analytics can take several hours, days or weeks to run, simply due to the data volumes involved. For example, a ratings prediction algorithm for the Netflix Prize challenge took nearly 20 hours to execute on 50 cores, and a large-scale image processing task to estimate geographic information took 3 days to complete using 400 cores. "The Large Synoptic Survey Telescope is expected to generate terabytes of data every night and eventually store more than 50 petabytes, while in the bioinformatics sector, the 12 largest genome sequencing houses in the world now store petabytes of data apiece. It is very difficult for a data scientist to trace an unknown or an unanticipated result. === Big data debugging === Big data analytics is the process of examining large data sets to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information. Machine learning, among other algorithms, is used to transform and analyze the data. Due to the large size of the data, there could be unknown features in the data. The massive scale and unstructured nature of data, the complexity of these analytics pipelines, and long runtimes pose significant manageability and debugging challenges. Even a single error in these analytics can be extremely difficult to identify and remove. While one may debug them by re-running the entire analytics through a debugger for stepwise debugging, this can be expensive due to the amount of time and resources needed. Auditing and data validation are other major problems due to the growing ease of access to relevant data sources for use in experiments, the sharing of data between scientific communities and use of third-party data in business enterprises. As such, more cost-efficient ways of analyzing data intensive scale-able computing (DISC) are crucial to their continued effective use. === Challenges in Big Data debugging === ==== Massive scale ==== According to an EMC/IDC study, 2.8 ZB of data were created and replicated in 2012. Furthermore, the same study states that the digital universe will double every two years between now and 2020, and that there will be approximately 5.2 TB of data for every person in 2020. Based on current technology, the storage of this much data will mean greater energy usage by data centers. ==== Unstructured data ==== Unstructured data usually refers to information that doesn't reside in a traditional row-column database. Unstructured data files often include text and multimedia content, such as e-mail messages, word processing documents, videos, photos, audio files, presentations, web pages and many other kinds of business documents. While these types of files may have an internal structure, they are still considered "unstructured" because the data they contain doesn't fit neatly into a database. The amount of unstructured data in enterprises is growing many times faster than structured databases are growing. Big data can include both structured and unstructured data, but IDC estimates that 90 percent of Big Data is unstructured data. The fundamental challenge of unstructured data sources is that they are difficult for non-technical business users and data analysts alike to unbox, understand and prepare for analytic use. Beyond issues of structure, the sheer volume of this type of data contributes to such difficulty. Because of this, current data mining techniques often leave out valuable information and make analyzing unstructured data laborious and expensive. In today's competitive business environment, companies have to find and analyze the relevant data they need quickly. The challenge is going through the volumes of data and accessing the level of detail needed, all at a high speed. The challenge only grows as the degree of granularity increases. One possible solution is hardware. Some vendors are using increased memory and parallel processing to crunch large volumes of data quickly. Another method is putting data in-memory but using a grid
Read more →