Digital cinematography is the process of capturing (recording) a motion picture using digital image sensors rather than through film stock. As digital technology has improved in recent years, this practice has become dominant. Since the 2000s, most movies across the world have been captured as well as distributed digitally. Many vendors have brought products to market, including traditional film camera vendors like Arri and Panavision, as well as new vendors like Red, Blackmagic, Silicon Imaging, Vision Research and companies which have traditionally focused on consumer and broadcast video equipment, like Sony, GoPro, and Panasonic. As of 2023, professional 4K digital cameras were approximately equal to 35mm film in their resolution and dynamic range capacity. Some filmmakers still prefer to use film picture formats to achieve the desired results. == History == The basis for digital cameras are metal–oxide–semiconductor (MOS) image sensors. The first practical semiconductor image sensor was the charge-coupled device (CCD), based on MOS capacitor technology. Following the commercialization of CCD sensors during the late 1970s to early 1980s, the entertainment industry slowly began transitioning to digital imaging and digital video over the next two decades. The CCD was followed by the CMOS active-pixel sensor (CMOS sensor), developed in the 1990s. Beginning in the late 1980s, Sony began marketing the concept of "electronic cinematography," utilizing its analog Sony HDVS professional video cameras. The effort met with very little success. However, this led to one of the earliest high definition video shot feature movies, Julia and Julia (1987). Rainbow (1996) was the world's first film to utilize extensive digital post production techniques. Shot entirely with Sony's first Solid State Electronic Cinematography cameras and featuring over 35 minutes of digital image processing and visual effects, all post production, sound effects, editing and scoring were completed digitally. The Digital High Definition image was transferred to a 35mm negative via an electron beam recorder for theatrical release. The first digitally videoed and post produced feature was Windhorse, shot in Tibet and Nepal in 1996 on the Sony DVW-700WS Digital Betacam and the prosumer Sony DCR-VX1000. The offline editing (Avid) and the online post and color work (Roland House / da Vinci) were also all digital. The film, transferred to 35mm negative for theatrical release, won Best U.S. Feature at the Santa Barbara Film Festival in 1998. In 1997, with the introduction of HDCAM recorders and 1920 × 1080 pixel digital professional video cameras based on CCD technology, the idea, now re-branded as "digital cinematography," began to gain traction in the market. Shot and released in 1998, The Last Broadcast is believed by some to be the first feature-length video shot and edited entirely on consumer-level digital equipment. In May 1999, George Lucas challenged the supremacy of the movie-making medium of film for the first time by including footage filmed with high-definition digital cameras in Star Wars: Episode I – The Phantom Menace. The digital footage blended seamlessly with the footage shot on film and he announced later that year he would film its sequels entirely on hi-def digital video. Also in 1999, digital projectors were installed in four theaters for the showing of The Phantom Menace. In May 2000, Vidocq, which was directed by Pitof, began principal photography shot entirely using a Sony HDW-F900 camera, with the video being released in September the next year. According to the Guinness World Records, Vidocq is the first full length feature filmed in digital high resolution. In June 2000, Star Wars: Episode II – Attack of the Clones began principal photography shot entirely using a Sony HDW-F900 camera as Lucas had previously stated. The film was released in May 2002. In May 2001 Once Upon a Time in Mexico was also shot in 24 frame-per-second high-definition digital video, partially developed by George Lucas using a Sony HDW-F900 camera, following Robert Rodriguez's introduction to the camera at Lucas' Skywalker Ranch facility whilst editing the sound for Spy Kids. A lesser-known movie, Russian Ark (2002), was also shot with the same camera and was the first tapeless digital movie, recorded on HDD instead of tape. In 2009, Slumdog Millionaire became the first movie shot mainly in digital to be awarded the Academy Award for Best Cinematography. The highest-grossing movie in the history of cinema, Avatar (2009), not only was shot on digital cameras as well, but also made the main revenues at the box office no longer by film, but digital projection. Major movies shot on digital video overtook those shot on film in 2013. Since 2016 over 90% of major films were shot on digital video. As of 2017, 92% of films are shot on digital. Only 24 major films released in 2018 were shot on 35mm. Since the 2000s, most movies across the world have been captured as well as distributed digitally. Today, cameras from companies like Sony, Panasonic, JVC and Canon offer a variety of choices for shooting high-definition video. At the high-end of the market, there has been an emergence of cameras aimed specifically at the digital cinema market. These cameras from Sony, Vision Research, Arri, Blackmagic Design, Panavision, Grass Valley and Red offer resolution and dynamic range that exceeds that of traditional video cameras, which are designed for the limited needs of broadcast television. == Technology == Digital cinematography captures motion pictures digitally in a process analogous to digital photography. While there is a clear technical distinction that separates the images captured in digital cinematography from video, the term "digital cinematography" is usually applied only in cases where digital acquisition is substituted for film acquisition, such as when shooting a feature film. The term is seldom applied when digital acquisition is substituted for video acquisition, as with live broadcast television programs. === Recording === ==== Cameras ==== Professional cameras include the Sony CineAlta (F) Series, Blackmagic Cinema Camera, Red One, Arri D-20, D-21 and Alexa, Panavision Genesis, Silicon Imaging SI-2K, Thomson Viper, Vision Research Phantom, IMAX 3D camera based on two Vision Research Phantom cores, Weisscam HS-1 and HS-2, GS Vitec noX, and the Fusion Camera System. Independent micro-budget filmmakers have also pressed low-cost consumer and prosumer cameras into service for digital filmmaking. Flagship smartphones like the Apple iPhone have been used to shoot movies like Unsane (shot on the iPhone 7 Plus) and Tangerine (shot on three iPhone 5S phones) and in January 2018, Unsane's director and Oscar winner Steven Soderbergh expressed an interest in filming other productions solely with iPhones going forward. ==== Sensors ==== Digital cinematography cameras capture digital images using image sensors, either charge-coupled device (CCD) sensors or CMOS active-pixel sensors, usually in one of two arrangements. Single chip cameras designed specifically for the digital cinematography market often use a single sensor (much like digital photo cameras), with dimensions similar in size to a 16 or 35 mm film frame or even (as with the Vision 65) a 65 mm film frame. An image can be projected onto a single large sensor exactly the same way it can be projected onto a film frame, so cameras with this design can be made with PL, PV and similar mounts, in order to use the wide range of existing high-end cinematography lenses available. Their large sensors also let these cameras achieve the same shallow depth of field as 35 or 65 mm motion picture film cameras, which many cinematographers consider an essential visual tool. Codecs Professional raw video recording codecs include Blackmagic Raw, Red Raw, Arri Raw and Canon Raw. ==== Video formats ==== Unlike other video formats, which are specified in terms of vertical resolution (for example, 1080p, which is 1920×1080 pixels), digital cinema formats are usually specified in terms of horizontal resolution. As a shorthand, these resolutions are often given in "nK" notation, where n is the multiplier of 1024 such that the horizontal resolution of a corresponding full-aperture, digitized film frame is exactly 1024 n {\displaystyle 1024n} pixels. Here the "K" has a customary meaning corresponding to the binary prefix "kibi" (ki). For instance, a 2K image is 2048 pixels wide, and a 4K image is 4096 pixels wide. Vertical resolutions vary with aspect ratios though; so a 2K image with an HDTV (16:9) aspect ratio is 2048×1152 pixels, while a 2K image with a SDTV or Academy ratio (4:3) is 2048×1536 pixels, and one with a Panavision ratio (2.39:1) would be 2048×856 pixels, and so on. Due to the "nK" notation not corresponding to specific horizontal resolutions per format a 2K image lacking, for example, the typical 35mm film soundtrack space, is only 182
Message queuing service
A message queueing service is a message-oriented middleware or MOM deployed in a compute cloud using software as a service model. Service subscribers access queues and or topics to exchange data using point-to-point or publish and subscribe patterns. It's important to differentiate between event-driven and message-driven (aka queue driven) services: Event-driven services (e.g. AWS SNS) are decoupled from their consumers. Whereas queue / message driven services (e.g. AWS SQS) are coupled with their consumers. Message queues can be a good buffer to handle spiky workloads but they have a finite capacity. According to Gregor Hohpe, message queues require proper mechanisms (aka flow controls) to avoid filling the queue beyond its manageable capacity and to keep the system stable. == Ordering Guarantees in Message Queues == Amazon SQS FIFO and Azure Service Bus sessions are queue-based messaging systems that provide ordering guarantees within a message group or session attempt but do not necessarily guarantee ordered delivery in cases of retries or failures. In SQS FIFO, messages in the same message group are processed in order, with subsequent messages held until the preceding message is successfully processed or moved to the dead-letter queue (DLQ). Once a message is placed in the DLQ, it is no longer retried, creating a gap in the sequence. However, the remaining messages continue to be delivered in order. Azure Service Bus sessions function similarly by maintaining ordering within a session, provided a single consumer processes messages sequentially. The implementation differs from SQS FIFO but follows the same fundamental ordering principle. In contrast, Apache Kafka is a distributed log-based messaging system that guarantees ordering within individual partitions rather than across the entire topic. Unlike queue-based systems, Kafka retains messages in a durable, append-only log, allowing multiple consumers to read at different offsets. Kafka uses manual offset management, giving consumers control over retries and failure handling. If a consumer fails to process a message, it can delay committing the offset, preventing further progress in that partition while other partitions remain unaffected. This partition-based design enables fault isolation and parallel processing while allowing ordering to be maintained within partitions, depending on consumer handling. == Vendors == Apache Kafka Apache Kafka is a distributed system consisting of servers that store and forward messages between producer client and consumer applications. IBM MQ IBM MQ offers a managed service that can be used on IBM Cloud and Amazon Web Services. Microsoft Azure Service Bus Service Bus offers queues, topics & subscriptions, and rules/actions in order to support publish-subscribe, temporal decoupling, and load balancing scenarios. Azure Service Bus is built on AMQP allowing any existing AMQP 1.0 client stack to interact with Service Bus directly or via existing .Net, Java, Node, and Python clients. Standard and Premium tiers allow for pay as you go or isolated resources at massive scale. Oracle Messaging Cloud Service This service provides a messaging solution for applications for asynchronous communication and is influenced by the Java Message Service (JMS) API specification. Any application platform that understands HTTP can also use Oracle Messaging Cloud Service through the REST interface. For Java applications, Oracle Messaging Cloud Service provides a Java library that implements and extends the JMS 1.1 interface. The Java library implements the JMS API by acting as a client of the REST API. Amazon Simple Queue Service Supports messages natively up to 256K, or up to 2GB by transmitting payload via S3. Highly scalable, durable and resilient. Provides loose-FIFO and 'at least once' delivery in order to provide massive scale. Supports REST API and optional Java Message Service client. Low latency. Utilizes Amazon Web Services. IronMQ Supports messages up to 64k; guarantees order; guarantees once only delivery; no delays retrieving messages. Supports REST API and beanstalkd open source protocol. Runs on multiple clouds including AWS and Rackspace. Scaling must be managed by user. RabbitMQ RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine. Supports AMQP, STOMP, MQTT StormMQ Open platform supports messages up to 50Mb. Uses AMQP to avoid vendor lock-in and provide language neutrality. Locate-It Option allows customers to audit the location of their data at all times and satisfy data protection principles. AnypointMQ An enterprise multi-tenant, cloud messaging service that performs advanced asynchronous messaging scenarios between applications. Anypoint MQ is fully integrated with Anypoint Platform, offering role based access control, client application management, and connectors.
Open Data-Link Interface
The Open Data-Link Interface (ODI) is an application programming interface (API) for network interface controllers (NICs) developed by Apple and Novell. The API serves the same function as Microsoft and 3COM's Network Driver Interface Specification (NDIS). Originally, ODI was written for NetWare and Macintosh environments. Like NDIS, ODI provides rules that establish a vendor-neutral interface between the protocol stack and the adapter driver. It resides in Layer 2, the Data Link layer, of the OSI model. This interface also enables one or more network drivers to support one or more protocol stacks.
WhoSay
WhoSay was an American social media service and branding platform for celebrities and their fans. Founded in Los Angeles in 2010, with financing by Creative Artists Agency (CAA), Amazon.com and other investors, it is notable for allowing its users to retain ownership rights over the content that they post to their accounts, through copyright branding, and for enabling users to post content to other social media sites like Twitter, Facebook, Instagram and Tumblr simultaneously. WhoSay describes itself as a "social celebrity magazine" whose editorial team keeps its users informed about the latest celebrity and entertainment news. Clients such as Dylan McDermott and Chris Rock lauded the service for its ability to add content to multiple social network sites easily. Rock in particular has commented on its ease of use for those who are not part of a tech-savvy demographic, commenting, "It's perfect for someone that's not 25." WhoSay's competitors included theAudience, which is operated by the William Morris Endeavor. == History == WhoSay was founded in March 2010, by Steve Ellis and the Los Angeles-based talent agency Creative Artists Agency (CAA). It was financed through investments Amazon.com (who along with CAA, holds a minority stake in the company), Comcast, Greylock Partners, and High Peak Ventures. The company's main headquarters are in The New York Times Building in Manhattan, with additional headquarters in CAA's office building in the Silicon Beach area of Los Angeles, and in London. The company was founded to protect celebrities' intellectual property and enable the celebrities themselves to profit themselves from their own content through copyright branding. Its chief executive is co-founder Steve Ellis, who, after leaving Getty Images, was contacted by CAA, who were looking to resolve the issue of celebrities losing the rights to their own photos and videos when uploading them to social network sites. Ellis explained WhoSay's mission thus: "We work with people who are constantly being utilized by third parties for the wrong reasons. [The company was formed] to give celebrities and other influential people a set of tools to allow them to manage and control their presence in the digital world." In this way, WhoSay is likened by Ellis to "a People magazine by the people themselves who are in it." The company started slowly, until CAA client Tom Hanks signed onto WhoSay three months after the service's launch. The company continued to maintain a low profile for the first three years of operation, during which it accumulated a client list of 1,500 actors, musicians and artists. Clients are accepted by the service on an invitation-only basis, although they are not restricted to Creative Artists clients. Among them are Kelly Clarkson, Julia Louis-Dreyfus, Paula Patton, Kevin Spacey, Jim Carrey, John Cusack, Bill Maher, Johnny Knoxville, Chelsea Handler, Eva Longoria, Spike Lee, Enrique Iglesias and Katie Couric. Clients are not charged for the service, and are given a share of any revenue that is generated by advertisements. They are also given the ability share in the database of e-mail addresses that come with registration, in order to communicate directly with fans. Actor Dylan McDermott was introduced to WhoSay by his agent, as a way of easily posting content to Facebook, Twitter, Tumblr and even China's Tencent social network with relative ease. McDermott comments, "When you put something out there, you can hit everything at one time. It makes it easy for me." Comedian Chris Rock has commented that WhoSay is ideal for people like him have developed difficulty in keeping track of different websites as they get older, saying, "It's perfect for someone that's not 25." In September 2013 WhoSay introduced a mobile application for consumers. By October 2013, the company's website attracted 12 million monthly visitors. In July 2014 Rob Gregory left his role as president of Newsweek's The Daily Beast to become WhoSay's chief revenue officer. Among his responsibilities are developing ways to monetize WhoSay's web and mobile products, such as premium advertising strategies and brand partnerships. WhoSay does not allow consumers to create accounts, nor does it include search features, making it difficult to access a celebrity's account unless a user is directed there from one of their other social pages. According to Ellis, consumers have enough social media choices, saying, "Frankly they don't really need the services that we provide, and there are a lot of very specific features built into our service that really only benefit someone who is of a high profile." By February 2015, WhoSay had amassed 4.8 million unique users, and expanded its accounts to companies that employ celebrities for branded content. Such companies include Lexus, which partnered with the company to promote a campaign in which actress Rosario Dawson, during the lead up to the 87th Academy Awards, released five short videos on her social media accounts. The videos feature her driving through Los Angeles in preparation for the grand opening of her pop-up store, which sells Studio One Eighty Nine, a clothing line tied to her foundation promoting African culture and content. That April, WhoSay partnered with Chevrolet's #BestDayEver social media campaign for April Fool's Day, enlisting Olivia Wilde, Norman Reedus, Alec Baldwin, Ian Somerhalder, and Nikki Reed to surprise students in four U.S. classrooms as their substitute teachers. For example, Baldwin, dressed as Abraham Lincoln, surprised students in an Occidental College class on U.S. Culture and Society. Other companies that WhoSay has partnered with include KFC, JCPenney, Dunkin' Donuts and Crest. In January 2018, the website was acquired by Viacom (now Paramount Global).
Cognos ReportNet
Cognos ReportNet (CRN) was a web-based software product for creating and managing ad hoc and custom-made reports. ReportNet was developed by the Ottawa-based company Cognos (formerly Cognos Incorporated), an IBM company. The web-based reporting tool was launched in September 2003. Since IBM's acquisition of Cognos, ReportNet has been renamed IBM Cognos ReportNet like all other Cognos products. ReportNet uses web services standards such as XML and Simple Object Access Protocol and also supports dynamic HTML and Java. ReportNet is compatible with multiple databases including Oracle, SAP, Teradata, Microsoft SQL server, DB2 and Sybase. The product provides interface in over 10 languages, has Web Services architecture to meet the needs of multi-national, diversified enterprises and helps reduce total cost of ownership. Multiple versions of Cognos ReportNet have since been released by the company. Cognos ReportNet was awarded the Software and Information Industry Association (SIIA) 2005 Codie awards for the "Best Business Intelligence or Knowledge Management Solution" category. CRN's capabilities have been further used in IBM Cognos 8 BI (2005), the latest reporting tool. CRN comes with its own software development kit (SDK). == Launch == Early adopters of Cognos ReportNet for their corporate reporting needs included Bear Stearns, BMW and Alfred Publishing. Around this same time of launch, Cognos competitor Business Objects released version 6.1 of its enterprise reporting tool. Cognos ReportNet has been successful since its launch, raising revenues in 2004 from licensing fees. == Controversy == Cognos rival Business Objects announced in 2005 that BusinessObjects XI significantly outperformed Cognos ReportNet in benchmark tests conducted by VeriTest, an independent software testing firm. The tests performed showed Cognos ReportNet performed poorly when processing styled reports, complex business reports and combination of both. The tests reported a massive 21 times higher report throughput for BusinessObjects XI than Cognos ReportNet at capacity loads. Cognos soon dismissed the claims by stating Business Objects dictated the environment and testing criteria and Cognos did not provide the software to participate in benchmark test. Cognos later performed their own test to demonstrate Cognos ReportNet capabilities. == Components == Cognos Report Studio – A Web-based product for creating complex professional looking reports. Cognos Query Studio - A Web-based product for creating ad-hoc reports. Cognos Framework Manager – A metadata modeling tool to create BI metadata for reporting and dashboard applications. Cognos Connection – Main portal used to access reports, schedule reports and perform administrator activities. == Versions == Cognos ReportNet 1.1 – Java EE-style professional web-based authoring tool. (base version) Cognos ReportNet IBM Special Edition – comes with an embedded version of IBM WebSphere as its application server and IBM DB2 as its data store. Cognos Linux – for Intel-based Linux platforms.
Learning to rank
Learning to rank (LTR) or machine-learned ranking (MLR) is the application of machine learning, often supervised, semi-supervised or reinforcement learning, in the construction of ranking models for information retrieval and recommender systems. Training data may, for example, consist of lists of items with some partial order specified between items in each list. This order is typically induced by giving a numerical or ordinal score or a binary judgment (e.g. "relevant" or "not relevant") for each item. The goal of constructing the ranking model is to rank new, unseen lists in a similar way to rankings in the training data. == Applications == === In information retrieval === Ranking is a central part of many information retrieval problems, such as document retrieval, collaborative filtering, sentiment analysis, and online advertising. A possible architecture of a machine-learned search engine is shown in the accompanying figure. Training data consists of queries and documents matching them together with the relevance degree of each match. It may be prepared manually by human assessors (or raters, as Google calls them), who check results for some queries and determine relevance of each result. It is not feasible to check the relevance of all documents, and so typically a technique called pooling is used — only the top few documents, retrieved by some existing ranking models are checked. This technique may introduce selection bias. Alternatively, training data may be derived automatically by analyzing clickthrough logs (i.e. search results which got clicks from users), query chains, or such search engines' features as Google's (since-replaced) SearchWiki. Clickthrough logs can be biased by the tendency of users to click on the top search results on the assumption that they are already well-ranked. Training data is used by a learning algorithm to produce a ranking model which computes the relevance of documents for actual queries. Typically, users expect a search query to complete in a short time (such as a few hundred milliseconds for web search), which makes it impossible to evaluate a complex ranking model on each document in the corpus, and so a two-phase scheme is used. First, a small number of potentially relevant documents are identified using simpler retrieval models which permit fast query evaluation, such as the vector space model, Boolean model, weighted AND, or BM25. This phase is called top- k {\displaystyle k} document retrieval and many heuristics were proposed in the literature to accelerate it, such as using a document's static quality score and tiered indexes. In the second phase, a more accurate but computationally expensive machine-learned model is used to re-rank these documents. === In other areas === Learning to rank algorithms have been applied in areas other than information retrieval: In machine translation for ranking a set of hypothesized translations; In computational biology for ranking candidate 3-D structures in protein structure prediction problems; In recommender systems for identifying a ranked list of related news articles to recommend to a user after he or she has read a current news article. == Feature vectors == For the convenience of MLR algorithms, query-document pairs are usually represented by numerical vectors, which are called feature vectors. Such an approach is sometimes called bag of features and is analogous to the bag of words model and vector space model used in information retrieval for representation of documents. Components of such vectors are called features, factors or ranking signals. They may be divided into three groups (features from document retrieval are shown as examples): Query-independent or static features — those features, which depend only on the document, but not on the query. For example, PageRank or document's length. Such features can be precomputed in off-line mode during indexing. They may be used to compute document's static quality score (or static rank), which is often used to speed up search query evaluation. Query-dependent or dynamic features — those features, which depend both on the contents of the document and the query, such as TF-IDF score or other non-machine-learned ranking functions. Query-level features or query features, which depend only on the query. For example, the number of words in a query. Some examples of features, which were used in the well-known LETOR dataset: TF, TF-IDF, BM25, and language modeling scores of document's zones (title, body, anchors text, URL) for a given query; Lengths and IDF sums of document's zones; Document's PageRank, HITS ranks and their variants. Selecting and designing good features is an important area in machine learning, which is called feature engineering. == Evaluation measures == There are several measures (metrics) which are commonly used to judge how well an algorithm is doing on training data and to compare the performance of different MLR algorithms. Often a learning-to-rank problem is reformulated as an optimization problem with respect to one of these metrics. Examples of ranking quality measures: Mean average precision (MAP); DCG and NDCG; Precision@n, NDCG@n, where "@n" denotes that the metrics are evaluated only on top n documents; Mean reciprocal rank; Kendall's tau; Spearman's rho. DCG and its normalized variant NDCG are usually preferred in academic research when multiple levels of relevance are used. Other metrics such as MAP, MRR and precision, are defined only for binary judgments. Recently, there have been proposed several new evaluation metrics which claim to model user's satisfaction with search results better than the DCG metric: Expected reciprocal rank (ERR); Yandex's pfound. Both of these metrics are based on the assumption that the user is more likely to stop looking at search results after examining a more relevant document, than after a less relevant document. == Approaches == Learning to Rank approaches are often categorized using one of three approaches: pointwise (where individual documents are ranked), pairwise (where pairs of documents are ranked into a relative order), and listwise (where an entire list of documents are ordered). Tie-Yan Liu of Microsoft Research Asia has analyzed existing algorithms for learning to rank problems in his book Learning to Rank for Information Retrieval. He categorized them into three groups by their input spaces, output spaces, hypothesis spaces (the core function of the model) and loss functions: the pointwise, pairwise, and listwise approach. In practice, listwise approaches often outperform pairwise approaches and pointwise approaches. This statement was further supported by a large scale experiment on the performance of different learning-to-rank methods on a large collection of benchmark data sets. In this section, without further notice, x {\displaystyle x} denotes an object to be evaluated, for example, a document or an image, f ( x ) {\displaystyle f(x)} denotes a single-value hypothesis, h ( ⋅ ) {\displaystyle h(\cdot )} denotes a bi-variate or multi-variate function and L ( ⋅ ) {\displaystyle L(\cdot )} denotes the loss function. === Pointwise approach === In this case, it is assumed that each query-document pair in the training data has a numerical or ordinal score. Then the learning-to-rank problem can be approximated by a regression problem — given a single query-document pair, predict its score. Formally speaking, the pointwise approach aims at learning a function f ( x ) {\displaystyle f(x)} predicting the real-value or ordinal score of a document x {\displaystyle x} using the loss function L ( f ; x j , y j ) {\displaystyle L(f;x_{j},y_{j})} . A number of existing supervised machine learning algorithms can be readily used for this purpose. Ordinal regression and classification algorithms can also be used in pointwise approach when they are used to predict the score of a single query-document pair, and it takes a small, finite number of values. === Pairwise approach === In this case, the learning-to-rank problem is approximated by a classification problem — learning a binary classifier h ( x u , x v ) {\displaystyle h(x_{u},x_{v})} that can tell which document is better in a given pair of documents. The classifier shall take two documents as its input and the goal is to minimize a loss function L ( h ; x u , x v , y u , v ) {\displaystyle L(h;x_{u},x_{v},y_{u,v})} . The loss function typically reflects the number and magnitude of inversions in the induced ranking. In many cases, the binary classifier h ( x u , x v ) {\displaystyle h(x_{u},x_{v})} is implemented with a scoring function f ( x ) {\displaystyle f(x)} . As an example, RankNet adapts a probability model and defines h ( x u , x v ) {\displaystyle h(x_{u},x_{v})} as the estimated probability of the document x u {\displaystyle x_{u}} has higher quality than x v {\displaystyle x_{v}} : P u , v ( f ) = CDF ( f ( x u ) − f ( x v ) ) , {\displaystyle P_{u,v}(f)={\text{CDF}
Data exchange
Data exchange is the process of moving data from one information system to another. It often involves transforming data that is native to the source system into a form that is consumable by the target system or to a standardized form that is consumable by any compatible system. In particular, data exchange allows data to be shared between computer programs. Data exchange is similar to data integration except that data may be restructured with possible loss of content. There may be no way to transform a particular collection based on exchange constraints. Conversely, there may be multiple ways to transform the data, in which case one option must be identified in order to achieve compatibility between source and target. There are two main types of data exchange: broadcast and peer-to-peer (a.k.a. unicast). For broadcast, data is transmitted simultaneously to all consumers. Just as a conference call, all participants get the same information from the speaker at the same time. For peer-to-peer, data is sent to a single receiver, defined by a specific address. For example, a letter goes to just one mail box. == Single-domain == In some domains, a multiple source and target schema (proprietary data formats) may exist. An exchange or interchange format is often developed for a single domain, and then necessary routines (mappings) are written to (indirectly) transform/translate each and every source schema to each and every target schema by using the interchange format as an intermediate step. That requires less work than writing and debugging the many routines that would be required to directly translate each source schema directly to each target schema. Examples of these transformative interchange formats include: Standard Interchange Format for geospatial data; Data Interchange Format for spreadsheet data; Open Document Format for spreadsheets, charts, presentations and word processing documents; GPS eXchange Format or Keyhole Markup Language for describing GPS data; GDSII for integrated circuit layout. == Representation == A data exchange (a.k.a. interchange) language defines a domain-independent way to represent data. These languages have evolved from being markup and display-oriented to support the encoding of metadata that describes the structural attributes of the information. Practice has shown that certain types of formal languages are better suited for this task than others, since their specification is driven by a formal process instead of particular software implementation. For example, XML is a markup language that was designed to enable the creation of dialects (the definition of domain-specific sublanguages). However, it does not contain domain-specific dictionaries or fact types. Beneficial to a reliable data exchange is the availability of standard dictionaries-taxonomies and tools libraries such as parsers, schema validators, and transformation tools. === XML === The popularity of XML for data exchange on the World Wide Web has several reasons. First of all, it is closely related to the preexisting standards Standard Generalized Markup Language (SGML) and Hypertext Markup Language (HTML), and as such a parser written to support these two languages can be easily extended to support XML as well. For example, XHTML has been defined as a format that is formal XML, but understood correctly by most (if not all) HTML parsers. === YAML === YAML was designed to be human-readable and authored via a text editor with notion similar to reStructuredText and wiki syntax. YAML 1.2 also includes a shorthand notion that is compatible with JSON, and as such any JSON document is also valid YAML; this however does not hold the other way. === REBOL === REBOL was designed to be human-readable and authored via a text editor. It uses a simple free-form syntax with minimal punctuation and a rich set of data types (such as URL, email, date and time, tuple, string, tag) that respect common standards. It is designed to not need any additional meta-language, being designed in a metacircular fashion which is why the parse dialect used for definitions and transformations of REBOL dialects is also itself a dialect of REBOL. REBOL was used as a source of inspiration for JSON. === Gellish === Gellish English is a formalized subset of natural English (language), which includes a simple grammar and a large, extensible dictionary (taxonomy) that defines the general and domain specific terminology, whereas the concepts are arranged in a hierarchy, which supports inheritance of knowledge and requirements. The dictionary also includes standardized fact types. The terms and relation types together can be used to create and interpret expressions of facts, knowledge, requirements and other information. Gellish can be used in combination with SQL, RDF/XML, OWL and various other meta-languages. The Gellish standard is a combination of ISO 10303-221 (AP221) and ISO 15926. === List === The following describes and compares popular data exchange languages. Columns Schemas – Whether supports representing domain specific data structure definition Flexible – Whether supports extension of the semantic expression capabilities without modifying the schema Semantic verification – Whether supports semantic verification of the correctness of expressions in the language Dictionary – Whether includes a dictionary and a taxonomy (hierarchy) of concepts with inheritance Information model – Whether supports an information model Synonyms and homonyms – Whether supports the use of synonyms and homonyms in expressions Dialecting – Whether is available in multiple natural languages or dialects Web standard – Whether is standardized by a recognized body Transformations – Whether includes a translation to other standards Lightweight – Whether a lightweight version is available Human readable – Whether expressions are understandable without training Compatibility – Which other tools can be used or are required