Adobe ImageReady was a bitmap graphics editor that was shipped with Adobe Photoshop for six years. It was available for Windows, Classic Mac OS and Mac OS X from 1998 to 2007. ImageReady was designed for web development and closely interacted with Photoshop. == Function == ImageReady was designed for web development rather than effects-intensive photo manipulation. To that end, ImageReady has specialized features such as animated GIF creation, image compression optimization, image slicing, adding rollover effects, and HTML generation. Photoshop versions with which ImageReady was released have an "Edit in ImageReady" button that enables editing of image directly in ImageReady. ImageReady, in turn, has an "Edit in Photoshop" button. ImageReady has strong resemblances to Photoshop; it can even use the same set of Photoshop filters. One set of tools that does not resemble the Photoshop tools, however, is the Image Map set of tools, indicated by a shape or arrow with a hand that varied depending upon the version. This toolbox has several features not found in Photoshop, including: Toggle Image Map Visibility and Toggle Slice Visibility tools: toggle between showing and hiding image maps and slices, respectively Export Animation Frames as Files option: saves all or specified frames for an alternate use, e.g., to e-mail slides for review Preview Document tool: provides a preview of rollover effects in ImageReady rather than previewing them in a browser Preview in Default Browser tool: previews the image in a browser, including any rollover or animation effects Edit in Photoshop button: opens the current image in Photoshop == History == Adobe ImageReady 1.0 was released in July 1998 as a standalone application. Version 2.0 was packaged with Photoshop 5.5, and the program was included with Photoshop through version 9.0 (CS2). Starting with Photoshop 7.0, Adobe changed the version numbers of ImageReady to match. With the release of the Creative Suite 3, ImageReady was discontinued. According to Adobe, ImageReady's most popular features were merged into Photoshop. (Even before discontinuation, some of ImageReady's web optimization functionality could be found in Photoshop's Save For Web & Devices tool.) Around the same time, Adobe purchased rival software developer Macromedia, whose application Fireworks had been a competitor to ImageReady.
Human Race Machine
The Human Race Machine (HRM) is a computerized console composed of four different programs. The Human Race Machine program allows participants to see themselves with the facial characteristics of six different races: Asian, White, African, Middle Eastern, and Indian, mapped onto their own face. The Age Machine allows viewers see an aged version of his or her face. A version of this methodology has been used for over twenty years by the FBI and the National Center for Missing and Exploited Children to help locate kidnap victims and missing children. The Couples Machine combines photographs of two people in different percentages to show the appearance of their child. The Anomaly Machine lets viewers see themselves with facial anomalies. The HRM was created by artist Nancy Burson and David Kramlich; it uses morphing technology. It was shown on Oprah on 2006-02-16.
Data stream management system
A data stream management system (DSMS) is a computer software system to manage continuous data streams. It is similar to a database management system (DBMS), which is, however, designed for static data in conventional databases. A DBMS also offers a flexible query processing so that the information needed can be expressed using queries. However, in contrast to a DBMS, a DSMS executes a continuous query that is not only performed once, but is permanently installed. Therefore, the query is continuously executed until it is explicitly uninstalled. Since most DSMS are data-driven, a continuous query produces new results as long as new data arrive at the system. This basic concept is similar to complex event processing so that both technologies are partially coalescing. == Functional principle == One important feature of a DSMS is the possibility to handle potentially infinite and rapidly changing data streams by offering flexible processing at the same time, although there are only limited resources such as main memory. The following table provides various principles of DSMS and compares them to traditional DBMS. == Processing and streaming models == One of the biggest challenges for a DSMS is to handle potentially infinite data streams using a fixed amount of memory and no random access to the data. There are different approaches to limit the amount of data in one pass, which can be divided into two classes. For the one hand, there are compression techniques that try to summarize the data and for the other hand there are window techniques that try to portion the data into (finite) parts. === Synopses === The idea behind compression techniques is to maintain only a synopsis of the data, but not all (raw) data points of the data stream. The algorithms range from selecting random data points called sampling to summarization using histograms, wavelets or sketching. One simple example of a compression is the continuous calculation of an average. Instead of memorizing each data point, the synopsis only holds the sum and the number of items. The average can be calculated by dividing the sum by the number. However, it should be mentioned that synopses cannot reflect the data accurately. Thus, a processing that is based on synopses may produce inaccurate results. === Windows === Instead of using synopses to compress the characteristics of the whole data streams, window techniques only look on a portion of the data. This approach is motivated by the idea that only the most recent data are relevant. Therefore, a window continuously cuts out a part of the data stream, e.g. the last ten data stream elements, and only considers these elements during the processing. There are different kinds of such windows like sliding windows that are similar to FIFO lists or tumbling windows that cut out disjoint parts. Furthermore, the windows can also be differentiated into element-based windows, e.g., to consider the last ten elements, or time-based windows, e.g., to consider the last ten seconds of data. There are also different approaches to implementing windows. There are, for example, approaches that use timestamps or time intervals for system-wide windows or buffer-based windows for each single processing step. Sliding-window query processing is also suitable to being implemented in parallel processors by exploiting parallelism between different windows and/or within each window extent. == Query processing == Since there are a lot of prototypes, there is no standardized architecture. However, most DSMS are based on the query processing in DBMS by using declarative languages to express queries, which are translated into a plan of operators. These plans can be optimized and executed. A query processing often consists of the following steps. === Formulation of continuous queries === The formulation of queries is mostly done using declarative languages like SQL in DBMS. Since there are no standardized query languages to express continuous queries, there are a lot of languages and variations. However, most of them are based on SQL, such as the Continuous Query Language (CQL), StreamSQL and ESP. There are also graphical approaches where each processing step is a box and the processing flow is expressed by arrows between the boxes. The language strongly depends on the processing model. For example, if windows are used for the processing, the definition of a window has to be expressed. In StreamSQL, a query with a sliding window for the last 10 elements looks like follows: This stream continuously calculates the average value of "price" of the last 10 tuples, but only considers those tuples whose prices are greater than 100.0. In the next step, the declarative query is translated into a logical query plan. A query plan is a directed graph where the nodes are operators and the edges describe the processing flow. Each operator in the query plan encapsulates the semantic of a specific operation, such as filtering or aggregation. In DSMSs that process relational data streams, the operators are equal or similar to the operators of the Relational algebra, so that there are operators for selection, projection, join, and set operations. This operator concept allows the very flexible and versatile processing of a DSMS. === Optimization of queries === The logical query plan can be optimized, which strongly depends on the streaming model. The basic concepts for optimizing continuous queries are equal to those from database systems. If there are relational data streams and the logical query plan is based on relational operators from the Relational algebra, a query optimizer can use the algebraic equivalences to optimize the plan. These may be, for example, to push selection operators down to the sources, because they are not so computationally intensive like join operators. Furthermore, there are also cost-based optimization techniques like in DBMS, where a query plan with the lowest costs is chosen from different equivalent query plans. One example is to choose the order of two successive join operators. In DBMS this decision is mostly done by certain statistics of the involved databases. But, since the data of a data streams is unknown in advance, there are no such statistics in a DSMS. However, it is possible to observe a data stream for a certain time to obtain some statistics. Using these statistics, the query can also be optimized later. So, in contrast to a DBMS, some DSMS allows to optimize the query even during runtime. Therefore, a DSMS needs some plan migration strategies to replace a running query plan with a new one. === Transformation of queries === Since a logical operator is only responsible for the semantics of an operation but does not consist of any algorithms, the logical query plan must be transformed into an executable counterpart. This is called a physical query plan. The distinction between a logical and a physical operator plan allows more than one implementation for the same logical operator. The join, for example, is logically the same, although it can be implemented by different algorithms like a Nested loop join or a Sort-merge join. Notice, these algorithms also strongly depend on the used stream and processing model. Finally, the query is available as a physical query plan. === Execution of queries === Since the physical query plan consists of executable algorithms, it can be directly executed. For this, the physical query plan is installed into the system. The bottom of the graph (of the query plan) is connected to the incoming sources, which can be everything like connectors to sensors. The top of the graph is connected to the outgoing sinks, which may be for example a visualization. Since most DSMSs are data-driven, a query is executed by pushing the incoming data elements from the source through the query plan to the sink. Each time when a data element passes an operator, the operator performs its specific operation on the data element and forwards the result to all successive operators. == Examples == AURORA, StreamBase Systems, Inc. Archived 23 March 2009 at the Wayback Machine Hortonworks DataFlow IBM Streams NIAGARA Query Engine NiagaraST: A Research Data Stream Management System at Portland State University Odysseus, an open source Java-based framework for Data Stream Management Systems Pipeline DB PIPES Archived 24 December 2016 at the Wayback Machine, webMethods Business Events QStream SAS Event Stream Processing SQLstream STREAM StreamGlobe StreamInsight TelegraphCQ WSO2 Stream Processor
Public Services Network
The Public Services Network (PSN) is a UK government's high-performance network, which helps public sector organisations work together, reduce duplication and share resources. It unified the provision of network infrastructure across the United Kingdom public sector into an interconnected "network of networks" to increase efficiency and reduce overall public expenditure. It is now a legacy network and public sector organisations are being migrated to using services on the public internet. == Origins == The Public Services Network (PSN) was launched officially as part of the Transformational Government Strategy commencing in 2005, under the original name of the Public Sector Network. Prior to this, some parts of local government had already successfully implemented the concept. The Hampshire Public Services Network (HPSN) was the first PSN, launched in 1999, followed closely by Kent County Councils partnerships with the KPSN. The HPSN, encompassing all of the borough, district and unitary councils, with the County Council, as well as the Fire Services, the Isle of Wight Council and 540 schools. National PSN technical and architecture compliance criteria were established from 2007, by GDS working with local government leaders from Socitm (the Society of Information Technology Management) on the National CIO Council and the Local CIO Council. The PSN's aim was to bring public services organisations with a common interest onto a single, coherent and standards-based ‘network of networks’. This would create influence, economies of scale and a commonality of standards for secure and easy inter-connection between public service organisations. The original concept of a network of networks strategy was based upon the work already undertaken in local government and recognition of Communities of Interest (COI) within the Criminal Justice Sector during work by the Office for Criminal Justice Reform (OCJR) between 2005 and 2007 to enable data sharing across business units. In this context a COI was defined as groups of Government departments and external partners who in combination provided services within a specific area of operation and used the same data, with a similar risk profile, shared risk appetite and common governance framework. Historically each group member had implemented their own networks and standards of operation in isolation with little or no consideration as to how services and data may be shared and resulting in increased costs of operation. The Network of Networks strategy proposed within OCJR recommended the creation of specific networks based upon these Communities of Interest which were joined together through data interchange gateways supporting common standards. Under this approach networks would be arranged by data type and business functions such as Criminal Justice, Health and Social Care, Defence and Intelligence or Public Finance rather than solely on established departmental boundaries. Within a COI, trust relationships and data interchange are readily supported, enabling data sharing without a need to cross network boundaries and providing benefits of scale without the challenges and compromises intrinsic to homogeneous cross sector networks. Data is made available without a need to transport it between organisations and control is retained by the data originator. In early 2007 a group of UK Government department CTOs in conjunction with the Office for Government Commerce Buying Solutions (OGC BS) established the vision for a single commonly provided, procured and managed public sector voice and data network infrastructure to replace the multitude of separately procured and managed networks serving various segments of the UK public sector; Education, Health, Central Government, Local Government etc. In 2008 an Industry Working Group was established to document the objectives and requirements more clearly. Their report set out the architectural and commercial principles as well as anticipated security, service management, governance and transition arrangements. == Architecture == The PSN comprises a core network, the Government Conveyancing Network or GCN provided by GCN Service Providers or GCNSPs. The GCN interconnects multiple operator networks, termed Direct Network Service Providers or DNSPs. Subscriber organisations contract to a connection from a local participating DNSP, connect via that to GCN and hence onwards to other interconnected networks and services. The GCN network is entirely based on IPv4 and MPLS and the GCNSPs are not currently mandated to provide IPv6, though they should have a roadmap to implementing it if and when required. == Commercial framework == In 2010 Virgin Media Business, BT, Cable & Wireless and Global Crossing signed Deeds of Undertaking (DoU) and subsequently achieved accreditation for providing GCN and IP VPN services. In March 2012, BT, Cable & Wireless, Capita Business Services, Eircom, Fujitsu, Kcom, Level 3, Logicalis, MDNX, Thales, Updata and Virgin Media Business were successful bidders for the initial two-year PSN Connectivity framework. In June 2012, 29 companies were confirmed as suppliers of ICT services to the UK public sector under the Government's PSN Services framework contract. Apart from most of the previous suppliers, additional companies also included 2e2, Airwave Solutions, Azzurri Communications, Cassidian, CSC Computer Sciences, Computacenter, Daisy Communications, Easynet Global Services, EE, Freedom Communications, Icom Holdings, NextiraOne, PageOne Communications, Phoenix IT Group, Siemens Communications, Specialist Computer Centres, Telefónica, telent Technology Services, Uniworld Communications and Vodafone. == Governance == The PSN is managed within the Cabinet Office where it is part of the Government Digital Service. == Early implementations == There were already notable initiatives in progress in county council areas, demonstrating public sector network integration in both the Hampshire HPSN2 network and in Kent's community network. Project Pathway was established as a pilot linking these two county-wide networks, with Virgin Media Business and Global Crossing the subscriber and GCN network elements. Staffordshire County Council was the first council in England to establish a PSN that included the county's NHS Health partners. Other county councils have since followed the leads of these councils. == Transition == Centrally procured public sector networks are expected to migrate across to the PSN framework as they reach the end of their contract terms, either through an interim framework or directly. The Government Secure Intranet (GSi) contracts expired in September 2011, running on to 12 February 2012 and were replaced by the transitional Government Secure Intranet Convergence Framework (GCF). The Managed Telephony Service (MTS) contract expired on 31 December 2011 and was replaced by the Managed Telephony Convergence Framework (MTCF). == Future plan == In a blog post published on 20 January 2017, Government Digital Service announced that the Technology Leaders Network (TLN) had agreed that government was starting a journey away from the PSN. This was because using the Internet was considered suitable for the vast majority of the work that the public sector does. The blog post confirmed that the 'move was not going to happen immediately' and stated that 'there's quite a bit of work to do across the public sector to prepare for the changes'. It also stated that it was too early for a full timeline to be provided, although all PSN-connected organisations would be updated as the process evolved. The blog post confirmed that organisations that need to access services that are only available on the PSN would still need to connect to it for the time being and continue to meet its assurance requirements. In a blog post published on 16 March 2017, Government Digital Service (GDS) set out its plans for PSN assurance. The blog post confirmed that the PSN compliance process wasn't 'going anywhere, certainly for a while yet'. It explained that the TLN agreed that – as one of the only recognised, externally accredited, cross-government common assurance standards – it 'needs to live on far beyond the end of the physical PSN network'. Government Digital Service, along with the National Cyber Security Centre (NCSC) and the Cyber and Government Security Directorate, are now looking at ways to expand and reframe PSN compliance in a new context that, while retaining the assurance principles that are the basis of the existing process, will aim to improve the process. A GDS blog post titled 'The road to closing down the PSN' published on 8 September 2020 describes how the public sector will migrate away from the PSN. The Cabinet Office has set up a programme called Future Networks for Government (FN4G) to help organisations move away from the PSN.
Data exchange
Data exchange is the process of moving data from one information system to another. It often involves transforming data that is native to the source system into a form that is consumable by the target system or to a standardized form that is consumable by any compatible system. In particular, data exchange allows data to be shared between computer programs. Data exchange is similar to data integration except that data may be restructured with possible loss of content. There may be no way to transform a particular collection based on exchange constraints. Conversely, there may be multiple ways to transform the data, in which case one option must be identified in order to achieve compatibility between source and target. There are two main types of data exchange: broadcast and peer-to-peer (a.k.a. unicast). For broadcast, data is transmitted simultaneously to all consumers. Just as a conference call, all participants get the same information from the speaker at the same time. For peer-to-peer, data is sent to a single receiver, defined by a specific address. For example, a letter goes to just one mail box. == Single-domain == In some domains, a multiple source and target schema (proprietary data formats) may exist. An exchange or interchange format is often developed for a single domain, and then necessary routines (mappings) are written to (indirectly) transform/translate each and every source schema to each and every target schema by using the interchange format as an intermediate step. That requires less work than writing and debugging the many routines that would be required to directly translate each source schema directly to each target schema. Examples of these transformative interchange formats include: Standard Interchange Format for geospatial data; Data Interchange Format for spreadsheet data; Open Document Format for spreadsheets, charts, presentations and word processing documents; GPS eXchange Format or Keyhole Markup Language for describing GPS data; GDSII for integrated circuit layout. == Representation == A data exchange (a.k.a. interchange) language defines a domain-independent way to represent data. These languages have evolved from being markup and display-oriented to support the encoding of metadata that describes the structural attributes of the information. Practice has shown that certain types of formal languages are better suited for this task than others, since their specification is driven by a formal process instead of particular software implementation. For example, XML is a markup language that was designed to enable the creation of dialects (the definition of domain-specific sublanguages). However, it does not contain domain-specific dictionaries or fact types. Beneficial to a reliable data exchange is the availability of standard dictionaries-taxonomies and tools libraries such as parsers, schema validators, and transformation tools. === XML === The popularity of XML for data exchange on the World Wide Web has several reasons. First of all, it is closely related to the preexisting standards Standard Generalized Markup Language (SGML) and Hypertext Markup Language (HTML), and as such a parser written to support these two languages can be easily extended to support XML as well. For example, XHTML has been defined as a format that is formal XML, but understood correctly by most (if not all) HTML parsers. === YAML === YAML was designed to be human-readable and authored via a text editor with notion similar to reStructuredText and wiki syntax. YAML 1.2 also includes a shorthand notion that is compatible with JSON, and as such any JSON document is also valid YAML; this however does not hold the other way. === REBOL === REBOL was designed to be human-readable and authored via a text editor. It uses a simple free-form syntax with minimal punctuation and a rich set of data types (such as URL, email, date and time, tuple, string, tag) that respect common standards. It is designed to not need any additional meta-language, being designed in a metacircular fashion which is why the parse dialect used for definitions and transformations of REBOL dialects is also itself a dialect of REBOL. REBOL was used as a source of inspiration for JSON. === Gellish === Gellish English is a formalized subset of natural English (language), which includes a simple grammar and a large, extensible dictionary (taxonomy) that defines the general and domain specific terminology, whereas the concepts are arranged in a hierarchy, which supports inheritance of knowledge and requirements. The dictionary also includes standardized fact types. The terms and relation types together can be used to create and interpret expressions of facts, knowledge, requirements and other information. Gellish can be used in combination with SQL, RDF/XML, OWL and various other meta-languages. The Gellish standard is a combination of ISO 10303-221 (AP221) and ISO 15926. === List === The following describes and compares popular data exchange languages. Columns Schemas – Whether supports representing domain specific data structure definition Flexible – Whether supports extension of the semantic expression capabilities without modifying the schema Semantic verification – Whether supports semantic verification of the correctness of expressions in the language Dictionary – Whether includes a dictionary and a taxonomy (hierarchy) of concepts with inheritance Information model – Whether supports an information model Synonyms and homonyms – Whether supports the use of synonyms and homonyms in expressions Dialecting – Whether is available in multiple natural languages or dialects Web standard – Whether is standardized by a recognized body Transformations – Whether includes a translation to other standards Lightweight – Whether a lightweight version is available Human readable – Whether expressions are understandable without training Compatibility – Which other tools can be used or are required
Dave's Redistricting
Dave's Redistricting App (DRA) is an online web app originally created by Dave Bradlee that allows anyone to simulate redistricting a U.S. state's congressional and legislative districts. == Purpose == According to Bradlee, the software was designed to "put power in people's hands," and so that they "can see how the process works, so it's a little less mysterious than it was 10 years ago." Bradlee has noticed that many citizens are taking this process seriously and using his app to create legitimate redistricting maps that could be put in place. Some websites have called Bradlee the pioneer and cause of the rise of do-it-yourself redistricting. States such as Montana in 2021 allowed the general population to use it to submit redistricting proposals following the 2020 United States Census. Dave's Redistricting has frequently been mentioned as a resource that can be used to combat gerrymandering, given that the public has free access to it. Political science firms such as FiveThirtyEight have used the website to draw examples of gerrymandered districts, including on their famous Atlas of Redistricting. Dave Bradlee built the first generation of DRA. DRA 2020 is built by a small team of volunteers—Dave Bradlee, Terry Crowley, Alec Ramsay, and David Rinn—all with a shared passion for technology & democracy and all Microsoft veterans. Their mission is to empower civic organizations and citizen activists to advocate for fair congressional and legislative districts and increased transparency in the redistricting process. == Functions == Users can redraw the congressional and state legislative districts for all 50 states, the District of Columbia, and Puerto Rico using a variety of census and election datasets including Cook PVI. Maps can be optimized for different criteria. DRA 2020 added several major features to the first generation app: Sharing & collaborative editing of maps, like Google Docs Multiple statewide elections for all 50 states including the ability to import your own data Comprehensive analytics for evaluating and comparing maps Custom overlays, and Block-level editing DRA remains free to use. == Versions == 2.2: This uses Bing Maps, an outdated software that projects the districts of a single state onto a map of the United States. 2.5: After Bing Maps announced that it would no longer be updating for the foreseen future, the U.S. Map feature was removed. DRA 2020: At the end of 2018, a beta version of 2020 was released. This version that did not require Microsoft Silverlight and could be used in any web browser. DRA 2020 has been under continuous development since and is built using React (JavaScript library), Mapbox, OpenStreetMap, TypeScript, Node.js, Amazon Web Services, as well as many open source components, tools, and icons.
Bitmap index
A bitmap index is a special kind of database index that uses bitmaps. Bitmap indexes have traditionally been considered to work well for low-cardinality columns, which have a modest number of distinct values, either absolutely, or relative to the number of records that contain the data. The extreme case of low cardinality is Boolean data (e.g., does a resident in a city have internet access?), which has two values, True and False. Bitmap indexes use bit arrays (commonly called bitmaps) and answer queries by performing bitwise logical operations on these bitmaps. Bitmap indexes have a significant space and performance advantage over other structures for query of such data. Their drawback is they are less efficient than the traditional B-tree indexes for columns whose data is frequently updated: consequently, they are more often employed in read-only systems that are specialized for fast query - e.g., data warehouses, and generally unsuitable for online transaction processing applications. Some researchers argue that bitmap indexes are also useful for moderate or even high-cardinality data (e.g., unique-valued data) which is accessed in a read-only manner, and queries access multiple bitmap-indexed columns using the AND, OR or XOR operators extensively. Bitmap indexes are also useful in data warehousing applications for joining a large fact table to smaller dimension tables such as those arranged in a star schema. == Example == Continuing the internet access example, a bitmap index may be logically viewed as follows: On the left, Identifier refers to the unique number assigned to each resident, HasInternet is the data to be indexed, the content of the bitmap index is shown as two columns under the heading bitmaps. Each column in the left illustration under the Bitmaps header is a bitmap in the bitmap index. In this case, there are two such bitmaps, one for "has internet" Yes and one for "has internet" No. It is easy to see that each bit in bitmap Y shows whether a particular row refers to a person who has internet access. This is the simplest form of bitmap index. Most columns will have more distinct values. For example, the sales amount is likely to have a much larger number of distinct values. Variations on the bitmap index can effectively index this data as well. We briefly review three such variations. Note: Many of the references cited here are reviewed at (John Wu (2007)). For those who might be interested in experimenting with some of the ideas mentioned here, many of them are implemented in open source software such as FastBit, the Lemur Bitmap Index C++ Library, the Roaring Bitmap Java library and the Apache Hive Data Warehouse system. == Compression == For historical reasons, bitmap compression and inverted list compression were developed as separate lines of research, and only later were recognized as solving essentially the same problem. Software can compress each bitmap in a bitmap index to save space. There has been considerable amount of work on this subject. Though there are exceptions such as Roaring bitmaps, Bitmap compression algorithms typically employ run-length encoding, such as the Byte-aligned Bitmap Code, the Word-Aligned Hybrid code, the Partitioned Word-Aligned Hybrid (PWAH) compression, the Position List Word Aligned Hybrid, the Compressed Adaptive Index (COMPAX), Enhanced Word-Aligned Hybrid (EWAH) and the COmpressed 'N' Composable Integer SEt (CONCISE). These compression methods require very little effort to compress and decompress. More importantly, bitmaps compressed with BBC, WAH, COMPAX, PLWAH, EWAH and CONCISE can directly participate in bitwise operations without decompression. This gives them considerable advantages over generic compression techniques such as LZ77. BBC compression and its derivatives are used in a commercial database management system. BBC is effective in both reducing index sizes and maintaining query performance. BBC encodes the bitmaps in bytes, while WAH encodes in words, better matching current CPUs. "On both synthetic data and real application data, the new word aligned schemes use only 50% more space, but perform logical operations on compressed data 12 times faster than BBC." PLWAH bitmaps were reported to take 50% of the storage space consumed by WAH bitmaps and offer up to 20% faster performance on logical operations. Similar considerations can be done for CONCISE and Enhanced Word-Aligned Hybrid. The performance of schemes such as BBC, WAH, PLWAH, EWAH, COMPAX and CONCISE is dependent on the order of the rows. A simple lexicographical sort can divide the index size by 9 and make indexes several times faster. The larger the table, the more important it is to sort the rows. Reshuffling techniques have also been proposed to achieve the same results of sorting when indexing streaming data. == Encoding == Basic bitmap indexes use one bitmap for each distinct value. It is possible to reduce the number of bitmaps used by using a different encoding method. For example, it is possible to encode C distinct values using log(C) bitmaps with binary encoding. This reduces the number of bitmaps, further saving space, but to answer any query, most of the bitmaps have to be accessed. This makes it potentially not as effective as scanning a vertical projection of the base data, also known as a materialized view or projection index. Finding the optimal encoding method that balances (arbitrary) query performance, index size and index maintenance remains a challenge. Without considering compression, Chan and Ioannidis analyzed a class of multi-component encoding methods and came to the conclusion that two-component encoding sits at the kink of the performance vs. index size curve and therefore represents the best trade-off between index size and query performance. == Binning == For high-cardinality columns, it is useful to bin the values, where each bin covers multiple values and build the bitmaps to represent the values in each bin. This approach reduces the number of bitmaps used regardless of encoding method. However, binned indexes can only answer some queries without examining the base data. For example, if a bin covers the range from 0.1 to 0.2, then when the user asks for all values less than 0.15, all rows that fall in the bin are possible hits and have to be checked to verify whether they are actually less than 0.15. The process of checking the base data is known as the candidate check. In most cases, the time used by the candidate check is significantly longer than the time needed to work with the bitmap index. Therefore, binned indexes exhibit irregular performance. They can be very fast for some queries, but much slower if the query does not exactly match a bin. == History == The concept of bitmap index was first introduced by Professor Israel Spiegler and Rafi Maayan in their research "Storage and Retrieval Considerations of Binary Data Bases", published in 1985. The first commercial database product to implement a bitmap index was Computer Corporation of America's Model 204. Patrick O'Neil published a paper about this implementation in 1987. This implementation is a hybrid between the basic bitmap index (without compression) and the list of Row Identifiers (RID-list). Overall, the index is organized as a B+tree. When the column cardinality is low, each leaf node of the B-tree would contain long list of RIDs. In this case, it requires less space to represent the RID-lists as bitmaps. Since each bitmap represents one distinct value, this is the basic bitmap index. As the column cardinality increases, each bitmap becomes sparse and it may take more disk space to store the bitmaps than to store the same content as RID-lists. In this case, it switches to use the RID-lists, which makes it a B+tree index. == In-memory bitmaps == One of the strongest reasons for using bitmap indexes is that the intermediate results produced from them are also bitmaps and can be efficiently reused in further operations to answer more complex queries. Many programming languages support this as a bit array data structure. For example, Java has the BitSet class and .NET have the BitArray class. Some database systems that do not offer persistent bitmap indexes use bitmaps internally to speed up query processing. For example, PostgreSQL versions 8.1 and later implement a "bitmap index scan" optimization to speed up arbitrarily complex logical operations between available indexes on a single table. For tables with many columns, the total number of distinct indexes to satisfy all possible queries (with equality filtering conditions on either of the fields) grows very fast, being defined by this formula: C n [ n 2 ] ≡ n ! ( n − [ n 2 ] ) ! [ n 2 ] ! {\displaystyle \mathbf {C} _{n}^{\left[{\frac {n}{2}}\right]}\equiv {\frac {n!}{\left(n-\left[{\frac {n}{2}}\right]\right)!\left[{\frac {n}{2}}\right]!}}} . A bitmap index scan combines expressions on different indexes, thus requiring only one index per column t