Airborne Networking

An Airborne Network (AN) is the infrastructure owned by the United States Air Force that provides communication transport services through at least one node that is on a platform capable of flight. == Background == === Definition === The intent of the US Air Force's Airborne Network is to expand the Global Information Grid (GIG) to connect the three major domains of warfare: Air, Space, and Terrestrial. The Transformational Satellite Communications System network currently provides connectivity for all communication through space assets. The Combat Information Transport System and Theater Deployable Communications provide terrestrial connectivity for theatre based operations. The Airborne Network is engineered to utilize all airborne assets to connect with space and surface networks building a seamless communications platform across all domains. === Capabilities === The capabilities identified by this type of system are vastly beyond that of our current military. This system will enable the Air Force to provide a transportable network, flexible enough to communicate with any air, space, or ground asset in the area. The network will provide a beyond line-of-sight (LoS) communications infrastructure that can be packed up and moved in and out of the designated battlespace, enabling the military to have a reliable and secure communications network that extends globally. The network is designed to be flexible enough to provide the right communication and network packages for a specific region, mission, or technology. Operationally, The AN is designed to be self-forming, self-organizing, and self-generating, with nodes joining and leaving the network as they enter and exit a specific region. The network consists of dedicated tactical links, wideband air-to-air links, and ad hoc networks constructed by the Joint Tactical Radio System (JTRS) networking services. JTRS is a software-defined radio that will work with many existing military and civilian radios. It includes integrated encryption and Wideband Networking Software to create mobile ad hoc networks. It also provides system performance analysis and fault diagnostics automatically, reducing the demand for human intervention and network maintenance. === Intended Use === The AN was designed as the cornerstone for the new military doctrine known as Network Centric Warfare. This doctrine was developed to use information superiority to equip warfighters with more precise information enabling commanders and shooters to make smarter decisions faster. The AN contributes to Network Centric Warfare by enabling commanders to provide real-time information to warfighters in the air and on the ground. Warfighters can then utilize more information and make more educated decisions about how to act in a particular situation. Once the act has been carried out commanders will have immediate information about the result and can make judgments on how to continue. All-in-all the AN was designed to reduce the time necessary to identify a target, make clear and educated decisions to pull or not to pull the trigger, and assess battle == Topologies == There are four main network topologies that will be deployed and vary based on the placement of backbone and subnet class networks. === Space, Air, Ground Tether === Establishing a direct connection to another aircraft or ground node, via a point-to-point link for nodes within LOS or via a Satellite Communications (SATCOM) link for nodes that are beyond line-of-sight is known as tethering. SATCOM links provide connectivity to a network ground entry point. Strike aircraft that accompany C2 aircraft such as an AWACS are tethered via point-to-point links. Finally, C2 or intelligence, surveillance, and reconnaissnce (ISR) aircraft may connect via a LOS link directly to a network ground entry point. Each of these tethered alternatives works exactly like a hub or switch that has an entry point to a larger network and allows their connected users access to that network. === Flat Ad Hoc === A flat ad hoc topology refers to establishing nonpersistent network connections as needed among AN nodes that are present at a given time. With this network the nodes dynamically “discover” other nodes to which they can interconnect and form the network. The specific interconnections between the nodes are not planned in advance, but are made as opportunities arise. The nodes join and leave the network at will, continually changing connections to neighbor nodes based upon their location and mobility characteristics. === Tiered Ad Hoc === Ad hoc networks can be flat in the sense that all nodes are peers of each other in a single network, as discussed above, or they can dynamically organize themselves into hierarchical tiers such that higher tiers are used to move data between more localized subnets. This network topology can be compared to any conventional deployed network that utilizes routers, switches, and hubs to temporarily connect users. === Persistent Backbone === A network topology characterized by a persistent backbone is established using relatively persistent wideband connections among high-value platforms flying relatively stable orbits. It provides the connectivity between the tactical subnets which are considered edge networks relative to the backbone. This provides concentration points for connectivity to the space backbone as well as to terrestrial networks. This type of network topology is comparable to a conventional permanent network with established data trunks, routers, switches, and hubs to connect users. == Architecture == === Network Management === The platform management system enables operators to manage all on-board network elements. It interfaces and interoperates with the Airborne Network management system to enable operators to manage remote network elements in the airborne network. The network management system monitors the health of the network by passively testing the network for faults and latency. The system will also actively troubleshoot faults with probes to identify and isolate faulty connections, and enables operators to apply network parameters and security changes to all systems based on the status of the network. === Routing/Switching === Routing and switching enables data to be dynamically transmitted over the network to other nodes. Routing protocols must be able to identify nodes transmitted within their own platform and data to be sent to other platforms regardless of the current topology. The routing protocol must also provide seamless roaming by ensuring that no routed packets are lost when a node changes its point of attachment to the network. Maintaining scalability is important in routing as the network is constantly changing. The network must be able to function with numerous levels of platforms, varying numbers of fast moving platforms, and varying amounts of traffic per platform. Routers and switches will use metrics to determine the best paths to take when routing data. The routing protocol utilized for the AN will be an Adaptive Quality of Service routing protocol. === Gateways/Proxies === Gateways and proxies enable the connection numerous technology types regardless of age to communicate across the IP-based network. Gateways and proxies are essential in the operation of this network because so many different technologies are used to communicate in each domain. These systems will facilitate the transition of the legacy on-board infrastructure, transmission systems, tactical data link systems, and user applications to the objective airborne network systems. Therefore, they are only temporary until all platforms use a standardized IP radio for transmission. === Performance Enhancing Proxies === Performance Enhancing Proxies improve the performance of user applications running across the Airborne Network by countering wireless network impairments, such as limited bandwidth, long delays, high loss rates, and disruptions in network connections. Proxy systems are implemented between the user application and the network and can be used to improve performance at the application and transport functional layers of the OSI model. Some techniques that can be employed include: Compression: Data compression or header compression can be used to minimize the number of bits sent over the network. Data bundling: Smaller data packets can be combined (bundled) into a single large packet for transmission over the network. Caching: A local cache can be used to save and provide data objects that are requested multiple times, reducing transmissions over the network (and improving response times). Store and forward: Message queuing can be used to ensure message delivery to users who become disconnected from the network or are unable to connect to the network for a period of time. Once the platform connects, the stored messages are sent. Pipelining: Rather than opening several separate network connections pipelining can be used to share a single networ

Digital Image Processing with Sound

DIPS (Digital Image Processing with Sound) is a set of plug-in objects that handle real-time digital image processing in Max/MSP programming environment. Combining with the built-in objects of the environment, DIPS enables to program the interaction between audio and visual events with ease, and supports the realization of interactive multimedia art as well as interactive computer music. == Summary of Features == A plug-in software for Max/MSP (Max 5 and 6) More than 300 Max external objects and abstractions More than 90 OpenGL objects included More than 110 visual effect objects (Dfx library, Core Image Filters) A utility library for the easy of programming (prefix Dlib) A comprehensive set of sample patches, and a detailed tutorial Handling images & movie files (QuickTime, OpenGL) Render and move 3D models (OpenGL) Video signal input (QuickTime, video texture) Video input analysis: motion detect, face tracking (OpenCV, OpenGL) Importing 3D models (.obj file) Importing Quartz Composer files OpenGL Shading Language (GLSL) programming interface Easy integration of visual events using DIPSWindowMixer (OpenGL) == Description == DIPS is a free plug-in software (a set of external objects) for Max/MSP. It supports the designing of the interaction between sound and visual events in Max using Apple’s Core Image, OpenGL and OpenCV technologies, and consequently, provides a powerful and user-friendly programming environment for the creation of interactive multimedia art. DIPS can be used to detect a performer’s motions and to track positions of subtle details, such as the face, mouth, and eyes. It can also be used to measure the distance between objects and a Kinect sensor system, and offers powerful tools for realtime image processing of incoming video stream and stored movie files. In addition, it can be used to create complex images in a virtual three-dimensional space. The DIPS consists of a library of more than 300 Max external objects and abstractions, a comprehensive set of sample patches, and a detailed tutorial. Some of its strong points, in comparison with other similar plug-ins and software, are its ease of programming, power, and efficiency. The sample patches and tutorial contained in the installation package allows composers and artists who are interested in the creation of interactive art to realize sophisticated realtime video effects on a live video signal at their first practice. And because of its ease of programming, it is likely that one will soon acquire skills needed to create state-of-the-art interactive performance works, multimedia installations, interactive multimedia artworks, and Max VJ applications using DIPS. == History == Initially developed by Shu Matsuda in 1997, DIPS was a plug-in software for Max/FTS running on SGI Octane and O2 computers. Since 2000, it has been developed by the DIPS Development Group supervised by Takayuki Rai. Current active group members are Shu Matsuda, Yota Morimoto, Takuto Fukuda, and Keitaro Takahashi. Previously, Chikashi Miyama, Daichi Ando and Takayuki Hamano also contributed to its development. 2013 DIPS5 for Max (Mac OS X) 2009 DIPS4 for Max/MSP (Mac OS X) 2006 DIPS3 for Max/MSP (Mac OS X) 2003 DIPS2 for jMax4 (Mac OS X) 2002 DIPS for jMax2 (Mac OS X & Linux) 2000 DIPS for jMax (Linux)

Data deduplication

In computing, data deduplication is a technique for eliminating duplicate copies of repeating data. Successful implementation of the technique can improve storage utilization, which may in turn lower capital expenditure by reducing the overall amount of storage media required to meet storage capacity needs. It can also be applied to network data transfers to reduce the number of bytes that must be sent. The deduplication process requires comparison of data 'chunks' (also known as 'byte patterns') which are unique, contiguous blocks of data. These chunks are identified and stored during a process of analysis, and compared to other chunks within existing data. Whenever a match occurs, the redundant chunk is replaced with a small reference that points to the stored chunk. Given that the same byte pattern may occur dozens, hundreds, or even thousands of times (the match frequency is dependent on the chunk size), the amount of data that must be stored or transferred can be greatly reduced. A related technique is single-instance (data) storage, which replaces multiple copies of content at the whole-file level with a single shared copy. While possible to combine this with other forms of data compression and deduplication, it is distinct from newer approaches to data deduplication (which can operate at the segment or sub-block level). Deduplication is different from data compression algorithms, such as LZ77 and LZ78. Whereas compression algorithms identify redundant data inside individual files and encodes this redundant data more efficiently, the intent of deduplication is to inspect large volumes of data and identify large sections – such as entire files or large sections of files – that are identical, and replace them with a shared copy. == Functioning principle == For example, a typical email system might contain 100 instances of the same 1 MB (megabyte) file attachment. Each time the email platform is backed up, all 100 instances of the attachment are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; the subsequent instances are referenced back to the saved copy for deduplication ratio of roughly 100 to 1. Deduplication is often paired with data compression for additional storage saving: Deduplication is first used to eliminate large chunks of repetitive data, and compression is then used to efficiently encode each of the stored chunks. In computer code, deduplication is done by, for example, storing information in variables so that they don't have to be written out individually but can be changed all at once at a central referenced location. Examples are CSS classes and named references in MediaWiki. == Benefits == Storage-based data deduplication reduces the amount of storage needed for a given set of files. It is most effective in applications where many copies of very similar or even identical data are stored on a single disk. In the case of data backups, which routinely are performed to protect against data loss, most data in a given backup remain unchanged from the previous backup. Common backup systems try to exploit this by omitting (or hard linking) files that haven't changed or storing differences between files. Neither approach captures all redundancies, however. Hard-linking does not help with large files that have only changed in small ways, such as an email database; differences only find redundancies in adjacent versions of a single file (consider a section that was deleted and later added in again, or a logo image included in many documents). In-line network data deduplication is used to reduce the number of bytes that must be transferred between endpoints, which can reduce the amount of bandwidth required. See WAN optimization for more information. Virtual servers and virtual desktops benefit from deduplication because it allows nominally separate system files for each virtual machine to be coalesced into a single storage space. At the same time, if a given virtual machine customizes a file, deduplication will not change the files on the other virtual machines—something that alternatives like hard links or shared disks do not offer. Backing up or making duplicate copies of virtual environments is similarly improved. == Classification == === Post-process versus in-line deduplication === Deduplication may occur "in-line", as data is flowing, or "post-process" after it has been written. With post-process deduplication, new data is first stored on the storage device and then a process at a later time will analyze the data looking for duplication. The benefit is that there is no need to wait for the hash calculations and lookup to be completed before storing the data, thereby ensuring that store performance is not degraded. Implementations offering policy-based operation can give users the ability to defer optimization on "active" files, or to process files based on type and location. One potential drawback is that duplicate data may be unnecessarily stored for a short time, which can be problematic if the system is nearing full capacity. Alternatively, deduplication hash calculations can be done in-line: synchronized as data enters the target device. If the storage system identifies a block which it has already stored, only a reference to the existing block is stored, rather than the whole new block. The advantage of in-line deduplication over post-process deduplication is that it requires less storage and network traffic, since duplicate data is never stored or transferred. On the negative side, hash calculations may be computationally expensive, thereby reducing the storage throughput. However, certain vendors with in-line deduplication have demonstrated equipment which performs in-line deduplication at high rates. Post-process and in-line deduplication methods are often heavily debated. === Data formats === The SNIA Dictionary identifies two methods: Content-agnostic data deduplication – a data deduplication method that does not require awareness of specific application data formats. Content-aware data deduplication – a data deduplication method that leverages knowledge of specific application data formats. === Source versus target deduplication === Another way to classify data deduplication methods is according to where they occur. Deduplication occurring close to where data is created, is referred to as "source deduplication". When it occurs near where the data is stored, it is called "target deduplication". Source deduplication ensures that data on the data source is deduplicated. This generally takes place directly within a file system. The file system will periodically scan new files creating hashes and compare them to hashes of existing files. When files with same hashes are found then the file copy is removed and the new file points to the old file. Unlike hard links however, duplicated files are considered to be separate entities and if one of the duplicated files is later modified, then using a system called copy-on-write a copy of that changed file or block is created. The deduplication process is transparent to the users and backup applications. Backing up a deduplicated file system will often cause duplication to occur resulting in the backups being bigger than the source data. Source deduplication can be declared explicitly for copying operations, as no calculation is needed to know that the copied data is in need of deduplication. This leads to a new form of link on file systems, called a reference-counted link, or reflink, in some systems (e.g. Linux), or a cloned file on macOS, where one or more inodes (file information entries) are made to share some or all of their data. It is named analogously to hard links, which work at the inode level, and symbolic links, which work at the filename level.The individual entries have a copy-on-write behavior that is non-aliasing, i.e. changing one copy afterwards will not affect other copies. Microsoft's ReFS also supports this operation. Target deduplication is the process of removing duplicates when the data was not generated at that location. Example of this would be a server connected to a SAN/NAS, The SAN/NAS would be a target for the server (target deduplication). The server is not aware of any deduplication, the server is also the point of data generation. A second example would be backup. Generally this will be a backup store such as a data repository or a virtual tape library. === Deduplication methods === One of the most common forms of data deduplication implementations works by comparing chunks of data to detect duplicates. For that to happen, each chunk of data is assigned an identification, calculated by the software, typically using cryptographic hash functions. In many implementations, the assumption is made that if the identification is identical, the data is identical, even though this cannot be true in all cases due to the pigeonhole principle; other implementations do not as

Social media coverage of the Olympics

Over the years, television broadcast rights have distinguished what Olympic-related content can be accessed by fans online. By doing so, mobile-friendly social platforms began to integrate into the Olympics. Athletes and fans use these platforms to share live updates, special moments, and behind-the-scenes specials. Various social media platforms have been used for Olympic content, including Twitter and Facebook. Some marketers credit social media for prompting the official U.S. broadcasters, NBC, to live stream events, including early rounds. == Background == The Olympics is able to advertise to its viewers and its host country with the use of data it collects through Social media marketing. Prominent social media platforms include: Twitter, Facebook, Instagram, Tumblr, YouTube, Google, MSN, Yahoo and many more. Campaign Initiatives and Artificial Intelligence technologies have been used to analyze the social media content of users. Information from consumers such as their preferences, demographics, age and locality are all analyzed to gain consumer insight. Campaign initiatives and AI technologies were used for such purposes in the 2010 Vancouver Winter Olympics and are in use currently. Social media marketing of the Olympics is a new phenomena, beginning prior to the 2008 Beijing Olympics == Variations == There are two classifications of social media marketing recognized by the IOC: Officially sanctioned content from rights holders and sponsors that maximizes the use of Olympic content (imagery, hashtag) Unofficial content that is generated by brands that leverage the excitement of the Olympics == 2008 Beijing Summer Olympics == Social media marketing emerged as a phenomenon during the 2008 Beijing Olympics, which progressed as a marketing and an advertising tactic ever since. The Beijing Olympics became the test subject for social media marketing initiatives started by advertising agencies. In 2008, social media marketing began the transition from one-sided communication to mass communication of the Olympic Games. Although social media marketing of the Olympic Games began in 2008, the audience to the Olympics was still primarily reached through television–reaching an audience of 4.3 billion viewers. At the time, the viewers of the Olympic Games through Internet website platforms made up an audience of approximately 390 million individuals. What was the beginning of Olympic social media marketing, was also the beginning of a more globalized experience of the Olympic Games via social media. Twitter, now a prominent social media platform, began in 2006 and grew to three million active users by the beginning of the 2008 Beijing Olympics. Members of Facebook, another prominent social media platform, tracking the Olympic Games grew from approximately one million during the Olympic Games of Athens 2004 to 90 million during the 2008 Beijing Olympics. Social media use, in general, increased by 24 percent between 2007 and 2008–from 63 percent of U.S. adults to 87 percent of U.S. adults. == 2010 Vancouver Winter Olympics == The International Olympic Committee (IOC) deemed The Vancouver Winter Olympics as "the first social media games” based on its fan base through social media platforms. The IOC launched their Facebook page a month before the games began, attracting 1.5 million fans. Shifting to online viewing attracted a younger audience than past Olympic games with over 60 percent of Facebook fans being under 24 years of age. Athletes like Lindsey Vonn and Shaun White reached fans on social media as the platform posted behind-the-scenes coverage on their experiences. The IOC used social media to create competitions between athletes and fans streamed online. Its YouTube channel hosted a “Best of Us” challenge in which the public could compete in games with their favorite athletes, acquiring three million viewers. Photos spread across social media platforms, such as Flickr, which had 11,000 photos posted by 600 photographers, bringing a new perspective to the games. Twitter contributed constant live updates of the competitions. The IOC's Twitter following doubled to 12,000 followers during the Vancouver Olympics, creating a larger viewer population for the games. The IOC created social media guidelines as more athletes and fans got online to interact with the Olympics. Social media was still relatively new as a marketing platform, so these guidelines confused many individuals. == 2012 London Summer Olympics == The London 2012 Olympic Games succeeded in broadcasting, participation and marketing. For the first time, the IOC broadcast the Olympic Games live and on-demand through YouTube, allowing fans to access the Games anytime, anywhere through live streaming. The combination of conventional broadcasting and mobile platforms reached a global audience of 4.8 billion people. Social media soared with Facebook, Twitter and Google+, attracting 4.7 million followers. Athletes shared photographs, interacted online with fans and updated daily, either in person or via an agent. Instagram was established by 2012, making itself a premier photo-sharing platform perfect for athletes to capture their emotions. Lewis Wiltshire, head of sport for Twitter UK said, "Never before have fans had such direct access to their sporting heroes." Social media created conversation on fan opinions regarding athletes, including 962,756 total mentions of Usain Bolt, “Fastest Man in History,” who defended the 100 meter and 200 meter gold medals. Michael Phelps followed with 828,081 total mentions. Olympic sponsors were active on social media; created several campaigns to promote their brands; and inspired viewers with mass participation and personalized events. The Adidas “Take the Stage” Campaign recognized talent around the world, installing a photo booth and inviting the 550 Olympics athletes to take the stage. (IOC Marketing Report 2012). David Beckham surprised fans at the photo booth in Westfield shopping centre, gaining popularity in UK media. Coca-Cola, Acer Inc., McDonald's, Visa Inc. and several others used similar tactics of participation to attract viewers. == 2014 Sochi Winter Olympics == === Channels === The 2014 Winter Olympic Games were held in Sochi, a city in Krasnodar Krai, Russia, establishing the first “social media Olympics” for Russia. The most popular Russian social media and networking service, VK, created an Olympic page, similar to Facebook's. The Olympic VK page has 2.8 million fans and—the most popular official community on the platform. Throughout the games, VK had 54 million Olympic mentions, an average of 1.5 million per day. Numbers grew on other social media pages: more than 2 million fans joined the Olympic Facebook page, 168,101 followed the Olympic Twitter, 150,000 followed the Olympic Instagram and three million visited the Olympic website in February 2014. There were 90,000 total updates on social media by Sochi 2014 Olympians and teams. The United States was the most active country during the games logging 22,598 posts across Facebook, Twitter, and Instagram. === Engagement === With social media there is also hashtags. The most popular hashtag was #sochi2014 with almost 11,000 uses. The next top five hashtags were #wearewinter, #teamusa, #olympics, #goaus and #wirfuerD. Another popular hashtag was #Sochiproblems, depicting local struggles. Photos of the poor state of Sochi on all platforms made the games the number one trending topic one week before the opening ceremony. #SochiFail and #SochiProblems gave multiple reports of the poor living arrangements, incomplete construction, broken elevators, and polluted waters. This was one way that social media provided awareness to its users. === Media Perceptions === Media perceptions varied during the games; the Olympics was viewed as a confrontation between Eastern and Western Civilizations. The LGBT community took a stand against the games. Sponsors for the games including Coca-Cola, Mcdonald's, and P&G protested against Russian authorities and Russian anti-LGBT laws. Many protests took a stand against Russian laws, which created a discussion between human rights advocates. Advocates believed organizations should not promote certain values in western markets while supporting an anti-human rights government in another market. == 2016 Rio Summer Olympics == Social media marketing was an influential tool in the promotion and analysis of the 2016 Rio Olympics. Thomas Bach, President of the International Olympic Committee said that the power of sport demonstrates that diversity and interconnectedness can enlighten us all. With over 25,000+ sources of accredited media covering the games, the 2016 games were the most consumed Olympic games to date. Marketing for the Rio Olympics began in 2013 and ultimately lasted 3 years. There were 26 million visits to Olympic.org, the official website of the Olympic games, and over 7 billion views of official Olympic content on social media. There were o

Social media measurement

Social media measurement, also called social media controlling, is the management practice of evaluating successful social media communications of brands, companies, or other organizations. Key performance indicators may be measured by extracting information from social media channels, such as blogs, wikis, micro-blogs such as Twitter, social networking sites, or video/photo sharing websites, forums from time to time. It is also used by companies to gauge current trends in the industry. The process first gathers data from different websites and then performs analysis based on different metrics like time spent on the page, click through rate, content share, comments, text analytics to identify positive or negative emotions about the brand. Some other social media metrics include share of voice, owned mentions, and earned mentions. The social media measurement process starts with defining a goal that needs to be achieved and defining the expected outcome of the process. The expected outcome varies per the goal and is usually measured by a variety of metrics. This is followed by defining possible social strategies to be used to achieve the goal. Then the next step is designing strategies to be used and setting up configuration tools that ease the process of collecting the data. In the next step, strategies and tools are deployed in real-time. This step involves conducting Quality Assurance tests of the methods deployed to collect the data. And in the final step, data collected from the system is analyzed and if the need arises, it is refined on the run time to enhance the methodologies used. The last step ensures that the result obtained is more aligned with the goal defined in the first step. == Data Acquisition == Acquiring data from social media is in demand of an exploring the user participation and population with the purpose of retrieving and collecting so many kinds of data(ex: comments, downloads etc.). There are several prevalent techniques to acquire data such as Network traffic analysis, Ad-hoc application and Crawling Network Traffic Analysis - Network traffic analysis is the process of capturing network traffic and observing it closely to determine what is happening in the network. It is primarily done to improve the performance, security and other general management of the network. However concerned about the potential tort of privacy on the Internet, network traffic analysis is always restricted by the government. Furthermore, high-speed links are not adaptable to traffic analysis because of the possible overload problem according to the packet sniffing mechanism Ad-hoc Application - Ad-hoc application is a kind of application that provides services and games to social network users by developing the APIs offered by social network companies (Facebook Developer Platform). The infrastructure of Ad-hoc application allows the user to interact with the interface layer instead of the application servers. The API provides a path for application to access information after the user login. Moreover, the size of the data set collected vary with the popularity of the social media platform i.e. social media platforms having high number of users will have more data than platforms having less user base. Scraping is a process in which the APIs collect online data from social media. The data collected from Scraping is in raw format. However, having access to these types of data is a bit difficult because of its commercial value. Crawling - Crawling is a process in which a web crawler creates indexes of all the words in a web-page, stores them, then follows all the hyperlinks and indexes on that page and again stores them. It is the most popular technique for data acquisition and is also well known for its easy operation based on prevalent Object-Orientated Programming Language (Java or Python etc.). And most important, social network companies (YouTube, Flicker, Facebook, Instagram, etc.) are friendly to crawling techniques by providing public APIs == Applications == === For branding === Monitoring social media allows researchers to find insights into a brand's overall visibility on social media, to measure the impact of campaigns, to identify opportunities for engagement, to assess competitor activity and share of voice, and to detect impending crises. It can also provide valuable information about emerging trends and what consumers and clients think about specific topics, brands or products. This is the work of a cross-section of groups that include market researchers, PR staff, marketing teams, social-engagement, and community staff, agencies and sales teams. Several different providers have developed tools to facilitate the monitoring of a variety of social media channels - from blogging to internet video to internet forums. This allows companies to track what consumers say about their brands and actions. Companies can then react to these conversations and interact with consumers through social media platforms. === In government === Apart from commercial applications, social media monitoring has become a pervasive technique applied by public organizations and governments. Monitoring is a tradition within the public sector, and social-media monitoring provides a real-time approach to detecting and responding to social developments. Governments have come to realize the need for strategies to cope with surprises from the rapid expansion of public issues. Sobkowicz introduced a framework with three blocks of social-media opinion tracking, simulating and forecasting. It includes: real-time detection of emotions, topics and opinions information-flow modelling and agent-based simulation modeling of opinion networks Bekkers introduced the application of social media monitoring in the Netherlands. Public organizations in the Netherlands (such as the Tax Agency and the Education Ministry) have started to use social media monitoring to obtain better insights into the sentiments of target groups. On the one hand, the public sector will be enabled to provide timely and efficient answers to the public by using social media monitoring techniques, but on the other hand, they also have to deal with concerns about ethical issues such as transparency and privacy. == Quantifying social media == Social media management software (SMMS) is an application program or software that facilitates an organization's ability to successfully engage in social media across different communication channels. SMMS is used to monitor inbound and outbound conversations, support customer interaction, audit or document social marketing initiatives and evaluate the usefulness of a social media presence. It can be difficult to measure all social media conversations. Due to privacy settings and other issues, not all social media conversations can be found and reported by monitoring tools. However, whilst social media monitoring cannot give absolute figures, it can be extremely useful for identifying trends and for benchmarking, in addition to the uses mentioned above. These findings can, in turn, influence and shape future business decisions. In order to access social media data (posts, Tweets, and meta-data) and to analyze and monitor social media, many companies use software technologies built for business. These range from in-platform analytics dashboards to dedicated third-party platforms, which offer more advanced capabilities including cross-platform audience intelligence, sentiment analysis, and trend detection at scale. == Location-based == Most social media networks allow users to add a location to their posts (reference all of our feeds). The location can be classified as either 'at-the-location' or 'about-the-location'. "'At-the-location' services can be defined as services where location-based content is created at the geographic location. 'About-the-location' services can be defined as services which are referring to a particular location but the content is not necessarily created in this particular physical place." The added information available from geotagged (link to Geotagging article) posts means that they can be displayed on a map. This means that a location can be used as the start of a social media search rather than a keyword or hashtag. This has major implications for disaster relief, event monitoring, safety and security professionals since a large portion of their job is related to tracking and monitoring specific locations. == Technologies used == Various monitoring platforms use different technologies for social media monitoring and measurement. These technology providers may connect to the API provided by social platforms that are created for 3rd party developers to develop their own applications and services that access data. Facebook's Graph API is one such API that social media monitoring solution products would connect to pull data from. Some social media monitoring and analytics companies use calls to data providers each time an end-user d

Wide-column store

A wide-column store (or extensible record store) is a type of NoSQL database. It uses tables, rows, and columns, but unlike a relational database, the names and format of the columns can vary from row to row in the same table. A wide-column store can be interpreted as a two-dimensional key–value store. Google's Bigtable is one of the prototypical examples of a wide-column store. == Wide-column stores versus columnar databases == Wide-column stores such as Bigtable and Apache Cassandra are not column stores in the original sense of the term, since their two-level structures do not use a columnar data layout. In genuine column stores, a columnar data layout is adopted such that each column is stored separately on disk. Wide-column stores do often support the notion of column families that are stored separately. However, each such column family typically contains multiple columns that are used together, similar to traditional relational database tables. Within a given column family, all data is stored in a row-by-row fashion, such that the columns for a given row are stored together, rather than each column being stored separately. Wide-column stores that support column families are also known as column family databases. == Notable examples == Notable wide-column stores include: Apache Accumulo Apache Cassandra Apache HBase Bigtable DataStax Enterprise (uses Apache Cassandra) DataStax Astra DB (uses Apache Cassandra) Hypertable Azure Tables ScyllaDB

Data stream management system

A data stream management system (DSMS) is a computer software system to manage continuous data streams. It is similar to a database management system (DBMS), which is, however, designed for static data in conventional databases. A DBMS also offers a flexible query processing so that the information needed can be expressed using queries. However, in contrast to a DBMS, a DSMS executes a continuous query that is not only performed once, but is permanently installed. Therefore, the query is continuously executed until it is explicitly uninstalled. Since most DSMS are data-driven, a continuous query produces new results as long as new data arrive at the system. This basic concept is similar to complex event processing so that both technologies are partially coalescing. == Functional principle == One important feature of a DSMS is the possibility to handle potentially infinite and rapidly changing data streams by offering flexible processing at the same time, although there are only limited resources such as main memory. The following table provides various principles of DSMS and compares them to traditional DBMS. == Processing and streaming models == One of the biggest challenges for a DSMS is to handle potentially infinite data streams using a fixed amount of memory and no random access to the data. There are different approaches to limit the amount of data in one pass, which can be divided into two classes. For the one hand, there are compression techniques that try to summarize the data and for the other hand there are window techniques that try to portion the data into (finite) parts. === Synopses === The idea behind compression techniques is to maintain only a synopsis of the data, but not all (raw) data points of the data stream. The algorithms range from selecting random data points called sampling to summarization using histograms, wavelets or sketching. One simple example of a compression is the continuous calculation of an average. Instead of memorizing each data point, the synopsis only holds the sum and the number of items. The average can be calculated by dividing the sum by the number. However, it should be mentioned that synopses cannot reflect the data accurately. Thus, a processing that is based on synopses may produce inaccurate results. === Windows === Instead of using synopses to compress the characteristics of the whole data streams, window techniques only look on a portion of the data. This approach is motivated by the idea that only the most recent data are relevant. Therefore, a window continuously cuts out a part of the data stream, e.g. the last ten data stream elements, and only considers these elements during the processing. There are different kinds of such windows like sliding windows that are similar to FIFO lists or tumbling windows that cut out disjoint parts. Furthermore, the windows can also be differentiated into element-based windows, e.g., to consider the last ten elements, or time-based windows, e.g., to consider the last ten seconds of data. There are also different approaches to implementing windows. There are, for example, approaches that use timestamps or time intervals for system-wide windows or buffer-based windows for each single processing step. Sliding-window query processing is also suitable to being implemented in parallel processors by exploiting parallelism between different windows and/or within each window extent. == Query processing == Since there are a lot of prototypes, there is no standardized architecture. However, most DSMS are based on the query processing in DBMS by using declarative languages to express queries, which are translated into a plan of operators. These plans can be optimized and executed. A query processing often consists of the following steps. === Formulation of continuous queries === The formulation of queries is mostly done using declarative languages like SQL in DBMS. Since there are no standardized query languages to express continuous queries, there are a lot of languages and variations. However, most of them are based on SQL, such as the Continuous Query Language (CQL), StreamSQL and ESP. There are also graphical approaches where each processing step is a box and the processing flow is expressed by arrows between the boxes. The language strongly depends on the processing model. For example, if windows are used for the processing, the definition of a window has to be expressed. In StreamSQL, a query with a sliding window for the last 10 elements looks like follows: This stream continuously calculates the average value of "price" of the last 10 tuples, but only considers those tuples whose prices are greater than 100.0. In the next step, the declarative query is translated into a logical query plan. A query plan is a directed graph where the nodes are operators and the edges describe the processing flow. Each operator in the query plan encapsulates the semantic of a specific operation, such as filtering or aggregation. In DSMSs that process relational data streams, the operators are equal or similar to the operators of the Relational algebra, so that there are operators for selection, projection, join, and set operations. This operator concept allows the very flexible and versatile processing of a DSMS. === Optimization of queries === The logical query plan can be optimized, which strongly depends on the streaming model. The basic concepts for optimizing continuous queries are equal to those from database systems. If there are relational data streams and the logical query plan is based on relational operators from the Relational algebra, a query optimizer can use the algebraic equivalences to optimize the plan. These may be, for example, to push selection operators down to the sources, because they are not so computationally intensive like join operators. Furthermore, there are also cost-based optimization techniques like in DBMS, where a query plan with the lowest costs is chosen from different equivalent query plans. One example is to choose the order of two successive join operators. In DBMS this decision is mostly done by certain statistics of the involved databases. But, since the data of a data streams is unknown in advance, there are no such statistics in a DSMS. However, it is possible to observe a data stream for a certain time to obtain some statistics. Using these statistics, the query can also be optimized later. So, in contrast to a DBMS, some DSMS allows to optimize the query even during runtime. Therefore, a DSMS needs some plan migration strategies to replace a running query plan with a new one. === Transformation of queries === Since a logical operator is only responsible for the semantics of an operation but does not consist of any algorithms, the logical query plan must be transformed into an executable counterpart. This is called a physical query plan. The distinction between a logical and a physical operator plan allows more than one implementation for the same logical operator. The join, for example, is logically the same, although it can be implemented by different algorithms like a Nested loop join or a Sort-merge join. Notice, these algorithms also strongly depend on the used stream and processing model. Finally, the query is available as a physical query plan. === Execution of queries === Since the physical query plan consists of executable algorithms, it can be directly executed. For this, the physical query plan is installed into the system. The bottom of the graph (of the query plan) is connected to the incoming sources, which can be everything like connectors to sensors. The top of the graph is connected to the outgoing sinks, which may be for example a visualization. Since most DSMSs are data-driven, a query is executed by pushing the incoming data elements from the source through the query plan to the sink. Each time when a data element passes an operator, the operator performs its specific operation on the data element and forwards the result to all successive operators. == Examples == AURORA, StreamBase Systems, Inc. Archived 23 March 2009 at the Wayback Machine Hortonworks DataFlow IBM Streams NIAGARA Query Engine NiagaraST: A Research Data Stream Management System at Portland State University Odysseus, an open source Java-based framework for Data Stream Management Systems Pipeline DB PIPES Archived 24 December 2016 at the Wayback Machine, webMethods Business Events QStream SAS Event Stream Processing SQLstream STREAM StreamGlobe StreamInsight TelegraphCQ WSO2 Stream Processor