AI Analytics Social Media

AI Analytics Social Media — independent reviews, comparisons, pricing and step-by-step guides on Aizhi.

• 

  Collaboration-oriented architecture

  Collaboration Oriented Architecture (COA) is a computer system that is designed to collaborate, or use services, from systems that are outside of the operators control. Collaboration Oriented Architecture will often use Service Oriented Architecture to deliver the technical framework. Collaboration Oriented Architecture is the ability to collaborate between systems that are based on the Jericho Forum principles or "Commandments". Bill Gates and Craig Mundie (Microsoft) clearly articulated the need for people to work outside of their organizations in a secure and collaborative manner in their opening keynote to the RSA Security Conference in February 2007. Successful implementation of a Collaboration Oriented Architecture implies the ability to successfully inter-work securely over the Internet and will typically mean the resolution of the problems that come with de-perimeterisation. == Etymology == The term Collaboration Oriented Architectures was defined and developed in a meeting of the Jericho Forum at a meeting held at HSBC on 6 July 2007. == Definition == The key elements that qualify a security architecture as a Collaboration Oriented Architecture are as follows; Protocol: Systems use appropriately secure protocols to communicate. Authentication: The protocol is authenticated with user and/or system credentials. Federation: User and/or systems credentials are accepted and validated by systems that are not under your (locus of) control. Network Agnostic: The design does not rely on a secure network, thus it will operate securely from an Intranet to raw-Internet Trust: The collaborating system have the capacity to be able to confirm to a specified degree of confidence that the components in a transaction chain have. Risk: The collaborating systems can make a risk assessment on any transaction based on the communicated levels of required trust, based on the required degree of identity, confidentiality, integrity, availability. == Authentication == Working in a collaborative multi-sourced environment implies the need for authentication, authorization and accountability which must interoperate / exchange outside of your locus / area of control. People/systems must be able to manage permissions of resources and rights of users they don't control There must be capability of trusting an organization, which can authenticate individuals or groups, thus eliminating the need to create separate identities In principle, only one instance of person / system / identity may exist, but privacy necessitates the support for multiple instances, or one instance with multiple facets, often referred to as personas Systems must be able to pass on security credentials /assertions Multiple loci (areas) of control must be supported

  Read more →
• 

  ESign (India)

  Aadhaar eSign is an online electronic signature service in India to facilitate an Aadhaar holder to digitally sign a document. The signature service is facilitated by authenticating the Aadhaar holder via the Aadhaar-based e-KYC (electronic Know Your Customer) service. To eSign a document, one has to have an Aadhaar card and a mobile number registered with Aadhaar. With these two things, an Indian citizen can sign a document remotely without being physically present. == Procedure == The notification issued by Government of India in this regard stipulates the following procedure for the e-authentication using Aadhaar e-KYC services. Authentication of an electronic record by e-authentication technique, which shall be done by the applicable use of e-authentication, hash function, and asymmetric cryptosystem techniques, leading to issuance of digital signature certificate by Certifying Authority, a trusted third party service by subscriber's key pair generation, storing of the key pairs on hardware security module and creation of digital signature provided that the trusted third party shall be offered by the certifying authority (the trusted third party shall send application form and certificate signing request to the Certifying Authority for issuing a digital signature certificate to the subscriber), issuance of digital signature certificate by Certifying Authority shall be based on e-authentication, particulars given in the prescribed format, digitally signed verified information from Aadhaar e-KYC services and electronic consent of digital signature certificate applicant, the manner and requirements for e-authentication shall be as issued by the Controller from time to time, the security procedure for creating the subscriber's key pair shall be in accordance with the e-authentication guidelines issued by the Controller, the standards referred to in rule 6 of the Information Technology (Certifying Authorities) Rules, 2000 shall be complied with, in so far as they relate to the certification function of public key of Digital Signature Certificate applicant, and the manner in which information is authenticated by means of digital signature shall comply with the standards specified in rule 6 of the Information Technology (Certifying Authorities) Rules, 2000 in so far as they relate to the creation, storage and transmission of Digital Signature. == eSign Service Providers == Organisations and individuals seeking to obtain the eSigning Service can utilize the services of various service providers. There are empanelled service providers with whom organisations can register as an Application Service Prover after submitting the requisite documents, getting UAT access, building the application around the service and going through an IT Audit by an CERT-IN empanelled auditor. However, the process of registering as an Application Service Provider is cumbersome, and requires huge investments of time, money and resources in complying with the regulations and building a suitable application. Most organisations prefer using services of plug-n-play gateway providers who take the responsibility of complying with the regulations, hence simplifying the process for the market.

  Read more →
• 

  Data exhaust

  Data exhaust (also exhaust data) is the trail of data generated as a by-product of users' online activity, behaviour, and transactions, rather than data they deliberately create or submit. It forms part of a broader category of unconventional data that also includes geospatial, network, and time-series data, and may be useful for predictive analytics. Data exhaust can take the form of cookies, temporary files, log files, clickstream records and stored preferences. Actions such as visiting a web page, following a link, or dwelling on an element may all generate exhaust data that is recorded without the user's active awareness. Unlike primary content — which the user intentionally creates — exhaust data is a passive side effect of interaction. A bank, for example, might treat the amounts and parties involved in a transaction as primary data, while secondary data could include whether the transaction was carried out at a cash machine rather than a branch. == Uses == Data exhaust collected by companies is often information that is not immediately useful in isolation, but can be aggregated and analysed to improve products, personalise content, identify trends, and support quality control. Companies may also store exhaust data for future analysis or sell it to third parties. Shoshana Zuboff has described this practice as a core mechanism of what she terms surveillance capitalism, in which behavioural data generated by users is converted into predictive products. Kosciejew notes that large quantities of often raw data are collected in this way, much of which is never analysed. == Medical exhaust data == Many medical devices — including pacemakers, dialysis machines and surgical cameras — generate exhaust data as a by-product of their operation. The majority of this data is never captured or analysed, and is typically discarded once a procedure ends or a device completes its routine monitoring cycle. The potential use of data generated by implanted devices such as pacemakers raises additional legal and ethical questions around ownership and consent. Using electronic health records for research also creates challenges because of the volume of data involved, creating a need for automated algorithms to process it. == Privacy and regulation == The collection and distribution of data exhaust is not in itself illegal in most jurisdictions, but its use raises questions of privacy and informed consent. Steps commonly taken to address these concerns include data anonymisation, offering users an opt-out from the sale of their data, and publishing explicit privacy policies that disclose what data is collected and how it is used.

  Read more →
• 

  Social Media Working Group Act of 2014

  The Social Media Working Group Act of 2014 (H.R. 4263) is a bill that would direct the United States Secretary of Homeland Security to establish within the United States Department of Homeland Security (DHS) a social media working group (the Group) to provide guidance and best practices to the emergency preparedness and response community on the use of social media technologies before, during, and after a terrorist attack. The bill was introduced into the United States House of Representatives during the 113th United States Congress. == Background == === Social media === Social media is the social interaction among people in which they create, share or exchange information and ideas in virtual communities and networks. Andreas Kaplan and Michael Haenlein define social media as "a group of Internet-based applications that build on the ideological and technological foundations of Web 2.0, and that allow the creation and exchange of user-generated content." Furthermore, social media depend on mobile and web-based technologies to create highly interactive platforms through which individuals and communities share, co-create, discuss, and modify user-generated content. They introduce substantial and pervasive changes to communication between organizations, communities, and individuals. Social media differ from traditional or industrial media in many ways, including quality, reach, frequency, usability, immediacy, and permanence. === Virtual Social Media Working Group === First responders have increasingly used social media in emergency response and recovery operations. Social media tools are used to connect with citizens after a disaster and share information. The Virtual Social Media Working group (VSMWG) is an online platform that gives advice to first responders on how to safely and effectively use social media in emergency response operations. The working group is made up of subject matter experts from across the U.S. It was created by DHS in December 2010 and gives first responders guidance and best practices regarding the use of social media during emergencies. The DHS S&T and the VSMWG work with local and state governments, academics and nonprofits. Meetings of the VSMWG are chaired by the Under Secretary of Homeland Security for Science and Technology. == Provisions of the bill == This summary is based largely on the summary provided by the Congressional Research Service, a public domain source. The Social Media Working Group Act of 2014 would amend the Homeland Security Act of 2002 to direct the United States Secretary of Homeland Security to establish within the United States Department of Homeland Security (DHS) a social media working group (the Group) to provide guidance and best practices to the emergency preparedness and response community on the use of social media technologies before, during, and after a terrorist attack. The bill would require the Group to submit an annual report that includes: (1) a review of current and emerging social media technologies being used to support preparedness and response activities related to terrorist attacks, of best practices and lessons learned on the use of social media during the response to terrorist attacks that occurred during the period covered by the report, and of available training for government officials on the use of social media in response to a terrorist attack; (2) recommendations to improve DHS's use of social media and to improve information sharing among DHS and its components and among state and local governments; and (3) a summary of coordination efforts with the private sector to discuss and resolve legal, operational, technical, privacy, and security concerns. == Congressional Budget Office report == This summary is based largely on the summary provided by the Congressional Budget Office, as ordered reported by the House Committee on Homeland Security on June 11, 2014. This is a public domain source. H.R. 4263 would direct the Department of Homeland Security (DHS) to establish a working group to provide guidance and best practices on the use of social media technologies, specifically during a terrorist attack or other emergency. The group would prepare guidance for the emergency preparedness and response community. The bill would define the membership of the working group, which would include more than 20 experts from federal, state, local, and tribal governments along with nongovernmental organizations. The working group would be exempt from the Federal Advisory Committee Act and would be authorized to hold virtual meetings to fulfill the requirement to meet twice a year. The working group would be required to submit an annual report on emerging trends and best practices for emergency response through social media. Based on the cost of similar activities carried out under the DHS Acquisition and Accountability Efficiency Act and the Critical Infrastructure Research and Development Advancement Act of 2013, the Congressional Budget Office (CBO) estimates that the new DHS responsibilities and the annual report required by H.R. 4263 would cost a total of less than $500,000 annually, assuming the availability of appropriated funds. Enacting the legislation would not affect direct spending or revenues; therefore, pay-as-you-go procedures do not apply. H.R. 4263 contains no intergovernmental or private-sector mandates as defined in the Unfunded Mandates Reform Act and would impose no costs on state, local, or tribal governments. == Procedural history == The Social Media Working Group Act of 2014 was introduced into the United States House of Representatives on March 14, 2014, by Rep. Susan W. Brooks (R, IN-5). It was referred to the United States House Committee on Homeland Security and the United States House Homeland Security Subcommittee on Emergency Preparedness, Response, and Communications. On June 19, 2014, it was reported (amended) alongside House Report 113-480. On July 8, 2014, the House voted in Roll Call Vote 369 to pass the bill 375–19. == Debate and discussion == Nate Elliott, a social media expert at Forrester Research, explains that "the hope is when government or another authority tweets something, people will share it for them," but that this often doesn't happen. This problem, that "messages wash away very quickly," is the reason that the federal government is trying to formulate a better social media strategy. Rep. Steven Palazzo (R-MS), who co-sponsored the bill, stated that "social media has played a crucial role in emergency preparedness and response in Mississippi, including during disasters like Hurricane Isaac and the tornadoes that hit the Hattiesburg area a little over a year ago." He said that their goal with the bill was to "build upon existing public-private partnerships and use social media in a more strategic way in order to help save lives and property."

  Read more →
• 

  MoFA Mitra

  MoFA Mitra is a mobile application launched by the Ministry of Foreign Affairs of Nepal to provide digital consular services, emergency support, rescue coordination, and complaint registration facilities for Nepali citizens living and working abroad. The application allows Nepali migrant workers, students, tourists, and Non-Resident Nepalis (NRNs) to access embassy services, emergency help, and official information directly from their smartphones. == Background == The need for a centralized digital support platform for Nepalis abroad had been discussed for several years due to increasing complaints related to labor exploitation, rescue delays, documentation problems, and lack of communication with Nepali diplomatic missions. Media organizations and migrant rights advocates had continuously highlighted issues faced by Nepali workers abroad, including human trafficking, fraudulent recruitment, delayed repatriation, and difficulties in receiving emergency assistance. In response, the Ministry of Foreign Affairs developed the MoFA Mitra app to digitize complaint handling, improve communication between embassies and citizens, and make emergency response faster and more accessible. == Features == The app includes several services and features for Nepali citizens abroad, including complaint registration, rescue coordination, embassy communication, and digital consular support services. Features of the application include: Online complaint registration Emergency rescue request system Direct contact with Nepali embassies and consulates Digital consular information Passport and document-related assistance Labor and migration support information Emergency hotline access Real-time notifications and alerts Location-based embassy information Tracking and coordination support for stranded citizens According to reports, the application was designed to simplify access to diplomatic services and strengthen emergency response coordination for Nepalis abroad. == Launch == The application was officially launched by Nepal’s Ministry of Foreign Affairs in Kathmandu in May 2026. Government officials stated that the app would strengthen Nepal’s digital governance system and improve support mechanisms for Nepali citizens residing overseas. Officials said the platform would help improve communication between Nepali diplomatic missions and citizens during emergencies and rescue operations. == Reception == The launch of the app received positive coverage from Nepali and international media outlets. Commentators described the initiative as a significant step toward modernization of Nepal’s diplomatic and consular services and digital governance infrastructure. Some observers also emphasized the importance of effective implementation, rapid response mechanisms, and continuous monitoring to ensure practical benefits for migrant workers abroad.

  Read more →
• 

  Sumazi

  Sumazi is a social media and social intelligence platform for enterprises, brands, and celebrities. Its technology performs social data analysis across social networking services including Facebook, Twitter and LinkedIn, to identify key people in his/her network who are experts, influencers or are located in a specific area for marketing, advertising or sales campaigns. The technology company was founded in 2011 by former Sun Microsystems employee Sumaya Kazi. The company was headquartered in San Francisco, California. The company was out of business by 2017. == Reception == Sumazi was one of 25 startups selected out of more than 1,200 to compete at TechCrunch Disrupt Startup Battlefield, where it won the Omidyar Network award for the startup "Most Likely to Change the World." Sumazi, which was based out of San Francisco, California, had been profiled in The New York Times as well as USA Today, which commented the advantages of the startup's location in the Silicon Valley. American Express OPEN Forum also featured Sumazi as a "Startup of the Week". Sumazi has additionally been mentioned in articles by Mashable, The Wall Street Journal, Current Editorials, Harvard Business Review, Smashing Magazine, and TechCrunch.

  Read more →
• 

  Tumblr

  Tumblr ( TUM-blər) is a microblogging and social media platform founded by David Karp in 2007 and operated by American company Tumblr, Inc., a subsidiary of Automattic. The service allows users to post multimedia and other content to a short-form blog. It has attracted significant attention and controversy for hosting a wide range of progressive user-generated content. == History == === Beginnings (2006–2012) === Development of Tumblr began in 2006 during a two-week gap between contracts at David Karp's software consulting company, Davidville. Karp had been interested in tumblelogs (short-form blogs, hence the name Tumblr) for some time and was waiting for one of the established blogging platforms to introduce their own tumblelogging platform. As none had done so after a year of waiting, Karp and developer Marco Arment began working on their own platform. Tumblr was launched in February 2007, and within two weeks had gained 75,000 users. Arment left the company in September 2010 to work on Instapaper. In June 2012, Tumblr featured its first major brand advertising campaign in collaboration with Adidas, who launched an official soccer Tumblr blog and bought ad placements on the user dashboard. This launch came only two months after Tumblr announced it would be moving towards paid advertising on its site. === Ownership by Yahoo! (2013–2018) === On May 20, 2013, it was announced that Yahoo and Tumblr had reached an agreement for Yahoo! Inc. to acquire Tumblr for $1.1 billion in cash. Many of Tumblr's users were unhappy with the news, causing some to start a petition, achieving nearly 170,000 signatures. David Karp remained CEO and the deal was finalized on June 20, 2013. Advertising sales goals were not met and in 2016 Yahoo wrote down $712 million of Tumblr's value. Verizon Communications acquired Yahoo in June 2017, and placed Yahoo and Tumblr under its Oath subsidiary. Karp announced in November 2017 that he would be leaving Tumblr by the end of the year. Jeff D'Onofrio, Tumblr's president and COO, took over leading the company. The site, along with the rest of the Oath division (renamed Verizon Media Group in 2019), continued to struggle under Verizon. In March 2019, Similarweb estimated Tumblr had lost 30% of its user traffic since December 2018, when the site had introduced a stricter content policy with heavier restrictions on adult content (which had been a notable draw to the service). In May 2019, it was reported that Verizon was considering selling the site due to its continued struggles since the purchase (as it had done with another Yahoo property, Flickr, via its sale to SmugMug). Following this news, Pornhub's vice president publicly expressed interest in purchasing Tumblr, with a promise to reinstate the previous adult content policies. === Automattic (2019–present) === On August 12, 2019, Verizon Media announced that it would sell Tumblr to Automattic, the operator of blog service WordPress.com and corporate backer of the open source blog software of the same name. The sale was for an undisclosed amount, but Axios reported that the sale price was less than $3 million, less than 0.3% of Yahoo's original purchase price. Automattic CEO Matt Mullenweg stated that the site will operate as a complementary service to WordPress.com, and that there were no plans to reverse the content policy decisions made during Verizon ownership. In November 2022, Mullenweg stated that Tumblr will add support for the decentralized social networking protocol ActivityPub. In November 2023, most of Tumblr's product development and marketing teams were transferred to other groups within Automattic. Mullenweg stated that focus would shift to core functionality and streamlining existing features. In February 2024, Automattic announced that it would begin selling user data from Tumblr and WordPress.com to Midjourney and OpenAI. Tumblr users are opted-in by default, with an option to opt out. In August 2024, Automattic announced that it would migrate Tumblr's backend to an architecture derived from WordPress, in order to ease development and code sharing between the platforms. The company stated that this migration would not impact the service's user experience and content, and that users "won't even notice a difference from the outside". In January 2025, Mullenweg stated that the migration, once completed, would also "unlock" ActivityPub access for Tumblr, including native support for the company's official ActivityPub plugin for WordPress. In April 2025, Automattic announced layoffs for 16% of its workforce, reducing a large portion of Tumblr staff. On March 16, 2026, Tumblr implemented a change to how notes were assigned to reblogs, making it more similar to sites like Twitter and Bluesky. The change was rolled back the next day after heavy user backlash. == Features == === Blog management === Dashboard: The dashboard is the primary tool for the typical Tumblr user. It is a live feed of recent posts from blogs that they follow. Through the dashboard, users are able to comment, reblog, and like posts from other blogs that appear on their dashboard. The dashboard allows the user to upload text posts, images, videos, quotes, or links to their blog with a click of a button displayed at the top of the dashboard. Users are also able to connect their blogs to their Twitter and Facebook accounts, so that whenever they make a post, it will also be sent as a tweet and a status update. As of June 2022, users can also turn off reblogs on specific posts through the dashboard. Queue: Users are able to set up a schedule to delay posts that they make. They can spread their posts over several hours or even days. Tags: Users can help their audience find posts about certain topics by adding tags. If someone were to upload a picture to their blog and wanted their viewers to find pictures, they would add the tag #picture, and their viewers could use that word to search for posts with the tag #picture. HTML editing: Tumblr allows users to edit their blog's theme using HTML to control the appearance of their blog. Custom themes are able to be shared and used by other users, or sold. Custom domains: Tumblr allows users to use custom domains for their blogs. Users must purchase a domain from Tumblr Domains, an in-house registrar that provides domains that can only be used with Tumblr unless removed from the user's blog and transferred to another registrar. Blogs previously were able to be linked with any domain/subdomain from any registrar, however following the introduction of the Tumblr Domains service, now requires you to purchase a domain directly from Tumblr to be used with a blog. Users who kept their blogs connected to a domain after the introduction got to keep their custom domain, as long as they do not disconnect it from Tumblr or let the domain expire. === Tags === The tagging system on the website operates on a hybrid tagging system, involving both self-tagging (user write their own tags on their posts) and an auto-manual function (the website will recommend popular tags and ones that the user has used before.) Only the first 20 tags added to any post will be indexed by the site. The tags are prefaced by a hashtag and separated by commas, and spaces and special characters are allowed, but only up to 140 characters total per tag. There are two main types used by Tumblr users: descriptive tagging, and opinion or commentary tagging. Descriptive tags are usually introduced by the original poster, and describe what is in the post (e.g. #art, #sky). These are important for the original poster to use, so their post will be indexed and searchable by others wishing to view that subject of content. Tags used as a form of communication are unique to Tumblr, and are typically more personal, expressing opinions, reactions, meta-commentary, background information, and more. Instead of adding onto the reblogged post (with their comments becoming an addition to each subsequent reblog from them) a user may add their comments in the tags, not changing the content or appearance of the original post in any way. Not all users choose to use tags this way, but those who do use tags for commentary may prefer it over adding a comment on the actual post. === Mobile === With Tumblr's 2009 acquisition of Tumblerette, an iOS application created by Jeff Rock and Garrett Ross, the service launched its official iPhone app. The site became available to BlackBerry smartphones on April 17, 2010, via a Mobelux application in BlackBerry World. In June 2012, Tumblr released a new version of its iOS app, Tumblr 3.0, allowing support for Spotify integration, hi-res images and offline access. An app for Android is also available. A Windows Phone app was released on April 23, 2013. An app for Google Glass was released on May 16, 2013. === Inbox and messaging === Tumblr blogs have the option to allow users to submit questions, either as themselves or anonymously, to the blog for a response. Tumblr

  Read more →
• 

  Data lineage

  Data lineage refers to the process of tracking how data is generated, transformed, transmitted and used across systems over time. It documents data's origins, transformations and movements, providing detailed visibility into its life cycle. This process simplifies the identification of errors in data analytics workflows, by enabling users to trace issues back to their root causes. Data lineage facilitates the ability to replay specific segments or inputs of the dataflow. This can be used in debugging or regenerating lost outputs. In database systems, this concept is closely related to data provenance, which involves maintaining records of inputs, entities, systems and processes that influence data. Data provenance provides a historical record of data origins and transformations. It supports forensic activities such as data-dependency analysis, error/compromise detection, recovery, auditing and compliance analysis: "Lineage is a simple type of why provenance." Data governance plays a critical role in managing metadata by establishing guidelines, strategies and policies. Enhancing data lineage with data quality measures and master data management adds business value. Although data lineage is typically represented through a graphical user interface (GUI), the methods for gathering and exposing metadata to this interface can vary. Based on the metadata collection approach, data lineage can be categorized into three types: Those involving software packages for structured data, programming languages and Big data systems. Data lineage information includes technical metadata about data transformations. Enriched data lineage may include additional elements such as data quality test results, reference data, data models, business terminology, data stewardship information, program management details and enterprise systems associated with data points and transformations. Data lineage visualization tools often include masking features that allow users to focus on information relevant to specific use cases. To unify representations across disparate systems, metadata normalization or standardization may be required. == Representation of data lineage == Representation broadly depends on the scope of the metadata management and reference point of interest. Data lineage provides sources of the data and intermediate data flow hops from the reference point with backward data lineage, leading to the final destination's data points and its intermediate data flows with forward data lineage. These views can be combined with end-to-end lineage for a reference point that provides a complete audit trail of that data point of interest from sources to their final destinations. As the data points or hops increase, the complexity of such representation becomes incomprehensible. Thus, the best feature of the data lineage view is the ability to simplify the view by temporarily masking unwanted peripheral data points. Tools with the masking feature enable scalability of the view and enhance analysis with the best user experience for both technical and business users. Data lineage also enables companies to trace sources of specific business data to track errors, implement changes in processes and implementing system migrations to save significant amounts of time and resources. Data lineage can improve efficiency in business intelligence BI processes. Data lineage can be represented visually to discover the data flow and movement from its source to destination via various changes and hops on its way in the enterprise environment. This includes how the data is transformed along the way, how the representation and parameters change and how the data splits or converges after each hop. A simple representation of the Data Lineage can be shown with dots and lines, where dots represent data containers for data points, and lines connecting them represent transformations the data undergoes between the data containers. Data lineage can be visualized at various levels based on the granularity of the view. At a very high-level, data lineage is visualized as systems that the data interacts with before it reaches its destination. At its most granular, visualizations at the data point level can provide the details of the data point and its historical behavior, attribute properties and trends and data quality of the data passed through that specific data point in the data lineage. The scope of the data lineage determines the volume of metadata required to represent its data lineage. Usually, data governance and data management of an organization determine the scope of the data lineage based on their regulations, enterprise data management strategy, data impact, reporting attributes and critical data elements of the organization. == Rationale == Distributed systems like Google Map Reduce, Microsoft Dryad, Apache Hadoop (an open-source project) and Google Pregel provide such platforms for businesses and users. However, even with these systems, Big Data analytics can take several hours, days or weeks to run, simply due to the data volumes involved. For example, a ratings prediction algorithm for the Netflix Prize challenge took nearly 20 hours to execute on 50 cores, and a large-scale image processing task to estimate geographic information took 3 days to complete using 400 cores. "The Large Synoptic Survey Telescope is expected to generate terabytes of data every night and eventually store more than 50 petabytes, while in the bioinformatics sector, the 12 largest genome sequencing houses in the world now store petabytes of data apiece. It is very difficult for a data scientist to trace an unknown or an unanticipated result. === Big data debugging === Big data analytics is the process of examining large data sets to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information. Machine learning, among other algorithms, is used to transform and analyze the data. Due to the large size of the data, there could be unknown features in the data. The massive scale and unstructured nature of data, the complexity of these analytics pipelines, and long runtimes pose significant manageability and debugging challenges. Even a single error in these analytics can be extremely difficult to identify and remove. While one may debug them by re-running the entire analytics through a debugger for stepwise debugging, this can be expensive due to the amount of time and resources needed. Auditing and data validation are other major problems due to the growing ease of access to relevant data sources for use in experiments, the sharing of data between scientific communities and use of third-party data in business enterprises. As such, more cost-efficient ways of analyzing data intensive scale-able computing (DISC) are crucial to their continued effective use. === Challenges in Big Data debugging === ==== Massive scale ==== According to an EMC/IDC study, 2.8 ZB of data were created and replicated in 2012. Furthermore, the same study states that the digital universe will double every two years between now and 2020, and that there will be approximately 5.2 TB of data for every person in 2020. Based on current technology, the storage of this much data will mean greater energy usage by data centers. ==== Unstructured data ==== Unstructured data usually refers to information that doesn't reside in a traditional row-column database. Unstructured data files often include text and multimedia content, such as e-mail messages, word processing documents, videos, photos, audio files, presentations, web pages and many other kinds of business documents. While these types of files may have an internal structure, they are still considered "unstructured" because the data they contain doesn't fit neatly into a database. The amount of unstructured data in enterprises is growing many times faster than structured databases are growing. Big data can include both structured and unstructured data, but IDC estimates that 90 percent of Big Data is unstructured data. The fundamental challenge of unstructured data sources is that they are difficult for non-technical business users and data analysts alike to unbox, understand and prepare for analytic use. Beyond issues of structure, the sheer volume of this type of data contributes to such difficulty. Because of this, current data mining techniques often leave out valuable information and make analyzing unstructured data laborious and expensive. In today's competitive business environment, companies have to find and analyze the relevant data they need quickly. The challenge is going through the volumes of data and accessing the level of detail needed, all at a high speed. The challenge only grows as the degree of granularity increases. One possible solution is hardware. Some vendors are using increased memory and parallel processing to crunch large volumes of data quickly. Another method is putting data in-memory but using a grid

  Read more →
• 

  Neural network Gaussian process

  A Neural Network Gaussian Process (NNGP) is a Gaussian process (GP) obtained as the limit of a certain type of sequence of neural networks. Specifically, a wide variety of network architectures converges to a GP in the infinitely wide limit, in the sense of distribution. The concept constitutes an intensional definition, i.e., a NNGP is just a GP, but distinguished by how it is obtained. == Motivation == Bayesian networks are a modeling tool for assigning probabilities to events, and thereby characterizing the uncertainty in a model's predictions. Deep learning and artificial neural networks are approaches used in machine learning to build computational models which learn from training examples. Bayesian neural networks merge these fields. They are a type of neural network whose parameters and predictions are both probabilistic. While standard neural networks often assign high confidence even to incorrect predictions, Bayesian neural networks can more accurately evaluate how likely their predictions are to be correct. Computation in artificial neural networks is usually organized into sequential layers of artificial neurons. The number of neurons in a layer is called the layer width. When we consider a sequence of Bayesian neural networks with increasingly wide layers (see figure), they converge in distribution to a NNGP. This large width limit is of practical interest, since the networks often improve as layers get wider. And the process may give a closed form way to evaluate networks. NNGPs also appears in several other contexts: It describes the distribution over predictions made by wide non-Bayesian artificial neural networks after random initialization of their parameters, but before training; it appears as a term in neural tangent kernel prediction equations; it is used in deep information propagation to characterize whether hyperparameters and architectures will be trainable. It is related to other large width limits of neural networks. === Scope === The first correspondence result had been established in the 1995 PhD thesis of Radford M. Neal, then supervised by Geoffrey Hinton at University of Toronto. Neal cites David J. C. MacKay as inspiration, who worked in Bayesian learning. Today the correspondence is proven for: Single hidden layer Bayesian neural networks; deep fully connected networks as the number of units per layer is taken to infinity; convolutional neural networks as the number of channels is taken to infinity; transformer networks as the number of attention heads is taken to infinity; recurrent networks as the number of units is taken to infinity. In fact, this NNGP correspondence holds for almost any architecture: Generally, if an architecture can be expressed solely via matrix multiplication and coordinatewise nonlinearities (i.e., a tensor program), then it has an infinite-width GP. This in particular includes all feedforward or recurrent neural networks composed of multilayer perceptron, recurrent neural networks (e.g., LSTMs, GRUs), (nD or graph) convolution, pooling, skip connection, attention, batch normalization, and/or layer normalization. === Illustration === Every setting of a neural network's parameters θ {\displaystyle \theta } corresponds to a specific function computed by the neural network. A prior distribution p ( θ ) {\displaystyle p(\theta )} over neural network parameters therefore corresponds to a prior distribution over functions computed by the network. As neural networks are made infinitely wide, this distribution over functions converges to a Gaussian process for many architectures. The notation used in this section is the same as the notation used below to derive the correspondence between NNGPs and fully connected networks, and more details can be found there. The figure to the right plots the one-dimensional outputs z L ( ⋅ ; θ ) {\displaystyle z^{L}(\cdot ;\theta )} of a neural network for two inputs x {\displaystyle x} and x ∗ {\displaystyle x^{}} against each other. The black dots show the function computed by the neural network on these inputs for random draws of the parameters from p ( θ ) {\displaystyle p(\theta )} . The red lines are iso-probability contours for the joint distribution over network outputs z L ( x ; θ ) {\displaystyle z^{L}(x;\theta )} and z L ( x ∗ ; θ ) {\displaystyle z^{L}(x^{};\theta )} induced by p ( θ ) {\displaystyle p(\theta )} . This is the distribution in function space corresponding to the distribution p ( θ ) {\displaystyle p(\theta )} in parameter space, and the black dots are samples from this distribution. For infinitely wide neural networks, since the distribution over functions computed by the neural network is a Gaussian process, the joint distribution over network outputs is a multivariate Gaussian for any finite set of network inputs. == Discussion == === Infinitely wide fully connected network === This section expands on the correspondence between infinitely wide neural networks and Gaussian processes for the specific case of a fully connected architecture. It provides a proof sketch outlining why the correspondence holds, and introduces the specific functional form of the NNGP for fully connected networks. The proof sketch closely follows the approach by Novak and coauthors. ==== Network architecture specification ==== Consider a fully connected artificial neural network with inputs x {\displaystyle x} , parameters θ {\displaystyle \theta } consisting of weights W l {\displaystyle W^{l}} and biases b l {\displaystyle b^{l}} for each layer l {\displaystyle l} in the network, pre-activations (pre-nonlinearity) z l {\displaystyle z^{l}} , activations (post-nonlinearity) y l {\displaystyle y^{l}} , pointwise nonlinearity ϕ ( ⋅ ) {\displaystyle \phi (\cdot )} , and layer widths n l {\displaystyle n^{l}} . For simplicity, the width n L + 1 {\displaystyle n^{L+1}} of the readout vector z L {\displaystyle z^{L}} is taken to be 1. The parameters of this network have a prior distribution p ( θ ) {\displaystyle p(\theta )} , which consists of an isotropic Gaussian for each weight and bias, with the variance of the weights scaled inversely with layer width. This network is illustrated in the figure to the right, and described by the following set of equations: x ≡ input y l ( x ) = { x l = 0 ϕ ( z l − 1 ( x ) ) l > 0 z i l ( x ) = ∑ j W i j l y j l ( x ) + b i l W i j l ∼ N ( 0 , σ w 2 n l ) b i l ∼ N ( 0 , σ b 2 ) ϕ ( ⋅ ) ≡ nonlinearity y l ( x ) , z l − 1 ( x ) ∈ R n l × 1 n L + 1 = 1 θ = { W 0 , b 0 , … , W L , b L } {\displaystyle {\begin{aligned}x&\equiv {\text{input}}\\y^{l}(x)&=\left\{{\begin{array}{lcl}x&&l=0\\\phi \left(z^{l-1}(x)\right)&&l>0\end{array}}\right.\\z_{i}^{l}(x)&=\sum _{j}W_{ij}^{l}y_{j}^{l}(x)+b_{i}^{l}\\W_{ij}^{l}&\sim {\mathcal {N}}\left(0,{\frac {\sigma _{w}^{2}}{n^{l}}}\right)\\b_{i}^{l}&\sim {\mathcal {N}}\left(0,\sigma _{b}^{2}\right)\\\phi (\cdot )&\equiv {\text{nonlinearity}}\\y^{l}(x),z^{l-1}(x)&\in \mathbb {R} ^{n^{l}\times 1}\\n^{L+1}&=1\\\theta &=\left\{W^{0},b^{0},\dots ,W^{L},b^{L}\right\}\end{aligned}}} ==== ==== z l | y l {\displaystyle z^{l}|y^{l}} is a Gaussian process We first observe that the pre-activations z l {\displaystyle z^{l}} are described by a Gaussian process conditioned on the preceding activations y l {\displaystyle y^{l}} . This result holds even at finite width. Each pre-activation z i l {\displaystyle z_{i}^{l}} is a weighted sum of Gaussian random variables, corresponding to the weights W i j l {\displaystyle W_{ij}^{l}} and biases b i l {\displaystyle b_{i}^{l}} , where the coefficients for each of those Gaussian variables are the preceding activations y j l {\displaystyle y_{j}^{l}} . Because they are a weighted sum of zero-mean Gaussians, the z i l {\displaystyle z_{i}^{l}} are themselves zero-mean Gaussians (conditioned on the coefficients y j l {\displaystyle y_{j}^{l}} ). Since the z l {\displaystyle z^{l}} are jointly Gaussian for any set of y l {\displaystyle y^{l}} , they are described by a Gaussian process conditioned on the preceding activations y l {\displaystyle y^{l}} . The covariance or kernel of this Gaussian process depends on the weight and bias variances σ w 2 {\displaystyle \sigma _{w}^{2}} and σ b 2 {\displaystyle \sigma _{b}^{2}} , as well as the second moment matrix K l {\displaystyle K^{l}} of the preceding activations y l {\displaystyle y^{l}} , z i l ∣ y l ∼ G P ( 0 , σ w 2 K l + σ b 2 ) K l ( x , x ′ ) = 1 n l ∑ i y i l ( x ) y i l ( x ′ ) {\displaystyle {\begin{aligned}z_{i}^{l}\mid y^{l}&\sim {\mathcal {GP}}\left(0,\sigma _{w}^{2}K^{l}+\sigma _{b}^{2}\right)\\K^{l}(x,x')&={\frac {1}{n^{l}}}\sum _{i}y_{i}^{l}(x)y_{i}^{l}(x')\end{aligned}}} The effect of the weight scale σ w 2 {\displaystyle \sigma _{w}^{2}} is to rescale the contribution to the covariance matrix from K l {\displaystyle K^{l}} , while the bias is shared for all inputs, and so σ b 2 {\displaystyle \sigma _{b}^{2}} makes the z i l {\displaystyle z_{i}^{l}} for different datapoints more similar and

  Read more →
• 

  Hybrid cryptosystem

  In cryptography, a hybrid cryptosystem is one which combines the convenience of a public-key cryptosystem with the efficiency of a symmetric-key cryptosystem. Public-key cryptosystems are convenient in that they do not require the sender and receiver to share a common secret in order to communicate securely. However, they often rely on complicated mathematical computations and are thus generally much more inefficient than comparable symmetric-key cryptosystems. In many applications, the high cost of encrypting long messages in a public-key cryptosystem can be prohibitive. This is addressed by hybrid systems by using a combination of both. A hybrid cryptosystem can be constructed using any two separate cryptosystems: a key encapsulation mechanism, which is a public-key cryptosystem a data encapsulation scheme, which is a symmetric-key cryptosystem The hybrid cryptosystem is itself a public-key system, whose public and private keys are the same as in the key encapsulation scheme. Note that for very long messages the bulk of the work in encryption/decryption is done by the more efficient symmetric-key scheme, while the inefficient public-key scheme is used only to encrypt/decrypt a short key value. == Implementations and standards == All practical implementations of public key cryptography today employ a hybrid system. Examples include the TLS protocol and the SSH protocol, that use a public-key mechanism for key exchange (such as Diffie-Hellman) and a symmetric-key mechanism for data encapsulation (such as AES). The OpenPGP file format and the PKCS#7 file format are other examples. Hybrid Public Key Encryption (HPKE, published as RFC 9180) is a modern standard for generic hybrid encryption. HPKE is used within multiple IETF protocols, including Messaging Layer Security (MLS), Oblivious DNS over HTTPS, Oblivious HTTP, Privacy Preserving Measurement, and TLS Encrypted Client Hello. Envelope encryption is an example of a usage of hybrid cryptosystems in cloud computing. In a cloud context, hybrid cryptosystems also enable centralized key management. == Example == To encrypt a message addressed to Alice in a hybrid cryptosystem, Bob does the following: Obtains Alice's public key. Generates a fresh symmetric key for the data encapsulation scheme. Encrypts the message under the data encapsulation scheme, using the symmetric key just generated. Encrypts the symmetric key under the key encapsulation scheme, using Alice's public key. Sends both of these ciphertexts to Alice. To decrypt this hybrid ciphertext, Alice does the following: Uses her private key to decrypt the symmetric key contained in the key encapsulation segment. Uses this symmetric key to decrypt the message contained in the data encapsulation segment. == Security == If both the key encapsulation and data encapsulation schemes in a hybrid cryptosystem are secure against adaptive chosen ciphertext attacks, then the hybrid scheme inherits that property as well. However, it is possible to construct a hybrid scheme secure against adaptive chosen ciphertext attacks even if the key encapsulation has a slightly weakened security definition (though the security of the data encapsulation must be slightly stronger). == Envelope encryption == Envelope encryption is term used for encrypting with a hybrid cryptosystem used by all major cloud service providers, often as part of a centralized key management system in cloud computing. Envelope encryption gives names to the keys used in hybrid encryption: Data Encryption Keys (abbreviated DEK, and used to encrypt data) and Key Encryption Keys (abbreviated KEK, and used to encrypt the DEKs). In a cloud environment, encryption with envelope encryption involves generating a DEK locally, encrypting one's data using the DEK, and then issuing a request to wrap (encrypt) the DEK with a KEK stored in a potentially more secure service. Then, this wrapped DEK and encrypted message constitute a ciphertext for the scheme. To decrypt a ciphertext, the wrapped DEK is unwrapped (decrypted) via a call to a service, and then the unwrapped DEK is used to decrypt the encrypted message. In addition to the normal advantages of a hybrid cryptosystem, using asymmetric encryption for the KEK in a cloud context provides easier key management and separation of roles, but can be slower. In cloud systems, such as Google Cloud Platform and Amazon Web Services, a key management system (KMS) can be available as a service. In some cases, the key management system will store keys in hardware security modules, which are hardware systems that protect keys with hardware features like intrusion resistance. This means that KEKs can also be more secure because they are stored on secure specialized hardware. Envelope encryption makes centralized key management easier because a centralized key management system only needs to store KEKs, which occupy less space, and requests to the KMS only involve sending wrapped and unwrapped DEKs, which use less bandwidth than transmitting entire messages. Since one KEK can be used to encrypt many DEKs, this also allows for less storage space to be used in the KMS. This also allows for centralized auditing and access control at one point of access.

  Read more →
• 

  Content inventory

  A content inventory is the process and the result of cataloging the entire contents of a website. An allied practice—a content audit—is the process of evaluating that content. A content inventory and a content audit are closely related concepts, and they are often conducted in tandem. == Description == A content inventory typically includes all information assets on a website, such as web pages (HTML), meta elements (e.g., keywords, description, page title), images, audio and video files, and document files (e.g., .pdf, .doc, .ppt). A content inventory is a quantitative analysis of a website. It simply logs what is on a website. The content inventory will answer the question: “What is there?” and can be the start of a website review. A related (and sometimes confused term) is a content audit, a qualitative analysis of information assets on a website. It is the assessment of that content and its place in relationship to surrounding Web pages and information assets. The content audit will answer the question: “Is it any good?” Over the years, techniques for creating and managing a content inventory have been developed and refined in the field of website content management. A spreadsheet application (e.g., Microsoft Excel or LibreOffice Calc) is the preferred tool for keeping a content inventory; the data can be easily configured and manipulated. Typical categories in a content inventory include the following: Link — The URL for the page Format — For example, .HTML, .pdf, .doc, .ppt Meta page title — Page title as it appears in the meta