Bidirectional encoder representations from transformers (BERT) is a language model introduced in October 2018 by researchers at Google. It learns to represent text as a sequence of vectors using self-supervised learning. It uses the encoder-only transformer architecture. BERT dramatically improved the state of the art for large language models. As of 2020, BERT is a ubiquitous baseline in natural language processing (NLP) experiments. BERT is trained by masked token prediction and next sentence prediction. With this training, BERT learns contextual, latent representations of tokens in their context, similar to ELMo and GPT-2. It found applications for many natural language processing tasks, such as coreference resolution and polysemy resolution. It improved on ELMo and spawned the study of "BERTology", which attempts to interpret what is learned by BERT. BERT was originally implemented in the English language at two model sizes, BERTBASE (110 million parameters) and BERTLARGE (340 million parameters). Both were trained on the Toronto BookCorpus (800M words) and English Wikipedia (2,500M words). The weights were released on GitHub. On March 11, 2020, 24 smaller models were released, the smallest being BERTTINY with just 4 million parameters. == Architecture == BERT is an "encoder-only" transformer architecture. At a high level, BERT consists of 4 modules: Tokenizer: This module converts a piece of English text into a sequence of integers ("tokens"). Embedding: This module converts the sequence of tokens into an array of real-valued vectors representing the tokens. It represents the conversion of discrete token types into a lower-dimensional Euclidean space. Encoder: a stack of Transformer blocks with self-attention, but without causal masking. Task head: This module converts the final representation vectors into one-shot encoded tokens again by producing a predicted probability distribution over the token types. It can be viewed as a simple decoder, decoding the latent representation into token types, or as an "un-embedding layer". The task head is necessary for pre-training, but it is often unnecessary for so-called "downstream tasks," such as question answering or sentiment classification. Instead, one removes the task head and replaces it with a newly initialized module suited for the task, and finetune the new module. The latent vector representation of the model is directly fed into this new module, allowing for sample-efficient transfer learning. === Embedding === This section describes the embedding used by BERTBASE. The other one, BERTLARGE, is similar, just larger. The tokenizer of BERT is WordPiece, which is a sub-word strategy like byte-pair encoding. Its vocabulary size is 30,000, and any token not appearing in its vocabulary is replaced by [UNK] ("unknown"). The first layer is the embedding layer, which contains three components: token type embeddings, position embeddings, and segment type embeddings. Token type: The token type is a standard embedding layer, translating a one-hot vector into a dense vector based on its token type. Position: The position embeddings are based on a token's position in the sequence. BERT uses absolute position embeddings, where each position in a sequence is mapped to a real-valued vector. Each dimension of the vector consists of a sinusoidal function that takes the position in the sequence as input. Segment type: Using a vocabulary of just 0 or 1, this embedding layer produces a dense vector based on whether the token belongs to the first or second text segment in that input. In other words, type-1 tokens are all tokens that appear after the [SEP] special token. All prior tokens are type-0. The three embedding vectors are added together representing the initial token representation as a function of these three pieces of information. After embedding, the vector representation is normalized using a LayerNorm operation, outputting a 768-dimensional vector for each input token. After this, the representation vectors are passed forward through 12 Transformer encoder blocks, and are decoded back to 30,000-dimensional vocabulary space using a basic affine transformation layer. === Architectural family === The encoder stack of BERT has 2 free parameters: L {\displaystyle L} , the number of layers, and H {\displaystyle H} , the hidden size. There are always H / 64 {\displaystyle H/64} self-attention heads, and the feed-forward/filter size is always 4 H {\displaystyle 4H} . By varying these two numbers, one obtains an entire family of BERT models. For BERT: the feed-forward size and filter size are synonymous. Both of them denote the number of dimensions in the middle layer of the feed-forward network. the hidden size and embedding size are synonymous. Both of them denote the number of real numbers used to represent a token. The notation for encoder stack is written as L/H. For example, BERTBASE is written as 12L/768H, BERTLARGE as 24L/1024H, and BERTTINY as 2L/128H. == Training == === Pre-training === BERT was pre-trained simultaneously on two tasks: Masked language modeling (MLM): In this task, BERT ingests a sequence of words, where one word may be randomly changed ("masked"), and BERT tries to predict the original words that had been changed. For example, in the sentence "The cat sat on the [MASK]," BERT would need to predict "mat." This helps BERT learn bidirectional context, meaning it understands the relationships between words not just from left to right or right to left but from both directions at the same time. Next sentence prediction (NSP): In this task, BERT is trained to predict whether one sentence logically follows another. For example, given two sentences, "The cat sat on the mat" and "It was a sunny day", BERT has to decide if the second sentence is a valid continuation of the first one. This helps BERT understand relationships between sentences, which is important for tasks like question answering or document classification. ==== Masked language modeling ==== In masked language modeling, 15% of tokens would be randomly selected for masked-prediction task, and the training objective was to predict the masked token given its context. In more detail, the selected token is: replaced with a [MASK] token with probability 80%, replaced with a random word token with probability 10%, not replaced with probability 10%. The reason not all selected tokens are masked is to avoid the dataset shift problem. The dataset shift problem arises when the distribution of inputs seen during training differs significantly from the distribution encountered during inference. A trained BERT model might be applied to word representation (like Word2Vec), where it would be run over sentences not containing any [MASK] tokens. It is later found that more diverse training objectives are generally better. As an illustrative example, consider the sentence "my dog is cute". It would first be divided into tokens like "my1 dog2 is3 cute4". Then a random token in the sentence would be picked. Let it be the 4th one "cute4". Next, there would be three possibilities: with probability 80%, the chosen token is masked, resulting in "my1 dog2 is3 [MASK]4"; with probability 10%, the chosen token is replaced by a uniformly sampled random token, such as "happy", resulting in "my1 dog2 is3 happy4"; with probability 10%, nothing is done, resulting in "my1 dog2 is3 cute4". After processing the input text, the model's 4th output vector is passed to its decoder layer, which outputs a probability distribution over its 30,000-dimensional vocabulary space. ==== Next sentence prediction ==== Given two sentences, the model predicts if they appear sequentially in the training corpus, outputting either [IsNext] or [NotNext]. During training, the algorithm sometimes samples two sentences from a single continuous span in the training corpus, while at other times, it samples two sentences from two discontinuous spans. The first sentence starts with a special token, [CLS] (for "classify"). The two sentences are separated by another special token, [SEP] (for "separate"). After processing the two sentences, the final vector for the [CLS] token is passed to a linear layer for binary classification into [IsNext] and [NotNext]. For example: Given "[CLS] my dog is cute [SEP] he likes playing [SEP]", the model should predict [IsNext]. Given "[CLS] my dog is cute [SEP] how do magnets work [SEP]", the model should predict [NotNext]. === Fine-tuning === BERT is meant as a general pretrained model for various applications in natural language processing. That is, after pre-training, BERT can be fine-tuned with fewer resources on smaller datasets to optimize its performance on specific tasks such as natural language inference and text classification, and sequence-to-sequence-based language generation tasks such as question answering and conversational response generation. The original BERT paper published results demonstrating that a small amount of fine
CloudPassage
CloudPassage is a company that provides an automation platform, delivered via software as a service, that improves security for private, public, and hybrid cloud computing environments. CloudPassage is headquartered in San Francisco. == History == CloudPassage was founded by Carson Sweet, Talli Somekh, and Vitaliy Geraymovych in 2010. The company used cloud computing and big data analytics to implement security monitoring and control in a platform called Halo. CloudPassage spent a year in stealth developing the Halo technology, coming out of stealth mode to a closed beta in January 2011. In June 2012, the company launched the commercial product that included configuration security monitoring, network microsegmentation, and two-factor authentication for privileged access management. By 2013, CloudPassage expanded Halo to support large enterprises with advanced security and compliance requirements with a product called Halo Enterprise. The first round of venture funding for the company raised $6.5 million. In April 2012, CloudPassage raised $14 million. The financing round was led by Tenaya Capital. In February 2014, CloudPassage announced that it had raised $25.5 million in funding led by Shasta Ventures. In total, the company has invested over $30 million in its technology and raised approximately $88 million in capital. == Product == The CloudPassage platform provides cloud workload security and compliance for systems hosted in public or private cloud infrastructure environments, including hybrid cloud and multi-cloud workload hosting models. The flagship product the company offers is called Halo. Halo secures virtual servers in public, private, and hybrid cloud infrastructures and provides file integrity monitoring (FIM) while also administering firewall automation, vulnerability monitoring, network access control, security event alerting, and assessment. The Halo platform also provides security applications such as privileged access management, software vulnerability scanning, multifactor authentication, and log-based IDS. In December 2013, CloudPassage set up six servers with Microsoft Windows and Linux operating systems and combinations of popular programs and invited hackers to attempt to hack into the servers. The top prize was $5,000 and the winning hacker was a novice that completed the task in four hours. CloudPassage programmed the servers to use basic default security settings to show how vulnerable cloud computing programs can be to security threats. == Awards and recognition == In May 2011, Gigaom named CloudPassage in its list of the Top 50 Cloud Innovators. That same month, eWeek recognized CloudPassage as one of 16 Hot Startup Companies Flying Under the Radar. SC Magazine named CloudPassage an Industry Innovator in the Virtualization and Cloud Security category in 2012. Also in 2012, The Wall Street Journal named CloudPassage a runner-up in the Information Security category of its Technology Innovation Awards. The CloudPassage large-scale security program, Halo, won Best Security Solution in 2014 at the SIIA Codie awards.
Computer network engineering
Computer network engineering is a technology discipline within engineering that deals with the design, implementation, and management of computer networks. These systems contain both physical components, such as routers, switches, cables, and some logical elements, such as protocols and network services. Computer network engineers attempt to ensure that the data is transmitted efficiently, securely, and reliably over both local area networks (LANs) and wide area networks (WANs), as well as across the Internet. Computer networks often play a large role in modern industries ranging from telecommunications to cloud computing, enabling processes such as email and file sharing, as well as complex real-time services like video conferencing and online gaming. == Background == The evolution of network engineering is marked by significant milestones that have greatly impacted communication methods. These milestones particularly highlight the progress made in developing communication protocols that are vital to contemporary networking. This discipline originated in the 1960s with projects like ARPANET, which initiated important advancements in reliable data transmission. The advent of protocols such as TCP/IP revolutionized networking by enabling interoperability among various systems, which, in turn, fueled the rapid growth of the Internet. Key developments include the standardization of protocols and the shift towards increasingly complex layered architectures. These advancements have profoundly changed the way devices interact across global networks. == Network infrastructure design == The foundation of computer network engineering lies in the design of the network infrastructure. This involves planning both the physical layout of the network and its logical topology to ensure optimal data flow, reliability, and scalability. === Physical infrastructure === The physical infrastructure consists of the hardware used to transmit data, which is represented by the first layer of the OSI model. ==== Cabling ==== Copper cables such as ethernet over twisted pair are commonly used for short-distance connections, especially in local area networks (LANs), while fiber optic cables are favored for long-distance communication due to their high-speed transmission capabilities and lower susceptibility to interference. Fiber optics play a significant role in the backbone of large-scale networks, such as those used in data centers and internet service provider (ISP) infrastructures. ==== Wireless networks ==== In addition to wired connections, wireless networks have become a common component of physical infrastructure. These networks facilitate communication between devices without the need for physical cables, providing flexibility and mobility. Wireless technologies use a range of transmission methods, including radio frequency (RF) waves, infrared signals, and laser-based communication, allowing devices to connect to the network. Wi-Fi based on IEEE 802.11 standards is the most widely used wireless technology in local area networks and relies on RF waves to transmit data between devices and access points. Wireless networks operate across various frequency bands, including 2.4 GHz and 5 GHz, each offering unique ranges and data rates; the 2.4 GHz band provides broader coverage, while the 5 GHz band supports faster data rates with reduced interference, ideal for densely populated environments. Beyond Wi-Fi, other wireless transmission methods, such as infrared and laser-based communication, are used in specific contexts, like short-range, line-of-sight links or secure point-to-point communication. In mobile networks, cellular technologies like 3G, 4G, and 5G enable wide-area wireless connectivity. 3G introduced faster data rates for mobile browsing, while 4G significantly improved speed and capacity, supporting advanced applications like video streaming. The latest evolution, 5G, operates across a range of frequencies, including millimeter-wave bands, and provides high data rates, low latency, and support for more device connectivity, useful for applications like the Internet of Things (IoT) and autonomous systems. Together, these wireless technologies allow networks to meet a variety of connectivity needs across local and wide areas. ==== Network devices ==== Routers and switches help direct data traffic and assist in maintaining network security; network engineers configure these devices to optimize traffic flow and prevent network congestion. In wireless networks, wireless access points (WAP) allow devices to connect to the network. To expand coverage, multiple access points can be placed to create a wireless infrastructure. Beyond Wi-Fi, cellular network components like base stations and repeaters support connectivity in wide-area networks, while network controllers and firewalls manage traffic and enforce security policies. Together, these devices enable a secure, flexible, and scalable network architecture suitable for both local and wide-area coverage. === Logical topology === Beyond the physical infrastructure, a network must be organized logically, which defines how data is routed between devices. Various topologies, such as star, mesh, and hierarchical designs, are employed depending on the network’s requirements. In a star topology, for example, all devices are connected to a central hub that directs traffic. This configuration is relatively easy to manage and troubleshoot but can create a single point of failure. In contrast, a mesh topology, where each device is interconnected with several others, offers high redundancy and reliability but requires a more complex design and larger hardware investment. Large networks, especially those in enterprises, often employ a hierarchical model, dividing the network into core, distribution, and access layers to enhance scalability and performance. == Network protocols and communication standards == Communication protocols dictate how data in a network is transmitted, routed, and delivered. Depending on the goals of the specific network, protocols are selected to ensure that the network functions efficiently and securely. The Transmission Control Protocol/Internet Protocol (TCP/IP) suite is fundamental to modern computer networks, including the Internet. It defines how data is divided into packets, addressed, routed, and reassembled. The Internet Protocol (IP) is critical for routing packets between different networks. In addition to traditional protocols, advanced protocols such as Multiprotocol Label Switching (MPLS) and Segment Routing (SR) enhance traffic management and routing efficiency. For intra-domain routing, protocols like Open Shortest Path First (OSPF) and Enhanced Interior Gateway Routing Protocol (EIGRP) provide dynamic routing capabilities. On the local area network (LAN) level, protocols like Virtual Extensible LAN (VXLAN) and Network Virtualization using Generic Routing Encapsulation (NVGRE) facilitate the creation of virtual networks. Furthermore, Internet Protocol Security (IPsec) and Transport Layer Security (TLS) secure communication channels, ensuring data integrity and confidentiality. For real-time applications, protocols such as Real-time Transport Protocol (RTP) and WebRTC provide low-latency communication, making them suitable for video conferencing and streaming services. Additionally, protocols like QUIC enhance web performance and security by establishing secure connections with reduced latency. == Network security == As networks have become essential for business operations and personal communication, the demand for robust security measures has increased. Network security is a critical component of computer network engineering, concentrating on the protection of networks against unauthorized access, data breaches, and various cyber threats. Engineers are responsible for designing and implementing security measures that ensure the integrity and confidentiality of data transmitted across networks. Firewalls serve as barriers between trusted internal networks and external environments, such as the Internet. Network engineers configure firewalls, including next-generation firewalls (NGFW), which incorporate advanced features such as deep packet inspection and application awareness, thereby enabling more refined control over network traffic and protection against sophisticated attacks. In addition to firewalls, engineers use encryption protocols, including Internet Protocol Security (IPsec) and Transport Layer Security (TLS), to secure data in transit. These protocols provide a means of safeguarding sensitive information from interception and tampering. For secure remote access, Virtual Private Networks (VPNs) are deployed, using technologies to create encrypted tunnels for data transmission over public networks. These VPNs are often used for maintaining security when remote users access corporate networks but are also used ion other settings. To enhance threat detection and r
Social network hosting service
A social network hosting service is a web hosting service that specifically hosts the user creation of web-based social networking services, alongside related applications. Such services are also known as vertical social networks due to the creation of SNSes which cater to specific user interests and niches; like larger, interest-agnostic SNSes, such niche networking services may also possess the ability to create increasingly niche groups of users. == List of social network hosting services == Federated Media Publishing's BigTent BroadVision Clearvale Ning Wall.fm
Content management
Content management (CM) are a set of processes and technologies that support the collection, managing, and publishing of information in any form or medium. When stored and accessed via computers, this information may be more specifically referred to as digital content, or simply as content. Digital content may take the form of text (such as electronic documents), images, multimedia files (such as audio or video files), or any other file type that follows a content lifecycle requiring management. The process of content development and management is complex enough that various commercial software vendors (large and small), such as Interwoven and Microsoft, offer content management software to control and automate significant aspects of the content lifecycle. == Process == Content management practices and goals vary by mission and by organizational governance structure. News organizations, e-commerce websites, and educational institutions all use content management, but in different ways. This leads to differences in terminology and in the names and number of steps in the process. For example, some digital content is created by one or more authors. Over time that content may be edited. One or more individuals may provide some editorial oversight, approving the content for publication. Publishing may take many forms: it may be the act of "pushing" content out to others, or simply granting digital access rights to certain content to one or more individuals. Later that content may be superseded by another version of the content and thus retired or removed from use (as when this wiki page is modified). Content management is an inherently collaborative process. It often consists of the following basic roles and responsibilities: Creator – responsible for creating and editing content. Editor – responsible for tuning the content message and the style of delivery, including translation and localization. Publisher – responsible for releasing the content for use. Administrator – responsible for managing access permissions to folders, collections and files, usually accomplished by assigning access rights to user groups or roles. Admins may also assist and support users in various ways. Consumer, viewer or guest – the person who reads or otherwise consumes the content after it is published or shared. A critical aspect of content management is the ability to manage versions of content as it evolves (see also version control). Authors and editors often need to restore older versions of edited products due to a process failure or an undesirable series of edits. Time-sensitive content may also require updates as the subject matter evolves over time. Another equally important aspect of content management involves the creation, maintenance, and application of review standards. Each member of the content creation and review process has a unique role and set of responsibilities in the development or publication of the content. Each review team member requires clear and concise review standards. These must be maintained on an ongoing basis to ensure the long-term consistency and health of the knowledge base. A content management system is a set of automated processes that may support the following features: Import and creation of documents and multimedia material Identification of all key users and their roles The ability to assign roles and responsibilities to different instances of content categories or types Definition of workflow tasks often coupled with messaging so that content managers are alerted to changes in content The ability to track and manage multiple versions of a single instance of content The ability to publish the content to a repository to support access The ability to personalize content based on a set of rules Increasingly, the repository is an inherent part of the system, and incorporates enterprise search and retrieval. Content management systems take the following forms: Web content management system—software for web site management (often what content management implicitly means) Output of a newspaper editorial staff organization Workflow for article publication Document management systems Knowledge management software Single source content management system—content stored in chunks within a relational database Variant management system—where personnel tag source content (usually text and graphics) to represent variants stored as single source "master" content modules, resolved to the desired variant at publication (for example: automobile owners manual content for 12 model years stored as single master content files and "called" by model year as needed)—often used in concert with database chunk storage (see above) for large content objects == Governance structures == Content management expert Marc Feldman defines three primary content management governance structures: localized, centralized, and federated—each having its unique strengths and weaknesses. === Localized governance === By putting control in the hands of those closest to the content, the context experts, localized governance models empower and unleash creativity. These benefits come, however, at the cost of a partial-to-total loss of managerial control and oversight. === Centralized governance === When the levers of control are strongly centralized, content management systems are capable of delivering an exceptionally clear and unified brand message. Moreover, centralized content management governance structures allow for a large number of cost-savings opportunities in large enterprises, realized, for example, through (1) the avoidance of duplicated efforts in creating, editing, formatting, repurposing and archiving content; (2) process management and the streamlining of all content related labor; and/or (3) an orderly deployment or updating of the content management system. === Federated governance === Federated governance models potentially realize the benefits of both localized and centralized control while avoiding the weaknesses of both. While content management software systems are inherently structured to enable federated governance models, realizing these benefits can be difficult because it requires, for example, negotiating the boundaries of control with local managers and content creators. In the case of larger enterprises, in particular, the failure to fully implement or realize a federated governance structure equates to a failure to realize the full return on investment and cost savings that content management systems enable. == Implementation == Content management implementations must be able to manage content distributions and digital rights in content life cycle. Content management systems are usually involved with digital rights management in order to control user access and digital rights. In this step, the read-only structures of digital rights management systems force some limitations on content management, as they do not allow authors to change protected content in their life cycle. Creating new content using managed (protected) content is also an issue that gets protected contents out of management controlling systems. A few content management implementations cover all these issues.
IruSoft
IruSoft (Arabic: آيروسوفت) is an insurance regulatory platform designated for licensing, supervision and inspection of the insurance sector within a country. The platform introduced unique supervision-technology (suptech), insurance-technology (insurtech) and regulatory-technology (regtech) automated modules by which a regulator requires less resources to ensure fairness, transparency and competition and to prevent conflicts of interest in the sector. IruSoft was founded by Abdullah Al-Salloum and owned by the Insurance Regulatory Unit in Kuwait. The Insurance Regulatory Unit optimized processing insurance-sector's customer complaints by issuing Resolution No. (1) of 2022 that introduced IruSoft's complaints public module; an automated resolution center, by which the process of receiving submitted complaints, passing them on to the platforms of licensed insurance companies, tracking matter-related discussions and updates and getting them escalated if unresolved to be discussed by a committee assigned by the unit is integrally automated and analyzed for better key performance indicators.
Data refuge
Data Refuge is a public and collaborative project designed to address concerns about federal climate and environmental data that is in danger of being lost. In particular, the initiative addresses five main concerns: What are the best ways to safeguard data? How do federal agencies play a crucial role in collecting, managing, and distributing data? How do government priorities impact data's accessibility? Which projects and research fields depend on federal data? Which data sets are of value to research and local communities, and why? Data Refuge began as a grassroots organization in opposition to government data on climate change and the environment not being archived systemically. Data Refuge's main goal is to collect and allocate data in multiple safe locations to create a sustainable way of archiving old and new data. Data Refuge was initiated in 2016 to protect federal climate and environmental data that is vulnerable under an administration that denies climate change. The system aims to make public research-quality copies of federal climate and environmental data. Data Refuge is supported by the National Geographic Foundation, private donors, Libraries+ Network, Preserving Electronic Governance Initiative (PEGI), the Union of Concerned Scientists (USC), and the Penn Program in Environmental Humanities (PPEH). == Types of data == Data Refuge collects public federal data on the climate and environment in the form of satellite imagery, PDFs, and stories. The data are stored in multiple trusted locations as they are less vulnerable if in only one location, and to ensure accessibility for researchers. Through the Data Rescue events, Data Refuge has accumulated 4 terabytes of data, 30,000 URLs, and 800 participants. === Storytelling === Data Refuge collects stories on vulnerable federal climate and environmental data through: surveys, oral history, photo essays, maps, video shorts, and animations. The stories are archived in a public bank that showcase how federal environmental data support health and safety in communities. Data Stories are collected at Data Rescue events, which are partnered with universities, city and town halls, and advocacy groups. Data stories are collected and used to emphasize the importance of Data Refuge, in how the data on climate change and the environment are being used by people in the United States and across the world for meaningful practices.