A message queueing service is a message-oriented middleware or MOM deployed in a compute cloud using software as a service model. Service subscribers access queues and or topics to exchange data using point-to-point or publish and subscribe patterns. It's important to differentiate between event-driven and message-driven (aka queue driven) services: Event-driven services (e.g. AWS SNS) are decoupled from their consumers. Whereas queue / message driven services (e.g. AWS SQS) are coupled with their consumers. Message queues can be a good buffer to handle spiky workloads but they have a finite capacity. According to Gregor Hohpe, message queues require proper mechanisms (aka flow controls) to avoid filling the queue beyond its manageable capacity and to keep the system stable. == Ordering Guarantees in Message Queues == Amazon SQS FIFO and Azure Service Bus sessions are queue-based messaging systems that provide ordering guarantees within a message group or session attempt but do not necessarily guarantee ordered delivery in cases of retries or failures. In SQS FIFO, messages in the same message group are processed in order, with subsequent messages held until the preceding message is successfully processed or moved to the dead-letter queue (DLQ). Once a message is placed in the DLQ, it is no longer retried, creating a gap in the sequence. However, the remaining messages continue to be delivered in order. Azure Service Bus sessions function similarly by maintaining ordering within a session, provided a single consumer processes messages sequentially. The implementation differs from SQS FIFO but follows the same fundamental ordering principle. In contrast, Apache Kafka is a distributed log-based messaging system that guarantees ordering within individual partitions rather than across the entire topic. Unlike queue-based systems, Kafka retains messages in a durable, append-only log, allowing multiple consumers to read at different offsets. Kafka uses manual offset management, giving consumers control over retries and failure handling. If a consumer fails to process a message, it can delay committing the offset, preventing further progress in that partition while other partitions remain unaffected. This partition-based design enables fault isolation and parallel processing while allowing ordering to be maintained within partitions, depending on consumer handling. == Vendors == Apache Kafka Apache Kafka is a distributed system consisting of servers that store and forward messages between producer client and consumer applications. IBM MQ IBM MQ offers a managed service that can be used on IBM Cloud and Amazon Web Services. Microsoft Azure Service Bus Service Bus offers queues, topics & subscriptions, and rules/actions in order to support publish-subscribe, temporal decoupling, and load balancing scenarios. Azure Service Bus is built on AMQP allowing any existing AMQP 1.0 client stack to interact with Service Bus directly or via existing .Net, Java, Node, and Python clients. Standard and Premium tiers allow for pay as you go or isolated resources at massive scale. Oracle Messaging Cloud Service This service provides a messaging solution for applications for asynchronous communication and is influenced by the Java Message Service (JMS) API specification. Any application platform that understands HTTP can also use Oracle Messaging Cloud Service through the REST interface. For Java applications, Oracle Messaging Cloud Service provides a Java library that implements and extends the JMS 1.1 interface. The Java library implements the JMS API by acting as a client of the REST API. Amazon Simple Queue Service Supports messages natively up to 256K, or up to 2GB by transmitting payload via S3. Highly scalable, durable and resilient. Provides loose-FIFO and 'at least once' delivery in order to provide massive scale. Supports REST API and optional Java Message Service client. Low latency. Utilizes Amazon Web Services. IronMQ Supports messages up to 64k; guarantees order; guarantees once only delivery; no delays retrieving messages. Supports REST API and beanstalkd open source protocol. Runs on multiple clouds including AWS and Rackspace. Scaling must be managed by user. RabbitMQ RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine. Supports AMQP, STOMP, MQTT StormMQ Open platform supports messages up to 50Mb. Uses AMQP to avoid vendor lock-in and provide language neutrality. Locate-It Option allows customers to audit the location of their data at all times and satisfy data protection principles. AnypointMQ An enterprise multi-tenant, cloud messaging service that performs advanced asynchronous messaging scenarios between applications. Anypoint MQ is fully integrated with Anypoint Platform, offering role based access control, client application management, and connectors.
Machine learning in video games
Artificial intelligence and machine learning techniques are used in video games for a wide variety of applications such as non-player character (NPC) control, procedural content generation (PCG) and deep learning-based content generation. Machine learning is a subset of artificial intelligence that uses historical data to build predictive and analytical models. This is in sharp contrast to traditional methods of artificial intelligence such as search trees and expert systems. Information on machine learning techniques in the field of games is mostly known to public through research projects as most gaming companies choose not to publish specific information about their intellectual property. The most publicly known application of machine learning in games is likely the use of deep learning agents that compete with professional human players in complex strategy games. There has been a significant application of machine learning on games such as Atari/ALE, Doom, Minecraft, StarCraft, and car racing. Other games that did not originally exists as video games, such as chess and Go have also been affected by the machine learning. == Overview of relevant machine learning techniques == === Deep learning === Deep learning is a subset of machine learning which focuses heavily on the use of artificial neural networks (ANN) that learn to solve complex tasks. Deep learning uses multiple layers of ANN and other techniques to progressively extract information from an input. Due to this complex layered approach, deep learning models often require powerful machines to train and run on. ==== Convolutional neural networks ==== Convolutional neural networks (CNN) are specialized ANNs that are often used to analyze image data. These types of networks are able to learn translation invariant patterns, which are patterns that are not dependent on location. CNNs are able to learn these patterns in a hierarchy, meaning that earlier convolutional layers will learn smaller local patterns while later layers will learn larger patterns based on the previous patterns. A CNN's ability to learn visual data has made it a commonly used tool for deep learning in games. === Recurrent neural network === Recurrent neural networks are a type of ANN that are designed to process sequences of data in order, one part at a time rather than all at once. An RNN runs over each part of a sequence, using the current part of the sequence along with memory of previous parts of the current sequence to produce an output. These types of ANN are highly effective at tasks such as speech recognition and other problems that depend heavily on temporal order. There are several types of RNNs with different internal configurations; the basic implementation suffers from a lack of long term memory due to the vanishing gradient problem, thus it is rarely used over newer implementations. ==== Long short-term memory ==== A long short-term memory (LSTM) network is a specific implementation of a RNN that is designed to deal with the vanishing gradient problem seen in simple RNNs, which would lead to them gradually "forgetting" about previous parts of an inputted sequence when calculating the output of a current part. LSTMs solve this problem with the addition of an elaborate system that uses an additional input/output to keep track of long term data. LSTMs have achieved very strong results across various fields, and were used by several monumental deep learning agents in games. === Reinforcement learning === Reinforcement learning is the process of training an agent using rewards and/or punishments. The way an agent is rewarded or punished depends heavily on the problem; such as giving an agent a positive reward for winning a game or a negative one for losing. Reinforcement learning is used heavily in the field of machine learning and can be seen in methods such as Q-learning, policy search, Deep Q-networks and others. It has seen strong performance in both the field of games and robotics. === Neuroevolution === Neuroevolution involves the use of both neural networks and evolutionary algorithms. Instead of using gradient descent like most neural networks, neuroevolution models make use of evolutionary algorithms to update neurons in the network. Researchers claim that this process is less likely to get stuck in a local minimum and is potentially faster than state of the art deep learning techniques. == Deep learning agents == Machine learning agents have been used to take the place of a human player rather than function as NPCs, which are deliberately added into video games as part of designed gameplay. Deep learning agents have achieved impressive results when used in competition with both humans and other artificial intelligence agents. === Chess === Chess is a turn-based strategy game that is considered a difficult AI problem due to the computational complexity of its board space. Similar strategy games are often solved with some form of a Minimax Tree Search. These types of AI agents have been known to beat professional human players, such as the historic 1997 Deep Blue versus Garry Kasparov match. Since then, machine learning agents have shown ever greater success than previous AI agents. === Go === Go is another turn-based strategy game which is considered an even more difficult AI problem than chess. The state space of is Go is around 10^170 possible board states compared to the 10^120 board states for Chess. Prior to recent deep learning models, AI Go agents were only able to play at the level of a human amateur. ==== AlphaGo ==== Google's 2015 AlphaGo was the first AI agent to beat a professional Go player. AlphaGo used a deep learning model to train the weights of a Monte Carlo tree search (MCTS). The deep learning model consisted of 2 ANN, a policy network to predict the probabilities of potential moves by opponents, and a value network to predict the win chance of a given state. The deep learning model allows the agent to explore potential game states more efficiently than a vanilla MCTS. The network were initially trained on games of humans players and then were further trained by games against itself. ==== AlphaGo Zero ==== AlphaGo Zero, another implementation of AlphaGo, was able to train entirely by playing against itself. It was able to quickly train up to the capabilities of the previous agent. === StarCraft series === StarCraft and its sequel StarCraft II are real-time strategy (RTS) video games that have become popular environments for AI research. Blizzard and DeepMind have worked together to release a public StarCraft 2 environment for AI research to be done on. Various deep learning methods have been tested on both games, though most agents usually have trouble outperforming the default AI with cheats enabled or skilled players of the game. ==== Alphastar ==== Alphastar was the first AI agent to beat professional StarCraft 2 players without any in-game advantages. The deep learning network of the agent initially received input from a simplified zoomed out version of the gamestate, but was later updated to play using a camera like other human players. The developers have not publicly released the code or architecture of their model, but have listed several state of the art machine learning techniques such as relational deep reinforcement learning, long short-term memory, auto-regressive policy heads, pointer networks, and centralized value baseline. Alphastar was initially trained with supervised learning, it watched replays of many human games in order to learn basic strategies. It then trained against different versions of itself and was improved through reinforcement learning. The final version was hugely successful, but only trained to play on a specific map in a protoss mirror matchup. === Dota 2 === Dota 2 is a multiplayer online battle arena (MOBA) game. Like other complex games, traditional AI agents have not been able to compete on the same level as professional human player. The only widely published information on AI agents attempted on Dota 2 is OpenAI's deep learning Five agent. ==== OpenAI Five ==== OpenAI Five utilized separate long short-term memory networks to learn each hero. It trained using a reinforcement learning technique known as Proximal Policy Learning running on a system containing 256 GPUs and 128,000 CPU cores. Five trained for months, accumulating 180 years of game experience each day, before facing off with professional players. It was eventually able to beat the 2018 Dota 2 esports champion team in a 2019 series of games. === Planetary Annihilation === Planetary Annihilation is a real-time strategy game which focuses on massive scale war. The developers use ANNs in their default AI agent. === Supreme Commander 2 === Supreme Commander 2 is a real-time strategy (RTS) video game. The game uses Multilayer Perceptrons (MLPs) to control a platoon’s reaction to encountered enemy units. Total of four MLPs are used, one for each platoon type: land, naval
Telligent Community
Telligent Community is a community and collaboration software platform developed by Telligent Systems and was first released in 2004. Telligent Community is built on the Telligent Evolution platform, with a variety of core applications running on top of it such as blogs, forums, media galleries, and wikis. Additional applications from third parties using the API's and REST stack can be installed or integrated with the platform. Telligent Community is built with ASP.NET, C#, and Microsoft SQL Server. It is available as downloadable software that can be installed on a web server or via hosting providers. The current version is Verint Community 12.0 which was released February 2012. The product used to be named Community Server before being rebranded as part of the 5.0 release. == History == Telligent Systems was founded by Rob Howard in 2004, who was previously part of Microsoft's ASP.NET team. Telligent introduced its first product, Community Server, in the fall of 2004. Community Server was one of the first integrated community platforms that brought together blogs, photo galleries, wikis, forums, user profiles and more. Community Server was based on the merger of three then-widely used open source ASP.NET projects: the ASP.NET Forums, nGallery photo gallery, and .Text blog engine. The people behind those projects (Scott Watermasysk, Jason Alexander, and Rob Howard) joined together as Telligent Systems and along with several other software developers created Community Server 1.0. Between 2004 and 2009 Community Server steadily grew in scope, features, and capabilities. In 2008 Telligent Systems released a second version of Community Server that targeted as an Enterprise Social Software platform used to create and manage internal employee communities and intranets. Originally branded as Community Server Evolution this was later renamed Telligent Enterprise. Telligent also announced a new Enterprise Reporting platform at its first Community Server Developers Conference in 2008, which was later renamed Harvest. It was one of the first analytics suites for enterprise collaboration software, and provides social analytics including sentiment analysis, social fingerprints, and buzz analysis on social networking sites such as Twitter. Telligent rebranded all of its products on June 23, 2009 at the Enterprise 2.0 conference when it launched its new Evolution platform product suite. Community Server became known as Telligent Community, Community Server Evolution became known as Telligent Enterprise and the underlying platform that both run on is now referred to as Telligent Evolution. The Social Analytics suite was renamed Telligent Analytics.
ElabFTW
eLabFTW is a web application written by Nicolas Carpi in PHP which can be used to create personal and common logbooks. It has been developed at the Curie Institute originally. Besides there, it is used on universities around the world eLabFTW is licensed under the GNU Affero General Public License as free software. It is translated into seven languages. == Description == eLabFTW is a free and open-source lab book. It is written in PHP and uses a MySQL database. Docker containers are also available. Among the various features are Secure. Entries and transmission are encrypted Timestamps. RFC 3161 compliant timestamping of experiments. Inventory management. Apart from experience logs, it also can manage the inventory Import and export. Entries can be imported and exported == Platforms == eLabFTW is a PHP package with Mysql database. Therefore, it can be executed on most servers. Furthermore, the docker containers allow to run it almost everywhere. == Usage == eLabFTW is used by various universities, like University of Alberta, Berkeley University, Hanover Medical School, Cardiff University and UMC Utrecht
Clesh
Clesh (clip load edit share) is a cloud-based video editing platform, created by Forbidden Technologies plc, designed for the consumers, prosumers, and online communities to integrate user-generated content. The core technology is based on FORscene which is geared towards professionals working for example in broadcasting, news media, post production. Video, audio, and graphical content is uploaded to Clesh via a standard web browser, a mobile device such as a phone / tablet, or desktop software for DV capture over FireWire. The hosted material can then be reviewed, searched, edited, and published online by anyone with a standard web browser or compatible mobile device. Clesh supports storyboard shot selection, frame-accurate editing, transitions and various other functions such as; pan, zoom, colour and light correction, and audio levels. Content can be published in formats for example; Podcast, Mpeg2, HTML video or in a proprietary Java format. Cloud-based software provides greater scope for sharing information and collaborating compared to LAN or desktop based systems. Users of cloud-based software rely on the cloud's owner for adequate security, performance and resilience. Clesh does not assert any rights over uploaded content in contrast to other platforms (such as YouTube). All rights to any content uploaded to Clesh remain with the Author. == Features == Some of the services available to Clesh users: Access via Java enabled desktops or Android smartphones or tablets Real-time video rendering including effects and transitions Multiple audio tracks Secured log-on Frame accurate timeline for fine cut editing Logging / meta-data annotation assigns text to portions of video (usable by Clesh and web search engines) Storyboard assembles rough cuts using drag-and-drop Import, host, organise and search for media (DV tape and various video, audio, and still image formats) Publish content to in formats such as podcast, MPEG-2, web (Java Applet), Flash, Ogg, HTML and JPEG Chatrooms to talk to other Clesh users Showreel (a gallery for publishing material visible to internet users) Moderation for approval of material prior to distribution downstream Re-branding and integration support for white-label deployment == Technology == Clesh is based on the same technology as FORscene. An array of servers on the internet backbone provide the cloud computing platform to host Clesh. As a white-label solution Clesh would be branded and hosted per the client requirement. == User interface == End-users access Clesh on clients such as standard Java-enabled Web Browsers and / or Android enabled mobile devices such as tablets and smartphones. == History == Clesh was launched January 2006 and subject to several upgrades during the year to extend functionality including; storyboard, podcasting, moderation, chat and a showreel. During 2007 consumers are offered Clesh via a subscription model. Upgrades include Web Start and graphics upload. Mr Paparazzi selects Clesh as the platform to host its video offering and TrueTube does the same in 2008 by choosing to use Clesh to manage its video portal. Several further upgrades are applied and include; better audio quality, image enhancement controls, transitions, fades, titles, and additional publishing options such as JPEG. In 2010 a version of Clesh is demonstrated on an Android OS tablet device (Samsung Galaxy S Tab), and several upgrades are applied including; HTML publishing, pan, zoom, and overlays.
Progressive Graphics File
PGF (Progressive Graphics File) is a wavelet-based bitmapped image format that employs lossless and lossy data compression. PGF was created to improve upon and replace the JPEG format. It was developed at the same time as JPEG 2000 but with a focus on speed over compression ratio. PGF can operate at higher compression ratios without taking more encoding/decoding time and without generating the characteristic "blocky and blurry" artifacts of the original DCT-based JPEG standard. It also allows more sophisticated progressive downloads. == Color models == PGF supports a wide variety of color models: Grayscale with 1, 8, 16, or 31 bits per pixel Indexed color with palette size of 256 RGB color image with 12, 16 (red: 5 bits, green: 6 bits, blue: 5 bits), 24, or 48 bits per pixel ARGB color image with 32 bits per pixel Lab color image with 24 or 48 bits per pixel CMYK color image with 32 or 64 bits per pixel == Technical discussion == PGF claims to achieve an improved compression quality over JPEG adding or improving features such as scalability. Its compression performance is similar to the original JPEG standard. Very low and very high compression rates (including lossless compression) are also supported in PGF. The ability of the design to handle a very large range of effective bit rates is one of the strengths of PGF. For example, to reduce the number of bits for a picture below a certain amount, the advisable thing to do with the first JPEG standard is to reduce the resolution of the input image before encoding it — something that is ordinarily not necessary for that purpose when using PGF because of its wavelet scalability properties. The PGF process chain contains the following four steps: Color space transform (in case of color images) Discrete Wavelet Transform Quantization (in case of lossy data compression) Hierarchical bit-plane run-length encoding === Color components transformation === Initially, images have to be transformed from the RGB color space to another color space, leading to three components that are handled separately. PGF uses a fully reversible modified YUV color transform. The transformation matrices are: [ Y r U r V r ] = [ 1 4 1 2 1 4 1 − 1 0 0 − 1 1 ] [ R G B ] ; [ R G B ] = [ 1 3 4 − 1 4 1 − 1 4 − 1 4 1 − 1 4 3 4 ] [ Y r U r V r ] {\displaystyle {\begin{bmatrix}Y_{r}\\U_{r}\\V_{r}\end{bmatrix}}={\begin{bmatrix}{\frac {1}{4}}&{\frac {1}{2}}&{\frac {1}{4}}\\1&-1&0\\0&-1&1\end{bmatrix}}{\begin{bmatrix}R\\G\\B\end{bmatrix}};\qquad \qquad {\begin{bmatrix}R\\G\\B\end{bmatrix}}={\begin{bmatrix}1&{\frac {3}{4}}&-{\frac {1}{4}}\\1&-{\frac {1}{4}}&-{\frac {1}{4}}\\1&-{\frac {1}{4}}&{\frac {3}{4}}\end{bmatrix}}{\begin{bmatrix}Y_{r}\\U_{r}\\V_{r}\end{bmatrix}}} The chrominance components can be, but do not necessarily have to be, down-scaled in resolution. === Wavelet transform === The color components are then wavelet transformed to an arbitrary depth. In contrast to JPEG 1992 which uses an 8x8 block-size discrete cosine transform, PGF uses one reversible wavelet transform: a rounded version of the biorthogonal CDF 5/3 wavelet transform. This wavelet filter bank is exactly the same as the reversible wavelet used in JPEG 2000. It uses only integer coefficients, so the output does not require rounding (quantization) and so it does not introduce any quantization noise. === Quantization === After the wavelet transform, the coefficients are scalar-quantized to reduce the amount of bits to represent them, at the expense of a loss of quality. The output is a set of integer numbers which have to be encoded bit-by-bit. The parameter that can be changed to set the final quality is the quantization step: the greater the step, the greater is the compression and the loss of quality. With a quantization step that equals 1, no quantization is performed (it is used in lossless compression). In contrast to JPEG 2000, PGF uses only powers of two, therefore the parameter value i represents a quantization step of 2i. Just using powers of two makes no need of integer multiplication and division operations. === Coding === The result of the previous process is a collection of sub-bands which represent several approximation scales. A sub-band is a set of coefficients — integer numbers which represent aspects of the image associated with a certain frequency range as well as a spatial area of the image. The quantized sub-bands are split further into blocks, rectangular regions in the wavelet domain. They are typically selected in a way that the coefficients within them across the sub-bands form approximately spatial blocks in the (reconstructed) image domain and collected in a fixed size macroblock. The encoder has to encode the bits of all quantized coefficients of a macroblock, starting with the most significant bits and progressing to less significant bits. In this encoding process, each bit-plane of the macroblock gets encoded in two so-called coding passes, first encoding bits of significant coefficients, then refinement bits of significant coefficients. Clearly, in lossless mode all bit-planes have to be encoded, and no bit-planes can be dropped. Only significant coefficients are compressed with an adaptive run-length/Rice (RLR) coder, because they contain long runs of zeros. The RLR coder with parameter k (logarithmic length of a run of zeros) is also known as the elementary Golomb code of order 2k. === Comparison with other file formats === JPEG 2000 is slightly more space-efficient in handling natural images. The PSNR for the same compression ratio is on average 3% better than the PSNR of PGF. It has a small advantage in compression ratio but longer encoding and decoding times. PNG (Portable Network Graphics) is more space-efficient in handling images with many pixels of the same color. There are several self-proclaimed advantages of PGF over the ordinary JPEG standard: Superior compression performance: The image quality (measured in PSNR) for the same compression ratio is on average 3% better than the PSNR of JPEG. At lower bit rates (e.g. less than 0.25 bits/pixel for gray-scale images), PGF has a much more significant advantage over certain modes of JPEG: artifacts are less visible and there is almost no blocking. The compression gains over JPEG are attributed to the use of DWT. Multiple resolution representation: PGF provides seamless compression of multiple image components, with each component carrying from 1 to 31 bits per component sample. With this feature there is no need for separately stored preview images (thumbnails). Progressive transmission by resolution accuracy, commonly referred to as progressive decoding: PGF provides efficient code-stream organizations which are progressive by resolution. This way, after a smaller part of the whole file has been received, it is possible to see a lower quality of the final picture, the quality can be improved monotonically getting more data from the source. Lossless and lossy compression: PGF provides both lossless and lossy compression in a single compression architecture. Both lossy and lossless compression are provided by the use of a reversible (integer) wavelet transform. Side channel spatial information: Transparency and alpha planes are fully supported ROI extraction: Since version 5, PGF supports extraction of regions of interest (ROI) without decoding the whole image. == Available software == The author published libPGF via a SourceForge, under the GNU Lesser General Public License version 2.0. Xeraina offers a free Windows console encoder and decoder, and PGF viewers based on WIC for 32bit and 64bit Windows platforms. Other WIC applications including File Explorer are able to display PGF images after installing this viewer. Digikam is a popular open-source image editing and cataloging software that uses libPGF for its thumbnails. It makes use of the progressive decoding feature of PGF images to store a single version of each thumbnail, which can then be decoded to different resolutions without loss, thus allowing users to dynamically change the size of the thumbnails without having to recalculate them again.
Apache Hama
Apache Hama is a distributed computing framework based on bulk synchronous parallel computing techniques for massive scientific computations e.g., matrix, graph and network algorithms. Originally a sub-project of Hadoop, it became an Apache Software Foundation top level project in 2012. It was created by Edward J. Yoon, who named it (short for "Hadoop Matrix Algebra"), and Hama also means hippopotamus in Yoon's native Korean language (하마), following the trend of naming Apache projects after animals and zoology (such as Apache Pig). Hama was inspired by Google's Pregel large-scale graph computing framework described in 2010. When executing graph algorithms, Hama showed a fifty-fold performance increase relative to Hadoop. Retired in April 2020, project resources are made available as part of the Apache Attic. Yoon cited issues of installation, scalability, and a difficult programming model for its lack of adoption. == Architecture == Hama consists of three major components: BSPMaster, GroomServers and Zookeeper. === BSPMaster === BSPMaster is responsible for: Maintaining groom server status Controlling super steps in a cluster Maintaining job progress information Scheduling jobs and assigning tasks to groom servers Disseminating execution class across groom servers Controlling fault Providing users with the cluster control interface. A BSP Master and multiple grooms are started by the script. Then, the bsp master starts up with a RPC server for groom servers. Groom servers starts up with a BSPPeer instance and a RPC proxy to contact the bsp master. After started, each groom periodically sends a heartbeat message that encloses its groom server status, including maximum task capacity, unused memory, and so on. Each time the BSP master receives a heartbeat message, it brings the groom server status up-to-date. The bsp master makes use of groom servers' status in order to assign tasks to idle groom servers - and returns a heartbeat response containing assigned tasks and others actions for a groom server to do. Currently BSP master has a FIFO job scheduler and simple task assignment algorithms. === GroomServer === A groom server (shortly referred to as groom) is a process that performs BSP tasks assigned by BSPMaster. Each groom contacts the BSPMaster, and it takes assigned tasks and reports its status by means of periodical piggybacks with BSPMaster. Each groom is designed to run with HDFS or other distributed storages. Basically, a groom server and a data node should be run on one physical node. === Zookeeper === A Zookeeper is used to manage the efficient barrier synchronisation of the BSPPeers.