High-volume, distributed guaranteed message delivery

Problem

In large-scale distributed cloud environments, large volumes of messages must be moved to subscribing instances reliably. Message volumes may be continuous or in bursts. This is especially desired in social networking applications, monitoring and logging, distributing messages to map-reduce workers, news feeds to multiple subscribers, web sites, etc. Most often, consumers are geographically distributed across multiple clusters or availability zones. Messages are typically delivered in the order in which they are received, at least once to their subscribers.

Solution

This is a composite pattern that implements a ‘distributed message delivery” as a service.  The underlying implementation leverages ‘topic brokers’ that:

• Accepts request from ‘producers’ and provide a handle to topic boxes
• Enable ‘producers’ to post to topic mail boxes
• Push posted messages to ‘consumer group brokers’

Leverages ‘Consumer Group Brokers’ (could be based on MQ) that:

• Assign subscribers into logical groups (logical groups of subscribers interested in the similar topics)
• Ensure all consumers have had access to the subscribed message at least once (e.g., posting a message to a subscribers wall on Facebook)
• Ensure message delivery adheres to committed SLAs

Click to Enlarge

Application

• Broadcast of news feeds, blogs, updates to social networking walls
• Distribution of emergency alerts, messages
• Distribution of data to map-reduce workers

Impacts

Anti-Patterns

Related Patterns

• Relational Data Store as-a-service Pattern•Simple Data Store as-a-service Pattern
• Pub-Sub as-a-service Pattern
• Notification as-a-service Pattern
• Auto Scale as-a-service Pattern
• Load Balancing as-a-service Pattern
• Cloud Federation Pattern
• Cloud Bus Pattern
• Cloud Monitoring as-a-Service Pattern

References

• Kafka
• HedWig
• BigData