Lambda architecture

http://lambda-architecture.net

eric baldeschwieler

The Lambda Architecture aims to satisfy the needs for a robust system that is fault-tolerant, both against hardware failures and human mistakes, being able to serve a wide range of workloads and use cases, and in which low-latency reads and updates are required. The resulting system should be linearly scalable, and it should scale out rather than up.

1. All data entering the system is dispatched to both the batch layer and the speed layer for processing. 
2. The batch layer has two functions: (i) managing the master dataset (an immutable, append-only set of raw data), and (ii) to pre-compute the batch views. 
3. The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way. 
4. The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only. 
5. Any incoming query can be answered by merging results from batch views and real-time views.

Example realtime application:
Recommender
Newsfeed

From eric baldeschwieler 

http://www.slideshare.net/jeric14/hadoop-where-did-it-come-from-and-whats-next-padadena-oc

Message bus like Kafka, Flume, Scribe
Realtime engine: Kafka, Storm, Samza, DataTorent
Serving Store: Cassandra, MySQL
Service: Slider, Twill, Hbase, Sqooq