How to Study These Architectures
Don’t focus on nitty gritty details for the following articles, instead:- Identify shared principles, common technologies, and patterns within these articles
- Study what problems are solved by each component, where it works, where it doesn’t
- Review the lessons learned
Source: Twitter timelines at scale
Data Processing Systems
| System | Description | Reference |
|---|---|---|
| MapReduce | Distributed data processing from Google | research.google.com |
| Spark | Distributed data processing from Databricks | slideshare.net |
| Storm | Distributed data processing from Twitter | slideshare.net |
Data Store Systems
| System | Description | Reference |
|---|---|---|
| Bigtable | Distributed column-oriented database from Google | harvard.edu |
| HBase | Open source implementation of Bigtable | slideshare.net |
| Cassandra | Distributed column-oriented database from Facebook | slideshare.net |
| DynamoDB | Document-oriented database from Amazon | harvard.edu |
| MongoDB | Document-oriented database | slideshare.net |
| Spanner | Globally-distributed database from Google | research.google.com |
| Memcached | Distributed memory caching system | slideshare.net |
| Redis | Distributed memory caching system with persistence and value types | slideshare.net |
File Systems
| System | Description | Reference |
|---|---|---|
| Google File System (GFS) | Distributed file system | research.google.com |
| Hadoop File System (HDFS) | Open source implementation of GFS | apache.org |
Infrastructure Systems
| System | Description | Reference |
|---|---|---|
| Chubby | Lock service for loosely-coupled distributed systems from Google | research.google.com |
| Dapper | Distributed systems tracing infrastructure | research.google.com |
| Kafka | Pub/sub message queue from LinkedIn | slideshare.net |
| Zookeeper | Centralized infrastructure and services enabling synchronization | slideshare.net |
Company Architectures
Tech Giants
Amazon Architecture
Learn about Amazon’s distributed systems architecture
Google Architecture
Explore Google’s approach to building scalable systems
Facebook Engineering
Scaling memcached, TAO distributed data store, and photo storage
Netflix Stack
A 360 degree view of the entire Netflix stack
Social Media & Communication
| Company | Key Topics | Reference |
|---|---|---|
| Timelines at scale, 150M active users, 300K QPS | Making Twitter 10000% faster Timelines at scale | |
| 14 million users, terabytes of photos | Architecture | |
| Architecture Facebook bought for $19 billion | Architecture | |
| From 0 to 10s of billions of page views | Scaling Pinterest |
Other Notable Architectures
| Company | Key Topics | Reference |
|---|---|---|
| Uber | Real-time market platform, scaling to 2000 engineers | Scaling Uber |
| Dropbox | How Dropbox scaled their infrastructure | Video |
| YouTube | YouTube scalability and architecture | Scalability |
| Stack Overflow | Stack Overflow architecture | Architecture |
| Salesforce | Handling 1.3 billion transactions a day | Architecture |
Specialized Systems
| Company | Specialty | Reference |
|---|---|---|
| ESPN | Operating at 100,000 duh nuh nuhs per second | Architecture |
| Flickr | Photo sharing architecture | Architecture |
| Tumblr | 15 billion page views a month | Architecture |
| TripAdvisor | 40M visitors, 200M dynamic page views, 30TB data | Architecture |
| Cinchcast | Producing 1,500 hours of audio every day | Architecture |
| DataSift | Realtime datamining at 120,000 tweets per second | Architecture |