Distributed Systems Concepts for Interviews

Senior system design interviews probe the theory behind the boxes: what consistency your design gives, what happens during a network partition, how nodes agree on a leader. You do not need academic depth — just accurate concepts and their practical consequences.

1. Explain the CAP theorem and PACELC.

CAP: during a network Partition, a distributed store must choose between Consistency (every read sees the latest write, or errors) and Availability (every request gets a response, possibly stale). PACELC adds: Else, when there is no partition, you trade Latency against Consistency — strong consistency needs coordination, which costs time.

2. What consistency models should you know?

• Strong (linearisable): reads always see the latest committed write — like a single machine.
• Read-your-writes: a user always sees their own updates.
• Monotonic reads: a user never sees data go backwards.
• Eventual: replicas converge if writes stop — fine for likes and view counts, not for balances.

3. How does replication work, and what is replication lag?

In leader–follower replication, writes go to the leader and are copied to followers, which serve reads. With asynchronous replication, followers can lag, so a user may not see a write they just made — fix with reading your own writes from the leader or sticky sessions. Synchronous replication avoids lag but slows writes. Multi-leader and leaderless (quorum) designs trade more complexity for availability.

4. What are quorums?

In leaderless systems with N replicas, a write waits for W acknowledgements and a read queries R replicas. If W + R > N, every read overlaps at least one replica with the latest write. For N = 3, W = 2 and R = 2 is a common balance of consistency and availability.

5. How do nodes agree on a leader or a value?

With a consensus algorithm such as Raft or Paxos: nodes elect a leader by majority vote, the leader replicates a log, and an entry is committed once a majority has stored it. A cluster of 2f + 1 nodes tolerates f failures. You rarely implement it — you use systems built on it (etcd, ZooKeeper, Consul) for configuration, locks and leader election.

6. How do you partition (shard) data well?

Choose a key that spreads load evenly and keeps related data together. Hash partitioning spreads evenly but makes range queries hard; range partitioning supports ranges but risks hot spots (all new timestamps on one shard). Consistent hashing minimises data movement when nodes join or leave. Watch for hot keys such as a celebrity account.

7. Why is idempotency essential in distributed systems?

Networks fail ambiguously — a request may have succeeded even if you never got the response — so clients retry and messages are delivered at least once. Make operations idempotent: use idempotency keys for payments and orders, upserts instead of blind inserts, and de-duplicate processed message ids.

POST /payments
Idempotency-Key: 7f3c-2026-order-1042

server:
  if key seen → return the stored response (no second charge)
  else        → charge, store (key → response), return response

8. Why can't you trust clocks across machines?

Physical clocks drift and are adjusted by NTP, so timestamps from different machines can be out of order — “last write wins” can silently drop data. Use logical clocks (Lamport timestamps, version vectors) to order events causally, or generate ordering from a single source such as a database sequence or the consensus leader.