Load Balancing in System Design Interviews: Algorithms, Layers and Failure
Load balancers appear in every system design diagram and get about four seconds of thought. That is fine until the interviewer asks "which algorithm, and what happens when one server is slow?" — at which point the box on the whiteboard needs to have a design behind it.
L4 or L7
Layer 4 balances on TCP: source and destination IP and port. It never looks at the payload, which makes it fast and protocol-agnostic, and means it can forward anything. It cannot route on URL path, read a header, or terminate TLS meaningfully.
Layer 7 terminates the connection and understands HTTP. Now you can route /api/* to one pool and /static/* to another, retry idempotent requests, rewrite headers, and do TLS termination in one place. You pay in CPU and a little latency.
Most systems want L7 at the edge. Reach for L4 when you need raw throughput or you are balancing something that is not HTTP — a database proxy, a message broker, a game server.
The algorithms, and when each is wrong
Round robin. Next server in rotation. Correct only when requests cost roughly the same and servers are identical. A single slow request type skews everything, because the balancer keeps feeding a struggling server its share regardless.
Weighted round robin. Same, with weights for heterogeneous hardware. Fixes capacity differences, not request-cost differences.
Least connections. Send to whichever server has the fewest in-flight requests. This is the sensible default for variable work: a server stuck on slow requests naturally stops receiving new ones, because its connection count stays high. If you only remember one upgrade from round robin, make it this.
Least response time. Least connections, weighted by observed latency. Better, and more sensitive to measurement noise.
Consistent hashing on a key. Routes the same key to the same server. You want this when the server holds state for that key — a cache tier, or a websocket session. Why hash mod N breaks here, and what the ring fixes is its own topic.
Health checks are the interesting part
A load balancer's real job is noticing that a server is broken. Be ready to say more than "it health-checks them."
Passive checks watch real traffic: enough 5xxs or timeouts from a server and it comes out of rotation. Free, but it finds the problem by serving errors to users first.
Active checks poll an endpoint on a timer. They cost a little traffic and find failures before users do.
The subtlety worth raising: a health endpoint that returns 200 as long as the process is alive is nearly useless. It should check what the service actually needs — database reachable, dependencies responding — or the balancer will keep routing to a server that is up but cannot do anything.
And the failure mode most people miss: if the health check is too aggressive, a brief dependency blip marks every server unhealthy at once, and the balancer takes the whole fleet out of rotation. Good implementations refuse to evict below a floor — if fewer than half are healthy, serve to all of them rather than to none.
Sticky sessions, and why to avoid them
Sticky sessions pin a user to a server so in-memory session state keeps working. They also break autoscaling (you cannot drain a server without dropping sessions), skew load, and turn one server's death into a visible logout for its users.
The better answer is to make servers stateless and put session state in Redis or a signed token. If an interviewer asks about sticky sessions, saying "I'd avoid needing them, and here's where the state goes instead" is stronger than explaining how to configure them.
How traffic reaches the balancer
A question that catches people: your load balancer is itself a single point of failure. The standard layers:
- DNS returns several IPs, or a region-aware answer. Coarse — DNS caching means you cannot fail over quickly.
- Anycast advertises one IP from many locations; the network routes to the nearest. This is how global load balancers and CDNs steer traffic.
- A redundant pair at the edge sharing a virtual IP, so one can die without taking the address with it.
For a multi-region design, the usual shape is: anycast or geo-DNS picks a region, then an L7 balancer inside the region picks a server.
What a good answer sounds like
"L7 at the edge so I can route by path and terminate TLS. Least connections rather than round robin, because our request costs vary a lot — image processing and a profile fetch shouldn't get equal shares. Active health checks against a dependency-aware endpoint, with a floor so a blip can't evict the whole fleet. Servers stateless, sessions in Redis, so I can scale in without draining sticky sessions."
Four sentences, every claim justified by a property of the system rather than a default. That is the difference between a box labelled "LB" and a design.