Scalability: latency, throughput, and backpressure

A scalable system preserves an acceptable user experience as its workload grows. Adding machines is one possible technique. It is not the definition. If every extra application instance opens enough database connections to overwhelm the same database, horizontal scaling has made the incident worse.

The useful starting point is a workload and a constraint. What operations arrive, how much work do they require, and which resource becomes scarce first? This guide develops that reasoning through latency, concurrency, queues, caching, and overload behavior.

Measure work rather than counting users

A million registered users says little about instantaneous load. A service with infrequent profile updates has a different workload from a live video platform with the same user count. Separate registrations, daily active users, concurrent sessions, requests per second, and in-flight requests.

Latency is the time one operation takes. Throughput is the number completed per unit time. Concurrency is the number in progress. These quantities are related, but improving one does not automatically improve the others. Batching may increase throughput while making individual requests wait longer.

Report a distribution of latency. A median of 50 milliseconds says nothing about whether one request in a hundred takes ten seconds. Track p50, p95, and p99 at meaningful boundaries, including the complete user operation. Averaging percentiles from separate instances does not produce the percentile of the combined request population; aggregate the observations or a compatible distribution representation.

Use Little’s Law as a consistency check

For a stable system over an appropriate measurement interval, Little’s Law relates average work in progress $L$, average arrival rate $\lambda$, and average time in the system $W$:

If an endpoint receives 200 requests per second and its average response time is 0.25 seconds, approximately 50 requests are in flight on average. If response time rises to two seconds at the same arrival rate, that becomes 400. Those requests occupy memory, connections, and scheduling capacity even if their original work did not become more computationally expensive.

Use consistent boundaries. Requests waiting in an application queue belong in both the concurrency measurement and the response-time measurement if that queue is inside the chosen system. An unstable queue that grows indefinitely does not have a useful steady-state average waiting time.

Little’s Law does not tell you the required number of threads or machines directly. An asynchronous server can manage many waiting operations with few threads, while a CPU-heavy operation needs actual compute. The law tells you how much work is present; implementation determines which resources that work consumes.

Understand why latency rises before capacity is exhausted

A simple single-server queue illustrates the danger of operating near full utilization. In an M/M/1 model, arrivals are Poisson, service times are exponential, and the system has one server with arrival rate $\lambda$ below service rate $\mu$. The mean time in the system is:

Suppose service takes ten milliseconds on average, so $\mu=100$ operations per second. At 80 arrivals per second, the model predicts 50 milliseconds in the system. At 95 arrivals per second, it predicts 200 milliseconds. A relatively small increase in arrivals causes a much larger increase in waiting.

Real systems have bursts, multiple workers, correlated dependencies, and non-exponential service times. The numerical formula is not a production capacity guarantee. Its lesson is qualitative: variability and saturation create nonlinear waiting. Leave headroom and measure the point where latency begins to deteriorate, rather than treating 100% utilization as an efficient steady operating target.

A slow dependency can start a feedback loop. Requests wait longer, more requests accumulate, clients time out, and retries create even more work. The incident may look like an application memory problem while its initiating cause is a database or network bottleneck.

Find the bottleneck before parallelizing

A request can spend time on CPU, database execution, lock waits, network transfer, external APIs, or queues. A trace helps separate those components. “The endpoint is slow” is not a diagnosis.

Parallelizing independent work can shorten the critical path. If three independent lookups take 40, 60, and 80 milliseconds, sequential execution takes roughly 180 milliseconds before overhead, while concurrent execution can approach the longest lookup. But concurrency also increases simultaneous demand on the dependencies. A faster endpoint under light load can become less stable at peak load.

Amdahl’s Law provides another limit. If fraction $s$ of an operation remains serial and the rest can be divided perfectly among $n$ workers, the theoretical speedup is:

With a 20% serial fraction, even infinitely many workers cannot exceed a fivefold speedup under this model. Real coordination costs usually make the limit less favorable. Eliminate unnecessary work and serial bottlenecks before increasing parallel fan-out.

Reduce database work at its source

An N+1 query pattern can turn one page load into dozens of round trips. Fetch the required relationships deliberately, inspect query plans, and measure rows scanned as well as rows returned. An index is useful when it matches the access pattern and its maintenance cost is justified.

For a learner’s activity history, an index beginning with learner_id can support a filter on that learner, followed by a stable sort key. PostgreSQL’s multicolumn index documentation explains why column order and query conditions matter. Adding every column to an index is not a substitute for understanding the query.

Connection pools should bound database concurrency. More connections do not create more disk throughput or remove lock contention. If each of fifty application instances opens a pool of forty, the potential connection count is two thousand. Coordinate pool sizes with the database’s measured capacity and with any transaction-pooling layer.

Read replicas can move some read work, but replication lag changes semantics. A learner who just submitted an assignment may not immediately see it through a lagging replica. Route read-after-write operations appropriately or make the consistency behavior explicit.

Treat caching as a consistency decision

If cache-eligible request rate is $\lambda_c$ and hit rate is $h$, the corresponding origin request rate is approximately:

A 90% hit rate on 1,000 eligible requests per second leaves 100 origin reads. The approximation assumes hits really avoid the origin operation and ignores refresh and invalidation traffic. Measure the actual reduction rather than inferring it from a dashboard percentage alone.

Every cache needs a key, ownership rule, freshness policy, and invalidation strategy. Include tenant and authorization context where required. A fast cache that serves another customer’s private response is a correctness failure, regardless of its hit rate.

Popular keys can create stampedes when they expire. Techniques include coalescing concurrent refreshes, spreading expiration times, and serving a bounded stale value while one refresh occurs. These techniques trade freshness against availability. They are appropriate for some catalog data and inappropriate for other decisions, such as authorizing a financial operation or enforcing a hard seat limit.

A queue needs a policy, not just a name

Queues decouple producers from workers and absorb temporary bursts. They cannot absorb a permanent mismatch between arrival and processing rates. If jobs arrive at 50 per second and workers complete 20, the backlog grows by roughly 30 per second while those rates continue.

Monitor the age of the oldest job as well as queue depth. Ten thousand tiny jobs may drain quickly; one hundred expensive jobs may not. Separate interactive, scheduled, and bulk work so a large export does not block password-reset delivery.

Backpressure means reducing or rejecting incoming work when the next stage cannot accept it safely. Bound queue sizes, bound in-flight operations, and set maximum useful job ages. Decide whether an expired job should be dropped, recomputed from newer data, or escalated. A job that starts after its result is no longer useful still consumes capacity.

The Google SRE chapter on handling overload discusses degradation and rejection strategies. In a learning platform, a sensible degraded mode might preserve course access and progress writes while temporarily disabling expensive recommendations. Define the priority before the incident.

Make timeouts and retries part of the capacity plan

A deadline bounds how long the entire operation may take. Individual dependency timeouts should fit within it, leaving time to handle failure and return a response. A caller that times out after one second gains little from a server continuing expensive work for another minute.

Retry only when the failure is plausibly transient and repeating the operation is safe. Use bounded attempts, exponential backoff, jitter, and a retry budget. If three layers independently make three total attempts, a single original operation can cause up to 27 calls to the deepest dependency in the worst case.

A timeout does not prove that a write failed. The server may have committed the operation while its response was lost. Stable idempotency keys or another explicit deduplication contract are needed for safe retries. The AWS Builders’ Library treatment of timeouts and retries is useful background, but each application still needs to define its own operation semantics.

Test the system you intend to operate

Load tests should reproduce the request mix, payload distribution, think time, hot keys, and dependency behavior you expect. A benchmark of one cached endpoint cannot establish the capacity of a platform whose expensive paths are uncached writes and reports.

An arrival-rate test can reveal overload that a closed loop hides. In a closed-loop test, a fixed number of clients waits for each response before sending another request; as the server slows, offered load falls. That may be realistic for some users, but it can conceal the effect of independent arrivals continuing during an incident.

Increase load gradually, hold it long enough to observe steady behavior, then test a burst and a dependency slowdown. Record latency distributions, successful throughput, rejection rate, queue age, database saturation, and cost. Test recovery too: a system that survives the peak but takes hours to drain is still operationally weak.

Practice: locate the knee in the curve

Choose a single endpoint backed by a realistic dataset. Measure its latency and throughput as offered load rises. Identify the resource that saturates first. Improve one cause, such as an N+1 query or an unbounded connection pool, and repeat the same experiment.

Then deliberately slow the database and compare behavior with bounded concurrency, deadlines, and no automatic retries against behavior with unbounded waiting. The most useful result is a clear explanation of where work accumulates and which policy prevents that accumulation from becoming a cascade.