Rate limiting is one of the most frequently asked topics in System Design and Senior Backend Engineering interviews. Whether designing public APIs, microservice gateways, or AI token billing, engineers must protect systems against resource exhaustion, malicious abuse, and cascading failures.
1. Why Rate Limiting Matters
Without rate limiting, any public or internal API is vulnerable to severe operational risks:
- Resource Exhaustion & Denial of Service (DoS): Unchecked spikes saturate database connection pools, thread pools, and memory.
- Cost Overruns (LLMs & Third-Party APIs): Consuming upstream APIs (e.g. OpenAI, Stripe, AWS) without quotas can generate massive unexpected bills.
- The "Noisy Tenant" Problem: In multi-tenant SaaS architectures, one aggressive user can consume 90% of available capacity, causing high latency, timeouts, and downtime for innocent users sharing the same infrastructure.
- Scraping & Credential Stuffing: Brute-force login attempts and automated data scraping depend on rapid-fire request loops.
2. The 4 Fundamental Rate Limiting Algorithms
Interviewers expect you to compare these 4 core algorithms based on memory consumption, accuracy, and burst tolerance:
| Algorithm | Burst Tolerance | Memory per Key | Complexity | Best For |
|---|---|---|---|---|
| Token Bucket | High (up to capacity) | Minimal (O(1)) | Low | General public APIs (Stripe, AWS) |
| Leaky Bucket | None (constant rate) | Low (buffer size) | Low | Egress queues, video streaming, traffic smoothing |
| Fixed Window Counter | High (2x burst at boundary) | Minimal (O(1)) | Very Low | Simple coarse-grained quotas |
| Sliding Window Log / Counter | Accurate | Moderate (O(N) or O(1) hybrid) | Moderate | High-precision finance & security APIs |
A. Token Bucket
429 Too Many Requests.
- Advantage: Handles bursts gracefully up to bucket capacity while strictly enforcing average throughput.B. Leaky Bucket
C. Fixed Window Counter
D. Sliding Window Counter (Production Standard)
Requests in Window = Current Count + Previous Count × (1 - Elapsed in Current Window / Window Size)3. Standard HTTP Headers for Rate Limiting
A polite, production-ready API must communicate rate limit status using standardized headers:
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 30
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1712839230
{
"error": "Too Many Requests",
"message": "Quota exceeded. Please retry after 30 seconds."
}- •
429 Too Many Requests: RFC 6585 standard status code. - •
Retry-After: Number of seconds (or HTTP date) to wait before retrying. - •
RateLimit-Remaining: Remaining allowed calls in the current active window. - •
RateLimit-Reset: Unix epoch timestamp when quota refreshes.
4. Distributed Rate Limiting with Redis & Lua
In a distributed multi-node cluster, in-memory Node.js state cannot enforce global limits across multiple server instances. Centralizing rate counters in Redis is the industry standard.
The Race Condition Problem
INCR. Under high concurrency, hundreds of requests can execute between the read and increment steps, completely bypassing the limit.The Atomic Solution: Redis Lua Script
-- KEYS[1]: rate limit key (e.g. "ratelimit:user_123")
-- ARGV[1]: window in seconds
-- ARGV[2]: max allowed requests
local current = redis.call('INCR', KEYS[1])
if tonumber(current) == 1 then
redis.call('EXPIRE', KEYS[1], ARGV[1])
end
if tonumber(current) > tonumber(ARGV[2]) then
return 0 -- Rejected
else
return 1 -- Allowed
end5. Practical In-Memory Implementation
Here is a clean sliding-window rate limiter implementation in modern TypeScript/JavaScript suitable for coding interviews:
interface RateLimiterOptions {
limit: number;
windowMs: number;
}
interface RateLimiter {
tryAcquire(key: string): boolean;
getRemaining(key: string): number;
reset(key?: string): void;
}
function createRateLimiter({ limit, windowMs }: RateLimiterOptions): RateLimiter {
// Key -> array of valid request timestamps
const records = new Map<string, number[]>();
const getValidTimestamps = (key: string): number[] => {
const now = Date.now();
const cutoff = now - windowMs;
const timestamps = (records.get(key) || []).filter((t) => t > cutoff);
records.set(key, timestamps);
return timestamps;
};
return {
tryAcquire(key: string): boolean {
const valid = getValidTimestamps(key);
if (valid.length < limit) {
valid.push(Date.now());
return true;
}
return false;
},
getRemaining(key: string): number {
const valid = getValidTimestamps(key);
return Math.max(0, limit - valid.length);
},
reset(key?: string): void {
if (key !== undefined) {
records.delete(key);
} else {
records.clear();
}
},
};
}6. Client-Side Resilience: Exponential Backoff with Jitter
When a client receives a 429 Too Many Requests response, retrying immediately causes retry storms that hammer recovering servers. Clients should use Exponential Backoff with Full Jitter:
async function fetchWithRetry(url, maxRetries = 4, baseDelayMs = 250) {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
const res = await fetch(url);
if (res.status !== 429) return res;
// Parse server Retry-After header or calculate exponential backoff
const serverRetryAfter = res.headers.get("Retry-After");
const delay = serverRetryAfter
? parseInt(serverRetryAfter, 10) * 1000
: Math.random() * (baseDelayMs * Math.pow(2, attempt)); // Full Jitter
await new Promise((resolve) => setTimeout(resolve, delay));
}
throw new Error("Exceeded max retry attempts due to rate limiting.");
}Practice Rate Limiter Live on Pairlet
Frequently Asked Questions
What is the primary difference between Token Bucket and Leaky Bucket?
Token Bucket allows short bursts of traffic up to the bucket capacity as long as tokens are available, refilling at a steady rate. Leaky Bucket smooths traffic to a strictly constant outbound processing rate, dropping or queueing any requests that exceed buffer depth.
How do you prevent race conditions when rate limiting across distributed servers?
Use an in-memory data store like Redis with atomic Lua scripts. By executing rate check and increment operations in a single atomic Lua script, Redis avoids concurrent read-modify-write race conditions without requiring expensive distributed locks.
What is the 'Noisy Tenant' problem in multi-tenant APIs?
A noisy tenant is a single customer or API consumer who monopolizes shared infrastructure resources (CPU, DB connections, downstream quotas) with excessive requests, degrading response times and causing outages for all other tenants.
What HTTP response headers should a rate-limited API return?
When a request is throttled, return status code 429 Too Many Requests along with Retry-After (seconds until retry is permitted). Standard APIs also include RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset on every response.
Conduct Live Coding Interviews in the Browser
No candidate sign-up required. Create a room, share the link, and code together in real time with shared code execution.