Maxing out a $12 server.

The app

A small social media API, something like Twitter. Express 5 and TypeScript in one Node process. Four endpoints:

The database was seeded with 50,000 users, 500,000 posts and 2,016,005 likes, the same every time (fixed seed). That is 357 MB on disk.

The stack

Everything runs on the same box. There is no separate database server. Postgres memory was sized to the machine (shared_buffers 512 MB), and I raised Nginx and systemd connection limits so a default setting would not end the test early.

Generating load

I used k6 to create virtual users. Each virtual user is one distinct account with its own login token, and it runs this loop forever:

1. Load the feed (GET /feed)
2. Wait 3 to 7 seconds
3. Open a post (GET /posts/:id)
4. Wait 3 to 8 seconds
5. 15% chance to like it, 2% chance to write a post
6. Wait 5 to 15 seconds, then start over

That works out to about 20 seconds of thinking per loop and about 0.1 requests per second per user. So 1,000 users is roughly 100 requests per second. This matters. A number like "10,000 requests per second" tells you nothing about users until you know how often a user makes a request.

The load generator

k6 ran on a second droplet in the same datacenter: 4 vCPU and 8 GB, about $0.13 per hour. Round trip between the two boxes was under 1 ms. It peaked at 37% of one core, so it was never the bottleneck. I rented it for about four and a half hours, which cost about $0.60.

What "handling it" means

A load level passes if, over the whole run:

p95 is the time 95% of requests finish under. Averages hide pain. The slowest few percent of requests are the ones your users notice.

Finding the limit

I started small (10, 50, 100 users), then jumped to 1,000, then searched: 1,000 passed, 2,000 passed, 4,000 failed, 3,000 failed, 2,500 passed, 2,750 failed. Then I ran 2,500 three more times for five minutes each to confirm it.

Drag the slider to see what happened at each level:

Two things stand out. Up to 1,000 users the median is about 5 ms. That is the floor: one ~1 ms Postgres query plus JSON, through Nginx. And the tail goes first. At 2,000 users the median barely moved, but p95 jumped from 19 ms to 161 ms. Between 2,500 and 3,000 the median itself explodes. That is a queue forming.

The answer: 2,500 concurrent users

What broke

The CPU. Nothing else.

Every request costs about 3.6 ms of the one core. There are 1,000 ms in a second. That is the whole story. With no think time, hammering GET /feed directly, the box topped out at about 280 requests per second no matter how many connections I used. 2,500 realistic users was 232 req/s, about 83% of that ceiling.

The database was never the problem. Postgres sat idle 60% of the time while the site fell over. The expensive part was Node doing the same work over and over: running the feed query and turning the same 20 posts into JSON for every single request.

Change 1: cache the feed inside Node for one second

The feed is the same for everyone, and it was over half of all requests. So render it once, keep the JSON in memory for one second, and hand that out to everyone who asks in that second. No Redis and no extra server. If you only have one server, the cache can live inside your process.

let feedCache = null; // { body, expires }

get('/feed', async (req, res) => {
  if (feedCache && feedCache.expires > Date.now())
    return res.send(feedCache.body);   // HIT
  const body = await renderFeed();
  feedCache = { body, expires: Date.now() + 1000 };
  res.send(body);
});

The real version also shares one in-flight render between requests that miss at the same time, and drops the cache when someone writes a post. Like counts can be up to one second stale.

Result: 4,000 users (2 of 3 five-minute runs passed, median p99 998 ms, right on the line). The raw feed ceiling went from about 280 to about 1,200 requests per second.

Change 2: move the cache into Nginx

Node was still waking up for every feed request just to return the cached copy. Nginx can do that itself. A one-second micro-cache in Nginx means Node never sees most feed requests.

location = /feed {
  proxy_cache feed_cache;
  proxy_cache_valid 200 1s;
  proxy_cache_lock on;
}

proxy_cache_lock makes sure only one request goes to Node when the cache expires. Everyone else waits for that one.

Result: 5,250 users at p95 160 ms and p99 454 ms. The raw feed ceiling went past 10,800 requests per second, and at that point the load generator was the limit, not the server.

All three versions

Same 4,000 users on each version, to make it concrete: the baseline collapsed (p50 2.7 seconds), the Node cache passed right on the line (p99 998 ms), and the Nginx cache passed easily (p99 295 ms).

The new limit is the part I did not cache: GET /posts/:id and likes. At 8,000 users the box started returning real errors (504s and 502s) as Node's connection queue overflowed.

What 5,250 concurrent users means

5,250 concurrent means 5,250 people actively using the app at the same moment. People don't use your app all day. They open it a few times, for a few minutes. If 5 to 10% of your daily users are online at any given moment, 5,250 concurrent is tens of thousands of daily active users, on a $12 server.

That ratio depends entirely on your app, so treat it as a rough guide. The concurrent number is what I measured.

Caveats

• The 5,250 number is a single 2-minute probe. The 2,500 number was confirmed with three 5-minute runs. 4,000 passed 2 of 3.
• The load generator sat in the same datacenter, so network latency is close to zero. Real users add their own round trip on top.
• The user model is one I chose. A write-heavy app, or one with heavier pages, will hit the wall sooner.
• The box was on the public internet the whole time. It also served a handful of scanners probing for /.git/config.

The takeaway

You don't always need a distributed system. Measure first. In this case the bottleneck was one thing, and the fix was a one-second cache and a few lines of Nginx config. Total cost of the whole experiment: $12 for the month and about $0.60 for the load generator.