naveenopslearn
Search
Course outline
  1. 01How a computer works
    1. 1.1What is a computer?(completed)
    2. 1.2The CPU: the brain(completed)
    3. 1.3Memory: RAM vs storage(completed)
    4. 1.4How the CPU and memory work together(completed)
    5. 1.5What happens when you turn on a computer(completed)
  2. 02How the Internet works
    1. 2.1What is the Internet?(completed)
    2. 2.2Addresses and names: IP and DNS(completed)
    3. 2.3Packets: how information travels(completed)
    4. 2.4Client and server: opening a website(completed)
    5. 2.5Inside a busy server(completed)
    6. 2.6Why this matters for AI(completed)
  3. 03Introducing the GPU
    1. 3.1What is a GPU?(completed)
    2. 3.2CPU vs GPU(completed)
    3. 3.3The GPU's own memory: VRAM(completed)
    4. 3.4How the CPU and GPU work together(completed)
    5. 3.5Why GPUs suit AI(completed)
  4. 04Tensors and models
    1. 4.1What is a tensor?soon
    2. 4.2What is an AI model?soon
    3. 4.3Training vs inferencesoon
  5. 05Splitting work across GPUs
    1. 5.1Why split work across GPUs?soon
    2. 5.2Data parallelismsoon
    3. 5.3Tensor parallelismsoon
    4. 5.4Pipeline parallelismsoon
    5. 5.5Expert parallelismsoon
    6. 5.6Which to use whensoon
  6. 06Serving LLMs with vLLM
    1. 6.1What is LLM serving?soon
    2. 6.2Meet vLLMsoon
    3. 6.3Serving a model with vLLMsoon

Lesson 2.5

Inside a busy server

What a server's CPU and memory do when many requests arrive at once, and what happens when it gets too busy.

beginner 4 min readChecked

In one line

A busy server shares its CPU cores between many requests, gives each one a little RAM, makes the rest wait in a queue, and turns some away when it is full.

Real-life analogy

It is Friday night and our restaurant is packed.

  • The restaurant has 4 chefs (CPU cores). Each chef can only cook one dish at any instant, but they switch between dishes quickly.
  • Every order needs space on the counter (RAM) for its ingredients.
  • New orders that cannot be started yet go on the order rail to wait (the queue).
  • When the counter is full and the rail is overflowing, the host at the door says: “Sorry, we are full, please come back later” (a 503).

Where the comparison stops working: a chef needs minutes per dish. A CPU core can handle the steps of a simple web request in a tiny fraction of a second, so a single server can often handle a very large number of requests per second. Also, a server rarely spends all its time “cooking”: much of a request is waiting, for example for a database or for packets to arrive, and the core works on other requests meanwhile.

Diagram

ARRIVINGreqreqreqreqWAITING LINE (QUEUE)waiting...waiting...waiting...THE SERVERCPU cores: one request each at a timecorecorecorecoreRAM: a little space for each requestreq 1req 2req 3req 4Full RAM or a long line: new requestswait longer, or are turned away.503: busy,try again later
Requests queue up, cores work on them, and each one uses some RAM. When the server is full, requests wait or get a 503.

Step by step

In the last lesson, one browser sent one request to one server. But a popular website gets requests from many people at the same time. What happens inside the server’s CPU and memory?

Remember: a server is a computer

A server has the same parts you met in Module 1: a CPU with several cores, RAM, and storage. It runs an operating system and a web server program that listens for requests.

Step 1: requests arrive

Requests arrive as packets from all over the world. The operating system collects them and hands them to the web server program.

Step 2: the work is shared across cores

The web server program usually runs as several workers. For example, the popular web server nginx runs one main process plus several worker processes, and it can be set to start one worker per CPU core.

Each core can only run one thing at a given instant. But the operating system switches between tasks very quickly, so many requests make progress at what feels like the same time.

Step 3: each request uses some RAM

While a request is being handled, the server needs working space for it: who asked, what they asked for, and the answer being built. That space lives in RAM.

More requests at once means more RAM in use. When a request is finished, its space is freed for the next one.

Step 4: the waiting line (queue)

What if requests arrive faster than the cores can handle them? They wait in a queue, a line of requests that have arrived but not started yet.

A short queue is fine: people just wait a tiny bit longer. A long queue means pages feel slow.

Step 5: when it gets too busy

A server cannot grow its queue and RAM use for ever. When it hits its limits, it has a few choices:

  • slow down: every request waits longer,
  • turn some requests away with a 503 Service Unavailable response (“too busy, try again later”),
  • or, in the worst case, run out of memory and crash, which is why turning requests away early is often the safer option.

Some services also limit how many requests one person may send in a short time. They answer 429 Too Many Requests to anyone who goes over.

How big websites cope

Big websites run many servers side by side and spread the requests between them. That way no single server gets overwhelmed. We will meet this idea again when we spread AI work across many GPUs.

Key takeaways

  • A server is a computer with cores, RAM and storage, running a web server program.
  • The cores share the work, switching quickly between requests.
  • Each request needs a little RAM while it is being handled.
  • Extra requests wait in a queue. When the server is full, requests are slowed down or turned away with 503.

Recap: a packed restaurant on a Friday night: chefs juggle dishes, every order needs counter space, new orders wait on the rail, and when it is truly full, the host asks people to come back later.

Now you understand what happens inside a server when lots of people arrive at once. Next, the big link to AI: what happens when your request needs a GPU.

Quick check

Pick an answer, then press Check answer. Nothing is recorded.

  1. A server has 4 cores and 100 requests arrive at the same moment. What happens?
  2. Why does each request use some RAM?
  3. What does the 503 status code tell you?

Sources

This lesson was checked against these pages.

  1. Beginner's Guide (master and worker processes) · nginx
  2. About Processes and Threads · Microsoft Learn
  3. 503 Service Unavailable · MDN Web Docs
  4. 429 Too Many Requests · MDN Web Docs