The GPU's own memory: VRAM

In one line

A separate GPU has its own memory, called VRAM, right next to its workers; data must be copied there from RAM first, and how much fits and how fast it flows decide how much work the GPU can do.

Real-life analogy

Diagram

Step by step

A GPU can have a huge number of workers. But workers are useless if they have nothing to work on. Where does a GPU keep its data?

Two sides, two memories

In a computer with a separate GPU (for example a graphics card, or a GPU in a server), there are two sides:

• the host: the CPU and its memory, RAM;
• the device: the GPU and its own memory, VRAM (video RAM, often just called GPU memory).

RAM and VRAM are separate memories, each attached to its own chip.

Why give the GPU its own memory?

Remember: a GPU has a huge number of workers, all hungry for numbers at the same moment. To keep them busy, it needs memory that can deliver a lot of data every second.

So GPUs come with high-bandwidth memory attached right next to them. Bandwidth means how much data can flow per second, like the width of a road.

How data gets to the GPU

When a program wants the GPU to work on something:

1. The data starts in storage, and the CPU loads it into RAM (just like in Module 1).
2. The CPU copies it from RAM into VRAM, across the connection between them (commonly PCIe; some systems use faster links such as NVLink).
3. The GPU’s workers now read and write VRAM directly, very quickly.
4. When the work is done, results can be copied back to RAM.

That trip across the connection is much slower than the GPU working inside its own memory. So a well-built program copies data over once and keeps it on the GPU as long as possible.

Why size matters

VRAM size decides what fits on the GPU.

For AI, the biggest thing to fit is usually the model itself, which is a huge collection of numbers (Module 4). On top of that, every request being answered needs some working space too, just like each request needed a little RAM in lesson 2.5.

If it does not fit, you run out of memory. The usual fixes are to split the model across several GPUs (Module 5) or to shrink the model so it needs less memory.

Why speed matters

VRAM speed (bandwidth) decides how fast the workers can be fed.

Many AI steps do only a little maths for each number they read. For those steps, the GPU spends much of its time waiting for data, not calculating. A faster memory means less waiting.

Key takeaways

• A separate GPU has its own memory, VRAM, apart from the CPU’s RAM.
• Data is copied from RAM to VRAM over a connection such as PCIe; that trip is relatively slow, so do it once.
• VRAM size decides what fits (for AI: the model plus working space).
• VRAM speed (bandwidth) decides how fast the workers can be fed.

Recap: the helpers have their own prep table. Carry the ingredients in once, make sure the table is big enough, and keep the supplies flowing fast.

Now you understand what VRAM is and why AI people talk about “GPU memory” so much. Next, we will see how the CPU and GPU work together as a team.