Massed Compute

The wall every RAG project hits

Building a system that answers questions over your own documents follows a familiar path.

You get a small demo working in an afternoon. A hundred documents, a language model, some retrieval glue. It feels great.

Then you point it at the real collection. Fifty thousand documents. Or five million. The part that took thirty seconds on the demo now takes all night.

That slow part is the vector index build. Almost nobody plans for it, and it decides whether your project ships this month or next quarter.

Moving your vector database to GPU is what removes that wall. On the NVIDIA RTX PRO 4500 Blackwell it costs $0.76 an hour, and the full guide to the RTX PRO 4500 covers the card in detail.

What a vector index is and why it takes so long

Retrieval works by turning text into numbers. Each chunk of a document becomes a long list of numbers called an embedding, and chunks with similar meaning end up with similar numbers.

To answer a question, you turn the question into numbers the same way and find the chunks whose numbers sit closest.

Checking every chunk one at a time works fine for a hundred documents and falls apart at a million. So the numbers get organized into an index first, a structure that lets you find close matches without checking everything.

Building that index is heavy math over your entire collection. That is the all night job.

Two things make it slow on regular servers. Generating the embeddings takes a model run per chunk. Then organizing millions of embeddings into an index takes many rounds of distance calculations.

Both of those are exactly what a GPU is built for.

What GPU accelerated vector search delivers

NVIDIA compared a server with eight RTX PRO 4500 cards against an AMD 9654 server with 192 vCPUs. The job was indexing 33 million vectors in Milvus. The CPU used HNSW, a common index type. The GPU used cuVS with an algorithm called CAGRA.

The GPU server delivered up to 50 times the performance.

Read that number carefully, because it does not divide down to one card. A GPU index build needs the whole index in GPU memory, and Milvus sizes GPU CAGRA at roughly 1.8 times the raw vector data. At 33 million vectors that is well past 100 GB, which is why the test used eight cards. A single 32 GB card cannot run that job at all.

So the honest framing is this. The 50x figure tells you what GPU vector search does at scale. A single RTX PRO 4500 gives you the same kind of speedup on a collection that fits its memory. For most teams that means low millions of vectors rather than tens of millions.

That is still the change that matters. An overnight job becomes a job you wait through with a coffee, and you stop being afraid to rebuild.

Why rebuilding matters more than it sounds

When an index takes all night, you build it once and avoid touching it. That quietly limits your project in ways that add up.

You stop testing different chunk sizes, even though chunk size strongly affects answer quality. You stop trying other embedding models. You let the index drift out of date because refreshing it is a whole evening.

When a rebuild takes minutes, all of that opens back up. You test three chunking strategies before lunch. You refresh nightly without thinking about it. You try a better embedding model when one comes out.

The speed is worth having for the hours it saves. What it really buys you is the freedom to keep improving, which is where answer quality actually comes from.

What cuVS does for your vector database

cuVS is NVIDIA’s library for vector search on GPU. It plugs into tools you may already use. Milvus has shipped GPU CAGRA through cuVS since version 2.4, and FAISS added a cuVS backend in version 1.10 through a separate faiss-gpu-cuvs build.

CAGRA is the index type it uses. It builds a graph that connects similar vectors, and searching that graph finds close matches quickly. Published figures put CAGRA build times up to 12 times faster than CPU HNSW, with search latency up to 4.7 times lower.

So your existing stack mostly carries over. Two practical notes before you start.

First, Milvus needs a GPU enabled container image, which is a deployment change rather than a config flag. If you ask for a GPU index on a standard image, Milvus quietly falls back to CPU HNSW with no error. That is a bad failure mode, because everything works and you never see the speedup you came for. Check your image first.

Second, the FAISS cuVS build runs on Linux x86-64 only, and GPU CAGRA caps query results at 1,024 per search.

Three places the GPU helps a RAG pipeline

Generating embeddings. Every chunk needs a model run. This work batches well, so a GPU chews through it. On a first build over a large collection, this is usually the biggest share of the wall clock time.

Building the index. The distance math that organizes vectors into a searchable structure. This is where the published GPU speedups are largest, and it is the step that scales worst on CPU as your collection grows.

Answering queries. Once built, searching is already fast. GPU helps most when you have very high query volume or very large collections.

Worth being precise about which of the first two dominates, since it depends on your situation. Embedding generation usually takes the most time on a first pass through a big collection. Index building is the step that punishes you every time you rebuild, and it is the one that gets disproportionately worse as you scale. A GPU speeds up both, which is why moving the whole pipeline matters more than optimizing either step alone.

Query speed is rarely the thing standing between you and a working system.

Fitting a whole RAG pipeline in 32 GB

A RAG pipeline has a few pieces in memory at once, and 32 GB handles them together.

The workhorse embedding models are small. Something like bge-large at 335 million parameters is under a gigabyte at 16-bit. That leaves plenty of room for the batches being processed.

Worth knowing that the top of the embedding leaderboards has moved up in size. Several current leaders are 4 billion and 8 billion parameter models, and an 8 billion parameter embedding model runs about 16 GB at 16-bit. Those will fit on this card, though not alongside much else.

If you want the language model that writes the final answer on the same card, that fits comfortably with a smaller embedding model. A 7 or 8 billion parameter model at 4-bit is around 5 GB. So a single RTX PRO 4500 can hold a full pipeline while you build and test it.

Collection size is the real constraint, and it is a hard one rather than a soft one. A GPU index build needs the index in GPU memory, and Milvus sizes GPU CAGRA at roughly 1.8 times your raw vector data. Work out that number for your collection before you pick a configuration, because if it does not fit, the build fails rather than running slowly.

What a vector index build actually costs

Here is the part that surprises people.

Index building is bursty. You need real horsepower for a few hours, then almost nothing until the next refresh. That pattern fits hourly rental well, and it fits owning hardware poorly.

Say a build takes four hours on a single card. That is about $3.

Now say you move that same build to four cards. You are paying $3.04 an hour instead of $0.76, so the interesting question is whether it finishes four times sooner. Embedding generation splits across cards well, so a good part of it does. You end up paying roughly the same few dollars and getting your afternoon back.

That is the shape of the tradeoff worth understanding. More cards on the same job mostly buys you time rather than costing you money. More cards are also what let you build a bigger index at all, since capacity is a hard limit.

Either way the numbers land in single or double digit dollars. The reason index builds feel expensive is the waiting, not the money.

Since on demand has no contract, you can scale up for the build and scale back down when it finishes.

How to move your vector database to GPU

1. Launch an RTX PRO 4500 instance
2. Install your vector database with GPU support turned on, or install cuVS directly
3. Run a slice of your collection first, maybe a few thousand documents
4. Check that retrieval returns sensible chunks for questions you know the answers to
5. Once it looks right, scale up and build the full index

Testing retrieval quality on a small slice first is worth the time. A fast index that returns the wrong chunks is not progress. A few thousand documents will tell you that for a few cents, instead of after a full build at five million.

You can also skip the manual setup. The Massed Compute MCP server connects an AI assistant like Claude, Cursor, or ChatGPT straight to your account. Ask it for a machine sized to your index and it checks live inventory, launches the instance, and hands back your SSH details. That makes it easy to scale up for a build and shut everything down the moment it finishes. See what the MCP server can do and how to set it up

Common questions

Which vector databases support GPU?

Milvus has supported GPU CAGRA through cuVS since version 2.4, and FAISS added a cuVS backend in version 1.10. Milvus does need a GPU enabled container image, and on a standard image it falls back to CPU without telling you.

How much faster is GPU vector search?

NVIDIA measured up to 50 times the performance of a 192 vCPU server, using eight RTX PRO 4500 cards on 33 million vectors in Milvus. That result needs eight cards, since a 33 million vector index will not fit in 32 GB. Published CAGRA figures put build times up to 12 times faster than CPU HNSW, which is a better guide to what one card does on a collection that fits.

How large a collection fits on one card?

Work backwards from memory. Milvus sizes a GPU CAGRA index at roughly 1.8 times your raw vector data. On a 32 GB card that lands in the low millions of vectors, depending on your embedding dimensions. Above that, add cards.

Do I need a GPU to run queries, or just to build the index?

Building the index is where the biggest gain sits. Queries are already fast once the index exists. GPU helps at query time mainly with very high volume or very large collections.

Can one GPU hold my embedding model and my language model together?

On 32 GB, usually yes. An embedding model is small, and a 7 or 8 billion parameter language model at 4-bit leaves comfortable room alongside it.

Where this fits with everything else

Vector search is one of three workloads this card handles well. The other two are serving language models and understanding video.

Most teams building RAG need the language model side too, so it is worth reading how the RTX PRO 4500 handles serving language models.

For the full picture on memory, bandwidth, and pricing, read our guide to the NVIDIA RTX PRO 4500 Blackwell. It also covers how this card compares to the L40S, A100, and H100.