Massed Compute

Most production AI runs on a smaller model than you would guess

The headlines go to the largest models. The work goes somewhere else.

That gap should change how you shop. The best GPU for LLM work is rarely the biggest one on the list.

Look at what teams actually ship and you find models in the 8 billion to 32 billion parameter range doing the bulk of it. Chat features. Coding help. Summarizing documents. Classifying support tickets. Pulling structured data out of messy text. Powering the reasoning loop inside an agent.

A small language model in that range is not a compromise for those jobs. It is the right size. It answers fast, it costs little to run, and the quality gap on focused tasks closed a long time ago.

So the card you want is the smallest one your model fits on with room to work.

For the 8B to 32B tier, that is the NVIDIA RTX PRO 4500 Blackwell at $0.76 an hour. The full guide to the RTX PRO 4500 covers the card itself. This article is about what it does with language models.

The best GPU for LLM work is the one your model fits on

Two things go in GPU memory. The model weights, and the context, meaning the conversation or documents the model is working through.

Weights are easy to estimate. Take the parameter count and multiply by the bytes per parameter. A 16-bit model uses 2 bytes per parameter, an 8-bit model uses 1, and a 4-bit model uses about half.

So an 8 billion parameter model at 16-bit is about 16 GB of weights. The same model at 4-bit is about 4 GB.

Here is how the common combinations land on a 32 GB card.

Those figures are weights only. Real usage runs higher once context and overhead are counted, so treat them as a floor rather than a total. Independent testing measured Phi-4 14B at 4-bit using about 8 GB in practice against the 7 GB the weights alone suggest.

Two readings worth taking from that table.

First, 4-bit is what makes the 32B tier possible on this card. At 4-bit a 32 billion parameter model uses about half the card and leaves the rest for context.

Second, the ceiling is real. A 70 billion parameter model at 4-bit is about 35 GB of weights, which is more than the card holds. The only way onto a single card is 3-bit, where the weights drop to about 26 GB and quality loss becomes measurable rather than negligible.

Benchmark numbers on a single card

Two sets of numbers are worth looking at, and they measure different things.

Independent single-user results. Exxact, a server maker that builds systems around this card, ran several models through Ollama at Q4_K_M, which is a 4-bit quantization. These are interactive speeds, meaning what one person sees while text streams back.

Exxact puts the usable baseline at about 30 tokens per second, with 50 and above as the comfortable zone. Nemotron Nano and Phi-4 clear that easily. Gemma 4 31B sits just over the line, which is fine for reading but leaves less headroom.

The spread across those three is worth understanding, and it is not about size. Nemotron Nano is a mixture of experts model. It holds a much larger parameter space, around 253 billion, and activates about 30 billion of them for any given token. Only the active part does work, so it runs faster than Phi-4, a dense 14 billion parameter model where every weight participates every time.

So architecture matters as much as parameter count. A number on a model card does not tell you how fast it will feel.

Our own results. We ran Meta’s Muse Glimmer 30B on this card and published the full method and raw data. Through llama.cpp on Meta’s own 4-bit build, one stream at a time, it decoded at 35.2 tokens per second.

That sits in the same range as the Gemma 4 31B result above, which is reassuring for a model of similar size at similar precision. The two are not a strict comparison, since one ran through Ollama and the other through llama.cpp. They do agree on what this card does with a 30 billion parameter model for one user.

One thing to hold onto before the next section. Every number above is one person streaming, which lands in the tens of tokens per second. Total output across many users at once lands in the hundreds or thousands. Same card, different question. If a benchmark does not say which it measured, it is not telling you much.

Ollama for one person, vLLM for many

The software you serve with matters as much as the card.

Ollama is built for getting a model running quickly. It is the right choice while you are trying things out, or when one person is using the model at a time. It handles concurrent requests, but that is not what it is tuned for.

vLLM is built for serving. It batches incoming requests together so the GPU works on many at once, which is what turns tens of tokens per second into hundreds. If you are putting a model behind an API for a real application, this is the tier you want.

The practical path is to start on Ollama to confirm the model does what you need, then move to vLLM before you put it in front of users.

Our own batched result shows the size of that gap. Running Muse Glimmer 30B through vLLM at thirty-two concurrent requests, one card produced 945.8 output tokens per second. That is the same card and the same model size as the 35 tokens per second single-stream figure. Batching is the main reason for the difference, though the two runs also used different engines.

One caveat on that number. It used a community 4-bit conversion of the model rather than one published by Meta or NVIDIA, because Meta’s official 4-bit build comes in a format vLLM does not serve. It is labeled that way in our results.

Why time to first token decides how fast your app feels

Streaming speed gets the attention. Time to first token is what users actually notice.

Time to first token is the pause between someone hitting send and the first word appearing. As a rule of thumb, around 100 milliseconds reads as instant, and past half a second people notice they are waiting.

This is where the Blackwell architecture earns its place. Exxact ran Llama 3.1 8B on a single RTX PRO 4500 through vLLM twice, once at NVFP4 and once at BF16. NVFP4 is a 4-bit floating point format that Blackwell runs directly, with no step converting weights back up to a higher precision first.

Same card, same hourly rate, 2.4 times the work and half the wait.

One thing to plan for. NVFP4 needs a checkpoint built for it and a runtime that knows about it, which today means vLLM or TensorRT-LLM. A plain 4-bit download will not get you there.

One GPU per session beats one big shared GPU

Here is the result that changes how you size a deployment.

We ran the same model on one, two, four, and eight cards. Single-stream speed held between 35 and 36 tokens per second the whole way up. Adding cards to a model that already fits on one card does not make it faster.

What more cards do is serve more people at once.

So the pattern that works is one card per active session rather than one large card shared between everyone. Four RTX PRO 4500 cards cost $3.04 an hour and serve four sessions at full speed. One larger card at a similar price serves them by taking turns.

The eight card configuration gives you 256 GB of total memory for $6.08 an hour, which suits a team serving several different mid-size models side by side. Think of it as capacity rather than speed.

Running your own model instead of calling an API

A lot of teams reach this card because they want a private LLM deployment rather than a metered API. Usually one of three things is driving it. The per-token bill is growing faster than the business. They want to serve their own fine tuned weights. Or they would rather their prompts did not become training data for someone else’s model.

Hourly rental fits all three. There is no per-token meter, so a busy month costs the same as a quiet one at the same uptime. Your prompts go to your instance rather than a third party provider.

Be precise about what that does not give you. This is a hosted instance in our data center, not hardware in your building. If data can never physically leave your own facility, which is common under strict regulatory obligations, LocalMetal™ is the product to look at instead.

The tradeoff is work. You run the serving stack and idle time still costs money. For steady workloads that favors your own deployment. For spiky, low-volume traffic an API often still wins.

When you actually need a bigger card

Being straight about the ceiling saves everyone a wasted afternoon.

A 70 billion parameter model is past this card. At 4-bit the weights alone exceed what it holds, so you drop to 3-bit and speed lands in the single digits or low teens. Published single-card figures disagree with each other enough that we would rather run it ourselves than quote one.

Two cards do make it usable, at roughly 20 tokens per second in independent testing, and past two the gains mostly stop. So two will run a 70B model, but a card with more memory beats working around a shortage of it. Start with the RTX PRO 6000 Blackwell at 96 GB or an H100.

The same goes for serving hundreds of people from one machine, where high bandwidth memory earns its price.

If your models live between 8B and 32B, this card is the better economics. Above that, size up.

What an hour of inference costs

Cost per token is how most teams end up judging an AI inference GPU. It swings enormously based on how you serve.

All three ran on the same card at the same $0.76 an hour. Everything else changed. The model, the precision, the serving software, and how many requests were in flight at once. That spread is the point. Your hardware bill is fixed and your cost per token is not.

Treat the 4 cent figure as a best case ceiling rather than a planning number. It assumes a small model, a batching-friendly runtime, and a card kept busy. Real traffic arrives unevenly, so real applications land higher.

The middle row is the useful one for most teams. The bottom row is the warning. Serving one request at a time leaves most of the card idle, so the cost per token climbs hard. Those two rows are both a 30 billion parameter model on one card, but they ran through different engines on different 4-bit builds, so read the gap as a direction rather than a controlled test.

How to launch an instance for LLM inference

1. Sign in to the Massed Compute marketplace and open Deploy
2. Under Select GPU Type, pick the RTX PRO 4500 configuration you want
3. Choose a Category, then an Image or setup recipe that matches your serving stack
4. Add an SSH key and a startup command if you want them. Both are optional.
5. Launch

The instance comes up in a few minutes with a public IP. On demand has no contract, so a test costs you the hour it takes.

There is a faster way to do all of this. The Massed Compute MCP server connects an AI assistant like Claude, Cursor, or ChatGPT straight to your account. Describe the model you want to serve and it picks a matching configuration, launches it, and hands back your SSH details. It can also report what you are spending and shut instances down when you finish testing. See what the MCP server can do and how to set it up

Common questions

What is the best GPU for LLM inference?

The smallest card your model fits on with room for context. For models between 8 billion and 32 billion parameters, the RTX PRO 4500 covers that tier at $0.76 an hour with native FP4 support. Above 32 billion parameters, move up to the RTX PRO 6000 Blackwell or an H100.

How many tokens per second can one RTX PRO 4500 do?

For one person streaming, the tens. Independent testing puts Phi-4 14B near 76 and Gemma 4 31B near 34. For total output under batching, the hundreds. Our own 30 billion parameter test hit 945.8 output tokens per second across 32 concurrent requests.

Can it run a 70B model?

Not well on one card. At 4-bit the weights are about 35 GB, more than the card holds, so a single card means dropping to 3-bit and living with single digit speeds. Two cards reach roughly 20 tokens per second. For that tier a card with more memory is the better answer.

Do I need more than one GPU?

Not for speed. Adding cards to a model that already fits on one card leaves single-stream speed flat. Add cards to serve more people at once, or to run a model that does not fit.

Is this good enough for a private LLM deployment?

For models in the 8B to 32B range, yes. You run your own serving stack on your own instance and there is no per-token meter. For a requirement that data stay in your own building, look at LocalMetal™ instead.

Is this the best GPU for AI work in general?

No, and no single card is. It fits language models from 8 to 32 billion parameters, video understanding, and vector search. For training, for one very large model, or for hundreds of simultaneous users, a card with high bandwidth memory is the right tool.

Which serving software should I use?

Ollama while you are evaluating a model. vLLM once real users are involved, because batching is what makes cost per token reasonable.

Where this fits with everything else

Serving language models is one of three workloads this card handles well. The other two are understanding video and speeding up vector search for RAG. Teams building retrieval systems usually need both this article and that one, since a working setup means a language model and a vector index side by side.

For the full picture on memory, bandwidth, and pricing, read our guide to the NVIDIA RTX PRO 4500 Blackwell. It also covers how this card compares to the L40S, A100, and H100.