Have you ever downloaded the model everyone is praising, run it on your own machine, and found that it answers everything exactly as before? Nothing breaks. No errors. The answers just feel flatter, in a way you cannot point at.
There is an explanation for that feeling, and the explanation is a number the model makers publish themselves.
Following part one, on Physical AI, the next session at AI Update Bangkok 2026 belonged to Dr. Jim (Jimmy Tejasen), who hosts the event, and it was the longest one of the day. Almost all of it is machines and numbers. Heard in passing it sounds like a graphics card review. The structure underneath is not a review. It is an order of questions you can actually decide with.
Productize keeps his order rather than reorganising it, because the order is the most valuable part: start at the machine, separate the quantities that must not be mixed, then reach the cost of compression, and finish with how to read a number somebody hands you.
Part 1Four classes of machine, and the seller says the lines are blurry
He starts by sorting machines into four classes: gaming graphics cards, workstation cards with a lot of memory, appliance-style boxes you carry home and plug in, and rack cards for a datacentre.
Then he warns, himself, that these lines are not sharp. The same silicon climbs across classes, and the marketing between a gaming card and a workstation card is not straightforward.
The point he makes directly: a gaming card and a workstation card differ in memory, and the chip is the same chip. The word Pro makes people expect far faster silicon, when what the extra money buys is more memory on the same die.
Two rules fall out of that. First, compare machines only inside the same class. Second, price does not tell you how fast a machine is. It tells you what kind of work the machine was built for.
He sells these machines himself, and still said on stage that some kinds of work should not use them at all.
Part 2Three quantities you must not mix
This is the spine of the session, because people who buy the wrong machine usually do it by collapsing three separate things into one word, fast.
His image is a water tank. Capacity is the size of the tank, read speed is the size of the tap. A big tank says how large a model you can pour in. A wide tap says how fast the water comes out. A big tank with a small tap gives you water a drop at a time, and that is exactly the machine that loads a larger model and then types too slowly to sit through.
| Quantity | What it decides |
|---|---|
| VRAM, capacity | How large a model can load at all |
| bandwidth, memory read speed | How many tokens per second come back |
| compute, processing power | How long before the first character appears |
The formula he worked live on stage: tokens per second equals bandwidth divided by the size the model occupies in memory. The reason is mechanical. Every time the model produces one more token it has to read all of itself again, plus the whole conversation so far. A larger model is therefore slower by nature, not because the machine is bad.
The numbers he put up: a personal machine with 128 GB of capacity reading memory at 273 GB/s gets about 7 tokens per second, against a datacentre machine with 288 GB reading at about 8 TB/s, which gets about 177 tokens per second. Roughly 25 times faster, on read speed alone. The model size used for either calculation was never stated.
The figures here are the ones given at the event. Use them for ratios, not as product specifications.
One more stop on the word compute, because several kinds of chip do this job and they do not do the same job.
| Chip | Character | Suited to |
|---|---|---|
| CPU | Few cores, roughly 4 to 64, each strong; good at work in sequence | General work, and directing the order of jobs |
| GPU | Thousands of cores working at once; good at multiplying matrices | Training and running AI, and graphics |
| TPU | Google's special-purpose chip; multiplies matrices and nothing else | AI work on Google Cloud |
| NPU | Small chip that only does the answering stage, not training; low power draw | AI running on your own device |
The warning attached: the units for GPUs and NPUs are not directly comparable. One side counts floating point operations, the other counts integer operations. A bigger number does not mean a faster chip.
The other thing printed on the box that people skip is the type of memory, not just the capacity figure, because the type is what sets read speed.
| Type | Character | Found in |
|---|---|---|
| HBM | Stacked in layers with a very wide path; the fastest to read | Datacentres |
| GDDR6 and GDDR7 | Medium speed, cheaper than HBM | Ordinary graphics cards |
| unified memory | One pool shared by CPU and GPU, with nothing to copy across | Appliance machines and Apple's M-series |
| LPDDR5X | Low power draw, medium read speed | Combined chips in small machines |
| discrete VRAM | Memory soldered to the graphics card, separate from system RAM | Replaceable graphics cards |
That table also explains what comes next: two machines with the same capacity printed on the box can answer at different speeds, if the memory is a different grade.
The half people miss is that a slow first character comes from a long prompt, not from a large model. The mechanism a model reads with compares every token against every other token, so the cost grows with the square of prompt length. A 5,000 token prompt means roughly 5,000 times 5,000 units of work. And prompts today are not one sentence. They are an entire document pasted in.
Capacity has one more trap folded inside it. VRAM has to leave room for the conversation as well. A 30 billion parameter model on a 32 GB card does not fail because of the model. It fails because only about 2 GB is left over.
Part 3Two arms races, in different arenas
Once the three quantities are separated, something else becomes visible that explains why each class of machine runs in its own direction. Datacentres and personal machines are not competing with each other. They are competing at different things.
The datacentre side keeps pushing memory read speed up, and keeps shrinking number precision inside the chip itself, from 16 bits to 8 to 4. The reason is not elegance. Smaller numbers mean one machine serves more users.
The personal machine side chases something else: total memory capacity shared between chip and system. The goal is to get a large model into one box. It is not competing on how many people it can serve at once.
The use of this frame is in reading launch announcements. A leap in read speed means they are playing the datacentre game. Growth in capacity means they are playing the personal machine game. Then you decide which game is yours.
Part 4The 1% nobody locates
Before the price, a word on what compression compresses. Precision is the number of bits used to store one value. Fewer bits means coarser, but faster. From finest to fastest: 32 bits, 16 bits, 8 bits, then 4 bits.
The rough exchange rate is that halving the bits roughly doubles the speed. His example: one datacentre chip at 4 bits does roughly four times the work of the same chip at 16 bits. On the capacity side, compressing model weights to 8 or 4 bits cuts VRAM use by 2 to 4 times.
And there is something that has to be said in full. The coarseness of 4 bits can be partly compensated. Newer 4-bit number formats use fine-grained scaling to claw back some of the lost precision. Productize has written about that format already, in vLLM on a rented GPU, where vLLM is the software used to run the model. The point is that it recovers some of the loss, not that nothing is lost.
Which brings us to the heart of the session. The damage figure is not a secret. The vendors write it down: about 0.1% lost at 8 bits, about 1% at 4 bits.
Most people read that and move on, because 1% sounds small. His argument is that it is not small.
One evening of letting an AI help write code or work through documents passes a million tokens comfortably. One percent of that is ten thousand degraded tokens, with nothing marking which ones.
What it looks like from outside is wrong answers, and invented details you do not catch. Not an error. Not a crash. Compressed all the way to 2 bits, the damage is certainly past 1%. And the sentence it seems fine to me, it answers everything is the real trap, because a language model answers everything anyway. That is what it was built to do.
So his conclusion is not a preference. It is a condition. Compress, if you already have an eval set. Without one, compressing means trading quality for speed without knowing what you paid.
How to read numbers other people hand you is in part two, on reading benchmarks. How to pick a model from the work itself is in choosing a local LLM by the job, which goes into the mechanism of compression in more detail than this piece does.
Part 5Open weight does not mean you can run it
The last stretch is where open models came from, which was not idealism. In 2023 Llama, Meta's model, shipped under a licence you had to sign with a long list of restrictions, and the weight file was posted on a public forum. Everyone downloaded it. The entire open model lineage grew out of something that was never meant to be open.
What decides which model people actually build on is not the score. It is the licence terms. Most Chinese models ship under licences with no legal ambiguity, which makes them the default starting point people reach for.
But downloadable does not mean your machine can hold it. A frontier open-weight model of 280 billion parameters, run with no compression at all, needs two appliance machines, or four Mac Minis.
What makes a model that size runnable at all is an architecture that splits it into many parts and wakes only the parts relevant to the question. 280 billion in total, roughly 10 billion actually awake. A second mechanism reduces the every-token-against-every-token reading to certain layers only, summarising the rest, which is how very long prompts become possible without memory growing in step.
It closes with a buying trap. Second-hand cards retired from datacentres are a trap. They are cheap and they genuinely have a lot of memory, but that silicon was designed when datacentres still ran high-precision numbers. Give it a compressed model and it turns out slow. Plenty of memory, on its own, is not a reason to buy.
Part 6What to take away
- Before buying a machine, ask whether your work is bound by capacity, by memory read speed, or by compute, then compare only within one class.
- Before compressing a model, have an eval set of your own. Not compress first and see how it feels afterwards.
- Before believing a score table, ask who measured, which version, which mode, and whether the conditions matched.
- Before deciding the machine is slow, check whether it is slow to the first character or slow while producing tokens. Those two symptoms have different causes.
- Before buying a second-hand card with a lot of memory, check that it is fast at the low precision you will actually use.
What comes home from this session is not which machine to buy. It is this: stop reading a spec sheet as the word fast, and read it as the question of where your work is bound. Change the question and most of the comparison tables that used to be confusing read themselves.
Source
Everything here comes from AI Update Bangkok 2026, held on 21 August 2026 at The Cloud Bangkok by Jimmy Tejasen. This part covers the session he presented himself.
- Event livestream: watch the recording
- Jimmy Tejasen's work: aiserver.in.th, news and reviews of AI server hardware, and agentic-press.com, a publishing house for AI books
The datacentre machine's read speed was given two ways during the day, 7.1 and about 8 TB/s; we take the second because it appeared more often and the ratio barely moves either way. Every figure here comes from what was said on stage, not from vendor spec sheets. Use them for ratios and for asking better questions, not as product specifications. Productize has not reproduced anything shown at the event.