What is an LLM benchmark?

An LLM benchmark is a standard test set used to score language models against each other, covering things like solving programming problems, using tools, or reasoning. The problem is not the test set. It is what people do with the score, because the same number changes meaning depending on which version it was compared against and which mode was switched on during the run.

How do you read an LLM benchmark without being fooled?

Ask four questions before believing any number. Who measured it, the model vendor or a neutral party. Which version, the same model or an older one. Which mode, was extended thinking on and were tools allowed. And were the conditions identical, the same harness and the same scoring. Then ask one more: did it get better at the thing you actually do.

Why can you not compare scores from two different leaderboards?

Because they measure different dimensions. One ranks general intelligence, the other ranks work the model has to carry out itself plus safety. The leader on the first scores 63 and the leader on the second scores 83. Those two numbers are not measuring the same thing, so putting them side by side proves nothing. The same applies to test sets with similar names but different versions.

Which numbers can you actually trust?

The ones you measure yourself against your own work. The minimum version is to collect 20 to 30 real questions with answers you know are correct, then run that set before and after every model change or every time you shrink a model. Standard benchmarks measure what the world cares about, which may not be the field you are competing in.

How to Read an LLM Benchmark Without Being Fooled

Have you ever read a model launch and noticed that everybody wins? Every table has one row in bold and it is always their own. Every chart has one line on top and it is always theirs. Then you put the new one to work and it is barely different from what you were already using.

Following part one, on Physical AI, AI Update Bangkok 2026 had two more sessions run by Dr. Jim (Jimmy Tejasen), who hosts the event, over two hours between them, going from hardware to models to training to which layer of the AI stack collects the money.

Listening to it, the sessions sound unrelated. Laying the notes back out afterwards, every session is teaching the same thing: the number that gets announced is not the number you get to use.

Productize writes this part as a map of both sessions, so the shape is clear before the later parts go into each piece.

Terms, defined once, here

LLM a large language model, the thing that reads text and writes text back; the engine underneath the chatbots and assistants people use
benchmark a standard test set fired at models so their scores can be compared
eval a set of questions with known-correct answers that you build yourself, to measure whether a model is better or worse at your work
open weight a model whose weight file you can download and run; not the same as open source code or open training data
quantize shrinking the numbers inside a model so a large model fits the machine you already own
token the chunk of text a model reads and emits one at a time, roughly a word or half a word
harness the program that wraps a model, holds the tools, holds the permissions, and runs the loop

Part 1The same trap, repeating at every layer

If you take one line home from both sessions, take this one. Every layer of AI has two numbers, one for announcing and one for using, and they are never the same number.

Layer	The announced number	The usable number
Machine	The capacity printed on the box	What is left after reserving room for the conversation
Model	The weights are downloadable	Whether your machine can hold it, which depends on total parameters
File size	Parameter count at 16 bits	The file people download has already been compressed, a different size entirely
Speed	Tokens per second in the press release	Meaningless unless it names the machine and the software stack
Test scores	Beats the competition	Beats which version of them, and in the arena you actually work in
Training	Train it and it gets better	How much better, once you subtract how good the starting point already was

Once that shape is visible, everything that looked scattered across the day turns into one subject, and the question narrows to one: which of the two numbers is this.

Part 2A map of both sessions

The material splits into four blocks, ordered from the tangible to the structural.

Block 1, the machine. Machines that run AI get sorted into four classes by the question of what you buy and where you put it, not by raw power. Then it separates capacity from memory read speed. Capacity decides how large a model can load at all. Read speed decides how many tokens per second come back.

Block 2, models and compression. Downloadable does not mean your machine can hold it, and the file people actually download has always been compressed. Compression has a price, and it is a price you cannot see, because the model still answers every question exactly as before.

Block 3, training. Draws a hard line between what creates intelligence and what merely adjusts behaviour, then says plainly that most of the time training your own is not worth it, because the starting point is already good.

Block 4, where the money sits. The models themselves are being turned into a commodity that competes on price, while the layer that actually collects money moves to the pipe and to the thing that wraps the model.

Notice that none of the four blocks answers which one should I buy. All four answer how do I know what the number in front of me means. That is the part worth taking home, because the models change every quarter and the reading method does not.

Part 3Reading a benchmark with four questions

This is the block with the most detail in it, because it is where people get fooled most often. It starts with something that stops you short.

The industry uses two rulers, and they measure different dimensions. One ranks general intelligence. The other ranks work the model carries out itself, plus safety. The leader on the first scores 63. The leader on the second scores 83. Those two numbers cannot be set side by side, because they are not measuring the same thing.

Then come the three traps that show up most often.

Mixed versions. The same test set at different versions. Scores from those are not comparable.
Vendor-measured is not neutral-measured. Ask who ran the test before reading the result.
The mode changes the score. The same model with extended thinking on and off produces different numbers, and a variant with Max in its name is not the standard release.

From there it becomes a four-item checklist for any number at all.

Question	What to look for
Who measured	The model vendor, or a neutral party
Which version	Genuinely the same model, or an older release
Which mode	Was extended thinking on, were tools allowed
Same conditions	Same harness, same prompts, same scoring

The worked example is Qwen 3.8-27B, an open-weight model having a moment, announcing wins over a closed competitor across three arenas. On SWE-bench Pro, for writing code, it scores 61.7 against 53.4. On OSWorld-Verified, for driving a computer itself, 84.3 against 72.7. On AndroidWorld, for driving a phone, 81.9 against 62.0.

It did win all three. But he attached a note to it himself, saying the model it was compared against is Claude Opus 4.6 Max, a second-tier release, not the Opus 5 on sale today. Beating last year's model and beating the one on sale today are different claims. Another slide makes the same point: the headlines reporting that Qwen beat Opus usually leave out which Opus.

The last question is which direction it improved in, and this is the sharpest point in the session. Against its own previous release, this model moved a long way on work it carries out itself and on reading images, up 20.4 to 56.7 points depending on the arena. Its knowledge and reasoning barely moved. That side is measured across two arenas, GPQA and HLE, and the slide says it still loses to Opus 4.6 Max on both. GPQA reads 89.2, up 1.4 from the previous release's 87.8. HLE reads 30.8, which the slide files under still behind. The table that went out shows only the three arenas it won.

His summary was that this model did not get smarter. It got able to work on its own, and able to see.

And there he had it both ways: this model beats the competitor's second-tier release on autonomous work and on vision, and loses on reasoning. The table that went out to the world showed only the winning half.

Upgrading without testing the dimension you actually rely on can return exactly nothing. If your work is reasoning and knowledge, this set of numbers says nothing about you at all.

There is a side note he scoped himself, applying it to announcements from Chinese labs: when a speed figure is published, ask what it ran on, because most of the time nobody says, and it may be spread across several machines.

Part 4So which number can you trust

His answer is short, and it repeated often enough across the day to become the spine of it. The number you can trust is the one you measured yourself, on your own work.

He did not say it as an ideal. He said it as a precondition. Talking about compressing a model, he said you can only do it if you already have an eval set. Talking about training, he said the same thing again. Two different points in the day, one condition.

The reasoning is plain, and it is a line Productize holds on its own work too. Standard benchmarks measure what the world cares about. Your work is your work. A model that scores high in an arena you never compete in has told you nothing.

Part 5Which part to read next

This part is the map. The parts after it go into each block.

Source

Everything here comes from AI Update Bangkok 2026, held on 21 August 2026 at The Cloud Bangkok by Jimmy Tejasen. This part covers the two sessions he presented himself.

Event livestream: watch the recording
Jimmy Tejasen's work: aiserver.in.th, news and reviews of AI server hardware, and agentic-press.com, a publishing house for AI books

The figures here come from what was said on stage, not from vendor spec sheets. Use them for ratios and for asking better questions, not as product specifications. Productize has not reproduced anything shown at the event.

AI Update Bangkok 2026, Part 2