We tested NVIDIA Kumo: it shrinks the pile, but can't decide alone

Scroll your feed and you have probably seen a "What is Nvidia Kumo" post with a long list of use cases: predict churn, catch fraud, plan your supply chain. Some say it can make decisions for your team. If you have a database of customers, invoices and payments, the first question is likely "what can it really do, and could I use it on my own data?"

We wondered too, so we measured it. Productize tested Kumo Relational on a public multi-table database, set pass criteria before running, and put every number here. We also compare Kumo with a decision model (an AI that picks one answer from a fixed list of options), which we tested before, because both get pitched as "AI that decides" but work differently.

Part 1What is Kumo Relational, and what can it predict?

Kumo Relational is a foundation model (trained up front on lots of data, then pointed at new tasks right away) for data spread across linked tables. Give it your customer, invoice and payment tables, and it predicts an outcome for each customer from past cases with known outcomes.

Think of a veteran collections clerk. No formula, but they have seen plenty of customers like this one, so they can make a rough call on a new one. Kumo does that with the examples we hand it. A "case" is one thing you want predicted at one point in time: in our test, one case was one driver over the next 30 days or so; in a business, one customer this month, or one invoice.

"No training" means this: when you ask for predictions, you hand it past cases with known outcomes as examples (about 1,000 in our runs), and it predicts the new ones right away. There is nothing to train or tune.

It predicts only 2 kinds of thing: yes or no (will this customer pay late?) and a single number (how much will they buy next month?). For yes or no it returns a probability: 0.83 means an 83% chance it happens. A churn question ("will this customer stop in the next 30 days?") fits this yes-or-no shape, but we have not tested it; we did not test fraud either.

What the hype posts leave out is what it can't do. The model card lists no recommendation task. It can't predict several targets at once or read free text such as notes. Each request is capped at 30MB. And it doesn't tell you why.

Part 2How we tested it

The test ground was a Formula 1 dataset from RelBench, a public benchmark. It holds nobody's customer data and has layers of linked tables, much like a business database.

All 3 tasks ask, for each driver over roughly the next 30 days: will they fail to finish a race, will they finish in the top 3, and what position will they finish? The first 2 use AUROC: the chance the model ranks a case where the event happened above one where it didn't (0.5 is random, 1.0 is perfect). The third uses MAE: how many places off on average, lower is better.

Our main comparison was a baseline, a simple method to compare against. Ours used LightGBM, a traditional method, without pulling in the other tables. That makes it weak.

Before any real run, we locked 2 criteria in a file. First, Kumo must beat our baseline on at least 2 of the 3 tasks. Second, the "can it decide on its own?" test, which applies only to the 2 yes-or-no tasks: a call is confident when the probability is 0.7 or above, or 0.3 or below. Confident cases must be at least half of all cases, and at least 90% of them must be right.

For this run we sent the data to NVIDIA's online service.

Part 3It beats the baseline, so why can't it decide on its own?

Because ranking cases well and being trustworthy when "confident" are different things. Kumo passed criterion 1 and failed criterion 2 on both yes-or-no tasks.

Kumo beat our baseline on all 3 tasks: "did not finish", AUROC 0.688 vs 0.617; "top 3", 0.832 vs 0.590; finishing position, MAE 3.889 vs 4.381.

Our baseline is deliberately simple: it looks at one table only.

The problem is when it says it's confident. Criterion 2, set before the run, required the cases it is confident about to be at least half of all cases, and at least 90% of them to be right.

The "did not finish" task: Kumo was confident on 75% of all cases, and right on 76% of those, short of 90%. All 529 confident calls said "yes": the driver had a race they didn't finish in the next 30 days or so. For scale, 70.5% of drivers in the test did have such a race, so always answering yes would already be right that often, and Kumo's 76% is only a little better.

The "top 3" task: Kumo was confident on 85% of all cases, and right on 88% of those, also short of 90%. All 618 confident calls said "no". Only 17.6% of drivers finished top 3.

If a collections model were only as accurate as Kumo on "did not finish" and decided alone whenever confident, about 24 of every 100 of its decisions would be wrong. And since every confident call said "yes", every mistake would go the same way: chasing people who shouldn't be chased.

So can it sort a queue? After the run, we also counted how many of Kumo's top 30, 70 and 140 scored cases actually happened. We picked these cutoffs after the run; they were not criteria set in advance.

• "Did not finish" (702 cases in all): the top 30 had 29 right (97%), the top 70 89%, the top 140 86%. Our baseline's top 30 had only 19 right (63%).
• "Top 3" (726 cases in all, 17.6% made the top 3): only 4 of the top 30 made it (13%). But 46% of the top 70 did, and 48% of the top 140, about 2.7 times random.

So the list does shrink the pile, but not on every task. On "top 3", the top 30 was no better than picking at random, and it only started to help from the top 70 down. Before your team phones anyone, open a batch from the top of the list yourself and count how many really happened, the way we counted the top 30, 70 and 140.

Limits: we ran once, no repeats.

Part 4How is it different from an AI that picks from options?

A decision model is an AI that picks one answer from a fixed list of options. The "decision" in its name means picking among options, not deciding in place of a person. You give it a single item and the list, and it returns one, with a confidence for each option. Hand it a receipt and a list of expense categories, and it picks one category.

Kumo works differently. It takes linked tables with history and predicts yes or no, or a single number, for each case. So you can't hand Kumo that receipt and ask for a category.

The one we tried is Clef-flash, which anyone with a Cloudflare account can call on Workers AI. We had it categorize posts on our own blog; the details are in that post.

Which one when? Look at the data's shape. A question answered from history across several tables is a job for Kumo. A question answered from the one document in front of you, like which expense category this receipt is, is a job for a decision model.

Part 5What data do you need to try Kumo?

You need tables linked by IDs, a date on every event, and enough past cases with known outcomes. Accounting software data fits: invoices carry a customer ID and issue date, payments carry an invoice number and payment date. That is enough to try predicting which invoices will be paid late. Without dates, Kumo can't tell what came first.

The last item matters most. We could use NVIDIA's online service only because F1 data is public. Real customer data should not go there as is. To try it on real data, there are 2 ways: run Kumo's open weights on your own GPU, or get PDPA (Thailand's Personal Data Protection Act) advice and strip whatever identifies people first.

NVIDIA publishes the open weights on Hugging Face, free to download and allowed for commercial use, but they need an NVIDIA GPU, NVIDIA publishes no exact spec, and we haven't tried this way. On our 6 GB machine with no GPU, Kumo ran out of memory, so we used NVIDIA's online service instead. It has no public price.

To try it, start with one queue your team already sorts, such as who to call about overdue payments first. Build a simple baseline, such as ranking by days overdue, and write your criteria before running. If Kumo can't beat the baseline, stop there. If it does, use the results as a review list, and don't let it press the button yet. Every Kumo number here comes from F1. How it does on your data, you'll only know by trying.

Sources and references

Keep reading: All posts on Productize / Can a Jev-style decision model cut accounting work? / Where a decision model gets auto-tagging right, and where it slips / Human in the Loop Is Not All-or-Nothing: A Three-Tier Model