Can a Jev-style decision model cut accounting work? We tried it on Thai withholding tax

Every month end, the accounting team works through the payments line by line. Air conditioner repair. Copier rental. A freelancer building the website. Each line needs an answer: how many percent to withhold. The work is repetitive, there is a lot of it, and a mistake costs money.

Lately there is a new group of AI that people loosely call decision models. The one people talk about most is Jev, from a company called TypeSafe. It does not write long answers the way a chatbot does. It just decides. So we wanted to know whether it can really help with work like this, and how much we should let it decide.

We tried it on 2 sets of expense lines we made up ourselves, 51 cases in all. Not a single line of real customer data.

Part 1What a decision model is, and what it is normally used for

The AI we are used to is the LLM, the kind you type to and it answers in paragraphs. A decision model writes nothing at all.

We send it 3 things: the text to decide on (here, one expense line), the question (how many percent to withhold on this line), and every option it can answer (0%, 1%, 2%, 3%, 5%, 10% and the progressive rate, which means withholding on a sliding scale like salary tax). What comes back is a number for every option, saying how likely each one is to be right. Not one word of explanation.

Think of a multiple-choice exam where you also have to say how sure you are of each answer. It looks about like that.

The number on the top option is its confidence, and this is exactly why it is interesting, because we can draw a confidence cut-off line. In this experiment we set it at 0.7. When a case reaches 0.7, the system uses the answer with no person looking, which means letting the AI decide on its own. Cases below the line go to a person.

The work people use decision models for is choosing again and again from options already known. Think of a ticket a customer opens: which team should get it? There are only a few teams, thousands of tickets a day, and when the model is not sure, a person looks instead. Approval requests and possibly risky transactions look the same.

With millions of items, letting a decision model choose is cheaper and faster than calling an LLM to write an answer every time. That is the cost argument people make. We did not measure it ourselves.

Jev itself is still in early access, available only through its developer's service (reference 1). So we did not try Jev directly. We tried 2 Cloudflare models that take and return data the same way Jev does: Clef-flash, the small one at 9B, and Clef, the large one at 27B (B is the number of parameters in billions, and the more there are, the bigger the model). Other decision models in this group are in the references at the end.

Part 2Why we thought it could cut accounting work

This part is our opinion, not a test result.

Accounting work is full of questions that look alike. What is this payment for? Which category does it go in? Which rate applies? Who should approve it? Every question has known options, the data to read is text someone typed, and it arrives hundreds of lines at a time.

That shape is exactly what decision models are good at. If confidence really can tell which lines need a person, the lines the AI is sure of go through by themselves and the unsure ones reach a person. Instead of going through every line, the accountant looks only at the lines that really need thought.

So the one thing we most wanted to know was this.

When it is wrong, does it know?

And why withholding tax? Because it is a good test ground in 2 ways. First, the rules are written down clearly, both in Tor.Por.4/2528 (the Revenue Department order that sets withholding rates by type of payment) and in the Revenue Department's rulings, whose numbers start with Kor.Kor. (กค), short for the Ministry of Finance. So every case has a right answer to check against.

Second, a wrong answer costs real money, and it can go wrong in 2 ways that do not cost the same. Withhold too much and the supplier calls to complain, and it gets fixed. But with a tax shortfall, withholding less than you should or nothing when you must, the payer pays the missing tax out of their own pocket, plus a surcharge of 1.5% per month or part of a month (Revenue Code section 27), which is separate from penalties. So we care about tax shortfall more than about how many cases were answered right.

Part 3How we tested it

We made up 2 sets of cases. Each case is one expense line, with an answer key saying the right rate. The first set has 29 cases, for trying things out and tuning the method. The new set has 22 cases that do not repeat the first, for seeing whether the results still hold on cases it has never seen.

Before running the new set, we locked everything: the 0.7 line, the text we send the model (the prompt), and the code used to run it. That way we could not slip into editing things afterwards to make the results look nice.

We also checked the answer key against Tor.Por.4/2528 and the rulings, and found our own key was wrong on some cases. That story is in Part 7. Every number below is scored against the corrected key.

We tried 2 ways.

1. Let the AI pick the rate itself. Send the line in and have it choose from 0%, 1%, 2%, 3%, 5%, 10% and the progressive rate.
2. Split the job. The AI answers only what the payment is for, out of 17 categories such as service or contract work, property rent, or interest. Who the recipient is and how much is paid, code reads from the kind of data an accounting system holds (in this experiment we typed it in ourselves). The rate comes from a table, and exceptions go to a person (Part 6).

We came up with the split only after seeing the errors on both sets, so its results on the new set cannot count as a fresh test. This matters, and we will come back to it at the end.

Each way was tried with 2 models and prompts in 2 languages, so 4 runs per way: Clef-flash in Thai, Clef-flash in English, Clef in Thai and Clef in English. We agreed up front that Clef-flash in Thai would be the primary run, the one that decides pass or fail. The other 3 runs were for comparison.

Before testing, we set the pass mark. Among the cases the AI decided on its own, no case may withhold less than the key (including not withholding when it must), and no case may use the wrong rate type. A case that breaks either rule counts as a criteria miss, for example a copier rental withheld at 3% when it should be 5%. The first set also had to get at least 26 of 29 right.

Take a freelancer's fee. Income under section 40(2) (pay for work done for someone) is withheld at the progressive rate and reported on PND1, like salary under 40(1). Withhold a flat 3% instead, and the payment lands on PND3. Answering on the wrong side of progressive versus flat rate like this is a wrong rate type.

But a wrong rate type does not always mean a tax shortfall. We only saw that clearly when we worked the numbers out in baht.

Item 2.6 of the instructions on the PND1 form says that for income where you cannot tell how many times a year it will be paid, you compute the tax on that one payment under section 48(1), and "if the computation gives no tax payable, there is no need to withhold". A 30,000-baht fee for an MC (master of ceremonies) paid once, less the 50% expense deduction, leaves 15,000 baht. After the 60,000-baht personal allowance, nothing is left to tax. The tax is 0 baht.

Every case in our sets whose key is the progressive rate comes out at 0 baht like this (except 2 cases of employee income, which need the salary first). If the AI answers a flat rate, that is withholding too much, not too little.

So we count 2 things side by side. The criteria miss still decides pass or fail, as before. Tax shortfall is counted in baht, to see whether money really fell short. We added the second one later, after seeing the results, so it does not replace the pass mark.

Part 4Letting it pick every rate on its own does not hold up

A copier rental in the new set needed 5%, and it answered 3% at confidence 0.75. The line sits at 0.7, so the case went through and nobody saw it. So letting it pick every rate is not a good fit yet, and high confidence did not mean a right answer.

Without the rules, it just guesses

The first time, we gave only the 7 options and no rules at all. It got 3 to 8 of 29 cases right, close to random guessing. No model knew Thai withholding tax on its own.

When we added 1 rule line per option to the prompt, summarized ourselves from Tor.Por.4/2528, the results moved. With the Thai prompt, Clef got 21 of 29 right and Clef-flash 19. Clef-flash with the English prompt went up to 25, but that is still short of 26. None passed.

At the start of the experiment we also tried 2 more small models that run on our own machine (references 3 and 4), and dropped them. One gave a confidence of 0.999 to a case we slipped in and deliberately wrote to be ambiguous. The other, given Thai text, gave the same single answer to every case and got only 3 of 29 right.

The cases it decided alone looked good on the first set, then the new set failed

No run reached 26 of 29, but if you look only at the cases it decided on its own, the first set looked good, better than it really was. We picked its 0.7 line after seeing the results (we tried several values at the time: 0.5, 0.7, 0.9), so the numbers that came out were the best case on the same data used to pick the line.

On the first set, Clef-flash in Thai decided 15 of 29 cases on its own and got 13 right. The other 2 both withheld too much, so neither left tax short. The first was a 25,000-baht fee to a freelance web developer. It answered 3% at confidence 0.78, but that pay is withheld at the progressive rate, so this is a wrong rate type and counts as a criteria miss. In baht, it withheld 750 baht too much. The second was a 48,000-baht staff seminar at a hotel. It answered 3% at 0.73 when it should be 0%. That is too much, but on the same flat-rate side, so the pass mark does not count it.

On the new set, with everything locked in advance, the primary run failed. It had 3 criteria misses that no one saw, and all 3 were real tax shortfalls.

The 3 cases that slipped through:

• Copier rental: answered 3% instead of 5%, at confidence 0.75
• Loan interest paid to the parent company: answered 0% instead of 1%, at 0.74
• Land rent paid to an individual: answered the progressive rate instead of 5%, at 0.81

The other 3 runs came out differently. Clef-flash in English and Clef in Thai missed nothing. Clef in English missed 1 case, the MC we come to in Part 6 (a wrong rate type, with no tax shortfall). But the run that decides is the primary run, and the primary run failed.

Part 5Where it is good, and where it slips

In the new set, while it still picked the rate itself, a one-off carpet cleaning for 800 baht was withheld at 3% in 3 of the 4 runs, at confidence 0.77 to 0.92, though a contract under 1,000 baht needs no withholding. That is withholding too much, not a tax shortfall. It read the line fine. What it missed was the amount. Most of its mistakes were like that: not reading mistakes, but comparing amounts, knowing who the recipient is, and exceptions. What it is good at is reading text and choosing one answer from the options given, such as what this payment is for, though the reading still slips now and then.

When we lined up the errors from all 51 cases, they fell into 3 kinds.

• Comparing amounts. It does not compare the amount with 1,000 baht. The 900-baht air conditioner repair in the first set was wrong in 3 of the 4 runs with rules in the prompt. The carpet cleaning above is the same mistake. Tor.Por.4/2528 clause 12/7 says no withholding is needed when the contract amount is under 1,000 baht.
• Who the recipient is. Individual or company can change the whole answer, for example the freelance web developer, interest to the parent company, and land rent to an individual. The expense line often does not say it all.
• Exceptions that need the rulings. They are in Tor.Por.4/2528 or the rulings, but the rules in the prompt did not mention them. For example hotel services, which clause 12/1 exempts, plain machine rental, whose rate differs from hiring a service, or an MC, whom a ruling does not count as a performer.

What is left is reading what the payment is for, which is a real language task, and it is the part it got mostly right. It did misread some, such as warehouse rent, which it called a service, and the MC, which it called a performer.

Along the way the question changed. Instead of asking which model can decide the tax, we started asking which part it should decide, and whose job the rest should be.

Someone who builds pure rules systems (rules engines) might ask: if you already have the table and the recipient and amount data, why use a model at all? Our answer is that expense lines are text someone typed, so reading what a payment is for is a language task. But we did not compare it with categorizing by keywords, so we cannot yet say how big the gap is.

Part 6What has to sit around it

Let the AI do only the text-reading layer. The rest comes from the accounting system, a table and people. Take the copier rental that the primary run once withheld at 3% instead of 5%. Split the answer into parts: what the payment is for comes from the model, the recipient and amount from the accounting system, the rate from a table, and exceptions go to a person. That gives the 4 layers in the picture.

Two things the picture does not show. First, the model never guesses who the recipient is or how much is paid from the text; the system reads them from the vendor master and the invoice or contract. Second, layer 2 has to keep a running total for each recipient, because when you pay the same person again in the same year, the tax has to be worked out again on the total (Part 7).

The 3 kinds of error from Part 5 land in different layers. Comparing amounts and who the recipient is go to layer 2. Exceptions that can be written as a row, such as hotels, go to layer 3. Exceptions outside the table, such as the MC, are the person's job in layer 4.

The layer 3 table looks roughly like this (a few sample rows).

This table is no smarter than the model. But if a row is wrong, we can point to the clause it cites and fix that one row.

Before letting the model try, we checked the table first, by feeding in the correct category for every case ourselves. The table gave rates matching the key on all 51 cases. That sounds good, but the rows were built from these same cases, so it only says the table matches our key. It says nothing about new cases, and it does not measure the model at all.

The "individual working alone" row needs something the expense line does not say. An individual who takes work on their own and has not set up a business earns 40(2) income, withheld at the progressive rate. With costs high enough to make it 40(8) income (business income), it is withheld at 3%. The data we simulated held only the recipient type (individual, company, financial institution, government body) and the contract amount, with no field saying the person works alone. So our table treats every individual contractor as working alone.

What you need to prepare for the system and people to draw on comes in 2 pieces. The first is the rate table where every row cites its source, for layer 3. We actually used this one in the experiment. The second is a list of known exceptions, for the person in layer 4. That one is still a proposal, in the box at the end of this part.

What splitting the job changed

After the split, the same copier rental got 5%, matching the key, and no run had a case the AI decided on its own that left tax short. But the criteria misses across all 4 runs stayed the same, because the MC case still slipped through. Tax shortfall went away completely, but it still does not pass.

We ran the same 22 cases of the new set again, in the same 4 runs, with the same 0.7 line. 3 things changed: the question sent to the model (only what the payment is for, out of 17 categories), the rate table, and the recipient type and contract amount data that code reads. We filled in the recipient type and amount ourselves from the case text, standing in for real data from a vendor master and an invoice or contract. And because 3 things changed at once, we cannot tell which one helped, or by how much.

Left of each arrow is the AI picking the rate itself, and right is the split. The primary run got better in every column and its tax shortfall fell to 0, but 1 criteria miss is left, and that column has to be 0 to pass.

Every run got more right and decided more on its own. The 2 English runs decided all 22 cases on their own, so no case reached a person at all. Tax shortfall across the 4 runs fell from 3 to 0.

More right answers, yet criteria misses did not fall, because that column counts only the cases the AI decided on its own, and it now decided more of them. The primary run improved from 3 to 1, but the other 3 runs missed more or the same, and the case left in every run is the same MC case.

In the primary run, the cases that used to be a problem changed like this.

• Copier rental used to get 3%. Asked only for the category, the model answered property rent at 0.81, which gives 5%, matching the key.
• Interest to the parent company got 1%, because the recipient type data says it is not a financial institution.
• Land rent to an individual was still put in the wrong category, but at a confidence of only 0.40, so the system sent it to a person as designed.
• Warehouse rent was put in the service category, which is wrong, but at only 0.29, so it went to a person too.
• Carpet cleaning for 800 baht got 0%, because code compares the amount itself.

The first-set cases came out right too, but on the first set we had already seen the errors before designing the table.

• The freelance web developer was put in service or personal work by all 4 runs (confidence 0.55 to 0.81) and got the progressive rate, because the recipient type data says individual. It is right only because our table treats every individual as working alone.
• The hotel seminar used to get 3% in all 4 runs. Asked only for the category, all 4 answered hotel services (confidence 0.85 to 0.98), so the table gave 0% under clause 12/1.

The case left over is the MC

Before the split, the 4 criteria misses came from 4 different cases: copier rental, interest to the parent company, land rent to an individual, and the MC. After the split, all 4 are the same case.

That case is "Paid MC fee for a product launch event to Ms. Pim, a Thai actress (individual), 30,000 baht."

The text says the recipient is an actress, so the model thinks of the performer fee category. All 4 runs answered that category at confidence 0.78 to 0.85. The category means public entertainers, such as singers, musicians, stage or film actors, and professional athletes who perform, and it is withheld at a flat 5%. But ruling Kor.Kor. 0811/7831 (กค 0811/7831) says MC work does not belong there. It is 40(2) income, withheld at the progressive rate (so this case withholds too much rather than leaving tax short, with the details in Part 7).

Before the split, the primary run also got this case wrong (5% at confidence 0.36), but the confidence was low, so the case went to a person. After the split, the model put it in performer fee at 0.81 and decided it on its own. So the primary run traded 3 old cases with tax shortfalls for 1 MC case that used to reach a person.

The thing to think about is that the word "MC" is already in a category description the model sees. We described the commission or personal work category as "commission, or pay for work a person does for you, such as an MC".

The knowledge was right in front of it, and the model still went with performer.

A case like this is an exception that accounting people have to know, and high confidence did nothing to get it to a person. The way we have in mind is in the box below.

Part 7What to watch out for

High confidence does not mean right

A system like this rests its safety on the confidence number. But in this experiment that number often failed to pick out the wrong cases.

• The small model we dropped at the start gave 0.999 to a case we wrote to be ambiguous
• The freelance web developer was wrong at 0.78 to 0.93
• Carpet cleaning was wrong at 0.77 to 0.92
• The MC was wrong at 0.78 to 0.85

After the split, both English runs, Clef-flash and Clef, decided all 22 of 22 cases on their own, with no case reaching a person, and both still missed the MC case. In runs like these, a person never gets the chance to see the case that went wrong.

If we write the rule wrong, it goes wrong with us

We only checked part of the key against Tor.Por.4/2528 or the rulings, and found mistakes of our own.

3 wrong out of 17 checked, so the key may be wrong on cases we have not checked too.

The first is the freelance web developer: "Paid website development fee to Ms. Malee, a freelancer (individual), 25,000 baht." Our first key said 3%, and all 4 runs with rules in the prompt also answered 3%, at confidence 0.78 to 0.93.

But ruling Kor.Kor. 0702/1232 covers a freelancer who takes work on their own and has not set up a business. That is income under section 40(2) of the Revenue Code and is withheld at the progressive rate. Our corrected key sets up Malee's facts this way. If the freelancer had costs high enough to make it 40(8) income, the rate would be 3% instead. The rules in the prompt never separated these two cases, so the model went wrong in the same place we did, and about as sure of it as we were.

The second is the hotel seminar, a staff seminar package with room and meals paid to a hotel, 48,000 baht. The first key said 3%. But hotel services are exempt under clause 12/1 of Tor.Por.4/2528 (see ruling Kor.Kor. 0706(Kor.Mor.05)/1022 alongside), so it must be 0%. All 4 Clef and Clef-flash runs answered 3%, following our wrong first key, because the 3% rule line in the prompt said nothing about hotels.

In these 2 cases, the mistake was in the rule a person wrote, before the model ever saw it. The model followed the rules it was given to the letter, gaps included. If we had not checked the key against the rulings, the freelance web developer would still count as a case the model got right, and the numbers we report would keep looking better than they are.

We caught it because we checked the key against the clauses of Tor.Por.4/2528 and the ruling numbers case by case. If the rules in a system carry no reference numbers, we would never find the line that is wrong.

Wording and language can change the results

We wrote one prompt per language and did not try other wordings. We wrote every rule line and category description ourselves, and translated the English ourselves. Change the wording and every number may move. The Thai and English results differ too, as the table shows.

The prompt rules themselves have a spot that does not match the law. When the AI picked the rate itself, the 0% rule said "payment under 1,000 baht at a time", while clause 12/7 counts the contract amount. We left it that way so the new set would use the same prompt.

As for model size, we cannot conclude anything yet. In some runs the larger model (Clef) did better, in others worse.

A wrong rate type may not leave tax short, but the job is not done

Counted in baht, the MC case leaves no tax short. An MC fee paid once comes to 0 tax under item 2.6, so withholding 5% means withholding 1,500 baht too much, and the payment goes on PND3 instead of PND1. But that 0-baht figure comes with one condition and one duty.

• Paying the same person again in the same year. Item 2.6 says to add the earlier payment and compute again, so the figure holds only for a first or one-off payment. The facts layer has to keep a running total for each recipient.
• 0 tax still has to be reported. A payer of 40(2) income has to list the recipient on PND1A, filed within February of the following year (Revenue Code section 58(2) and the instructions on the PND1A form).

So the tax-shortfall figures above assume 2 things: each progressive-rate case is computed on its own amount, and there is no repeat payment to the same person in the same year. They also leave out the 2 cases of employee income (a top-salesperson award in the first set and an annual bonus in the new set), because those need the salary, which the cases do not give.

The results do not pass yet, and we still need a third set of cases

If you ask what keeps us from using this for real, the first thing is that it has not passed the bar we set ourselves. No run passed on the new set, and criteria misses across the 4 runs did not fall. The second weighs more: we came up with the split after seeing the errors on both sets, so it has never met cases it had not seen. A third set of cases has to come before any talk of real use.

Two more things to know. We typed the recipient and amount data in ourselves from the case text, so the numbers do not measure errors in pulling that data from a real system at all. And the cases are few, 29 in the first set and 22 in the new set, all made up; one wrong case moves the numbers visibly. The numbers in this post show a direction, nothing more.

What we have not tried yet

The two things we most want to try next are Jev itself, because the results here come from models that take and return data the same way but are not Jev, and keyword-only categorizing, because if keywords come close, the model may not be needed at all. We have also not tried other decision models with the same API (references 5 and 6), RAG, or fine-tuning, which means training the model further on our own data.

About the RAG in the proposals box: we wondered ourselves whether it could replace the table. For now we cannot say whether it helps. All we can say is the signal from the MC case (1 case, 4 runs) that putting the knowledge in the prompt alone is not enough. If we tried it, we would put RAG at layer 4, as in the proposals box in Part 6.

But isn't a table a dead thing, when new rulings keep coming out? True. But a table where every row has a reference number shows which row to change when a new ruling comes out, and cases the table does not cover still end up with a person.

If you want to try this approach, an accounting system that already has a vendor master and keeps amounts from invoices or contracts already holds the raw material for the facts layer, although we have not tried it with real data. What you still need is a field saying whether an individual supplier works alone, and a table where every row can say where it came from.

If we were to try again with real work, we would start by preparing cases with known answers and checking the key against the rulings first, and only then ask which model gets them right.

The AI we want helping with accounting is one that knows which lines to send to a person.

In this experiment, we have not got there yet.