Running GPU inference on a shoestring: 7 submerged stumps before the first green email

Part 1The problem: bursty GPU workloads on a ฿1,000 budget

Our bill-scanning pilot gives each user 30 free uploads a month, with a system-wide cap of 3,000 bills. The budget was set at ฿1,000 per month (about $30). Do the math and the per-bill cost ceiling comes out to ฿0.30 (about $0.009). Go over that and the budget breaks.

The catch: the vision model that reads the bills needs a real GPU. Monthly GPU rentals start in the low thousands of baht and climb into the tens of thousands, and you pay for the whole month even if the workload only runs a few hours. Off-the-shelf vision APIs, on the other hand, charge several times our per-bill ceiling.

That narrowed the problem to one requirement: find a GPU that only charges for the seconds it actually works, and let it shut itself down the rest of the time. That's why we ended up on Modal.

Part 2The shape that keeps costs low: cheap queue, expensive worker, kept strictly separate

The whole design comes down to one rule: never let the GPU wake up and wait around for someone.

• The upload queue lives on a Cloudflare Worker with D1, which costs next to nothing.
• Every 5 minutes, a small cron job on Modal checks the queue. This part is pure CPU and cheap to run.
• If the queue is empty, the cron does nothing and exits. When there's work, it wakes the GPU class to process the whole batch at once, then the machine shuts itself down within 2 minutes of finishing.

With this shape, a month with no traffic costs the whole system a matter of tens of baht. A month with a full queue still only pays for bills actually processed.

Part 3The 7 stumps we hit along the way

Reality was never as smooth as the paragraphs above make it sound. Here is the list of submerged stumps we hit, in the order we hit them, so whoever comes next doesn't have to hit them twice.

1. The base image has to be cuda-devel, not runtime. vLLM compiles kernels at install time. Use an image without nvcc and the build fails, with an error message that doesn't tell you directly what's missing.
2. The vLLM version and the model's tokenizer have to match. Some newer models need a newer vLLM than pip resolves by default. The symptom is a model that won't load even though every file is present.
3. Calling a job that outlives the timeout needs .spawn(), not .remote(). Our cron has a 120-second lifetime, but the GPU's cold start eats 344.6 seconds. Call it as a blocking request and the cron dies before the job even starts. It has to be fired off fire-and-forget instead, letting the job keep running after the cron itself has exited.
4. .spawn() only works inside a deployed app. Test-run it locally with modal run and spawn can't find the class. You have to deploy first and let the cron be the caller. The lesson: test through the same path as production from the start.
5. A crash inside @enter is a crash loop that silently burns money. The function that runs on container wake, if it throws, Modal keeps retrying the wake indefinitely. The GPU spun up and shut down in a loop for nearly an hour before we noticed it on the dashboard. The fix is to run modal app stop immediately and diagnose afterward, and to watch the dashboard closely for the first stretch of every deploy.
6. A transcription-tuned model won't emit JSON no matter how nicely you ask. The first model we used was tuned for transcription. Ask it to answer in JSON and it rambles until it burns through its token budget. The fix wasn't a stronger prompt, it was switching to an instruct model and enforcing the format with structured outputs, so the grammar controls the output instead of a polite request.
7. An API that silently disappears on a version bump is more dangerous than one that breaks outright. vLLM 0.24 renamed the class GuidedDecodingParams to StructuredOutputsParams. Our code had a fallback in case the import failed. The result: the system never errored, but it quietly fell back to the old mode without anyone noticing, until we caught it line by line in the logs. This was the most expensive lesson of the whole project. If a fallback matters for correctness, you need to assert that the log confirms the primary mode is running, not just that nothing threw an error.

Part 4The real numbers from the logs

• First cold start (load model + compile): 344.6 seconds, paid once per wake, not per bill.
• On a warm container, the 2B model reads a bill in 7-10 seconds per page, working out to about ฿0.144 (roughly $0.004) per bill.
• Batched 30 bills per wake, the average drops to about ฿0.236 (roughly $0.007) per bill, still under the ฿0.30 ceiling.
• Upgrading to the 4B model for better accuracy pushes per-bill time to 12-22 seconds, with cost landing right at the ceiling, in exchange for correctly reading every bill in our test set.

These numbers come from our own system's logs. Your bills and hardware will differ. But the underlying pattern, an expensive cold start paid once and cheaper costs once batched, holds up anywhere.

Part 5The email we were waiting for

After clearing all seven stumps, an email from the system landed early one morning. A utility bill had cleared every check: the amounts matched, the totals matched, and it even came with a tax receipt number instead of just a plain receipt. That was the first time the entire path, from upload to inbox, had worked end to end.

Looking back, the money burned on the crash loop and the time lost chasing the silent fallback were the tuition for this system. Still cheaper than renting a GPU by the month.

If you're about to run bursty AI workloads on a tight budget, the seven stumps above are our gift to you. Check them before your first deploy. And if you want to see the system this all built toward, it's live for testing now on our AI bill scanner page.

Frequently asked questions

How does Modal charge?

It bills by the second the container actually runs, broken down by machine type. Our pipeline uses an L4 GPU for the processing step and a small CPU instance for the cron that checks the queue every 5 minutes. No queue means almost no cost.

How long is Modal's cold start?

For our system, the first wake takes 344.6 seconds, since it has to load the model and compile before running. That's paid once per wake. The way to soften the impact is to batch jobs so each wake covers as much work as possible.

Is running OCR on Modal expensive?

From our own measurements, the 2B model runs about ฿0.144 (roughly $0.004) per bill on a warm container, and about ฿0.236 (roughly $0.007) averaged across a 30-bill batch. The more accurate 4B model runs close to ฿0.30 per bill. The numbers mostly depend on model size and document length.

Should I use Modal or rent a GPU by the month?

For bursty, unpredictable workloads, Modal has the edge because you don't pay when there's no work. For workloads that need the GPU running continuously for several hours a day, every day, a dedicated rental or on-premise hardware starts to win out. The real breakeven point depends on your own GPU-hours per month.