A FinOps Loop for AI Workloads That Survives Past the First Cleanup

Most AI cost programs die the same way. Someone gets paged by the invoice. A team burns two weeks deleting idle GPUs and capping output tokens, the bill drops 30%, everyone feels virtuous, and two quarters later the number is back where it started.

The drift is structural. The system that generates costs changes every single week: new features, new models, new prompts, new retry logic shipped by people who were not in the cleanup meeting. A one-time project fights a snapshot. You need a loop instead.

This is the one I run: figure out where the money actually goes, pull the levers in the right order, detect anomalies on billing data, encode budgets as code. I built the automated half as CostSentinel, a Temporal-orchestrated demo pipeline I use to test these ideas against real billing exports.

Where the money actually goes

Decompose the bill before touching anything. For API-based workloads the math is boring:

cost = sum over models of (input tokens x input price) + (output tokens x output price)

Boring, but informative. Output tokens typically run 3-5x the input price, so anything that shortens responses pays disproportionately. For self-hosted models the shape changes completely: GPU hours times utilization. And utilization is where the bodies are buried. A 70B model serving four requests a minute on two 80GB GPUs is not an inference service. It is a very expensive space heater.

Then comes the spend nobody forecasts. Dev and staging environments that faithfully mirror prod's model calls. Eval suites that re-run on every merge. Retry storms during provider incidents, when your client helpfully asks the same expensive question five times. Embedding backfills. Abandoned fine-tuning endpoints still billed by the hour because nobody filed the ticket to shut them down.

In every review I have done, at least one of those was material. Usually two.

The levers, ranked

Honest ranges below, assuming you actually measure. Your traffic decides where you land, not my table.

Lever	Typical savings	Effort	Main risk
Prompt discipline	10-30% of total spend	Days	Output truncation bugs
Prompt and semantic caching	20-50% of input spend	Days to weeks	Stale cached context
Model routing	30-60% of model spend	Weeks, needs evals	Silent quality regression
Quantization	40-60% of serving infra	Weeks, self-hosted only	Accuracy loss on hard tasks
Spot capacity	60-90% on eligible batch	Weeks, self-hosted only	Mid-job interruptions

Ranked by effort-adjusted payback: prompt discipline and caching first, routing next behind an eval gate, quantization and spot only when the bill or sovereignty requirements justify self-hosting. The full version with sizing math lives in the LLM cost levers cheatsheet.

Two cautions before you start pulling. First, routing without a per-task eval gate is how you learn about quality regressions from customer tickets, which is the most expensive possible monitoring system. Second, below some bill size the optimization does not cover the engineering. At $5K a month, a heroic 40% saving is $2K. That is a fine number, but it does not pay for three weeks of routing infrastructure. Do the breakeven before the work.

The weekly loop

Thirty to sixty minutes, every week, same time. Five steps.

1. Reconcile. The billing export lands daily; the first job is confirming it arrived, parsed, and matches the provider console. A FinOps loop running on stale data is worse than no loop at all, because it looks informed while being wrong.
2. Attribute. Cost per model, per feature, per environment, per tenant where relevant. This requires cost-per-request telemetry in the gateway, not just the invoice. If you cannot state cost per successful request by feature today, that is the only project this month. Everything else waits.
3. Detect anomalies. Run detection over per-service daily cost series. Details in the next section.
4. Act. Every finding becomes a ticket with an owner and an expected saving. No owner, no finding: it goes in the notebook as context, not into the tracker as noise.
5. Verify. Next week's numbers confirm the fix moved the bill, or they do not. Optimization work that is never re-measured is a rumor.

Anomaly detection on billing data

Billing data is a hostile time series. Strong weekly seasonality. Step changes from deploys. Heavy tails from batch jobs. Fancy forecasting models miss the point entirely, because the job is to answer one narrow question: did anything move more than its history says it should, and where.

What works is unglamorous. Per-service daily series. A robust baseline, meaning a rolling median with median absolute deviation, or an IQR fence. Computed separately for weekdays and weekends, because your Saturday traffic is not your Tuesday traffic. Flag a service-day when cost lands outside the fence.

Robust statistics matter here. One embedding backfill should not permanently widen your thresholds, and mean-and-standard-deviation methods do exactly that. The outlier wins, the baseline inflates, and the next real anomaly sails through.

Then the LLM does what it is actually good at. For each flagged anomaly, the system hands the model the anomaly record, recent deploy events, and that service's usage metrics, and gets back a short diagnosis: what probably changed, what to check first. The detection itself stays deterministic. The LLM never sees the raw billing dump, which keeps both cost and hallucination down, and it means a wrong narrative never changes which anomalies get flagged. The model writes commentary, not verdicts.

This is the CostSentinel pipeline. A Temporal-scheduled daily workflow pulls billing and usage data, runs the detector, fans out LLM analysis for the top anomalies, and posts to Slack with links into a small dashboard. Why Temporal? Because the pipeline is a dozen flaky API calls strung together, and durable retries plus a schedule the server owns remove the two classic ways this kind of job silently dies: the transient error nobody retries and the cron host nobody noticed was decommissioned.

Budgets as code

Budgets that live in a console get edited by whoever is logged in during the incident. Budgets in Terraform get reviewed.

Per account or project, define budgets with alerts at 50%, 80%, 100%, and 120% of the monthly figure, wired to the channel the team actually reads. Not the channel they should read. The one they read.

The interesting part is deriving the figure. Expected volume times measured unit cost, per feature. When a feature's cost per successful request is known, its budget is arithmetic, and a breach has an interpretation: volume grew, unit cost grew, or both. Compare that to the red bar in a console, which tells you nothing except that you should feel bad.

Commit the budgets next to the infrastructure they cover. When a PR doubles expected request volume, the budget diff shows up in the same review, and the reviewer gets to ask the obvious question before the invoice does.

What does not work

Monthly invoice review: too slow, thirty days of drift before anyone looks. Account-level alerting: a 40% spike in one service hides inside a quiet account. Optimizing before measuring: the levers above are useless without attribution, because you cannot verify what you cannot assign. And trusting provider dashboards for attribution. They answer "what did we spend," never "which feature spent it," and that second question is the only one that leads to a fix.

The loop is deliberately boring. That is the point. Cost control that depends on heroics fails the first busy quarter; cost control that runs on a schedule just runs.