All blogs
9 mins
LLM Unit Economics: Measuring the Cost of AI Requests
Piyush-Kalra
I have seen vendors celebrate price reductions for their cheap model and then soon after find their AI features incur losses at scale. The problem is not just about the model. Often, teams do not account for the full cost of the workload the model is embedded within.
The cost of LLMs is often roughly estimated as the sum of input and output tokens at the price of the model per million tokens. This answer approximates the expected cost of sending API requests. This cost metric is far from sufficient to consider the AI product's economics with respect to the cost of infra, retries, and the work to actually complete the task. These optimization efforts for the token cost of LLMs can lead to unrealized costs of the workload.
In this article, I'll break down how to calculate the cost of an AI request, why cost per successful outcome can be a more useful unit-economics metric, and where the biggest hidden costs are in production AI systems.
What Is LLM Unit Economics?
LLM unit economics refers to the estimation of the full cost of delivering a single unit of value with the aid of AI, whether it be a request, completed workflow, or successful outcome. This goes beyond the simple cost calculation of an LLM to determine the cost of an operational AI feature.
Generally, this cost calculation will progress through four stages:
Token cost: What the model charges for input and output tokens.
Request cost: The cost of a single model call, including the cost of allocated infrastructure, if any.
Workflow cost: The total cost of model calls, tool calls, retries, and any other resources used to complete the task.
Cost per successful outcome: Total AI spend divided by the number of outcomes that meet the defined success criteria.
How to Calculate the Cost of an AI Request
Basic LLM request cost:
Request cost = (input tokens × input price) + (output tokens × output price)
The above is the cost-per-token calculation for LLMs used by most LLM cost calculators. Input and output tokens are priced differently, since prompt processing and autoregressive output generation have different computational complexities, though the pricing ration varies by model and vendor.
Cost of an AI request in the real world:
Total request cost = model cost + allocated infrastructure cost + supporting services + retries/rework
Model cost is the API or token cost described above.
Cost of allocated infrastructure is the GPU or compute capacity, whether it is rented for a single request or comes from a self-hosted environment.
Supporting services include vector databases, storage, logging, networking, and other services needed to run the workflow.
Retries and rework cover failed generations, low-confidence outputs, and repeated model calls.
Example: calculating LLM cost per request
Consider an AI support system where the cost per request for the LLM API is $0.03, GPU compute, and database and network costs are $0.02 and $0.07, respectively, with the cost for retries and failed generations being $0.03. The dashboard shows a cost of $0.03 for an AI request. However, the true cost of the overall workload would be $0.15.
The token cost of LLMs is useful to an extent; however, when there are significant costs for infrastructure and retries, the token cost does not reflect the true cost of workloads.
Cost Per Request vs. Cost Per Outcome
A $0.15 request doesn’t translate to a $0.15 outcome. Not every request yields an acceptable outcome.
Why Requests Aren't Always Successful
There are several ways production AI workflows can fail in a predictable manner. For example, a request can be formatted incorrectly, a tool call can fail, an answer can be sent for review that has low confidence, a multi-step agent requires more internal calls to the model to complete a task, etc. All of these result in cost and no successful outcome.
Calculating Cost Per Successful Outcome
Cost per successful outcome = total AI spend ÷ successful outcomes
Considering the $0.15 average cost per request, if 75% of requests produce an acceptable outcome, then the cost per successful outcome is:
$0.15 ÷ 0.75 = $0.20
This is a better cost of delivering a successful outcome than considering the model's token cost in isolation.
Why Cost Per Outcome Is a Better LLM Unit Economics Metric
Cost per outcome evaluates the economics of the work the system actually accomplishes, rather than the model’s consumption. Two systems can have identical token costs and vastly different unit economics if one system has a 95% success rate and the other has a 60% success rate. Focus on outcomes over calls.
Hidden Costs Behind Every AI Request
Context and token repetition: Agentic workflows can resend context at every step. If a five-step workflow sends a 5000-token context on each call, it can process 25,000 tokens before output is generated.
Agent loops and multiple model calls: A single user request can cause a model call, a tool call, an error, a retry, context resending, and a verification call. What is effectively a single request in your application can incur many billable model calls.
Retries and failed requests: Retries can cause token spending and infrastructure costs. If the JSON structure or total is not validated, then the model call is rerun. It is cheaper to do JSON structural validation and ensure that the total is justified by line items.
Retrieval and data infrastructure: RAG systems use vector databases, storage, and retrieval pipelines. These databases and pipelines support systems and can be a large part of the total cost, sometimes exceeding the cost of the model.
Networking, logging, and observability: Production AI systems require monitoring, logging, and network capacity. These costs are not included with an LLM API, but they add to the total cost of a cloud computing workload.
How Does GPU Utilization Affect LLM Unit Economics?
If you choose to self-host inference rather than use a managed API, GPU utilization becomes the primary driver of the cost per inference. Unlike API costs, you have to pay for the GPU capacity, even if it is not fully utilized or sitting idle.
Idle and Overprovisioned GPU Capacity
Suppose a company spends $100,000 a month on GPU infrastructure and gets an average of 40% utilization. There are infrastructure costs associated with productive capacity. With full utilization, these infrastructure costs are lower. This example explains how idle capacity leads to an increased effective cost per inference.
Autoscaling and Variable AI Demand
AI workloads often exhibit demand that combines steadiness and bursts that are unpredictable. Infrastructure optimized for peak traffic can lead to higher costs per inference if it is underutilized for the remainder of the time. Autoscaling can help align infrastructure to actual demand, as long as a workload allows for it.
Cost Per Productive GPU Hour
Productive AI work per hour is the actual utility. GPU hours purchased in isolation do not give an adequate metric of GPU infrastructure. Higher productivity utilization of self-hosted inference can bring notable changes to the economics if performance and reliability are not adversely impacted.
How to Optimize LLM Infrastructure Costs
Right-size compute: Underprovision compute resources and align them with projected workload demand rather than overprovisioning for peak traffic.
Eliminate idle resources: Resources operating at super low utilization should be shut down, scaled down, or moved to more appropriate configurations.
Optimize GPU and compute utilization: Optimize scheduling and enhance batching and capacity allocation prior to purchasing more hardware. Higher productive utilization can reduce the infrastructure cost allocated to each inference.
Use autoscaling where appropriate: Capacity should be aligned with demand rather than operating fixed infrastructure at all times. This is most beneficial for workloads with large variation in traffic or demand for inference.
Optimize cloud commitments: For predictable and stable workloads, commitment-based pricing helps lower the cost of infrastructure. On AWS, Savings Plans and Reserved Instances can help customers save up to 66%-72%, depending on the plan and the commitment. On Google Cloud, customers can save up to 55% on many machine configurations with resource-based Committed Use Discounts. For Azure, eligible VM reservations can help customers save up to 72%, depending on their commitment. The actual savings vary by the resource, region, and usage.
Review storage and networking costs: AI workloads incur significant costs for storage, networking, logging, vector databases, and data. These should be included in the cost modeling rather than treating the cost model bill as the total cost for AI.
This cost optimization applies to the layers underneath a model call: the compute resources and infrastructure.
How Pump Helps Improve LLM Unit Economics
Pump doesn't optimize your prompts or model choice. It optimizes the cloud infrastructure bill underneath your AI workload. That's an important distinction: LLM optimization reduces the model and inference costs, while cloud optimization reduces the infrastructure cost required to run those workloads.
Get visibility into cloud spend: Pump View brings AWS, GCP, Azure, and engineering-tool spend into one dashboard, while integrations with OpenAI and Anthropic provide visibility into AI API usage and spend.
Identify infrastructure waste: Pump continuously monitors infrastructure and surface optimization opportunities, including idle and underutilized resources.
Right-size infrastructure: Infrastructure recommendations can help teams align capacity more closely with actual usage and reduce overprovisioned resources.
Optimize cloud commitments: Pump automatically manages commitment-based savings across AWS, GCP, and Azure, including Reserved Instances, Savings Plans, and Committed Use Discounts.
Identify more efficient infrastructure: Pump's optimization capabilities can surface opportunities to reduce unnecessary infrastructure spend as workloads and usage patterns change.
Better infrastructure economics lower the cost of delivering each AI workload, which can improve the overall unit economics of the AI product built on top of it.
A Practical Framework for Measuring LLM Unit Economics
Measure model cost: Track API usage, input, and output tokens. Also, track total API spend per model.
Measure infrastructure cost: Account for compute, GPUs, storage, and networking. When possible, charge shared costs to the workloads.
Measure request and workflow volume: Understand the full extent of model, tool, and other API calls and other tasks one user request initiates.
Measure successful outcomes: Define what counts as a successful result for your product or workflow.
Calculate cost per outcome: Cost per successful outcome = total AI costs ÷ successful outcomes.
Find the largest optimization opportunities: Prioritize the model, infrastructure, workflow, or failure points that have the greatest cost impact.
Metric | What it tells you |
Cost per token | Model consumption efficiency |
Cost per request | Basic AI request economics |
Cost per workflow | Multi-step workload economics |
Infrastructure cost | Cloud efficiency |
GPU utilization | How effectively GPU capacity is being used |
Cost per successful outcome | Outcome-level unit economics |
Conclusion
When defining AI economics, it’s imperative to think beyond model prices. The real figure is the total cost of delivering an outcome. This includes the cost of model calls, retry calls, compute, GPUs, storage, network, and the infrastructure cost of running the workload.
For cloud-based AI workload teams, optimizing workload infrastructure can be an important lever to improve the unit economics of LLMs. Focus on measuring cost per successful outcome in addition to cost per token to see the actual cost of running your AI workloads.
Frequently Asked Questions
What is the difference between LLM cost and LLM unit economics?
When referring to LLM cost, we mean the price charged by a model provider for token usage, such as an API call. LLM unit economics is using outcome-based indicators to analyze the cost of an AI service, including model calls, infrastructure, retries, and ancillary services, among others.
How do you calculate LLM cost per token?
Define the cost of each input token by the number of tokens used and multiply this by the input token price, then do the same for output tokens and multiply the cost of output tokens by the price of output tokens. Output tokens are usually priced higher than input tokens. Model providers set output token prices based on the complexity of the model.
What is the cost per successful AI outcome?
Cost per successful outcome refers to the total AI spend (model spend + infrastructure + relevant retries) divided by the number of requests that return an acceptable outcome. In our example, each request costs an average of $0.15, and 75% of the requests return an acceptable outcome, resulting in a cost per successful outcome of $0.20.
What is the best way to track LLM costs?
Users should track model costs, infra costs, and volume of workflow. Cost per outcome = model cost + infra cost. Token pricing is helpful to track model consumption. However, it does not capture carbon/infrastructure spending, retries, retrieval, or other support services.
Similar Blog Posts
Amazon Nova Pricing: Cost Breakdown & Savings Guide
Rightsizing vs Autoscaling in Kubernetes: A Complete Guide
