Quantifying LLM Cost Savings from Cache-Aware Inference Arbitrage

In this benchmark, the same LLM session could cost up to 4x as much depending on the provider used. The gap comes from token pricing, provider prompt-caching behavior, and users’ request patterns. When those factors diverge, a cache-aware cost-arbitrage layer can reduce spend. The question is how much it saves and how reliably.

We benchmarked Auriko's cache-aware cost-arbitrage engine against five inference providers and an inference routing peer. The benchmark sent over 80,000 API requests to 37 models across 3 workload types. These requests form over 22,000 sessions. Auriko reduced dollar-weighted spend against six tested baselines: 32.8% against the routing peer (95% CI [30.6%, 34.9%]) and 7.7–38.3% across five single-provider baselines.

This report presents the benchmark results, explains the cost drivers behind the savings, and documents the methodology, statistical tests, and robustness checks.

The Opportunity: Cost Dispersion Across Providers

Inference cost for the same requested model varies across providers. For some models we tested, the most expensive path costs 4x the cheapest for an identical request.

We compared 11 models across providers. For each model, we divided each provider's average cost by the lowest-cost provider's average cost and multiplied by 100. A score of 100 means cheapest. A score of 300 means three times the cost for the same request.

The variation appears across models in the study. The pattern holds across workload types. Coding agent workloads show the widest gap; single-turn and conversation workloads show comparable spreads (details in Robustness and Sensitivity).

This cost dispersion means that within the tested provider set, a cheapest provider choice exists for each measured workload. The rest of this report presents a statistical study of the cost reduction observed for Auriko's cache-aware routing approach. The next section describes how we conducted the study.

Experimental Design

This study adopts a matched-pairs design. For each model and workload combination, every call to Auriko and each comparator uses identical prompts. Request parameters — temperature, top-p, frequency penalty, presence penalty, and maximum output length — are held constant across all calls. Both sides are dispatched concurrently to control for time-of-day effects. Each combination is repeated multiple times. If either side of a pair errors, both sides are excluded before analysis.

Coverage

Requests made counts individual API calls. LLM sessions group related calls into workload sessions, including multi-turn conversations and multi-step agent runs. Requested paired sessions are target-level session pairings before error exclusion; clean paired target comparisons are the same pairings after symmetric exclusion when either side errored.

Metric

Cost is the dollar amount returned by each API response. For session-level analysis, request costs are summed within each workload session. For the routing peer, cost is adjusted for their platform fee, matching what a user of that service effectively pays:

adjusted cost = API-returned cost × (1 + platform fee %)

Since Auriko charges no platform fees, no cost adjustment is needed.

Statistical Methods

Validity Controls

The Evidence: Cost Reduction

The dispersion section showed that the same requested model can cost materially different amounts on different providers. This section presents Auriko's realized cost savings against each comparator target.

Primary Findings

Auriko reduced dollar-weighted spend against six comparator targets, with target-level savings ranging from 7.7% to 38.3%.

The chart below shows the distribution of session-level cost differences for each comparator target, together with the percentage of wins, ties, and losses — where a win is a session where Auriko costs less than the comparator. Auriko costs less in 60–90% of non-tie sessions across six targets.

The chart below shows session win rates with 95% confidence intervals, ties excluded. Auriko costs less in the majority of non-tie sessions against each comparator target, with confidence intervals above the 50% even-odds line.

For each comparator target, the chart below shows dollar-weighted savings with 95% confidence intervals. Savings range from 7.7% (Model provider E) to 38.3% (Model provider A). Confidence intervals are computed via stratified bootstrap.

These panels show how cost accumulates over a multi-turn conversation. Each line is the average cumulative cost across models tested against that comparator. The gap between the lines is the savings, and it widens with each turn, a pattern consistent with routing choices and reusable prompt context becoming more valuable over successive turns. This is conversation-scenario data only; single-turn workloads have no turn progression to show.

The per-model breakdown below shows the cumulative-cost gap is also visible at the individual-model level.

Cost Reduction Across Workloads

Each cell represents one comparator target paired with one workload scenario, annotated with the dollar-weighted savings percentage for that pair. Color encodes magnitude. Cost reduction is positive across workload types and comparator targets.

For most targets, multi-turn workloads show larger cost reduction than single-turn requests, consistent with reusable prompt context becoming more valuable over successive turns.

Cost Reduction Across Models

A common objection to average savings is that one model with a large cost difference could drive the entire result. This subsection tests whether savings are broad across the tested model set.

Holm-significant models survived Holm–Bonferroni multiple-comparison correction, a conservative family-wise correction across all model-level tests within the comparator group.

Each dot is one model's dollar-weighted savings with a 95% bootstrap confidence interval. Blue markers indicate models that survived Holm–Bonferroni correction. Models near or below zero have wide confidence intervals reflecting small sample sizes.

Statistical Tests

For each comparator target, we computed the session-level cost difference d = comparator cost − Auriko cost, where positive values indicate Auriko was cheaper. We tested the null hypothesis that Auriko has no lower session-level cost (H₀: the distribution of d is centered at zero) against the one-sided alternative (H₁: d is shifted positive) using two nonparametric tests.

The sign test counts non-tie sessions where Auriko costs less than the comparator versus more, and tests whether the lower-cost proportion exceeds 50%. The Wilcoxon signed-rank test incorporates magnitude by ranking the absolute differences, then testing whether positive ranks systematically outweigh negative ranks. Zero differences are included in the ranking (Wilcox method). We report both tests because they are complementary: the sign test is robust to outliers (a single extreme session cannot drive the result), while the Wilcoxon is more powerful because it uses rank information.

Both tests reject H₀ at p < 0.001 for six comparator targets, with p-values ranging from 10⁻¹² to 10⁻²⁵⁴.

Summary Table

Dollar-weighted savings = (total comparator spend − total Auriko spend) / total comparator spend. 95% CI is a stratified bootstrap interval (5,000 replicates, model + scenario strata) around dollar-weighted savings.

Model provider D has 416 clean paired sessions and 2 models. It is useful as a measured target result, but its narrow coverage means it should not be treated as broad evidence.

Model provider E has the narrowest target-level savings and the highest tie rate. Its raw lower-cost rate is 42.5% because 30.2% of sessions are ties; excluding ties, Auriko costs less in 60.9% of non-tie sessions.

Dropping the highest-savings direct baseline from the cohort, model provider cohort savings remain at approximately 16.8%. The cohort result is not carried by a single anonymized baseline.

Cost Reduction Consistency

The preceding sections reported statistically significant cost differences across six comparators. Statistical significance provides evidence against the null hypothesis, but does not quantify magnitude — with sufficient sample size, even a negligible difference produces a significant p-value. Effect size measures how large the savings are relative to session-to-session cost variability, independent of sample size.

We report Hedges' g: the mean per-session dollar difference (comparator minus Auriko) divided by its standard deviation, with a small-sample bias correction applied. Higher values indicate savings that are consistent across sessions; lower values indicate savings that are positive on average but more variable per session.

Hedges' g ranges from 0.21 (Model provider E, N = 3,077) to 0.65 (Model provider A, N = 1,647). Six comparators produce positive effect sizes.

The target-level results answer the magnitude question. The next question is which measured cost components are associated with these savings.

The Mechanics: Cache-Aware Arbitrage

The actual cost of an LLM request depends on more than headline token price. On the provider side, five variables shape the realized cost: minimum token thresholds for cache activation, block granularity, cache expiration timing, write and storage costs, and context-length pricing tiers. On the user side, the workload pattern (prefix lengths, token reuse rates, request timing, and conversation depth) determines how effectively each provider's caching mechanics activate. The interaction between these two sides means the lowest-cost provider varies per request.

This section examines two measured patterns in the test data: provider cost arbitrage, where Auriko's realized route is compared with the tested provider cost range, and higher prompt-cache hit measures for the tested workloads. For a full treatment of the cost model, see How Auriko Optimizes LLM Inference Cost.

Provider Cost Arbitrage

The cost dispersion section showed that the same requested model can cost up to 4× more on different providers. This chart shows where Auriko's realized cost appears within that dispersion. For each model, the gray bar spans the cost range across the five tested single-provider baselines. Auriko's realized cost is near the low end of the range. The routing peer's cost is higher in the range.

Prompt-Cache Hit Measures

Providers charge a reduced rate for prompt tokens served from cache. Auriko had a higher token-weighted cache hit rate (cache-read tokens as a share of total input tokens) against six comparator targets, with deltas ranging from +2.9 to +11.6 percentage points.

The previous chart measured the share of input tokens served from cache. This chart measures a different dimension: the share of requests with any cache hit. Auriko has a higher request-level hit rate against five of six targets. Model provider C is the exception, where the comparator leads by 2.5 percentage points.

Session-level cache hit rate delta correlates positively with cost reduction across six comparator targets (Spearman ρ = 0.26 to 0.61).

Auriko's higher cache hit rate is consistent with selecting provider routes whose caching mechanics fit the measured usage pattern. Over successive conversation turns, the reusable prompt context grows and the measured cache-hit gap widens.

Robustness and Sensitivity

We tested the all-comparison aggregate result against 242 perturbations of the analysis sample: removing each scenario, comparator target, or model one at a time; re-including error sessions; normalizing completion-token costs; and restricting the analysis to sessions with similar output volumes. The result remained positive across every reported robustness-check family. The lowest aggregate estimate was 16.4% in the completion-matched subset.

Stress-Test Summary

The baseline dollar-weighted cost reduction across six comparator targets is 24.6% (95% CI [23.6%, 25.7%]). The table below reports the lowest estimate from each robustness-check family — the single variant in that family that produces the lowest cost reduction — along with its bootstrap confidence interval and the more conservative p-value from the sign test and Wilcoxon signed-rank test. It summarizes the all-comparison aggregate floor estimates, not all 242 robustness rows.

The full robustness package contains 242 rows across target-level, direct-provider-cohort, and all-comparison scopes: 24 scenario leave-one-out rows, 11 baseline leave-one-out rows, 175 model leave-one-out rows, 8 error-inclusion rows, and 24 completion-matched rows.

Five panels show per-target results under each stress test; the sixth shows the maximum shift from baseline for each test family. The table above reports the floor from each panel.

The error-inclusion row reintroduces 1,158 platform-complete sessions with at least one side marked as error; aggregate savings remain 18.1%.

Leave-One-Out Stability

The three leave-one-out checks remove each scenario, comparator target, or model one at a time and recompute dollar-weighted cost reduction on the remaining sample. Scenario leave-one-out produces the tightest range (23.6%–26.4%), indicating that the result is not dependent on any one workload type in the leave-one-out check. Model leave-one-out shows the largest single-element influence: excluding deepseek-v4-pro yields 19.3% savings (95% CI [18.7%, 20.0%]), reflecting that model's unusually large measured savings in the benchmark.

Baseline leave-one-out ranges from 20.1% to 30.5%, depending on which comparator is excluded. This spread reflects the natural variation in per-target cost reduction, which ranges from 7.7% to 38.3%. Across scenario, baseline, and model leave-one-out checks, the aggregate CI lower bound remains above 18%.

Per-target detail for output-volume controls follows.

Completion Normalization

We test output-volume sensitivity by scaling output costs symmetrically: whichever side produced fewer completion tokens is scaled up to match the other side. Prompt and cache costs are preserved as observed. After normalization, savings remain positive at 9.9–38.4% across targets, indicating that output-volume differences do not fully explain the result. Every per-target bootstrap interval remains above zero; the lowest lower bound is 9.5%.

Completion-Matched Sensitivity

As a second output-volume check, we restrict the analysis to sessions where Auriko and the comparator produced similar output volumes. Even in the tightest band reported in the robustness table (±10%), savings remain positive for most targets. A stricter ±5% output-matched diagnostic, tracked separately from the 242-row robustness table, retains 4,547 sessions (47.4% of paired sessions) and still produces 16.1% aggregate savings. This is evidence that output-volume mismatch does not explain the full measured savings.

Workload-Controlled Dispersion

One objection to provider cost dispersion is that different providers might serve different workload mixes. We therefore compare uncontrolled provider spread against spread measured only within common workload cells: identical prompts sent to all providers for each model. The dispersion persists after this workload control, consistent with the cost-dispersion pattern shown earlier.

The controlled audit covers 36 models across 1,556 common workload cells. The median controlled spend multiple is 1.57x; 32 of 36 models remain above 1.1x, 20 remain above 1.5x, and 7 remain above 2x.

Across the reported all-comparison stress-test families, aggregate savings ranged from 16.4% to 30.5%. Where aggregate confidence intervals are reported, their lower bounds remained above zero; completion-normalized per-target intervals also remained above zero. The next section defines the scope and boundaries of these results.

Scope of Results

This benchmark compared Auriko's realized costs against six comparator targets (one routing peer and five model providers) across 37 models and three workload types. The benchmark covers over 80,000 API requests forming over 22,000 sessions, collected over the benchmark window described in Experimental Design. Cost reduction was positive for each comparator target, each workload type, and 18 target-scenario cells, with dollar-weighted cost reduction ranging from 7.7% to 38.3% by target.

Comparator coverage. The benchmark tests six targets. The inference provider market includes providers not tested here.

Model coverage. 37 models are represented. Production workloads may include models outside this set. The model-level analysis shows cost reduction is distributed across models, not concentrated in a few.

Workload specificity. Cost reduction magnitudes are specific to this benchmark's workload mix. Different prompt lengths, conversation depths, or model distributions would produce different magnitudes.

Cost only. This benchmark measures cost. It does not measure latency, quality, or reliability differences across routes.

Validity analysis. Known limitations, confound controls, and threats to validity are examined in the Robustness and Sensitivity section.

Methodology

Metric definitions, experimental design, and statistical methods.

Glossary of Metrics

Statistical Methods

Confidence intervals. Percentile bootstrap on dollar-weighted savings. 5,000 replicates, stratified by model and scenario. 2.5th–97.5th percentile (95% CI). Fixed seed for reproducibility.

Aggregate hypothesis tests. For each comparator target, session-level cost differences (d = comparator cost − Auriko cost) are tested with two one-sided nonparametric tests. The sign test counts non-tie sessions where Auriko costs less than the comparator versus more, and tests whether the lower-cost proportion exceeds 50%. The Wilcoxon signed-rank test ranks absolute differences and tests whether positive ranks systematically outweigh negative ranks.

Per-model hypothesis tests. Within each target, per-model cost differences are tested for normality (Shapiro-Wilk). Normal distributions use a paired t-test; non-normal distributions use a Wilcoxon signed-rank test. Both are two-sided.

Multiple comparisons. Holm–Bonferroni step-down procedure at α = 0.05 on per-model p-values within each target.

Effect size. Hedges' g: mean of paired cost differences divided by their standard deviation, with small-sample bias correction.

Win rate inference. Binomial test against H₀: win rate = 50% (ties excluded). Wilson score interval on session win rate.

Heterogeneity. Cochran's Q tests whether model-level savings are homogeneous. I² estimates the proportion of variance due to true heterogeneity. τ² (DerSimonian-Laird) estimates between-model variance.

Limitations

See cost optimization live

See how Auriko reduces inference cost