[arXiv]score: 0.23

Output Compression Cuts API Costs 1.4–2.4x; Input Compression Raises Them

June 24, 2026

Evaluated across 8 models and 5 datasets, output compression (forcing shorter model responses) reduces realized API cost 1.4–2.4x per model with up to 3x in best cases, while input compression (shortened prompts) raises net cost ~1.15x on average and up to 2.7x under aggressive compression because models compensate with longer outputs. The CAVEWOMAN benchmark provides a two-channel evaluation protocol measuring task accuracy, realized cost, and reference agreement.

HOW THIS AFFECTS YOU

●

builderCompressing your prompts to save tokens likely backfires — focus cost optimization on output length constraints, not input brevity.

●

researcherThe two-channel evaluation protocol separating input vs. output compression effects is a useful framework for future inference cost studies.

read original ↗arxiv.org

← back to feed