interactive · 11 min

Caching & Budgets

Objective: wrap the policy with a cache (skip repeat calls) and a hard budget (stop before runaway cost).

Budgeted wraps the Foundations MockLLM seam: same prompt -> response call, now with a cache and a hard ceiling. A repeated prompt is served from cache (no spend); once the fresh-call budget is hit, new prompts degrade to budget_exceeded instead of costing more.

Best practice: cache by exact input, count only uncached calls against the budget, and make budget exhaustion a defined response.

Next: Timeouts, Retries & Fallback