Skip to content

Commit dd41b8b

Browse files
authored
Merge pull request #3664 from docker/feat/custom-model-pricing
feat(config): declare explicit per-model token pricing with cost
2 parents b84ebbe + 6cb5d6f commit dd41b8b

11 files changed

Lines changed: 352 additions & 2 deletions

File tree

agent-schema.json

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1659,6 +1659,10 @@
16591659
"capabilities": {
16601660
"$ref": "#/definitions/CapabilitiesConfig",
16611661
"description": "Explicit attachment capability override for models the models.dev catalogue does not describe correctly (custom OpenAI-compatible providers, local models like Ollama, or dropped model versions). When set, the declared flags are authoritative and no models.dev lookup is performed; when omitted, capabilities are detected automatically. Without it, such models fall back to text-only and their image/PDF attachments are silently dropped."
1662+
},
1663+
"cost": {
1664+
"$ref": "#/definitions/CostConfig",
1665+
"description": "Explicit token pricing (USD per 1M tokens), overriding the models.dev catalogue. Used for per-turn cost computation, session cost tracking, the /model picker, and the after_llm_call hook's cost field. Makes an uncatalogued model (custom base_url provider, local or private deployment) 'priced' instead of billing $0. Prices must not be negative; an all-zero table means 'priced, free'. Cannot be combined with first_available."
16621666
}
16631667
},
16641668
"additionalProperties": false
@@ -1678,6 +1682,33 @@
16781682
},
16791683
"additionalProperties": false
16801684
},
1685+
"CostConfig": {
1686+
"type": "object",
1687+
"description": "Explicit token pricing for a model, in USD per one million tokens. Takes precedence over the models.dev catalogue and prices models the catalogue does not know.",
1688+
"properties": {
1689+
"input": {
1690+
"type": "number",
1691+
"minimum": 0,
1692+
"description": "USD price per 1M input tokens"
1693+
},
1694+
"output": {
1695+
"type": "number",
1696+
"minimum": 0,
1697+
"description": "USD price per 1M output tokens"
1698+
},
1699+
"cache_read": {
1700+
"type": "number",
1701+
"minimum": 0,
1702+
"description": "USD price per 1M cached input tokens"
1703+
},
1704+
"cache_write": {
1705+
"type": "number",
1706+
"minimum": 0,
1707+
"description": "USD price per 1M cache-write tokens"
1708+
}
1709+
},
1710+
"additionalProperties": false
1711+
},
16811712
"RoutingRule": {
16821713
"type": "object",
16831714
"description": "A single routing rule that maps example phrases to a target model",

docs/configuration/hooks/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -346,7 +346,7 @@ Notes:
346346
- `prompt` is also populated for `user_followup_submit`, carrying the text of the dequeued follow-up message (a user message queued for end-of-turn processing via the FollowUp API / queue, as opposed to mid-turn steering).
347347
- `stop_response` carries the model's final assistant text for `stop`, `after_llm_call`, and `subagent_stop`. `last_user_message` carries the latest user message at dispatch time.
348348
- `model_id` is populated for `after_llm_call` (and `before_llm_call`) in the canonical `<provider>/<model>` form (e.g. `anthropic/claude-sonnet-4-5`). For harness agents, `model_id` is the harness label (e.g. `claude-code`) rather than a canonical model name — see [Coding Harnesses](../../features/harnesses/index.md).
349-
- `usage` and `cost` are populated for `after_llm_call` only. `usage` is the per-call token usage object (`input_tokens`, `output_tokens`, `cached_input_tokens`, `cached_write_tokens`, and `reasoning_tokens` — the last is itself omitted for non-reasoning models); the whole object is absent when the provider reported no usage. `cost` is the USD price of that one model response. For a **native model call** it is the price computed from `usage` and the model's pricing table, and equals the cost the session records for the turn: it is **absent** when the response is unpriced (no pricing data on file, or no usage) and an explicit `0` for a priced call that was free — so a present `cost` is authoritative and an absent one means "unpriced", with no need to cross-check `usage`. (For harness agents the meaning differs — see the next note.) A cost ledger can therefore record per-call spend from the payload alone, without subscribing to the runtime event channel.
349+
- `usage` and `cost` are populated for `after_llm_call` only. `usage` is the per-call token usage object (`input_tokens`, `output_tokens`, `cached_input_tokens`, `cached_write_tokens`, and `reasoning_tokens` — the last is itself omitted for non-reasoning models); the whole object is absent when the provider reported no usage. `cost` is the USD price of that one model response. For a **native model call** it is the price computed from `usage` and the model's pricing table, and equals the cost the session records for the turn: it is **absent** when the response is unpriced (no pricing data on file, or no usage) and an explicit `0` for a priced call that was free — so a present `cost` is authoritative and an absent one means "unpriced", with no need to cross-check `usage`. Models the catalogue does not price (custom endpoints, local models) can be priced explicitly with the model-level [`cost`](../models/index.md#custom-token-pricing) config. (For harness agents the meaning differs — see the next note.) A cost ledger can therefore record per-call spend from the payload alone, without subscribing to the runtime event channel.
350350
- For [harness agents](../../features/harnesses/index.md), `cost` is the harness's own reported total for the call rather than a computed price, and is present only when the harness reported a non-zero cost (some harnesses, e.g. `codex`, report token counts but no cost — those turns carry `usage` with `cost` absent, even though the recorded message stores `0`).
351351
- `after_llm_call` fires for **every** model call, including calls made inside sub-sessions (transferred tasks, background agents, skills). For those, `session_id` is the sub-session's id. Summing `cost` across `after_llm_call` events therefore captures **all** spend, including sub-sessions (and even sub-sessions that error before their cost is persisted). Do **not** add a separately-queried session cost total on top: the runtime's own total already recurses into and includes completed sub-session spend, so combining the two double-counts. Pick one source — the summed hook costs — as the authoritative ledger.
352352
- `context_limit` is `0` when the model definition is unavailable (treat `0` as "unknown", not as a real limit).

docs/configuration/models/index.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,11 @@ models:
3737
capabilities: # Optional: override attachment capabilities
3838
image: boolean # Optional: whether the model accepts image attachments
3939
pdf: boolean # Optional: whether the model accepts PDF attachments
40+
cost: # Optional: explicit token pricing (USD per 1M tokens)
41+
input: float # Optional: price per 1M input tokens
42+
output: float # Optional: price per 1M output tokens
43+
cache_read: float # Optional: price per 1M cached input tokens
44+
cache_write: float # Optional: price per 1M cache-write tokens
4045
provider_opts: # Optional: provider-specific options
4146
key: value
4247
title_model: string # Optional: model used for session-title generation
@@ -65,6 +70,7 @@ models:
6570
| `track_usage` | boolean | ✗ | Track and report token usage for this model |
6671
| `routing` | array | ✗ | Rule-based routing to different models. See [Model Routing](../routing/index.md). |
6772
| `capabilities` | object | ✗ | Override attachment capabilities for this model. See [Attachment Capability Overrides](#attachment-capability-overrides). |
73+
| `cost` | object | ✗ | Explicit token pricing in USD per 1M tokens, overriding the built-in catalogue. See [Custom Token Pricing](#custom-token-pricing). |
6874
| `provider_opts` | object | ✗ | Provider-specific options (see provider pages) |
6975
| `title_model` | string | ✗ | Model used for session-title generation. Can be a named model from the `models:` section or an inline `provider/model` string. When omitted, the agent's primary model generates titles. Cannot be combined with `first_available`. |
7076
| `compaction_model` | string | ✗ | Model used for session compaction (summary generation). Can be a named model or an inline `provider/model` string. When omitted, the primary model compacts. Cannot be combined with `first_available`. See [Delegating Session Compaction](#delegating-session-compaction). |
@@ -111,6 +117,53 @@ conservative text-only fallback).
111117

112118
See [`examples/capability-overrides.yaml`](https://github.com/docker/docker-agent/blob/main/examples/capability-overrides.yaml) for a complete example.
113119

120+
## Custom Token Pricing
121+
122+
docker-agent prices each model call from the [models.dev](https://models.dev/)
123+
catalogue. Models the catalogue does not know — custom OpenAI-compatible
124+
providers, local models, private deployments — are "unpriced": every call is
125+
recorded at $0 despite consuming tokens, with only a log warning.
126+
127+
Declare `cost` to price a model explicitly, in **USD per one million tokens**.
128+
When set, it takes precedence over the catalogue and makes an uncatalogued
129+
model priced:
130+
131+
```yaml
132+
models:
133+
internal-gpt:
134+
provider: internal-llm
135+
model: gpt-4o
136+
cost:
137+
input: 1.25 # USD per 1M input tokens
138+
output: 5.00 # USD per 1M output tokens
139+
cache_read: 0.125 # USD per 1M cached input tokens
140+
cache_write: 1.5625 # USD per 1M cache-write tokens
141+
142+
# Also works for catalogued models, e.g. a negotiated enterprise discount:
143+
discounted-sonnet:
144+
provider: anthropic
145+
model: claude-sonnet-4-5
146+
cost:
147+
input: 2.4
148+
output: 12.0
149+
```
150+
151+
| Field | Type | Description |
152+
| ------------------ | ----- | --------------------------------------- |
153+
| `cost.input` | float | USD price per 1M input tokens |
154+
| `cost.output` | float | USD price per 1M output tokens |
155+
| `cost.cache_read` | float | USD price per 1M cached input tokens |
156+
| `cost.cache_write` | float | USD price per 1M cache-write tokens |
157+
158+
The declared prices feed per-turn cost computation, session cost tracking, the
159+
`/model` picker, and the [`after_llm_call` hook](../hooks/index.md)'s `cost`
160+
field. Prices must not be negative; omitted fields default to `0`. An all-zero
161+
table means "priced, free" — distinct from omitting `cost` entirely
162+
(unpriced). Cannot be combined with `first_available` (set it on the candidate
163+
models instead).
164+
165+
See [`examples/custom-pricing.yaml`](https://github.com/docker/docker-agent/blob/main/examples/custom-pricing.yaml) for a complete example.
166+
114167
## Delegating Session-Title Generation
115168

116169
The `title_model` field lets a heavyweight primary model hand off the cheap

examples/custom-pricing.yaml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Demonstrates the `cost` override that declares a model's token pricing
2+
# explicitly, in USD per one million tokens. It takes precedence over the
3+
# models.dev catalogue and — more importantly — prices models the catalogue
4+
# does not know at all: custom OpenAI-compatible providers, local models, and
5+
# private deployments otherwise bill $0 despite consuming tokens.
6+
#
7+
# A declared cost feeds per-turn cost computation, session cost tracking, the
8+
# /model picker, and the after_llm_call hook's `cost` field. Prices must not
9+
# be negative; an all-zero table means "priced, free" (distinct from unpriced).
10+
providers:
11+
# A self-hosted OpenAI-compatible gateway with negotiated pricing.
12+
# models.dev knows nothing about the models it serves.
13+
internal-llm:
14+
api_type: openai_chatcompletions
15+
base_url: https://llm.internal.example.com/v1
16+
token_key: INTERNAL_LLM_API_KEY
17+
18+
models:
19+
# Custom endpoint: without `cost` every call would be recorded at $0.
20+
internal-gpt:
21+
provider: internal-llm
22+
model: gpt-4o
23+
cost:
24+
input: 1.25 # USD per 1M input tokens
25+
output: 5.00 # USD per 1M output tokens
26+
cache_read: 0.125 # USD per 1M cached input tokens
27+
cache_write: 1.5625 # USD per 1M cache-write tokens
28+
29+
# Catalogued model with a discounted enterprise contract: the override
30+
# replaces the models.dev list price so recorded spend matches the invoice.
31+
discounted-sonnet:
32+
provider: anthropic
33+
model: claude-sonnet-4-5
34+
cost:
35+
input: 2.4
36+
output: 12.0
37+
38+
agents:
39+
root:
40+
model: internal-gpt
41+
description: Assistant on a custom endpoint with accurate cost tracking
42+
instruction: You are a helpful assistant.

pkg/config/latest/cost_test.go

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
package latest
2+
3+
import (
4+
"testing"
5+
6+
"github.com/goccy/go-yaml"
7+
"github.com/stretchr/testify/assert"
8+
"github.com/stretchr/testify/require"
9+
)
10+
11+
func TestCostConfigValidate(t *testing.T) {
12+
t.Parallel()
13+
14+
tests := []struct {
15+
name string
16+
cost *CostConfig
17+
wantErr string
18+
}{
19+
{name: "nil is valid"},
20+
{name: "all zero means priced free", cost: &CostConfig{}},
21+
{name: "positive prices", cost: &CostConfig{Input: 2.5, Output: 10, CacheRead: 0.25, CacheWrite: 3.125}},
22+
{name: "negative input", cost: &CostConfig{Input: -1}, wantErr: "cost.input must not be negative, got -1"},
23+
{name: "negative output", cost: &CostConfig{Output: -0.5}, wantErr: "cost.output must not be negative, got -0.5"},
24+
{name: "negative cache_read", cost: &CostConfig{CacheRead: -2}, wantErr: "cost.cache_read must not be negative, got -2"},
25+
{name: "negative cache_write", cost: &CostConfig{CacheWrite: -0.1}, wantErr: "cost.cache_write must not be negative, got -0.1"},
26+
}
27+
28+
for _, tt := range tests {
29+
t.Run(tt.name, func(t *testing.T) {
30+
t.Parallel()
31+
err := tt.cost.validate()
32+
if tt.wantErr != "" {
33+
require.ErrorContains(t, err, tt.wantErr)
34+
return
35+
}
36+
require.NoError(t, err)
37+
})
38+
}
39+
}
40+
41+
func TestModelConfigCostYAMLRoundTrip(t *testing.T) {
42+
t.Parallel()
43+
44+
const in = `provider: internal-llm
45+
model: gpt-4o
46+
cost:
47+
input: 1.25
48+
output: 5
49+
cache_read: 0.125
50+
cache_write: 1.5625
51+
`
52+
var f FlexibleModelConfig
53+
require.NoError(t, yaml.Unmarshal([]byte(in), &f))
54+
55+
require.NotNil(t, f.Cost, "cost should be parsed")
56+
assert.InEpsilon(t, 1.25, f.Cost.Input, 0.0001)
57+
assert.InEpsilon(t, 5.0, f.Cost.Output, 0.0001)
58+
assert.InEpsilon(t, 0.125, f.Cost.CacheRead, 0.0001)
59+
assert.InEpsilon(t, 1.5625, f.Cost.CacheWrite, 0.0001)
60+
61+
// A model carrying a cost override must not collapse to the
62+
// "provider/model" shorthand on marshal, or the override would be lost.
63+
assert.False(t, f.isShorthandOnly(), "cost override must defeat shorthand marshalling")
64+
65+
out, err := yaml.Marshal(f)
66+
require.NoError(t, err)
67+
68+
var rt FlexibleModelConfig
69+
require.NoError(t, yaml.Unmarshal(out, &rt))
70+
require.NotNil(t, rt.Cost, "cost should survive a marshal round-trip; got:\n%s", out)
71+
assert.InEpsilon(t, 1.25, rt.Cost.Input, 0.0001)
72+
assert.InEpsilon(t, 5.0, rt.Cost.Output, 0.0001)
73+
}
74+
75+
func TestConfigValidateRejectsNegativeCost(t *testing.T) {
76+
t.Parallel()
77+
78+
cfg := Config{
79+
Models: map[string]ModelConfig{
80+
"main": {Provider: "openai", Model: "gpt-4o", Cost: &CostConfig{Input: -1}},
81+
},
82+
Agents: Agents{
83+
{Name: "root", Model: "main"},
84+
},
85+
}
86+
require.ErrorContains(t, cfg.Validate(), "models.main: cost.input must not be negative")
87+
}

pkg/config/latest/types.go

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1030,6 +1030,48 @@ type ModelConfig struct {
10301030
// Capabilities optionally declares the model's attachment capabilities,
10311031
// overriding the automatic models.dev-based detection. See [CapabilitiesConfig].
10321032
Capabilities *CapabilitiesConfig `json:"capabilities,omitempty"`
1033+
// Cost optionally declares the model's token pricing explicitly,
1034+
// overriding the models.dev catalogue. See [CostConfig].
1035+
Cost *CostConfig `json:"cost,omitempty"`
1036+
}
1037+
1038+
// CostConfig declares a model's token pricing explicitly, in USD per one
1039+
// million tokens. When set it takes precedence over the models.dev catalogue
1040+
// for per-turn cost computation, session cost tracking, and the after_llm_call
1041+
// hook's cost field. It also makes an uncatalogued model "priced", so custom
1042+
// endpoints (base_url providers, local models, private deployments) can carry
1043+
// accurate spend accounting instead of billing $0.
1044+
type CostConfig struct {
1045+
// Input is the USD price per 1M input tokens.
1046+
Input float64 `json:"input,omitempty"`
1047+
// Output is the USD price per 1M output tokens.
1048+
Output float64 `json:"output,omitempty"`
1049+
// CacheRead is the USD price per 1M cached input tokens.
1050+
CacheRead float64 `json:"cache_read,omitempty"`
1051+
// CacheWrite is the USD price per 1M cache-write tokens.
1052+
CacheWrite float64 `json:"cache_write,omitempty"`
1053+
}
1054+
1055+
// validate rejects negative prices. All-zero is valid and means "priced,
1056+
// free" — distinct from an unset Cost (unpriced).
1057+
func (c *CostConfig) validate() error {
1058+
if c == nil {
1059+
return nil
1060+
}
1061+
for _, f := range []struct {
1062+
name string
1063+
value float64
1064+
}{
1065+
{"input", c.Input},
1066+
{"output", c.Output},
1067+
{"cache_read", c.CacheRead},
1068+
{"cache_write", c.CacheWrite},
1069+
} {
1070+
if f.value < 0 {
1071+
return fmt.Errorf("cost.%s must not be negative, got %v", f.name, f.value)
1072+
}
1073+
}
1074+
return nil
10331075
}
10341076

10351077
// CapabilitiesConfig declares a model's attachment capabilities explicitly,
@@ -1225,7 +1267,8 @@ func (f *FlexibleModelConfig) isShorthandOnly() bool {
12251267
f.TitleModel == "" &&
12261268
f.CompactionModel == "" &&
12271269
f.CompactionThreshold == nil &&
1228-
f.Capabilities == nil
1270+
f.Capabilities == nil &&
1271+
f.Cost == nil
12291272
}
12301273

12311274
// RoutingRule defines a single routing rule for model selection.

pkg/config/latest/validate.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ func (t *Config) Validate() error {
3131
if err := validateCompactionThreshold(m.CompactionThreshold); err != nil {
3232
return fmt.Errorf("models.%s: %w", name, err)
3333
}
34+
if err := m.Cost.validate(); err != nil {
35+
return fmt.Errorf("models.%s: %w", name, err)
36+
}
3437
if err := m.Auth.Validate(EffectiveProviderType(m, t.Providers)); err != nil {
3538
return fmt.Errorf("models.%s: %w", name, err)
3639
}
@@ -141,6 +144,9 @@ func (m *ModelConfig) validateFirstAvailable() error {
141144
if m.CompactionThreshold != nil {
142145
return errors.New("first_available cannot be combined with compaction_threshold")
143146
}
147+
if m.Cost != nil {
148+
return errors.New("first_available cannot be combined with cost (set it on the candidate models instead)")
149+
}
144150
for i, ref := range m.FirstAvailable {
145151
if strings.TrimSpace(ref) == "" {
146152
return fmt.Errorf("first_available[%d] must not be empty", i)

pkg/config/latest/validate_test.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ func TestModelConfigValidateFirstAvailable(t *testing.T) {
7373
{name: "with title_model", model: ModelConfig{FirstAvailable: candidates, TitleModel: "small"}, wantErr: "first_available cannot be combined with title_model"},
7474
{name: "with compaction_model", model: ModelConfig{FirstAvailable: candidates, CompactionModel: "small"}, wantErr: "first_available cannot be combined with compaction_model"},
7575
{name: "with compaction_threshold", model: ModelConfig{FirstAvailable: candidates, CompactionThreshold: new(0.5)}, wantErr: "first_available cannot be combined with compaction_threshold"},
76+
{name: "with cost", model: ModelConfig{FirstAvailable: candidates, Cost: &CostConfig{Input: 1}}, wantErr: "first_available cannot be combined with cost"},
7677
}
7778

7879
for _, tt := range tests {

0 commit comments

Comments
 (0)