-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathbuild_static_page.py
More file actions
112 lines (102 loc) · 6.97 KB
/
Copy pathbuild_static_page.py
File metadata and controls
112 lines (102 loc) · 6.97 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
#!/usr/bin/env python3
"""Build a self-contained STATIC results page from the VERIFIED consolidate_harden
gating (Sections 1/2/3). Captures consolidate_harden.py stdout verbatim so the
published numbers are exactly the resolved-gated, per-bucket-priced, schema-unified
figures — never the (broken) interactive-board aggregator in consolidate_scores.py.
Output: public/v760.html (served directly, no build/JSON dependency).
The interactive leaderboard render is deferred (fix-later, post-sliver).
"""
import subprocess, sys, html
from pathlib import Path
HERE = Path(__file__).parent
HARDEN = HERE / "consolidate_harden.py"
OUT = HERE / "public" / "v760.html"
import re
table = subprocess.run([sys.executable, str(HARDEN)], capture_output=True, text=True).stdout
# Strip internal needle IDs (→NNNN) and the retired-bench housekeeping note —
# these are internal ticket refs / noise, not for a public results page.
table = re.sub(r"→\d+", "", table)
table = "\n".join(l for l in table.splitlines()
if "[note]" not in l and "retired" not in l.lower() and "legacy pre" not in l.lower())
table_esc = html.escape(table)
PAGE = f"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>needle-bench — v7.6.0 schema-unified results (2026-06-17)</title>
<meta name="robots" content="index,follow">
<style>
:root {{ color-scheme: light dark; }}
body {{ font: 15px/1.6 -apple-system,Segoe UI,Roboto,sans-serif; max-width: 980px;
margin: 0 auto; padding: 1.5rem; color:#1a1a1a; background:#fff; }}
@media (prefers-color-scheme: dark) {{ body {{ color:#e6e6e6; background:#141414; }} a{{color:#7db3ff}} }}
h1 {{ font-size: 1.5rem; margin-bottom: .2rem; }}
.sub {{ color:#888; margin-top:0; }}
.banner {{ background:#fff3cd; border:1px solid #ffe08a; color:#5c4400; padding:.7rem 1rem;
border-radius:6px; margin:1rem 0; font-size:.9rem; }}
@media (prefers-color-scheme: dark) {{ .banner{{background:#2a2410;border-color:#5c4d12;color:#e8d98a}} }}
.box {{ background:rgba(127,127,127,0.08); border:1px solid rgba(127,127,127,0.3); border-radius:8px; padding:1rem 1.2rem; margin:1rem 0; }}
table.tldr {{ border-collapse:collapse; width:100%; font-size:.92rem; }}
table.tldr th, table.tldr td {{ text-align:left; padding:.35rem .6rem; border-bottom:1px solid rgba(127,127,127,0.25); }}
.win {{ color:#1a7f37; font-weight:700; }} .neg {{ color:#cf222e; font-weight:700; }}
@media (prefers-color-scheme: dark) {{ .win{{color:#3fb950}} .neg{{color:#f85149}} }}
pre {{ background:rgba(127,127,127,0.10); border:1px solid rgba(127,127,127,0.3); border-radius:8px; padding:1rem;
overflow-x:auto; font: 12px/1.45 ui-monospace,SFMono-Regular,Menlo,monospace; }}
code {{ font-family: ui-monospace,Menlo,monospace; }}
h2 {{ font-size:1.15rem; margin-top:1.8rem; border-bottom:1px solid #e1e4e8; padding-bottom:.2rem; }}
ul li {{ margin:.3rem 0; }}
.foot {{ color:#888; font-size:.85rem; margin-top:2rem; }}
</style>
</head>
<body>
<p><a href="/">← needle-bench.cc</a></p>
<h1>needle-bench — v7.6.0 schema-unified results</h1>
<p class="sub">Native vendor CLI vs ostk kernel · single-shot (samples=1) · captured 2026-06-17</p>
<div class="banner"><strong>Preliminary static results.</strong> These are the authoritative, accuracy-verified numbers for the v7.6.0 run.
The interactive leaderboard on the home page is being rebuilt on this dataset and should not be read as current yet.
More testing (cache-sliver projection) is in progress.</div>
<h2>TL;DR</h2>
<div class="box">
<p><strong>Apples-to-apples cost/token is honestly measurable only on Anthropic models</strong> (claude-code is the
only native harness that reports cost + cache split + real turns). Cost is summed per token bucket
(fresh 1×, cache-read 0.1×, cache-create 1.25×/2×) on a rate card validated to within 0.5% of real
Anthropic billing; efficiency deltas are computed <em>only over cells both arms solved</em>.</p>
<table class="tldr">
<tr><th>Model</th><th>Solve (native → kernel)</th><th>Cost Δ</th><th>Token Δ</th></tr>
<tr><td>claude-opus-4-8</td><td>97% → 97% (parity)</td><td class="win">−17%</td><td class="win">−49%</td></tr>
<tr><td>claude-sonnet-4-6</td><td>93% → <strong>100%</strong> (kernel +2)</td><td class="win">−38%</td><td class="win">−65%</td></tr>
<tr><td>devstral-2512 <span style="color:#888">(solve-only)</span></td><td><strong>19% → 97%</strong></td><td>n/a</td><td>n/a</td></tr>
<tr><td>gemini-3.1-pro <span style="color:#888">(solve-only)</span></td><td>97% → 100%</td><td>n/a</td><td>n/a</td></tr>
<tr><td>gpt-5.5 <span style="color:#888">(solve-only)</span></td><td>94% → 97%</td><td>n/a</td><td>n/a</td></tr>
<tr><td>kimi-k2.6 <span style="color:#888">(solve-only)</span></td><td>97% → 95%</td><td>n/a</td><td>n/a</td></tr>
<tr><td>grok-4.3 / deepseek-v4-pro <span style="color:#888">(B*)</span></td><td>native key-blocked — kernel-only</td><td>n/a</td><td>n/a</td></tr>
</table>
<p style="margin-bottom:0"><strong>Read:</strong> at equal-or-better solve rate the kernel is cheaper and far lighter on tokens on the
honestly-comparable (Anthropic) models, and a large capability multiplier for weak tool-users
(devstral 19%→97%). This is a <em>floor</em> — the cache-sliver projection is not yet in the kernel arm.</p>
</div>
<h2>Full results (verified gating)</h2>
<p>Generated verbatim from <code>consolidate_harden.py</code> — resolved-gated efficiency, per-bucket pricing,
split-resolve / both-fail / infra broken out. 38 benchmarks.</p>
<pre>{table_esc}</pre>
<h2>Methodology & caveats (read these)</h2>
<ul>
<li><strong>Single-shot (samples=1).</strong> Individual cells are noisy; trust per-model aggregates, not any one cell.</li>
<li><strong>Anthropic-only cost/tokens.</strong> gemini-cli / codex / opencode native arms don't report cost (=$0)
and under/mis-count tokens & turns, so non-Anthropic models are <em>solve-rate + kernel-absolutes only</em> — cross-arm
cost/token deltas are omitted as measurement artifacts.</li>
<li><strong>Resolved-gated efficiency.</strong> Cost/token deltas are computed only over cells <em>both</em> arms solved.
Split-resolve (one arm only), both-fail, and zero-work infra cells are excluded from the deltas and listed separately.</li>
<li><strong>Floor, not ceiling.</strong> The kernel arm here does <em>not</em> exercise the cache-sliver projection
(deferred). That mechanism makes the kernel's tokens cheap-cached and is expected to widen the cost win.</li>
<li><strong>B vs B*:</strong> true <strong>B</strong> = kernel-cpu (native driver). <strong>B*</strong> = generic OpenRouter kernel
(no hand-written driver for that provider).</li>
</ul>
<p class="foot">needle-bench · v7.6.0 · single-shot native-vs-kernel · static snapshot 2026-06-17.
Interactive render rebuilding on this data.</p>
</body>
</html>
"""
OUT.write_text(PAGE)
print(f"wrote {OUT} ({len(PAGE):,} bytes)")