Agent can lower val_bpb without improving the model — add an optional experiment integrity log?

[pulkit6732]

The loop keeps a commit when the printed val_bpb improves (program.md:99-103).
But the agent owns train.py, and nothing binds that printed number to real
training — so an agent optimizing for the metric can "win" without learning
anything. I verified the following against master (228791f).

Four ways the number can move without a better model:

1. Skip the optimizer, still finish the budget. step / dt /
   total_training_time are plain locals (train.py:540, 578-579, 603). Advance
   step and accrue total_training_time without loss.backward() /
   optimizer.step() and the loop completes on an untrained model.
2. Short-circuit the metric. train.py:613 calls evaluate_bpb and
   train.py:622 prints it; the agent can replace/wrap that call. The
   "do not change" on evaluate_bpb is prose only (program.md:28-31) — nothing
   enforces it.
3. Val set is never fingerprinted (prepare.py:353-354) — shrinking
   EVAL_TOKENS or swapping the shard leaves no trace. (Related to the cache-trust
   work in Harden cache artifact trust boundary in prepare.py #41 / Harden downloaded dataset shard cache in prepare.py #215, but here it's the eval side.)
4. Results are print()ed, never bound to code/data/model (train.py:621-630),
   so results.tsv keeps a commit + a number that can't be reproduced later.

Minimal reproducer (stdlib only, no torch/GPU) : https://gist.github.com/pulkit6732/a5c3ff9113bfac7e0b6ae50e69b8b567
a fabricated val_bpb beats
the current best and gets KEPT, while a one-line receipt exposes it:

[honest          (real training)]   val_bpb: 0.317639   -> KEEP (best)
[game_eval_const (skip optim+fake)] val_bpb: 0.050000   -> KEEP  <-- fake result recorded

experiment                                      val_bpb  rep_step  opt_step  untrained?
honest          (real training)                0.317639        86       688        no
game_eval_const (skip optimizer + fake eval)       0.05        12         0       YES

real_opt_steps is read from the optimizer state (not the loop var) and the
model hash equals the untrained baseline — so the gamed run is unmistakable.

Proposed fix — a ~30-line stdlib integrity log. One line per run binding
val_bpb to sha256 of train.py + prepare.py + the val shard + the final
model state, plus the real optimizer-step count and wall-time. Detection, not
prevention; no new deps; keeps the repo's minimal spirit. Integration is ~11
lines after the eval, wrapped so it can never fail a run.

Is an optional log like this something you'd want in the repo? If so I'm happy to
send a small PR (module + the hook + the standalone reproducer). And if this is
intentionally left to the human supervisor, that's a fair answer too — figured
it was worth raising.

[safal207]

The proposed integrity log looks like the right minimal primitive. One useful extension would be to make each row an explicit "ExperimentReceipt" that binds the claimed improvement to the exact causal inputs and execution evidence.

For example:

{
"experiment_id": "exp_042",
"parent_experiment_id": "exp_041",
"code_hash": "sha256:...",
"prepare_hash": "sha256:...",
"evaluation_data_hash": "sha256:...",
"initial_model_hash": "sha256:...",
"final_model_hash": "sha256:...",
"optimizer_steps": 688,
"wall_time_seconds": 300.2,
"claimed_val_bpb": 0.317639,
"verification": "verified | rejected | inconclusive",
"reason_code": "metric_bound_to_training_evidence"
}

The important property is not cryptographic sophistication; it is causal binding:

"code + data + initial state + real training steps -> final state -> reported metric"

That gives a few useful guarantees while keeping the project small:

1. A kept result can be traced to the exact code, data, and model state that produced it.
2. Fabricated or short-circuited metrics become distinguishable from real training.
3. Retries or reruns can be compared against the same parent experiment.
4. Downstream analysis can separate "verified", "rejected", and "inconclusive" runs instead of trusting every printed number equally.
5. The experiment history becomes replayable without requiring the full terminal transcript.

A practical acceptance test could be: changing only the printed metric must never produce a "verified" receipt when optimizer steps and final model hash do not support the claim.

This still fits the repo's minimal spirit: one append-only JSONL file, stdlib hashes, no prevention layer, and no dependency on the agent being honest.

[pulkit6732]

The ExperimentReceipt extension is a solid upgrade — especially parent_experiment_id for chain tracing and separating verified / rejected / inconclusive in downstream analysis. The acceptance test ("changing only the printed metric must never produce a verified receipt") is exactly right.
One gap I keep coming back to: the JSONL is writable by the same process that generates it. Software hashes are self-attesting. An agent controlling the file can stamp "verified" on a fabricated run, and nothing external disagrees. Your reason_code field is useful metadata but it's still self-reported.
The deeper fix needs an external witness — something that can't be produced without real compute having run. I've been working on this at the inference layer: a 152-byte receipt binding output hash + optimizer evidence to a hardware attestation (TPM2 or NVIDIA Confidential Computing), verifiable offline without trusting the agent. Prototype is at github.com/pulkit6732/aetherproof. The information-fidelity piece has a more formal treatment here: zenodo.org/records/20782418.
Different scope (per-inference vs per-experiment), same root problem.

[safal207]

@pulkit6732 Agreed — that is the correct boundary to draw.

The "ExperimentReceipt" I described is a process-integrity record, not an independent proof that compute physically occurred. It can bind code, data, model state, optimizer evidence, and the reported metric into one replayable object, but if the same process controls both execution and receipt generation, the record is still self-attested.

I would separate the assurance levels explicitly:

1. Process receipt — internal causal binding and replayability.
2. Signed receipt — authorship and tamper evidence.
3. External witness — independent confirmation of the execution event.
4. Hardware-backed attestation — confirmation tied to a measured execution environment.

For this repository, level 1 may still be the right minimal default because it catches accidental shortcuts, metric-printing bugs, broken lineage, and inconsistent reruns without changing the project’s deployment model. But the schema should not label that as externally verified.

A small compatibility point could be an optional field such as:

{
"verification_scope": "self_attested | externally_witnessed | hardware_attested",
"attestation_ref": null,
"verifier_id": null
}

Then "verified" would mean “verified within the declared assurance scope,” not “universally proven true.”

That keeps the local JSONL useful while leaving a clean upgrade path for independent witnesses or hardware-backed evidence. Your distinction is important: integrity, authenticity, and truth-of-execution are three different guarantees.

[pulkit6732]

Aleksey ,the assurance-level separation you've drawn is exactly right, and the verification_scope field is a clean way to make the schema honest without breaking the minimal-default case.
One thing worth noting: levels 2–4 in your taxonomy have meaningfully different threat models. A signed receipt (level 2) catches post-hoc tampering but not fabrication during execution. An external witness (level 3) adds independence but is still vulnerable if the witness and the executor collude. Hardware-backed attestation (level 4) is the only level where the receipt itself carries evidence that a specific measured environment produced a specific output the verifier doesn't have to trust any party's claim.
The upgrade path you're describing is exactly what I've been building for AI inference specifically a 152-byte receipt schema that starts at level 2 and has explicit fields for hardware evidence and log anchoring at levels 3–4, all verifiable offline with only a public key and an append-only log. The schema is designed so level 1 self-attested records can carry a verification_scope: self_attested flag and remain compatible as the assurance level rises.
Happy to share the receipt schema if useful it may be worth aligning field names early if this direction gets traction here.