Guldo

Limits

What this does not show

The most useful part of a verification site is a list of what it has not verified. This page is kept as plain as the results.

What a checked result still does not prove

A certificate can be checked by anyone, and checking it proves the arithmetic and the fairness. It does not, by itself, prove the timings were honest.

  • Proven by checking: the seeds' fingerprints were public first, the questions are exactly the ones the seeds produce, the rules are the ones named, and the credit follows from the recorded timings.
  • Not proven by checking: that the recorded timings are what a real run took. Someone who runs the test could write plausible numbers. What helps: the locked-box environment (never yet run on a real machine), and other people re-measuring on their own computers. A re-measurement only counts as independent if you chose to trust whoever made it: a signature shows who signed, not that they are unconnected to the operator.
  • One way to prove a cost exactly: instead of seconds, count the Python instructions a solver executes. That number is the same on every computer, so anyone can re-run a few questions and must get identical counts, trusting nobody's machine or clock. It has limits: it is not a time, it cannot see work done inside a built-in C function (which counts as one instruction however long it runs), it needs a recent Python and pure-Python solvers, and, unless a counted round declared the sizes in its first log entry before any submission, the question sizes are chosen by whoever ran it, so it is not an official round result.
  • The baseline decides what counts: a speedup is only as meaningful as the solver it beats. A weak baseline makes big speedups cheap. Each problem's author is asked to say what the baseline is and what is known to beat it, and the problem's card shows when they have not.
  • Computers differ: a speedup is a ratio, which mostly cancels differences in machine speed, but the spread across different machines has not been measured yet, so the tolerance for re-measurement is deliberately loose.
  • Private solvers are hidden from the public, not from the operator: a submitter may keep their solver private, and then the public certificate shows only its fingerprint. Everyone can still verify the log, the questions and the arithmetic, but nobody except the operator can inspect or re-run it, so its timings rest on the operator's sandbox alone. If the owner later publishes the file, anyone can check it against the fingerprint. Hiding a solver from the operator as well would need special hardware or a zero-knowledge proof, and neither is feasible for solvers that run for minutes.

Open limits

  • The Docker sandbox that makes it safe to run other people's code has not been verified against a real Docker daemon or a Linux machine. Every run on this page used bare local processes and my own solvers, so none of it can be an official result.
  • Official credit needs three independent seeds, each with a published commitment, the sandbox, and the family's full instance count and CPU cap. No seed on this page meets all of those.
  • The credit rule was designed while looking at seed A, so seed A is not an independent test. The out-of-sample test is a pre-registered run on a fresh seed, with its pass criterion written down before the data existed.
  • The certifiable range is limited. At the official ladder (24 instances per size, 120 s cap), a simulation with real per-instance noise certifies a constant speedup of 1.5 to 3 bits with roughly 65 to 100% probability (each figure is good to about 10 points), 1 bit rarely (6.5%) and 4 bits almost never (1%), because too few hard instances remain for the bound to stay stable. A round can ask for more instances per size (48: 68% at 4 bits; 96: 100% at 4 and 98% at 5 bits), but about 5 bits is the ceiling at a 120 s cap whatever the count, since a very fast submission falls under the 0.5 s cost floor wherever the baseline finishes. Results outside the range are reported as inconclusive, never credited, and speedups of tens of bits cannot be certified by this design at all.
  • The paper's own code (the "egg") cannot yet be used as a baseline: its sieve does not reach the target past dimension 30.
  • Several jobs at once: runs use several processes on one computer at the same time. Re-measuring one job at a time shifted the baseline by -0.050 bits and the tuned solver by -0.084, so the speedup was slightly underestimated, not inflated. The pass line was a shift under 0.10.
  • That effect depends on the solver. On the factoring control, re-measured one job at a time, the baseline (a plain Python loop) moved by -0.033 bits, but the numpy solver ran 1.16 bits faster than it had with six jobs at once, against a pre-registered limit of 0.10. So that check failed. The bias happened to favour the baseline, which makes the credit cautious, but for another pair of solvers it could point the other way. Runs with several jobs at once are therefore valid only after re-measuring both sides one job at a time, and official runs use one question at a time in the locked box.
  • The factoring control's known effect (about 4.4 bits) was calibrated on 44 to 52-bit numbers. The 4.2 bits measured at 58 to 64 bits agrees with it partly because the several-jobs bias cancels a real increase with size. It is not a clean recovery of a known number.
  • One seed is one sample. The pre-registered result is a single test on a fresh seed, not a replication.

What would close each limit

Some limits can be closed with more work, and this table says what that work is. Others are properties of the method and will stay on this page.

LimitWhat would close itStatus
The sandbox that runs other people's code has not been verified on a real computerA Linux host where the sandbox's own tests and self-test passOpen
No result is officialAn official round: three independent secret numbers, run inside the verified sandbox at the problem's full sizeOpen
Runs used several jobs at onceOfficial runs evaluate one question at a time inside the sandboxOpen
One seed is one sampleThree or more fresh-seed runs, each registered before its data and judged against its written criteriaOpen
The factoring control is not a clean recovery of a known numberA calibrated pair at the ladder's own sizes, run one job at a time in the sandboxOpen
Differences between computers are not measuredRe-measurements from two or more different computersOpen
The paper's own code cannot be used as a baselineA stronger solver for the baseline (research, not engineering)Open
The certifiable range is about 1.5 to 3 bits, with a ceiling near 5Nothing removes it: it follows from the 0.5 s cost floor and the time limit. Bigger rounds stretch itStays on this page
A private solver's timings rest on the operator's sandboxWould need trusted hardware or proofs of execution, which are not feasible for solvers that run for minutesStays on this page
The baseline decides what countsIt is a judgement: authors state what beats their baseline and independent baselines helpStays on this page

Related work

Public challenges already exist in this area. The Darmstadt SVP challenge and its LWE counterpart publish instances and record solutions. Yukon, from Eigen Labs, is an open platform where people and AI agents compete on posted technical challenges. This project is about checking claims: seeds committed before the questions exist, a public log, and certificates anyone can recompute.