Results
What has been measured
Every number here comes from saved timings you can download and recompute. Each run below shows the data behind its number: the difficulty groups, every question, and a table of the same figures.
Leaderboard
| Submission | Family | Credit (bits) | Status | Per seed |
|---|---|---|---|---|
| factor_trial_np | factor | 4.1 | Provisional PROVISIONAL (0/3 official seeds) | 4.1 |
Not official because
| ||||
| tuned_bkz | gm | 0.1 | Provisional PROVISIONAL (0/3 official seeds) | 0.1, 2.6 |
Not official because
| ||||
The credit on the board is the lowest over a submission's seeds, so one weak or small run pulls it down. A row that reads lower than the pre-registered result below it usually includes an early design run (12 questions per size, 60 s limit, the data the credit rule was built from). That is why each row lists its per-seed values and the reasons it is not official.
Pre-registered test, seed 755a57fd
The pass criteria were written before any data from this seed existed and are applied here without interpretation.
- Primary claim
- 2.58 bits: outcome (a) credit >= 1.0 bit, which SUPPORTS the claim that this ladder size can certify a pruning-sized speedup.
- Exploratory hard-pool claim
- lower bound 2.94 bits: first out-of-sample support for promoting it (lower bound >= 1.0 bit).
- Concurrency check
- not recorded on this page; see the remeasure output
The registration, verbatim
PRE-REGISTRATION, seed D (supersedes the VOID seed-B and seed-C registrations). Written BEFORE any seed-D data exists. date: 2026-11-18T20:34:47Z seed commitment (sha256): 755a57fd4dd7d65fec1c77318096fc5bbc233fafe1336745d535615634b34b47 config: family gm, dims [64, 66, 68, 70, 72], n=24/dim, cap 120 s, --jobs 6 (safe max on this machine: 8), local mode (own solvers only), baseline_bkz vs tuned_bkz, this machine. PRIMARY estimator (as in the code hashes below): difficulty-stratified credit, JOINT bootstrap re-selecting bins every resample; bins = octaves of baseline cost from 0.5 s; usable = >=5 instances and submission median cost >=0.5 s; top 3 hardest usable bins; baseline-timeout instances excluded; one-sided 95% lower bound. UNCONDITIONAL criterion for the primary estimator: (a) credit >= 1.0 bit -> supports the claim that this ladder size can certify a pruning-sized speedup (b) 0 <= credit < 1.0, conclusive -> FAILS (c) inconclusive (< 2 usable bins) -> FAILS (insufficient power for gm 24/dim at a 120 s cap) EXPLORATORY estimator (reported beside, never instead): hard-pool claim, baseline cost >= 4 s, n >= 12, submission median >= 0.5 s. Its 4 s threshold was chosen AFTER the author saw seed A's bin table (in-sample there). A hard-pool lower bound >= 1.0 bit on seed D is the first out-of-sample support for promoting it; below 1.0 or inconclusive means it is NOT promoted. CONCURRENCY check (after the ladders finish): remeasure.py re-runs 4 instances per cost octave of the BASELINE ladder sequentially. Criterion: overall median |shift| <= 0.10 bits AND no octave with n>=3 beyond +-0.20 bits; otherwise the concurrency bias is flagged and the seed-D credit must not be called clean. Known before the run (so they are not later presented as findings): on seed A (12/dim, 60 s cap) the primary estimator gave 0.1 bits (median speedup 2.2) with ~0% simulated power; the exploratory claim gave 2.3 (in-sample). The primary rule's design (bins, top-3, floor) was influenced by seed A; the joint-bootstrap correction came from simulation. Production-code simulations at n=120: lower bound exceeds truth 2.5-5% of the time; 63% power at a true 1.5 bits, 100% at 2.5. Real runs may have fewer eligible instances (timeouts, floor). Bare local mode: a solver can escape cleanup (setsid) and macOS does not kill SIGXCPU-ignorers at the CPU limit; both solvers here are mine and well-behaved, so neither applies. No Docker is used (the sandbox meter is unverified). code hashes (sha256): 2feb07a369fcfdb7c08e8fc416f5a846f29c7fa644096d4b12be140c8a82123b extrapolate.py 5a8393b185fbfaeed37c0208ab2f6fb2f468d262ced14bbd88db1cda56fc6e4b families.py ea2a61628f280fa37262b86a9f61fbfebec8164fab43ac6a4f4edc3876929067 instances.py 6da9b5b0b3fc232b3d162ff4d03248964dca1902722c2715aaae7a10c60f1ed6 harness.py fde3e74fe6a0442d7d63246c2b29052fb9b3fe40bccf2fcb6f707dc79a92068f runner.py 8d33943015f2c2089f98c738f868451a63fddf1b8663d80293e3b6a731f20a1d solvers/baseline_bkz.py 4d011945d3f33adf3bc856f977784b5c572fe3cb0d831bb4db2307ef853247ee solvers/tuned_bkz.py
Positive control, seed 88ecb627
The criteria were written before any data existed and are applied here without interpretation. The true effect was calibrated in-process at about 4.4 bits.
- Primary claim
- 4.12 bits (median 4.24): outcome (a) credit at least 2.5 and median within [3.5, 5.5], which means the pipeline RECOVERS a known large speedup.
- Concurrency check
- not recorded on this page yet
Does the compiler's prediction match real solvers? (dlog)
Two real solvers were run through the full pipeline on three fresh seeds. What would count as success was written down before any data existed (the registration), and each criterion below is applied exactly as written, with no reinterpretation. Local mode, one computer, no locked box: even a full pass is provisional.
| Criterion | Outcome | The numbers |
|---|---|---|
| code unchanged | Not met | changed since registration: harness.py. Results measured with different code are not the registered experiment |
| A1: every seed conclusive | Met | claim 1.54 bits (per seed ['1.61', '1.67', '1.54']); pooled paired median 1.61 bits over 201 eligible pairs (all-pairs median 1.60) |
| A2: claim in [0.7, 1.7] and not above the paired median + 0.3 | Met | claim 1.54 bits (per seed ['1.61', '1.67', '1.54']); pooled paired median 1.61 bits over 201 eligible pairs (all-pairs median 1.60) |
| A3: claim >= 1.0 bit | Met | claim 1.54 bits (per seed ['1.61', '1.67', '1.54']); pooled paired median 1.61 bits over 201 eligible pairs (all-pairs median 1.60) |
| B1: not credited at 5 bits or more (the ceiling is real) | Met | claim inconclusive bits (per seed ['inconclusive', 'inconclusive', 'inconclusive']); pooled paired median 12.82 bits over 201 eligible pairs (all-pairs median 12.61) |
| B2: paired median reported beside the claim | Met | claim inconclusive bits (per seed ['inconclusive', 'inconclusive', 'inconclusive']); pooled paired median 12.82 bits over 201 eligible pairs (all-pairs median 12.61). The gap between them is the ceiling's size, not a failure |
Overall: FAIL.
Runs
dlog_stride4 on dlog, seed 725ff238
InconclusiveFewer than two difficulty bins had at least 5 instances and a submission cost of at least 0.5 s, so no credit can be certified from this run.
- Instances
- 9 generated, 4 where the baseline hit the 12 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- inconclusive (needs 12 instances at 4 s or more and a submission median cost of at least 0.5 s)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 1 | +2.11 | 0.138 | not usable |
| 2–4 | 2 | +1.66 | 0.931 | not usable |
| 8–16 | 1 | +1.61 | 3.726 | not usable |
dlog_bsgs on dlog, seed d00100a7
InconclusiveFewer than two difficulty bins had at least 5 instances and a submission cost of at least 0.5 s, so no credit can be certified from this run.
- Instances
- 72 generated, 0 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- inconclusive (needs 12 instances at 4 s or more and a submission median cost of at least 0.5 s)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 4 | +9.68 | 0.001 | not usable |
| 1–2 | 9 | +10.60 | 0.001 | not usable |
| 2–4 | 7 | +11.65 | 0.001 | not usable |
| 4–8 | 12 | +12.40 | 0.001 | not usable |
| 8–16 | 9 | +13.63 | 0.001 | not usable |
| 16–32 | 12 | +14.37 | 0.001 | not usable |
| 32–64 | 9 | +15.13 | 0.001 | not usable |
| 64–128 | 6 | +16.01 | 0.001 | not usable |
dlog_stride4 on dlog, seed d00100a7
1.7 bits certified (median speedup 1.7; the credit is the one-sided 95% lower bound)
- Instances
- 72 generated, 0 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- at least 1.6 bits (median 1.7, 48 instances)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 4 | +1.58 | 0.281 | not usable |
| 1–2 | 9 | +1.38 | 0.567 | usable |
| 2–4 | 7 | +1.40 | 0.987 | usable |
| 4–8 | 12 | +1.34 | 2.241 | usable |
| 8–16 | 9 | +1.73 | 4.058 | usable |
| 16–32 | 12 | +1.67 | 6.517 | in the credit |
| 32–64 | 9 | +1.80 | 12.954 | in the credit |
| 64–128 | 6 | +1.97 | 19.227 | in the credit |
dlog_bsgs on dlog, seed e5dab2c8
InconclusiveFewer than two difficulty bins had at least 5 instances and a submission cost of at least 0.5 s, so no credit can be certified from this run.
- Instances
- 72 generated, 0 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- inconclusive (needs 12 instances at 4 s or more and a submission median cost of at least 0.5 s)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 6 | +9.56 | 0.001 | not usable |
| 1–2 | 8 | +10.44 | 0.001 | not usable |
| 2–4 | 9 | +11.68 | 0.001 | not usable |
| 4–8 | 12 | +12.29 | 0.001 | not usable |
| 8–16 | 5 | +13.50 | 0.001 | not usable |
| 16–32 | 12 | +14.57 | 0.001 | not usable |
| 32–64 | 6 | +15.11 | 0.001 | not usable |
| 64–128 | 6 | +14.16 | 0.004 | not usable |
dlog_stride4 on dlog, seed e5dab2c8
1.6 bits certified (median speedup 1.8; the credit is the one-sided 95% lower bound)
- Instances
- 72 generated, 0 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- at least 1.6 bits (median 1.7, 41 instances)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 6 | +1.55 | 0.25 | not usable |
| 1–2 | 8 | +1.67 | 0.473 | not usable |
| 2–4 | 9 | +1.41 | 1.156 | usable |
| 4–8 | 12 | +1.43 | 1.841 | usable |
| 8–16 | 5 | +1.75 | 2.878 | usable |
| 16–32 | 12 | +1.78 | 7.004 | in the credit |
| 32–64 | 6 | +1.87 | 10.745 | in the credit |
| 64–128 | 6 | +1.75 | 23.737 | in the credit |
dlog_bsgs on dlog, seed e70be0d8
InconclusiveFewer than two difficulty bins had at least 5 instances and a submission cost of at least 0.5 s, so no credit can be certified from this run.
- Instances
- 72 generated, 0 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- inconclusive (needs 12 instances at 4 s or more and a submission median cost of at least 0.5 s)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 3 | +9.74 | 0.001 | not usable |
| 1–2 | 4 | +10.44 | 0.001 | not usable |
| 2–4 | 13 | +11.60 | 0.001 | not usable |
| 4–8 | 15 | +12.39 | 0.001 | not usable |
| 8–16 | 11 | +13.52 | 0.001 | not usable |
| 16–32 | 10 | +14.47 | 0.001 | not usable |
| 32–64 | 9 | +15.59 | 0.001 | not usable |
| 64–128 | 4 | +16.19 | 0.001 | not usable |
dlog_stride4 on dlog, seed e70be0d8
1.5 bits certified (median speedup 1.6; the credit is the one-sided 95% lower bound)
- Instances
- 72 generated, 0 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- at least 1.5 bits (median 1.6, 49 instances)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 3 | +1.35 | 0.288 | not usable |
| 1–2 | 4 | +1.32 | 0.556 | not usable |
| 2–4 | 13 | +1.33 | 1.246 | usable |
| 4–8 | 15 | +1.50 | 1.986 | usable |
| 8–16 | 11 | +1.62 | 3.367 | in the credit |
| 16–32 | 10 | +1.60 | 7.336 | in the credit |
| 32–64 | 9 | +1.58 | 18.072 | in the credit |
| 64–128 | 4 | +1.73 | 24.966 | not usable |
factor_trial_np on factor, seed 88ecb627
4.1 bits certified (median speedup 4.2; the credit is the one-sided 95% lower bound)
- Instances
- 96 generated, 7 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 6 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- at least 3.8 bits (median 3.8, 89 instances)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 4–8 | 19 | +3.78 | 0.472 | not usable |
| 8–16 | 27 | +3.69 | 0.912 | usable |
| 16–32 | 12 | +3.84 | 1.812 | in the credit |
| 32–64 | 13 | +4.16 | 2.399 | in the credit |
| 64–128 | 18 | +4.50 | 4.013 | in the credit |
tuned_bkz on gm, seed 755a57fd
2.6 bits certified (median speedup 4.1; the credit is the one-sided 95% lower bound)
- Instances
- 120 generated, 41 where the baseline hit the 120 s cap (excluded)
- Run
- bare local process, 6 concurrent job(s), Apple M4 Pro
- Hard-pool claim (exploratory)
- at least 2.9 bits (median 4.1, 30 instances)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 15 | -0.86 | 0.951 | usable |
| 1–2 | 11 | +0.65 | 0.698 | usable |
| 2–4 | 10 | +2.28 | 0.663 | usable |
| 4–8 | 11 | +2.56 | 0.819 | in the credit |
| 16–32 | 13 | +4.91 | 0.815 | in the credit |
| 32–64 | 5 | +5.04 | 1.015 | in the credit |
| 64–128 | 1 | +6.46 | 1.144 | not usable |
tuned_bkz on gm, seed e78e4a8f
0.1 bits certified (median speedup 2.2; the credit is the one-sided 95% lower bound)
- Instances
- 108 generated, 19 where the baseline hit the 60 s cap (excluded)
- Run
- bare local process, 1 concurrent job(s), unknown CPU
- Hard-pool claim (exploratory)
- at least 2.3 bits (median 4.6, 13 instances)
in the credit usable not usablecredit (lower bound)
Median over all paired instances, with a 90% bootstrap interval; where the baseline hit its cap its cost is taken as the cap, so those instances can only understate the speedup. Hollow points are dimensions the estimator does not use ("capped": more than half the baseline runs hit the cap; "< floor": costs under 0.5 s): they are not measured, shown only as lower bounds or for reference. Intervals of neighbouring dimensions often overlap, so a rising median is a tendency, not a measured rate. Growth beyond the last measured dimension is not measured either: no line is drawn across a gap, and nothing here is extrapolated.
Table view
| Baseline cost (s) | Instances | Median bits | Submission median (s) | Role |
|---|---|---|---|---|
| 0.5–1 | 15 | +0.17 | 0.53 | usable |
| 1–2 | 5 | +0.54 | 0.997 | in the credit |
| 2–4 | 8 | +2.21 | 0.674 | in the credit |
| 4–8 | 4 | +1.91 | 1.524 | not usable |
| 8–16 | 1 | +4.57 | 0.381 | not usable |
| 16–32 | 7 | +5.02 | 0.744 | in the credit |
| 32–64 | 1 | +4.87 | 1.28 | not usable |
Reading these charts
- Each bar is the typical (median) speedup in bits for questions grouped by how hard the baseline found them. Each bit is a factor of two.
- Solid bars are the three hardest groups that count, and they make up the credit. Light bars count but are not among the hardest. Dashed bars have too few questions, or sit below the 0.5 s floor, so they do not count.
- The dashed line is the credit. It sits below the bars on purpose: it is the cautious number, not the typical one.
- The "speedup by problem size" chart asks a different question: does the speedup grow as the problem gets harder? It shows the typical speedup at each size with a 90% range. Where the baseline ran out of time, its cost is taken as the time limit, so those questions can only understate the speedup. Hollow points are sizes the credit does not use (too many baseline time-outs, or costs under the 0.5 s floor). They are lower bounds or reference points, not measurements, and nothing is guessed beyond them or joined by a line.
- A group can show a negative speedup: a tuned solver with a fixed setup cost is slower on trivial questions. That is why the floor exists, and it is shown, not hidden.