Method
How a result is checked
The test assumes whoever submits a solver may be mistaken, careless or cheating, and asks what can still be concluded. Every rule below closes a specific way a result could be gamed or misread.
The rules
Seed first, reveal later
Before the deadline we publish a fingerprint of a secret number, the seed. After the deadline we reveal the seed. Every test question is computed from it, so nobody could have tuned a solver to the questions, and anyone can recompute them. You can run that check yourself on the pre-registrations page.
Every question has a known answer
Each question is built around an answer we hid in it: a short point in a lattice, or two known primes multiplied together. So no question is impossible, and a table of old answers cannot cover new questions.
Answers are checked exactly
Whole-number arithmetic only. For lattices we check that the point is on the lattice and short enough; for factoring, that the number divides evenly. A solver can claim anything. The checker decides.
Same question, both solvers
The speedup on a question is log2(baseline cost / your cost), measured on that same question. Runs under 0.5 s are ignored: a fixed startup cost would otherwise look like a speedup or a slowdown.
Failing costs the maximum
If the baseline runs out of time on a question, that question is dropped. If your solver fails, it is charged the full time limit. Dropping questions only when the baseline fails closes the trick of failing the hard ones on purpose.
Grouped by difficulty
Questions are grouped by how hard the baseline found them, each group costing about twice the one before. A group counts only if it has at least 5 questions and your solver's median cost there is at least 0.5 s. The credit comes from the three hardest groups that count.
For example: if the baseline takes about 1, 2, 4 and 8 seconds on four groups of questions, a group is used only when it holds at least 5 questions and your solver needs half a second or more on them. A solver that is so fast it finishes the easy groups in a blink simply has no say about those groups.
The credit
The credit is deliberately cautious: a number we are 95% sure the true speedup reaches or beats. It is found by re-running the whole analysis on many reshuffled copies of the data, including the step that picks the groups, so the choice of groups is part of the uncertainty. What the credit still cannot show is on the Limits page.
A second, exploratory claim covers only the hardest questions (baseline cost of 4 s or more). It picks nothing after the fact and is reported separately. It is promoted only if a pre-registered test on a fresh seed supports it.
"Official" means all of this
- The locked-box environment, with a meter that bills every process a solver starts
- At least three independent seeds, each with its fingerprint published first
- The problem's full question count and time limit
- The same computer, the same solver file and the same baseline file for every seed
- The credit is the lowest over the seeds, and a seed with no conclusion counts as zero
Pre-registration
Before the test on a fresh seed, we wrote the setup, the fingerprints of the code and the pass criteria into a file, and kept the file. The criteria are fixed in advance: whichever outcome occurs is the one the file says it is. A registration that is abandoned is marked void and kept, never edited; every one is listed on the pre-registrations page.
What a certificate adds
A certificate is one file that carries everything needed to check a result on your own machine: the questions' seeds, every measured time, both solvers and the log showing the fingerprints came first. The guide says exactly what checking it proves, and what it cannot.
Beyond speedups
The same file can back other claims, such as a benchmark score on questions fixed in advance, or a result that nobody in a contest managed to beat a target. They are on the Other claims page, each with what it does not show.