Myopic pointsHow frontier models fail expert STEM problems

The ZORA team at VedronOctober 20, 2026

Every ZORA problem is written by a domain expert to sit just past what frontier models can do, and most arrive with the author's record of a model failing it: the response, the line where it went wrong, and why. Until now we have used those records the way most evaluation work does, as a pass/fail stamp on the problem. This report reads them as data.

We coded every documented failure from four frontier models (GPT-5.5, GPT-5.4, GPT-5.3 and Claude Opus 4.8) on problems in physics, engineering, applied mathematics and quantitative finance. We then recomputed the longest chain in the set, an 18-step adversarial VWAP benchmark, end to end. None of the documented failures is a knowledge gap: the guidelines exclude them, and the critiques bear that out. They are failures of control: models adopt values they never check, read one rule as another, add structure the problem does not contain, and stop when the problem is fully determined.

Summary

Verification failures dominate. The two most common mechanisms are unverified anchoring and false precision or invented structure. At least one of them appears in more than half of the failed responses.
Passing checks proved nothing. With GPT-5.3's anchored root (residual 0.0364), all five of the task's mandatory consistency checks still pass.
Upstream errors can shrink. The anchored root moves the circuit-breaker time by 4.6% but the final P/L by only 0.26%. On its own, that is a wrong answer under exact grading and a correct one under any tolerance of 1% or more.
3 of 12 GPT-5.5 failures were refusals. Each declared a fully specified problem underdetermined and returned no number.
Eight runs, eight answers. Claude Opus 4.8's eight independent runs on one problem returned eight different exact fractions. Four landed within 5.1–6.5% of the key.

Want the problems themselves? Get the five-problem sample.

1What we studied

The problems were submitted between July and August 2026 by domain experts, university faculty among them, under guidelines aligned with the Humanity's Last Exam task rules. Each must be original, have exactly one verifiable answer, and be solvable by hand by an expert in under an hour. The bar for acceptance is adversarial: fewer than four correct in eight attempts (Pass@8 below 4/8) on a frontier model. A failure counts only if the final answer is wrong and the reasoning contains a genuine error, and a wrong answer within about 5% of the key does not count at all.

Each problem comes with its author's complete worked solution. Where our reviewers re-derived one independently, as for the VWAP task's full 18-step chain with unrounded quantities, the report says so.

Authors tested against whichever model they could reach, so the records come in three shapes. GPT-5.5 and GPT-5.4 have one critiqued response per problem. GPT-5.3 has eight successive passes at the VWAP task in a single conversation (ChatGPT Pro), each critiqued. Claude Opus 4.8 has eight independent runs at the replicas problem. 3 further problems arrived with a designed trap but no model record; we discuss one of them below but exclude all three from every count.

Figure 1. The testing record

	GPT-5.512 responses	GPT-5.41 response	GPT-5.38 responses	Claude Opus 4.88 responses	None3 problems
Physics	9	1	–	–	–
Engineering	2	–	–	8	–
Mathematics	1	–	–	–	–
Quantitative finance	–	–	8	–	–
No model record	–	–	–	–	3

Each square is one documented failed response; the column says which model, the row which domain. Read across a row to see that no two models faced the same problem. That is the property that makes this a record of failures rather than a benchmark.

How to read the numbers

Every problem here was kept because its target model failed it, so every documented pass rate is zero by construction, and no two models attempted the same problem. Nothing in this report ranks one model above another. What the records do support is a study of how models fail: which mechanism, where in the chain, how far off the answer. A head-to-head comparison needs the same problems, the same prompt and the same number of attempts for every model. That controlled run is the next ZORA study.

We coded each response from its author's Where/What/Why/Impact critique, not from the raw transcript. Codes are multi-label: a response that anchors on a bad value and then claims to have verified it carries both codes. Where a critique was ambiguous we coded only what it states. The six mechanisms are defined in Appendix A, and every response's codes are listed in Appendix B.

2How wrong, and in what way

Figure 2. How far each final answer missed

	GPT-5.512 responses	GPT-5.41 response	GPT-5.38 responses	Claude Opus 4.88 responses
Within 5–10%	2	–	–	4
10–50% off	1	1	–	3
50–100% off	2	–	–	1
Over 100% off or wrong sign	2	–	–	–
Wrong functional form	2	–	–	–
No numerical answer	3	–	1	–
Final value not in the record	–	–	7	–

One square per failed response, binned by the relative distance between its final answer and the verified key. GPT-5.3's critiques record the reasoning in passes 2–8 but not their final tuples. We show bands rather than exact errors because several of these wrong answers are already public, and an exact error beside a public wrong answer would give away the key.

Of the 16 responses that produced a number, 6 missed by less than 10%. These are structurally wrong answers that land numerically close: a five-subinterval quadrature with Simpson's weights (8.6% off), a sign error in a forcing term (5.4%), and four of the replica runs. The 5% acceptance rule exists because this region is where grading becomes a judgment call. A tolerance band is part of the measurement, not a neutral setting.

At the other end, two GPT-5.5 answers were off by more than 100% or had the wrong sign, and two had the wrong functional form entirely: a scaling exponent and a polynomial whose coefficients do not follow from the system.

Figure 3. Failure mechanisms by model

	GPT-5.512 responses	GPT-5.41 response	GPT-5.38 responses	Claude Opus 4.88 responses
Unverified anchoring	–	–	7	3
False precision and invented structure	2	–	5	5
Constraint misreading	3	–	3	–
Specification drift	2	–	1	3
Template substitution	2	1	–	–
Premature abstention	3	–	–	–

Squares count responses showing each mechanism; one response can show several, so columns sum to more than their response counts. Column totals differ because the records differ (see Figure 1). Compare shapes, not heights.

Single-response records and multi-attempt records fail differently. 8 of 12 GPT-5.5 failures are errors of framing (a misread constraint, a substituted template, an abstention), committed before any real computation. The multi-attempt records are dominated by errors of verification: values adopted unchecked, exactness claimed and not earned. Part of that contrast is the protocol. A single response cannot anchor across passes, and only a long record can show a model repeating itself. We return to what can and cannot be concluded from it in the section on model signatures.

3Model pathologies

3.1 Logic anchoring and error propagation

The VWAP benchmark liquidates a long position after an adverse liquidity shock. Order flow raises network load, load sets a latency, and a circuit breaker switches the favourable execution rate partway through that latency window. The breaker fires when the unaffected price recovers to a set level:

S(t)=S_0-3e^{-5t}-1.5e^{-10t}-0.5e^{-20t},\qquad S(t_c)=S_0-4.1.

Substituting

x=e^{-5t}

turns this into a quartic whose unique root in

(0,1]

fixes the regime-switch time:

f(x)=0.5x^4+1.5x^2+3x-4.1=0,\qquad t_c=-\frac{\ln x_c}{5}.

Most of what follows step 6 of 18 depends on

t_c

: the fill ratio

p

, the three price integrals

J(a,r)

, the conditional VWAP, impact, fee, offset and realised P/L.

Figure 4. Eight passes, one wall

First recorded error
A further recorded error
Depends on an earlier error
No error recorded
Not reached

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
Pass 1	No error recorded	No error recorded	No error recorded	No error recorded	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	Not reached	×First recorded error
Pass 2	No error recorded	No error recorded	No error recorded	No error recorded	No error recorded	×First recorded error	No error recorded	Depends on an earlier error	Depends on an earlier error	No error recorded	!A further recorded error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error
Pass 3	No error recorded	No error recorded	No error recorded	No error recorded	No error recorded	×First recorded error	No error recorded	Depends on an earlier error	Depends on an earlier error	No error recorded	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	!A further recorded error	Depends on an earlier error
Pass 4	No error recorded	No error recorded	No error recorded	No error recorded	No error recorded	×First recorded error	No error recorded	Depends on an earlier error	Depends on an earlier error	No error recorded	!A further recorded error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error
Pass 5	No error recorded	No error recorded	No error recorded	No error recorded	No error recorded	×First recorded error	No error recorded	Depends on an earlier error	Depends on an earlier error	No error recorded	!A further recorded error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error
Pass 6	No error recorded	No error recorded	No error recorded	No error recorded	No error recorded	×First recorded error	No error recorded	Depends on an earlier error	Depends on an earlier error	No error recorded	Depends on an earlier error	!A further recorded error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error
Pass 7	No error recorded	No error recorded	No error recorded	No error recorded	No error recorded	×First recorded error	No error recorded	Depends on an earlier error	Depends on an earlier error	No error recorded	Depends on an earlier error	!A further recorded error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error
Pass 8	No error recorded	No error recorded	No error recorded	No error recorded	No error recorded	×First recorded error	No error recorded	Depends on an earlier error	Depends on an earlier error	No error recorded	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	Depends on an earlier error	!A further recorded error

Solution step (1–18)

Pass 1. Establishes $\rho<1$ , then declares UNSTABLE by reading the step-18 checks as an instability rule.
Pass 2. Accepts $x_c\approx0.88498$ unchecked; its $J$ -values do not follow from its own formula.
Pass 3. Treats the rough root as exact; rounds executed shares despite the instruction not to.
Pass 4. Claims high precision for an $(x_c,t_c)$ pair that violates $t_c=-\ln x_c/5$ .
Pass 5. Sees the $J$ -values are too small and nudges them up instead of recomputing.
Pass 6. Invents an “internal balance” and rejects an admissible VWAP as too negative.
Pass 7. Declares upstream values final because earlier passes repeated them.
Pass 8. Claims exactness and treats passing the inequality checks as proof.

Each row is one GPT-5.3 pass over the 18-step solution. Steps 7 and 10 stay clear because they do not depend on the root. A step marked “no error recorded” was not criticised; it was not independently re-checked.

Show the numbers

Pass	What the critique records
Pass 1	Establishes $\rho<1$ , then declares UNSTABLE by reading the step-18 checks as an instability rule.
Pass 2	Accepts $x_c\approx0.88498$ unchecked; its $J$ -values do not follow from its own formula.
Pass 3	Treats the rough root as exact; rounds executed shares despite the instruction not to.
Pass 4	Claims high precision for an $(x_c,t_c)$ pair that violates $t_c=-\ln x_c/5$ .
Pass 5	Sees the $J$ -values are too small and nudges them up instead of recomputing.
Pass 6	Invents an “internal balance” and rejects an admissible VWAP as too negative.
Pass 7	Declares upstream values final because earlier passes repeated them.
Pass 8	Claims exactness and treats passing the inequality checks as proof.

Pass 2 accepts

x_c\approx0.88498

without substituting it back. One line of arithmetic would have shown that

f(0.88498)\approx0.0364\neq0

; the admissible root is

x_c=0.879793\ldots

. Passes 3 to 8 inherit the value. Pass 7 makes the mechanism explicit: it declares the upstream quantities final because earlier passes repeated them. The conversation carries values forward, but not whether anyone checked them.

How much damage does the root do on its own? Re-running the verified chain with only

x_c

replaced gives an exact answer. Because

t_c=-\ln x_c/5

\frac{\delta t_c}{t_c}=\frac{1}{\ln x_c}\,\frac{\delta x_c}{x_c}\approx-7.81\,\frac{\delta x_c}{x_c},

so a

+0.59\%

root error becomes a

-4.59\%

error in the switch time. From there it is damped, not amplified. The switch time only moves the boundary between two execution rates inside a short latency window, so the fill ratio moves by

0.28\%

, the VWAP offset by

0.027\%

and the P/L by

0.26\%

Figure 5. How the anchored root propagates

Root

x_c

(the input error)+0.59%

Circuit-breaker time

t_c

−4.59%

Temporary impact

I

−0.41%

Latency mitigation

M_{\mathrm{latency}}

−0.34%

Fill ratio

p

−0.28%

Realised P/L+0.26%

Unaffected VWAP offset−0.027%

Net offset

\Delta_{\mathrm{net}}

−0.019%

Signed relative error of each quantity when GPT-5.3's root replaces the true one and everything else is computed exactly. Logarithmic scale. Only the switch time crosses the 1% line.

Show the numbers

Quantity	Relative error
Root $x_c$ (the input error)	+0.5895%
Circuit-breaker time $t_c$	−4.5898%
Temporary impact $I$	−0.4078%
Latency mitigation $M_{\mathrm{latency}}$	−0.3425%
Fill ratio $p$	−0.2823%
Realised P/L	+0.2631%
Unaffected VWAP offset	−0.0266%
Net offset $\Delta_{\mathrm{net}}$	−0.0192%

This has two consequences, and both bear on verification.

First, the task's five mandatory checks,

0<t_c<\tau

0<p<1

-5<S^{(0)}_{\mathrm{VWAP}}-S_0<0

\Delta_{\mathrm{net}}<S^{(0)}_{\mathrm{VWAP}}-S_0

and

P/L<0

, all pass with the anchored root. Pass 8 treated passing them as proof of correctness. They are necessary conditions, and a value 0.6% off satisfies every one.

Second, the root cannot explain the size of the errors the critiques record. Pass 6 reports a VWAP offset near

-4.35

; the verified offset is about 7% more negative, and the root accounts for 0.03 percentage points of that. The rest comes from the three

J

-integrals, which, in the reviewer's words, “do not follow from the displayed piecewise formula, even using its own inputs.” The model spent its checking effort on plausibility and never recomputed the one quantity that was badly wrong.

Claude Opus 4.8's runs on the replicas problem show the same habit in exact arithmetic. Run 1 produced an intermediate probability greater than 1 and continued. Run 4 presented contradictory fractions, met an impossible divisibility claim, and substituted a new final value. Run 6 reported a verified solution to its reduced 4×4 system whose residuals are nonzero in all four rows. Run 7 replaced a derived fraction with 459/515 after an unshown “re-examination”.

3.2 Misapplication of constraints

The VWAP prompt contains exactly one termination rule: if

\rho=L_{\mathrm{eff}}/C\ge1

the answer is UNSTABLE. Pass 1 established

\rho<1

(here

\rho=984.42/1200\approx0.820

), then answered UNSTABLE anyway. It reasoned that the mandatory consistency checks would fail, and added an “economic instability” the prompt never defines. The prompt says what a failed check means: the derivation is wrong and must be revisited. The model turned a diagnostic about its own work into a statement about the market, and never computed the integrals it predicted would fail.

The same confusion between a rule and the conditions under which it applies recurs in three GPT-5.5 responses:

A boundary that is not a root

In the Floquet problem the model solves

4\mu(T)^3-27\eta(T)^2=0

inside the admissible interval and reports the interior root near

5.71\,\mathrm{h}

as the first loss of the left well. But on the reported branch

\sigma_J(T)\to+\infty

T\downarrow T_0

, so

\mu(T)=\mu_0-a\,\sigma_J(T)\to-\infty

and the left basin never exists at onset. The answer sits at the endpoint, which a root search over the open interval cannot find.

Operations that do not commute

The projective-response problem requires projecting onto monomials invariant under

\boldsymbol\epsilon\mapsto e^{i\alpha}\boldsymbol\epsilon

before averaging over

SO(3)

. The fifth recorder channel contains

\mathbf X=\operatorname{Re}\boldsymbol\epsilon

, which is not invariant. Averaging first keeps it, and the response flips sign.

A rule outside its domain

Composite Simpson-

1/3

needs an even number of subintervals. Six nodes give five. The model applied the weights

(1,4,2,4,2,1)

anyway, where the stated construction averages the two admissible

1/3+3/8

splits, with weights proportional to

(8.5,29.5,22,22,29.5,8.5)

In each case the model keeps the form of a rule and drops its scope. Nothing in the response is false in isolation; it is applied where it does not hold.

3.3 False precision and hallucinated structure

In pass 6 the model rejected a VWAP offset near

-4.35

as “too negative” by appeal to an internal balance condition that appears nowhere in the prompt. The only admissibility condition is

-5<S^{(0)}_{\mathrm{VWAP}}-S_0<0

, which

-4.35

satisfies, and the verified value is more negative still. The invented constraint was unnecessary, and it pointed the wrong way.

False precision is the textual form of the same failure. Pass 4 described its solution as high-precision while quoting a root and switch time that do not satisfy

t_c=-\ln x_c/5

. Pass 8 claimed exact intermediate logic and end-only rounding while reusing the invalid root. Claude Opus 4.8's run 6 claimed a verification its own displayed system contradicts. Run 5 printed the quantity “160133/183eastward” and moved on.

In the physics problems the same failure takes the form of terms the problem does not contain:

A factor that is not there

In the anyon-collider problem, GPT-5.5 asserted that the primitive term is proportional to

I_0\int_0^\infty dt\,t\,[\pi T/\sinh(\pi T t)]^{2\delta}

. After contact terms are subtracted, the retained thermal asymptotic scales as

T^{s_0-1}

with

s_0=2\delta

and carries no explicit

I_0t

. The extra factor shifts the scaling exponent and the encoded answer with it.

A normal form asserted, not derived

In the adaptive-oscillator problem the response wrote

d_\star/d_\star^{(0)}=1+\alpha+\tfrac{\alpha^2}{2}+4\eta^2-\eta\chi+O_3

without deriving the fold conditions. The cusp must come from the averaged branch function

B

: solve

B_{xx}=0

for the moving minimizer, then evaluate

\delta=x-B/B_x

. That calculation renormalizes every coefficient. The response's

4\eta^2

has no source.

Where asymptotic analysis was required

Two of the records fail at the step where matched asymptotics should start. In the hybrid-plasmon problem the endpoint is resolved by the inner balance

\beta=\sqrt{\delta y}

, which turns the stability boundary into a moving-minimizer problem with a unique series. GPT-5.5 declared the coefficient undetermined instead. In the oscillator problem above, the same moving-minimizer step is the one the response skipped.

A third problem, singularly perturbed advection–diffusion

\frac{d}{dX}\bigl(\epsilon T'-u(X)T\bigr)=Q\,\delta(X)

with flow directed away from an interior interface, has no model record yet. Its trap is the classic one: treating the prescribed endpoint temperatures as the outer solution. With outward flow, the boundary layers sit at the outflow endpoints, and the interior is fixed by continuity and the flux jump, not by the boundary values. It enters the controlled study.

3.4 Premature abstention

3 of the 12 GPT-5.5 failures produced no number. Each response declared the problem underdetermined, and in each case the prompt named the construction that determines the missing object.

Hybrid plasmon

“The full dissipative stability condition is absent, so

p_8

is not determined.” The near-threshold framework fixes it: substitute

\Gamma=(1-\delta)/\sqrt2

and

\beta=\sqrt{\delta y}

, and the threshold function's minimizer determines the series uniquely.

Soft-memory detector

“The

O(\Omega)

calibration and the

W_+

projection are unspecified.” The prompt fixes

W_+

as the helicity-

+

component of the radiative field, so its projection is

\varepsilon_+^\mu

. The calibration is the particle-wise timing deformation of the leading memory.

Cherenkov counts

“The collection kernel

P_a(z,\lambda)

is missing.” The prompt asks for the leading diffusion solution with an extrapolated-zero boundary. That means the half-space Green function, an image source, and the transport-length shift of the Cherenkov source.

Each of these refusals came instead of a long but standard derivation. An abstention reads as cautious, but here it was the wrong answer.

3.5 Template substitution

The magnetar problem asks for the gravitational-wave energy of a neutron star whose deformation axis wobbles at

\theta_w=20^\circ

to its spin. GPT-5.4 used the textbook orthogonal-rotator luminosity,

L_{\mathrm{GW}}\propto32\sin^4\theta_w

, which radiates only at

2\Omega

. Writing out the deformation axis

\mathbf n(t)=(\sin\theta_w\cos\Omega t,\ \sin\theta_w\sin\Omega t,\ \cos\theta_w)

shows that the

n_xn_z

and

n_yn_z

components of the inertia tensor radiate at

\Omega

as well. The angular factor becomes

K(\theta_w)=2\sin^2\theta_w\left(16\sin^2\theta_w+\cos^2\theta_w\right),

and the share of power the template discards is

\cos^2\theta_w/(16\sin^2\theta_w+\cos^2\theta_w)

. That share is 32.1% at

20^\circ

. It vanishes at

\theta_w=90^\circ

Figure 6. What the orthogonal-rotator template leaves out

Share of gravitational-wave power radiated at the spin frequency Ω, which the 2Ω-only template drops, as a function of wobble angle. The template is exact only at 90°.

Show the numbers

Wobble angle	Share of power at Ω
0°	100.0%
10°	66.8%
20°	32.1%
30°	15.8%
45°	5.9%
60°	2.0%
75°	0.4%
90°	0.0%

That is why template substitution survives: the standard case validates the template, so checking it against the textbook confirms it. The two other cases have the same shape.

In the soliton problem, GPT-5.5 used the textbook reheating condition

H_R=\Gamma

with the total charge-loss rate. The problem defines decay through the fractional rate,

\epsilon=\left.(\Gamma_Q/Q)/H\right|_{T_{\mathrm{eq}}}

with

H_R=\epsilon H_{\mathrm{eq}}

, and the slip roughly doubles the reheating temperature.

In the turbine-blade problem, it correctly remapped the film-cooling effectiveness to the coolant inlet temperature, then wrote the wall balance

T_w=\frac{(B+r)T_h+T_{c,\mathrm{exit}}}{1+B+r}

in the textbook exit-referenced form, which counts the coolant warming twice.

3.6 Building the wrong process

The replicas problem describes a recovery process for five replicas, with round types that alternate under a rule and a fatigue flag that remembers the previous outcome. The state that makes it Markovian has three parts: replica count, round type and flag. Three of Claude Opus 4.8's runs built the wrong process from a correct reading. Run 2 subtracted complementary probabilities wrongly (13/30 for a stay probability of 8/15). Run 3 misapplied the unordered-pair factor (3/20 for 1/20). Run 8 made both errors at once.

Figure 7. Eight independent runs, eight answers

28153647

Final values reported by Claude Opus 4.8 on the replicas problem, each an exact fraction. The verified key is withheld. No two runs agree.

Show the numbers

Run	Reported fraction	Value
1	3271968/4057763	0.8063
2	7816/20175	0.3874
3	85403/97244	0.8782
4	56135/63248	0.8875
5	85313/101920	0.8371
6	25058392/28378305	0.8830
7	459/515	0.8913
8	4474/9317	0.4802

The spread matters for two reasons. Majority voting cannot rescue a distribution with no mode. The disagreement itself, though, is a free and reliable signal that something is wrong. And four runs landed between 5.1% and 6.5% of the key. Under exact grading all eight fail. Under a 10% tolerance, four would pass, Pass@8 would be 4/8, and the problem would no longer clear the acceptance bar.

4Model signatures, not rankings

Because every record is conditioned on failure and no two models saw the same problem, we cannot say any model outperforms another. The records do differ in kind, and the differences line up with how each model was run. We give them as hypotheses that fit the record, not as measurements of model internals.

Single responses fail at the framing

8 of 12 GPT-5.5 failures are committed before real computation begins: a template chosen, a rule's scope dropped, a problem declared unsolvable. A single forward pass commits to a framework early, and nothing in the response revisits that choice once the algebra starts.

Long conversations remember values, not provenance

GPT-5.3's eight passes share a context. Each pass can see the previous root but not whether it was ever checked, so repetition looks like confirmation. This is the memoryless state tracking the anchoring pattern implies: the model carries the state forward, and drops its verification status.

Independent runs fail at bookkeeping

Claude Opus 4.8's errors cluster in exact rational elimination and transition tables, long arithmetic carried in text. The runs disagree with each other, and a response that claims verification can still be contradicted by its own displayed system. This is exactly where executed computation, rather than computation narrated in text, would change outcomes. The records do not show whether tools were available.

Horizon length is not the binding constraint

The VWAP chain is 18 steps long, but no pass failed for running out of horizon: seven of eight fell at step 6 and stayed there. What limits long-horizon performance here is not how far a model can reason, but whether it checks a value before building on it.

None of the critiques records an executed check; every verification they mention is a claim in text. The controlled study separates model from protocol: the same prompts across six models, eight independent attempts each.

5What this means for post-training

Reward the check, not the claim. Words like “verified”, “exact” and “final correct” appear throughout these failed responses. A reward model that reads them is rewarding verification theatre. Verifiers should execute checks: substitute the root, recompute the integral, take the residual of the displayed system.
Grade the steps, or grade exactly. An anchored root that moves the final P/L by 0.26% is invisible to any outcome grader with 1% tolerance, so a tolerance-based outcome reward cannot teach the habit that matters. Step-level rewards on intermediate quantities can.
Necessary checks are not terminal rewards. Consistency inequalities pass for wrong values. Environments that treat “all checks pass” as success teach models to stop at plausibility.
Penalise false indeterminacy, not abstention. If hedging is rewarded as honesty, specification-complete problems teach models to bail. Training needs paired items: problems where the answer is fixed and abstaining scores zero, beside genuinely underdetermined ones where abstaining is correct. That way models learn calibration rather than refusal.
Revise from the first invalid node. Self-correction data should reward recomputing from the earliest bad value. Pass 5 noticed its integrals were too small and patched the outputs, which looks like correction and fixes nothing.
Perturb the canonical case. Template substitution survives because the textbook case confirms the template. Training and evaluation problems should sit close to canonical setups but off them: a 20° wobble, not a 90° one.
Use disagreement. Eight runs with eight answers is a free uncertainty signal. Route it to a tool or to an abstention, rather than to a vote.

6Limitations

The records are conditioned on failure, from four models tested through whatever interface each author had, including a consumer subscription in one case. We coded the authors' critiques, not full transcripts. The failure-mechanism codes are our own and are listed in full so they can be disputed. Problems named in this report are retired from held-out use.

Get the five-problem sample

Five problems from this report as a PDF: the full problem statements, each with the documented failure of the frontier model it was tested against. Enter your work email and the download starts.

Or talk to us about the full set Back to ZORA

Citation

@misc{vedron2026myopic,
  title  = {Myopic points: how frontier models fail expert STEM problems},
  author = {{Vedron ZORA team}},
  year   = {2026},
  month  = {oct},
  url    = {https://vedron.ai/zora/reasoning-limits}
}

AAppendix A: the codebook

Mechanism	Definition	Example from the record
Unverified anchoring	An intermediate value is adopted without the check that would expose it, then carried forward.	A quartic root accepted without substituting it back, reused for seven passes.
False precision and invented structure	The response adds a term, condition or coefficient the problem does not contain, or claims an exactness or verification it did not perform.	An “internal balance” condition used to reject an admissible VWAP.
Constraint misreading	A stated rule or validity condition is applied to the wrong object, outside its scope, or not at all.	A failed consistency check read as a market-instability verdict.
Specification drift	A stated definition, formula or process is transcribed or constructed incorrectly.	An empirical exponent moved from one factor onto a ratio of two.
Template substitution	A familiar special-case result replaces the derivation the stated, more general setup requires.	The orthogonal-rotator formula applied at a 20° wobble.
Premature abstention	The response declares a fully specified problem underdetermined and returns no answer.	A detector kernel called “missing” when the prompt names its construction.

BAppendix B: every coded response

Problem	Field	Model	Response	Mechanisms	Distance from key
Cooling sensitivity of a turbine blade	Aerospace heat transfer	GPT-5.5	Response	Template substitution	50–100% off
First loss of a Floquet landscape well	Optics	GPT-5.5	Response	Constraint misreading	Over 100% off or wrong sign
A dissipative hybrid-plasmon endpoint coefficient	Condensed matter	GPT-5.5	Response	Premature abstention	No numerical answer
Projective enantiomeric response	Optics	GPT-5.5	Response	Constraint misreading	Over 100% off or wrong sign
Maximum length of a slurry pipeline	Chemical engineering	GPT-5.5	Response	Specification drift	10–50% off
Laser-driven proton focusing response	Nuclear & particle	GPT-5.5	Response	Constraint misreading	Within 5–10%
Radial quantum-response stationarity	Quantum mechanics	GPT-5.5	Response	Specification drift	Within 5–10%
Scaling in a fractional-quantum-Hall anyon collider	Condensed matter	GPT-5.5	Response	False precision and invented structure	Wrong functional form
Soliton domination and induced gravitational waves	Astrophysics & cosmology	GPT-5.5	Response	Template substitution	50–100% off
Cusp shift of an adaptive oscillator	Applied mathematics	GPT-5.5	Response	False precision and invented structure	Wrong functional form
A retarded soft-memory waveform at a real detector	Quantum field theory	GPT-5.5	Response	Premature abstention	No numerical answer
Cherenkov light counted through tissue	Medical physics	GPT-5.5	Response	Premature abstention	No numerical answer
The gravitational-wave budget of a wobbling magnetar	Astrophysics & cosmology	GPT-5.4	Response	Template substitution	10–50% off
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 1	Constraint misreading; False precision and invented structure	No numerical answer
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 2	Unverified anchoring	Final value not in the record
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 3	Unverified anchoring; Specification drift	Final value not in the record
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 4	Unverified anchoring; False precision and invented structure	Final value not in the record
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 5	Unverified anchoring	Final value not in the record
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 6	Unverified anchoring; False precision and invented structure; Constraint misreading	Final value not in the record
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 7	Unverified anchoring; False precision and invented structure	Final value not in the record
Adversarial latency and a conditional VWAP	Execution modelling	GPT-5.3	Pass 8	Unverified anchoring; False precision and invented structure; Constraint misreading	Final value not in the record
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 1	Unverified anchoring; False precision and invented structure	10–50% off
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 2	Specification drift	50–100% off
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 3	Specification drift	Within 5–10%
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 4	Unverified anchoring; False precision and invented structure	Within 5–10%
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 5	False precision and invented structure	10–50% off
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 6	Unverified anchoring; False precision and invented structure	Within 5–10%
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 7	False precision and invented structure	Within 5–10%
Five replicas, two round types and a fatigue flag	Distributed systems	Claude Opus 4.8	Run 8	Specification drift	10–50% off