Skip to content

Nullable probe-field validation correction

The user authorized this correction after the Phase 20 benchmark exposed repeated injection probe requires a string field failures. It is a separate correction on fix-nullable-probe-validation, based on frozen benchmark branch commit daeefe7. It is not a tuning change applied to the retained baseline.

Excel's create_table.table_name is declared as anyOf: [string, null]. The old review validator checked only a top-level type: string, rejecting this field even though it accepts a string. The same check also affected oversized-probe bindings and whether a tool with only nullable string fields was eligible for a probe plan. Malformed type arrays could additionally raise an unhashable-type error. Retained errors and request schemas identify this mismatch; discarded failed responses are not recoverable and have not been reconstructed as captures.

One shared helper now classifies declared primitive types, type arrays, and nested anyOf branches for all three checks. Numeric-only, unknown, and malformed untyped fields remain ineligible. This is type classification for existing inert probe templates, not a full schema satisfiability check or expanded Docker schema support. Runtime schema validation, baseline prerequisites, probe payloads, prompts, finding identities, and JSON/SARIF schemas are unchanged.

Regression tests failed before the fix (six failures, four passing controls) and pass after it (all ten cases). They cover nullable strings, type-array syntax, nested unions, nullable containers, numeric-only unions, and malformed/untyped fields. Both binding validation and eligibility use the same cases. Reproduce:

python -m pytest tests/test_gpt_review.py -k probe_plan_union_field_types --no-cov -q
python -m pytest tests/test_gpt_review.py tests/test_phase20_measurements.py --no-cov -q
make check

Evidence is retained under artifacts/corrections/nullable-probe-validation. All 17 previously accepted captures still validate through the corrected native reviewer in offline replay (accepted-capture-check.json). This is compatibility verification, not a new detection measurement or recovery of rejected responses. The historical baseline in docs/phase20-verification.md, the corpus, approved request packet, capture ledger, all successful captures, and failed-attempt accounting remain unchanged. The benchmark's paid-capture source-identity guard rejects the corrected scanner against the original approval; its test explicitly checks this boundary. Any measurement of the corrected scanner needs separately identified preparation and the selected budget approval before new paid calls. No paid call is part of this correction's verification. The subsequently approved completion measurement captured the remaining requests; Phase 20 is now accepted.