A Fix Is Not a Verdict: How Apex Verifies Security Patches
Fix Review found 97 issue-category entries in 62 of 193 fix pull requests submitted by clients. Here is why a passing build is not enough to close a security finding.
Most security fixes follow a familiar path: change a few lines, add a regression test, and wait for CI to turn green. The harder question comes next. Is the vulnerability actually gone?
A green build proves that the code compiles, and the new test covers the reported case. A sibling caller may still reach the same sink. The check may run after a side effect, or the patch may create a different weakness nearby.
That gap was visible in a three-month production snapshot. Apex reviewed 193 client fix pull requests that reached a durable verdict, and 62 contained at least one issue. Fix Review recorded 97 issue-category entries. Of those entries, 63 showed that the original bug was still reachable, while 34 described a new security problem introduced by the patch.
97
issue-category entries across 62 fix pull requests
Why ordinary patch review stops too early
Engineers usually start with the evidence closest to the change: the diff, the new test, and CI. That makes sense, but security bugs rarely stay inside the edited lines.
Consider a guard added to one request path. The reported input now fails, the regression test passes, and the change looks ready. If a sibling route reaches the same operation without that guard, the original finding remains open. The edited function is safer, but the vulnerable behavior is still reachable.
01 · Partial coverage
The reported input is blocked, but an equivalent encoding or sibling caller remains reachable.
Required proof
Probe alternate representations and every caller that shares the vulnerable operation.
02 · Late validation
The right check runs after the mutation, commit, or external side effect.
Required proof
Establish the invariant before the first irreversible operation.
03 · Supported-flow regression
The exploit disappears because a legitimate retry, recovery, or success path also disappears.
Required proof
Exercise the named supported flows as well as negative cases.
04 · New attack surface
The change adds attacker-controlled state, fallback behavior, or a new trust transition.
Required proof
The same problem appears in less obvious forms. Validation may run after state has changed. A broad rejection may remove a legitimate recovery flow. New fallback logic may introduce attacker-controlled state. These are questions about behavior and dataflow, not just syntax.
For this analysis, we counted initiated reviews rather than unique patches. Each review is one attempt against a linked pull request or the repository's current branch. A substantive review passes the no-change preflight and returns Fix confirmed or Fix Incorrect.
- reviews run
- 1,258
- findings checked
- 644
- repositories
- 51
- organizations
- 10
Code target for each review
460 Linked pull requestExact PR checkout
798 Current branchRepository state at review time
The figures contain disclosure-safe aggregates only. Client and repository identifiers are excluded. The counts describe the client work Apex observed, not a general failure rate for security patches.
How Fix Review follows the original security argument
By the time a patch arrives, much of the useful context already exists. The original investigation identified the exploit path, attacker preconditions, and security property that failed. Fix Review carries that record into the submitted checkout instead of asking a reviewer to reconstruct the threat from the pull request description and changed lines.
The review packet includes the raw finding, validation record, expected invariant, negative cases, supported flows, and prohibited patch shapes. It also keeps the relevant code anchors, nearby state transitions, and deployment assumptions in view. When evidence is missing, the reviewer reports the gap rather than silently filling it in.
- Question
- Diff-only checkDoes the changed code look reasonable?
- Apex Fix ReviewDoes every required safety property hold?
- Evidence
- Diff-only checkChanged lines and local test results
- Apex Fix ReviewOriginal finding, validation record, fix contract, and checkout
- Scope
- Diff-only checkThe edited path
- Apex Fix ReviewEquivalent inputs, sibling paths, ordering, and supported flows
- Decision
- Diff-only checkApprove or request changes
- Apex Fix ReviewFix confirmed, Fix Incorrect, or insufficient context
Target identity matters as much as context. Pull-request mode checks the linked checkout; branch mode checks the current repository state after a merge. Combining results from those targets would make the verdict impossible to reproduce.
If the original location has not changed, Apex stops before analysis. This preflight returned Fix Pending for 154 attempts within 2.2 seconds. Those runs were skipped, not confirmed.
01 · Setup
Check out the selected PR or branch and load the evidence packet.
02 · Scope
Map the original exploit and every required safety property into the checkout.
03 · Hunt
Exercise negative cases, adjacent paths, ordering, supported flows, and deployment assumptions.
04 · Validate
Reject unsupported candidates and preserve missing evidence as uncertainty.
05 · Synthesize
Return Fix confirmed, Fix Incorrect, or insufficient context with evidence.
Fix confirmed appears only when every first-class safety requirement has supporting evidence from the reviewed checkout. Evidence that contradicts a requirement returns Fix Incorrect. If the reviewer cannot establish a requirement, the result is insufficient context rather than a pass.
What the engineer gets back
When Fix Review returns Fix Incorrect, the result separates Original finding still needs attention from Additional issue found. Each writeup points to the relevant code and describes the security condition that remains unsatisfied, giving the engineer a useful starting point for the next commit.
Time to verdict
- substantive reviews
- 1,003
- median time to verdict
- 37 min
- middle half
- 27–51 min
Evidence returned for surfaced issues
- deduplicated issue writeupsall include analysis
- 93
- include file and linespecific code location
- 70
- include a supported next fixactionable remediation
- 66
Patch review is often iterative. Of 399 findings that eventually reached Fix confirmed, 254 did so after one review, 99 after two, and 46 after three or more. These are counts of review passes, not failed patches. Any new commit makes the earlier verdict stale because the code under review has changed.
An unmerged pull request can earn Fix confirmed, but it remains Pending Merge and the finding stays in Todo. After merge, the status changes to Merged and the finding moves to Fixed.
Where Fix Review stops
Fix Review answers one bounded question: does this checkout close the known finding without adding a validated problem attributable to the patch? It does not claim that the rest of the system is secure. Changes to authentication, trust boundaries, or undocumented business rules can still require a broader review.
Complete evidence gives the team a defensible reason to close the finding. When evidence is missing, the engineer sees the remaining security condition and where to continue.