Measurement method
How we measure, and within what limits.
The measurement conditions, the results and their limits, presented as they are.
The conditions
Two alert sets, two roles
Each alert receives a verdict, noise or attack. It is compared with an expected answer, set before the measurement and never shown to the model.
The tuning set, 60 alerts
30 noise alerts and 30 attacks, hand-written. This set was used to tune the model, so it does not measure its performance. We send it on request, so that you can run the measurement again.
Request the set of 60 alerts →The control set, 320 alerts
160 noise alerts and 160 attacks, never used for training. The published results rest on this set. It is not distributed, in order to keep its value as a control, and can be replayed in front of you in a demonstration.
Request a demonstration →The rules
Six rules, applied to every measurement
- One pass per alert, with no second try.
- An unreadable answer is counted as an error.
- The model and the prompt are identified by their fingerprint in every report.
- The model answers at temperature zero, to limit variability from one measurement to the next.
- Each percentage comes with the numbers of alerts it is drawn from.
- Every complete measurement is reported, including the less favourable ones.
The results
Results on the control set
Enterprise edition, model fine-tuned on 30 September 2026, 320 alerts.
| Measurement | Hardware | Correct verdicts |
|---|---|---|
| 13 October 2026 | processor | 86.9% |
| 10 November 2026 | graphics card | 87.8% |
Detail of the 10 November measurement: 144 noise alerts recognised out of 160, 137 attacks escalated out of 160, 23 attacks judged to be noise by the model alone, 16 noise alerts escalated needlessly, no unreadable answer, a median of 2.3 s per alert.
The measurement has a veto over the model. Two fine-tunes made since 30 September were evaluated on the same set (81.6%, then 82.2 to 85.3%). As their results were not better than those of the reference version, they were not retained. The 30 September version remains our reference.
The limits
What these results do not say
- We wrote these alerts ourselves. They do not reproduce your flow: on your own alerts, the result may differ. That is why the measurement can be run again on your side.
- Out of 320 alerts, one alert weighs 0.3 point. A smaller gap is not significant.
- These results describe the performance observed under the conditions of the protocol. They are not a guarantee that no error occurs in real conditions.