Sovereign cybersecurity, within everyone's reachSovereign security for allSOC AI on your servers · free edition Request a demoDemo

Measurement method

How we measure, and within what limits.

The measurement conditions, the results and their limits, presented as they are.

The conditions

Two alert sets, two roles

Each alert receives a verdict, noise or attack. It is compared with an expected answer, set before the measurement and never shown to the model.

On request

The tuning set, 60 alerts

30 noise alerts and 30 attacks, hand-written. This set was used to tune the model, so it does not measure its performance. We send it on request, so that you can run the measurement again.

Request the set of 60 alerts →
Reserved for control

The control set, 320 alerts

160 noise alerts and 160 attacks, never used for training. The published results rest on this set. It is not distributed, in order to keep its value as a control, and can be replayed in front of you in a demonstration.

Request a demonstration →

The rules

Six rules, applied to every measurement

  • One pass per alert, with no second try.
  • An unreadable answer is counted as an error.
  • The model and the prompt are identified by their fingerprint in every report.
  • The model answers at temperature zero, to limit variability from one measurement to the next.
  • Each percentage comes with the numbers of alerts it is drawn from.
  • Every complete measurement is reported, including the less favourable ones.

The results

Results on the control set

Enterprise edition, model fine-tuned on 30 September 2026, 320 alerts.

MeasurementHardwareCorrect verdicts
13 October 2026processor86.9%
10 November 2026graphics card87.8%

Detail of the 10 November measurement: 144 noise alerts recognised out of 160, 137 attacks escalated out of 160, 23 attacks judged to be noise by the model alone, 16 noise alerts escalated needlessly, no unreadable answer, a median of 2.3 s per alert.

The measurement has a veto over the model. Two fine-tunes made since 30 September were evaluated on the same set (81.6%, then 82.2 to 85.3%). As their results were not better than those of the reference version, they were not retained. The 30 September version remains our reference.

The limits

What these results do not say

  • We wrote these alerts ourselves. They do not reproduce your flow: on your own alerts, the result may differ. That is why the measurement can be run again on your side.
  • Out of 320 alerts, one alert weighs 0.3 point. A smaller gap is not significant.
  • These results describe the performance observed under the conditions of the protocol. They are not a guarantee that no error occurs in real conditions.

This site uses cookies. Technical cookies are required for the site to work and stay on at all times. Functional and audience-measurement cookies wait for your consent.

Customise