Tasks completed
Baseline10/24Sentinel P324/24
ICISELFLAB / SENTINEL
Stress-test AI agents beyond benchmark scores
ICI Sentinel examines how failures enter, propagate and survive correction across agents, memory, tools and orchestrators. It is designed to expose mechanisms that a single benchmark score cannot explain.
Core implementation, evaluation logic, benchmark design, scoring rules and proprietary methods are not disclosed. Publication of this demonstration does not grant any right to reproduce, redistribute or commercially exploit the underlying methodology, except as permitted by applicable law.
AGENTTHREATBENCH · 5 OCT 202624 CASES · GATE PASS
More legitimate tasks completed. Fewer successful attacks.
In the matched, self-run AgentThreatBench comparison, Sentinel P3 with host controls completed all 24 observed tasks and prevented all 19 labelled attacks. Qwen 7B and generation settings stayed unchanged.
Successful attacks
Baseline5/19Sentinel P30/19
Safe completions
Baseline5/24Sentinel P324/24
Per-case regressions: 048 native runsQwen2.5 7B Instruct · Q4_K_MAudit VERIFIED
132/132 available regression checks passed; one archived reporting fixture was unavailable for rerun.
Observed Development / regression data; not a fresh holdout or independent validation. The measured advantage applies to this 24-case benchmark. Generalization to unseen tasks remains untested.
01
The problem
An agent can pass familiar evaluations and still fail when evidence is incomplete, tools disagree, memory is contaminated or several agents reinforce the same wrong assumption. A correct final answer does not reveal whether the evidence chain was independent, whether tool authority was respected or whether the same failure will return after the context changes.
02
How ICI approaches it
Sentinel separates observation, inference, verification, provenance, memory and action authority. It forms causal hypotheses, attacks them with targeted counter-tests and re-runs the scenario under changed conditions. The live Trace Audit also distinguishes OBSERVED, INFERRED, VERIFIED, UNVERIFIED and CONTRADICTION states before producing an ALLOW, WARN, BLOCK or HUMAN_REVIEW tool-authority verdict.
03
Operational example
A four-agent workflow reaches consensus, but every agent ultimately depends on the same upstream source. Sentinel treats the agreement as correlated rather than independent evidence, changes source availability or authority, then tests whether the consensus survives. If the result collapses, the shared dependency becomes a retained failure mechanism for later tests.
04
Evidence discipline
The public Live Trial currently exposes server-side Trace Audit, provenance, Goal Drift, Tool Authority, rate limiting, redaction and JSON audit export. Real-Web execution is kept separate and is shown as configured only when its provider credential is present on the server. Separately, on 3 October 2026, a frozen-policy offline deterministic action-trace replay over 48 author-created held-out traces produced 48/48 correct decisions, blocked 23/23 unsafe proposed actions and preserved 25/25 benign traces. No LLM inference occurs in that replay. It is internal contract validation, not an official external benchmark or independent third-party validation. The current canonical candidate is ici-sentinel@13.1.0-rc.2 (RC2). A separate real external suite is frozen for UK AISI Inspect AI 0.3.276 / Inspect Evals 0.23.0 (AgentDojo, AgentThreatBench, b3) and WebArena-Verified 1.2.3; official external results remain unpublished until native benchmark artifacts exist.
CONTACT
Work with ICI
Research collaboration, controlled pilots and technical evaluation.