How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2608.01454v2 Announce Type: replace-cross Abstract: Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols. We investigate this dependency by re-evaluating representative PIDS on public datasets that meet our audit, labeling, and calibration requirements. Focusing primarily on the audited DARPA TC E3 datasets, we apply a unified protocol with temporally separated test periods and validation-only checkpoint selection and threshold calibration, and ask which architectural claims are empirically supported. We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation. On three of the four primary datasets, a simple allowlist built from training executable names and paths matches or exceeds the selected learned baselines on key operating-point metrics, suggesting that much of these systems' measured performance reflects lexical novelty rather than richer provenance modeling. Quantifying semantic signal quality through feature completeness and field entropy helps explain why several audited E3 datasets support alerting performance without reliably separating model architectures, whereas Theia combines the richest semantic signal with the clearest improvements in ranking and node-level recovery by our reference model. Overall, these findings reinforce the importance of interpreting …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2608.01454
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.