Detecting Functional Memorization in Code Language Models

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

arXiv:2606.12764v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used to generate code at scale. Meanwhile, prior work has investigated whether training data may be recoverable from model outputs, by auditing the textual overlap between training examples and model generations. Code, however, can preserve the same logic while differing substantially in syntax and structure. We here study functional memorization: the leakage of training data logic from LLM generations in ways that textual audits fail to detect. We leverage AI coding agents to generate diverse test inputs for training data functionality and evaluate whether model-generated continuations produce the same outputs. We formalize this through a counterfactual framework, comparing target models (exposed to specific code) against reference models (not exposed) and requiring functional equivalence only for the target. We instantiate this framework across 4 open-source models and explicitly filter for functions with meaningful logic in 5 programming languages. We find that 0.3-3.4% of filtered functions are counterfactually functionally memorized, i.e., reproduced with equivalent behavior in restructured code that textual metrics fail to detect. We further show that LLM-based judges offer a scalable proxy for execution-based testing, achieving a true positive rate of 68% at 1% false positive rate, and find that functional memorization is associated with semantic duplication in the training corpus.

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://arxiv.org/abs/2606.12764
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.