Italian cybersecurity firm ISGroup fed 1.24 billion tokens across 12,000 model requests into large language models targeting a decade-hardened platform. The AI discovered 29 confirmed, real vulnerabilities. This demonstrates that LLMs can now surface legitimate security flaws in production-grade software.
TL;DR: ISGroup, an Italian cybersecurity company, consumed 1.24 billion tokens across 12,000 model requests against a platform hardened by a decade of security testing. The large language models found 29 confirmed, real vulnerabilities, proving that AI can surface legitimate security flaws in production-grade software.
What Did ISGroup’s Large-Scale LLM Security Study Actually Test?
ISGroup’s study tested whether large language models can independently discover real, exploitable vulnerabilities in software that has already undergone extensive professional security testing. The target was not a toy application or a deliberately weakened system. It was a platform hardened by a decade of continuous security assessments, penetration tests, and remediation cycles. The study, titled “What Can an Attacker Find With an LLM,” was published on July 31, 2026, by the Verona-based firm.
The researchers wanted to answer a simple question. Can an attacker armed with an LLM find bugs that professional human testers missed? To find out, they gave the models access to the platform’s source code and documentation, then let them analyze it at scale. The experiment involved 12,000 separate model requests, each probing different aspects of the system for weaknesses.
This was not a CTF challenge. It was real software.
The methodology focused on replicating what a determined adversary might do if they obtained access to a codebase. Instead of relying on automated scanners that pattern-match against known vulnerability databases, the LLMs reasoned about the code. They identified logic flaws, data flow issues, and edge cases that traditional static analysis tools routinely overlook.
How Many Real Vulnerabilities Did the LLMs Discover?
The LLMs discovered 29 confirmed, real vulnerabilities in the hardened platform. These were not theoretical issues or false positives flagged for manual review. Each vulnerability was independently verified by ISGroup’s security researchers, who confirmed that the flaws were genuine and exploitable.
The significance of this number cannot be understated. The target platform had already been subjected to a decade of professional penetration testing. Human experts had spent years probing its attack surface, fixing known issues, and strengthening its defenses. Despite that history, the AI found nearly three dozen previously unknown vulnerabilities that all prior testing had missed.
That is a striking result for automated security research.
The vulnerabilities spanned multiple categories, reflecting the models’ ability to reason about different classes of security defects. Rather than finding a single cluster of similar bugs, the AI surfaced a diverse set of issues across the codebase. This suggests that LLMs do not simply pattern-match against common vulnerability types. They can identify context-specific flaws rooted in business logic.
ISGroup verified each finding before counting it. No unconfirmed reports were included in the final tally.
What Does 1.24 Billion Tokens Reveal About AI Security Costs?
The study consumed 1.24 billion tokens across 12,000 model requests, revealing that AI-driven vulnerability discovery is a resource-intensive process. That token volume reflects the depth of analysis required to find real bugs in production software rather than simple test cases or code snippets.
Breaking down the numbers gives a clearer picture. With 12,000 requests and 29 confirmed vulnerabilities, the study averaged roughly 414 model requests per finding. Each confirmed vulnerability required extensive back-and-forth analysis, code review, hypothesis testing, and verification cycles. This is not a cheap or fast process.
The cost structure differs from traditional penetration testing. Human testers bill by the hour or by the engagement, and their time is finite. LLMs bill by the token, and they can run continuously. However, the sheer volume of tokens consumed in this study demonstrates that meaningful AI security analysis is not a low-cost endeavor. It requires serious computational investment.
At current API pricing, 1.24 billion tokens could cost anywhere from several hundred to several thousand USD depending on the model used.
The token consumption also reveals something about how LLMs approach code analysis. They do not simply scan and flag. They read, reason, hypothesize, and iterate. Each of the 12,000 requests represented a discrete analytical step in a larger investigation, building on previous context to narrow down potential issues.
Which LLM Models Were Used in the ISGroup Study?
ISGroup has not publicly disclosed the specific LLM models used in the study, but the experiment’s scale and methodology provide important context about the types of models required for this kind of work. The study involved 12,000 model requests consuming 1.24 billion tokens, which indicates the use of large-scale frontier models capable of handling extended context windows and complex reasoning tasks.
Security code analysis demands models that can maintain coherence over long codebases. A model must track data flow across multiple files, understand how user input propagates through different application layers, and reason about the interaction between components. Smaller or older models typically lack the reasoning depth needed for this caliber of analysis.
The token volume per request also matters. With 1.24 billion tokens across 12,000 requests, the average request consumed roughly 103,000 tokens. This suggests the models were processing substantial chunks of code and context in each interaction, far exceeding what budget-tier models can handle in a single call.
Frontier models from providers like OpenAI, Anthropic, and Google are the likely candidates for this type of work.
The study’s success in finding 29 real vulnerabilities also implies that the models used possessed strong code comprehension abilities. They needed to understand multiple programming languages, frameworks, and architectural patterns. They also needed to distinguish between intentional design choices and actual security defects, a task that requires sophisticated contextual reasoning.
How Does AI-Found Vulnerabilities Compare to Traditional Penetration Testing?
AI-found vulnerabilities differ from traditional penetration testing results in both methodology and the types of flaws discovered. Traditional penetration testing relies on human experts who use their experience, intuition, and tooling to probe a system’s attack surface over a fixed timeframe. ISGroup’s study showed that LLMs can complement this process by finding bugs that human testers missed entirely.
The 29 vulnerabilities were discovered in a platform that had already undergone a decade of professional security testing. This means human experts had repeatedly examined the same codebase without finding these flaws. The AI approached the problem differently, analyzing the code systematically and without the cognitive biases or time constraints that affect human testers.
Human testers excel at understanding business context and chaining exploits. AI excels at exhaustive pattern analysis.
The findings suggest a hybrid approach may be the most effective strategy. Human penetration testers bring deep domain expertise, creative thinking, and the ability to understand complex attack chains. LLMs bring tireless analytical capacity, the ability to process enormous codebases, and a fresh perspective unburdened by prior assumptions about what is safe.
The cost comparison is nuanced. A single penetration testing engagement might cost $15,000 to $50,000 depending on scope. The ISGroup study’s token consumption represents a comparable investment, but the AI can run continuously and re-analyze code after every change. This makes AI analysis more suitable for continuous security assessment rather than point-in-time audits.
What Types of Security Flaws Did the AI Models Identify?
The ISGroup study confirmed 29 real vulnerabilities across a platform hardened by over a decade of professional penetration testing. The flaws spanned multiple categories, including SQL injection, broken authentication, path traversal, and insecure direct object references. These are not theoretical bug classes. They are concrete, exploitable weaknesses in production-grade code.
LLMs identified both well-known vulnerability patterns and subtle logic flaws that traditional automated scanners often miss. The models analyzed source code, configuration files, and API endpoints to detect inconsistencies that could lead to security breaches. Some vulnerabilities required understanding business logic — a capability previously associated only with skilled human auditors.
The diversity of findings is notable. The AI did not simply find one category of bug repeatedly. It surfaced a spectrum of issues across different components and attack surfaces. This suggests LLMs can generalize across vulnerability types when given sufficient context and computational resources.
- SQL injection variants in database query layers
- Authentication bypass mechanisms in session handling
- Path traversal in file upload and download modules
- Insecure direct object references in API endpoints
- Cross-site scripting in dynamic content rendering
- Privilege escalation through parameter manipulation
- Information disclosure in error handling routines
- Race conditions in concurrent transaction processing
| Vulnerability Category | Detection Rate | Severity Range |
|---|---|---|
| Injection Flaws | High | Critical to High |
| Authentication Issues | Moderate | High to Medium |
| Access Control | Moderate | High to Medium |
| Logic Flaws | Lower | Variable |
| Information Disclosure | High | Medium to Low |
Can LLMs Autonomously Exploit the Vulnerabilities They Find?
ISGroup’s research focused on vulnerability discovery, not autonomous exploitation. The LLMs demonstrated the ability to identify potential security flaws and describe how they could be exploited conceptually. However, building functional exploit code for each finding required human oversight and validation.
The models could outline attack paths and describe exploitation strategies in abstract terms. For some vulnerabilities, they generated proof-of-concept snippets that illustrated the attack vector. But turning those concepts into reliable, working exploits demanded human expertise to handle environment-specific factors.
This distinction matters enormously. Finding a vulnerability is fundamentally different from weaponizing it. The ISGroup study shows AI excels at the discovery phase — scanning code, identifying patterns, and flagging potential issues — but the gap between identification and functional exploitation remains significant.
Security professionals should view LLMs as powerful discovery assistants rather than autonomous exploitation engines. The models accelerate the reconnaissance and analysis phases dramatically. The final steps of validation and exploitation still require skilled human judgment.
What Are the Limitations of Using AI for Vulnerability Research?
The ISGroup study required 1.24 billion tokens and 12,000 model requests to find 29 vulnerabilities. That is an enormous computational investment. The cost of running such analyses at scale raises serious questions about economic feasibility for routine security audits.
LLMs also produce significant false positives. Each flagged potential vulnerability required human verification to confirm it was real and exploitable. This validation overhead offsets some of the automation benefits. Models can hallucinate plausible-looking but ultimately incorrect vulnerability reports.
Context window limitations constrain how much code the models can analyze at once. Large codebases require chunking strategies that may miss vulnerabilities spanning multiple files or components. The models sometimes lose track of complex data flows across application boundaries.
- High token consumption increases operational costs substantially
- False positives require expert human validation
- Context window limits restrict analysis of large codebases
- Business logic flaws remain challenging for automated detection
- Model outputs vary between runs, affecting reproducibility
- Integration with existing security tooling requires custom engineering
- Specialized or proprietary frameworks may confuse general-purpose models
- Privacy concerns arise when sharing source code with external AI services
How Should Security Teams Integrate LLMs Into Their Workflow?
Security teams should deploy LLMs as a complementary layer alongside existing SAST, DAST, and manual review processes. The ISGroup findings demonstrate that AI excels at identifying patterns across large codebases quickly. This makes models ideal for initial triage and prioritization before human auditors begin detailed analysis.
A practical workflow starts with feeding source code and configuration files to the LLM for broad vulnerability scanning. The model flags potential issues. Human reviewers then validate each finding, eliminating false positives and confirming exploitability. Confirmed vulnerabilities enter the standard remediation pipeline.
This approach maximizes the strengths of both AI and human analysts. Models handle volume and pattern recognition. Humans handle context, business logic, and exploitation validation. The combination produces results neither could achieve alone.
Organizations should establish clear protocols for handling AI-generated findings. Not every flagged issue warrants immediate attention. Teams need scoring systems to prioritize based on severity, exploitability, and business impact. Regular calibration of prompts and analysis parameters helps improve detection accuracy over successive iterations.
What Does This Mean for the Future of Automated Security Testing?
The ISGroup study represents a shift from theoretical AI security research to practical, measurable outcomes. Finding 29 real vulnerabilities in a hardened platform proves LLMs can deliver value in genuine security contexts. This is not a laboratory experiment with synthetic targets.
As model capabilities advance and costs decrease, AI-driven vulnerability discovery will become increasingly accessible. Smaller organizations that cannot afford regular penetration tests may benefit from AI-assisted code review. The technology democratizes aspects of security auditing previously reserved for well-funded teams.
However, attackers have access to the same tools. The same LLMs that help defenders find vulnerabilities can help attackers identify targets. This creates pressure to adopt AI-driven security faster than adversaries adopt AI-driven attacks.
The security industry must prepare for a landscape where both sides wield powerful language models. Defense strategies, incident response procedures, and vulnerability management programs all need to account for AI-accelerated threat discovery on both sides of the equation.
Frequently Asked Questions
How many vulnerabilities did ISGroup’s LLM study find?
ISGroup’s study confirmed 29 real vulnerabilities in a platform that had been hardened by over a decade of professional penetration testing. All findings were validated by human security experts to ensure they represented genuine, exploitable security flaws rather than false positives.
How many tokens and model requests did the ISGroup study use?
The study consumed 1.24 billion tokens across approximately 12,000 model requests. This massive computational scale reflects the depth of analysis required to thoroughly examine a complex, hardened platform for security vulnerabilities.
Were the vulnerabilities found by AI in real production software?
Yes, all 29 vulnerabilities were discovered in a genuine platform that had undergone professional security testing for over ten years. The findings were not synthetic or planted bugs but real flaws surviving in hardened, production-grade software.
Can AI models autonomously exploit the vulnerabilities they discover?
The ISGroup study focused on vulnerability discovery rather than autonomous exploitation. While the models could describe exploitation strategies conceptually and sometimes generate proof-of-concept snippets, building reliable working exploits required human expertise and validation.
Summary
- ISGroup’s study proves LLMs can find real vulnerabilities in hardened production software, with 29 confirmed flaws discovered using 1.24 billion tokens and 12,000 model requests.
- The vulnerabilities spanned multiple categories including SQL injection, authentication bypass, path traversal, and business logic flaws across a decade-hardened platform.
- AI excels at vulnerability discovery and pattern recognition but cannot yet autonomously weaponize findings without significant human oversight and validation.
- Security teams should integrate LLMs as a complementary analysis layer alongside existing SAST, DAST, and manual review workflows for maximum effectiveness.
- The asymmetry between offensive and defensive AI capabilities will shape the future of security testing, making adoption of AI-driven vulnerability management increasingly urgent.
The full ISGroup study is available through their official press release. Additional coverage includes reports from Neo Bulletin and KDH News.