AI and LLMs in Cyber Security: How Artificial Intelligence Is Changing Threats
A finance employee at Arup sat through an entire video call with deepfaked versions of his own CFO and colleagues. He authorized a $25.6 million transfer before anyone realized none of the people on that call were real.
How is artificial intelligence changing cyber security threats?
AI and LLMs in cyber security are reshaping threats in two directions at once: attackers now weaponize generative AI for convincing fraud, while the AI applications businesses adopt for productivity introduce entirely new vulnerability classes traditional security tools were never built to catch.
Emerging threats in cyber security now include AI-powered social engineering realistic enough to fool trained employees, and attack surfaces that live inside the model’s own reasoning process rather than in code. AI-powered attacks don’t need to break into your network the traditional way anymore, they can talk their way past a human, or talk their way past an AI system your business already trusts.
Prompt injection: the OWASP #1 risk in LLM applications
Prompt injection is ranked LLM01, the number one risk in the OWASP Top 10 for LLM Applications 2025, because it exploits a fundamental architectural weakness: language models cannot reliably distinguish trusted instructions from untrusted data they’re simply asked to process.
Prompt injection now appears in over 73% of production AI deployments assessed during security audits. Unlike a traditional software vulnerability, it isn’t a bug developers can patch once, since the model’s core function, following instructions in text, is exactly what the attack abuses. NIST has called it “generative AI’s greatest security flaw,” a framing that reflects how central this single risk has become to the entire category.
Direct vs indirect injection: the attack you can’t see coming
Direct prompt injection happens in the chat itself, instruction override, role reassignment, encoding tricks. Indirect prompt injection plants malicious instructions inside content the AI reads on your behalf, a poisoned PDF, a scraped webpage, a GitHub issue, and the user never sees the attack at all.
Indirect injection is the more dangerous variant precisely because it bypasses every UI-level safeguard your employees might otherwise catch. A malicious instruction hidden in a document summary or an email the AI processes never appears as suspicious text to a human, it only ever exists inside the model’s own reasoning, which is exactly why it’s now the dominant real-world attack vector across documented incidents.
Real CVEs: Microsoft Copilot, GitHub Copilot, and Cursor IDE exploits
Real, critical CVEs confirm this threat moved from theory to production: EchoLeak (CVE-2025-32711) against Microsoft Copilot, a GitHub Copilot remote code execution flaw (CVE-2025-53773) that put the tool into an unsafe “YOLO mode” via a hidden instruction, and Cursor IDE’s CurXecute and MCPoison vulnerabilities, both permitting code execution without user interaction.
| Product | CVSS Score |
| Microsoft Copilot | 9.3 |
| GitHub Copilot | 9.6 |
| Cursor IDE | 9.8 |
A separate February 2026 finding, RoguePilot, showed the first confirmed case of an AI coding assistant fully weaponized through natural language alone, hidden HTML comments in a GitHub Issue triggered token theft and complete repository takeover, with no malware, no exploit code, just text. A December 2025 research effort called IDEsaster tested every major AI coding assistant and found 100% of them vulnerable to a prompt-injection-to-code-execution attack chain.
Agentic AI vulnerabilities: when an attacker hijacks the agent’s goal
Agentic AI vulnerabilities get their own dedicated framework, OWASP’s Top 10 for Agentic Applications 2026, because an AI agent doesn’t just answer a question, it plans and acts across multiple steps with real system access, making a hijacked goal far more damaging than a single bad response.
ASI01, Agent Goal Hijack, is the top entry, combining prompt injection with excessive autonomy so the impact compounds across an entire autonomous task chain rather than one output. Palo Alto Networks’ Unit 42 documented real “agent session smuggling,” where a malicious remote agent injects instructions mid-session, invisible to the end user, causing data exfiltration or unauthorized tool execution. Cisco’s 2026 State of AI Security report found 83% of organizations plan to deploy agentic AI, but only 29% feel genuinely ready to secure it, a gap this specific risk class explains directly.
Why no complete fix exists, even in frontier models
Even Anthropic’s own frontier model testing shows prompt injection succeeds at meaningful rates given enough attempts: indirect injection attacks against agentic coding environments succeeded 4.7% of the time on a single try, climbing to 33.6% across 10 attempts, and 63.0% across 100 attempts.
Techniques like retrieval-augmented generation and fine-tuning were expected to reduce this risk. Research shows they don’t fully mitigate it, since the underlying architectural problem, models can’t cleanly separate instructions from data, remains unsolved regardless of how the model gets deployed. As one industry researcher put it plainly: prompt injection is the new SQL injection, and guardrails alone aren’t enough to close it.
AI-generated malware and deepfake voice attacks: the offensive side
The Arup case is the clearest proof this threat has moved from novelty to operational reality: 62% of organizations reported experiencing at least one deepfake attack in the past 12 months, and a voice clone requires just three seconds of audio to reach roughly 85% accuracy.
Deloitte projects US generative-AI-enabled fraud losses will reach $40 billion by 2027, up from $12.3 billion in 2023. Regulators are responding directly: starting August 2, 2026, Article 50 of the EU AI Act mandates disclosure and watermarking for deepfake content, with fines reaching €15 million or 3% of global turnover for non-compliance. Any business relying purely on “does the voice sound right” as a verification method is now operating against a threat that specifically defeats that exact check.
What does the current OWASP GenAI LLM Top 10 actually cover?
The full 2025 list runs LLM01 through LLM10: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption.
Prioritize based on your actual deployment, not the full list uniformly. A customer-facing chatbot with read-only data access should weight LLM01, LLM02, and LLM09 heaviest. A system with tool access to production infrastructure needs LLM06, excessive agency, and LLM05, improper output handling, added as top concerns. Treat the list as a threat-modeling checklist matched to your specific configuration, not a box-ticking exercise applied identically everywhere.
Regulatory pressure: EU AI Act, NIST, and mapping your compliance
Prompt injection and related AI risks now map to at least seven overlapping frameworks, OWASP, MITRE ATLAS, NIST AI RMF, the EU AI Act, ISO 42001, GDPR, and NIS2, with the EU AI Act’s August 2026 deadlines making compliance mapping genuinely urgent for any business operating in or serving EU customers.
NIST’s AI Risk Management Framework treats prompt injection resilience as a core measurable control, not an optional add-on. For US and UK businesses without direct EU exposure, mapping against NIST’s framework still provides the clearest, most actionable baseline, since it’s built specifically around the same architectural risks OWASP’s LLM Top 10 documents in more technical detail.
Defense in depth: the only strategy that works
No single control stops prompt injection reliably, which is why defense in depth, input sanitization, output validation, restricted tool permissions, and human approval gates on irreversible actions, remains the only strategy with genuine track record against this risk.
Restrict what any AI agent can do without explicit human confirmation, especially anything touching credentials, financial transactions, or production systems. Businesses already running structured vulnerability assessment and penetration testing programs should extend that same discipline to AI-specific red teaming, since traditional web application testing methods don’t catch prompt injection or excessive agency flaws. Cyber Security Solutions Ltd helps clients build exactly this layered approach around AI deployments already in production.
Conclusion
The gap between what AI can now convincingly fake and what your current verification process assumes is where the next incident is most likely to land. Cyber Security Solutions Ltd can help you close that gap at cybersecuritysolutionsltd.com.
FAQs
Prompt injection is an attack technique that manipulates an AI model’s behavior by embedding malicious instructions in its input, exploiting the model’s inability to reliably distinguish trusted commands from untrusted data. It’s ranked the number one risk in the OWASP Top 10 for LLM Applications.
Direct injection happens within the chat interface itself. Indirect injection plants instructions inside external content the AI processes, like a document, webpage, or email, meaning the user never sees the attack. Indirect injection is now the dominant real-world attack vector.
Yes. Critical, confirmed CVEs exist for Microsoft Copilot, GitHub Copilot, and Cursor IDE, with CVSS scores ranging from 9.3 to 9.8. A December 2025 study found 100% of tested AI coding assistants vulnerable to a prompt-injection-based code execution chain.
ASI01, Agent Goal Hijack, is the top risk in OWASP’s Top 10 for Agentic Applications 2026. It describes an AI agent’s objectives being redirected by adversarial instructions, amplified because agents act autonomously across multiple steps rather than producing a single response.
No. Even frontier models show meaningful attack success rates when tested repeatedly, one study found indirect injection succeeded 63% of the time across 100 attempts. Techniques like RAG and fine-tuning reduce risk but don’t eliminate the underlying architectural weakness.
Very real. A finance employee at Arup authorized a $25.6 million transfer after a video call with deepfaked executives. 62% of organizations reported experiencing a deepfake attack in the past year, and a voice can be cloned convincingly from just three seconds of audio.
Yes, if they serve EU customers or process EU user data. From August 2026, Article 50 requires deepfake disclosure and watermarking, with fines up to €15 million or 3% of global turnover. US and UK-only businesses should still map against NIST’s AI Risk Management Framework.
