Synapsed
OWASP AI Testing Guide was at the center of Synapsed’s session at OWASP AppSec EU 2026 in Vienna, where Matteo Meucci and Marco Morana presented how organizations can move from AI threat modeling to concrete testing evidence for trustworthy AI systems. The session focused on the practical use of AITG for verifying trustworthy AI products and included AI testing domains, AI threat modeling, testing evidence and demo examples.
Artificial intelligence is changing the way software is designed, developed and validated. LLM applications, RAG systems and agentic workflows are not only code. They combine prompts, models, data, external tools, memory, permissions and runtime behavior. This creates a new assurance problem: we must test not only whether the application is secure, but whether the AI system behaves safely, reliably and within its intended boundaries.
An AI application may have no obvious injection flaw, no broken access control, no insecure dependency and no traditional web vulnerability. Yet it may still be unsafe. It may follow malicious instructions hidden in external content. It may leak sensitive information. It may hallucinate critical facts. It may produce toxic content, biased outputs or unauthorized actions through an agentic workflow.
This is why secure software is not the same as trustworthy AI.
Why the OWASP AI Testing Guide matters
The OWASP AI Testing Guide provides a structured methodology for trustworthiness testing of AI and LLM-based systems. OWASP describes AITG as a practical, technology-agnostic framework that evaluates not only security threats, but also broader trustworthiness properties required for responsible and regulatory-aligned AI deployments. The guide also organizes repeatable test cases across AI Application, AI Model, AI Infrastructure and AI Data layers.
This matters because traditional application security testing is no longer enough.
Secure coding, SAST, DAST, penetration testing and secure architecture remain essential. However, an AI application can pass many traditional security checks and still fail in production. It may leak sensitive information. It may follow malicious instructions hidden in external content. It may hallucinate critical facts. It may take unauthorized actions through an agentic workflow. It may behave differently when the model, data or context changes.
In other words, security is necessary, but it is not sufficient.
Secure Software is not Trustworthy AI
One of the central ideas of the session was that AI trustworthiness depends on behavior. Code can be secure, but the model can still fail. An application can have strong authentication, validated inputs and patched dependencies, while the AI component still produces unsafe, biased, misleading or non-compliant outputs.
This is the gap between software security and AI trustworthiness.
Traditional security asks questions such as:
- Is the code vulnerable?
- Are inputs validated?
- Are access controls enforced?
- Are dependencies secure?
Trustworthy AI testing asks additional questions:
- Can the model be manipulated?
- Can external content change the behavior of the system?
- Can the system disclose sensitive data?
- Are outputs grounded, safe and verifiable?
- Are agentic actions bounded?
- Are privacy, fairness and human oversight requirements respected?
The Vienna presentation highlighted this transition clearly: security alone is not enough, one-shot testing does not work for AI systems, and organizations need continuous, risk-driven testing focused on behavior, not just code.
From AI threat modeling to testing evidence
AI threat modeling helps teams understand what can go wrong. It identifies assets, trust boundaries, abuse cases, attack paths and possible impacts across the AI architecture.
But threat modeling alone is not enough.
The next step is testing evidence.
Testing evidence shows whether the AI system is actually exposed to a given threat scenario. It makes trustworthiness measurable. It gives security teams reproducible findings, engineering teams actionable remediation, governance teams documented assurance and executives a clearer view of residual risk.
This is the operational value of the OWASP AI Testing Guide: it turns AI risks into structured tests.
In Vienna, the AITG workflow was presented as a practical bridge between threat modeling and evidence-based validation. The session agenda explicitly moved from AI threat modeling to testing evidence and then to project walkthrough and testing demo examples.
The four testing domains of AITG
The OWASP AI Testing Guide covers four major testing domains:
- AI Application Testing verifies the behavior of LLM applications, RAG systems, agents, prompts, tools and application-level controls.
- AI Model Testing evaluates model robustness, poisoning, membership inference, inversion attacks, hallucinations and alignment issues.
- AI Infrastructure Testing focuses on the platforms, pipelines, plugins, serving environments and supply-chain components that support AI systems.
- AI Data Testing examines training data, runtime data, data exposure, minimization, consent, privacy and data quality.
This structure is important because AI risks rarely belong to only one layer. A weakness may originate in data, propagate through the model, appear at the application layer and create business impact through user interaction or agentic execution.
The comprehensive AITG testing suite shown in the Vienna session mapped these risks into specific test IDs and test categories across application, model, infrastructure and data layers.
Example: testing indirect prompt injection
A concrete example discussed in the session was indirect prompt injection.
In this scenario, malicious instructions are embedded in external content that the AI system processes, such as a webpage, document, email or retrieved knowledge source. The user may simply ask the system to summarize or process that content. However, the hidden instruction attempts to manipulate the AI system, bypass guardrails, reveal information or trigger unauthorized behavior.
The test objective is not to check whether the application has a classical input validation bug. The objective is to verify whether the AI application can be indirectly manipulated by external content and whether it executes unintended instructions.
A vulnerability is confirmed when the model follows malicious instructions embedded in external content, reveals sensitive information or performs unauthorized actions. The Vienna deck includes indirect prompt injection as AITG-APP-02 and defines expected outcomes and remediation guidance for this test scenario.
This example shows why AI testing must be evidence-based. A generic prompt is not enough. Teams need a defined test objective, controlled input, expected behavior, observed output, risk evaluation and remediation guidance.
What organizations should do next
The lesson from Vienna is clear: trustworthy AI must become part of the engineering lifecycle.
Organizations adopting LLM applications, RAG pipelines or AI agents should start by creating an AI inventory and identifying which systems require structured testing. Then they should map the AI architecture, perform AI threat modeling, select relevant AITG test cases, execute tests, collect evidence and track remediation.
This approach helps connect different stakeholders:
- Security teams can validate AI-specific attack scenarios.
- AI engineers can understand failure modes and improve controls.
- Risk and compliance teams can review documented evidence.
- Executives can see where AI systems are trustworthy enough for production.
The result is a more mature AI assurance process, where testing is not an isolated activity before release, but a continuous validation practice throughout the AI lifecycle.
Conclusion
The industry spent more than twenty years building practices for secure software. Now we need to extend that discipline to AI systems.
The OWASP AI Testing Guide is an important step in this evolution. It gives organizations a common language and a practical testing structure for AI trustworthiness. It helps teams validate not only whether software is secure, but whether AI behavior is safe, reliable, private, responsible and aligned with its intended purpose.
Secure software remains essential.
But in the AI era, the real objective is broader:
Trustworthy AI must be tested, documented and continuously validated.
OWASP AI Testing Guide references:
At OWASP Global AppSec USA 2026 in San Francisco, we will go deeper with a hands-on training:
“OWASP AI Testing Guide (AITG): Enabling Trustworthy AI Through Structured Validation”
Together with Marco Morana, we will show how to apply AITG to real AI systems, moving from AI threat modeling to structured validation across security, privacy, robustness, governance and responsible AI behavior.
Come join us in San Francisco 👇
Related Synapsed blog resources:
- OWASP Top 10 LLM 2025 Synapsed Research Study
https://synapsed.ai/rd-owasp-top-10-llm-2025-a-synapsed-research-study/