Synapsed
At the recent AI Festival 2026 in Milan, Synapsed.ai’s own Matteo Meucci took the stage to address a critical question facing the technology industry: after a quarter-century of experience with web applications, why do we still struggle to build secure software, and how can we hope to create trustworthy AI systems? This question formed the foundation of a compelling presentation that explored the evolution from software quality to the comprehensive framework of AI trustworthiness.
Since 2023, the evolution from generative AI to agentic AI systems has marked a fundamental shift: AI systems no longer merely produce content, but act autonomously to achieve goals. Like Little Red Riding Hood in the illustration, an AI agent is given an objective and a set of initial rules, then proceeds to plan, decide, and execute actions across environments, tools, and platforms. As agentic systems increasingly operate with reduced human oversight, they become susceptible to manipulation, context confusion, and instruction hijacking, highlighting why autonomy, while powerful, introduces new and systemic trust and security challenges.
The Uncomfortable Truth: Our Struggle with Secure Software
For over 25 years, the software development industry has been on a journey to improve the security of its products. This journey has led to the evolution of the Software Development Life Cycle (SDLC), a process focused on producing high-quality software. As security became a more pressing concern, the Secure SDLC emerged, integrating security practices into every phase of development. However, as Meucci pointed out, the challenge of building truly secure software persists.
This persistent struggle raises a crucial question as we enter the age of artificial intelligence. If we have not yet perfected the art of securing traditional software, how can we confidently build AI systems that are not just secure, but truly trustworthy? The answer, as Meucci’s talk highlighted, lies in expanding our perspective beyond security alone.
If Secure Software Is Still Hard, Trustworthy AI Is Even Harder
Web application security has been around for decades:
- OWASP Top 10 started in 2003
- Secure SDLC has been discussed for over 20 years
- Millions invested in tools, frameworks, training, and automation
And yet breaches are still systemic, vulnerabilities are still recurring and security is still often bolted on after development
Now let’s look at AI systems:
- Non-deterministic behavior
- Opaque decision processes
- Data-driven logic instead of explicit code
- Continuous evolution after deployment
- Agents that can act, not just respond
If we failed to consistently secure deterministic software, why should we assume AI will be easier?
The answer is simple: We cannot rely on the same mental models, processes, or shortcuts.
We need a new paradigm.
Continuous evaluation of AI Systems
One-Shot Testing: Why Continuous Evaluation is Crucial
In the world of traditional software, testing is often seen as a phase a gate that, once passed, signals a product is ready for release. This “one-shot” testing approach is fundamentally incompatible with the dynamic and evolving nature of AI systems. An AI application that is deemed safe and secure at launch can become vulnerable and unreliable over time, not because of a code change, but due to shifts in its data, user interactions, or even the subtle evolution of the AI model itself.
AI systems in general are not static, they are in a constant state of flux, influenced by:
•Evolving Prompts: teams continuously refine prompts to improve performance, but these changes can inadvertently introduce new vulnerabilities.
•Dynamic Data Sources: AI systems that retrieve information from ever-changing knowledge bases can be exposed to new risks as data is added or modified.
•Model Updates: upgrading to a new AI model or fine-tuning an existing one can lead to unpredictable changes in behavior.
•User Behavior: users constantly test the boundaries of AI systems, discovering new ways to interact that were not anticipated by the developers.
This continuous evolution means that risk is not a static variable but a dynamic one that can accumulate over time. A security vulnerability that doesn’t exist today might emerge tomorrow. The Gemini 2.5 model response image from the presentation provides a stark example of this, where a prompt injection attack is detected and thwarted, highlighting the need for real-time monitoring and defense mechanisms.
The Role of OWASP in Building Trustworthy AI
So, how do we navigate this complex and evolving landscape? The Open Web Application Security Project (OWASP), a long-standing authority in the field of software security, has stepped up to provide guidance and resources for the age of AI. The OWASP GenAI Security Project is a global initiative dedicated to addressing the security and safety risks of generative AI. It provides a wealth of resources, including:
•The OWASP Top 10 for Large Language Model Applications: This list identifies the most critical security vulnerabilities in LLM applications, providing a roadmap for developers and security professionals.
•The OWASP Top 10 for Agentic Applications: A guide to the unique security challenges posed by autonomous AI agents.
•OWASP AI Testing Guide: A standard for the trustworthiness testing of AI systems, moving beyond traditional security testing.
•OWASP AI Maturity Assessment: A structured framework to assess and improve the maturity of companies that adopt or develop AI systems.
By leveraging the resources and community of OWASP, organisations can move beyond the limitations of traditional security practices and embrace a more holistic and continuous approach to building trustworthy AI.
The Path Forward: A Call for a New Paradigm
Matteo Meucci’s talk at the AI Festival 2026 was a powerful call to action. It was a reminder that the lessons we have learned from 25 years of web application security are both a foundation and a warning. We cannot simply apply the old rules to a new game. Building trustworthy AI requires a paradigm shift, a move away from a narrow focus on security and towards a comprehensive framework that embraces all the pillars of trustworthiness. It demands a commitment to continuous learning, continuous testing, and continuous improvement. The journey is just beginning, but with the guidance of organizations like OWASP and the collective effort of the AI community, we can build a future where AI is not just powerful, but also worthy of our trust.
REFERENCES:
- OWASP Top 10 LLM 2025 – A Synapsed Research Study
Synapsed Research
https://synapsed.ai/rd-owasp-top-10-llm-2025-a-synapsed-research-study - Trustworthy AI – Principles and Foundations
Synapsed
https://synapsed.ai/trustworthy-sdlc-a-new-paradigm-for-ai-products-development/ - OWASP AI Testing Guide
OWASP Foundation
https://owasp.org/www-project-ai-testing-guide/ - OWASP AI Maturity Assessment
OWASP Foundation
https://owasp.org/www-project-ai-maturity-assessment/ - OWASP GenAI Security Project
OWASP Foundation
https://genai.owasp.org/ - Lessons from Defending Gemini Against Indirect Prompt Injections — Chongyang Shi et al— arXiv preprint — https://arxiv.org/abs/2505.14534