AI Inside: handle with Care!

Understanding the Challenge of Fragile AI systems

AI’s influence is everywhere. Yet, behind these capabilities lies a critical and often overlooked challenge: AI systems are inherently fragile. In this article, we’ll explore the roots of AI fragility, discuss why it matters, and offer strategies for building more resilient, trustworthy systems.

Why Are We Talking About Fragile Software?

When most people think of software, they envision neat lines of code executing deterministic logic: the same input always produces the same output. Traditional software is, by design, predictable and stable. AI software, however, operates differently. Its core strength—learning from data rather than following hard-coded rules—introduces an element of uncertainty that can lead to unexpected behavior.

What’s the difference between “traditional software” and “AI software”?
Gary McGraw from BIML explains very well the core differences here [1].

We can affirm the following regarding the main differences:

• Traditional Programming (WHAT): in traditional software development, developers explicitly write code that dictates every step of a process. For each problem, there’s a defined rule or set of instructions. It’s like handing the computer a step-by-step recipe. This approach ensures a deterministic output: the same input always leads to the same, predictable result.
• Artificial Intelligence (HOW): AI, on the other hand, is less about writing explicit instructions and more about enabling the system to learn from examples. Instead of giving the computer a precise recipe, we provide it with numerous examples and let it “learn” patterns. Over time, it can recognize new situations and make predictions, much like teaching a child to identify animals by showing them countless pictures rather than describing every possible feature of a cat, dog, or bird.

A Concrete Example: Traditional Programming vs. AI

Traditional Programming (Calculator Example)
Consider building a simple calculator for addition. The logic is straightforward: for given inputs a and b, the system returns their sum a + b.

Developer’s Thought Process:

1. Prompt the user for two numbers, a and b.
2. Pass these values to an add(a, b) function that simply calculates a + b.
3. Output the result.

The following is a sample Code in a Traditional Language (e.g. Python):

Execution of the Program:

• User Input: for example a=2, b=3
• Computer Execution: 2+3=5
  No matter when or where you run this code—this year or next year, on your laptop or a server in the cloud—the result will always be 5.

With AI, this Predictability Doesn’t Hold:

Let’s see how to “Teach” a Model to perform simple addition using Machine Learning.

1. Data Preparation:
   Create a training set of examples (e.g., (1, 1, 2), (1, 2, 3), (2, 3, 5)) where the first two numbers are inputs and the third is the sum.
2. Model Selection:
   Choose a simple neural network—a small feed-forward model with a few layers and neurons—to predict the sum from the two input numbers.
3. Training:
   During training, the model receives inputs (e.g., a=3, b=4), predicts an output (e.g., 6.9 instead of 7), then adjusts its internal weights via backpropagation to reduce the error between its prediction and the correct sum.
4. Evaluation and Optimization:
   Test the model on unseen examples. If performance is poor, adjust parameters (e.g., number of neurons or learning rate) or provide more training data.
5. Usage:
   Once trained and optimized, the model can take any two numbers as input and output their sum. Although this is a trivial example—simple arithmetic doesn’t require ML—it illustrates how a model “learns” patterns from data rather than following explicit, hard-coded rules.

Let’s say you want the system to recognize a cat in an image. Instead of telling it how to distinguish a cat’s ears or whiskers, you show it millions of images labeled “Cat” and “Not Cat” until it “learns” to identify feline features. Rather than producing a guaranteed outcome, the system offers a probabilistic guess. Sometimes it will say, “I’m 95% sure this is a cat.” But that percentage can vary slightly each time due to differences in model training, data samples, or even the randomness introduced by certain algorithms.

The Fragility of AI Systems: Root Causes

So we can understand that the first real difference using an AI system is the output that is not deterministic. We can identify at least 3 other root causes of AI fragility:

1. Human Misunderstanding:
   AI’s probabilistic nature can lead users astray if they expect deterministic outcomes. Without the right guidance, people may make misguided decisions based on misunderstood outputs. This risk is so recognized that, starting in February 2025, the AI Act will require that users be educated on how to interact effectively with AI systems [2].
2. Data Quality:
   AI models rely heavily on the data they’re trained on. Biased, inaccurate, or non-diverse datasets produce unreliable predictions and fragile behavior. Even seemingly minor flaws in data collection or labeling can lead to significant issues once the model is deployed. International standards like ISO/IEC 5338 [3)]highlight why high-quality data and proper methodologies are critical during AI system development.
3. Ethical Considerations:
   Fragile AI systems often fail to uphold core ethical principles such as fairness, transparency, and accountability. If an AI consistently misclassifies certain groups or fails to explain its reasoning, it not only diminishes user trust but can lead to real-world harm and reputational damage. By implementing ISO/IEC 42001, organizations can develop AI systems that uphold these ethical principles, thereby enhancing user trust and mitigating potential harms [4].

Building More Resilient AI Systems: Roles and Responsibilities

Creating durable and trustworthy AI requires collaboration and adherence to established standards and guidelines. Here’s how various roles can contribute:

• End-Users and Customers: must be educated on how AI works, its limitations, and how to interpret its outputs. With the upcoming regulations, user training isn’t optional—it’s necessary to ensure safe and effective AI adoption.
• Executives and Compliance Officers: need to implement frameworks that mandate human oversight. Before critical AI-driven decisions are executed, a human-in-the-loop should verify outcomes, ensuring that ethical standards and compliance requirements are met.
• Data Scientists and Engineers: must ensure models are built, tested, and validated using high-quality, diverse datasets. They should employ robust techniques to reduce bias, uncertainty, and overfitting.
• Project Managers and Product Owners: should communicate the probabilistic nature of AI outcomes to stakeholders, setting realistic expectations and ensuring that end-users understand how to interpret the results.

Conclusion

Fragile AI systems pose significant challenges, not only in terms of technical performance but also in their ethical implications. Unlike traditional software, AI operates in a probabilistic, ever-evolving space, which makes fragility a critical issue. If left unchecked, this fragility can lead to biased decisions, loss of trust, and even harm.

At Synapsed, we recognize that addressing AI fragility is both a technical and ethical imperative. By leveraging frameworks such as the EU AI Act and ISO standards, we can lay the foundation for resilient, transparent, and trustworthy AI systems. These systems should not only meet performance expectations but also adhere to principles of fairness, accountability, and safety.

The future of AI depends on our ability to build systems that are as robust as they are innovative. Through rigorous testing, comprehensive user education, and adherence to international standards, we can create AI solutions that inspire confidence and drive progress.

Are you ready to join us on this journey? Let Synapsed help you build AI systems that stand the test of time—contact us today to learn more

Ready to Learn More?
If you’d like to explore how Synapsed can help you build more reliable and trustworthy AI systems, reach out to us today.

References:

[1] “AN ARCHITECTURAL RISK ANALYSIS OF MACHINE LEARNING SYSTEMS:Toward More Secure Machine Learning” Gary McGraw, Ph.D., Harold Figueroa, Ph.D., Victor Shepardson, Richie Bonett:https://berryvilleiml.com

[2] EU AI ACT Article 4 on awareness: https://artificialintelligenceact.eu/article/4/

[3] ISO/IEC 5338 “Information technology — Artificial intelligence — AI system life cycle processes”: https://www.iso.org/standard/81118.html

[4] ISO/IEC 42001 “Information technology — Artificial intelligence — Management system”: https://www.iso.org/standard/81230.html