Secure AI Enablement 

1. Define the Use Case and Business Value – Purpose Before Technology

Specify the intended purpose, users, expected value and measurable success criteria before selecting technology. 

Most of this is not new, it’s been a part of procurement processes for quite some time. But what does it mean? For example, without having a purpose or intended use for an AI application, resources will not be invested in what supposed to support business processes.

Further, ensuring that the value can be measurable in different success criteria’s is a good method to evaluate if the choice fulfilled the intended use and value. A formula for this can be; Success criteria = Business outcome + AI quality + Adoption + Risk boundary, for example: 

The application is considered successful if it reduces manual processing time by >30%, produces correct outputs in >95% of evaluated cases, is actively used by >70% of the intended user group, and operates without exceeding defined security, privacy and compliance risk thresholds. 

2. Establish Risk Tolerance and Decision Thresholds – Set Clear Boundaries

Define acceptable and unacceptable outcomes, risk tiers, performance thresholds and clear go/no-go criteria. 

Before starting to measure different success criteria’s, risk and thresholds need to be in place. For example, having a generative AI system that is hallucinating and produces information that is incorrect, fabricated, misleading, or unsupported by the available data or context, while presenting it as if it were factual or reliable in 30% of the cases an acceptable or not acceptable outcome?

In addition to that, risk tiers need to be defined that reflects AI, this means that organizations need to identify, measure, prioritize and manage AI risks according to their own context, risk tolerance, and potential impact, which in some cases are not equal to how we traditionally measure risk, here NIST AI RMF 1.0 or ISO 42001 can be supporting standards.   

3. Define the Use Case and Business Value – Purpose Before Technology

Identify where and how the AI system will be used, foreseeable misuse, assumptions, dependencies and potential impacts. 

Perform an assessment in order to understand the potential impacts an AI system might have on the organization. For example, ask the following questions in these areas to assess and understand the impact of an AI system in the organization: 

• Impact on individuals: Could an incorrect AI decision materially affect a person?  
• Decision autonomy: Is AI merely assisting a human, or making decisions automatically?  
• Data sensitivity: Does it process personal, confidential, health, financial or security-sensitive information?  
• Business criticality: What happens if the AI produces an incorrect result?  
• Security exposure: Can the AI access systems, execute actions, retrieve sensitive information or interact with external systems? 
• Scale: How many users or individuals could be affected?  
• Reliability: What are the consequences of hallucination, bias or inaccurate results? 
• Human oversight: Is an appropriately qualified person reviewing consequential outputs? 
• Traceability: What kind of logs and audit trails are collected? Can we detect when something goes wrong? 

4. Select Technology Based on Requirements – Fit the Solution to the Need

Choose the model, platform and architecture only after functional and non-functional requirements, such as security, privacy and performance are clear. 

If one of the requirements is to ensure a very high accuracy of the AI system, for example, image recognition of 98%, AI model choice is essential, where the model needs to not only be trained specifically on images, but it also needs to show test results in accuracy of 98% or higher.

The same goes for all functional and non-functional requirements. All the requirements need to be aligned and support business processes, the potential values, and the goals of the organization.  

5. Design Security and Privacy Into the Architecture – Secure by Design

Apply least privilege, restrict data and tool access, validate inputs and outputs, and protect sensitive data and models. 

Secure by design means to build security and privacy into the AI system from the start, rather than adding controls at a later stage, and this applies for the entire system, not only the AI model.  

Apply least privilege: A very common mistake is that an AI system is given overly excessive permissions and does not follow least privilege, for example, an AI assistant that should only summarize information should not be able to send emails or change records. 

Restrict data and tool access: Enforce each user’s access rights, with on-behalf-of and OAuth passthrough or other methods when retrieving information. Rather use several agents with a small context for each agent and one to two tools, than one agent for all contexts with a lot of tools. 

Validate inputs and outputs: Always treat prompts, uploaded documents and retrieved content as untrusted, and verify the outputs with predefined validation checks. This shall be done before displaying them or passing them to another system and never execute generated instructions automatically. 

Traceability: Collect the right kind of logs and audit trails in a centralized solution so it is possible to detect rouge AI agents, misconfigurations, data accidentally deleted by an AI or human. Monitoring and logs are essential. 

These four areas do not cover everything within secure by design, but are fundamental and unfortunately, commonly missed.  

6. Threat-Model the Complete AI Solution – Cover the Full Attack Surface

Assess risks across models, data sources, retrieved content, integrations, plugins, suppliers, generated outputs and connected tools. 

Threat modeling is a cornerstone in secure by design and is something that should always be done before and during development of any system, AI or no AI. Threat modeling does not only give us an understanding of security risks in the system, but it also gives the team a common understanding on how the system actually works and be aligned on that. This is not as common as one might think.  

7. Make Human Oversight Operational – Give Oversight Authority

Assign accountable roles, define required competence and ensure authorised people can question, override, escalate or stop the system. 

Here it is necessary to define and differentiate on what AI does and does not. It needs to be established whether the AI system supports human decision-making or replaces parts of it. Accountability needs to be clear, within the organization’s governance structure with defined processes and responsibilities. Further, ensuring that the organization and individuals with roles and responsibilities for AI systems has the necessary competence and training.  

However, oversight can’t solely be human, assuming human-in-the-loop automatically creates effective oversight. AI oversight need to include monitoring where abnormalities automatically alerts responsible individuals.   

8. Test Before Deployment – Validate Under Realistic Conditions

Evaluate performance with representative users and test adversarial inputs, automation bias, failure scenarios and unsafe outcomes. 

As in any type of system or application, testing before production is a quite normal procedure. The same procedure applies here, perform automated test, negative tests, A/B testing, secrets scanning, static code analysis, dependency scanning app & container, security test/negative test, dynamic code analysis, TLS scanning, application security scanning, depending on which part of the lifecycle that is managed.  

Adversarial penetration tests are increasingly important as the usage of AI systems and specifically agentic AI systems, where risk such as prompt injection, improper output handling, tool misuse, identity and privilege abuse and many more.  

Without proper testing, there will be unknown risks that are accepted if the AI system is implemented.  

9. Deploy Gradually With Safeguards – Limit Initial Exposure

Begin with limited permissions, constrained scope and controlled access. Expand capabilities only when evidence supports doing so. 

Start small, go bigger. With a small context, limited access to tools and data, the implementation of AI systems is more controlled, and if it would go wrong, the risk and exposure is smaller. Once a small context and use case works, as intended, e.g. it satisfies the success criteria’s described in step 1, start exploring adding capabilities, this does not mean to add more tools to an AI agent, merely to give it the capabilities to support the business as a whole. 

Do remember, when capabilities increase, so does the need to increase all the other areas, such as the scope for threat modeling, logs collection, alerts and monitoring, to mention a few.  

10. Continuously Monitor and Improve – Manage Risk Throughout the Lifecycle

Monitor performance, security, intervention effectiveness and emerging risks. Connect findings to incident response and repeat risk assessments throughout the AI lifecycle. 

By performing continuous monitoring of performance, security, effectiveness and continuous measuring of the success criteria’s, we can quickly act when changes and/or abnormalities happen within an indecent response process and perform a new risk assessment to determine if it is still within our risk tolerance. Remember, a risk assessment shall always be performed when medium and major changes are in play. 

Conclusions 

Bringing AI into an organisation means connecting it to people, systems, data and business processes. There are many ways things can go wrong, and we should expect to learn along the way. The aim is not to eliminate every uncertainty before starting, but to create the conditions to identify problems, contain their impact and act on what we learn.