Offensive Use of AI: Attacks on AI/ML Systems

Offensive Use of AI (Part 2)

• This presentation covers the offensive use of AI, focusing on attacks targeting AI/ML systems.

Attacks to AI/ML

• AI applications have several components:
  • Data: Used to train and refine AI models.
  • Model: The core AI system, learning from data.
  • Decision-making: The model makes decisions based on learned data.
  • Outputs: The results generated by the model.
  • Model types: Own models, open-source models, hybrid models.
  • Deployment methods: Local deployment, API-accessed models via REST.
  • Frontend: User interface for interacting with the model.

Security Issues of an AI Application

• Data:
  • Data poisoning: Injecting malicious data into training sets. This manipulates the model's output, leading to incorrect or biased results.
  • Data exfiltration: Stealing sensitive data from the AI application through unauthorized access or breaches.
• Model:
  • Inherited vulnerabilities: Public/open-source models might have inherent vulnerabilities.
  • API risks: Unauthorized access, manipulation, or intellectual property theft via an API.
  • Adversarial Machine Learning: Exploiting vulnerabilities to manipulate model behavior.
• Frontend:
  • Prompt/input injection: Attackers craft malicious prompts/entries to exploit model vulnerabilities.
  • Software vulnerabilities: Common weaknesses in software can be exploited.

OWASP Top 10 for LLMs

• Open Worldwide Application Security Project (OWASP): Creates open-source resources for application security.
• OWASP Top Ten: Identifies the most critical risks in software development.
• Process: Collaboratively developed by security experts.
• Components: Data collection, risk assessment & prioritization, and community collaboration.
• Other OWASP Top 10 lists: Include Web Application Security Risks (2021), API Security Risks (2023), Mobile Security Risks (2024), and LLM Applications (2025).

LLM01. Prompt Injection

• User input manipulates LLM behavior/output in unintended ways.
• Exploits handling of prompts to generate harmful outcomes.
• Types: Direct (intentional or unintentional) and Indirect (external inputs alter behavior).
  • Jailbreaking is a specific type of prompt injection where attackers bypass safety protocols.
• Related Risks: Data disclosure, output manipulation, unauthorized access.
• Countermeasures: Constrain model behavior, validate output formats, input/output filtering, privilege control, human approval for high-risk actions, external content segregation and adversarial testing.

LLM02. Sensitive Information Disclosure

• LLMs unintentionally expose sensitive information (PII, financial, health, proprietary data).
• Related Risks: PII leakage, algorithms exposure, business data exposure.
• Countermeasures: Techniques to mask sensitive content before training, robust input validation, access controls, federated learning, homomorphic encryption, user education.

LLM03. Supply Chain

• External elements (software, tools, pre-trained models) can be manipulated.
• Attack vectors: Tampering, poisoning.
• Related risks: Outdated/deprecated components, vulnerable pre-trained models and software components, and unclear licensing risks.
• Countermeasures: Best Practices from OWASP A06:2021, regular auditing security, maintain model integrity, update assets inventory.

LLM04. Data and Model Poisoning

• Training or fine-tuning data is manipulated.
• Injection of adversarial training data.
• Impacts: Compromises model security, harmful or incorrect outputs, degraded model performance.
• Countermeasures: Tracking the data pipeline for integrity, validating providers/outputs, using Machine Learning Bill of Material (ML-BOM) tools, and Retrieval-Augmented Generation (RAG).

LLM05. Improper Output Handling

• Insufficient validation, sanitization, and handling of LLM outputs.
• Attack vectors: Remote code execution (RCE), Cross-Site Scripting (XSS), SQL Injection (SQLi), and phishing injections.
• Goal: Ensuring outputs are safe for downstream systems.
• Countermeasures: Zero-trust approach, following OWASP ASVS guidelines (input validation, output sanitization), context-aware encoding (HTML, SQL, JavaScript), and rate limiting.

LLM06. Excessive Agency

• LLM granted more functionality/permissions/autonomy than needed.
• Leads to harmful/unintended actions.
• Impact: Actions beyond the users' control, breaching security boundaries, unintentional data modification.
• Countermeasures: Limiting LLM extensions, human approval for critical actions, logging and monitoring LLM and extension activity, rate limiting, anomalous behavior analysis.

LLM07. System Prompt Leakage

• Unintended exposure of the system prompt or internal instructions.
• Vulnerable to attacks from attackers who exploit leaked sensitive information and bypass security controls.
• Countermeasures: Externalize sensitive information within prompts, implement guardrails outside of the LLM, privilege separation and regular review of system prompts.

LLM08. Vector and Embedding Weaknesses

• Vulnerabilities related to the use of Retrieval Augmented Generation (RAG).
• Risks: Data leakage, poisoning attacks, and unintended behavior shifts.
• Countermeasures: Fine-grained access control, data validation and source authentication, and monitoring of retrieval activities.

LLM09. Misinformation

• LLMs generating false or misleading information that appears credible.
• Causes: hallucinations, biases, incomplete information, overreliance.
• Risks: Security breaches, reputational harm, legal liability.
• Countermeasures: Implementing RAG (using verified data), model fine-tuning with high-quality datasets, and rigorous cross-validation and human oversight.

LLM10. Unbounded Consumption

• LLMs exploited for excessive or uncontrolled inference, depleting resources and degrading system performance.
• Malicious activities flood the model with requests, drain resources, or steal intellectual property.
• Attack types: Variable-length input flood, resource-intensive queries, denial of wallet, model extraction, functional model replication.
• Countermeasures: Input validation, rate limiting, dynamic resource management, sandboxing, and scalable infrastructure, mechanisms to detect unauthorized actions/replication.

MITRE ATLAS

• Adversarial Threat Landscape for Artificial-Intelligence Systems, a knowledge base of adversary tactics and techniques against AI-based systems.
• Derived from MITRE ATT&CK.
• Includes 14 Tactics, 91 Techniques and subtechniques.
• Contains general objectives of threat actors, specific methods for tactical goals, and best practices for mitigating attacks.

Regulations and Best Practices for AI Security

• Framework for AI Cybersecurity Practices (FAICP), from ENISA (European Union Agency for Cybersecurity): Layers I, II, and III cover general practices, AI-specific practices, and critical sector-specific practices.
• Artificial Intelligence Risk Management Framework (AI RMF 1.0), from NIST: Provides a standardized framework for managing risks.
• Artificial Intelligence Act - Regulation (EU): EU regulations regarding AI systems.

This quiz delves into the offensive use of artificial intelligence, focusing on various attacks targeting AI and machine learning systems. Key components covered include data integrity, model vulnerabilities, and the security issues inherent in AI applications. Test your knowledge on the tactics and implications of these malicious actions.