Offensive Use of AI: Attacks on AI/ML Systems

Podcast

Play an AI-generated podcast conversation about this lesson

Questions and Answers

What potential risk is associated with users from different contexts sharing the same vector database?

Data poisoning attacks

Embedding inversion attacks

Cross-context information leaks (correct)

Unauthorized access

What is a potential consequence of generating false or misleading information through LLMs?

Factual inaccuracies leading to reputational damage (correct)

Costly data storage requirements

Reduction in model efficiency

Increased model complexity

Which countermeasure can help prevent unauthorized data access within vector databases?

Data validation updates

Fine-Grained access control (correct)

Extensive data logging

Cross-verification processes

What issue arises from the model's tendency to over-rely on its outputs?

Excessive trust in generated outputs Signup and view all the answers

What is the primary role of data in an AI application?

Training and refining models Signup and view all the answers

What is a method for mitigating risks associated with LLMs generating unsafe code?

Model fine-tuning with verified datasets Signup and view all the answers

Which risk pertains specifically to the integrity of claims made by LLMs?

Unsupported claims Signup and view all the answers

Which of the following describes data poisoning in the context of AI security?

Injecting harmful data into training sets Signup and view all the answers

What is a major risk associated with models accessed via APIs?

Unauthorized access and manipulation Signup and view all the answers

What could be a result of embedding inversion attacks?

Data leakage including sensitive information Signup and view all the answers

What is a necessary step before data is added to a knowledge base to ensure its reliability?

Data validation and source authentication Signup and view all the answers

Which type of injection is an attacker likely to use against the frontend of an AI application?

Prompt/input injection Signup and view all the answers

What primarily drives the risk of inherited vulnerabilities in AI models?

Public/open-source model usage Signup and view all the answers

Which of the following is NOT a security issue affecting data in AI applications?

Model manipulation Signup and view all the answers

Which organization is known for its OWASP Top 10 project related to application security?

Open Worldwide Application Security Project (OWASP) Signup and view all the answers

In terms of AI application architecture, what is the role of the model?

Core engine that learns, decides, and generates outputs Signup and view all the answers

What is the primary concern associated with prompt injection in large language models?

Manipulation of model outputs leading to harmful consequences Signup and view all the answers

What does jailbreaking specifically refer to in the context of prompt injection?

A method to completely bypass safety protocols Signup and view all the answers

Which of the following is an example of indirect prompt injection?

Data from a compromised external database affecting model responses Signup and view all the answers

What type of output manipulation can result from prompt injection?

Exposure to sensitive information and inadvertent biases Signup and view all the answers

What is a proposed countermeasure to prevent the effects of prompt injection?

Constrain model behavior by enforcing strict output limits Signup and view all the answers

Which of the following best describes 'direct prompt injections'?

User input that directly alters the behavior of a model, whether malicious or not Signup and view all the answers

What is a potential risk associated with prompt injection regarding organizational decision-making?

Alteration of outputs influencing critical decisions Signup and view all the answers

Which statement accurately reflects the nature of multimodal injections?

Malicious prompts can be hidden within various media formats. Signup and view all the answers

What is a potential consequence of unbounded consumption in LLMs?

Financial losses Signup and view all the answers

What is the consequence of injecting adversarial training data into a model?

Worsening model performance Signup and view all the answers

Which of the following is a suggested countermeasure for managing resource-intensive queries in LLMs?

Dynamic resource management Signup and view all the answers

Which countermeasure can help ensure model outputs are grounded in trusted sources?

Retrieval-Augmented Generation (RAG) Signup and view all the answers

What is a consequence of Denial of Service (DoS) attacks on LLMs?

Service degradation Signup and view all the answers

What aspect does OWASP recommend to maintain for third-party models?

Regular audits of security and access controls Signup and view all the answers

What type of attack involves flooding the model with excessive requests?

Variable-Length Input Flood Signup and view all the answers

How should the data pipeline be managed to prevent model poisoning?

Tracking and validating data at all stages Signup and view all the answers

Which of the following is NOT a recommended security practice for LLM-generated code?

Ignoring limitations of LLMs Signup and view all the answers

What does insufficient validation of outputs generated by LLM lead to?

Increased risk of Remote Code Execution (RCE) Signup and view all the answers

What technique is used to prevent unauthorized use or replication of LLM outputs?

Watermarking mechanisms Signup and view all the answers

Which practice is part of maintaining model integrity and provenance?

Vendor signed models and code Signup and view all the answers

Which of the following attacks involves crafting inputs to exceed the LLM’s context window?

Input overflow Signup and view all the answers

What is a consequence of model output handling deficiencies?

Exploitation of downstream systems Signup and view all the answers

What can be a result of improperly designed LLM plugins?

Remote code execution Signup and view all the answers

Which tool is recommended for validating models and enhancing integrity?

Machine Learning Bill of Materials (ML-BOM) Signup and view all the answers

What is the primary objective of MITRE ATLAS?

To document adversary tactics and techniques against AI-based systems Signup and view all the answers

Which of the following is NOT part of the MITRE ATLAS Matrix?

Regulatory guidelines for AI practices Signup and view all the answers

What type of attack involves creating a proxy ML model?

ML Attack Staging Signup and view all the answers

Which tactic involves searching for publicly available research materials?

Reconnaissance Signup and view all the answers

Which of the following best describes a backdoor ML model?

An ML model that allows unauthorized access Signup and view all the answers

What kind of mitigation strategies does MITRE ATLAS provide?

Prescriptive actions against specific malicious tactics Signup and view all the answers

What type of access allows attackers to utilize AI models for inference?

AI Model Inference API Access Signup and view all the answers

Which of the following is an example of an adversarial ML attack documented in ATLAS?

Using Rank-One Model Editing to force false facts Signup and view all the answers

What does the acronym FAICP stand for?

Framework for AI Cybersecurity Practices Signup and view all the answers

Which document underpins the best practices for AI security as stated in the content?

Framework for AI Cybersecurity Practices (FAICP) Signup and view all the answers

Study Notes

Offensive Use of AI (Part 2)

This presentation covers the offensive use of AI, focusing on attacks targeting AI/ML systems.

Attacks to AI/ML

AI applications have several components:
- Data: Used to train and refine AI models.
- Model: The core AI system, learning from data.
- Decision-making: The model makes decisions based on learned data.
- Outputs: The results generated by the model.
- Model types: Own models, open-source models, hybrid models.
- Deployment methods: Local deployment, API-accessed models via REST.
- Frontend: User interface for interacting with the model.

Security Issues of an AI Application

Data:
- Data poisoning: Injecting malicious data into training sets. This manipulates the model's output, leading to incorrect or biased results.
- Data exfiltration: Stealing sensitive data from the AI application through unauthorized access or breaches.
Model:
- Inherited vulnerabilities: Public/open-source models might have inherent vulnerabilities.
- API risks: Unauthorized access, manipulation, or intellectual property theft via an API.
- Adversarial Machine Learning: Exploiting vulnerabilities to manipulate model behavior.
Frontend:
- Prompt/input injection: Attackers craft malicious prompts/entries to exploit model vulnerabilities.
- Software vulnerabilities: Common weaknesses in software can be exploited.

OWASP Top 10 for LLMs

Open Worldwide Application Security Project (OWASP): Creates open-source resources for application security.
OWASP Top Ten: Identifies the most critical risks in software development.
Process: Collaboratively developed by security experts.
Components: Data collection, risk assessment & prioritization, and community collaboration.
Other OWASP Top 10 lists: Include Web Application Security Risks (2021), API Security Risks (2023), Mobile Security Risks (2024), and LLM Applications (2025).

LLM01. Prompt Injection

User input manipulates LLM behavior/output in unintended ways.
Exploits handling of prompts to generate harmful outcomes.
Types: Direct (intentional or unintentional) and Indirect (external inputs alter behavior).
- Jailbreaking is a specific type of prompt injection where attackers bypass safety protocols.
Related Risks: Data disclosure, output manipulation, unauthorized access.
Countermeasures: Constrain model behavior, validate output formats, input/output filtering, privilege control, human approval for high-risk actions, external content segregation and adversarial testing.

LLM02. Sensitive Information Disclosure

LLMs unintentionally expose sensitive information (PII, financial, health, proprietary data).
Related Risks: PII leakage, algorithms exposure, business data exposure.
Countermeasures: Techniques to mask sensitive content before training, robust input validation, access controls, federated learning, homomorphic encryption, user education.

LLM03. Supply Chain

External elements (software, tools, pre-trained models) can be manipulated.
Attack vectors: Tampering, poisoning.
Related risks: Outdated/deprecated components, vulnerable pre-trained models and software components, and unclear licensing risks.
Countermeasures: Best Practices from OWASP A06:2021, regular auditing security, maintain model integrity, update assets inventory.

LLM04. Data and Model Poisoning

Training or fine-tuning data is manipulated.
Injection of adversarial training data.
Impacts: Compromises model security, harmful or incorrect outputs, degraded model performance.
Countermeasures: Tracking the data pipeline for integrity, validating providers/outputs, using Machine Learning Bill of Material (ML-BOM) tools, and Retrieval-Augmented Generation (RAG).

LLM05. Improper Output Handling

Insufficient validation, sanitization, and handling of LLM outputs.
Attack vectors: Remote code execution (RCE), Cross-Site Scripting (XSS), SQL Injection (SQLi), and phishing injections.
Goal: Ensuring outputs are safe for downstream systems.
Countermeasures: Zero-trust approach, following OWASP ASVS guidelines (input validation, output sanitization), context-aware encoding (HTML, SQL, JavaScript), and rate limiting.

LLM06. Excessive Agency

LLM granted more functionality/permissions/autonomy than needed.
Leads to harmful/unintended actions.
Impact: Actions beyond the users' control, breaching security boundaries, unintentional data modification.
Countermeasures: Limiting LLM extensions, human approval for critical actions, logging and monitoring LLM and extension activity, rate limiting, anomalous behavior analysis.

LLM07. System Prompt Leakage

Unintended exposure of the system prompt or internal instructions.
Vulnerable to attacks from attackers who exploit leaked sensitive information and bypass security controls.
Countermeasures: Externalize sensitive information within prompts, implement guardrails outside of the LLM, privilege separation and regular review of system prompts.

LLM08. Vector and Embedding Weaknesses

Vulnerabilities related to the use of Retrieval Augmented Generation (RAG).
Risks: Data leakage, poisoning attacks, and unintended behavior shifts.
Countermeasures: Fine-grained access control, data validation and source authentication, and monitoring of retrieval activities.

LLM09. Misinformation

LLMs generating false or misleading information that appears credible.
Causes: hallucinations, biases, incomplete information, overreliance.
Risks: Security breaches, reputational harm, legal liability.
Countermeasures: Implementing RAG (using verified data), model fine-tuning with high-quality datasets, and rigorous cross-validation and human oversight.

LLM10. Unbounded Consumption

LLMs exploited for excessive or uncontrolled inference, depleting resources and degrading system performance.
Malicious activities flood the model with requests, drain resources, or steal intellectual property.
Attack types: Variable-length input flood, resource-intensive queries, denial of wallet, model extraction, functional model replication.
Countermeasures: Input validation, rate limiting, dynamic resource management, sandboxing, and scalable infrastructure, mechanisms to detect unauthorized actions/replication.

MITRE ATLAS

Adversarial Threat Landscape for Artificial-Intelligence Systems, a knowledge base of adversary tactics and techniques against AI-based systems.
Derived from MITRE ATT&CK.
Includes 14 Tactics, 91 Techniques and subtechniques.
Contains general objectives of threat actors, specific methods for tactical goals, and best practices for mitigating attacks.

Regulations and Best Practices for AI Security

Framework for AI Cybersecurity Practices (FAICP), from ENISA (European Union Agency for Cybersecurity): Layers I, II, and III cover general practices, AI-specific practices, and critical sector-specific practices.
Artificial Intelligence Risk Management Framework (AI RMF 1.0), from NIST: Provides a standardized framework for managing risks.
Artificial Intelligence Act - Regulation (EU): EU regulations regarding AI systems.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

This quiz delves into the offensive use of artificial intelligence, focusing on various attacks targeting AI and machine learning systems. Key components covered include data integrity, model vulnerabilities, and the security issues inherent in AI applications. Test your knowledge on the tactics and implications of these malicious actions.