Lecture 1 - Introduction.pdf

OFFENSIVE AI LECTURE 1: INTRODUCTION Dr. Yisroel Mirsky [email protected] 2 Course Overview Dr. Yisroel Mirsky [email protected]  Faculty Member in the SISE Department  Research: The intersection of machine learning and cyber security  Defensive AI  Offensive AI Dr. Yisroel Mirsky Dr. Yisro...

OFFENSIVE AI LECTURE 1: INTRODUCTION Dr. Yisroel Mirsky [email protected] 2 Course Overview Dr. Yisroel Mirsky [email protected]  Faculty Member in the SISE Department  Research: The intersection of machine learning and cyber security  Defensive AI  Offensive AI Dr. Yisroel Mirsky Dr. Yisroel Mirsky Course Overview Offensive AI Research Lab We are Researching: Malicious Deepfakes Attacks on AI and Machine Learning … https://offensive-ai-lab.github.io/ Looking for M.Sc./Ph.D. students and research assistants! Dr. Yisroel Mirsky 4 Course Overview Requirements ► This course will be taught in English ► Students are expected to know: ► ► Classical Machine Learning: Naive Bayes, SVM, Decision Trees... ► Deep Learning: Basic architectures (dense, CNN), training, loss functions Weekly reading: scientific articles Dr. Yisroel Mirsky 5 Course Overview Requirements Grade ► 10% Python exercise(s) ► 20% Final Project ► 70% Final Exam ► Must pass exam to pass the course Dr. Yisroel Mirsky 6 Course Overview Schedule 1. Attacks on AI 2. 3. 4. 5. Attacks using AI 6. 7. 8. 9. 10. Course Conclusion 11. Introduction to Machine Learning and Offensive AI Adversarial Examples – Whitebox Adversarial Examples – Blackbox Confidentiality and Poisoning Attacks Detection, Prevention and Mitigation of Adversarial ML Lab: hands on attacking ML Visual Deepfakes Audible Deepfakes Detection, Prevention and Mitigation of Deepfakes AI in Spyware and Cyber Attacks Student Presentations Dr. Yisroel Mirsky 7 Course Overview What is Offensive AI?  Offensive AI (OAI) refers to using or exploiting AI to perpetrate an attack  Contrast to Defensive AI (DAI) which is used to protect against attacks Ethics of Offensive AI  Some OAI acts are ethical (e.g., tracking terrorists, pen-testing models,...)  Others unethical (humiliation, scamming, damage, bodily harm,...) Dr. Yisroel Mirsky 8 Course Overview We are Whitehat not Blackhat AI should only be used to preserve human dignity and safety Not as a means to harm or take advantage of others Dr. Yisroel Mirsky 9 Course Overview The Purpose of this Course: To Understand the Threats  How the technologies work, their limitations, and applicability To Be Aware of the Defences  What can be done to stop them, what are effective countermeasures (and which are not!) To Observe their Trajectory  To inform policy, plan ahead, outsmart the attacker Dr. Yisroel Mirsky Today’s Agenda  Artificial Intelligence   Concepts, terms, scope Offensive AI (OAI)  What’s it all about? 10 Dr. Yisroel Mirsky 11 Artificial Intelligence What is AI? Dr. Yisroel Mirsky 12 Artificial Intelligence What is AI? Dr. Yisroel Mirsky 13 Introduction What is AI? AI (Artificial Intelligence): the ability of a digital computer or computercontrolled robot to perform tasks commonly associated with intelligent beings [Encyclopedia Britannica] Example from B.J. Copeland 14 Introduction What is ‘Intelligence’? Consider these cases:  A human unlocks his door to enter his home  A female digger wasp checks her home for intruders before bringing in food Why is the human’s behaviour considered intelligence but not the wasp’s? Example from B.J. Copeland 15 Introduction What is ‘Intelligence’? Consider these cases:  A human unlocks his door to enter his home  A female digger wasp checks her home for intruders before bringing in food Consider what happens if we move the food before the wasp returns......she will keep repeating the process over! Intelligence requires the ability to adapt to new circumstances Dr. Yisroel Mirsky 16 Introduction Human Intelligence  Reason  Discover  Learn meaning from past experience, or  Generalize Dr. Yisroel Mirsky Figuring out chess moves and saving best option 17 Introduction Artificial Intelligence (types) 1. Learning: To improve with experience  Trial and Error: Make mistakes to find better solutions  Generalization: Applying past experience to new situations 2. Reasoning: To draw inferences from a situation 3. Planning: To maximize utility through interactions with environment 4. Problem Solving: To perform a systematic search towards a goal 5. Perception: To interpret environments from sensors 6. Language: To understand, participate, and have social awareness in conversations Loss functions in deep learning(back propagation) Walk -> Walked Talk -> Talked Attack ->...? Bob is in New York, what country is he in? Eve is 5 years older than Mallory... How old is Eve? A swarm of drones working to build a wall Heuristic search to solve a Rubiks cube Find lowest cost budget for finance plan Autonomous vehicles finding the road and obj. AI chat bots Dr. Yisroel Mirsky 18 Introduction The Flexibility of AI So where are our AI overlords? There is a big gap between intelligence and sentience Applied AI vs General AI Dr. Yisroel Mirsky 19 Introduction The Flexibility of AI Helping master put on pants made him angry I will try helping him put on a shirt… Artificial General Intelligence – AGI (Strong AI)  Has flexibility in performing multiple intelligence tasks E.g., a robot whose goal is to help a human and make him happy Reason, plan, learn Tests for Strong AI: The Turing Test (Turing): Can a human tell the difference between a human and machine in conversation? The Coffee Test (Wozniak): Can a machine enter a home and figure out how to make coffee? The Robot College Student Test (Goertzel): Can a machine enroll in a university and obtaining a degree? The Employment Test (Nilsson): Can a machine perform an economically important job at least as well as humans in the same job? Didn’t work, back to planning. What is true happiness? Dr. Yisroel Mirsky 20 Introduction The Flexibility of AI Applied AI  Designed to solve exactly one problem   Examples: find cancer in CT scan, classify cat from dogs,... Complex systems will utilize multiple applied AIs  Example: Tesla’s autopilot = object recognition  Essentially all AI systems to day are Applied AI  AGI is considered a hard and unsolved problem + depth prediction + path planning Dr. Yisroel Mirsky 21 Introduction The Most Popular Applied AI Machine Learning >1950: AI Excitement Definition: Programs which improve with experience (data) Examples:  Decision Trees  Neural Networks  Guassian mixture models (GMM)  K-Means >1980: ML Flourishes >2010: Deep learning AI boom medium.com Dr. Yisroel Mirsky 22 Introduction Machine Learning Why is it so popular?  It can perform tasks that require some intelligence (mimic humans)  It can operate on massive amounts of data Motivations that Drive ML  Availability: we’ve got data, but can’t learn from it  Automation: we need automate a task with speed/scale/precision Dr. Yisroel Mirsky 23 Introduction Ubiquity of Machine Learning Surveillance Manufacturing Personal Assistants Healthcare Power Forecasting Social Media Self Driving Finance Security Dr. Yisroel Mirsky 24 Introduction Types of Machine Learning  Supervised →cat →cat → cat →car 𝑩 𝑨 →𝑨  Unsupervised  Semi-supervised →cat  Reinforcement → car? → cat →car -3 → house? -8 → lion? -1 → cat Dr. Yisroel Mirsky 25 Introduction The Modes of Machine Learning Train Mode 𝑥 Oracle 𝑥 (0.5kg, black, furry) 𝑥 (0.5kg, black, furry) ML Model ML Model 𝑦 (cat) (0.5kg, black, furry) ML Model unsupervised supervised reinforcement Execute Mode (test, predict,... aka decision time) 𝑥 𝑥 (0.5kg, black, furry) ML Model 𝑦ത (cat?) 𝑦ത (type A?) supervised, unsupervised, reinforcement (0.5kg, black, furry) ML Model lazy learners (KNN,...) 𝑦ത (cat?) 26 Offensive AI C’mon already... let’s troll some AIs! Dr. Yisroel Mirsky Dr. Yisroel Mirsky 27 Offensive AI What is Offensive AI? Definition: The use or abuse of AI to perform an attack Attacks using AI In practice: AI vs Humans Attacks on AI Humans vs AI Dr. Yisroel Mirsky 28 Offensive AI Perturbation Adversarial example Attacks on AI Adversarial Machine Learning The exploitation of a model to achieve a malicious goal:  Gain Control  Evade Detection  Denial of Service (DoS)  Steal Information = Milla Jovovich Dr. Yisroel Mirsky 29 Offensive AI Attacks using AI AI Attack Tools & Attack Automation The use of AI technology to Perform or Automate a malicious activity Dr. Yisroel Mirsky 30 Offensive AI OAI is Not Black & White Attacks targeting... humans machines Network Intrusion Targeting Systems Vulnerability detection Spam Intrusion Fingerprint-spoofing Drone warfare Evidence Tampering Cancer injection Adversarial deepfakes Impersonation Voice Recognition Password guessing Fooling Aut. cars Deepfakes Attacks on humans vs human intelligence Worms DGA evasion both Adversarial Machine Learning Attacks on machines vs intelligent machines (AI) Adversarial Examples Dr. Yisroel Mirsky 31 Offensive AI Physical vs Virtual  OAI can involve the virtual space, physical space, or both  The ‘space’ of an attack determines  Attack flexibility Access – can the attacker play with the gradient in advance? Preparation – what will the environment be? (view, traffic,..) Applicability – Some attacks wont work in physical space Timing – casual? timed? interactive?  Likelihood of success Robustness – Physical attacks are less robust/effective Knowledge – Virtual attacks have more static environments Dr. Yisroel Mirsky 32 Offensive AI Attacks on AI Physical Virtual + = Timing Jiefeng Chen, Xi Wu, Robust Attribution Regularization 21 Athalye A. et al, Synthesizing Robust Adversarial Examples 2018 Song D, et. Al. Physical adversarial examples for object detectors. In12th USENIX WOOT 18. No Sign Max speed 100 Setting Dr. Yisroel Mirsky 33 Offensive AI Attacks using AI Virtual Original Image Onetime Synthetic Content Confidence: 99.3% Attack = + Prediction: Benign Physical Content generated by AI Prediction: Malign Confidence: 99.9% Interactive Mirsky Y, et al. Malicious Tampering of 3D Medical Imagery using Deep Learning. USENIX 19 Global News. Too real? Deepfake Putin, Kim Jong Un ads pulled from U.S. debate DARPA Grand Challenge The Guardian. European MPs targeted by deepfake video calls imitating Russian opposition Army Researchers Advance Drone Swarm Learning Capabilities. Photonics Crime Detection Using Machine Learning. ATS journal Dr. Yisroel Mirsky 34 Offensive AI Which Types of AI Intelligence are used in OAI (today)? Stealing models Evading malware detectors Planting back doors (trojan)... Hiding a stop sign Impersonation glasses License plate dress (DoS)... Attacks on AI Exploit these AI: Learning Reasoning Planning Problem Solving Perception Language Spam evasion Fake reviews False authorship Exploiting LLMs... https://adversarialfashion.com/ Attacks using AI Utilize these AI: Learning Reasoning Planning Problem Solving Perception Language Fake news Automated phishing... Vulnerability detection User tracking, spyware, keylogging... Drone cooperation Botnet propagation... Impersonation Evidence tampering Target Selection... Dr. Yisroel Mirsky 35 types of AI are Vulnerable, and “AllAlltypes of AI can be used in Attacks THE MIRSKY CONJECTURE ” “All types of AI are Vulnerable”  Fact: All AI types either/both  Acquire Knowledge: learn from data, receive rules,...  Apply the Knowledge: making predictions (generalization) on new observations But, knowledge and observations can be tainted Test: If you can make a human perform a task wrong, then you can fool an Applied AI Dr. Yisroel Mirsky 36 types of AI are Vulnerable, and “AllAlltypes of AI can be used in Attacks THE MIRSKY CONJECTURE “All types of AI can be used in Attacks”  Fact: Every type of AI intelligence has a meaningful application  Fact: Ethical applications can be used unethically Test: Can repurpose a human’s skill for both an ethical an unethical task? ” Dr. Yisroel Mirsky 37 types of AI are Vulnerable, and “AllAlltypes of AI can be used in Attacks THE MIRSKY CONJECTURE ” So why don’t we see attacks on/using the other AI Types of Intelligence? The Golden Rule of Cyber Security: Attacks will only surface when profitable  Black Hat (criminals, vandals, state actors...): More effort than profit in targeting these applications  White Hat (security researchers): Lack of use cases/publishability to drive work in the area 38 Offensive AI: Attacks on AI Better known as “Trolling AI” Dr. Yisroel Mirsky Dr. Yisroel Mirsky 39 Offensive AI: Attacks on AI Taxonomy An adversarial ML attack can be categorized using the following these properties: Taxonomy modified and expanded from Barreno M, Nelson B, Joseph AD, Tygar JD. The security of machine learning. Machine Learning. 2010 Nov;81(2):121-48.  Goal: Evasion, Control, Confusion (DoS), Extraction  Influence: where the attacker can affect the ML pipeline  Security Violation: the type of security violation the attacker causes  Specificity: the selectivity (range) of the attack  Objective: the desired outcome (e.g, un/targeted mistake)  Assumption: the knowledge the attacker has on the victim Dr. Yisroel Mirsky 40 Offensive AI: Attacks on AI Taxonomy Train Mode (in/outside premises) Rest Mode (in storage) Test Mode (deployed) ML Model Causative Exploratory  Influence: the capability of the attacker. Either…  Causative: The attacker can influence the model’s parameters in train and rest modes (e.g., poison)  Exploratory: The attacker can interact with the model in execute or rest modes and possibly observe the results (e.g., adv examples, model inversion,...) Dr. Yisroel Mirsky 41 Offensive AI: Attacks on AI Taxonomy  Security Violation (CIA) Confidentiality:  Model inference (theft)  Training data inference (content, membership) Integrity:  Poisoning a dataset (make holes in defence, backdoors,...)  Manipulating a model (control model –adversarial examples) Availability:  Overloading systems with heavy/difficult queries  Cause too many false positives Dr. Yisroel Mirsky 42 Offensive AI: Attacks on AI Taxonomy Causative Exploratory  Specificity Selective  -which sample will be affected? Selective: Targets a specific instance or set of instances (E.g., specific malware accepted, membership inference...)  Indiscriminate: Indiscriminate Aims to cause a general failure (E.g., all malwares accepted, model inversion...) Dr. Yisroel Mirsky 43 Offensive AI: Attacks on AI Taxonomy  Objective  -what prediction will be made? Targeted: The objective is to  or  Cause a specific error (𝑓 𝑥 = 𝑐𝑖 changed to 𝑐𝑗 , where 𝑐𝑖 , 𝑐𝑗 ∈ 𝐶 )  Collect information on a specific item (e.g., is 𝑥 a member of 𝑋?) Untargeted: The objective is to  Cause any error to (𝑓 𝑥 = 𝑐𝑖 changed to 𝑐𝑗 ∈ 𝐶 | 𝑖 ≠ 𝑗 )  Collect general information (e.g., what is the parameters or training data of 𝑀?) Dr. Yisroel Mirsky 44 Offensive AI: Attacks on AI Taxonomy  Assumption  White-box: Attacker has full knowledge of the model (has a copy) Knows architecture, parameters, perhaps even datasets  Black-box: Attacker knows nothing (or little) about the model Perhaps knows its a CNN but that’s all  Gray-box: The attacker has limited knowledge on the model E.g., knows architecture and/or dataset –does not have parameters Dr. Yisroel Mirsky 45 Offensive AI: Attacks on AI Common Attack Classes Legend: Vulnerability Attack Rest Mode Train Mode (in/outside premises) Poisoning Model Tampering Model Evasion Backdoor ML Model Trust in the Training Data Extraction Model Extraction (deployed) Data Extraction Property inference Model Membership inference Inversion Runtime Corruption Skewing Execute Mode Manipulation Adversarial Examples degradation Trust in the hardware, supply chain, cyber security Sponge Examples System Exploits Design oversights Trust in the field data and environment Dr. Yisroel Mirsky 46 Offensive AI: Attacks on AI Design Oversights  Sometimes, it’s the AI system at fault and not a specific applied model  Planning AI are also susceptible  Edge cases must be considered  Can we trust the environment? Dr. Yisroel Mirsky 47 Offensive AI: Attacks on AI Design Oversights  Can we trust the user? In just 24 hrs, Tay assimilated the internet’s dark side into its personality... 2016 – Microsoft releases a twitter chat bot that learns from its users Dr. Yisroel Mirsky 48 Offensive AI: Attacks on AI Attacks Throughout the ML Lifecycle  Machine Learning Development Cycle Example Attack Goal Threats at every step... Evade Detection Plant Custom Trigger Attacker The attack… acts on Cause DoS Steal Trained Model targets both Learn Private Information Cause Task Malfunction Example Attack Vectors 1 Define Observations Plant insider, cause a diversion, … 2 Obtain Sample Data Insert malicious samples, sabotage sensor, … 3 Design Features Block assets, promote weak features, … 4 Select Model & Params Investigate weaknesses (e.g., model bias), … 5 Train Model 6 Evaluate Model 7 Deploy Model Send many instances, send crafted instances, … 8 Update Model Target decision boundary (gradual drift), … Sabotage hardware (GPU), generate local minima … Divert focus, target noise floor… 49 Offensive AI: Attacks using AI Dr. Yisroel Mirsky Dr. Yisroel Mirsky 50 Offensive AI: Attacks using AI Vulnerability discovery AI is an Enabler for Society  Ability to predict  Ability to automate  Ability to create But AI is a Double Edged Sword Drone delivery Special effects Zero-day attacks Automated bot campaigns Deepfake Impersonations If the good guys can use it to perform tasks......then so can the bad guys Dr. Yisroel Mirsky Offensive AI: Attacks using AI 51 Implementation: Attackers Do Not Have To Reinvent The Wheel Mirrored Tasks Same Code, adapted for same purpose but different intent  Vulnerability detection, tracking via facial or voice recognition, reverse engineering (code similarity), insider detection (cover tracks),... Repurposed Tasks Same algorithms adapted for different purpose (with changes to code/model)  Face replacement deepfakes, inferring private information from Social Media, explainable AI to cover artifacts (e.g., in malware)... Novel Tasks New algorithms and code developed for a new or existing attacks  Password guessing, keylogging from side channels, real-time deepfakes, botnet autonomy, malware obfuscation... With code from research opensourced on GitHub (attack or benign), it is easy for attackers to adopt AI quickly Dr. Yisroel Mirsky 52 Offensive AI: Attacks using AI But is the Sword Symmetric?  Some AI tools are better for the attacker   Example: Content Generation –deepfakes Some AI tools are better for the defender  Example: Anomaly Detection -network intrusion detection Corridor digital Kim H, Carrido P, Tewari A, Xu W, Thies J, Niessner M, Pérez P, Richardt C, Zollhöfer M, Theobalt C. Deep video portraits. ACM Transactions on Graphics (TOG). 2018 Jul 30;37(4):163. Dr. Yisroel Mirsky Offensive AI: Attacks using AI AI Attack tools are Changing the Battlefield   Attacker who use them have...  increased coverage (can reach new targets)  increased attack speed (can hit more targets a day)  increased stealth (can cover tracks better)  increased efficiency (costs less, fewer resources...) Attackers are using them to...  Enhance Existing Attacks Password Cracking, User tracking,...  Perform New Attacks Impersonation, tamper evidence,... 53 Dr. Yisroel Mirsky Offensive AI: Attacks using AI 54 Attack Tools Media Creation is the most Prominent Tool Oct 2021  Impersonation: Re-enactment, voice cloning,...  Image synthesis (e.g., fake persona)  Image tampering (e.g., insert or alter objects)  Record tampering (e.g., finance) Sept 2019 Avatartify - First order motion model for image animation. NeurIPs https://www.mcafee.com/blogs/privacy-identity-protection/artificial-imposterscybercriminals-turn-to-ai-voice-cloning-for-a-new-breed-of-scam/ Offensive AI: Attacks using AI Deepfake Voice Scams How likely are you to reply to a voice message from someone you know, saying that they are in need of money? 45% respondents said yes: 40% if partner or spouse 24% if mother 20% if child How likely are you to respond in these cases? They’ve been in a car accident (48%). They’ve been robbed (47%). They’ve lost their phone or wallet (43%). They needed help while traveling abroad (41%). 55 https://www.mcafee.com/blogs/privacy-identity-protection/artificial-imposterscybercriminals-turn-to-ai-voice-cloning-for-a-new-breed-of-scam/ Offensive AI: Attacks using AI Deepfake Voice Scams 56 Dr. Yisroel Mirsky Offensive AI: Attacks using AI Attack Tools (other prominent tools)  Hacking   Information Gathering   Botnets, password cracking, attack camouflage,... OSINT, inferring private info (e.g., blackmail), user tracking,... Military: Targeting, planning, terrain analysis,... Attack Automation  Offensive Swarm intelligence  Automated Attack Campaigns (e.g., phishing) 57 Dr. Yisroel Mirsky 58 Reccomended Reading: Week 1 Brundage M., et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228. 2018 Feb https://arxiv.org/pdf/1802.07228.pdf For more adversrail example papers see Nicolas Carlini’s website: https://nicholas.carlini.com/writing/2019/all-adversarial-example-papers.html 59 Dr. Yisroel Mirsky Derpfakes