CYB. Defensive AI (Part 3) Master's in Artificial Intelligence PDF

CYB. Defensive AI (part 3) Master in Artificial Intelligence 2024/25 ESEI – University of Vigo AI/ML in Malware Analysis Malware: Definition and Types Malware (malicious software) → software intentionally designed to harm, exploit, or compromise computer systems and data Typical types of...

CYB. Defensive AI (part 3) Master in Artificial Intelligence 2024/25 ESEI – University of Vigo AI/ML in Malware Analysis Malware: Definition and Types Malware (malicious software) → software intentionally designed to harm, exploit, or compromise computer systems and data Typical types of malware (can be a mixture) Self-replicating ˆ Viruses Programs that spread when the infected files are executed [Stuxnet] ˆ Worms Programs that spread across networks without user interaction [SQL Slammer] Auto-hiding malware ˆ Trojans Software that masquerades as legitimate but contains malicious code (backdoor, data theft) [Qbot/Qakbot, TrickBot] ˆ Rootkits Hides and makes difficult to detect or remove malicious software [Linfo, Pandora, HIDEDRV] 1 Malware: Definition and Types (II) Designed to harm ˆ Ransomware Encrypts a victim’s files and demands a ransom for decryption [CryptoLocker, Phobos/Dharma] ˆ Botnets Networks of compromised computers used for malicious activities (DDoS attacks, spam) [Mirai, Andromeda] ˆ Logic/Time Bombs Malicious code that triggers when specific conditions are met (causing damage to system/data) ˆ Keyloggers Records keystrokes to capture sensitive information ˆ Cryptojacking Cryptocurrency mining malware (hijack a computer’s processing power) [Kinsing, LoudMiner] ”Grayware” ˆ Spyware Collects information without the user’s consent (espionage, advertising) [CoolWebSearch, Gator] ˆ Adware Unwanted advertisements and can collect user data for targeted advertising [Fireball, Appearch] 2 Malware Analysis Understand the behavior and purpose of a suspicious file ˆ detection and mitigation a potential threat ˆ triage incidents by level of severity ˆ uncover hidden indicators of compromise (IOCs) Static analysis Examining the code and characteristics of a malware sample without executing it ˆ Study file’s structure, strings, metadata, embedded resources → identify known patterns/signatures ˆ Indicators: file names, hashes, strings, IP addr, domains, file headers ˆ Tools: disassemblers, static rules (example Yara Rules) ( signature-based ˆ (+) Effective detecting known malware heuristic analysis ˆ (—) May miss sophisticated or polymorphic threats. 3 Malware Analysis (II) Dynamic analysis Running malware in a controlled environment to observe its actual behavior   virtual machine  ˆ Malware executed within a sandbox: isolated system   etc,... (to prevent to harm to the host) ˆ Monitor malware’s actions: file system access and changes, network communication (TCP, DNS,..), library loading, system calls,... ˆ Useful for detecting/analyzing unknown or evolving malware ˆ (+) Accurate understanding of the malware’s behavior ˆ (—) Resource-intensive 4 Malware Analysis (III) Resources and online sandboxes ˆ MalwareBazaar ˆ VirusShare.com ˆ Malware Traffic Analysis ˆ Microsoft Malware Classification Challenge (BIG 2015) (Kaggle) ˆ Cuckoo Sandbox (open source, sandbox running in VirtualBox) ˆ Mobile Security Framework (MobSF) (Automated static and dynamic malware analysis for mobile apps) ˆ Joe Sandbox , Joe Sandbox reports examples ˆ Hybrid Analysis ˆ VirusTotal (VT APIv3) Critical aspect in IA/ML in Malware Analysis → Feature Generation (static vs dynamic) 5 Typical features in Malware analysis Static features ˆ Opcode Sequences [sequence of operation codes within the binary code] ˆ API Import and Export Functions [specific API functions to interact with the operating system for malicious purposes] ˆ File Metadata [size, creation/modific. dates, certificates], String Analysis ˆ Control Flow Graph (CFG) [flow of control between sections ( of the code] classification → malware unique CFG patterns ⇒ ML models use them for anomaly detection ˆ File Headers and Sections [features from file header (Portable Executable (PE) header in Windows), section names (.text,.data,.rsrc)] ˆ Image Representation [malware samples transformed into grayscale images] → malware families with distinctive visual patterns ⇒ use CNN (Conv. Neural Nets) ˆ Permissions and Manifest Information (for Mobile Malware) 6 Typical features in Malware analysis (II) Dynamic features ˆ API Call Sequences and Frequencies [malware make specific system calls, in certain order and frequency ⇒ ML models & behaviour analysis] ˆ Memory Access Patterns [how and when malware accesses specific memory regions] → clues about malware behavior (privilege escalation, sensitive memory regions) ˆ Network Traffic Patterns [malware communicates with remote servers for command and control (C2) purposes, DNS requests to known malicous sites] ˆ System Call Behavior [malware use specific system calls more frequently than benign programs (file, network, process operations)] ˆ Persistence Mechanisms (startup entries, scheduled tasks) and Registry Operations (for Windows Malware) 7 Microsoft Malware Classification Challenge (BIG 2015) Dataset: https://www.kaggle.com/c/malware-classification Paper: https://arxiv.org/pdf/1802.10135.pdf Multiclass problem: 9 types of malware Raw binary data (≈ 400GB) + metadata (function calls, strings,...) 8 Microsoft Malware Classification Challenge (BIG 2015) (II) First place approach in MMC Challenge (BIG 2015) (youtube) Features (static analysis on disassembled code) ˆ opcode n-gram (1-2-3-4-gram from the disassembled machine code) [> 70K sparse features] ˆ Segment names count [≈ 400 features] ˆ Feature selection using Random Forest ”importance” [≈ 4.4K opcode n-gram features, 19 segment count] ˆ First 800 pixel intensity from viewing the asm (assembler) file as an image ˆ Other features: ≈ 2K Final classification: XGBoost 9 AI/ML in Intrusion Detection IDS/IPS Intrusion Detection/Prevention Systems (IDS/IPS) ˆ Monitor network/hosts/applications to detect dangerous activity ˆ IDS → detect and alert (pasive) ˆ IPS → detect and block (proactive) ˆ Network-based IDS vs Host-based IDS ˆ NIDS → monitor network traffic [Snort, Suricata, Zeek] ˆ HIDS → monitor host activity (file system, system calls, logs) [Fail2Ban, OSSEC/Wazuh] ˆ Signature-based IDS vs Behavior-based IDS ˆ Signature-based → known attack patterns or signatures ˆ Behavior-based → deviations from ”normal” behavior baseline 10 Behavior-based / Anomaly-based IDS/IPS Unexpected events (anomalies) → system errors/misconfigurations or malicious activity ˆ Data exfiltration, malware activity (ransonware, virus), botnet activity, reconnaissance traffic,... Anomaly-based detection → detecting activities that are statistically unusual or abnormal ˆ Baseline establishment. Baseline of normal behavior for the system/network being monitored ˆ By analyzing historical data → define typical or acceptable behavior ˆ Behavioral profiling. Continuously monitor and profile the behavior of users/systems/network traffic ˆ Aspects to monitor: data transfer volumes, protocol usage, system resource usage, login times and frequency,... 11 Behavior-based / Anomaly-based IDS/IPS (II) ˆ Anomaly detection. Statistical and ML models to assess deviations from the established baseline ˆ Statistical approaches: deviation on moving average, time series analysis, Markov models,... ˆ Supervised learning: Random Forests, Recurrent Neural Nets (RNN, LSTM), k-Nearest Neighbors,... ˆ Unsupervised learning: clustering (k-means, DBSCAN), autoencoders, outlier/novelty detection (Isolation forests, One-Class-SVM),... ˆ Continuous learning Manage baseline evolution and behavior change (concept drifting) ˆ Adaptative models/thresholds Address data seasonality and changing trends 12 Anomaly detection techniques Finding events that don’t conform to an expectation (deviate significantly from the norm) ˆ Outlier detection: identifying data points that are significantly different from the majority of the data ˆ Learning from data containing both outliers (anomalies) and ”regular” data ˆ Novelty detection: identifying instances in the data that differ significantly from what was observed during the training phase ˆ Objective: detect instances that are unseen during model’s training ˆ Learning ”regular” data representation from data that does not contains novelties (anomalies) 13 Anomaly detection techniques (II) Types of Anomalies Point Anomalies. Anomalous individual data instances that are significantly different from the rest of the dataset. Contextual Anomalies. Anomalous behavior/instances that are considered abnormal in a particular context (periods fo time, regions) or under specific conditions (they may not be anomalies when considered in isolation) Collective Anomalies. A set of data points that, when considered together, exhibit anomalous behavior 14 Anomaly detection techniques (III) Source: Chiheb Chebbi, Mastering Machine Learning for Penetration Testing, 2019 15 Anomaly detection techniques (IV) Feature engineering for Anomaly detection Host intrusion detection Metrics/signals from host and OS activity ˆ OS instrumentation and auditing frameworks: OSquery (Cross-platform endpoint instrumentation), Auditd Daemon (GNU/Linux auditing system) ˆ Typical ”signals”: ˆ Running processes, Active/new user accounts, Permission changes ˆ DNS lookups, Network connections ˆ Kernel modules loaded, System scheduler changes, Startup operations ˆ Daemon/background/persistent processes, Syscalls, CPU utilization ˆ Filesystem changes ˆ Optionally, correlate signals from different sources 16 Anomaly detection techniques (V) Network intrusion detection Features from traffic between hosts ˆ Traffic metadata (info. from packet headers) vs. deep packet inspection (info. from packet payload) ˆ Aggregated info. (transactions/connections/bytes by IP address/subnet/geolocation) vs. Individual info. (single packet data) ˆ Protocol analyzers (Zeek, CICFlowMeter [python]) and firewall logs Web/Application intrusion detection Features from application level logs ˆ Server log info: IP-level statistics, malformed URLs, user agent patterns ˆ Application log info.: login attempts, application errors, out of order requests/accesses,... 17 NIDS Datasets NSL-KDD Dataset (improved version of Knowledge Discovery in Databases (KDD) Cup 1999 dataset) ˆ ”Classical” benchmark in ML-based intrusion detection ˆ Data collected over 9 weeks on a simulated military network environment (≈ 4.9 M connection records) ˆ Raw PCAP captures → 41 processed high level features (categorical and numerical) [see KDD names] ˆ 22 types of attacks [see attack types] in 4 general categories ˆ dos [denial of service] ˆ r2l [unauthorized access from remote servers] ˆ u2r [privilege escalation attempts] ˆ probe [brute-force probing attacks] 18 NIDS Datasets (II) Criticisms and limitations of KDD Cup 1999 dataset ˆ Outdated [does not adequately represent modern cyber threats challenges] ˆ Limited representation of real-world traffic [artificially generated dataset, does not accurately reflect the diversity and dynamics of actual network traffic] ˆ Limited variety of attacks ˆ Lack of network context ˆ Focus on signature-based detection ˆ Imbalance between normal and anomalous data Alternative Datasets ˆ UNSW-NB15 Dataset ˆ CIC-IDS2017 and CSE-CIC-IDS2018 on AWS ˆ UGR’16 Dataset 19