nvidia-notes.docx

11/09/2024 17:55 **1. Notes extracted from \"Running Your Own LLM\"** 1. NVIDIA LLM Inference Stack: - TRT-LLM uses tensor parallelism for efficient multi-GPU execution - Enables running large models like LLAMA2-70B across multiple GPUs without manual splitting...

11/09/2024 17:55 **1. Notes extracted from \"Running Your Own LLM\"** 1. NVIDIA LLM Inference Stack: - TRT-LLM uses tensor parallelism for efficient multi-GPU execution - Enables running large models like LLAMA2-70B across multiple GPUs without manual splitting - Supports various precisions: FP8, BF16, INIT4 for fine-grained quantization control 2. TRT-LLM Features: - In-flight batching (continuous batching) for higher GPU utilization - Modular Python API for easy customization and extension - Pre-built examples for popular model architectures 3. Inflight Batching: - Improves GPU usage and doubles throughput - Reduces energy costs and TOC - Immediately processes new requests when earlier ones complete 4. Model Quantization: - Can improve throughput and latency - Helps with portability and scale - Important to test for specific use cases 5. Running LLMs on Windows: - Uses Triton Inference Server on WSL - Provides low latency, high availability, and data privacy - Requires 64-bit Windows OS 6. NVIDIA AI Foundry: - DGX Cloud for training and fine-tuning at scale - Pre-trained and enterprise-optimized models available - Reference workflow for RAG implementation 7. NVIDIA AI Enterprise: - Provides branch support and maintenance - Includes bug fixes and critical security updates - NGC container scanning for vulnerability checks 8. Multimodal LLMs: - Understand text and images - Enable new applications impossible with text-only models - Example: NEMO Vision and Language Assistant 9. LLM Agents and Plugins: - Capable of reasoning and autonomous task execution - Use chain of action techniques - Plugins enhance capabilities for various tasks Glossary of NVIDIA-specific tooling: 1. TRT-LLM: TensorRT-based library for LLM inference optimization 2. TensorRT: Deep Learning compiler for NVIDIA GPUs 3. NVLink: High-bandwidth interconnect for NVIDIA GPUs 4. CUDA: Parallel computing platform for NVIDIA GPUs 5. NGC: NVIDIA GPU Cloud, a hub for GPU-optimized software 6. DGX Cloud: NVIDIA\'s cloud service for AI development 7. NEMO: NVIDIA\'s framework for generative AI models 8. Triton Inference Server: Scalable inference serving software 9. WSL: Windows Subsystem for Linux 10. NVIDIA AI Enterprise: Software suite for AI development and deployment 11. NVIDIA AI Foundry: Platform for building and customizing AI models 12. NVIDIA Inception: Program for AI startups 13. Deep Learning Institute: NVIDIA\'s educational platform for AI and deep learning **2. NVIDIA Glossary by Perplexity** **NVIDIA NeMo Framework** NeMo is NVIDIA\'s end-to-end, cloud-native enterprise framework for building, customizing, and deploying generative AI models. It includes: - **NeMo Core**: Foundational elements like the Neural Module Factory for training and inference. - **NeMo Collections**: Specialized modules and pre-trained models for ASR, NLP, and TTS. - **NeMo Curator**: GPU-accelerated data curation tool for preparing large-scale, high-quality datasets for LLM pretraining. - **NeMo Customizer**: Scalable microservice for fine-tuning and aligning LLMs for domain-specific use cases. - **NeMo Retriever**: High-performance, low-latency information retrieval for enhancing generative AI applications with enterprise-grade RAG capabilities. - **NeMo Guardrails**: Tool for adding programmable guardrails to control LLM application output. **NVIDIA TensorRT-LLM** An open-source library for optimizing LLM inference on NVIDIA GPUs. It supports FP8 format conversion and compilation to leverage optimized FP8 kernels on NVIDIA H100 GPUs. **NVIDIA AI Enterprise** A cloud-native suite of AI and data analytics software, including over 50 frameworks, pretrained models, and development tools optimized for GPU infrastructures. **NVIDIA AI Workbench** A platform for managing data, models, resources, and compute needs, enabling seamless collaboration and deployment for generative AI model development. **NVIDIA Base Command** Management and orchestration tool for large-scale AI workloads on multi-node, multi-GPU configurations. **NVIDIA AI Inference Manager (AIM) SDK** Provides a unified interface for orchestrating AI model deployment across various devices and inference backends. **NVIDIA RTX AI Toolkit** A suite of tools and SDKs for Windows developers to customize, optimize, and deploy AI models on RTX PCs and cloud. It includes: - **LlamaFactory GUI**: Tool for QLoRA fine-tuning of LLMs. - **AI Workbench LLaMa-Factory Project**: Reference project for LLM customization. **NVIDIA Triton Inference Server** Standardizes AI model deployment and enables analysis of model performance. **NVIDIA-specific Libraries and Tools** - **Megatron-LM**: Library for training large transformer language models. - **CUTLASS**: Collection of CUDA C++ templates for implementing high-performance matrix multiplication. - **RAFT**: Collection of GPU-accelerated machine learning primitives. - **cuDNN**: GPU-accelerated library of primitives for deep neural networks. **Deployment and Inference Options** - **TensorRT-LLM**: Supports both LoRA adapters and merged checkpoints for on-device inference. - **llama.cpp**: Supports LoRA adapters for on-device inference. - **ONNX Runtime - DML**: Supports LoRA adapters for on-device inference. - **vLLM**: Supports both LoRA adapters and merged checkpoints for cloud inference. - **NVIDIA Inference Microservices (NIMs)**: Supports LoRA adapters for cloud inference. **3. NVIDIA Specific Tooling from Perplexity** **spaCy** spaCy is an open-source Python library for advanced natural language processing (NLP). It provides fast and efficient tools for tasks like tokenization, part-of-speech tagging, named entity recognition, and dependency parsing\[9\]. **NumPy** NumPy is a fundamental library for scientific computing in Python. It provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays efficiently\[4\]. **NVIDIA RAPIDS** NVIDIA RAPIDS is an open-source GPU-acceleration platform for large-scale data analytics and machine learning. It includes various libraries that integrate with popular data science tools and accelerates them using GPUs\[5\]. **cuDF** cuDF is a GPU DataFrame library built on the Apache Arrow columnar memory format. It provides a pandas-like API for loading, joining, aggregating, filtering, and manipulating tabular data using NVIDIA GPUs\[7\]. **Dask cuDF** Dask cuDF extends cuDF to enable parallel computing across multiple GPUs. It allows for scaling out workflows and processing larger-than-memory datasets by partitioning them across GPU clusters\[7\]. **cuML** cuML is a suite of fast, GPU-accelerated machine learning algorithms designed for data science and analytical tasks. It provides both GPU-based and CPU-based execution capabilities with a scikit-learn-like API\[10\]. **cuGraph** cuGraph is a GPU-accelerated graph analytics library that is part of the NVIDIA RAPIDS suite. It provides high-performance implementations of graph algorithms for large-scale network analysis\[8\]. **Apache Arrow** Apache Arrow is a cross-language development platform for in-memory data, defining a standardized language-independent columnar memory format for flat and hierarchical data\[1\]. **NVIDIA NeMo** NVIDIA NeMo is an open-source toolkit for building state-of-the-art conversational AI models. It provides tools and pre-trained models for tasks like speech recognition, natural language processing, and text-to-speech\[3\]. **NVIDIA Triton** NVIDIA Triton Inference Server is an open-source software that streamlines AI inferencing. It enables teams to deploy AI models from multiple deep learning and machine learning frameworks across various hardware platforms\[2\]. **TensorRT** TensorRT is a C++ library developed by NVIDIA that facilitates high-performance inference on NVIDIA GPUs. It optimizes trained neural networks for inference, producing highly optimized runtime engines\[6\]. **NCCL (NVIDIA Collective Communications Library)** NCCL is a library providing inter-GPU communication primitives that are topology-aware. It implements collective communication operations like AllReduce, Broadcast, and AllGather, as well as point-to-point communication, optimized for NVIDIA GPUs\[12\]. Citations: \[1\] [https://arrow.apache.org](https://arrow.apache.org/) \[2\]  \[3\]  \[4\]  \[5\]  \[6\]  \[7\]  \[8\]  \[9\]  \[10\]  \[11\]  \[12\]