Untitled

Key Vocabulary Extraction

• ACID Transactions encompass Atomicity, Consistency, Isolation, and Durability, ensuring reliable database transaction processing.
• Batch Data involves data processed in groups, collected over time rather than individually.
• Data Analysis extracts insights from data, recognizing patterns and interpreting information types.
• Data Independence allows modifying data storage structures without affecting dependent applications.
• Data Literacy enables the competence to read, interpret, create, and communicate data effectively.
• Implicit Information is knowledge inferred from data analysis rather than explicitly stated.
• Relational Database Management Systems (RDBMS) manage data organized in relational schemas (tables).

Schema & Data Types

• Schema organizes or structures data, defining data storage in a database.
• Static Block Data includes fixed data stored in a constant form, typically in databases.
• Streaming Data involves continuously generated and updated data, often processed real-time.
• Structural Information regards the data's organization and arrangement within a dataset or database.
• Type Safety ensures values match data types, which avoids manipulation errors.
• Data types in Relational Tables include int, char, varchar, date, and decimal.

Data Management and Analysis

• Goals of Data Management focus on efficient large-dataset handling, transformation, and storage.
• Data Management concerns data storage and handling. Data Analysis seeks to extract insights and patterns from data.
• Static Data is fixed, while Streaming Data is continuously updated.
• Batch Processing methodology processes data in bulk rather than one at a time.

Relational Database Systems

• ACID Properties are principles ensuring transaction reliability in databases.
• Tables and Relationships comprise core relational-model components, structured as rows (tuples) and columns (attributes).
• Entity-Relationship Diagrams (ERD) visualize entities (tables), their attributes, and relationships.

Theoretical Foundations: Normal Forms and Keys

• First Normal Form (1NF) ensures all columns contain atomic values, which promotes data integrity by avoiding redundancy.
• Foreign Keys link tables within the relational model, enabling data relationships.

Methodologies

• Data Management Practices guide policies and techniques that ensure data consistency, availability, and fault tolerance.

Challenges in Pre-Big-Data Era

• Limited Data Volume Handling includes struggles with integrating diverse data sources.
• Slow processing capabilities exist due to older technologies.

Key Vocabulary Extraction

• Data Warehouse (DWH) is a centralized repository for storing and managing historical data from various sources.
• Decision Support System (DSS) supports business or organizational decision-making activities.
• Executive Information System (EIS) supports executive decision-making processes with internal and external information access.
• Integrations combine data from different sources into a single, unified view.
• Online Analytical Processing (OLAP) software enables insight into data through fast, consistent access.
• Online Transaction Processing (OLTP) software manages transaction-oriented applications for data entry and retrieval.
• Transitions can be Subject-Oriented, organized around areas of interest in decision-making.
• Transitions are Time Variant, the characteristic of data that changes over time for historical analysis.
• Transitions are Non-Volatile, where once loaded, data does not change or get deleted, which ensures data integrity for analysis.

ACID Properties Explained

• Atomicity means transactions are all-or-nothing.
• Consistency means transactions must transition data from one valid state to another.
• Isolation means transactions operate independently from one another.
• Durability means once a transaction commits, it remains so, even in case of system failure.

Typical Scale and SQL Purpose

• Typical Scale for Read/Write Operations: Operations occur on data scales ranging from MBytes to GBytes.
• SQL is the primary language for managing and enabling operations like data retrieval and manipulation of RDBMS.

Historical Perspectives and Data Privacy

• The Evolution of Database Models includes advancements from the Hierarchical Database Model to the Relational Database Model.
• Data Privacy necessitates understanding ethical and legal repercussions in data handling, especially with sensitive data.

Historical Context and DWH Role

• Historical Context: Data management systems evolved from 1960s operational databases to the integration of data warehouses in the 1990s.
• The cost-effective storage solutions include DWH storage for integrated data from various sources.

OLTP vs. OLAP

• Operational Systems (OLTP) focus on transaction processing with fast queries and are optimized for high throughput and real-time.
• Analytical Systems (OLAP) focus on complex queries over historical data, supporting strategic decision-making with slower response times.

Characteristics of Data Warehouses

• Integrated and Consistent Data integrates data from multiple operational sources into a consistent format for analysis.
• Read-Only Access data ensures continuous data analysis
• Support for Decision Making serves as the basis for DSS and BI applications, enabling deeper analysis via OLAP.

Multidimensional Data Modeling

• Dimensions and Hierarchies utilize data cubes to allow analysis from multiple perspectives.
• Corporate KPIs, like sales and profits, are the facts that are monitored across various dimensions.

Significant Definitions and OLAP Operations

• Data Warehouse is a subject-oriented, integrated, non-volatile, and time-variant collection of data in support of management decisions.
• Typical operations in OLAP include complex analytical queries with aggregate functions over large datasets.

Data Warehouse Characteristics & Principles

• Subject-Oriented, data are aligned aligning with relevant business metrics.
• Integrated Data means consistency across data sources enables reporting and analytics.
• Non-Volatility: Data is immutable, preserving historical accuracy for analyses.
• Time Variability: Tracking changes over time is critical for examining trends.
• OLAP systems require separate architectures from OLTP systems to properly handle complex queries without impacting real-time transaction processing.
• Data integration involves extracting data from a variety of sources, transforming it into a uniform format, and loading it into the DWH.

Key Vocab: ACID, BASE, CAP Theorem

• ACID ensures reliable processing of database transactions.
• BASE is used in NoSQL databases emphasizing availability over strict consistency.
• CAP Theorem: A distributed data store provides all guarantees: Consistency, Availability, and Partition Tolerance.

Key Vocab: Consistency Levels, Scaling, Key-Value Store, NoSQL

• Horizontal Scaling adds more machines to improve load handling (contrasts vertical scaling that improves a single machine's capacity).
• Key-Value Store NoSQL database stores data in pairs of keys and values for quick retrieval based on keys.
• NoSQL refers to a range of database systems that do not adhere strictly to the relational database model.

Introduction to NoSQL

• NoSQL movement began in the mid-2000s as the limitations of Traditional RDBMS in horizontally scaling to manage large volumes of unstructured data.
• Modern applications and web platforms generate massive datasets requiring flexible database solutions

Consistency Models & The CAP Theorem

• ACID vs. BASE helps offers higher availability but relaxes strict consistency.
• BASE helps in scenarios where temporary inconsistency is acceptable.
• Prioritize Consistency and Partition Tolerance but sacrifice Availability (CP Systems)

NoSQL Data Models

• Key-Value Stores stores data in pairs efficiently
• Document Stores store data as documents, often in JSON format, which allows complex queries.
• Wide Column Stores organize data into rows and columns but allow for flexible column families.
• Graph Databases focus on relationships and connections between data points.

Benefits and Challenges of NoSQL

• Benefits include high throughput, horizontal scalability, and simplified data models.
• Challenges include writing challenges, potential loss of consistency, and complex maintenance of databases.

Statistics and Formulas

• Horizontal Scaling distributes data across multiple nodes for improved performance
• Vertical Scaling increases resources for a single node because it is limited by hardware limitations.
• CAP Theorem has three properties and the any networked shared-data system can have at most of the two properties.
• Eventual Consistency updates are propagated at a later point
• Strong Consistency guarantees atomic actions

Additional Critical Aspects

• CAP Theorem outlines trade-offs among consistency, availability, and partition tolerance.

Methodologies: Partitioning & Replication

• Data Horizontal Partitioning breaks down large datasets into smaller partitions without altering the overall structure.
• Data Replication maintains multiple copies of data across nodes to ensure availability and fault tolerance
• Data includes Strings, Hashes, Lists, Sets, and Ordered Sets

ANN & Embedding

• ANN (Approximate Nearest Neighbor) improves search efficiency in database queries
• Embedding maps discrete objects represented by similar vectors
• MBR (Minimum Bounding Rectangle) contains a multi-dimensional space
• HNSW (Hierarchical Navigable Small World) efficiently supports approximate nearest neighbor searches

More Query Types

-K-NN (k-Nearest Neighbor): A query type that retrieves object -R-Tree balanced tree data structure -Similarity Query retrieves data similarity -Vector Database is optimized for analyzing and machine learning

Motivation & Funcionality for Vector DBs

• Traditional databases struggle with vector embeddings in Al applications.
• They facilitate similarity queries where intuitive meanings may not apply.

Indexing Embedding Spaces

• Hierarchical Trees: data is divided into pages
• Locality Sensitive uses computation speeds
• Learned Indexes: ML helps in optimizing queries

Structure & Definition: Database Queries

-Range Query: certain distance objects

• K-Nearest Neighbor Query: the “k* closest items
• Ranking Query is primarily proximity based for easy retrieval

R-Tree Structure

-Basic Structure: directory entries and leaf nodes

• Efficiency helps reduce search

Vector DB Distance

• Dist(x,y) = √∑i=1n(xi−yi)2
• Range Query = RQ(q, Є) = {o ∈ DB | dist(q, 0) < Є}.
• K-NN Query = NN(q, k) ⊆ DB such that |NN(q, k)| = k.

Data Structures

• R-Trees manage spatial data
• HNSW Graph supports efficient ANN queries.