SOEN 363 Lecture 1: Intro to Data Systems

• SOEN 363 is Data Systems for Software Engineers
• This is for Lecture 1: Introduction

Course Outline

• Motivation for studying data systems
• Course overview and administrative details
• A primer on databases

Motivation

• The 21st century is seeing breakthroughs in gene sequencing, biotechnology, ubiquitous computing, faster communication, and smaller, cheaper sensors
• A common theme across these breakthroughs is the increasing amount of data
• Data amount is rapidly growing; in 2010, there were 1.2 zettabytes (1ZB = 10^21 B or 1 billion TB)

Data Growth

• There are nearly 500 Exabytes generated per day by the Large Hadron Collider experiments
• 2.9 million emails are sent every second
• 20 hours of video are uploaded to YouTube every minute
• Google processes 24 PBs of data every day
• 50 million tweets are generated daily
• 700 billion minutes each month are spent on Facebook
• 72.9 items are ordered on Amazon every second

Data and Big Data

• Data's value as an organizational asset is widely recognized
• Data growth is occurring in three main dimensions: "Volume", "Velocity", and "Variety"
• Big Data is the proliferation of data that floods organizations daily
• Big Data is high volume, high velocity, and/or high variety information assets
• Fast mining, enhanced decision-making, insight discovery, and process optimization requires new forms of processing

Data Utilization

• Data is used for storing, sharing, querying, mining, and encrypting and must be done seamlessly and fast
• Data is accessed, shared, and processed from diverse interfaces and devices anytime, anywhere
• Data is becoming critical to health, education, environment, science, work, and finance

Studying Databases

• Data exists everywhere and is critical
• Data needs to be recorded, maintained, accessed, and manipulated correctly, securely, efficiently, and effectively
• DBMSs (Database management systems) are indispensable software for achieving such goals
• The principles and practices of DBMSs are now an integral part of computer science curricula
• They encompass OS, languages, theory, AI, multimedia, and logic, among others
• The study of database systems can prove to be richly rewarding

Database Modeling and Flat Files

• Example: Model a database for the university
• Problems with flat files include scaling, integrity, and system recovery issues as well as concurrent edits
• Other issues: Building another application and changes to how the data is physically stored

Relational Data Model

• In the 1970s, programmers were rewriting IMS code every time the database schema changed
• Abstract databases decouple logical structure from physical structure, store databases in a simple data structure and use high-level languages to access data
• Physical storage is left to the DBMS implementation

Course Objectives

• Design and implement databases from 'cradle-to-grave'
• Query and manipulate databases
• Refine and speed up data retrieval and manipulation
• Construct buffer and disk space managers, query optimizers, and concurrency managers for DBMSs
• Big Data, Hadoop, BigTable, parallel and distributed DBMSs, NoSQL and NewSQL databases

NoSQL

• A new class of databases that mainly follow the BASE properties emerged
• These were dubbed as NoSQL databases and include Amazon's Dynamo and Google's Bigtable
• Main characteristics include no strict schema requirements and adherence to ACID properties, while consistency is traded in favor of availability

Types of NoSQL databases

• Document Stores
• Graph Databases
• Key-Value Stores
• Columnar Databases

List of Topics

• Entity-Relationship Model
• The Relational Model
• SQL
• Data Storage and Organization
• Tree-Based and Hash-Based Indexing
• Query Evaluation and Optimization
• Advanced Topics: Distributed Databases, Hadoop, and NoSQL and NewSQL Databases

Learning Outcomes

• Describe a wide range of data involved in real-world organizations using the entity-relationship (ER) data model
• Explain how to translate an ER diagram into a relational database
• Indicate how SQL builds upon relational calculus and algebra and effectively apply SQL to create, query and manipulate relational databases
• Appreciate how DBMSs work
• Manipulate and manage files of fixed-length and variable-length records on disks
• Create and operate various static and dynamic tree-based (e.g., ISAM and B+ trees) and hash-based (e.g., extendable and linear hashing) indexing schemes
• Explain and evaluate various algorithms for relational operations (e.g., join) using techniques such as iteration, indexing and partitioning
• Analyze and apply different query evaluation plans and describe the various tasks of a typical relational query optimizer
• Identify alternative architectures for distributed databases, and describe how data can be partitioned and distributed across networked nodes of a DBMS
• Appreciate the scale of Big Data and discuss popular analytics engines for Big Data processing and denote the applicability of NoSQL databases for Big Data storage