Data Management Overview

Why Data Management Matters

• Enterprises need high-quality, accessible data to be effective
• Data should be complete and in context to be reliable
• It is estimated that organizations with at least 1,000 knowledge workers lose approximately $5.7 million annually due to employees spending time reformatting data across different applications

Goal of Data Management

• Data management focus on providing the infrastructure and tools necessary to convert raw data into high-quality, usable information

Data Management Challenges

• Exponentially increasing volumes of data
• The scattering of data throughout organizations
• Creation and usage of data offline without quality control
• Possible redundancy and staleness of data

Current Key Issues in Data Management

• Master Data Management (MDM) involves integrating data from various sources and enterprise applications to create a unified view
• Document Management System (DMS) offers hardware and software for file management, archiving, and purging electronic documents
• Green computing is about conserving resources and minimizing the environmental impact of computer usage

Costs of Data Errors

• Healthcare administrators waste 24%-30% of their time correcting data errors
• Each incorrect transaction can cost $60-$80 to fix
• Approximately 60% of invoices among supply chain partners contain errors, each costing $40-$400 to reconcile
• Billions of dollars are lost annually in the healthcare supply chain due to data disconnects

Benefits of Data Synchronization

• Faster and simpler product sourcing because of accurate and consistent item information
• Reduced fraud and unauthorized purchasing
• Reduced unnecessary stock
• Lower prices due to purchase volume awareness
• Increased patient safety

Stand-alone Systems

• Systems in isolation lead to redundancy, inconsistency, and isolation in data

Database Management Systems (DBMS)

• DBMS is a software for controlling the creation, maintenance, and usage of databases

Database Types

• Centralized database: a database located, stored, and maintained in a single location
• Distributed database: a database stored on multiple computers and locations

Distributed Databases: Advantages

• Security
• Reliability
• Speed
• Scalability

Distributed Databases: Disadvantages

• Availability
• Expense
• Security

Centralized Databases: Advantages

• Better control of data quality
• Ease of use
• Better IT security
• Better data integrity

Centralized Databases: Disadvantages

• Transmission delay
• Security
• Reliability
• Scalability

Data Warehouses, Data Marts, and Data Centers

• Data warehouses are integrated collections of data
• Data marts hold subsets of data from a data warehouse
• Data centers are facilities that house mission-critical ISs and components for data and IT service delivery

Current Database Types

• Operational: Stores detailed data; for example, a customer database uses Microsoft Access database management software
• Distributed: Replicates and distributes copies/parts of databases to network servers; for example, data on the World Wide Web, intranets, and extranets
• External: Provides access to online information, such as search engines like Google and Yahoo, and news outlets and magazines
• Hypermedia: Uses hypertext and hypermedia documents stored on websites, for example, graphics, images, photos, and video clips

Data Planning and Modeling

• Database administrators and designers work with corporate and end-user management to develop an enterprise model
• Data modeling is identifying the relationships between data elements
• Logical elements are data elements and relationships among data
• Physical - data are stored and accessed on the storage devices of a computer system

Online Transaction Processing (OLTP)

• OLTP is designed for managing transaction data
• Volatile and break down complex information into simpler data tables
• Balances transaction-processing efficiency and query efficiency
• Not optimized for data mining

Online Analytics Processing (OLAP)

• OLAP includes organizing large business databases
• Data is divided into one or more cubes that fit business needs

DBMSs (mid-2014)

• Oracle's MySQL
• Microsoft's SQL Server
• PostgreSQL
• IBM's DB2
• Teradata Database

Trend Toward NoSQL Systems

• Provides higher performance
• Allows easy data distribution across different nodes
• Enables scalability and fault tolerance
• Offers greater flexibility
• Easier to administrate

Dirty Data Explained

• Lacks integrity/validation, reduces user trust
• Incomplete, out of context, outdated, inaccurate, inaccessible, or overwhelming

Principle of Diminishing Data Value

• Data value decreases as it ages
• Blind data spots lasting 30 days or longer inhibit optimal performance
• Global financial service institutions need real-time data to hit peak performance

Principle of 90/90 Data Use

• Up to 90% of data is rarely accessed after 90 days, except for auditing
• Approximately 90% of data loses most of its value after three months

Data Life Cycle

• Internal and external data is combined into a central data warehouse
• Personal expertise and judgement is used to manipulate the data for analysis
• Results will be visualized
• Can be used in SCM, CRM, e-commerce etc

Data Silos

• Data silos are stand-alone data stores not accessible by other information systems or departments
• Information requirements include: understandable, relevant, timely accurate and secure

Elements of a DBMS

• Four important elements: data structure, data modeling language, data query language, and transaction mechanisms

Data Structure

• A specialized format for organizing and storing data
• General data structures include: files, record, table, tree

Data Hierarchy

• Bit: the smallest unit
• Byte: eight bits/one character
• Field: one item within record
• Example - last name
• Record: Set of related fields Example – employee #, name, pay rate, etc.
• File: Set of related records
• Database: Entire collection of files

File Access

• Sequential access: stores record in a sequence
• Random access: stores record in random order
• Indexed Sequential Access Method (ISAM): stores sequentially but also stores randomly

Limitations of the File Environment

• File processing has become cumbersome, costly, and inflexible in supplying needed data
• This makes data management more difficult
• There is also duplication of data
• This leads to data inconsistency and data isolation
• There are also data integration issues
• This leads to difficulty and time constraits

Structure Query Language (SQL)

• The most widely-used mainstream declarative language that works with any database
• Simplifies data access by requiring users to declare what data they want instead of telling the DBMS how to get it

Three Steps to move from Database to Data Warehouse or Data Mart

• Extracted from designated databases
• Transformed by standardizing formats, cleaning the data, and integrating them
• Loaded into a data warehouse

Active Data Warehouse (ADW)

• The ADW captures transactions as they change and integrates them into the warehouse with batch or scheduled cycle refreshes
• Companies using ADW can interact with customers for better service, respond to business events in near real-time, and share up-to-date data among merchants, vendors, customers, and associates

Data Lakes

• Data lakes provide a single store of structured, semi-structured, and unstructured enterprise data stored in its natural format

Data Governance

• Data governance are rules over availability, usability, integrity, and security of data
• Master data is a term used to describe business-critical information on customers, products, locations, employees and other things needed for operations and transactions
• Master data management (MDM) integrates data from various sources and enterprise applications for a unified view

Enterprise data warehouses (EDW)

• Data warehouses pull together data from disparate sources and databases
• Data Warehouses are the primary source of cleansed data for analysis, reporting, and Business Intelligence (BI)
• Data marts may subsidize warehouses high costs

Active Data Warehouse (ADW)

• Real-time data warehousing and analytics
• Transform by standardizing formats, cleaning the data, integration

Business Analytics

• The entire function of applying technologies, algorithms, human expertise, and judgment

Data Mining

• Software that enables users to analyze data from various dimensions or angles
• Helps categorize them and find correlative patterns among fields in the data warehouse

Data and Text Mining Concepts

• Text Mining: Consists of broad category involving interpreted words and concepts in context
• Sentimental Analysis: Understanding the consumer intent

Electronic Records Management

• Business Records is a documentation of a business event, action, decision, or transaction
• Electronic Records Management (EMR) are tools, scanners, and databases that manage and archive electronic documents, and image paper documents
• Index and store documents according to company policy or legal compliance
• Success depends on partnering with key players

Business Value of E-Records Management

• It is important for companies to abide by legal actions against them
• E-discovery is the process of gathering electronically stored information for a trail, legal, investigation or administrative action
• A company must produce what is requested and risk charges of obstructing justice