Data Integration Methods and Challenges

Data Strategy

• Data leadership is about understanding the organization's relationship with data and finding ways to meet goals using available tools.
• A data architect should understand business operation goals, the organization's overall goals, and the fundamental direction of the business.
• Answers to these questions lead to a detailed understanding of how to achieve organizational goals.
• Examples of questions include: how to source and market products, how to connect with customers, and how to deliver products.
• Data should support both the business' overarching goals and the processes that help achieve them.

Data Architecture

• Start with the most valuable data and consider how it supports the organization's primary objectives.
• Understand how the data relates to specific teams and their goals, and how it connects the technological and business aspects of the organization.
• Use data to generate relevant, tangible insights that benefit the organization.
• Data governance is essential for managing and controlling information within the architecture.
• Instead of focusing on a permanent framework, create one that adapts to the evolving needs of the organization.
• Data architectures should facilitate real-time information access for stakeholders.
• Data should be treated as a service to users.
• Data should be visualized to be more impactful.

Stakeholders in Data Architecture

• A data architect (big data architect) defines the data vision, translates it to technology requirements, and defines data standards.
• A project manager oversees data flow modifications and creations.
• A solution architect designs data systems to meet business requirements.
• A cloud architect or data center engineer prepares the infrastructure for data systems.
• A DBA or data engineer develops data systems, sets data quality, and manages data feeds.
• A data analyst uses the architecture for reports and insights.
• Data scientists use the architecture to find insights from the organization's data.

Data Architecture Frameworks

• DAMA-DMBOK 2.0 is a framework for data management.
• The Zachman Framework provides an enterprise ontology including architectural standards, semantic models, and logical/physical data models.
• TOGAF is an enterprise architecture methodology with Phase C for developing and roadmapping data architectures.

TOGAF Phase C1: Objectives

• Define the types and sources of data to support the business in a way that is understandable, complete, and consistent, as well as stable.
• Define the data entities relevant to the enterprise.
• Avoid designing logical or physical storage systems or databases.

TOGAF Phase C1: Overview

• The process involves defining reference materials, non-architectural inputs, architectural inputs and steps.
• The steps will output data architecture descriptions, perform a gap analysis and define roadmap components.
• Finally, generate a formal stakeholder review and create an architecture definition document.

TOGAF Phase C1: Approach-Key Considerations

• Data management: Understand and address data management issues by adhering to a structured and comprehensive approach.
• Data definition: Clearly define application components that serve as a system of record or reference for enterprise master data.
• Business function: Understanding how data entities are used in business function, processes, and services is crucial.
• Data transformation: Understand how data transformations are carried out.
• Data integration: Data integration with external organizations is important.

Data Migration

• Identify data migration requirements for new or changed applications.
• Establish high-quality data in the target application from the start.
• Establish enterprise-wide common data definitions to support transformations.

Data Governance

• Ensure the organization has necessary dimensions to facilitate data transformations.
• Use standards and bodies for successful management of data entities during transformation.
• Implement a data management system and programs.
• Identify the necessary data-related skills and roles within the organization.

TOGAF Phase C1: Outputs

• Improved and updated Architecture Vision phase deliverables (e.g., Statements of work, validated data principles and business drivers).
• Drafts of Architecture Definition Documents listing baseline data architecture, target data architecture, data management process models, data entity tables, views to address stakeholder concerns, and required technical specifications.

Why the DMBOK2?

• The DAMA-DMBOK Guide is a collection of processes and best practices.
• It defines data discipline-specific best practices and references.
• Data management includes processes like planning, specifying, enabling, creating, acquiring, maintaining, using, archiving, retrieving, controlling, and purging data.

What is the purpose of the DMBOK

• Standardize data activities, processes, and best practices, alongside clarifying roles, responsibilities, deliverables, and metrics.
• A comprehensive framework helps practitioners perform more consistently and effectively.

DAMA-DMBOK2

• 2013 knowledge areas: Data architecture, data quality, metadata, data warehousing and business intelligence, data modeling, data governance and more.

The 11 Data Management Knowledge Areas

• Data governance: Planning, oversight, and control of data usage.
• Data architecture: The structure of data and related resources within the enterprise.
• Data modeling and design: Analysis, design, and maintenance of data implementation.
• Data storage & operations: Physical data storage deployment and management.
• Data security: Enforces privacy, confidentiality, and appropriate access to data and ensuring network security.

Data Integration & Interoperability

• Data acquisition, extraction, transformation, movement, delivery, replication, federation, virtualization, and operational support.
• Handling documents, content, storing, protecting, indexing, and enabling access to data from unstructured sources.
• Establishing clear definitions and values for reference and master data.

Manual Data Integration

• Pros: low cost. Greater freedom
• Cons: Limited access, difficult scaling, room for error.

Middleware Data Integration

• Pros: Better data streaming, easier access
• Cons: Less access, limited functionality

Application-Based Integration

• Pros: Simplified processes, easier information exchange
• Cons: Limited access, inconsistent results, problematic setup, and difficult management.

Uniform Access Integration

• Pros: Lower storage requirements, easier access to data, simplified view for users
• Cons: Data integrity issues, strained systems.

Common Storage Integration

• Pros: Reduced processing burden, cleaner data appearance, improved data analytics
• Cons: Increased storage costs, higher maintenance needs.

Modern Data Architectures

• Cloud-native designs support high scalability, availability, security, and performance.
• Scalable data pipelines handle real-time streaming and micro-batch data bursts.
• Architectures support data integration using APIs for seamless functionality across systems.
• Data validation, classification, governance, and deployment should be automated using real-time data enablement.
• Loosely-coupled service deployment allows for minimal dependencies.

Data Integration Tools

• Presented list of data integration tools.

The Five Ws (5W1H)

• Basic questions utilized in information gathering and problem-solving.
• Includes questions like Who, What, When, Where, Why and How.

Scope/Executive/Planner

• Data analysis from the perspective of enterprise goals.

Business/Owner

• Identifying important data entities.
• Defining how information entities relate to one another.

Architect/Designer

• E/R model extraction
• E/R model normalization
• Identifying and linking data entities to processes.
• Extracting data entities and their identifiers.

Engineer/Builder

• Converting the E/R model into a data model.
• Normalizing the data model.
• Defining and analyzing transactions and questions that would be run on the data.
• Defining file structures, indices, and other relevant database attributes.

Technician/Subcontractor

• Creating database management systems and database architectures.
• Establishing access levels and data control information.
• Defining the data management program for user interfaces.
• Providing maintenance scenarios and managing database performance.

Data Integration Methodologies

• Manual Data Integration, Middleware Data Integration, Application-Based Integration, Uniform Access Integration, Common Storage Integration.