ECS Concepts and Features Participant Guide PDF

ECS CONCEPTS AND FEATURES PARTICIPANT GUIDE PARTICIPANT GUIDE Table of Contents Introduction to ECS 5 Data Transformation 5 Todays Challen...

ECS CONCEPTS AND FEATURES PARTICIPANT GUIDE PARTICIPANT GUIDE Table of Contents Introduction to ECS 5 Data Transformation 5 Todays Challenges 5 ECS Solution and Benefits 11 ECS Value Proposition 13 Object Storage Evolution 15 ECS Features 17 ECS Use Cases 18 ECS Products 32 ECS Software Licensing Options 33 Knowledge Check 34 Elevator Pitch So Far - Introduction to ECS 35 ECS Storage Structure 36 ECS Storage Layers 36 ECS Hardware Building Blocks 38 Physical Storage 42 Logical Storage 45 Users and Roles 50 ECS Data Storage Principles - Chunks 53 Knowledge Check 55 Elevator Pitch So Far - ECS Storage Structure 56 ECS Architectural Overview 57 ECS Architecture Layers 57 ECS Hardware Layer 59 ECS Node Options 60 ECS Infrastructure Layers 65 Fabric Layer 71 Storage Engine Layer 77 Data Services Layer 78 ECS Concepts and Features © Copyright 2023 Dell Inc Page 2  ECS Portal and Provisioning Layer 81 Knowledge Check 83 Elevator Pitch so Far - ECS Architectural Overview 84 ECS Features and Functionality 85 ECS Features and Functionality Overview 85 ECS Portal 87 Multi-tenancy and Namespaces 89 User Authentication and Authorization 91 Multi-protocol Support 92 S3 Feature Support 94 Knowledge Check 97 Local Data Protection 98 Multi-VDC, Multi-site Data Replication 104 Site Failure Handling 108 Data Encryption 113 Retention and Compliance 115 Quotas 121 Security 123 Identity and Access Management - IAM 125 Knowledge Check 127 Elevator Pitch So Far - ECS Features and Functionality 128 ECS Monitoring and Reporting 129 ECS Monitoring 129 ECS Portal Dashboard 131 ECS Portal Navigation - Monitor 133 Advanced Monitoring 135 Dell SRM Integration 137 Secure Connect Gateway 139 ECS Health Checklist 140 Knowledge Check 144 Elevator Pitch so Far - ECS Monitoring and Reporting 145 Final ECS Elevator Pitch Summary 146 ECS Concepts and Features © Copyright 2023 Dell Inc Page 3  Final ECS Elevator Pitch Summary 146 Course Completion 148 ECS Concepts and Features © Copyright 2023 Dell Inc Page 4  Introduction to ECS Introduction to ECS Data Transformation Lately, significant disruptions like the pandemic have quickly led organizations to reshape their operational strategies. The adoption of remote collaboration tools has become essential, triggering evaluations of IT infrastructure for energy efficiency, hybrid work models, and multi-cloud strategies. Despite facing challenges, these circumstances have also given rise to new prospects, fostering breakthroughs in vaccine development and digital connectivity. As the world becomes increasingly interconnected, the significance of cyber protection strategies has surged due to escalating cyber threats. The need to effectively manage the exponential growth of data across the enterprise is anticipated to be substantial. Todays Challenges In the present market landscape, clients are encountering significant storage-related hurdles. However, these issues can be effectively addressed or alleviated through ECS implementation within enterprises. ECS is tailored to fulfill object-based workload requirements. Select each tabs to learn more about the challenges: ECS Concepts and Features © Copyright 2023 Dell Inc Page 5  Introduction to ECS Unstructured Data The surge in third-platform technologies drives unstructured data1 growth, posing challenges related to its volume, variety, velocity, veracity, and value. Organizations without a solid strategy to handle this growth risk are outperformed by proactive competitors. 1Unstructured data fuels diverse traditional and modern use cases, fostering innovation. Notably, over 90% of global data is unstructured, highlighting its pivotal role in today's data- driven landscape. ECS Concepts and Features © Copyright 2023 Dell Inc Page 6  Introduction to ECS When formulating an unstructured data strategy, three fundamental challenges must be addressed: Ensuring data containment and protection2 across edge, core, and cloud environments. Extracting meaning from data through discovery, analysis for valuable insights, and effective management3. Converting data insights4 into actionable business value for tangible outcomes. 2 Ensuring the security and preservation of your data and systems across all levels—from edge devices to central infrastructure and cloud services. Modern infrastructure must be scalable to accommodate varying data growth patterns and prevent legacy platform limitations from impeding progress. 3 Developing the capability to comprehend your stored data is paramount. Organizations require tools to efficiently locate and interpret data that is spread across the enterprise. Effective management solutions are crucial to handle data appropriately. 4 Translating gathered insights into practical actions that yield tangible value. Having cutting- edge technology and tools is insufficient if lacking the expertise to seamlessly integrate into business operations. The goal is to generate measurable outcomes and a satisfactory return on investment. ECS Concepts and Features © Copyright 2023 Dell Inc Page 7  Introduction to ECS Inadequate Foundation Operational silos impede data sharing and process integration. Despite embracing data-driven decisions, companies struggle to harness available insights. Centralized data repositories are absent in many organizations, hindering value extraction from extensive data lakes. Efficient data storage and access are essential for uncovering meaningful insights. Enterprises are adopting data management to enhance functions, competitiveness, and revenue streams. Data Security and Compliance Ensuring data security and compliance is vital in the digital era. Organizations recognize data's value, with many prioritizing security in public cloud adoption. Data security and compliance drive the preference for on-premises object storage solutions. Failing to achieve compliance exposes enterprises to substantial penalties and irreparable reputation damage. Inflexible Stack Companies are adopting modern storage solutions for re-platforming and new application development. Traditional storage falls short of agile architectures like microservices and containers. ECS Concepts and Features © Copyright 2023 Dell Inc Page 8  Introduction to ECS Efficient Solutions for Data Management5 Challenges of Growing Data Volumes6 Agile Practices and Technologies7 Current Application Landscape8 Future Application Trends9 5 Addressing the cost and complexity of handling large data volumes is a priority. Enterprises aim to balance data availability with cost management. 6 Rising data volumes pose hurdles in migration, integration, governance, and analysis. This hampers realizing the full potential of data. 7 Embracing agile processes and technologies is vital for achieving effective digital outcomes and staying adaptable to evolving demands. 8 Now, most new applications (90%) are built using microservices, with a significant portion (35%) adopting a cloud-native approach. 9 Looking forward, a considerable proportion (over 80%) of upcoming applications that are developed in programming languages leverage container deployment, offering benefits such as speed, consistency, and portability. ECS Concepts and Features © Copyright 2023 Dell Inc Page 9  Introduction to ECS Cloud Repatriation Many customers are recognizing that the public cloud might not be the ideal environment for certain critical workloads. Hybrid and Multi-Cloud Approaches10 Multi-Cloud Management11 Security and Compliance Concerns12 Unanticipated Costs13 Data Portability Challenges14 Application Repatriation Trend15 10 Some applications are better suited to remain in-house, aligning with hybrid and multi-cloud strategies that offer flexibility and control. 11 In the future 90% of G1000 organizations will adopt a multi-cloud management strategy to efficiently handle diverse cloud environments. 12 An estimated 50% of organizations that have embraced the public cloud have encountered significant security and compliance issues, underscoring the importance of robust data protection measures. 13 A notable 64% of organizations have faced higher total service costs than initially expected when transitioning to the cloud. 14 Public cloud providers often facilitate easy data entry but pose difficulties for data extraction, often accompanied by high egress fees. 15 According to IDC, an estimated 50% of public applications are anticipated to be migrated back to on-premises environments within the next two years. ECS Concepts and Features © Copyright 2023 Dell Inc Page 10  Introduction to ECS ECS Solution and Benefits An essential solution for current issues in traditional infrastructure is Software-Defined Storage (SDS)16. This allows organizations to employ commodity platforms, bringing hyperscale capabilities to their data centers. 16In a software-defined approach, storage and data services are provided as software operating on storage devices. This software accommodates diverse data types and access protocols, including metering, monitoring, and reporting. ECS Concepts and Features © Copyright 2023 Dell Inc Page 11  Introduction to ECS 1: ECS is a unified global object storage platform, ideal for next-gen applications. It is the third- generation object platform of Dell, catering to mobile, cloud, big data, and social networking apps. ECS offers exceptional storage efficiency and management simplicity, accommodating diverse file sizes. It supports concurrent use by modern apps, Hadoop analytics, and traditional applications. 2: ECS is software-defined storage that is built from the ground up with the ability to scale up to the Exabyte level with superior economics. ECS supports multiple protocols for unstructured object and file workloads on a single cloud-scale storage platform. ECS natively provides storage for traditional archive applications and third-platform modern applications. 3: ECS can provide the benefits of a public cloud without data residency and compliance risks. Public cloud storage platforms are driven by compelling economic advantages over traditional SAN and NAS storage systems. However, public cloud storage platforms involve consideration of some fundamental trade-offs in the areas of data residency, compliance with local laws and regulations, and potential unforeseen costs for data retrieval. 4: ECS is easy to deploy, scale, migrate to, and distribute across the globe. ECS delivers simple storage management of globally distributed infrastructure under a single global namespace with anywhere access to content. ECS incorporates enterprise-grade features that are designed to reduce risks of data availability outages and eliminate the need for protocol translations, gateways, and data movement. ECS Concepts and Features © Copyright 2023 Dell Inc Page 12  Introduction to ECS ECS Value Proposition ECS key value propositions cover four main aspects, Select each tab to learn more about it. Multi-Purpose Platform Supports billions of files and major object protocols. Natively supports Hadoop, S3, Swift, and CAS. API extensions like Atomic appends and byte-range updates are available. Enables concurrent data access through industry protocols like HDFS and NFS. Smart Object Storage Provides intelligent storage with instant metadata search. Integrated HDFS and background erasure coding are included. Enables unified global file access and disaster recovery readiness. Handles outages efficiently. Native NFS v3 support, global namespace, and multi-protocol access are available. No file gateway or app-level changes are required. Integrated metadata management and search eliminate extra coding. Enterprise Class Provides multi-tenant capabilities for flexible storage service expansion. It caters to diverse clients while ensuring data integrity. Supports enterprises and service providers in delivering storage as a service. Superior Economics ECS Concepts and Features © Copyright 2023 Dell Inc Page 13  Introduction to ECS Provides enterprise-class storage with robust security and compliance capabilities. Ensures data encryption for enhanced protection. Supports various applications, offering data protection even during disk failures. Delivers high availability, simplicity, and scalability for modern applications. Enables easy access and offers superior economics for organizations. ECS Concepts and Features © Copyright 2023 Dell Inc Page 14  Introduction to ECS Object Storage Evolution Dell object storage solution has evolved through the years. 1: Dell created the object storage market with the introduction of Centera in 2001. Centera has a proprietary API and provides for WORM archive storage only. API (proprietary) Single Namespace Immutable Content Unstructured Content 2: Dell knew from customers and internal engineering, that the benefits of scale-out object storage would be useful for read/write workflows and collaboration across multiple sites. Atmos was introduced to offer storage at a larger scale, and the secure multi-tenant platform supports industry standard REST protocols. Soon the Cloud became an increasingly well-known concept. REST API Geo Replication Global Namespace Multi-tenant Unstructured Data ECS Concepts and Features © Copyright 2023 Dell Inc Page 15  Introduction to ECS 3: Innovation in third platform applications demands hardware flexibility, more intelligence from big data, and ever-expanding data access methods and device support. With ECS, you get much of the Atmos and Centera functionality in the same box, plus support for Hadoop and new interfaces. ECS delivers scale-out elastic object storage, and file storage at an efficient and affordable economic price point. Universal API Support Supports Analytics Highly efficient Geo Storage Unstructured Data Semi-structured Data Structures Data Hyperscale Economics 4: ECS has reached its third generation, featuring denser storage and enhanced computing capabilities for improved handling of larger workloads and data, demonstrating a continuous evolution to meet the highest customer demands. Similarly, ObjectScale marks the next step in unstructured data management, seamlessly integrated with ECS to transform data storage and utilization for modern enterprises. ECS Concepts and Features © Copyright 2023 Dell Inc Page 16  Introduction to ECS ECS Features ECS comes with a complete set of storage features that are desired by most organizations seeking to modernize their storage platform. 1: ECS scales to meet the ever-growing customer requirements. Both compute and capacity are added as additional ECS nodes and racks are added to the cluster. 2: Multi-tenancy is supported natively on ECS. ECS provides for both physical and logical separation, based on the organization's needs. 3: ECS is built from the ground up as an object storage platform with APIs supporting HTTP and HTTPS requests. The extensive APIs allow quick and easy web integration using commands native to a web environment. 4: ECS supports files and objects of any size. ECS can divide a large object into storage efficient chunks. Simultaneously, it can pack hundreds or thousands of tiny byte size objects into a single chuck for storage efficiency. 5: ECS is designed to scale from a small single site cluster to large clusters covering multiple sites with varied relationships between sites. 6: ECS has built in system metadata, and custom metadata capabilities to meet the needs or the organization and applications. Data objects can be extended to different policies for immutability, retention, access, and more. ECS Concepts and Features © Copyright 2023 Dell Inc Page 17  Introduction to ECS ECS Use Cases The ECS solution with its multi-site, active-active architecture, global namespace, and support for multiple types of data access clients has several interesting use cases. Big data Analytics: Storage for IoT Challenges: Unstructured data growth from IoT sensors "No Public Cloud" policy: on-premises required Compatibility with APIs like S3 and OpenStack Streamline Hadoop analytics workflow ECS Concepts and Features © Copyright 2023 Dell Inc Page 18  Introduction to ECS ECS Benefits17: Low cost per GB for protected and replicated data High availability with on-premises security Compatible with S3, OpenStack, and other APIs HDFS compatible: Enables data in place analytics Keep up with increases in capacity or retention requirements 17 The Internet of Things opens revenue opportunities as ECS efficiently collects and analyzes massive customer data. ECS simplifies analytics, supporting direct analysis on the platform without lengthy ETL processes. It seamlessly integrates with Hadoop clusters, handles increasing data volumes, and maintains data integrity. Rich metadata allows tagging and retention management, ensuring faster time-to-market, cloud scalability, and robust data privacy, security, and compliance for businesses. ECS Concepts and Features © Copyright 2023 Dell Inc Page 19  Introduction to ECS Big data Analytics: Data Protection for Hadoop Challenges: Exponential data growth Wide range of data types High speeds of data read and write ECS Concepts and Features © Copyright 2023 Dell Inc Page 20  Introduction to ECS ECS Benefits18: Integrated metadata search Native multi-protocol support and geo capabilities Ability to analyze faster and protect big data by distributing data across multiple data centers 18As data is growing exponentially, there is a need to deal with huge amounts of data, store a wide range of datatype and sources, and analyze data streams rapidly. ECS offers metadata search, multi-protocol support, and geo capabilities for data analysis and protection. It allows distributed data storage, fast analysis without conversions, and versatile data storage in one platform. ECS Concepts and Features © Copyright 2023 Dell Inc Page 21  Introduction to ECS Geo-Protected Archive Challenges: Data growth: Reclaim Tier 1 storage Scalable and cost efficient archival Instant access to cold-stored data "No Public Cloud" policy: On-premises required ECS Concepts and Features © Copyright 2023 Dell Inc Page 22  Introduction to ECS ECS Benefits:19 On-premises security Quick retrieval of archived data 19 Organizations face growing unstructured data storage needs. ECS provides cost-efficient solutions by moving inactive content from Tier 1 storage, and optimizing resources without compromising protection. Unlike tape solutions with high service costs, and cloud options with unpredictable costs and slow data retrieval. ECS offers an integrated on-premises answer, tackling both expense and access time issues. ECS Concepts and Features © Copyright 2023 Dell Inc Page 23  Introduction to ECS Video Surveillance Challenges: High capacity footprint per file Maintain data authenticity ECS Concepts and Features © Copyright 2023 Dell Inc Page 24  Introduction to ECS ECS Benefits:20 Low in cost Metadata used to tag files with important details like read-only status, camera location, retention requirement, and data protection requirement 20In contrast to IoT data, video surveillance data has a smaller object storage count, but a higher capacity footprint per file. While data authenticity is important, data retention is not as critical. ECS can be a low-cost landing area or secondary storage location for this data. ECS Concepts and Features © Copyright 2023 Dell Inc Page 25  Introduction to ECS Global Content Repository Challenges: Scale for explosive growth in unstructured content Store data efficiently at geo-scale Support modern web, mobile, and cloud applications Establish retention policies across data archives ECS Concepts and Features © Copyright 2023 Dell Inc Page 26  Introduction to ECS ECS Benefits:21 Global access to geo-replicated content Data accessible by web, mobile, and cloud apps Geo-protection and policy-based retention 21Current high-cost storage systems store unstructured content like images and videos. This cost prevents effective management of data growth. On-premises clouds are desired for easy cold/archive data management. Emerging applications leverage massive data through open APIs, causing shadow IT in public clouds. ECS allows policy-based retention for data compliance and governance. ECS Concepts and Features © Copyright 2023 Dell Inc Page 27  Introduction to ECS Modern Applications Challenges: Optimize storage architecture for mobile access Reduce multi-site access limitations and complexity Support public cloud APIs with private cloud flexibility Implement strong consistency at geo-scale, simplifying application development ECS Concepts and Features © Copyright 2023 Dell Inc Page 28  Introduction to ECS ECS Benefits:22 Strong consistency of data regardless of application location Multiple, industry standard APIs and protocols 22Traditional client/server architectures are limited by protocols and lack data latency consideration, causing development challenges. ECS offers various access methods and geo-capability, easing developer focus on next-gen applications. Multi-site operations support and consistent performance simplify development. ECS supports industry APIs like Amazon S3, OpenStack Swift, and HDFS, allowing the reuse of existing applications. The REST API enables direct ECS integration with preferred tools. ECS Concepts and Features © Copyright 2023 Dell Inc Page 29  Introduction to ECS Data Lake Challenges: Manage cost, volume, and velocity of data Hadoop native replication requires a large footprint Reduce manual data preparation Analyze the data quickly ECS Concepts and Features © Copyright 2023 Dell Inc Page 30  Introduction to ECS ECS Benefits:23 Cost-effective storage for PBs of large or small files Brings analytics to geo-distributed data and archives 23Global content repository analysis is crucial for business success, yet the diversity of data endpoints and types, along with changing data lifetimes, creates unmanageable unstructured data. ECS offers cost-effective storage and a unified infrastructure for global data analysis, eliminating ETL processes and enabling direct analysis for faster insights. ECS Concepts and Features © Copyright 2023 Dell Inc Page 31  Introduction to ECS ECS Products ECS can be delivered as an Appliance or as a Software-only solution. ECS Appliance Provides a complete storage system managed and readied by Dell. Customers receive an all-in-one hardware and software package. The solution ensures optimal configuration for object storage environments. ECS Software Allows customers to use their own commodity hardware for running object and HDFS data services. This approach combines the cost-effectiveness of commodity hardware with enterprise storage capabilities. Suitable for large deployments and needs Dell's qualification. Approved hardware is necessary for running ECS Software Defined storage. Dell collaborates with customers to adapt ECS software to their platform. ECS Concepts and Features © Copyright 2023 Dell Inc Page 32  Introduction to ECS ECS Software Licensing Options ECS software only licensing options include a free trial edition and a fully licensed production version. Community Edition Free trial Version Unlimited Capacity and time Development and testing Online community support Licensed Edition Free-based version Raw capacity based Production environment Dell Support ECS Concepts and Features © Copyright 2023 Dell Inc Page 33  Introduction to ECS Knowledge Check Context: Infinity Solutions is a technology company. They are struggling with managing the exponential growth of data across their enterprise. They are also looking to adopt agile practices and technologies to support their digital outcomes effectively. 1. What aspect of ECS would be most relevant to address the company's challenges of managing data growth and adopting agile practices? a. ECS's support for S3 and Swift protocols b. ECS's focus on hybrid and multi-cloud strategies c. ECS's integrated metadata management and search d. ECS's containerized version for free download 2. Infinity Solutions is concerned about data security and compliance as they plan to adopt new storage solutions. They have sensitive customer information and need a solution that ensures data protection and meets compliance requirements. Which features of ECS would be most beneficial for addressing a company's concerns about data security and compliance? a. ECS's encryption of data at rest and in-flight b. ECS's support for Hadoop and multi-protocol access c. ECS's integrated metadata management and metadata search d. ECS's compatibility with public cloud APIs 3. The client is evaluating the licensing options for implementing ECS within their organization. They want to understand the different licensing models and features available. Which statements accurately describe the ECS licensing options and features? (Select all that apply.) a. The ECS Community Edition is a containerized version of ECS available for free download, suitable for production environments. b. ECS software-only licensing is based on the amount of raw storage that is deployed per customer site. c. The ECS Community Edition is supported through the online community for testing. d. ECS software-only licensing includes software licenses, premium, and professional service support costs, and is available for production environments. ECS Concepts and Features © Copyright 2023 Dell Inc Page 34  Introduction to ECS Elevator Pitch So Far - Introduction to ECS The web version of this content contains an interactive activity. Infinity Solutions was struggling with managing the overwhelming growth of data across the enterprise. They needed a solution that not only efficiently handles data growth but also supports their goal of adopting agile practices. The metadata management and search features of ECS assist in managing increasing data volumes and supporting agile practices. Encryption of data within ECS ensures the security of customer information. ECS licensing is established based on deployed storage, including a free Community Edition for unlimited testing. Licensing for software-only ECS includes both software and support for production environments. ECS Concepts and Features © Copyright 2023 Dell Inc Page 35  ECS Storage Structure ECS Storage Structure ECS Storage Layers 1: The ECS Virtual Data Center – VDC is both a physical and logical construct. VDCs are physically connected racks that are logically defined as a cluster. VDCs are the top-level resource that represents the collection of ECS infrastructure to manage as a unit, also called an ECS cluster. Multiple VDCs can be located within a single physical location or site. 2: The storage pool provides a mechanism for creating physical separation, by logically grouping the storage on different sets of nodes. The grouping logically separates that storage from the storage on other sets of nodes. The storage pool is also used in determining which nodes are used when protecting copies of data. 3: ECS Replication Group is a logical construct that defines the storage pool protection relationship. Similar to federation between VDCs, a replication group defines a relationship between storage pools, or locally within the storage pool. Storage pools are placed into a replication group to provide flexibility where data can be accessed. Replication groups can be local or global. Local replication groups protect objects within the same VDC against disk or node failures. Ex: The replication group on VDC1 and VDC2 is a local replication group. Global replication groups protect objects against disk, node, and site failures. Ex: The replication group on VDC1 is a Global replication group. Both global and local replication groups can be created for later configuration selection. 4: A namespace is a logical construct that provides the equivalent to a dynamic logical volume. Each namespace creates a logically defined storage separation within a replication group. A ECS Concepts and Features © Copyright 2023 Dell Inc Page 36  ECS Storage Structure namespace is associated with one replication group. Multiple namespaces can be associated with the same replication group. When this occurs, the objects that are written through multiple namespaces are stored within and replicated across the same physical storage. 5: Buckets are object containers that are created on top of a namespace that can be used to control access to objects and to set properties that define attributes for all contained objects, such as retention periods and quotas. As namespaces are also global resources, an object can be addressed using its bucket and namespace from any linked VDC. 6: VDCs can be federated together and share a global configuration. Two or more VDCs, separated locally by just a few meters or globally by hundreds of kilometers, can be federated together. Federation is the first step to protecting and sharing objects for global access. ECS Concepts and Features © Copyright 2023 Dell Inc Page 37  ECS Storage Structure ECS Hardware Building Blocks ECS systems are built using fundamental building blocks to form clusters in a distributed storage architecture. Nodes and network switches are combined to form racks. Racks are added to expand the cluster. Nodes Nodes are the fundamental building blocks. A node consists of compute resources, or server components, and storage capacity in the form of a storage enclosure with drives. Each chassis contains either a single node EX5000S or dual node EX5000D in one 5U chassis. The EX500 and EXF900 chassis contains one node in a 2U chassis. ECS Concepts and Features © Copyright 2023 Dell Inc Page 38  ECS Storage Structure Switches Top of rack switches are part of the integrated solution and provide the communications infrastructure from the nodes to the customer network, and between nodes. ECS Gen3 uses two 10/25 Gbps front-end switches and two 10/25 Gbps back-end switches. The front-end switches are connected to the nodes and to the customer network. All traffic between nodes, between racks, and to the customer applications are carried over the data switches. The back-end switches currently provide connectivity for in-band and out-of-band node management. ECS Concepts and Features © Copyright 2023 Dell Inc Page 39  ECS Storage Structure The Aggregation switches24 used with the EXF900 ECS Appliance Nodes and are required when expanding beyond one rack. 24Dell provides two 100GbE S5232F back-end aggregation switches (AGG1 and AGG2) with four 100GbE VLT cables. These switches are known as the Eagle (AGG1) and the Falcon (AGG2) switches. ECS Concepts and Features © Copyright 2023 Dell Inc Page 40  ECS Storage Structure Racks Nodes and network switches are combined into racks. Racks are both physically separated, and logically separated in building an ECS cluster. Racks provide a physical separation for ECS configurations. Racks must contain a single node type. Racks must contain a minimum number of nodes. Racks are identified in ECS configuration files as a separate logical element. Racks are individually configured with each rack having a master configuration node. ECS Concepts and Features © Copyright 2023 Dell Inc Page 41  ECS Storage Structure Physical Storage Physical storage or constructs25 are groupings and separations of physical components. These components include racks and nodes. Virtual Data Centers The VDC defines26 which racks are grouped into the cluster. The racks are physically networked together, and virtually defined in the ECS software. Connecting the racks within a VDC requires the use of management or backend switches. This network of interconnected ECS appliances is known as the Nile Area Network (NAN)27. 25 A construct is a way to define a grouping. 26 - A VDC can be configured to contain one or multiple racks. - Increasing the number of racks expands capacity and adds compute resources to the VDC. - The total storage available across all racks in a VDC can be presented as a single storage space. 27 When multiple racks are combined within a VDC through the NAN, the management and capacity of all the nodes in all the racks are combined and managed as a single cluster. ECS Concepts and Features © Copyright 2023 Dell Inc Page 42  ECS Storage Structure Storage Pools The storage that is associated with the nodes can be partitioned into storage pools. Storage pools28 allow the organization of storage resources based on business requirements. For example, an organization requires physical separation of data, the nodes can be grouped into multiple different storage pools. 28.All storage within a given node can belong to only one storage pool. A storage pool requires at least four nodes. However, five nodes are recommended as a minimum to enable enhanced node loss protection. -The first step in the configuration is to create at least one storage pool, which occurs before creating a VDC. -The first storage pool that is created in a VDC may not be deleted. -You may join additional nodes to the storage pool from the same or different racks later in the implementation process. ECS Concepts and Features © Copyright 2023 Dell Inc Page 43  ECS Storage Structure View the displayed figure for examples29 of possible storage pool configurations within a VDC. 29 - VDC A and VDC B each have a single storage pool and VDC C has two storage pools. - The racks in VDC B are connected together through the Nile Area Network (NAN), and Storage Pool 2 includes the storage that is attached to all the nodes in both Rack 1 and Rack 2. - The rack in VDC C has been split into two storage pools, each storage pool includes storage from only part of the nodes in the rack. ECS Concepts and Features © Copyright 2023 Dell Inc Page 44  ECS Storage Structure Logical Storage Logical storage or constructs are software-defined groupings and separations. Logical storage is built on top of the physical storage. Most ECS storage is defined and related together using logical storage. Federation The establishment of a federation enables the connection of VDCs, facilitating the replication of data between different locations to enhance data availability and durability. Also, it ensures the resilience of ECS in the face of site failures. ECS Concepts and Features © Copyright 2023 Dell Inc Page 45  ECS Storage Structure Replication Groups ECS Replication Group is a logical construct that defines where storage pool content is protected. Storage pools are placed into a replication group to provide flexibility wherein data can be accessed. Replication groups can be local30 or global31. 30 Local replication groups protect objects within the same VDC against disk or node failures. 31 Global replication groups protect objects against disk, node, and site failures. ECS Concepts and Features © Copyright 2023 Dell Inc Page 46  ECS Storage Structure Replication Group X32 Replication Groups Y and Z33 are local replication groups. When an object is written to a replication group that contains nodes from a single VDC, the data for the object is protected only by distributing that data across the nodes in the storage pool. 32Replication Group X is a global replication group consisting of Storage Pool 1 in VDC A and Storage Pool 2 in VDC B. Placing federated VDCs and storage pools within a replication group enhances data protection by identifying participating nodes and VDCs. When data is written to a replication group with federated VDC nodes, it is protected by distributing the data across local VDC nodes and copying it to a storage pool in another VDC. In the event of a VDC failure, data can be recovered from the remaining VDCs in the replication group. Also, including federated VDCs in a replication group improves data access by establishing multiple local access points for the stored data. 33 Replication Groups Y and Z are local replication groups. When an object is written to a replication group that contains nodes from a single VDC, the data for the object is protected only by distributing that data across the nodes in the storage pool. For example, data that are stored in Replication Group Y on VDC C is only written to disks on the nodes within Storage Pool 3. If additional racks contained nodes also in Storage Pool 3, the data protection is extended to include all nodes in the replication group and not only the nodes in the single rack. ECS Concepts and Features © Copyright 2023 Dell Inc Page 47  ECS Storage Structure Namespaces The namespace34 provides the capability to separate ECS users using logical boundaries35 around storage pools. 34 Namespaces can be used to represent a company, a department within an organization, a group within a department, or isolated storage for a specific application. 35 The logical separation enables ECS to handle multi-tenant operations. Each tenant is defined by the namespace and the set of users who can store and access objects within that namespace. Users of one namespace cannot access objects from another namespace unless granted permission using ACLs. - For a service provider, this architecture can be used to separate one tenant from the other. - For enterprise IT, this architecture could be used to isolate one business unit from another such that it has its own user management, replication configuration, as well as encryption policy. ECS Concepts and Features © Copyright 2023 Dell Inc Page 48  ECS Storage Structure Buckets Objects are stored by writing into a bucket. Objects are retrieved by reading from a bucket. Writing36 and reading occur when the object application connects to the IP address37 of one of the nodes within the namespace containing the desired bucket. S3 or Swift buckets38 can be configured to support file system access for NFS and HDFS. 36 When writing an object, the node to which the application is connected determines which VDC owns the object. Note, however, that regardless of the VDC ownership, the application can access the object by connecting to the nodes in any other VDC in the same replication group. 37 The port on the IP address that the application connects through depends on whether the application uses the S3 protocol, OpenStack Swift protocol, or Centera CAS protocol. 38 A bucket that is configured for file system access can be read and written by using its object protocol and by using the NFS or HDFS protocol. ECS Concepts and Features © Copyright 2023 Dell Inc Page 49  ECS Storage Structure Users and Roles ECS supports a hierarchy of different types of users, who are defined as roles. Each role has a set of tasks a user can perform. 1: The system admin role is the highest privileged role for ECS. Users with a system admin role can perform all configuration and management tasks that are required for ECS. System admins configure and manage the VDC, storage pools (SP), replication groups (RP), authentication providers, namespaces, buckets, and users. System admins have full access to all monitoring and reporting available with an ECS system. A system admin can configure namespaces, and perform namespace administration, add buckets to a namespace, and create object users. The system admin role can create namespace administrators, and add them to specific namespaces. ECS has a root user account. The root account is assigned to the system admin role. The root account can be used to perform initial configuration for some ECS versions. 2: Users assigned the system monitor role can only view all configuration data, but cannot make any configuration changes. System monitor role users can view the same information available to the system admins. Local system monitors can only modify their own passwords. 3: The lock admin is a pre-defined, pre-provisioned local user named emcsecurity. The lock admin role can lock and unlock nodes through ECS Portal or the ECS Management API. Locking a node disables remote SSH access to the node for security purposes. The lock ECS Concepts and Features © Copyright 2023 Dell Inc Page 50  ECS Storage Structure admin can only change their passwords, and lock and unlock nodes. The role cannot be assigned to another user. 4: Users assigned the namespace admin role can configure namespace settings, such as creating buckets, and administering quotas and retention periods. Namespace admins can map domain users to the namespace, and assign local users as object users for that namespace. Namespace admin operations can be performed using the ECS Portal interface, using the ECS Rest API for programmed application integration. 5: Object users are end-users of ECS object storage. Object users access storage through object clients using the ECS supported access protocols, for example, S3, OpenStack Swift, CAS, or client applications. Object users are assigned privileges to perform operations within the namespace they are assigned to. Object users may be enabled to create or delete buckets, and assigned read, write, and modify privileges for data object access. 6: ECS system administration - The system admin, system monitor, and lock admin roles are used to perform, or monitor tasks associated with cluster or VDC level configuration and management. 7: User space administration - Name space administrator and object user roles are used to manage the logical storage space defined by the system admin role. 8: The security administrator is the only management user who can lock and unlock nodes from the ECS Portal or the ECS Management REST API. Locking a node is the ability to disable remote SSH access to the node. The security administrator is authenticated by local authentication (not through AD/LDAP) and cannot access to metering and monitoring UI pages and APIs. The security administrator role can be assigned to another management user only by users with security administrator role. Role capabilities: Upload certificates. Add authentication providers. Create, edit, and delete management users and/or AD/LDAP users/groups. Lock and unlock nodes. Change its own password. 9: EMCSecurity user can prevent remote SSH access to nodes by locking them. The password for this user should be changed after system installation and securely recorded. ECS Concepts and Features © Copyright 2023 Dell Inc Page 51  ECS Storage Structure Below are the mentioned role permissions for EMCSecurity: Upload certificates. Add authentication providers. Assign Security Administrator role to one or more local users and/or AD/LDAP users/ groups. Disable, delete any management user or Disable, delete any management user or change its password, including root account. Lock and unlock nodes. Change its own password. 10: Domain users are users, who are defined in an Active Directory AD/LDAP database. ECS must talk to the AD or LDAP server to authenticate user login request using a self-service capability. 11: ECS IAM provides control and secure access to the ECS S3 resources. IAM uses identities to manage access to the data. Namespace root user is an admin user in the namespace who can also access ECS UI by specifying a password during namespace creation or later. IAM user represents a person or application in the namespace that can interact with ECS resources. IAM users can be put together to form an IAM group. IAM role is an identity that is assumable by anyone who needs it and is associated to policies with a role that determines the access to ECS resources. ECS Concepts and Features © Copyright 2023 Dell Inc Page 52  ECS Storage Structure ECS Data Storage Principles - Chunks 1: ECS creates separate chunks to store actual data objects and the object's associated metadata. Whether the data is file data or object data, ECS stores the data using the same method and mixes the data within the same chunks. 2: As data is added to the ECS system, the objects are added to the next available chunk regardless of the size of the object. Multiple smaller objects are added to the same chunk, and larger objects are divided into multiple chunks. 3: Chunks are seen as open when less than a 128 MB of data is added to the container or a set time has not been reached. New data and modified data is written as an append-only to the next available open chunk. 4: Existing data blocks are never modified in place, which prevents the need to lock a file for modification. ECS continues to add data to an open chunk until it is filled with 128 MB or a set time has been reached. When a chunk is filled or after a set time, the chunk is closed. Closing a chunk is referred as sealing. 5: ECS data protection methodology protects the data at the chunk level rather than the individual object or file level. ECS always provides a minimum protection level to guarantee ECS Concepts and Features © Copyright 2023 Dell Inc Page 53  ECS Storage Structure data availability. Metadata chunks are triple mirrored, open data chunks use a combination of mirroring and erasure encoding, and sealed data chunks are fully erasure encoded with forward error correction. 6: ECS data storage principles use the concept of equal sized logical containers. ECS stores data and metadata in continuous logical containers called chunks. Each chunk is always 128 megabytes in size. ECS Concepts and Features © Copyright 2023 Dell Inc Page 54  ECS Storage Structure Knowledge Check Context: After a comprehensive sales pitch and marketing presentation from a Dell representative, Infinity Solutions has decided to implement ECS for their workflow requirements. As part of the sales team, aim to showcase proficiency by helping in selecting the most suitable ECS Storage system for the implementation of their solution. This involves thorough evaluation and consideration of various factors to ensure the chosen system aligns perfectly with their needs and demonstrates our expertise in delivering optimal storage solutions. 1. When discussing the ECS Gen3 infrastructure, the client frequently encountered the term "nodes." Can you explain the role that nodes play in the ECS Gen3 infrastructure? a. Nodes serve as the physical building blocks, consisting of both compute resources and storage capacity within the same enclosure. b. Nodes are responsible for seamless connectivity and communication between the ECS Gen3 system and the customer network. c. Nodes provide physical separation for ECS configurations, ensuring efficient resource allocation and management. d. Nodes enable seamless expansion and enhanced connectivity for the EXF900 ECS Appliance Nodes, facilitating scalable storage solutions. 2. In the role as a system administrator, why would you create storage pools in the ECS Gen3 infrastructure? a. Storage pools define the logical boundaries for separating ECS users. b. Storage pools allow for the organization and grouping of storage resources based on business requirements. c. Storage pools provide flexibility in accessing data and protect objects against failures. d. Storage pools enable the connection of VDCs and facilitate data replication between different locations. 3. As a client exploring the features and capabilities of the ECS Gen3 system, what is the key role and significance of namespace administrators in ensuring effective system management and data organization? a. Namespace Administrators can configure namespace settings, create buckets, and manage quotas and retention periods. b. Namespace Administrators have read-only access to view configuration data but cannot make any changes. ECS Concepts and Features © Copyright 2023 Dell Inc Page 55  ECS Storage Structure c. Namespace Administrators are responsible for uploading certificates and managing authentication providers. Elevator Pitch So Far - ECS Storage Structure The web version of this content contains an interactive activity. Infinity Solutions decided to use ECS after our presentation. Our team helps them pick the right ECS Storage system, matching their needs and showing our expertise. ECS nodes combine computing and storage, optimizing resources for seamless operation. Storage pools in ECS Gen3 efficiently organize storage resources, dividing and managing capacity. Namespace Administrators in ECS control settings, like creating buckets, and manage data's use and lifespan. They also assign user privileges, ensuring secure access. ECS Concepts and Features © Copyright 2023 Dell Inc Page 56  ECS Architectural Overview ECS Architectural Overview ECS Architecture Layers ECS provides a software-defined cloud storage platform that can be deployed on a set of qualified industry-standard hardware or a turnkey storage appliance. At a high level, ECS has many different components: 1: Data Services – The data services layer provides the services, tools and APIs to support Object and file access to the system. 2: Portal and Provisioning Services – Portal services provides a web-based portal that allows self-service access, management automation, reporting, and ECS node configuration and management. The service also handles licensing, authentication, multitenancy, and provisioning services. 3: Storage Engine – The storage engine is responsible for storing and retrieving data, managing transactions, data protection, and data replication. 4: Fabric – Fabric provides clustering, health, software and configuration management, upgrade capabilities, and alerting. ECS Concepts and Features © Copyright 2023 Dell Inc Page 57  ECS Architectural Overview 5: Infrastructure – Infrastructure is the core node operating system, and networking components. ECS uses SUSE Linux Enterprise Server 12 as the base operating system. 6: Hardware – Hardware provides the physical compute and storage components, or nodes. Nodes are based on ECS appliance, or customer provided qualified industry standard hardware. Certified network switches complete the hardware layer. Deep Dive: Go to ECS Architectural Guide for more information. ECS Concepts and Features © Copyright 2023 Dell Inc Page 58  ECS Architectural Overview ECS Hardware Layer ECS foundation begins with the hardware that the ECS software runs on. The hardware layer specifically sees the node hardware that is combined with the rack designations in the VDC. ECS appliance options are defined by the generation of the associated introduction of hardware models. ECS appliance hardware remains consistent within a generation. ECS Concepts and Features © Copyright 2023 Dell Inc Page 59  ECS Architectural Overview ECS Node Options 1: The ECS EXF900 nodes add extreme performance at scale for modern workloads. Built on Dell server technology, the EXF900 combines industry-standard server technology and ECS enterprise-grade storage software. Drive Capacity Disks per Disk Rack min/max Rack minimum Rack maximum node type nodes capacity capacity 3.84 TB drives 12 or 24 HDD Min 5 / Max 16 57 TB 1,474 TB drives 7.68 TB drives 12 or 24 HDD Min 5 / Max 16 115 TB 2,949 TB drives 15.36 TB 12 or 24 HDD Min 5 / Max 16 230 TB 5,898 TB drives drives ECS Concepts and Features © Copyright 2023 Dell Inc Page 60  ECS Architectural Overview 2: EX500 is a versatile option for midsized enterprises looking to support either modern applications or deep archive use cases. It has a denser capacity than EX300 but still maintains a high computing performance. The EX500 is built on the same Dell server technology as the EX300, combining industry-standard server technology and ECS enterprise-grade storage software. Drive Disks per Disk type Rack Rack Rack Capacity node min/max minimum maximum nodes capacity capacity 2 TB drives 12 or 24 HDD Min 5 / Max 120 TB 768 TB drives 16 4 TB 12 or 24 HDD Min 5 / Max 240 TB 1,536 TB drives drives 16 8 TB drives 12 or 24 HDD Min 5 / Max 480 TB 3,072 TB drives 16 12 TB drives 12 or 24 HDD Min 5 / Max 720 TB 4,608 TB drives 16 16 TB drives 12 or 24 HDD Min 5 / Max 960 TB 6,144 TB drives 16 20 12 or 24 HDD Min 5 / Max 1,200 TB 7,680 TB TB drives drives 16 ECS Concepts and Features © Copyright 2023 Dell Inc Page 61  ECS Architectural Overview 3: EX5000S single node and EX5000D dual node, high-density, hot disk-swappable, object storage system. EX5000 is an ideal platform for long-term retention, storage consolidation, and multipurpose object storage requirements that span S3, HDFS, and archive workloads. The below table shows the EX5000S Capacity comparison: Drive Disks per Disk type Rack Rack Rack Capacity node min/max minimum maximum nodes capacity capacity 16 TB drives 25, 50, 75, or HDD Min 5 / Max 7 1,600 TB 11,200 TB 100 drives 20 TB drives 25, 50, 75, or HDD Min 5 / Max 7 2000 TB 14,000 TB 100 drives 4: EXF900 nodes are two rack units or U high. EXF900 is an all-flash object storage solution of hyper-converged nodes for low-latency ECS deployments. Similar to EX500, EXF900 also needs a minimum of five nodes of the same node type to create an ECS cluster. Maximum of 16 nodes per rack. Each node has 12 or 24 NVMe SSD drives. Each node has 4 x 25 GbE networking. 5: EX500 nodes are two rack units or 2U high. Up to 16 nodes can be configured in a single rack. EX500 nodes with different drive sizes can be placed within the same rack. Each node contains only a single drive size. The minimum recommendation is five nodes containing drives of the same size. Nodes are connected to each front-end and back-end switch using 10 GbE SFP+ ports for high availability. The nodes support up to 10 Gbps and connect seamlessly to the new 25 ECS Concepts and Features © Copyright 2023 Dell Inc Page 62  ECS Architectural Overview Gbps switches. The Dell iDRAC port on each node is connected using a 1 GbE connector to the Fox back-end switch for direct-access management. Each rack contains Ethernet switches. Four 100 GbE QSFP ports are also available for customer uplink. A high-availability approach is now implemented for management switches using the same 25 Gbps switches used for the front-end network. 6: The EX5000S and EX5000D models share the same server or storage chassis. While the EX5000S chassis accommodates a single node per chassis. EX5000S have single compute modules per 5U chassis. The compute modules are the same and each contains the same processors and the same amount of RAM. The EX5000S requires a minimum of five chassis or nodes to create a rack. A single rack can contain up to seven nodes. Single-node incremental upgrades for later rack expansion are available. 7: EX5000D dual node, high-density, hot disk-swappable, object storage system. EX5000 is an ideal platform for long-term retention, storage consolidation, and multipurpose object storage requirements that span S3, HDFS, and archive workloads. The below table shows the EX5000D Capacity comparison: Drive Disks per Disk type Rack Rack Rack Capacity node min/max minimum maximum nodes capacity capacity 16 TB drives 25 or 50 HDD Min 8 / Max 1,920 TB 11,200 TB drives 14 20 TB drives 25 or 50 HDD Min 8 / Max 2,400 TB 14,000 TB drives 14 8: The EX5000S and EX5000D models share the same server or storage chassis. While the EX5000D chassis is equipped with double nodes. ECS Concepts and Features © Copyright 2023 Dell Inc Page 63  ECS Architectural Overview EX5000D nodes have dual compute modules per 5U chassis. The compute modules are the same and each contains the same processors and the same amount of RAM. The EX5000D requires a minimum of Four chassis with eight nodes to create a rack. A single rack can contain up to seven chassis with 14 nodes. Two-node incremental upgrades are available for later rack expansion. ECS Concepts and Features © Copyright 2023 Dell Inc Page 64  ECS Architectural Overview ECS Infrastructure Layers Infrastructure Layer Overview ECS infrastructure consists of the software operating system39 running on the nodes, and the networking infrastructure used for connectivity, communications, and virtual network separation. 39 OS version SuSE Linux Enterprise 12, hosting the docker to provide the operating system infrastructure for ECS appliances and approved customer-provided industry-standard hardware. Custom deployments for RedHat Enterprise Linux or CoreOS may be approved through a formal special request and extensive validation process. ECS Concepts and Features © Copyright 2023 Dell Inc Page 65  ECS Architectural Overview Docker is installed on the infrastructure to deploy the encapsulated ECS layers. The ECS software is written in Java, and the Java Virtual Machine (JVM) is installed as part of the infrastructure. The node connectivity and communications are provided using physical and virtual networking infrastructure. ECS Concepts and Features © Copyright 2023 Dell Inc Page 66  ECS Architectural Overview Networking The network traffic is responsible for all communications between the applications and clients and ECS nodes, between ECS nodes, between ECS racks, and between network functions. The ECS network consists of physical switches40, cabling, and virtual network components41. Physical Components have Front-end or Data switches42 and Back-end or Management switches43. 40 Network switches serve separate infrastructure functions. 41 Virtual network components are used to separate and route different types of traffic. 42 Front-end or Data switches are used to connect to the customer network, and provide a logical pathway to the back-end or Management switches. ECS Concepts and Features © Copyright 2023 Dell Inc Page 67  ECS Architectural Overview Virtual network components are classified as below: − North - South communications44 and East - West communications45. − Public, private, and virtual IP addresses46, and Public and private47 Every front-end switch is redundantly connected for high availability to the nodes in the rack, and to the customer network. All application, client, rack-to-rack, node-to-node, and site-to-site traffic uses the front-end switches for communications. 43 Back-end or Management switches are used to connect the nodes for in-band, and out-of- band management. The back-end switches are connected to the front-end switches to enable management access from the front-end switches. 44 North - South communications are between the applications and clients on the customer network to the ECS nodes. 45 East - West communications are between ECS nodes, and between racks in the VDC. 46 Public, private, and virtual IP addresses are assigned for use by different access and ECS functions. As an example, nodes are assigned public IP addresses for client access, and for node management. Private IP addresses are used for internal traffic functions. 47 Public and private networks define the types of traffic communications. Virtual local area network, or VLAN separation, which is known as VLAN tagging, separates, and routes the associated traffic within the ECS VDC. ECS Concepts and Features © Copyright 2023 Dell Inc Page 68  ECS Architectural Overview Gen3 Switches ECS Gen3 switches48 use S5148F49, S5248F50, and Aggregation Switches51. The switches are installed in redundant high-availability pairs for both the front-end, and back-end networks. 48 Switches use Virtual Link Trunks, or VLT, which is similar to and provides the same capabilities as Arista's Multi-Channel Link Aggregation, or MLAG, and Cisco's Virtual Port Channel, or vPC. 49 The Dell S5148F switches feature 25 Gbps, and 100 Gbps connectivity options to the customer network. 50 S5248F 25 GbE IU Ethernet switches with 48 x 25 GbE SFP ports and 4 x 100 GbE uplink ports. 51 Aggregation Switches are used for all models but are required for EXF900 with two or more racks. Dell provides two 100GbE S5232F back-end aggregation switches (AGG1 and AGG2) with four 100GbE VLT cables. ECS Concepts and Features © Copyright 2023 Dell Inc Page 69  ECS Architectural Overview Each pair of front-end and back-end switches are connected together with redundant high- speed links, and using spanning-tree protocol. The switches form the primary link from the nodes to the customer network. The front-end switches are used for most of the traffic. The front-end switches are often referred as Hare and Rabbit. The back-end switch functionality remains for management also provide network separation for replication traffic. The back-end switches are often referred as Hound and Fox. ECS Concepts and Features © Copyright 2023 Dell Inc Page 70  ECS Architectural Overview Fabric Layer Fabric Layer Components The fabric layer52 includes different components to manage the overall system. Node agent53 52 The fabric layer is a distributed piece of software that runs on every node. The fabric is a lightweight Java agent that runs on each node and runs natively. The agent manages Docker and handles Docker-related issues. The node agent provides an API and semantics to deploy and manage the applications dynamically. The fabric layer provides constant monitoring and dynamically reacts to changes. 53 Node agent – The agent manages node resources; disks, network, Docker containers, and so on; and system processes. The node agent runs on each node. ECS Concepts and Features © Copyright 2023 Dell Inc Page 71  ECS Architectural Overview Hardware Manager54 Lifecycle manager55 Application resilience manager56 Registry57 Event library58 54 Hardware Manager - Provides status, event information and provisioning of the hardware layer to higher-level services. These services have been integrated to support commodity hardware. 55 Lifecycle manager – Manages the application life cycle, which involves starting services, recovery, notification, and failure detection. 56 Application resilience manager – Provides the coordination and synchronization of the ECS distributed environment. 57 Registry – Stores all the Docker images for ECS. 58 Event library – Records and stores events occurring on the system. ECS Concepts and Features © Copyright 2023 Dell Inc Page 72  ECS Architectural Overview Fabric Layer Responsibilities The fabric responsibilities include provisioning, installing, and upgrading the ECS software components and system environment. The fabric plays a key role in maintenance 59 and upgrade60 activities. 59 The fabric monitors and maintains the various services, and is responsible to keep the services running. The fabric monitors, and manages the health of node resources, and provides alerting services. If a container is down, or a port needs to be opened, or disk health is failing, the fabric alerts and reports accordingly. The fabric attempts to restore whatever resources are missing, or not working properly. 60 The fabric alerts and query capabilities are available via REST APIs. The fabric provides a uniform command line experience to system administrators to enable node management and allow access to real-time events within the fabric. ECS Concepts and Features © Copyright 2023 Dell Inc Page 73  ECS Architectural Overview Encapsulated Docker Container ECS runs on top of the operating system as a Java application and is encapsulated within several Docker containers. The containers are isolated but share the underlying operating system resources, Node Agent, and HAL61. 61 HAL stands for Hardware Abstraction Layer. It is seen as the hardware manager and is integrated into the Fabric Agent in order to support industry-standard hardware. It runs on every ECS node. Its main purpose is to provide hardware-specific status, event information, and provisioning of the hardware layer to higher level services within ECS. ECS Concepts and Features © Copyright 2023 Dell Inc Page 74  ECS Architectural Overview The Encapsulated Docker Container consists of the following node components: object-main62 fabric-lifecycle63 fabric-zookeeper64 object-throttler65 object-grafana66 object-telegraf67 62 Object-main: Contains the resources and processes relating to the data services, storage engine, portal, and provisioning services. Runs on every node in ECS. All nodes can service read/write requests. 63 Fabric-lifecycle: Contains the processes, information, and resources that are required for system-level monitoring, configuration management, and health management. Depending on the number of nodes in the system, there will be an odd number of fabric-lifecycle instances running. Three instances ran on a four-node system and five for an eight-node system. 64 Fabric-zookeeper: Centralized service for coordinating and synchronizing distributed processes, configuration information, groups, and naming services. Seen as the application resilience Manager, runs on an odd number of nodes. Three instances ran on a four-node system and five for an eight–node system. 65 Object-throttler: Preferred way to throttle ingest rate is to stop reporting some metrics. Throttler will cause the lowest layer (services themselves) to stop providing some metrics. This ensures that monitoring the system under stress will not die because of memory restrictions. Runs on three nodes of the cluster. 66 Object-grafana: Grafana is responsible for visualizing metrics on the appropriate dashboards. Grafana loads all necessary metrics over the Flux queries (queries based on the Flux query language, a lightweight scripting language for querying databases like InfluxDB and working with its data), using the Grafana Flux query language plug-in. Grafana uses Fluxd as a data source. Runs on each cluster node. ECS Concepts and Features © Copyright 2023 Dell Inc Page 75  ECS Architectural Overview object-fluxd68 object-influxdb69 fabric-registry70 67 Object-telegraf: Telegraf is an agent for collecting, processing, aggregating, and writing metrics. Telegraf is used for filtering the incoming metrics and pushing them to the underlying InfluxDB instances. Runs on each cluster node. 68 Object-fluxd: Fluxd is a component responsible for running the incoming Flux queries (queries based on the Flux query language - a lightweight scripting language for querying databases like InfluxDB and working with its data). It is responsible to issue queries to the underlying InfluxDB instances and merge data. Runs on each cluster node. 69 Object-influxdb: InfluxDB is a time series database that is designed to handle high write and query loads. InfluxDB is used for collecting ECS service and system metrics. Runs on the last three nodes of each VDC. 70 Fabric-registry: Holds and maintains the ECS Docker images that are used to start or restart the different software containers that contain the ECS services that run on the server nodes. Runs on two nodes, usually node 1 and another node. ECS Concepts and Features © Copyright 2023 Dell Inc Page 76  ECS Architectural Overview Storage Engine Layer 1: Resource Service - Stores information such as user, namespace, bucket, and more. 2: Transaction Service - Parses object requests, and manages reads and writes of object data to and from chunk operations. 3: Index Service - Manages file-name or data-range to chunk mapping, and secondary indexes. 4: Chunk Management Service - Manages chunk information, for example location, and per chunk operations. 5: Storage Server Management Service - Monitors the storage server and disks, and reprotects the data upon hardware failures. 6: Partitions Record Service - Records and manages the owner node of a partition, and records B+ tree and journal entries. 7: Storage Server Service - Chunk I/O - Manages direct I/O operations to the disks. ECS has a layered architecture, with every function in the system built as an independent layer. This design principle makes each layer horizontally scalable across all nodes in the system and ensures high availability. ECS Concepts and Features © Copyright 2023 Dell Inc Page 77  ECS Architectural Overview Data Services Layer Data Services Layer Overview Client access to data stored in ECS is through object, HDFS, and file protocols. The data services, which are also seen as head services, are responsible for handling client requests, extracting the required information, and passing the request to the storage engine for further processing (for example read, write, so on). The data services layer acts as an interpreter between the protocols and the storage engine. ECS Concepts and Features © Copyright 2023 Dell Inc Page 78  ECS Architectural Overview Data Services The data head services71 are referenced as access head services and the ECS client access layer. There are some exceptions to multiprotocol access due to protocol semantics and representations of protocol design. The following table highlights the access methods and which protocols interoperate. 71All head services are combined into one process running on the infrastructure layer to handle each of the protocols called dataheadsvc, in order to reduce overall memory consumption. All protocols share the same access method to the storage engine layer. The shared access method enables multi-protocol access to the same data stored on ECS. ECS Concepts and Features © Copyright 2023 Dell Inc Page 79  ECS Architectural Overview Protocol Supported Interoperability Object S3 Extra capabilities like Byte Range HDFS, NFS, Swift Updates and Rich ACLS Swift V2 APIs and Swift and Keystone v3 HDFS, NFS, S3 Authentication CAS SDK v3.1.544 or later N/A File HDFS Hadoop 2.7 compatibility S3, NFS, Swift NFS NFSv3 S3, Swift, HDFS, Atmos (path-based objects only and not object ID style based). ECS Concepts and Features © Copyright 2023 Dell Inc Page 80  ECS Architectural Overview ECS Portal and Provisioning Layer Management of ECS is done through the ECS portal and provisioning services. 1: ECS provides a web-based user interface (WebUI) to manage, license, and provision ECS nodes. The portal has comprehensive reporting capabilities that include: Capacity utilization per site, storage pool, node, and disk. Performance monitoring on latency, throughput, and replication progress. Diagnostic information, such as node and disk recovery status. The ECS dashboard offers system health and performance insights, enhancing overall visibility. Users receive alerts for critical events like capacity limits, quota limits, disk, node failures, or software failures. Detailed performance reports are accessible in the UI under the Advance Monitoring folder, presented in a Grafana dashboard. 2: ECS provides command-line tools to install, upgrade, and monitor the ECS system. SSH access to the nodes is used for command-line management. 3: ECS can also be managed using RESTful APIs. The management API allows users to administer ECS within their own tools, scripts, and new or existing applications. The ECS web UI and command-line tools are built using the ECS REST Management APIs. 4: SRM incorporates ECS monitoring information and provides dashboards and reports relating to object utilization. For instance, object dashboards include a summary of configured ECS Concepts and Features © Copyright 2023 Dell Inc Page 81  ECS Architectural Overview usable capacity, capacity trends, and configurable usable capacity by service level. The inventory namespace report provides detailed information about the quota that is used and the percentage by namespace. The namespace charge-back reports show total used capacity local and remote, total number of objects for each namespace to identify service levels, cost contributors and charges. Bucket level reports provide details on number of objects, quota, capacity used and percentage of quota used by bucket. Also through SRM, performance views and capacity trends over a specified period can be observed for ECS. ECS Concepts and Features © Copyright 2023 Dell Inc Page 82  ECS Architectural Overview Knowledge Check Context: Infinity Solutions is a growing enterprise in need of a scalable and efficient cloud storage solution. They are looking for a platform that can be easily deployed on industry- standard hardware and offers comprehensive management and data services. They come across ECS, a software-defined cloud storage platform, and decide to explore its features and capabilities. 1. Why is the fabric layer an essential component in the ECS cloud storage platform? a. It ensures high availability and scalability of the system. b. It provides physical and virtual networking infrastructure for communication. c. It manages the application life cycle and monitors system processes. d. It stores Docker images and manages container deployment. 2. Infinity Solutions wants to provide a logical pathway for communication to customer. Which component in ECS facilitates the establishment of a logical pathway for communication to customer? a. Front-end or Data switches b. Back-end or Management switches c. Virtual network components d. Physical switches 3. Upon discovering the Grafana feature, the client became interested in understanding the role that it plays in the ECS cloud storage platform. What is the role of Grafana in the ECS cloud storage platform? a. It visualizes metrics and provides dashboards for monitoring and analysis. b. It manages node resources, such as disks and network connectivity. c. It coordinates and synchronizes distributed processes and configuration information. ECS Concepts and Features © Copyright 2023 Dell Inc Page 83  ECS Architectural Overview Elevator Pitch so Far - ECS Architectural Overview The web version of this content contains an interactive activity. Infinity Solutions, a growing enterprise, seeks scalable cloud storage on standard hardware. The ECS fabric layer ensures high availability and scalability through clustering, health monitoring, software management, upgrades, and alerts. Front-end or data switches in ECS create seamless connectivity between nodes and the customer network, ensuring effective communication. Grafana in ECS visualizes metrics, providing interactive dashboards. It monitors performance metrics like latency and throughput, aiding effective system monitoring and management. ECS Concepts and Features © Copyright 2023 Dell Inc Page 84  ECS Features and Functionality ECS Features and Functionality ECS Features and Functionality Overview ECS is a robust enterprise-grade multiprotocol object storage platform that offers a wide array of features. It is a well-designed platform that allows for easy configuration and management, catering to diverse enterprise and workflow needs. This lesson covers some of the notable features and functionalities that ECS provides. ECS Management interfaces and their value Native multitenancy significance User authorization and authentication Multiprotocol support Metadata search functionality Local data protection Multisite data protection Data at Rest Encryption Advanced data retention capabilities Quotas ECS Concepts and Features © Copyright 2023 Dell Inc Page 85  ECS Features and Functionality Security IAM Object Lock ECS Concepts and Features © Copyright 2023 Dell Inc Page 86  ECS Features and Functionality ECS Portal The ECS Portal is a graphical user interface (GUI) for configuring, managing, and monitoring an ECS VDC. Management users have specific roles like System Admin, System Monitor, and namespace Admin. Namespace admins can create and configure buckets and assign object users. The ECS Portal displays relevant information based on the user's role. Object storage users access ECS by supported clients and APIs, not through the ECS Portal. 1: The dashboard provides a system status view, and management overview of the VDC. 2: Logged in user 3: The Monitor section provides access to detailed monitoring information. 4: The Manage section provides management users, or administrator access to configure ECS storage and users. 5: The Settings section provides management users, or administrator access to configure the object base URL, Secure Remote Services, licensing, event notifications, and more. 6: Advanced Monitoring dashboards provide critical information about the ECS processes on the VDC you are logged in to. ECS Concepts and Features © Copyright 2023 Dell Inc Page 87  ECS Features and Functionality 7: Connected VDC 8: The help icon link in the ECS Portal opens the appropriate help guide based on the selected menu area. The help is context-sensitive and links to the appropriate guide, and sections within the guide. Context-sensitive help enables administrators to quickly identify the correct configuration settings, or interpret the information that is displayed in the ECS Portal page. The example displays help for the ECS Portal Dashboard in the ECS Monitoring Guide. Select image to enlarge. ECS Concepts and Features © Copyright 2023 Dell Inc Page 88  ECS Features and Functionality Multi-tenancy and Namespaces Multi-tenancy ECS provides native multi-tenancy72 capabilities. Namespaces provide a logical separation for administration and tenant data. Buckets provide an additional logical separation for subtenant data isolation. The enterprise-level features and simple management design enable customers and services providers to offer storage as a service. 72 ECS provides a simple to deploy, and easy to manage multi-tenant storage environment. Multi-tenancy expands ECS capabilities to meet multiple demands for a wide range of customers. ECS Concepts and Features © Copyright 2023 Dell Inc Page 89  ECS Features and Functionality Namespace Administrator The n