Storage Cloud Services PDF

IT2314 Storage Cloud Services Elastic Volume Service Elastic Volume Service (EVS) offers scalable block storage for cloud servers. EVS disks offer high reliability and excellent performance. They can be used for distributed file systems, development and testing environments, data warehouse applicat...

IT2314 Storage Cloud Services Elastic Volume Service Elastic Volume Service (EVS) offers scalable block storage for cloud servers. EVS disks offer high reliability and excellent performance. They can be used for distributed file systems, development and testing environments, data warehouse applications, and high-performance computing (HPC). A distributed file system (DFS) is a hierarchical file system, whose physical storage resources may not be directly connected to local nodes, but connected to local nodes through compute networks (compute nodes) or a group of logical partitions or volumes. Advantages of EVS: Various disk types: EVS provides a variety of disk types for you to choose from, and EVS disks can be used as data disks and system disks for servers. You can select an appropriate disk type that best suits your budget and service requirements. Elastic scalability: You can expand capacity on-demand and without interrupting services. Real-time monitoring: With Cloud Eye, you can monitor EVS disk health in real time. High security and reliability: Both system disks and data disks support data encryption to ensure data security. Data protection functions, such as backups and snapshots, safeguard the disk data, preventing incorrect data caused by application exceptions or attacks. EVS Architecture EVS disks are like the hard disks on your local computer, except on the cloud. They need to be attached to cloud servers before you can use them. You can initialize EVS disks, create file systems, and then use them for persistent data storage. Alternatively, you can create backups and snapshots for your EVS disks to improve data reliability. EVS Performance and Disk Types EVS performance metrics include: IOPS (Input/Output Operations per Second) – number of read/write operations performed by an EVS disk per second 05 Handout 1 *Property of STI  [email protected] Page 1 of 9  IT2314 Throughput – the amount of data read from and written into an EVS disk per second Read/write I/O latency – minimum interval between two consecutive read/write operations on an EVS disk In Huawei CLOUD, EVS disks are classified into the following types by I/O performance: Extreme SSD – super fast disks for workloads demanding ultra-high bandwidth and ultra-low latency Ultra-high I/O – high performance disks excellent for enterprise mission-critical services as well as workloads demanding high throughput and low latency General Purpose SSD – cost-effective disks designed for enterprise applications with medium performance requirements High I/O – disks suitable for commonly accessed workloads EVS Device Types There are two EVS device types: Virtual Block Device (VBD) is the default EVS device type. VBD EVS disks support only basic read/write SCSI commands. Small Computer System Interface (SCSI) EVS disks support transparent SCSI command transmission and allow the server OS to directly access the underlying storage media. Besides basic read/write SCSI commands, SCSI disks support advanced SCSI commands. Shared EVS Disks Shared EVS disks are block storage devices that support concurrent read/write operations and can be attached to multiple servers. Shared EVS disks feature multiple attachments, high-concurrency, high-performance, and high-reliability. They are usually used for enterprise business-critical applications that require cluster deployment for high availability (HA). Multiple servers can access the same shared EVS disk at the same time. 05 Handout 1 *Property of STI  [email protected] Page 2 of 9  IT2314 EVS Encryption In case your services require encryption for the data stored on EVS disks, EVS provides you with the encryption function. You can encrypt newly created EVS disks. System disk encryption: System disks are purchased along with servers and cannot be purchased separately. So whether a system disk is encrypted or not depends on the image selected during the server creation. Data disk encryption: Data disks can be purchased along with servers or separately. Whether data disks are encrypted depends on their data sources. See the following table for details. EVS Backup Cloud Disk Backup provided by Cloud Backup and Recovery (CBR) allows you to create backups for your EVS disks while servers are running. If data loss or damage occurs due to virus invasions, accidental deletions, or software/hardware faults, you can use backups to restore data, guaranteeing your data integrity and security. EVS Snapshot You can create snapshots to rapidly save the disk data at specified time points. In addition, you can use snapshots to create new disks so that the created disks will contain the snapshot data in the beginning. Snapshots and backups are different in that a backup saves the data as another copy in the storage system other than on the disk, whereas a snapshot establishes a relationship between the snapshot and disk data. The following example describes the snapshot principle by creating snapshots s1 and s2 for disk v1 at different time points: 1. Create disk v1, which contains no data. 2. Write data d1 and d2 to disk v1. Data d1 and d2 are written to new spaces. 3. Create snapshot s1 for disk v1 that is modified in 2. Data d1 and d2 are not saved as another copy elsewhere. Instead, the relationship between snapshot s1 and data d1 and d2 is established. 4. Write data d3 to disk v1 and change data d2 to d4. Data d3 and d4 are written to new spaces, and data d2 is not overwritten. The relationship between snapshot s1 and data d1 and d2 is still valid. Therefore, snapshot s1 can be used to restore data if needed. 5. Create snapshot s2 for disk v1 that is modified in 4. The relationship between s2 and data d1, d3, and d4 is established. 05 Handout 1 *Property of STI  [email protected] Page 3 of 9  IT2314 Differences Between EVS Backups and EVS Snapshots Both EVS backups and EVS snapshots provide redundancies for improved disk data reliability. Metric Storage Solution Data Synchronization DR Range Service Recovery Backup Backups are stored A backup is a copy of a A backup and its To restore data and in OBS, instead of disk taken at a given point source disk recover services, you disks. This ensures of time and is stored in a reside in can restore the data restoration different location. different AZs. backups to their upon disk damage Automatic backup can be original disks or create or corruption. performed based on new disks from the backup policies. Deleting backups. a disk will not delete its backups. Snapshot Snapshots are A snapshot is the state of A snapshot and You can use a stored on the a disk at a specific point in its source disk snapshot to roll back same disk as the time and is stored on the reside in the its original disk or original data. same disk. If the disk is same AZ. create a disk from the deleted, all its snapshots snapshot. will also be deleted. For example, if you reinstalled or changed the server OS, snapshots of the system disk were also automatically deleted. Snapshots of the data disks can be used as usual. EVS Three-Copy Redundancy The backend storage system of EVS employs three-copy redundancy to guarantee data reliability. With this mechanism, one piece of data is by default divided into multiple 1 MiB data blocks. Each data block is saved in three copies, and these copies are stored on different nodes in the system according to the distributed algorithms. Three-copy redundancy has the following characteristics: The storage system saves the data copies on different disks of different servers across cabinets, ensuring that services are not interrupted if a physical device fails. The storage system guarantees strong consistency between the data copies. 05 Handout 1 *Property of STI  [email protected] Page 4 of 9  IT2314 For example, for data block P1 on physical disk A of server A, the storage system backs up its data to P1'' on physical disk B of server B and to P1' on physical disk C of server C. Data blocks P1, P1', and P1'' are the three copies of the same data block. If physical disk A where P1 resides is faulty, P1' and P1'' can continue providing storage services, ensuring service continuity. Data Rebuild Each physical disk in the storage system stores multiple data blocks, whose copies are scattered on the nodes in the system according to certain distribution rules. When a physical server or disk fault is detected, the storage system automatically rebuilds the data. Since the copies of data blocks are scattered on different nodes, the storage system will start the data rebuild on multiple nodes simultaneously during a data restore, with only a small amount of data on each node. In this way, the system eliminates the potential performance bottlenecks that may occur when a large amount of data needs to be rebuilt on a single node, and therefore minimizes the adverse impacts exerted on upper-layer applications. The figure below shows the data rebuild process. If the physical disks on server F are faulty, the data blocks on these physical disks will be rebuilt on the physical disks of other servers. Object Storage Service Object Storage Service (OBS) is a scalable service that provides secure, reliable, and cost-effective cloud storage for massive amounts of data. OBS provides unlimited storage capacity for objects of any format, catering to the needs of common users, websites, enterprises, and developers. There is no limitation on the storage capacity of the entire OBS system or of a single bucket, and any number of objects can be stored. As a web service, OBS supports APIs over 05 Handout 1 *Property of STI  [email protected] Page 5 of 9  IT2314 Hypertext Transfer Protocol (HTTP) and Hypertext Transfer Protocol Secure (HTTPS). You can use OBS Console or OBS tools to access and manage data stored in OBS anytime, anywhere. With OBS SDKs and APIs, you can easily manage data stored in OBS and develop upper-layer applications. Advantages of OBS Data durability and service continuity: OBS provides storage for cloud albums of Huawei mobile phones to support access of hundreds of millions of users. It delivers a data durability of up to 99.9999999999% and service continuity of up to 99.995% by using cross-region replication, cross-AZ disaster recovery, device and data redundancy in an AZ, slow disk or bad sector detection, and other technologies. Multi-level protection and authorization management: OBS has passed the Trusted Cloud Service (TRUCS) certification. Measures, including versioning, server-side encryption, URL validation, virtual private cloud (VPC)-based network isolation, access log audit, and fine-grained access control are provided to keep data secure and trusted. 100-billion level objects, 10-million level concurrency: With intelligent scheduling and response, optimized data access paths, and technologies such as transmission acceleration, and big data vertical optimization, you can store hundreds of billions of objects in OBS, and still experience smooth concurrency, ultra-high bandwidth, and low latency. Easy use and management: OBS provides standard REST APIs, SDKs in different programming languages, and data migration tools to help you quickly move your workloads to cloud. Storage resources are linearly, infinitely scalable, without compromising performance. You do not have to plan storage capacity beforehand or worry expansion or reduction. When needed, you can ask Huawei Cloud to perform online upgrade or capacity expansion on your behalf. Tiered storage and on-demand use: Both pay-per-use and yearly/monthly billing are available for OBS. Data in each of the Standard, Infrequent Access, and Archive storage classes is separately metered and charged, which reduces storage costs. OBS Architecture OBS basically consists of buckets and objects. A bucket is a container for storing objects in OBS. Each bucket is specific to a region and has specific storage class and access permissions. A bucket is accessible through its access domain name over the Internet. An object is the fundamental storage unit in OBS. An object consists of the following: A key that specifies the name of an object. An object key is a UTF-8 string up to 1,024 characters long. Each object is uniquely identified by a key within a bucket. Metadata that describes an object. The metadata is a set of key-value pairs that are assigned to objects stored in OBS. There are two types of metadata: system-defined metadata and custom metadata. o System-defined metadata is automatically assigned by OBS for processing objects. Such metadata includes Date, Content-Length, Last-Modified, ETag, and more. o You can specify custom metadata to describe the object when you upload an object to OBS. Data that refers to the content of an object. 05 Handout 1 *Property of STI  [email protected] Page 6 of 9  IT2314 Application Scenarios Big Data Analytics OBS enables inexpensive big data solutions that feature high performance with zero service interruptions. It eliminates the need for capacity expansion. Such solutions are designed for scenarios that involve mass data storage and analysis, query of historical data details, analysis of numerous behavior logs, and statistical analysis of public transactions. Mass data storage and analysis: storage of petabytes of data, batch data analysis, and data query in milliseconds. Query of historical data details: account statement audit, analysis on device energy consumption history, playback of trails, analysis on vehicle driving behavior, and refined monitoring. Analysis of numerous behavior logs: analysis of learning habits and logs. Statistical analysis on public transactions: crime tracking, associated case queries, traffic congestion analysis, and scenic spot popularity statistics. Static Website Hosting OBS provides a website hosting function that is cost-effective, highly available, and scalable to traffic changes. By combining the OBS static website hosting, Content Delivery Network (CDN), and ECS, you can quickly build a website or an application system with separate static and dynamic content. The dynamic data on end user browsers and apps directly interacts with the service systems deployed on Huawei Cloud. Requests for dynamic data are sent to service systems for processing and then returned to end users. The static data is stored in OBS. Business systems can process static data over the intranet. End users directly request and read the static data from OBS through nearby high-speed nodes. Enterprise Cloud Boxes (Web Disks) OBS works with cloud services such as ECS, ELB, RDS, and VBS to provide enterprise web disks with a reliable, inexpensive storage system featuring low latency and high concurrency. The storage capacity automatically scales as the volume of stored data grows. 05 Handout 1 *Property of STI  [email protected] Page 7 of 9  IT2314 Dynamic data on devices such as mobile phones, PCs, and tablets interacts with the enterprise cloud disk service system built on Huawei Cloud. Requests for dynamic data are sent to the service system for processing and then returned to devices, and the static data is stored in OBS. Service systems can process static data over the intranet. End users can directly request and read the static data from OBS. In addition, OBS provides the lifecycle management function to automatically change storage classes for objects, reducing storage costs. Backup and Archive OBS offers a highly reliable, inexpensive storage system featuring high concurrency and low latency. It can hold massive amounts of data, meeting the archive needs for unstructured data of applications and databases. You can use the synchronization clients, Cloud Storage Gateway (CSG), DES, or mainstream backup software to back up your on-premises data to OBS. OBS also provides lifecycle rules to automatically transition objects between storage classes to save your money on storage. You can restore data from OBS to a DR or test host on the cloud. Synchronization clients – good for manual backup of a single database or program CSG – seamlessly compatible with on-premises backup systems DES – ideal for archiving massive volumes of data. It transfers data using Teleport devices and disks to cloud. Backup software – applicable to automatic backup for multiple applications or hosts, delivering strong compatibility Scalable File Service Scalable File Service (SFS) provides scalable, high-performance shared file storage. With SFS, you can enjoy shared file access spanning multiple Elastic Cloud Servers (ECSs), Bare Metal Servers (BMSs), and containers. Advantages of SFS File sharing: Servers in multiple availability zones (AZs) of a same region can access the same file system concurrently and share files. Elastic scaling: Storage can be scaled up or down on demand to dynamically adapt to service changes without interrupting applications. You can complete resizing with a few clicks. Superior performance and reliability: The service enables file system performance to increase as capacity grows, and delivers a high data durability to support rapid service growth. Seamless integration: SFS supports Network File System (NFS). With this standard protocol, a broad range of mainstream applications can read and write data in the file system. Easy operation and low costs: In an intuitive graphical user interface (GUI), you can create and manage file systems with ease. SFS slashes the cost as it is charged on a pay-per-use basis. Application Scenarios High Performance Computing In industries that require HPC, such as simulation experiments, biopharmacy, gene sequencing, image processing, and weather forecast, SFS provides superb compute and storage capabilities, as well as high bandwidth and low latency. 05 Handout 1 *Property of STI  [email protected] Page 8 of 9  IT2314 Media Processing Services of TV stations and new media are more likely to be deployed on cloud platforms than before. Such services include streaming media, archiving, editing, transcoding, content distribution, and video on demand (VoD). In such scenarios, a large number of workstations are involved in the whole program production process. Different operating systems may be used by different workstations, requiring file systems to share materials. In addition, HD/4K videos have become a major trend in the broadcasting and TV industry. Content Management and Web Services SFS can be used in various content management systems to store and provide information for websites, home directories, online releases, and archiving. File Sharing Office documents of enterprises or organizations can be saved in an SFS Turbo file system for high- performance shared access. Dedicated SFS Turbo provides shared file storage for enterprises, governments, and finance institutions based on dedicated compute and storage resource pools. Dedicated resource pools are physically isolated from public pools. The reliable, efficient cloud experience dedicated pools offer can help you meet specific performance, application, and compliance needs. Concepts Related to SFS Network File System (NFS) is a distributed file system protocol that allows different computers and operating systems to share data over a network. Common Internet File System (CIFS) is a protocol used for network file access. It is a public or open version of the Server Message Block (SMB) protocol, from Microsoft. CIFS allows applications to access files on computers over the Internet and send requests for file services. Using the CIFS protocol, network files can be shared easily between Windows hosts. A file system provides users with shared file storage service through NFS and CIFS. It is used for accessing network files remotely. After a user creates a mount point on the management console, the file system can be mounted to multiple ECSs and is accessible through the standard POSIX. Portable Operating System Interface (POSIX) is a set of interrelated standards specified by Institute of Electrical and Electronics Engineers (IEEE) to define the application programming interface (API) for software compatible with variants of the UNIX operating system. POSIX is intended to achieve software portability at the source code level. That is, a program written for a POSIX compatible operating system may be compiled and executed on any other POSIX operating system. Dynamic Host Configuration Protocol (DHCP) is a LAN network protocol. The server controls an IP address range, and a client can automatically obtain the IP address and subnet mask allocated by the server when logging in to the server. DHCP is not installed as a service component of Windows Server by default. Manual installation and configuration are required. 05 Handout 1 *Property of STI  [email protected] Page 9 of 9