VMware Cloud Foundation 5.2 Design Guide PDF

VMware Cloud Foundation Design Guide Modified on 09 OCT 2024 VMware Cloud Foundation 5.2 VMware Cloud Foundation Design Guide You can find the most up-to-date technical documentation on the VMware by Broadcom website at: https://docs.vmware.com/ VMware by Broadcom 3401 Hillview Ave. Palo Alto, C...

VMware Cloud Foundation Design Guide Modified on 09 OCT 2024 VMware Cloud Foundation 5.2 VMware Cloud Foundation Design Guide You can find the most up-to-date technical documentation on the VMware by Broadcom website at: https://docs.vmware.com/ VMware by Broadcom 3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com © Copyright 2021-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. For more information, go to https://www.broadcom.com. All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies. VMware by Broadcom 2  Contents About VMware Cloud Foundation Design Guide 7 1 VMware Cloud Foundation Concepts 10 Architecture Models and Workload Domain Types in VMware Cloud Foundation 10 Workload Domain Cluster to Rack Mapping in VMware Cloud Foundation 15 Networking Models in VMware Cloud Foundation 18 VMware Cloud Foundation Topologies 19 Single Instance - Single Availability Zone 20 Single Instance - Multiple Availability Zones 21 Multiple Instances - Single Availability Zone per Instance 23 Multiple Instances - Multiple Availability Zones per Instance 25 2 External Services Design for VMware Cloud Foundation 28 3 Physical Network Infrastructure Design for VMware Cloud Foundation 29 VLANs and Subnets for VMware Cloud Foundation 29 Leaf-Spine Physical Network Design Requirements and Recommendations for VMware Cloud Foundation 34 4 Supported Storage Types for VMware Cloud Foundation 41 5 vSAN Design for VMware Cloud Foundation 43 Logical Design for vSAN for VMware Cloud Foundation 43 Hardware Configuration for vSAN for VMware Cloud Foundation 45 Network Design for vSAN for VMware Cloud Foundation 47 vSAN Witness Design for VMware Cloud Foundation 48 vSAN Design Requirements and Recommendations for VMware Cloud Foundation 50 6 vSphere Design for VMware Cloud Foundation 58 ESXi Design for VMware Cloud Foundation 58 Logical Design for ESXi for VMware Cloud Foundation 59 Sizing Considerations for ESXi for VMware Cloud Foundation 59 ESXi Design Requirements and Recommendations for VMware Cloud Foundation 61 vCenter Server Design for VMware Cloud Foundation 65 Logical Design for vCenter Server for VMware Cloud Foundation 65 Sizing Considerations for vCenter Server for VMware Cloud Foundation 67 High Availability Design for vCenter Server for VMware Cloud Foundation 67 VMware by Broadcom 3 VMware Cloud Foundation Design Guide vCenter Server Design Requirements and Recommendations for VMware Cloud Foundation 68 vCenter Single Sign-On Design Requirements for VMware Cloud Foundation 71 vSphere Cluster Design for VMware Cloud Foundation 76 Logical vSphere Cluster Design for VMware Cloud Foundation 76 vSphere Cluster Lifecycle Method Design for VMware Cloud Foundation 79 vSphere Cluster Design Requirements and Recommendations for VMware Cloud Foundation 79 vSphere Networking Design for VMware Cloud Foundation 84 Logical vSphere Networking Design for VMware Cloud Foundation 84 vSphere Networking Design Requirements and Recommendations for VMware Cloud Foundation 89 7 NSX Design for VMware Cloud Foundation 93 Logical Design for NSX for VMware Cloud Foundation 95 NSX Manager Design for VMware Cloud Foundation 100 Sizing Considerations for NSX Manager for VMware Cloud Foundation 101 NSX Manager Design Requirements and Recommendations for VMware Cloud Foundation 101 NSX Global Manager Design Requirements and Recommendations for VMware Cloud Foundation 103 NSX Edge Node Design for VMware Cloud Foundation 106 Deployment Model for the NSX Edge Nodes for VMware Cloud Foundation 106 Sizing Considerations for NSX Edges for VMware Cloud Foundation 109 Network Design for the NSX Edge Nodes for VMware Cloud Foundation 109 NSX Edge Node Requirements and Recommendations for VMware Cloud Foundation 111 Routing Design for VMware Cloud Foundation 116 BGP Routing Design for VMware Cloud Foundation 118 BGP Routing Design Requirements and Recommendations for VMware Cloud Foundation 125 Overlay Design for VMware Cloud Foundation 135 Logical Overlay Design for VMware Cloud Foundation 136 Overlay Design Requirements and Recommendations for VMware Cloud Foundation 138 Application Virtual Network Design for VMware Cloud Foundation 141 Logical Application Virtual Network Design VMware Cloud Foundation 141 Application Virtual Network Design Requirements and Recommendations forVMware Cloud Foundation 143 Load Balancing Design for VMware Cloud Foundation 145 Logical Load Balancing Design for VMware Cloud Foundation 145 Load Balancing Design Requirements for VMware Cloud Foundation 147 8 SDDC Manager Design for VMware Cloud Foundation 150 Logical Design for SDDC Manager 150 SDDC Manager Repository Access Design 152 VMware by Broadcom 4 VMware Cloud Foundation Design Guide SDDC Manager Design Requirements and Recommendations for VMware Cloud Foundation 152 9 VMware Aria Suite Lifecycle Design for VMware Cloud Foundation 155 Logical Design for VMware Aria Suite Lifecycle for VMware Cloud Foundation 156 Network Design for VMware Aria Suite Lifecycle 158 Data Center and Environment Design for VMware Aria Suite Lifecycle 159 Locker Design for VMware Aria Suite Lifecycle 161 VMware Aria Suite Lifecycle Design Requirements and Recommendations for VMware Cloud Foundation 162 10 Workspace ONE Access Design for VMware Cloud Foundation 168 Logical Design for Workspace ONE Access 169 Sizing Considerations for Workspace ONE Access for VMware Cloud Foundation 173 Network Design for Workspace ONE Access 174 Integration Design for Workspace ONE Access with VMware Cloud Foundation 178 Deployment Model for Workspace ONE Access 179 Workspace ONE Access Design Requirements and Recommendations for VMware Cloud Foundation 180 11 Lifecycle Management Design for VMware Cloud Foundation 188 12 Logging and Monitoring Design for VMware Cloud Foundation 191 13 Information Security Design for VMware Cloud Foundation 192 Access Management for VMware Cloud Foundation 192 Account Management Design for VMware Cloud Foundation 193 Certificate Management for VMware Cloud Foundation 198 14 VMware Cloud Foundation Topology Design Blueprints 200 Topology Design Blueprint One: Single Instance - Single Availability Zone 200 Topology Design Blueprint Two: Consolidated Single Instance - Single Availability Zone 203 Topology Design Blueprint Three: Single Instance - Multiple Availability Zones 206 Topology Design Blueprint Four: Multiple Instance - Single Availability Zone 211 Topology Design Blueprint Five: Multiple Instance - Multiple Availability Zones 217 15 VMware Cloud Foundation vSphere Cluster Design Patterns 224 vSphere Cluster Design Pattern One: Multi-Rack Compute VI Workload Domain Cluster 224 16 VMware Cloud Foundation NSX Edge Cluster Design Patterns 227 NSX Edge Cluster Design Pattern One: Dedicated Edge Scale and Performance 227 NSX Edge Cluster Design Pattern Two: Multi-Rack Edge Availability 229 VMware by Broadcom 5 VMware Cloud Foundation Design Guide 17 Appendix: Design Elements for VMware Cloud Foundation 232 Architecture Design Elements for VMware Cloud Foundation 232 Workload Domain Design Elements for VMware Cloud Foundation 233 External Services Design Elements for VMware Cloud Foundation 233 Physical Network Design Elements for VMware Cloud Foundation 234 vSAN Design Elements for VMware Cloud Foundation 240 ESXi Design Elements for VMware Cloud Foundation 248 vCenter Server Design Elements for VMware Cloud Foundation 251 vSphere Cluster Design Elements for VMware Cloud Foundation 255 vSphere Networking Design Elements for VMware Cloud Foundation 260 NSX Design Elements for VMware Cloud Foundation 264 SDDC Manager Design Elements for VMware Cloud Foundation 290 VMware Aria Suite Lifecycle Design Elements for VMware Cloud Foundation 292 Workspace ONE Access Design Elements for VMware Cloud Foundation 297 Lifecycle Management Design Elements for VMware Cloud Foundation 304 Information Security Design Elements for VMware Cloud Foundation 307 VMware by Broadcom 6 About VMware Cloud Foundation Design Guide The VMware Cloud Foundation Design Guide contains a design model for VMware Cloud Foundation (also called VCF) that is based on industry best practices for SDDC implementation. The VMware Cloud Foundation Design Guide provides the supported design options for VMware Cloud Foundation, and a set of decision points, justifications, implications, and considerations for building each component. Intended Audience This VMware Cloud Foundation Design Guide is intended for cloud architects who are familiar with and want to use VMware Cloud Foundation to deploy and manage an SDDC that meets the requirements for capacity, scalability, backup and restore, and extensibility for disaster recovery support. Before You Apply This Guidance The sequence of the VMware Cloud Foundation documentation follows the stages for implementing and maintaining an SDDC. To apply this VMware Cloud Foundation Design Guide, you must be acquainted with the Getting Started with VMware Cloud Foundation documentation and with the VMware Cloud Foundation Release Notes. See VMware Cloud Foundation documentation. For performance best practices for vSphere, see Performance Best Practices for VMware vSphere 8.0 Update 1. Design Elements This VMware Cloud Foundation Design Guide contains requirements and recommendations for the design of each component of the SDDC. In situations where a configuration choice exists, requirements and recommendations are available for each choice. Implement only those that are relevant to your target configuration. VMware by Broadcom 7 VMware Cloud Foundation Design Guide Design Element Description Requirement Required for the operation of VMware Cloud Foundation. Deviations are not permitted. Recommendation Recommended as a best practice. Deviations are permitted. VMware Cloud Foundation Deployment Options in This Design This design guidance is for all architecture models of VMware Cloud Foundation. By following the guidance, you can examine the design for these deployment options: n Single VMware Cloud Foundation instance. n Single VMware Cloud Foundation instance with multiple availability zones (also known as stretched deployment). The default vSphere cluster of the workload domain is stretched between two availability zones by using Chapter 5 vSAN Design for VMware Cloud Foundation and configuring vSphere Cluster Design Requirements and Recommendations for VMware Cloud Foundation and BGP Routing Design for VMware Cloud Foundation accordingly. n Multiple VMware Cloud Foundation instances. You deploy several instances of VMware Cloud Foundation to address requirements for scale and co-location of users and resources. For disaster recovery, workload mobility, or propagation of common configuration to multiple VMware Cloud Foundation instances, you can deploy Chapter 7 NSX Design for VMware Cloud Foundation for the SDDC management and workload components. n Multiple VMware Cloud Foundation instances with multiple availability zones. You apply the configuration for stretched clusters for a single VMware Cloud Foundation instance to one or more additional VMware Cloud Foundation instances in your environment. vCenter Single Sign-On Options in This Design This design guidance covers the topology with a single vCenter Single Sign-On domain in a VMware Cloud Foundation instance and the topology with several isolated vCenter Single Sign- On domains in a single instance. See vCenter Single Sign-On Design Requirements for VMware Cloud Foundation. VMware Cloud Foundation Design Blueprints You can follow design blueprints for selected architecture models and topologies that list the applicable design elements. See Chapter 14 VMware Cloud Foundation Topology Design Blueprints. VMware by Broadcom 8 VMware Cloud Foundation Design Guide VMware Cloud Foundation Design Patterns A design pattern is a collection of design requirements and recommendations based on a specific component design, such as the design for vSphere clusters or NSX Edge clusters. A design pattern covers a particular aspect of the design rather than all design decisions for a full topology. See Chapter 15 VMware Cloud Foundation vSphere Cluster Design Patterns and Chapter 16 VMware Cloud Foundation NSX Edge Cluster Design Patterns. Information about Environment Configurations that Can be Converted or Imported into VMware Cloud Foundation In VMware Cloud Foundation 5.2, by using the VCF Import Tool, you can convert or import infrastructure into your VMware Cloud Foundation in the following way: n If you do not already have SDDC Manager deployed, you can deploy it on an existing vSphere environment and use the VCF Import Tool to convert that environment to the VMware Cloud Foundation management domain. n If SDDC Manager is already deployed, you can use the VCF Import Tool to import existing vSphere environments as VI workload domains. For more information, see Converting or Importing Existing vSphere Environments into VMware Cloud Foundationin the VMware Cloud Foundation Administration Guide. VMware Cloud Foundation Glossary See the VMware Cloud Foundation Glossary for constructs, operations, and other terms specific to VMware Cloud Foundation. It is important to understand these constructs before continuing with this design guidance. VMware by Broadcom 9 VMware Cloud Foundation Concepts 1 To design a VMware Cloud Foundation deployment, you need to understand certain VMware Cloud Foundation concepts. Read the following topics next: n Architecture Models and Workload Domain Types in VMware Cloud Foundation n Workload Domain Cluster to Rack Mapping in VMware Cloud Foundation n Networking Models in VMware Cloud Foundation n VMware Cloud Foundation Topologies Architecture Models and Workload Domain Types in VMware Cloud Foundation When you design a VMware Cloud Foundation deployment, you decide what architecture model, that is, standard or consolidated, and what workload domain types, for example, consolidated, isolated, or standard, to implement according to the requirements for hardware, expected number of workloads and workload domains, co-location of management and customer workloads, identity isolation, and shared or isolated networking and security. Architecture Models Decide on a model according to your organization's requirements and your environment's resource capabilities. Implement a standard architecture for workload provisioning and mobility across VMware Cloud Foundation instances according to production best practices. If you plan to deploy a small-scale environment, or if you are working on an SDDC proof-of-concept, implement a consolidated architecture. VMware by Broadcom 10 VMware Cloud Foundation Design Guide Figure 1-1. Choosing a VMware Cloud Foundation Architecture Model Start choosing an architecture model for VMware Cloud Foundation Do you need to No minimize hardware requirements? Yes No Can management Yes Use the standard Use the consolidated and customer architecture model architecture model workloads be co-located? Table 1-1. Architecture Model Recommendations for VMware Cloud Foundation Requirement ID Design Requirement Justification Implication VCF-ARCH-RCMD-CFG-001 Use the standard n Aligns with the Requires additional architecture model of VMware best hardware. VMware Cloud Foundation. practice of separating management workloads from customer workloads. n Provides better long- term flexibility and expansion options. Workload Domain Types A workload domain represents a logical unit of application-ready infrastructure that groups ESXi hosts managed by a vCenter Server instance with specific characteristics according to VMware recommended practices. A workload domain can consist of one or more vSphere clusters, provisioned by SDDC Manager. VMware by Broadcom 11 VMware Cloud Foundation Design Guide Table 1-2. Workload Domain Types Workload Domain Type Description Benefits Drawbacks Management domain n First domain deployed. n Guaranteed sufficient n You must carefully n Contains the resources for size the domain to following management management accommodate planned appliances for all components deployment of VI workload domains: n Makes it possible to workload domains and use specific hardware additional management n vCenter Server to meet the needs only components. n NSX Manager of the management n Hardware might not be n SDDC Manager components fully utilized until full- n Optional. VMware n Makes it possible to scale deployment has Aria Suite use dedicated physical been reached. components compute, network and n Optional. storage separately Management from those used for domain NSX Edge additional workloads nodes n Enables separate n Has dedicated ESXi lifecycle management hosts of management and n First domain to workload components. upgrade. Consolidated domain n Represents a n Considers the minimum n Management management domain possible initial hardware components and which also runs and management customer workloads customer workloads. component footprint. are not isolated. n Uses resource n Can be scaled to a n You must constantly pools to ensure standard architecture monitor it to ensure sufficient resources model. sufficient resources for management for management components. components. n Migrating customer workloads to dedicated VI workloads domains is more complex. VMware by Broadcom 12 VMware Cloud Foundation Design Guide Table 1-2. Workload Domain Types (continued) Workload Domain Type Description Benefits Drawbacks VI workload domain n Represents an n Can share an NSX This workload domain type additional workload Manager instance with cannot provide distinct domain for running other VI workload vCenter Single Sign-On customer workloads. domains. domains for customer n Shares a vCenter n All workload domains workloads. Single Sign-On domain can be managed with the management through a single pane domain. of glass. n Shares identity provider n Reduces password configuration with the management overhead. management domain. n Enables independent n Has dedicated ESXi life cycle management. hosts. Isolated VI workload n Represents an n Can provide distinct n Workload domain domain additional workload vCenter Single Sign-On vCenter Server domain for running domains for customer instances are managed customer workloads. workloads. through different panes n Has a distinct vCenter n You can scale up to 24 of glass. Single Sign-On domain. VI workload domains n Additional password n Has a distinct identity per VMware Cloud management overhead provider configuration. Foundation instance. exists for administrators n Enables independent of VMware Cloud n Has dedicated ESXi life cycle management. Foundation. hosts. VMware by Broadcom 13 VMware Cloud Foundation Design Guide Figure 1-2. Choosing a VMware Cloud Foundation Workload Domain Type for Customer Workloads Table 1-3. Workload Domain Recommendations for VMware Cloud Foundation Recommendation ID Design Recommendation Justification Implication VCF-WLD-RCMD-CFG-001 Use VI workload domains n Aligns with the Requires additional or isolated VI workload VMware best hardware. domains for customer practice of separating workloads. management workloads from customer workloads. n Provides better long term flexibility and expansion options. VMware by Broadcom 14 VMware Cloud Foundation Design Guide Workload Domain Cluster to Rack Mapping in VMware Cloud Foundation VMware Cloud Foundation distributes the functionality of the SDDC across multiple workload domains and vSphere clusters. A workload domain, whether it is the management workload domain or a VI workload domain, is a logical abstraction of compute, storage, and network capacity, and consists of one or more vSphere clusters. Each cluster can exist vertically in a single rack or be spanned horizontally across multiple racks. The relationship between workload domain clusters and data center racks in VMware Cloud Foundation is not one-to-one. While a workload domain cluster is an atomic unit of repeatable building blocks, a rack is a unit of size. Because workload domain clusters can have different sizes, you map workload domain clusters to data center racks according to your requirements and physical infrastructure constraints. You determine the total number of racks for each cluster type according to your scalability needs. Table 1-4. Workload Domain Cluster to Rack Configuration Options Workload Domain Cluster to Rack Configuration Description Workload domain cluster in a single rack n The workload domain cluster occupies a single rack. n Can be used for shared edge and compute workloads in the same cluster. n Can be dedicated for compute-only workloads or for NSX Edge-only clusters. Workload domain cluster spanning multiple racks n The management domain default cluster can span multiple racks if the data center fabric can provide Layer 2 adjacency, such as VXLAN overlay in the fabric, between racks. If the Layer 3 fabric does not support this requirement, then the management default cluster must be mapped to a single rack. n A VI workload domain cluster dedicated to compute- only workloads, without SDDC Manager deployed NSX Edge clusters, can span racks when using NSX Overlay in conjuction with a Layer 3 network fabric without Layer 2 adjacency between racks. n A vSphere cluster that is to host only an NSX Edge cluster, deployed from SDDC Manager, must be deployed in a single rack. To increase redundancy, you must deploy two vSphere clusters to two separate racks with edge nodes in both racks. Workload domain cluster with multiple availability zones, n To span multiple availability zones, the network fabric each zone in a single rack must support stretched Layer 2 networks and Layer 3 routed networks between the availability zones. n A cluster spanning multiple racks is not supported with multiple availability zones. VMware by Broadcom 15 VMware Cloud Foundation Design Guide Figure 1-3. Workload Domain Cluster in a Single Rack Data Center Fabric ToR ToR Switch Switch VI Workload Domain Cluster Management Domain Cluster Rack VMware by Broadcom 16 VMware Cloud Foundation Design Guide Figure 1-4. Workload Domain Cluster Spanning Multiple Racks Data Center Fabric ToR ToR ToR ToR Switch Switch Switch Switch VI Workload Domain Cluster Management Domain Cluster Rack Rack VMware by Broadcom 17 VMware Cloud Foundation Design Guide Figure 1-5. Workload Domain Cluster with Multiple Availability Zones, Each Zone in One Rack Data Center Fabric Availability Zone 1 Availability Zone 2 ToR ToR ToR ToR Switch Switch Switch Switch VI Workload Domain Stretched Cluster Management Domain Stretched Cluster Rack Rack Networking Models in VMware Cloud Foundation VMware Cloud Foundation supports multiple networking models that provide different levels of network availability, resilience, and scale. Networking Model Description Benefits Drawbacks NSX overlay Overlay-backed NSX n Supports AVNs in the n Requires routing segments management domain. configured on the n Supports deployment physical network fabric of NSX Edge clusters to route traffic from the from SDDC Manager. NSX Edge nodes. NSX VLAN-backed VLAN-backed NSX n Deploying NSX Edge n Requires changes in the segments clusters is not required. physical network fabric n Routing configuration to add new networks on the ToR switches for and VLANs. NSX Edge nodes is not required. VMware by Broadcom 18 VMware Cloud Foundation Design Guide VMware Cloud Foundation Topologies VMware Cloud Foundation supports multiple topologies that provide different levels of availability and scale. Availability Zones and VMware Cloud Foundation Instances Availability zone An availability zone is a fault domain at the SDDC level. You create multiple availability zones for the purpose of creating vSAN stretched clusters. Using multiple availability zones can improve availability of management components and workloads running within the SDDC, minimize downtime of services, and improve SLAs. Availability zones are typically located either within the same data center, but in different racks, chassis, rooms, or in different data centers with low-latency high-speed links connecting them. Note Only stretched clusters created by using the Stretch Cluster API, and are therefore vSAN storage based, are considered by and treated as stretched clusters by VMware Cloud Foundation. VMware Cloud Foundation Instance Each VMware Cloud Foundation instance is a separate VMware Cloud Foundation deployment and might contain one or two availability zones. VMware Cloud Foundation instances may be geographically separate. VMware Cloud Foundation Topologies Several topologies of VMware Cloud Foundation exist according to the number of availability zones and VMware Cloud Foundation instances. Table 1-5. VMware Cloud Foundation Topologies Topology Description Single Instance - Single Availability Zone Workload domains are deployed in a single availability zone. Single Instance - Multiple Availability Zones Workload domains might be stretched between two availability zones. Multiple Instances - Single Availability Zone per VMware Workload domains in each instance are deployed in a Cloud Foundation instance single availability zone. Multiple Instances - Multiple Availability Zones per Workload domains in each instance might be stretched VMware Cloud Foundation instance between two availability zones. VMware by Broadcom 19 VMware Cloud Foundation Design Guide Figure 1-6. Choosing a VMware Cloud Foundation Topology Start choosing a VMware Cloud Foundation topology Do you need No disaster recovery Yes or more than 24 VI workload domains? No Do you need Yes No Do you need Yes vSAN stretched vSAN stretched clusters? clusters? Use the Single Instance - Use the Single Instance - Use the Multiple Instance - Use the Multiple Instance - Single Avalibility Multiple Avalibility Single Avalibility Multiple Avalibility Zone topology Zones topology Zones topology Zones topology What to read next n Single Instance - Single Availability Zone Single Instance - Single Availability Zone is the simplest VMware Cloud Foundation topology where workload domains are deployed in a single availability zone. n Single Instance - Multiple Availability Zones You protect your VMware Cloud Foundation environment against a failure of a single hardware fault domain by implementing multiple availability zones. n Multiple Instances - Single Availability Zone per Instance You protect against a failure of a single VMware Cloud Foundation instance by implementing multiple VMware Cloud Foundation instances. n Multiple Instances - Multiple Availability Zones per Instance You protect against a failure of a single VMware Cloud Foundation instance by implementing multiple VMware Cloud Foundation instances. Implementing multiple availability zones in an instance protects against a failure of a single hardware fault domain. Single Instance - Single Availability Zone Single Instance - Single Availability Zone is the simplest VMware Cloud Foundation topology where workload domains are deployed in a single availability zone. The Single Instance - Single Availability Zone topology relies on vSphere HA to protect workloads against host failures. VMware by Broadcom 20 VMware Cloud Foundation Design Guide Figure 1-7. Single VMware Cloud Foundation Instance with a Single Availability Zone VCF Instance VI Workload Domain Management Domain Table 1-6. Single Instance - Single Availability Zone Attributes Attributes Detail Data centers Single data center Workload domain cluster rack mapping n Workload domain cluster in a single rack n Workload domain cluster spanning multiple racks Scale n Up to 25 workload domains n Up to 15 workload domains in a single vCenter Single Sign-On domain including the management domain. Resilience vSphere HA protects against host failures Single Instance - Multiple Availability Zones You protect your VMware Cloud Foundation environment against a failure of a single hardware fault domain by implementing multiple availability zones. Incorporating multiple availability zones in your design can help reduce the blast radius of a failure and can increase application availability. You usually deploy multiple availability zones across two independent data centers. VMware by Broadcom 21 VMware Cloud Foundation Design Guide Figure 1-8. Single VMware Cloud Foundation Instance with Multiple Availability Zones VCF Instance Availability Zone 1 Availability Zone 2 Stretched vSAN Cluster VI Workload Domain VI Workload Domain VI Workload Domain Stretched vSAN Cluster VI Workload Domain Stretched vSAN Cluster Management Domain VMware by Broadcom 22 VMware Cloud Foundation Design Guide Table 1-7. Single Instance - Multiple Availability Zone Attributes Attributes Detail Workload domain cluster rack mapping n Workload domain cluster in a single rack. n Workload domain cluster spanning multiple racks. n Workload domain cluster with multiple availability zones, each zone in a single rack. n Workload domain cluster with multiple availability zones, each zone spanning multiple racks (not supported with a multi-rack compute cluster). Stretched cluster n Because availability zones use VMware vSAN™ stretched clusters, the bandwidth between the zones must be at least 10 Gbps and the round-trip latency must be less than 5 ms. n Having the management domain on a vSAN stretched cluster is a prerequisite to configure and implement vSAN stretched clusters in your VI workload domains. In this way, the critical management components to manage the VI workload domains are still available if a site failure occurs. n You can have up to two availability zones. Scale n Up to 25 workload domains. Up to 15 workload domains in a single vCenter Single Sign-On domain. Resilience n vSphere HA protects workloads against host failures. n Multiple availability zones protect against data center failures. Multiple Instances - Single Availability Zone per Instance You protect against a failure of a single VMware Cloud Foundation instance by implementing multiple VMware Cloud Foundation instances. Incorporating multiple VMware Cloud Foundation instances in your design can help reduce the blast radius of a failure and can increase application availability across larger geographical distances than cannot be achieved by using multiple availability zones. You usually deploy this topology in the same data center for scale or across independent data centers for resilience. VMware by Broadcom 23 VMware Cloud Foundation Design Guide Figure 1-9. Multiple VMware Cloud Foundation Instances with a Single Availability Zone per Instance VCF Instance 1 VI Workload Domain Management Domain VCF Instance 2 VI Workload Domain Management Domain VMware by Broadcom 24 VMware Cloud Foundation Design Guide Table 1-8. Multiple Instance - Single Availability Zone Attributes Attributes Detail Workload domain cluster rack mapping n Workload domain cluster in a single rack n Workload domain cluster spanning multiple racks Multiple instances Using multiple VMware Cloud Foundation instances can facilitate the following use cases: n Disaster recovery across different VMware Cloud Foundation instances at longer distances n Scale beyond the maximums of a single VMware Cloud Foundation instance. n Co-location of end users and resources If you plan to use NSX Federation between VMware Cloud Foundation instances, the following considerations exist: n Up to four locations when using medium-size NSX Global Managers n Up to 16 locations when using large or extra large NSX Global Managers n Up to four locations per cross-instance Tier-0 gateway n Check https://configmax.vmware.com/ to select the right NSX Global Managers form factor for your scale needs. n Lifecycle management must be planned carefully Scale n Up to 25 workload domains per VMware Cloud Foundation instance Up to 15 workload domains in a single vCenter Single Sign-on domain per instance Resilience n vSphere HA protects workloads against host failures. n Deploying multiple instances can protect against natural disasters by providing recovery locations at greater geographical distances. Multiple Instances - Multiple Availability Zones per Instance You protect against a failure of a single VMware Cloud Foundation instance by implementing multiple VMware Cloud Foundation instances. Implementing multiple availability zones in an instance protects against a failure of a single hardware fault domain. Incorporating multiple VMware Cloud Foundation instances into your design can help reduce the blast radius of a failure and can increase application availability across larger geographical distances that cannot be achieved using multiple availability zones. VMware by Broadcom 25 VMware Cloud Foundation Design Guide Figure 1-10. Multiple VMware Cloud Foundation Instances with Multiple Availability Zones per Instance VCF Instance 1 Availability Zone 1 Availability Zone 2 Stretched vSAN Cluster VI Workload Domain VI Workload Domain VI Workload Domain Stretched vSAN Cluster VI Workload Domain Stretched vSAN Cluster Management Domain VCF Instance 2 Availability Zone 1 Availability Zone 2 Stretched vSAN Cluster VI Workload Domain VMware by Broadcom 26 VMware Cloud Foundation Design Guide Table 1-9. Multiple Instance - Multiple Availability Zone Attributes Attributes Detail Workload domain cluster rack mapping n Workload domain cluster in a single rack n Workload domain cluster spanning multiple racks n Workload domain cluster with multiple availability zones, each zone in a single rack n Workload domain cluster with multiple availability zones, each zone spanning multiple racks (not supported with multi-rack compute clusters) Multiple instances Using multiple VMware Cloud Foundation instances can facilitate the following: n Disaster recovery across different VMware Cloud Foundation instances at longer distances n Scale beyond the maximums of a single VMware Cloud Foundation instance n Co-location of end users and resources If you plan to use NSX Federation between instances, VMware Cloud Foundation consider the following: n Up to four locations when using medium-size NSX Global Managers n Up to 16 locations when using large-size NSX Global Managers n Up to four locations per stretched Tier-0 gateway n Lifecycle management must be planned carefully Stretched cluster n Because availability zones use VMware vSAN™ stretched clusters, the bandwidth between the zones must be at least 10 Gbps and the round-trip latency must be less than 5 ms. n You can have up to two availability zones. n Having the management domain on a vSAN stretched cluster is a prerequisite to configure and implement vSAN stretched clusters in your VI workload domains. Scale n Up to 25 workload domains per VMware Cloud Foundation instance Up to 15 workload domains in a single vCenter Single Sign-On domain per instance Resilience n vSphere HA protects workloads against host failures. n Multiple availability zones protect against data center failures. n Multiple instances can protect against natural disasters by providing recovery locations at greater geographical distances. VMware by Broadcom 27 External Services Design for VMware Cloud Foundation 2 IP addressing scheme, name resolution, and time synchronization must support the requirements for VMware Cloud Foundation deployments. Table 2-1. External Services Design Requirements for VMware Cloud Foundation Requirement ID Design Requirement Justification Implication VCF-EXT-REQD-NET-001 Allocate statically assigned Ensures stability across the You must provide precise IP addresses and host VMware Cloud Foundation IP address management. names for all workload instance, and makes it domain components. simpler to maintain, track, and implement a DNS configuration. VCF-EXT-REQD-NET-002 Configure forward and Ensures that all You must provide reverse DNS records components are accessible DNS records for each for all workload domain by using a fully qualified component. components. domain name instead of by using IP addresses only. It is easier to remember and connect to components across the VMware Cloud Foundation instance. VCF-EXT-REQD-NET-003 Configure time Ensures that all An operational NTP service synchronization by using an components are must be available in the internal NTP time source synchronized with a valid environment. for all workload domain time source. components. VCF-EXT-REQD-NET-004 Set the NTP service Ensures that the None. for all workload domain NTP service remains components to start synchronized after you automatically. restart a component. VMware by Broadcom 28 Physical Network Infrastructure Design for VMware Cloud Foundation 3 Design of the physical data center network includes defining the network topology for connecting physical switches and ESXi hosts, determining switch port settings for VLANs and link aggregation, and designing routing. A software-defined network (SDN) both integrates with and uses components of the physical data center. SDN integrates with your physical network to support east-west transit in the data center and north-south transit to and from the SDDC networks. Several typical data center network deployment topologies exist: n Core-Aggregation-Access n Leaf-Spine n Hardware SDN Note Leaf-Spine is the default data center network deployment topology used for VMware Cloud Foundation. Read the following topics next: n VLANs and Subnets for VMware Cloud Foundation n Leaf-Spine Physical Network Design Requirements and Recommendations for VMware Cloud Foundation VLANs and Subnets for VMware Cloud Foundation Configure your VLANs and subnets according to the guidelines and requirements for VMware Cloud Foundation. When designing the VLAN and subnet configuration for your VMware Cloud Foundation deployment, consider the following guidelines: VMware by Broadcom 29 VMware Cloud Foundation Design Guide Table 3-1. VLAN and Subnet Guidelines for VMware Cloud Foundation NSX Federation Between Multiple VMware Cloud Multi-Rack Compute VI All Deployment Topologies Multiple Availability Zones Foundation Instances Workload Domain Cluster n Ensure your n For network segments n An RTEP network Use the RFC 1918 IPv4 subnets are scaled which are stretched segment should have a address space for these appropriately to allow between availability VLAN ID and Layer 3 subnets and allocate one for expansion as zones, the VLAN range that is specific octet by rack and another expanding at a later ID must meet the to the VMware Cloud by network function. time can be disruptive. following requirements: Foundation instance. n Use the IP address n Be the same in both n In a VMware Cloud of the floating availability zones Foundation instance interface for Virtual with the same with multiple availability Router Redundancy Layer 3 network zones, the RTEP Protocol (VRPP) or segments. network segment must Hot Standby Routing n Have a Layer 3 be stretched between Protocol (HSRP) as the gateway at the first the zones and assigned gateway. hop that is highly the same VLAN ID and n Use the RFC 1918 available such that IP range. IPv4 address space it tolerates the n All Edge RTEP networks for these subnets and failure of an entire must reach each other. allocate one octet availability zone. by VMware Cloud n For network segments Foundation instance of the same type and another octet by which are not stretched function. between availability zones, the VLAN ID can be the same or different between the zones. When deploying VLANs and subnets for VMware Cloud Foundation, they must conform to the following requirements according to the VMware Cloud Foundation topology: VMware by Broadcom 30 VMware Cloud Foundation Design Guide Figure 3-1. Choosing a VLAN Model for Host and Management VM Traffic Start choosing a management VLAN model Is separate security Yes access required for ESXi host and VM management? No Yes Is isolating host No Use separate VLANs for Use the same VLAN for from VM management managing VMs and ESXi hosts managing VMs and ESXi hosts traffic required? Table 3-2. VLANs and Subnets for Availability Zones and Instances for VMware Cloud Foundation VMware Cloud Foundation Instances VMware Cloud Foundation Instances Function with a Single Availability Zone with Multiple Availability Zones VM management n Required when separate from n Required when separate from host management host management n Highly available gateway within n Must be stretched within the the instance instance n Highly available gateway across availability zones within the instance Host management - first availability n Required n Required zone n Highly available gateway within n Highly available gateway across the instance availability zones within the instance vSphere vMotion - first availability n Required n Required zone n Highly available gateway within n Highly available gateway in the instance first availability zone within the instance VMware by Broadcom 31 VMware Cloud Foundation Design Guide Table 3-2. VLANs and Subnets for Availability Zones and Instances for VMware Cloud Foundation (continued) VMware Cloud Foundation Instances VMware Cloud Foundation Instances Function with a Single Availability Zone with Multiple Availability Zones vSAN - first availability zone n Required for the default cluster of n Required the management domain n Highly available gateway in n Highly available gateway within first availability zone within the the instance instance Host overlay - first availability zone n Required n Required n Highly available gateway within n Highly available gateway in the instance first availability zone within the instance NFS n Required if using NFS as principal n Not Supported storage in the VI workload domain Uplink01 n Required n Required n Gateway optional n Gateway optional n Must be stretched within the instance Uplink02 n Required n Required n Gateway optional n Gateway optional n Must be stretched within the instance Edge overlay n Required n Required n Highly available gateway within n Must be stretched within the the instance instance n Highly available gateway across availability zones within the instance Host management - second n Not required n Required availability zone n Highly available gateway in second availability zone within the instance vSphere vMotion - second availability n Not required n Required zone n Highly available gateway in second availability zone within the instance vSAN - second availability zone n Not required n Required n Highly available gateway in second availability zone within the instance Host overlay - second availability n Not required n Required zone n Highly available gateway in second availability zone within the instance VMware by Broadcom 32 VMware Cloud Foundation Design Guide Table 3-2. VLANs and Subnets for Availability Zones and Instances for VMware Cloud Foundation (continued) VMware Cloud Foundation Instances VMware Cloud Foundation Instances Function with a Single Availability Zone with Multiple Availability Zones Edge RTEP n Required for NSX Federation only n Required for NSX Federation only n Highly available gateway within n Must be stretched within the the instance instance n Highly available gateway across availability zones within the instance Management and Witness - witness n Not required n Required appliance at a third location n Highly available gateway at the witness location Table 3-3. VLANs and Subnets for Multi-Rack Deployments with VMware Cloud Foundation VMware Cloud Foundation Instances VMware Cloud Foundation Instances With Multi-Rack Compute VI with Multi-Rack NSX Edge Function Workload Domain Cluster Availability VM management n Not required as compute-only n Required when separate from host management n Highly available gateway at the ToR switched or leaf nodes in the rack Host management n Required per rack n Required per rack n Highly available gateway at the n Highly available gateway at the ToR switched or leaf nodes in the ToR switched or leaf nodes in the rack rack vSphere vMotion n Required per rack n Required per rack n Highly available gateway at the n Highly available gateway at the ToR switched or leaf nodes in the ToR switched or leaf nodes in the rack rack vSAN n Required per rack n Required per rack if using vSAN n Highly available gateway at the n Highly available gateway at the ToR switched or leaf nodes in the ToR switched or leaf nodes in the rack rack Host overlay n Required per rack n Required per rack n Highly available gateway at the n Highly available gateway at the ToR switched or leaf nodes in the ToR switched or leaf nodes in the rack rack NFS n Not supported n Required if using NFS as principal storage for the clusters with the NSX Edge nodes. Uplink01 n Not required n Required per rack VMware by Broadcom 33 VMware Cloud Foundation Design Guide Table 3-3. VLANs and Subnets for Multi-Rack Deployments with VMware Cloud Foundation (continued) VMware Cloud Foundation Instances VMware Cloud Foundation Instances With Multi-Rack Compute VI with Multi-Rack NSX Edge Function Workload Domain Cluster Availability Uplink02 n Not required n Required per rack Edge overlay n Not required n Required per rack n Highly available gateway at the ToR switched or leaf nodes in the rack Leaf-Spine Physical Network Design Requirements and Recommendations for VMware Cloud Foundation Leaf-Spine is the default data center network deployment topology used for VMware Cloud Foundation. Consider network bandwidth, trunk port configuration, jumbo frames and routing configuration for NSX in a deployment with a single or multiple VMware Cloud Foundation instances. Leaf-Spine Physical Network Logical Design Each ESXi host has redundant connectivity to the top-of-rack (ToR) switches of the SDDC network fabric by two or more ports with a recommended speed of 25-GbE or higher. The ToR switches are configured to provide all necessary VLANs using an 802.1Q trunk. These redundant connections use features in vSphere Distributed Switch and NSX to guarantee that no physical interface is overrun and available redundant paths are used. VMware by Broadcom 34 VMware Cloud Foundation Design Guide Figure 3-2. Leaf-Spine Physical Network Logical Design Data Center Fabric Spine ToR ToR ToR ToR Switch Switch Switch Switch ESXi ESXi Hosts Hosts Leaf-Spine Physical Network Design Requirements and Recommendations The requirements and recommendations for the leaf-spine network configuration determine the physical layout and use of VLANs. They also include requirements and recommendations on jumbo frames, and on network-related requirements such as DNS, NTP and routing. VMware by Broadcom 35 VMware Cloud Foundation Design Guide Table 3-4. Leaf-Spine Physical Network Design Requirements for VMware Cloud Foundation Requirement ID Design Requirement Justification Implication VCF-NET-REQD-CFG-001 Do not use EtherChannel n Simplifies configuration None. (LAG, LACP, or vPC) of top-of-rack switches. configuration for ESXi host n Teaming options uplinks. available with vSphere Distributed Switch provide load balancing and failover. n EtherChannel implementations might have vendor-specific limitations. VCF-NET-REQD-CFG-002 Use VLANs to separate n Supports physical Requires uniform physical network functions. network connectivity configuration and without requiring many presentation on all the NICs. trunks that are made n Isolates the different available to the ESXi hosts. network functions in the SDDC so that you can have differentiated services and prioritized traffic as needed. VCF-NET-REQD-CFG-003 Configure the VLANs as All VLANs become Optionally, the members of a 802.1Q trunk. available on the same management VLAN can act physical network adapters as the native VLAN. on the ESXi hosts. VCF-NET-REQD-CFG-004 Set the MTU size to at least n Improves traffic When adjusting the MTU 1,700 bytes (recommended throughput. packet size, you must 9,000 bytes for jumbo n Supports Geneve by also configure the entire frames) on the physical increasing the MTU size network path (VMkernel switch ports, vSphere to a minimum of 1,600 network adapters, virtual Distributed Switches, bytes. switches, physical switches, vSphere Distributed Switch and routers) to support the n Geneve is an extensible port groups, and N-VDS same MTU packet size. protocol. The MTU switches that support the In an environment with size might increase following traffic types: multiple availability zones, with future capabilities. n Overlay (Geneve) While 1,600 bytes the MTU must be n vSAN is sufficient, an MTU configured on the entire size of 1,700 bytes network path between the n vSphere vMotion provides more room for zones. increasing the Geneve MTU size without the need to change the MTU size of the physical infrastructure. VMware by Broadcom 36 VMware Cloud Foundation Design Guide Table 3-5. Leaf-Spine Physical Network Design Requirements for Multi-Rack Compute VI Workload Domain Cluster for VMware Cloud Foundation Requirement ID Design Requirement Justification Implication VCF-NET-L3MR-REQD- For a multi-rack compute A Layer 3 leaf-spine fabric Requires additional VLANs CFG-001 VI workload domain cluster, has a Layer 3 boundary at for each rack. provide separate VLANs the leaf switches in each per rack for each network rack creating a Layer 3 function. boundary between racks. n Host management n vSAN n vSphere vMotion n Host overlay VCF-NET-L3MR-REQD- For a multi-rack compute Ensures the traffic for each Requires additional physical CFG-002 VI workload domain cluster, network can flow between network configuration to the subnets for each racks. make networks routable network per rack must be between racks. routable and reachable to the leaf switches in the other racks. n Host management n vSAN n vSphere vMotion n Host overlay VMware by Broadcom 37 VMware Cloud Foundation Design Guide Table 3-6. Leaf-Spine Physical Network Design Requirements for NSX Federation in VMware Cloud Foundation Requirement ID Design Requirement Justification Implication VCF-NET-REQD-CFG-005 Set the MTU size to at least n Jumbo frames are When adjusting the MTU 1,500 bytes (1,700 bytes not required between packet size, you must preferred; 9,000 bytes VMware Cloud also configure the entire recommended for jumbo Foundation instances. network path, that is, frames) on the components However, increased virtual interfaces, virtual of the physical network MTU improves traffic switches, physical switches, between the VMware throughput. and routers to support the Cloud Foundation instances n Increasing the RTEP same MTU packet size. for the following traffic MTU to 1,700 types. bytes minimizes n NSX Edge RTEP fragmentation for standard-size workload packets between VMware Cloud Foundation instances. VCF-NET-REQD-CFG-006 Ensure that the latency A latency lower than 500 None. between VMware Cloud ms is required for NSX Foundation instances that Federation. are connected in an NSX Federation is less than 500 ms. VCF-NET-REQD-CFG-007 Provide a routed Configuring NSX You must assign unique connection between the Federation requires routable IP addresses for NSX Manager clusters in connectivity between the each fault domain. VMware Cloud Foundation NSX Global Manager instances that are instances, NSX Local connected in an NSX Manager instances, and Federation. NSX Edge clusters. VMware by Broadcom 38 VMware Cloud Foundation Design Guide Table 3-7. Leaf-Spine Physical Network Design Recommendations for VMware Cloud Foundation Recommendation ID Design Recommendation Justification Implication VCF-NET-RCMD-CFG-001 Use two ToR switches for Supports the use of Requires two ToR switches each rack. two 10-GbE (25-GbE or per rack which might greater recommended) increase costs. links to each server, provides redundancy and reduces the overall design complexity. VCF-NET-RCMD-CFG-002 Implement the following n Provides availability n Might limit the physical network during a switch failure. hardware choices. architecture: n Provides support for n Requires dynamic n At least one 25-GbE BGP dynamic routing routing protocol (10-GbE minimum) port protocol configuration in the on each ToR switch for physical network. ESXi host uplinks (Host to ToR). n Layer 3 device that supports BGP. VCF-NET-RCMD-CFG-003 Use a physical network n Supports design Requires BGP configuration that is configured for BGP flexibility for routing in the physical network. routing adjacency. multi-site and multi- tenancy workloads. n BGP is the only dynamic routing protocol that is supported for NSX Federation. n Supports failover between ECMP Edge uplinks. VCF-NET-RCMD-CFG-004 Assign persistent IP n Ensures that endpoints Adding more hosts to configurations for NSX have a persistent TEP the cluster may require tunnel endpoints (TEPs) IP address. the static IP pools to be that use static IP pools n In VMware Cloud increased.. instead of dynamic IP pool Foundation, TEP IP addressing. assignment by using static IP pools is recommended for all topologies. n This configuration removes any requirement for external DHCP services. VCF-NET-RCMD-CFG-005 Configure the trunk ports Reduces the time to Although this design connected to ESXi NICs as transition ports over to the does not use the STP, trunk PortFast. forwarding state. switches usually have STP configured by default. VMware by Broadcom 39 VMware Cloud Foundation Design Guide Table 3-7. Leaf-Spine Physical Network Design Recommendations for VMware Cloud Foundation (continued) Recommendation ID Design Recommendation Justification Implication VCF-NET-RCMD-CFG-006 Configure VRRP, HSRP, or Ensures that the VLANs Requires configuration of a another Layer 3 gateway that are stretched high availability technology availability method for between availability zones for the Layer 3 gateways in these networks. are connected to a the data center. n Management highly- available gateway. Otherwise, a failure in the n Edge overlay Layer 3 gateway will cause disruption in the traffic in the SDN setup. VCF-NET-RCMD-CFG-007 Use separate VLANs for Reduces the size of the Increases the overall the network functions for Layer 2 broadcast domain number of VLANs that are each cluster. to a single vSphere cluster. required for a VMware Cloud Foundation instance. Table 3-8. Leaf-Spine Physical Network Design Recommendations for Dedicated Edge Scale and Performance for VMware Cloud Foundation Recommendation ID Design Recommendation Justification Implication VCF-NET-DES-RCMD- Implement the following Supports the requirements Requires 100-GbE network CFG-001 physical network for high bandwidth and switches. architecture: packets per second for n Two 100-GbE ports on large-scale deployments. each ToR switch for ESXi host uplinks (Host to ToR). n Layer 3 device that supports BGP. Table 3-9. Leaf-Spine Physical Network Design Recommendations for NSX Federation in VMware Cloud Foundation Recommendation ID Design Recommendation Justification Implication VCF-NET-RCMD-CFG-008 Provide BGP routing BGP is the supported None. between all VMware Cloud routing protocol for NSX Foundation instances that Federation. are connected in an NSX Federation setup. VCF-NET-RCMD-CFG-009 Ensure that the latency A latency lower than 150 None. between VMware Cloud ms is required for the Foundation instances that following features: are connected in an NSX n Cross vCenter Server Federation is less than 150 vMotion ms for workload mobility. VMware by Broadcom 40 Supported Storage Types for VMware Cloud Foundation 4 Storage design for VMware Cloud Foundation includes the design for principal and supplemental storage. Principal storage is used during the creation of a workload domain and is capable of running workloads. Supplemental storage can be added after the creation of a workload domain and can be capable of running workloads or be used for data at rest storage such as virtual machine templates, backup data, and ISO images. Special considerations apply if you plan to add clusters to the management domain, for example, to separate additional management components that require specific hardware resources or might impact the performance of the main management components in the default cluster, or, in the case of the consolidated architecture of VMware Cloud Foundation, to separate customer workloads from the management components. VMware Cloud Foundation supports the following principal and supplemental storage combinations for clean deployments of VMware Cloud Foundation. Note For information about the supported storage types for vSphere environments converted or imported into VMware Cloud Foundation, see Converting or Importing Existing vSphere Environments into VMware Cloud Foundation in the VMware Cloud Foundation Administration Guide. Table 4-1. Supported Storage Types in VMware Cloud Foundation Management Domain - Management Domain - Storage Type Default Cluster Additional Clusters VI Workload Domain vSAN Original Storage Principal Principal Principal Architecture (OSA) vSAN Expr