Information Technology Sites & Critical Loads PDF

Reference: Information Technology Sites and Critical Loads INTRODUCTION utilities and emergency and recovery/business continu- An Information Technology Equipment (ITE) center is ity procedures. an area designed to house and protect...

Reference: Information Technology Sites and Critical Loads INTRODUCTION utilities and emergency and recovery/business continu- An Information Technology Equipment (ITE) center is ity procedures. an area designed to house and protect information tech- Three important definitions included in NFPA 75 are nology equipment and allow for proper connectivity as follows: and usage of the IT system. The equipment can vary IT Equipment area - An area of a building where the widely from computers, servers and routers to phone information technology equipment room is located, switches and PBXs. ITE centers are typically supported including support rooms served by the same special air- by: electrical power, heating/cooling, equipment alarms conditioning/air-handling equipment as the IT equip- and monitoring, fire suppression, and security systems. ment room. IT Equipment room - A room within the IT equip- ment area that contains the IT equipment. IT Equipment system - Any electronic digital or analog computer, along with all peripheral, support, memory, programming, or other directly associated equipment, records, storage, and activities. WHY ARE ITE CENTERS IMPORTANT? The consequences of a loss of power to the load can have major impact on ITE sites. Recent studies show that industrial and digital business firms are losing $45.7 billion per year due to power outages. Across all business sectors, an estimated $104 billion to $164 bil- lion is lost to outages with another $15 billion to $24 billion to power quality problems. (1) In industrial automatic processing, whole production lines can go out of control, creating hazardous situations for onsite personnel and expensive material waste. Loss of processing in a large financial site can cost Electrical power is a key element as all other systems many unrecoverable thousands of dollars per minute of are totally dependent upon it. Because of this, power down time, as well as many hours of recovery time to systems have been designed so that they attempt to follow. Program and data corruption caused by a power eliminate any weaknesses that might interrupt the interruption can create problems for software person- 7 × 24 power availability of the site. These designs vary nel for days, weeks and sometimes months after the in levels of complexity, according to the criticality of incident. the intended ITE installation. Article 645 of the NEC is titled “Information Tech- WHAT IS DIFFERENT ABOUT WORKING AT nology Equipment” and it is applicable to facilities THESE SITES? housing computers and peripherals, server networks, All work, whether it be installing, testing, routine telecommunications equipment, process control com- maintenance, trouble-shooting or repair of electrical puters and electronic control and instrumentation. The systems in a critical mission environment will usually term “Information Technology Equipment” replaces involve procedural requirements, policies and planning other terms that describe computer-based business, with ITE management personnel. They are responsible personal, and industrial equipment. It has been ad- for the continuing uptime of the installation and con- opted by standards organizations globally as a more in- sider work performed on the critical infrastructure a clusive term. high-risk operation. It has been reported that “Human ITE is also addressed in NFPA 75: Standard for the error during operating procedures and hands-on main- Protection of Information Technology Equipment. This tenance have consistently accounted for 54% of critical standard covers the minimum requirements for the system shutdowns”. (2) Because of this, it is not an un- protection of IT equipment from damage by fire or its common requirement that any “hands on” procedures associated effects — namely, smoke, corrosion, heat, take place when the load exposure is minimal and the and water. It identifies risk factors, and sets rules for data traffic is low. building construction and materials, IT equipment, This kind of scheduling makes allowance for recovery fire protection and detection, record keeping/storage operations to be implemented should there be problems RE10830–1 © electrical training ALLIANCE 2017 2 Reference Information Technology Sites and Critical Loads in bringing the system back to a normal operating Extra care must be exercised around cabling, fire de- mode. The conduct and performance of anyone work- tectors, water sprinkler heads, water detection systems ing at these sites must be at a very high standard. and other sub floor devices. Clean-up operations must Every site will have it’s own customized set of rules be meticulous and special filters must be installed on and procedures and it is important to understand the vacuums to avoid particle contamination affecting sen- local requirements before work can proceed. Some of sitive smoke detectors. the policies may be familiar and apply to a wide variety of sites while others will be unique to the ITE environ- WHAT IS THE DIFFERENCE BETWEEN ment. For example, it may be necessary to use bypass ITE SITES? operations in parts of an electrical system when work is ITE centers have been categorized (3) into four tiers of to be performed, and some alarm systems may need to infrastructure support; the levels are based upon the be disabled. amount of redundancy that is built into the supporting Welding or cutting operations must be performed systems. The systems are described in terms of N, where with great care and awareness of nearby equipment and N refers to the number of units required to support the will likely require permits and separate ventilation. Ra- load. For example, in a system with 100 kW load, N dios and cellular phones are usually banned in the ITE would be the number of units required to support center as their signals can interfere with data transmis- 100 kW. sion. Particular care is required when opening panels or working around breakers and controls to avoid acci- TIER ONE (N SYSTEM) dentally tripping active circuits. Tier one ITE centers are most common, and are typi- Some ITE rooms have raised floors that are used as a cally smaller, less critical operations, (5 to 100 racks of large air plenum, delivering cold air from Computer equipment). The electrical and mechanical layout for Room Air Conditioning (CRAC) units to the under- the ITE center may or may not incorporate a standby side of IT equipment racks. This area sometimes con- generator, a UPS and dedicated CRAC. However, at tains power and data cables; access to the under floor this level, the data center will be somewhat susceptible area may require authorization. Removal of a large to system failure, as the critical power path contains number of tiles to access power cabling will lower the several “single points of failure”, (SPOF). static pressure, reducing the airflow and causing the IT equipment temperature to rise. RE10830–2 © electrical training ALLIANCE 2017 3 Reference Information Technology Sites and Critical Loads NOTE: A single point of failure (SPOF) is an element TIER THREE (2N) of the electrical distribution system that at some point will Tier three ITE centers are concurrently maintainable, cause downtime, if a means to bypass it is not developed in i.e., they have the capability of being maintained with- the system. out any interruption to the ITE center operation. They Tier one sites are also more susceptible to shutdowns are often sited in a dedicated facility. Planned activities because of changes needed on the building’s infrastruc- include preventive and programmable maintenance, ture, such as annual planned or unplanned mainte- repair and replacement of components, addition or re- nance to the main switchgear, or annual chilled water moval of capacity components, testing of components system maintenance. and systems, and more. For large sites using chilled water, this means two independent sets of pipes. TIER TWO (N + 1) Sufficient electrical power capacity and distribution Tier two ITE centers have a higher mission critical role must be available to simultaneously carry the load on than tier one sites, so redundancy is built into the data one path while performing maintenance or testing on center design. This means that, for example, alternate the other path. Unplanned activities such as errors in switchgear may provide a choice of power sources for operation or spontaneous failures of facility infrastruc- multiple UPSs as well as alternate power paths to ture components will still cause a data center disrup- CRAC units in the ITE center, so that the system is tion. Tier 3 sites are often designed for upgrade to somewhat tolerant of planned or unplanned mainte- Tier 4 when the client’s business case justifies the cost nance operations. Tier two data centers will still require of additional protection. a shutdown should maintenance be required on the main switch-gear, or building HVAC systems. RE10830–3 © electrical training ALLIANCE 2017 4 Reference Information Technology Sites and Critical Loads TIER FOUR (2N + 1) This requires simultaneously active distribution Tier four sites are fault tolerant. They are typically very paths, typically in a System + System configuration. large dedicated sites with highly mission-critical roles. Electrically, this means two separate UPS systems in They provide infrastructure capacity and capability to which each system has a backup UPS ready to take over permit any planned activity without disruption to the the load from the primary unit. Tier 4 requires all com- critical load. Fault-tolerant functionality also provides puter hardware to have dual power inputs. Because of the ability of the site infrastructure to sustain at least fire and electrical safety codes, there will still be down- one worst-case unplanned failure or event with no crit- time exposure due to fire alarms or people initiating an ical load impact. emergency power off (EPO) operation. RE10830–4 © electrical training ALLIANCE 2017 5 Reference Information Technology Sites and Critical Loads WHAT ARE THE ENTRY AND EXIT Often, other switches, such as an automatic door ac- PROCEDURES FOR AN ITE CENTER? tuation switch, may be mounted in close proximity to It is very common for ITE rooms to have restricted ac- the required disconnecting means. In this arrangement, cess doors. Physical security — controlling personnel personnel unfamiliar with the ITE room could get the access to facilities — is critical to achieving data center switches confused and accidentally initiate the EPO. availability goals. Methods of identifying people fall Inadvertent activation is one of the major causes of un- into three general categories of increasing reliability — planned ITE system shutdowns. and increasing equipment cost. To help prevent this, it is recommended that the dis- What you have is something you wear or carry — a connect means should require two-step operation such key, a card, or a small object (a token) that can be worn as a hinged transparent cover or have a break-the-glass- or attached to a key ring. It is the least reliable form of to-operate design. When a pushbutton is used, the but- identification, since there is no guarantee it is being ton should be PUSH to operate with some type of sig- used by the correct person — it can be shared, stolen, naling to indicate operation. or lost and found. Unauthorized personnel should only activate the dis- What you know is a password, code, or procedure for connect means in the event of a life-threatening situa- something such as opening a coded lock, verification at tion occurring inside the ITE room. Fire fighters or a card reader, or keyboard access to a computer. What ITE management personnel are usually the only autho- you know is more reliable than what you have, but rized users. In certain applications the requirement for passwords and codes can still be shared, and if written disconnect means might be waived or modified; these down they carry the risk of discovery. applications are covered in NEC Article 685, Integrated Who you are refers to identification by recognition of Electrical Systems. unique physical characteristics — this is the natural way people identify one another with nearly total cer- tainty. When accomplished (or attempted) by techno- logical means, it’s called biometrics. Biometric devices are generally very reliable, if recognition is achieved — that is, if the device thinks it recognizes you, then it almost certainly is you. Contractors are typically granted temporary access on a day-to-day basis. This may be a basic sign in/sign out procedure, use of temporary access badges, a secu- rity escort, or a combination of these methods. Regard- less of the method used, it is important that contractors strictly adhere to the site security policies for entry and exit. ITE centers may also have special procedures for emergency exit. Some sites may have, what is generally referred to as an EPO (Emergency Power Off) device. The NEC refers to this as a “disconnecting means” and it is required to be “readily accessible at principal exit doors” for any area designated as an Information Tech- nology Equipment room. The disconnect device must shut down power going to all electronic equipment in the room. Similarly, all air conditioning units must be shut down and fire/smoke dampers must be activated. It is allowable and very common to shut down both the power and the HVAC by using a single device. RE10830–5 © electrical training ALLIANCE 2017 6 Reference Information Technology Sites and Critical Loads WHY IS A RAISED FLOOR USED? distributed upwards to the IT equipment via cutouts Medium and large ITE centers may be installed on or perforated tiles. raised floors. A typical raised floor is made up of Power Distribution Units (PDUs) are often used in 2 × 2 ft. cast aluminum tiles with vinyl inserts precisely ITE rooms and sometimes distribute power under the mounted on each corner on a supporting pedestal. The raised floor. Intersystem connecting cables may also be tiles closely mate together to form supportive flooring distributed underneath the raised floor. The amount of that can be used in a small room, or cover many thou- cabling must be carefully managed so that it does not sands of square feet of ITE space. build up to the extent that it interferes with air distri- bution paths. Water-sensing alarms, transient surge suppression plates, and smoke detector networks may also be in- stalled under a raised floor. Some sites may have clean agent fire extinguishing systems below the floor. Some sites use a “signal reference structure” (SRS) below the raised floor. An SRS is intended to help re- duce noise-induced voltages to levels that minimize in- terference with IT equipment operation. Some high speed IT equipment is sensitive to minute potential differences between equipment enclosures, which can also affect signal cable shields. Raised floors were originally intended to provide the following: 1. A cold air distribution system. 2. Space for data / power cabling and other utilities Data is transferred at high frequencies and the direct (ex. chilled water). cabling between equipment can behave like an an- 3. An integral component of a signal reference tenna, radiating noise that can affect nearby equip- grounding system. ment. Bonding the enclosures of IT equipment to a The under floor area is typically part of a Com- SRS can suppress this radiated noise. The grid also puter Room Air Conditioning (CRAC) system. This minimizes ground potentials between IT equipment system may be comprised of several dedicated cool- enclosures and suppresses electromagnetic interference ing devices strategically sited throughout the ITE (EMI) and electrostatic discharge (ESD). room. It is designed so that hot air is drawn in from The SRS can be formed by connecting a grid of bare overhead and fed through the CRAC units. The conductors under the floor and installation of metal chilled air is then directed underneath the raised foil covering the entire sub-floor. Another method is to floor, building up a slight pressure. The cooling air is bond raised floor pedestals together with bare copper RE10830–6 © electrical training ALLIANCE 2017 7 Reference Information Technology Sites and Critical Loads wire. Although the purpose of this grounding system is primarily for noise suppression, it must be bonded to building ground in order to comply with Article 250 of the NEC. Panel board enclosures, metallic raceways, conduits, alarm panels, telephone equipment and any conduc- tive enclosure should be bonded to the signal reference structure to establish a zero reference potential. The need for and the effectiveness of an SRS in ITE envi- ronments is debatable and they are rarely used today. Modern IT equipment network connections are usu- ally isolated using optical coupling or by a transformer integrated into the IT equipment. Many small and medium sized ITE sites substitute the raised floor environment with overhead power and data cabling routed in suspended trays. The IT equip- ment racks often use integrated cross-flow aisle cool- ing, commonly referred to as a “hot aisle/cool aisle” system. Overhead power and data distribution can allow SUMMARY greater flexibility for site expansion and cable access. ITE installations are the heart of many businesses. Quite Data cabling has gradually been upgraded from bulky often they were not originally designed to provide the multi-conductor cables to thin fiber and copper net- ideal environment for IT equipment. Newer computer work cabling most often run overhead for easy access. room environments have evolved to keep pace with Some of these sites use conductive floor tiles for EMI trends such as server consolidation, technology compac- suppression. tion, and the convergence of voice, video, and data. RE10830–7 © electrical training ALLIANCE 2017 8 Reference Information Technology Sites and Critical Loads This has presented several challenges, including: get- ADDITIONAL REFERENCES ting enough power and power receptacles to each of the 1. “The Cost of Power Disturbances to Industrial & racks to support today’s increasing power densities, ef- Digital Economy Companies” – White Paper fectively cooling all of the IT equipment deployed, or- copyright 2001, Electric Power Research Institute, ganizing the large amount of data and power cables, Inc. providing security, and managing the system. 2. “We Have Met The Downtime Enemy, and He Is In all cases, special attention to the critical physical Us!” – White Paper by Kenneth G. Brill, The Up- infrastructure is required when deploying racks of time Institute. servers, switches and other IT equipment. The most 3. “Industry Standard Tier Classifications Define current designs include UPS, racks, PDUs, air- Site Infrastructure Performance” by W. Pitt Turner conditioning/airflow and environmental manage- IV, P.E. and Kenneth – White Paper, 2001. ment systems that are integrated into one package. RE10830–8 © electrical training ALLIANCE 2017