Data center from zero: facilities, servers, virtualization, storage and the network

What you will learn in this module. By the end of these chapters you will be able to walk into a data center and name every layer you see: the building and power, the racks, the servers and hypervisors, the storage, and the network that joins them. You need no prior data center experience. If you know what an IP address and a switch are, you are ready. Everything else in the CCNP Data Center track (spine-leaf, VXLAN, EVPN, vPC, ACI, SAN, UCS, automation) builds on the picture you form here.

Start with an analogy: a data center is a hotel for computers

Think of a big hotel. Guests (applications) check in and need a room (a server or virtual machine), electricity, air-conditioning, security at the door, a corridor system so they can reach the restaurant and each other (the network), and a luggage room where they keep their bags safely (storage). The hotel management does not care which guest arrives. It cares that every room is powered, cooled, reachable and safe, 24 hours a day.

A data center is exactly that for computers: a building, or a room inside a building, designed to host many servers and give them power, cooling, physical security, network connectivity and storage, with enough redundancy that one failure does not stop the business.

Why companies build data centers at all

Early businesses kept a few servers under a desk or in a cupboard. That breaks down quickly: the room overheats, a single power cut stops sales, anybody can walk in and unplug a server, and there is nowhere to add the fiftieth machine. Putting servers in one engineered place gives four big wins:

• Availability. Dual power feeds, backup generators and redundant network paths mean the applications stay up when a component fails.
• Scale. Racks, power and cooling are planned so that you can add hundreds of servers without redesigning everything.
• Security. Badges, cameras, cages and locked racks protect the hardware; firewalls and segmentation protect the data.
• Efficiency. Sharing cooling, power and staff across thousands of servers is far cheaper than spreading them across offices.

Types of data center you will hear about

Type	Who owns what	Typical example
Enterprise (on-premises)	The company owns the building or room and everything inside	A bank's own DC in its head office campus
Colocation (colo)	A provider owns the building, power and cooling; you rent racks or cages and bring your own servers and switches	A startup renting two racks in a colo facility
Cloud / hyperscale	The provider owns everything; you rent virtual machines and services by the hour	Public cloud regions with hundreds of thousands of servers
Edge	Small sites close to users for low latency	A mini DC at a telecom tower or a factory

Most real companies are hybrid: some workloads in their own DC or a colo, some in the public cloud, connected together. The skills in this track apply to all of them, because cloud providers run the same spine-leaf, VXLAN and EVPN designs inside their own buildings.

North-south and east-west: the two directions of traffic

Draw the data center with the internet at the top. Traffic that enters or leaves the building (a customer opening your website) travels up and down the drawing, so it is called north-south. Traffic that stays inside, between servers (the web server asking the application server, which asks the database), moves sideways and is called east-west.

Modern applications are split into many small services, so a single click from a user can cause dozens of east-west conversations. Today east-west traffic is usually far larger than north-south traffic. This one fact is the reason data center networks moved from the old three-tier design to spine-leaf, which you will study in the network modules.

Availability in numbers: the "nines"

Data centers are measured by how much of the year they stay up. 99.9% ("three nines") allows about 8.8 hours of downtime per year. 99.99% ("four nines") allows about 53 minutes. 99.999% ("five nines") allows about 5 minutes. Each extra nine costs much more money, because every single point of failure must be removed: two power feeds, two switches, two links, two storage paths.

You will see this "always two" rule everywhere in this track: two spines, two uplinks per leaf, vPC pairs, SAN fabric A and fabric B, two fabric interconnects in UCS.

Before a single switch is configured, a data center must answer two boring but vital questions: where does the electricity come from, and where does the heat go? Every watt a server draws becomes heat in the room. If power stops, everything stops. If cooling stops, servers shut themselves down within minutes to protect their chips. A network engineer who understands the facility plans racks correctly, reads alarms calmly and earns the trust of the facilities team.

The power chain, from the grid to the server

Follow the electricity from outside to inside:

1. Utility feeds. The power company delivers medium-voltage electricity. Good sites take two feeds from different substations, so one fault on the grid does not kill both.
2. ATS and generators. An automatic transfer switch (ATS) watches the utility. If it fails, diesel generators start and the ATS moves the load to them, usually within seconds. Sites keep fuel on site for many hours and have refuelling contracts.
3. UPS. An uninterruptible power supply sits between the source and the IT load. Its batteries (or a flywheel) carry the load for the few seconds to minutes until generators take over. An online double-conversion UPS also cleans spikes and dips from the grid.
4. PDUs. Power distribution units split the power into circuits. The big floor PDUs feed rack PDUs, the vertical power strips inside each rack. Metered rack PDUs show the load; switched ones let you power-cycle a single outlet remotely.
5. Dual power supplies. Servers and switches have two power supply units (PSUs). One plugs into the A feed, the other into the B feed.

Redundancy words: N, N+1 and 2N

N is the amount of equipment you need to carry the load. N+1 adds one spare (five UPS modules where four are enough). 2N duplicates the whole system: a complete A side and a complete B side. 2N costs more but survives the loss of an entire path, including maintenance on it.

Power budgets per rack

Every rack gets a power budget in kilowatts (kW). Typical enterprise racks are about 5 to 10 kW, dense virtualisation racks 15 to 30 kW, and GPU racks for AI training can go far beyond 40 kW and often need liquid cooling. The budget limits how many servers you can install, often long before you run out of space.

Cooling: hot aisle and cold aisle

Servers pull cool air in at the front and blow hot air out of the back. So racks are placed in rows facing each other: fronts face fronts across a cold aisle, backs face backs across a hot aisle. Cold air is supplied to the cold aisle (through perforated floor tiles or from overhead), and hot air returns to the cooling units.

Containment puts doors and a roof over one aisle so hot and cold air cannot mix: cold aisle containment or hot aisle containment. Blanking panels close empty rack units so hot air cannot loop back to the front.

The cooling units themselves: a CRAC (computer room air conditioner) has its own refrigerant compressor, like a big split AC. A CRAH (computer room air handler) has fans and a coil fed with chilled water from a central chiller plant. Industry guidance (ASHRAE) recommends server inlet air of about 18 to 27 °C.

PUE: how efficient is the building?

Power Usage Effectiveness = total facility power ÷ IT equipment power. A perfect building would score 1.0 (every watt goes to IT). Cooling, UPS losses and lighting push it higher.

Uptime Institute Tiers

Tier	Meaning	Commonly quoted availability
I	Basic capacity, single path, no redundancy	99.671%
II	Redundant components (N+1), still a single path	99.741%
III	Concurrently maintainable: any component can be taken down for maintenance without stopping IT	99.982%
IV	Fault tolerant: survives any single unplanned failure, fully independent paths (typically 2N)	99.995%

Fire suppression and physical security

Detection comes first: VESDA (very early smoke detection apparatus) sucks air through pipes and detects smoke long before a normal detector. Suppression is often a clean agent gas (such as FM-200 or inert gas blends) that puts out fire without damaging electronics, or pre-action sprinklers whose pipes stay dry until detection confirms a fire. An EPO (emergency power off) button cuts power for safety and is always guarded against accidental presses.

Physical security is layered like an onion: perimeter fence and guards, reception with ID check, mantrap (a two-door airlock that lets one person through at a time), badge plus biometric at the data hall, CCTV everywhere, locked cages and locked racks, and visitor escort logs.

Think of a rack as a bookshelf for computers and the cabling as the roads between the shelves. A data center with messy roads is like a city with no street names: things still move, but every repair takes hours. In this chapter you learn how equipment is mounted, how switches are placed in a row, which cables and optics carry which speeds, and how to label everything so the next engineer can find it at 3 a.m.

Racks and rack units

A standard data center rack is 19 inches wide between its mounting rails. Height is counted in rack units (RU or U): 1U = 1.75 inches = 44.45 mm. A 1U switch is thin; a 2U server is twice as tall. The most common rack is 42U (about 2 m tall), with 45U to 48U racks also popular. Racks for network gear are often 800 mm wide to leave room for cable managers, and 1000 to 1200 mm deep for long servers.

A rack elevation is a drawing that shows what sits in every U of a rack. It is one of the most important documents in a data center.

Where do the switches go? ToR, EoR and MoR

Design	How it works	Good	Watch out
Top of rack (ToR)	One or two switches in every rack; servers connect with short cables	Very short server cables, rack is a self-contained unit, only a few fibre uplinks leave the rack	Many more switches to manage and power
End of row (EoR)	Switches sit in a network rack at the end of each row; servers reach them through structured cabling	Fewer, larger switches	Long cable runs and heavy cable bundles
Middle of row (MoR)	Like EoR, but the network rack is in the middle	Roughly halves the longest cable run	Still lots of horizontal cabling

Modern spine-leaf fabrics almost always use ToR leaf switches, usually a pair per rack for redundancy.

Structured cabling and patch panels

Structured cabling means permanent, tested cable runs between fixed points, ending on patch panels. You never run a long cable straight from a server to a far switch. Instead, short patch cords connect the device to the local panel, the permanent trunk runs to the far panel, and another patch cord finishes the path. The TIA-942 standard names the areas: the MDA (main distribution area, where core gear lives), the HDA (horizontal distribution area) and the EDA (equipment distribution area, the server racks).

Copper versus fibre

Copper twisted pair (RJ45) is cheap and familiar. Cat6A carries 10GBASE-T up to 100 m, but 10GBASE-T uses more power and adds more latency than optical or DAC links, so it is mostly used for management and older servers.

Fibre carries light and comes in two families:

• Multimode (MMF): wider 50 µm core, cheap 850 nm short-reach optics. Grades OM3, OM4 (aqua jacket) and OM5 (lime green). Reach is short: for example 10GBASE-SR about 300 m on OM3 and 400 m on OM4, but 25GBASE-SR and 100GBASE-SR4 only about 70 m on OM3 and 100 m on OM4.
• Single-mode (SMF): thin 9 µm core (OS2, usually a yellow jacket), 1310/1550 nm optics, reach from 500 m to 10 km and beyond. Used between buildings, and increasingly inside large DCs for 100G/400G.

Connectors: LC duplex (two fibres, one transmit and one receive) and MPO/MTP (12 or 16 fibres in one connector) for parallel optics such as 100GBASE-SR4.

Optics, DAC and AOC

Form factor	Typical speed	Lanes
SFP	1G	1 × 1G
SFP+	10G	1 × 10G
SFP28	25G	1 × 25G
QSFP+	40G	4 × 10G
QSFP28	100G	4 × 25G
QSFP-DD	400G	8 × 50G (PAM4)

A transceiver (optic) plugs into a switch port and converts electrical signals to light. A DAC (direct attach copper) is a twinax cable with the modules fixed on both ends: cheapest and lowest power, but passive DACs are only a few metres long, so they suit server-to-ToR links inside one rack. An AOC (active optical cable) is fibre with fixed optics on both ends: longer reach (tens of metres), lighter and thinner than DAC.

Speeds you will meet: 1G and 10G (older servers), 25G (today's common server speed, one lane in the same small SFP form factor), 40G (legacy uplinks), 100G (common leaf-to-spine uplink) and 400G (spines and new fabrics).

Breakout: one fast port into four slower ones

Because a QSFP28 port is really four 25G lanes, you can split one 100G port into 4 × 25G with a breakout DAC/AOC or an SR4 optic with an MPO-to-4×LC cable. Similarly 40G breaks out to 4 × 10G and some 400G ports to 4 × 100G. On a Nexus 9000:

! split port 1/49 into four 25G interfaces
interface breakout module 1 port 49 map 25g-4x
! the new interfaces are Ethernet1/49/1 to Ethernet1/49/4
interface Ethernet1/49/1
  description SRV-DB01-NIC1
  no shutdown

LEAF-01# show interface brief | include Eth1/49
Eth1/49/1     1       eth  access up      none                       25G(D) --
Eth1/49/2     1       eth  access down    Link not connected         auto(D) --
Eth1/49/3     1       eth  access down    Link not connected         auto(D) --
Eth1/49/4     1       eth  access down    Link not connected         auto(D) --

Labelling and documentation

Label both ends of every cable with a unique ID and where it goes, for example R12-U42-P03 to R01-SPINE1-E1/12. Put the same information in the switch description, in the cable schedule (a spreadsheet or DCIM tool) and on the rack elevation. Use colour conventions agreed by the site (for example one colour for A-side, another for B-side, another for management).

A server is just a computer built for the data center: more CPU, more memory, faster network cards, two power supplies, and a tiny second computer inside that lets you manage it remotely. As a network engineer you will spend a lot of time at the point where the server's cable meets your switch port. If you understand what sits on the other side of that cable, most "the network is down" tickets become easy.

Server anatomy

CPU

The processor. Servers often have one or two sockets, each CPU with many cores (dozens is normal today). With simultaneous multithreading each core can run two threads. More cores means more virtual machines per server.

RAM

Memory in DIMM slots, often hundreds of gigabytes to several terabytes per server. Virtualisation hosts usually run out of RAM before CPU.

NIC

Network interface card. Speeds of 1G (management), 10G, 25G and 100G. Often two ports per card, and usually two cards for redundancy. A SmartNIC or DPU has its own processor to offload work such as encryption, overlay tunnels or virtual switching from the main CPU.

HBA

Host bus adapter. For Fibre Channel storage a server uses an FC HBA (16G, 32G or 64G FC). A converged network adapter (CNA), such as the Cisco VIC, carries both Ethernet and storage traffic on one card.

PCIe

The internal bus that connects NICs, HBAs, GPUs and NVMe drives to the CPU. Each generation roughly doubles bandwidth per lane.

Local disks

SATA/SAS drives or NVMe SSDs, often behind a RAID controller, plus small boot drives (for example M.2).

Rack servers and blade servers

	Rack server	Blade server
Form	Standalone 1U or 2U box with its own power, fans and NICs	Thin server that slides into a shared chassis
Shared parts	None	Chassis power, fans and network modules
Cabling	Every server cabled to the ToR	Only the chassis uplinks are cabled
Cisco example	UCS C-Series	UCS B-Series and the modular UCS X-Series

Blades save space and cables, but tie you to one vendor's chassis. Rack servers are simpler and more flexible. In Cisco UCS both kinds are managed through a pair of fabric interconnects, which you will study in the compute module.

Out-of-band management: the computer inside the computer

Every modern server has a BMC (baseboard management controller), a small independent computer with its own network port. It works even when the server is powered off or the operating system has crashed. Vendor names:

• Cisco CIMC (Cisco Integrated Management Controller)
• HPE iLO (Integrated Lights-Out)
• Dell iDRAC (integrated Dell Remote Access Controller)

Through the BMC you can power the server on and off, open a remote KVM console (see the screen, use the keyboard), mount an ISO as virtual media, read temperatures and fan speeds, and update firmware. It supports standard APIs such as IPMI and Redfish. The BMC port connects to the separate out-of-band (OOB) management network, never to the production fabric.

BIOS/UEFI and PXE boot

When a server powers on, its firmware (UEFI, the modern replacement for BIOS) tests the hardware and chooses a boot device: local disk, SAN LUN, virtual media or the network. PXE (Preboot Execution Environment) boots from the network, which is how large fleets are installed automatically:

1. The NIC sends a DHCP request.
2. The DHCP server returns an IP address plus the boot server and boot file name.
3. The NIC downloads a small boot loader (over TFTP, or from a web server on newer UEFI systems).
4. The loader pulls the OS installer, which installs the server with no one touching it.

For the network team, PXE means: the DHCP server must be reachable (a DHCP relay on the gateway if it is in another subnet) and the build VLAN must be untagged (native) on the port, because PXE firmware usually does not tag.

Connecting one server to two switches

Each server connects to two leaf switches so that a switch, cable or NIC failure does not isolate it. The operating system combines the NICs, which is called NIC teaming (Windows, VMware) or bonding (Linux). Two basic styles:

• Active/standby (Linux bond mode 1, active-backup): one link forwards, the other waits. No special switch configuration.
• LACP (802.3ad, Linux bond mode 4): both links forward as one logical link. The switch side must be a port-channel, and because the two links land on two different switches, the switches must act as one, using vPC on Nexus. You will build this in the vPC module.

A typical leaf port facing a virtualisation host is a trunk carrying the host's VLANs:

! NX-OS port toward ESXi host NIC1
interface Ethernet1/10
  description ESX01-VMNIC0
  switchport mode trunk
  switchport trunk allowed vlan 10,20,30
  spanning-tree port type edge trunk
  mtu 9216
  no shutdown

LEAF-01# show interface status | include Eth1/10
Eth1/10       ESX01-VMNIC0       connected trunk     full    25G     SFP-H25GB-CU3M

Imagine a large house split into flats. One building, one roof, one electricity connection, but many families, each with its own locked door. Virtualization does the same with a server: one physical machine is split into many virtual machines (VMs), each believing it has its own CPU, memory, disk and network card. Almost every server you connect in a modern data center runs a hypervisor, so the "device" at the end of your switch port is often a whole small network of VMs.

Hypervisors: type 1 and type 2

A hypervisor is the software that creates and runs VMs and shares the real hardware between them.

Type	Runs on	Examples	Used for
Type 1 (bare metal)	Directly on the server hardware	VMware ESXi, KVM (built into the Linux kernel), Microsoft Hyper-V	Production data centers
Type 2 (hosted)	As an application on a normal OS	VirtualBox, VMware Workstation	Labs and laptops

Each VM gets vCPUs, virtual RAM, a virtual disk (a file such as a VMDK or qcow2 on local or shared storage) and one or more vNICs, each with its own MAC address. A management platform such as vCenter controls many ESXi hosts as a cluster.

Virtual switches and port groups

VMs inside one host need a switch to talk to each other and to the outside. That switch is software: the virtual switch. Its "uplinks" are the physical NICs of the host (in VMware called vmnics), which plug into your leaf switches.

• vSphere Standard Switch (vSS): configured separately on every ESXi host.
• vSphere Distributed Switch (vDS): configured once in vCenter and pushed to all hosts, so every host is consistent.
• On KVM: the Linux bridge or Open vSwitch; on Hyper-V: the Hyper-V Virtual Switch.

A port group is a group of virtual ports with the same settings, most importantly the VLAN ID. A VM in port group "WEB-VLAN10" is placed in VLAN 10. The virtual switch adds the 802.1Q tag on the way out of the uplink. This is called virtual switch tagging, and it means the physical leaf port must be a trunk that allows every VLAN the host's port groups use.

Live migration and why it shapes the network

Live migration (VMware vMotion, Hyper-V Live Migration, KVM live migration) moves a running VM from one host to another with no reboot. The hypervisor copies the VM's memory to the target host over a dedicated migration network, repeats the copy for pages that changed, pauses the VM for a fraction of a second, and resumes it on the new host.

The VM keeps its IP address and MAC address. Its users should not notice. That creates two network requirements:

1. Layer-2 mobility. The VM's VLAN (and its default gateway) must exist on the new host's leaf ports too. Traditionally that meant stretching VLANs across many racks; today VXLAN overlays give the same Layer-2 reach over a routed fabric.
2. MAC moves. After the move, the physical switches still think the MAC is behind the old port. The hypervisor sends a notification frame (ESXi sends a RARP by default) from the new host so switches update their MAC tables. You will see the MAC "move" from one port or leaf to another.

LEAF-02# show mac address-table address 0050.56ab.1234
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   10     0050.56ab.1234   dynamic  0         F      F    Eth1/12

Before the vMotion this MAC was learned on another leaf; now it appears on LEAF-02 port Eth1/12. One move is normal. The same MAC bouncing between two ports many times a second is MAC flapping, a sign of a loop or a teaming misconfiguration.

Containers and Kubernetes (concept level)

A container packages an application with its libraries but shares the host's operating system kernel. It starts in seconds and uses far less memory than a VM. Kubernetes runs containers across many servers (nodes): it groups containers into pods, gives each pod its own IP address, restarts failed pods and scales them up and down. Networking is provided by a CNI plugin; some build an overlay between nodes, others peer with the leaf switches using BGP to advertise pod subnets.

What this means for the network team

• Trunk the right VLANs to every host in a cluster, identically on both leaves.
• Support larger MTU (jumbo frames, often 9000 bytes on hosts) for migration, storage and overlay traffic, end to end.
• Expect many more MAC and IP addresses per port, and constant change.
• Work closely with the virtualization and platform teams: a port group VLAN change on their side is a network change.

Go back to the hotel analogy. Guests do not keep their suitcases in their rooms forever; the hotel has a secure luggage room, and staff bring bags to whichever room the guest moves to. In a data center, storage is that luggage room. Servers and VMs come and go, move between hosts and fail, but the data must stay safe and reachable from anywhere. That is why most important data lives on shared storage systems, not inside one server.

Why storage is separate from the server

• Mobility. vMotion can move a VM in seconds only because its disk sits on shared storage that both hosts can reach.
• Protection. A storage array has redundant controllers, many disks, snapshots and replication, far more than one server can offer.
• Efficiency. Capacity is pooled and given to whoever needs it, instead of sitting unused inside hundreds of servers.

The media: HDD, SSD and NVMe

Media	How it works	Typical random IOPS per drive	Used for
HDD	Spinning platters and a moving head (7,200 to 15,000 RPM)	About 100 to 200	Cheap bulk capacity, backups, archives
SATA/SAS SSD	Flash memory behind an older disk interface	Tens of thousands	General performance storage
NVMe SSD	Flash connected directly over PCIe with a protocol built for flash	Hundreds of thousands or more	Databases, all-flash arrays, low latency

Storage is judged by four numbers: capacity (TB), IOPS (input/output operations per second), throughput (MB/s or GB/s) and latency (how long one operation takes, in milliseconds or microseconds).

RAID: surviving a disk failure

RAID (redundant array of independent disks) combines several disks so data survives a disk failure, and often runs faster.

Level	Method	Usable capacity	Survives	Minimum disks
RAID 0	Striping, no protection	N × disk	Nothing	2
RAID 1	Mirroring	50%	1 disk of the pair	2
RAID 5	Striping with single parity	(N − 1) × disk	1 disk	3
RAID 6	Striping with double parity	(N − 2) × disk	Any 2 disks	4
RAID 10	Mirrored pairs, then striped	50%	1 disk per mirror pair	4

LUNs: slicing the storage

A storage array pools hundreds of disks and then carves out volumes. A LUN (logical unit number) is one such volume presented to a server. The server sees it as a local disk, formats it and uses it. The array uses LUN masking so only the right servers can see each LUN, and in a Fibre Channel SAN the switches add zoning so only the right devices can talk at all.

Block, file and object

Block storage

The server gets raw blocks (a LUN) and puts its own file system on it. Fast and low-level; used for databases and VM datastores. Protocols: Fibre Channel, iSCSI, NVMe over Fabrics.

File storage

The storage system owns the file system and shares folders over the network. Many clients can open the same files. Protocols: NFS (common with Linux and VMware) and SMB (Windows shares).

Object storage

Data is stored as objects with an ID and rich metadata in a flat namespace, accessed through web-style REST APIs (S3-compatible APIs are the common standard). Scales to billions of objects; used for backups, media, logs and data lakes.

DAS, NAS and SAN

• DAS (direct-attached storage): disks inside or cabled directly to one server. Simple and fast, but not shared.
• NAS (network-attached storage): a file server (filer) that shares file systems over normal Ethernet with NFS or SMB.
• SAN (storage area network): a dedicated network that gives servers block access to LUNs on shared arrays, using Fibre Channel or iSCSI. Built as two separate fabrics, A and B, and servers use multipathing software to use both.

Why storage needs a lossless or very reliable network

Storage traffic carries disk reads and writes. If a frame is lost, the host must wait for a timeout and retry, and applications freeze. Fibre Channel is designed to be lossless: a sender only transmits when the receiver has buffer space (buffer-to-buffer credits). iSCSI and NFS run over TCP, which recovers lost packets, but drops still cause latency spikes. FCoE carries Fibre Channel over Ethernet and therefore needs lossless Ethernet features (DCB: Priority Flow Control and Enhanced Transmission Selection).

Preview of the storage module: FC with WWNs, VSANs and zoning on Cisco MDS; iSCSI (TCP port 3260) with IQN names; NFS (TCP port 2049).

Picture a city's road system. Houses (servers) need streets to reach each other, a highway to the airport (the internet), a special armoured route to the bank vault (storage), and a separate service road that ambulances can use even when the main roads are jammed (the management network). The data center network is that road system. In this chapter you will see what it must deliver, why the old design struggled, and the shape of the modern answer that the rest of this track teaches in depth.

What the DC network must do

• Connect servers to each other (east-west) with high bandwidth and low, predictable latency.
• Connect servers to storage reliably (iSCSI, NFS over Ethernet, or a separate FC SAN).
• Connect the DC to the outside world (north-south): internet, WAN to branches, other data centers and the public cloud.
• Survive failures of any link or switch without users noticing.
• Scale from a few racks to hundreds without redesign.
• Separate applications, environments and customers from each other, and allow workloads to move.

The old way: three-tier with spanning tree

Campus networks, and older data centers, use three layers: access (servers plug in), aggregation or distribution (gateways, services) and core (fast transport between aggregation blocks). Each access switch has two uplinks for redundancy. But Layer-2 networks cannot tolerate loops, so Spanning Tree Protocol (STP) blocks one of the two uplinks.

ACCESS-01# show spanning-tree vlan 10
Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/49          Root FWD 2         128.49   P2p
Eth1/50          Altn BLK 2         128.50   P2p

Problems with this design for a modern DC:

• Half the uplink bandwidth sits idle because of blocked ports.
• Big Layer-2 domains stretched for VM mobility mean a single loop or broadcast storm can take down many racks.
• Unequal paths: east-west traffic between two racks may climb to the core and back, adding hops and latency.
• Scaling up means buying bigger aggregation boxes, not simply adding more.

Technologies such as vPC (which you will study) let both uplinks forward by making two switches look like one, and they are still widely used at the edge of the fabric. But the core design changed.

The modern idea: spine-leaf (preview)

In a spine-leaf (Clos) fabric there are just two layers. Leaf switches connect servers, storage and services. Spine switches connect only to leaves. Rules: every leaf connects to every spine; leaves never connect to other leaves for transport; spines never connect to each other.

The links between leaves and spines are usually routed (Layer 3), so there is no spanning tree to block them. The routing protocol installs several equal-cost paths, and ECMP (equal-cost multipath) spreads flows across all of them:

LEAF-01# show ip route 10.0.0.13
10.0.0.13/32, ubest/mbest: 2/0
    *via 10.1.1.0, Eth1/49, [110/41], 2d03h, ospf-UNDERLAY, intra
    *via 10.1.2.0, Eth1/50, [110/41], 2d03h, ospf-UNDERLAY, intra

Benefits: every server is the same distance from every other (predictable latency), all links carry traffic, and you scale by adding: more spines for more bandwidth, more leaves for more ports.

Border leaf, services and the internet edge

Traffic leaves the fabric through a border leaf, which peers (usually with BGP or OSPF) with WAN routers, the internet edge and other data centers. Firewalls and load balancers often attach to a border or service leaf. A load balancer publishes a virtual IP (VIP) and spreads client connections across a pool of servers, checking their health. At the internet edge, edge routers talk BGP to one or more ISPs, and firewalls protect the DMZ where public-facing servers live.

The out-of-band management network

Every switch has a mgmt0 port, every server a BMC port, every PDU a network port. These connect to a separate, simple management network with its own switches, plus console servers wired to each device's serial console. If a bad change breaks the production fabric, you can still log in through this path and fix it. On Nexus, mgmt0 lives in its own management VRF, separate from the data-plane routing table.

Overlays and multitenancy (preview)

Routed spine-leaf solved bandwidth, but VMs still need Layer-2 mobility, and different customers or environments need isolation. The answer is an overlay: VXLAN wraps Layer-2 frames inside UDP (port 4789) packets that cross the routed fabric between VTEPs on the leaves. VXLAN uses a 24-bit VNI, about 16 million segments, compared with only 4,094 usable VLANs. BGP EVPN acts as the control plane that tells every leaf where each MAC and IP lives. Cisco ACI builds on the same VXLAN fabric but is driven by policy from the APIC controller. Multitenancy uses VRFs and VNIs so tenant A and tenant B can even use the same IP addresses without seeing each other.

A hospital is not only surgeons. It runs on nurses watching monitors, shift handovers, patient files, checklists before every operation and a clear process when something goes wrong. A data center is the same. Building the fabric is maybe 10% of the job; operating it safely for years is the other 90%. This chapter shows you how that work is organised, because it is exactly what your first data center job will look like.

Who does what: the NOC and its levels

• NOC (network operations center): the team that watches the infrastructure 24x7, usually in shifts, with dashboards on the wall.
• L1: watches alerts, acknowledges them, runs first checks from runbooks, opens and routes tickets, coordinates remote hands (colocation staff who can reseat a cable or swap an optic).
• L2: deeper troubleshooting, executes approved changes, owns most incidents.
• L3 / engineering: design, complex problems, root cause analysis, new builds and automation.
• Alongside: facilities (power and cooling), server and virtualization, storage, security and application teams.

Monitoring: how you know something is wrong

SNMP

Simple Network Management Protocol. The monitoring server polls devices on UDP 161 for counters (interface traffic, errors, CPU) identified by OIDs defined in MIBs. Devices can also send traps on UDP 162 when something happens. Use SNMPv3, which adds authentication and encryption.

Syslog

Devices send event messages (link down, neighbor lost, fan failure) to a central syslog server, traditionally on UDP 514. Each message has a severity from 0 (emergencies) to 7 (debugging): 0 emergencies, 1 alerts, 2 critical, 3 errors, 4 warnings, 5 notifications, 6 informational, 7 debugging.

Streaming telemetry

Instead of being polled every few minutes, the switch pushes structured data (based on YANG models) every few seconds, often over gRPC/gNMI. Much finer detail, far less load than heavy SNMP polling.

Flow data

NetFlow/IPFIX records who talked to whom, how much and on which ports, useful for capacity planning and security.

NTP

Every device must have the same accurate time, or you cannot correlate logs from different switches during an incident.

! Typical NX-OS monitoring baseline, all through the management VRF
ntp server 10.99.0.10 use-vrf management
logging server 10.99.0.20 5 use-vrf management
snmp-server user nocmon network-operator auth sha Str0ngAuthPass priv aes-128 Str0ngPrivPass

2026 Sep 29 02:14:07 LEAF-03 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/49 is down (Link failure)
2026 Sep 29 02:14:07 LEAF-03 %OSPF-5-ADJCHANGE: ospf-UNDERLAY [1234] Nbr 10.0.0.1 on Ethernet1/49 went DOWN

Read a syslog line from left to right: timestamp, device, facility, severity number, mnemonic, then the human message. Here, severity 5 events tell you an uplink failed and the OSPF neighbor on it went down. Because the leaf has a second uplink, traffic continues; the ticket is still urgent because redundancy is gone.

Tickets: incident, problem and change

Type	Meaning	Goal	Example
Incident	Unplanned interruption or degradation of a service	Restore service as fast as possible	Rack 12 servers lost connectivity
Problem	The underlying cause of one or more incidents	Find and remove the root cause	Why do optics from one batch keep failing?
Change	A planned modification to production	Make it safely, approved and reversible	Add VLAN 30 to the ESX cluster trunks

Incidents get a priority (P1 critical, full outage, to P4 low). Changes are standard (pre-approved, low risk, repeatable), normal (reviewed and approved, often by a change advisory board, CAB) or emergency (urgent fix, approved quickly, reviewed afterwards).

Change management, MOPs and rollback

Risky changes happen in a maintenance window, a pre-agreed low-traffic time such as Sunday 01:00 to 04:00. Every change has a MOP (method of procedure) that someone else could follow:

1. Pre-checks: capture the current state (interfaces up, neighbors, routes, MAC counts).
2. Implementation steps: exact commands, device by device.
3. Verification: the same checks again, plus an application test with the owner.
4. Rollback plan: exact steps to undo, and the time by which you must decide to roll back.
5. Communication: who to inform at start, end, and if something goes wrong.

NX-OS makes rollback easy with checkpoints:

! before the change
checkpoint PRE-CHG-4521
! ... make the change, verify ...
! if verification fails
rollback running-config checkpoint PRE-CHG-4521

Documentation: the memory of the data center

• IPAM (IP address management): every subnet, VLAN, VRF and reserved address.
• Rack elevations and cable plans (which port goes where), ideally in a DCIM tool.
• Logical and physical diagrams, kept current.
• Runbooks for common alerts, and automatic configuration backups after every change.

A day in the life of a DC network engineer

09:00 shift handover: two open incidents, one change tonight. 09:30 review overnight alerts: a flapping optic on LEAF-07, so raise a remote-hands ticket to replace it. 11:00 peer review a colleague's MOP. 14:00 new rack request from the server team: allocate ports, VLANs and IPs in IPAM, update the cable plan. 16:00 problem review on last week's outage. 01:00 maintenance window: checkpoint, change, verify, close the ticket with evidence.

Think of an apartment complex. There is a gate with a guard (perimeter), each tower has its own entrance (segmentation), each flat has its own lock (the workload), the society office keeps a list of who may enter which flat (access control), and there is a spare set of keys and important papers kept in a bank locker across town (backup and disaster recovery). Data center security follows the same idea: defence in depth, many independent layers, so one mistake is not a disaster.

Layer 1: physical security (recap)

From the facility chapter: fences, guards, mantraps, badge plus biometric, CCTV, locked cages and racks, escort logs. Remember that anyone with physical access to a switch's console port or a server's disks can bypass most software controls.

Layer 2: network segmentation

Segmentation divides the network so that systems only reach what they need. Tools, from simple to advanced:

• VLANs and subnets separate broadcast domains and IP ranges (web, app, database, management).
• VRFs (virtual routing and forwarding) create completely separate routing tables inside the same switch. Two VRFs cannot reach each other unless you deliberately leak routes or send traffic through a firewall. Different VRFs can even reuse the same IP ranges.
• Tenants: in a multitenant DC, each customer, business unit or environment (production, test) gets its own VRF, VLANs/VNIs and policies. In Cisco ACI, "tenant" is a formal object that contains VRFs, bridge domains and EPGs.

! NX-OS: put the PROD web gateway into its own VRF
vrf context PROD
interface Vlan10
  vrf member PROD
  ip address 10.10.10.1/24
  no shutdown

LEAF-01# show vrf
VRF-Name                           VRF-ID State   Reason
default                                 1 Up      --
management                              2 Up      --
PROD                                    3 Up      --

East-west firewalling and micro-segmentation

Traditional firewalls sit at the edge and inspect north-south traffic. But attackers who get into one server usually move sideways (east-west) to the next. So modern DCs also control traffic between internal tiers:

• East-west firewalling: traffic between zones or VRFs (for example web to database) is routed through a firewall, often attached to a service leaf.
• Micro-segmentation: policy applied per workload or group, not per subnet. Two VMs in the same subnet can still be blocked from talking. Examples: ACI EPGs and contracts, the VMware NSX distributed firewall, Cisco Secure Workload.

Zero trust is the idea behind this: never trust a connection just because it is "inside"; verify identity, allow only what is needed (least privilege), and log everything.

Protecting the management plane

Whoever controls your switches controls your data center. So:

• AAA (authentication, authorization, accounting) with a central server. TACACS+ (TCP 49) encrypts the whole packet and separates authorization so you can control and log every command; it is the usual choice for device administration. RADIUS (UDP 1812/1813) encrypts only the password and is common for network access.
• RBAC (role-based access control): operators get read-only roles (on NX-OS, network-operator), engineers get network-admin only when needed.
• SSH only (no Telnet), access only from the management VRF and jump hosts, ACLs on management access, and a tested local break-glass account in case the AAA servers are unreachable.
• Control-plane policing (CoPP) protects the switch CPU from floods.

! NX-OS: TACACS+ for logins and accounting, through the management VRF
feature tacacs+
tacacs-server host 10.99.0.30 key Str0ngTacKey
aaa group server tacacs+ NOC-TACACS
  server 10.99.0.30
  use-vrf management
aaa authentication login default group NOC-TACACS
aaa accounting default group NOC-TACACS

Backup and disaster recovery

Security also means being able to recover: from ransomware, a deleted database or the loss of a whole site. Two numbers drive every design:

• RPO (recovery point objective): how much data, measured in time, you can afford to lose. An RPO of 15 minutes means copies at least every 15 minutes.
• RTO (recovery time objective): how long the service may be down before it must be running again.

Good backup practice follows the 3-2-1 rule: three copies, on two different media, one off-site, and today at least one immutable or offline copy that ransomware cannot encrypt.

For site loss, companies use a second data center:

• Active/standby: the DR site waits, with data replicated to it. Simpler, but failover takes time and the standby capacity sits idle.
• Active/active: both sites serve users at the same time. Better use of money and faster recovery, but applications and networks are more complex.
• Synchronous replication gives an RPO of near zero, but latency limits it to short distances (typically up to about 100 km). Asynchronous replication works over any distance with an RPO of seconds to minutes.

DCI (data center interconnect) links the sites: dark fibre or DWDM, and on top of it technologies such as VXLAN EVPN Multi-Site or ACI Multi-Site. The golden rule: stretch only what you must, because a stretched Layer-2 domain also stretches failures.

You have now walked through the whole building: power and cooling, racks and cables, servers, hypervisors, storage, the network, operations and security. Before you open the network modules, this chapter gives you a map. First, which Cisco products live at each layer. Second, how the 350-601 DCCOR exam is organised and which module of this track covers each part. Finally, a checklist, a glossary and the facts that are tested most often.

The Cisco data center portfolio, layer by layer

Product	What it is	Where you study it
Nexus 9000 + NX-OS	The main data center switch family today, used as leaf and spine. Runs standalone NX-OS (you configure VXLAN EVPN, vPC and routing yourself) or ACI mode	Network modules
Nexus 7000/7700	Older modular core/aggregation chassis with features such as VDCs and OTV; being replaced by Nexus 9000 in new designs	Awareness only
MDS 9000	Fibre Channel SAN switches and directors: VSANs, zoning, FC up to 64G	Storage module
UCS	Cisco servers (B-Series blades, C-Series rack, X-Series modular) connected through a pair of fabric interconnects and managed with service profiles	Compute module
Intersight	Cloud-operated (SaaS, or on-premises appliance) management and automation for UCS and more	Compute module
ACI + APIC	Application Centric Infrastructure: a policy-driven VXLAN fabric of Nexus 9000 switches controlled by an APIC cluster (usually three controllers). Tenants, EPGs and contracts	ACI module
Nexus Dashboard / NDFC	Operations platform. NDFC (Nexus Dashboard Fabric Controller, formerly DCNM) builds and manages NX-OS fabrics; Insights gives analytics; Orchestrator manages multi-site	Automation module

How DCCOR maps to this track

CCNP Data Center requires the core exam 350-601 DCCOR plus one concentration exam of your choice. Passing DCCOR alone also earns a Specialist certification. The blueprint has five domains (weights as published for v1.1; always check the current blueprint before you book):

DCCOR domain	Weight	Modules in this track
1.0 Network	25%	DC networking refresher; spine-leaf and NX-OS first steps; underlay; VXLAN; BGP EVPN; vPC; ACI
2.0 Compute	25%	Cisco UCS, service profiles and Intersight
3.0 Storage Network	20%	Fibre Channel, zoning, VSANs, FCoE, iSCSI and NFS
4.0 Automation	15%	NX-API, Ansible, POAP, telemetry, Nexus Dashboard
5.0 Security	15%	DC security (AAA, RBAC, CoPP) plus ACI contracts, building on chapter 9 here

Checklist: you should now be able to explain

• The layers of a data center and the "always two, truly independent" rule.
• The power path (utility, ATS, generator, UPS, PDU, dual PSU), A/B feeds, N+1 vs 2N, and a rack power budget.
• Hot/cold aisles and containment, CRAC vs CRAH, how to calculate PUE, Tier I to IV.
• Rack units, ToR vs EoR, multimode vs single-mode, SFP28/QSFP28/QSFP-DD, DAC vs AOC, 100G to 4 × 25G breakout.
• Server parts, BMC (CIMC/iLO/iDRAC), PXE boot, teaming vs LACP and why LACP across two switches needs vPC.
• Type 1 vs type 2 hypervisors, vSS vs vDS, port groups and trunks, why vMotion needs Layer-2 mobility.
• RAID capacity math, LUNs, block vs file vs object, DAS vs NAS vs SAN, why storage needs a lossless or reliable network.
• Why three-tier with STP gave way to spine-leaf, oversubscription, border leaf, OOB management, overlays.
• Incident vs problem vs change, MOPs with rollback, SNMP/syslog/telemetry.
• VRFs and tenants, micro-segmentation, TACACS+ vs RADIUS, RPO vs RTO.

Mini glossary

A/B power

Two independent power paths to every device.

AOC

Active optical cable: fibre with fixed optics on both ends.

APIC

Controller cluster that manages a Cisco ACI fabric.

BMC

Out-of-band server controller (CIMC, iLO, iDRAC).

Border leaf

Leaf that connects the fabric to the outside.

CRAH / CRAC

Cooling unit using chilled water / using its own refrigerant.

DAC

Direct attach copper cable for short links.

DCI

Data center interconnect between sites.

ECMP

Equal-cost multipath: traffic spread over several equal routes.

EoR / ToR

End-of-row / top-of-rack switch placement.

Hypervisor

Software that runs virtual machines.

LUN

Block volume presented from storage to a host.

MOP

Method of procedure for a change, including rollback.

NAS / SAN

File-level shared storage / block-level storage network.

OOB

Out-of-band management network, separate from production.

Oversubscription

Ratio of downstream to upstream capacity on a leaf.

PDU

Power distribution unit.

PUE

Total facility power ÷ IT power.

PXE

Network boot using DHCP and TFTP.

RAID

Combining disks for protection and speed.

RPO / RTO

Acceptable data loss / acceptable downtime.

Spine-leaf

Two-layer fabric; every leaf connects to every spine.

UPS

Battery-backed supply that bridges power gaps.

vMotion

VMware live migration of a running VM.

VRF

Separate routing table inside a device.

VXLAN / VNI

Layer-2 overlay over UDP 4789 / its 24-bit segment ID.

Most-tested facts

• 1U = 1.75 inches; racks are 19 inches wide.
• PUE = total facility ÷ IT; Tier III = concurrently maintainable, Tier IV = fault tolerant.
• QSFP28 = 4 × 25G = 100G; QSFP-DD = 400G; multimode is short reach, single-mode long reach.
• RAID 5 = N − 1, RAID 6 = N − 2, RAID 10 = 50%.
• Spine-leaf: every leaf to every spine, no spine-to-spine links, ECMP.
• VXLAN: UDP 4789, 24-bit VNI (about 16 million); VLAN: 12-bit (4,094 usable).
• SNMP 161/162, syslog 514, TACACS+ TCP 49, RADIUS UDP 1812/1813, iSCSI TCP 3260, NFS TCP 2049.