What Network Software Cannot Detect About Physical Cabling, and Why LLDP Is Not Enough

• LLDP is one hop between two powered, configured switch ports. It never sees the patch panels and cross connects in between.
• Cable validation tools compare LLDP and optical readings with an intended topology, so they need power, agents and a correct plan to say anything at all.
• NVIDIA's own documentation says an operationally up link does not prove it reaches the intended host or port.
• Transceiver telemetry and error counters show that loss or corruption exists on a link, not where along the link the cause is.
• In a Microsoft study, only 8.2% of links with packet corruption corrupted in both directions, so a degraded link can look healthy from one side.
• Network redundancy was only about 40% effective at reducing the median impact of failure in a published study of Microsoft data centers.
• Ports, dust caps, labels and bends can be checked before power on, which is when fixing them costs the least.

Short answer: LLDP tells each switch port who is directly on the other end of its link: the neighbor device's identity and the port it is using. It only works between two powered devices that both run it, and it only reaches one hop.

The Link Layer Discovery Protocol, defined in IEEE 802.1AB, lets a device advertise its identity and capabilities to a physically adjacent peer. Every LLDP message carries the sender's device identity, the sending port and a time to live, and each switch keeps a table of what its neighbors last said.

The key word is adjacent. LLDP messages are not forwarded by standard switches, so each one travels exactly one link and stops. Your leaf switch learns that port 12 hears spine switch 3, port 40, and nothing about the path the light took to get there.

LLDP also needs both ends to be alive and talking. The local port has to be up and configured to send and receive, the far device has to be powered, booted and running LLDP, and the link has to pass light well enough to carry the messages.

What validation tools do with it

On its own, an LLDP table is just a list of neighbors. Validation tools turn it into a test by comparing each neighbor with the cable plan: if leaf 1, port 12 should face spine 3, port 40, and LLDP says it does, the link passes.

That catches the most common commissioning mistake on a live network, a cable plugged into the wrong port or the wrong switch. It is fast, it scales to thousands of links, and it runs again every time a link changes state.

LLDP answers one question well: which powered port is at the far end of this live link.

Short answer: They compare what the switches report, mostly LLDP neighbors plus optical readings, with an intended topology from a cable plan or source of truth, and mark each link pass, fail or unknown. They need power, agents or management access, and a correct plan.

The tools share one design. An intended topology lives in a file or a database, the switches report what they actually see, and the tool compares the two link by link.

• Cumulus Linux PTM: reads a cabling plan written as a topology file and compares it with LLDP on every link transition, reporting pass, fail or N/A when the interface hears no LLDP at all.
• NVIDIA UFM Cable Validation Tool: agents are installed on all switches and servers in the cluster. It flags wrong neighbor, wrong port, missing cable, extra cable, flapping links, out of range transceiver readings and receive and transmit power mismatches greater than 3 dB.
• NVIDIA Config Manager: compares what each switch reports over LLDP with the cabling recorded in the DCIM, falling back to forwarding and ARP tables where LLDP is missing, and writes connected, disconnected or invalid back to each cable record.

The plan is only as good as its source

Every one of these tools trusts its intended state. NetBox, a widely used source of truth, says plainly that it intends to represent the desired state of a network versus its operational state, and that data should be vetted by a person before it goes in.

That is the right design for a source of truth, and it has a consequence. The record says what should be on the floor, not what is, and the technicians, contractors and commissioning engineers we interviewed told us records go stale within about a month as equipment moves.

A validation tool can only be as right as the plan it validates against, and it can only test links that are already lit.

Short answer: Because patch panels and cross connects are passive. They pass light from one fiber to the next without powering up, sending messages or taking any part in the protocol, so to LLDP they do not exist.

A real data center link is rarely one cord between two switches. It runs from a switch port through a patch cord to a panel, into a trunk cable, across the room to another panel, and through another cord to the far switch. Every panel adds two more connectors and two more chances for a mistake.

The invisible middle

Follow one link end to end and note what each tool can see at each step:

1. 1. Switch port A: LLDP sees it, DOM reports its transmit and receive power, counters log its errors.
2. 2. Transceiver A: DOM reports its temperature and optical power.
3. 3. Patch cord A: no tool sees it. The cutsheet says which cord should be here.
4. 4. Patch panel A: no tool sees it. The cutsheet says which panel port it should land on.
5. 5. Trunk cable: no tool sees it. The cutsheet says which trunk and which fibers.
6. 6. Patch panel B: no tool sees it. The cutsheet says which panel port it should land on.
7. 7. Patch cord B: no tool sees it. The cutsheet says which cord should be here.
8. 8. Transceiver B: DOM reports its temperature and optical power.
9. 9. Switch port B: LLDP sees it, DOM reports its power, counters log its errors.

LLDP sees the two ends, DOM sees power at each end, counters see errors on the link as a whole, and the cutsheet says what should be in between. Only physical inspection sees every step in the middle.

This is why two links can be swapped at a panel and still pass validation if both reach plausible ports, and why a cord in the right switch port but the wrong panel port can sit undetected until someone traces it by hand. The software is working exactly as designed; the middle is simply outside its view.

Short answer: They tell you that something is wrong on a link: power is low, errors are rising, a module is hot. They cannot tell you where along the link the cause is, or which of the connectors, cords and panels in the middle is responsible.

Digital diagnostics

Modern transceivers report five live values: temperature, supply voltage, laser bias current, transmit power and receive power. Each value has warning and alarm thresholds programmed by the module's manufacturer, and the switch raises an event when one is crossed.

Receive power is the one people watch for cabling. If it is lower than expected, light is being lost somewhere between the far transmitter and this receiver. It might be a dirty connector, a tight bend, a poorly seated cord, a damaged trunk or simply a long run, and the number is the same either way.

Error counters

Counters record what the loss does to traffic. CRC errors count frames that arrived corrupted, and FEC counters show how many errors forward error correction had to fix, or could not fix. Both are symptoms measured at the receiving port, not locations.

The Microsoft CorrOpt study shows the limit clearly. Its authors used optical power together with corruption rates to infer likely causes, and noted that transceivers do not report on reflections, so some root causes could not be identified from telemetry every time.

Up does not mean right

A link can come up, pass light and carry traffic while still being connected to the wrong place. NVIDIA's Config Manager documentation states it directly: an operationally up link does not prove that it reaches the intended host or port.

The same page notes that when a neighbor is missing, an unpowered or misconfigured neighbor is the most common cause. In other words, the software's silence often means the far end is dark, not that the cable is wrong, and it cannot tell the two apart.

Telemetry says a link is sick. It cannot say which of its eight or more physical parts is the patient.

Short answer: A gray failure is a fault that leaves a component partly working, so the system's own health checks say it is fine while the applications using it suffer. Microsoft researchers call the core feature differential observability: different parts of the system see different realities.

The term comes from a Microsoft Research paper at HotOS 2017. Its authors argue that major availability breakdowns and performance problems in cloud systems are more often caused by subtle faults like these than by clean, total failures that are easy to detect.

A degraded fiber link is a textbook case. It stays up, LLDP still hears its neighbor, and a connectivity test passes, because a ping only needs a few packets through. Meanwhile a small share of real traffic is corrupted and retransmitted, and a synchronous job waits on every retry.

The CorrOpt data adds two details that make corruption especially gray. Only 8.2% of links with packet corruption corrupted in both directions, so the link can look clean from one end, and the corruption rate was stable over time and did not change with load. A bad link stays bad until someone fixes the physical cause.

A gray link passes the health check and fails the workload, quietly, every day.

Short answer: Any problem that has not yet caused an error, sits in the passive middle, or exists before power on: dust caps, labels, cable type, polarity, tight bends, slightly unseated connectors and shared components. Even problems that do cause errors show up as signals without a location.

Signal versus location

The Microsoft study above ranked the physical root causes of packet corruption in production data centers. The ranges are wide because technicians often took several actions without recording which one worked. Here is what each cause looks like to software, and why that signal cannot say where to look:

• Connector contamination (17 to 57% of cases): lower receive power and rising errors at the receiving port. Every connector along the path is a suspect, and each panel adds two more. See fiber connector contamination.
• Bent or damaged fiber (14 to 48%): lower receive power on one link. The bend could be at either end, behind a manager or inside a tray, and the number reads the same.
• Bad or loose transceiver (6 to 45%): errors or flaps on one link. From the counters alone it can look just like a dirty connector, and only a look or a reseat tells them apart.
• Shared component (10 to 26%): several links degrade at once. Software raises separate alarms on separate links and has no idea they share a panel, a cassette or a tray.

Problems with no signal at all

• Dust caps: a missing cap on an unused port produces no error until a contaminated connector is plugged in later.
• Labels: a wrong, missing or swapped label never affects traffic, but it sends the next technician to the wrong port.
• Cable type: the wrong cord type or length can work on a short link and still breach the design.
• Polarity before power on: a polarity error shows up only when the link is lit; before that, it is a job for test equipment.
• Bends that have not caused errors yet: a cord folded at a panel exit may pass today and fail after the next door slam or temperature swing. Cable management in AI halls covers where these hide.

Software can tell you a link is degraded. It cannot tell you which cap is missing, which label lies or which tray three degraded links share.

Short answer: Often not as independent as the topology says. Two logically separate paths can run through the same panel, cassette or tray, and published research found network redundancy far less effective than designers expect.

Redundancy assumes the two paths fail independently. In the logical design they do. On the floor, both paths may leave the rack through the same cassette, share a trunk to the next row, or lie in the same tray where one cut or one crushing load takes out both.

Networks track this risk with shared risk link groups, which tag links that share a physical component. Those groups are usually entered by hand from drawings, so they are only as accurate as the records behind them, and records drift.

The published evidence is sobering. A study of Microsoft data center networks presented at SIGCOMM 2011 found that network redundancy was only about 40% effective in reducing the median impact of failure, and observed the same configuration error made on both the primary and backup of a connection.

Large operators design around this deliberately. Google's SIGCOMM 2016 paper on high availability says it uses disjoint physical fibers and disjoint power feeds to be resilient to physical plant failures, and found that most failures it studied happened while a management operation was in progress nearby.

Software shows two healthy paths. Only the floor shows whether they are really two.

Short answer: Everything physical: connectors in their planned ports, dust caps, labels, bends, cable types and polarity tests, while the switches are still dark. It is the only window when the network itself cannot help, and the cheapest time to fix anything.

Between the last cord going in and the first switch booting, no software tool can see a thing. LLDP needs both ends powered, DOM needs a module talking to a switch, and counters need traffic. For that window, the only evidence is physical.

It is also the window that matters most for cost. The technicians, contractors and commissioning engineers we interviewed told us about 5% of connections need fixing while a row is built, and that even a good contractor can hand over with 1 to 2% still wrong after its own testing.

Fix a wrong port before power on and it is a two minute move. Find it after turn up and it becomes a troubleshoot, a change window and a retest, and late handover damages typically run $20,000 to $50,000 a day. Polarity is checked with test equipment before power on; port placement, caps, labels and bends can be checked by inspection.

Before power on, the network cannot vouch for its own cabling. Something else has to.

Short answer: We do not replace your validation tools. A FacilityOps robot checks the physical layer they cannot see, before power on and after every change, and confirms seating against your own error counters, without ever needing access to your network.

FacilityOps AI (RoboSkill Inc.) builds an autonomous inspection platform for data centers. Our inspection robots run fully autonomously, and a technician can take manual control at any time.

Your validation tools watch the two ends of every live link. A FacilityOps robot walks the row and checks the middle, the dark ports and everything physical, so the two together cover the whole path. It is the same split we describe in monitoring, inspection and robotic action.

What the robot checks

• Before power on: connector present in the planned port at both ends, including passive panels and cross connects that LLDP cannot see.
• Ports by position: every port is identified by its position in our 3D digital twin of the hall. Labels are read and judged against that position as matched, mismatched, missing, unreadable or misplaced. A label is evidence, never identity.
• Caps and bends: dust caps present on unused ports, and tight bends and kinks flagged from close up images.
• Seating: connectors visibly out of position flagged, then confirmed against your link error counters once the link is lit. We never call seating from a photo alone.
• Go to the bad link: when switches flag a link, the robot goes to that port and photographs the connector, bend, cap and label.
• After every change: change detection every round shows what moved, appeared or disappeared, with before and after images.

One interviewee found connections disturbed by other teams only by matching link down times to who had worked in the area the day before. Before and after records make that a lookup instead of an investigation.

No network access needed

The robot never connects to your switches. When we confirm seating or go to a bad link, we work from an export of counters you produce on your own schedule, and findings arrive in a inspection report your team can check against its own tools.

Built to be trusted

• We check the work. We never do the work. No plugging, unplugging, reseating, cleaning, part swaps or repair, in any phase, and nothing in the signal path.
• People stay in the loop. Anything the robot cannot read clearly is marked unreadable and goes to a person, and a person reviews every finding before it reaches you.
• Everything runs on site. Detection and the 3D model run on the robot's onboard computer, nothing has to leave the building, and you own every image and record.
• Hardware agnostic. Inspection robots and sensor hubs are the most common way we capture data, but the platform works with any capture source.
• Patent pending. Polarity stays with your test equipment.

Where it stands today

Whole facility rounds, the 3D digital twin and change detection run in a production data center today. Since July 28, 2026, a FacilityOps robot has run inspection rounds at the CRSC Data Center, powered by Solutions IS, in Houston, completing 19 autonomous inspection runs and more than 1,600 observations in its first three weeks.

Fiber verification runs on our own test rack, and the one row trial proves it on your racks, judged on your own data. Run it beside your validation tool and compare what each one finds.

• LLDP: Link Layer Discovery Protocol, IEEE 802.1AB; each device tells its directly connected neighbor who it is and which port it is using.
• Cable validation: comparing what live switches report with an intended cable plan, link by link.
• Intended state: what the plan or source of truth says should exist, as opposed to what is on the floor.
• DOM: digital optical monitoring, the live temperature, voltage, bias and power readings a transceiver reports.
• Rx power: the optical power arriving at a receiver, in dBm.
• FEC: forward error correction, which fixes some bit errors at the receiver and counts them.
• CRC: a check that flags frames corrupted in transit.
• Gray failure: a partial fault that health checks miss while applications suffer.
• Shared risk link group: a tag for links that share a physical component and can fail together.
• Passive panel: a patch panel or cross connect that passes light without power or protocol.

What can LLDP not detect?

Anything between the two switch ports, and anything on an unpowered link. LLDP cannot see patch panels, cross connects, trunks, dust caps, labels, cable type, tight bends or a connector seated just well enough to pass light. It also says nothing before power on, because both ends must be powered and running it.

Does LLDP see patch panels?

No. Patch panels and cross connects are passive: they pass light without power or protocol, so LLDP messages go straight through them. LLDP reports only the switch port at the far end, so a cord in the right switch port but the wrong panel port can pass validation.

How do I find which patch panel port a switch port connects to?

Today it usually means checking the cable plan, then tracing the cord by hand from the switch to the panel and reading both labels. Software cannot help, because the panel is passive. A physical record that identifies each panel port by position makes the answer a lookup rather than a trace.

Can network monitoring detect a dirty fiber connector?

It can detect the effect, not the connector. A dirty end face lowers receive power and can raise CRC or FEC errors, but those readings come from the receiving port, and every connector along the path is a suspect. Finding the dirty one takes a scope, connector by connector.

What does DOM receive power tell you?

It tells you how much light is arriving at a transceiver, in dBm. Low receive power means light is being lost somewhere between the far transmitter and this receiver, whether from dirt, a bend, a loose cord, a damaged trunk or distance. It does not say which of these, or where.

What causes CRC errors on fiber links?

Most often a physical problem that corrupts the signal: a dirty connector, a bent or damaged fiber, or a bad or loose transceiver, according to Microsoft's study of production data centers. Shared components can degrade several links at once. CRC counters confirm the corruption but cannot locate its cause.

What is a gray failure in a data center network?

It is a fault that leaves a link or device partly working, so health checks report it as fine while applications suffer. A fiber link corrupting a small share of packets is a classic example: it stays up and passes a ping, while real traffic is retransmitted and jobs slow down.

Can software validate cabling before switches are powered on?

No. LLDP, DOM and error counters all need powered, configured devices and lit links, so software has nothing to read while switches are dark. Before power on, port placement, dust caps, labels and bends can only be checked physically, and polarity with test equipment.

Do redundant links share the same cable tray?

Often, without anyone intending it. Two logically separate paths can leave a rack through the same cassette or share a tray or trunk, so one physical event takes out both. Shared risk groups are usually entered by hand, so they are only as accurate as the records behind them.

Is NetBox an accurate record of physical cabling?

It records what should be there. NetBox's own documentation says it represents the desired state of a network rather than its operational state. That is the right design for a source of truth, but it means the record must be checked against the floor, especially after moves and changes.

What is the difference between cable validation and cable certification?

Cable validation checks that live links reach the planned neighbor and port, using switch data such as LLDP. Certification tests the physical cabling itself with test equipment, measuring loss, length and polarity against a standard. One checks connectivity on a live network; the other checks the cable's performance.

Does FacilityOps need access to our network?

No. A FacilityOps robot never connects to your switches. To confirm seating or go to a link your switches flagged, we work from an export of counters you produce on your own schedule. Everything else we check, before power on and after every change, is physical.

↑ Back to contents

Network software is very good at what it does: it sees live links end to end and catches the wrong port the moment a link comes up. It cannot see the passive middle, the dark ports before power on or the shared tray under two redundant paths.

Physical inspection covers exactly that gap, without touching the network. Together they cover the whole link, from the switch port through every panel to the far end.

Software shows two healthy paths. Physical inspection shows whether they are really two.

• NVIDIA: Config Manager cable validation (vendor)
• NVIDIA: UFM Cable Validation Tool, introduction (vendor)
• NVIDIA: Cumulus Linux Prescriptive Topology Manager (vendor)
• NetBox Labs: NetBox introduction (vendor)
• Wireshark wiki: Link Layer Discovery Protocol (IEEE 802.1AB)
• Nokia: Digital diagnostics monitoring (vendor)
• Zhuo and colleagues, Microsoft Research: Understanding and Mitigating Packet Corruption in Data Center Networks, ACM SIGCOMM 2017
• Huang and colleagues, Microsoft Research: Gray Failure: The Achilles' Heel of Cloud Scale Systems, HotOS 2017
• Gill, Jain and Nagappan: Understanding Network Failures in Data Centers: Measurement, Analysis, and Implications, ACM SIGCOMM 2011
• Govindan and colleagues, Google: Evolve or Die: High Availability Design Principles Drawn from Google's Network Infrastructure, ACM SIGCOMM 2016
• Field figures: FacilityOps interviews with data center technicians, contractors and commissioning engineers, October 2026

• Why Fiber Cabling Errors Survive Data Center Commissioning, and How to Catch Them Before Power On
• Fiber Cable Management in AI Data Centers: Tight Bends, Blocked Airflow and Overheating Optics
• Reducing Fiber Commissioning Rework With Robotics: Catch Defects Before Turn Up, Not After