Intermittent Fiber Link Failures: How to Detect Them, Troubleshoot Them and Find the Physical Cause

• Intermittent links show up as flaps, CRC errors, rising corrected and uncorrectable FEC counts, or a link that is up but slow.
• In a Microsoft study of production data centers, contamination, bent fiber, bad or loose transceivers and shared components caused nearly all packet corruption.
• Receive and transmit power patterns point to a likely cause, but not to the exact connector, bend or panel.
• First repair attempts fixed the link only about half the time in that study, and each failed attempt added about two days.
• Moving cables during nearby work can cause transient loss in the cables touched, so check recent changes before replacing anything.
• In AI training, one flapping link can stall a synchronous job until a timeout fires.
• Troubleshoot physical first: counters, recent changes, the connection itself, the end face, then test equipment, and only then replace parts.

Short answer: A link that keeps working most of the time while showing signs of stress: it goes down and comes back, logs CRC errors, needs more and more FEC corrections, or reports low or wandering receive power. Sometimes the only symptom is a job that runs slower than it should.

People describe these faults in many ways, and they are usually describing the same few signals. Here is what to look for:

• Link flaps: the link drops and comes back on its own, sometimes once a day, sometimes several times an hour.
• CRC and FCS errors: frames arrive damaged and are thrown away, so traffic is retransmitted.
• Rising FEC counts: forward error correction fixes more and more errors, and eventually some codewords cannot be fixed at all.
• Low or fluctuating receive power: the optic reports less light than usual, or a value that moves.
• Up but slow: the link stays up and passes a ping, but real traffic suffers and jobs that use it fall behind.

What makes these faults hard is their timing. A link that flaps twice a day will pass any test you run in the ten minutes you spend at the rack, and the moment you close the ticket, it can start again.

That is the classic shape of a gray failure: the network's own health checks say the link is fine while the workload says it is not. The physical cause is still there, waiting for the next temperature swing or the next cord brushed by a passing hand.

An intermittent link is not a mystery. It is a physical fault that has not been looked at yet.

Short answer: Mostly physical problems: dirty or damaged end faces, bends from cable movement, loose or partly seated connectors and transceivers, disturbance during nearby work, heat at the optics and aging lasers. Microsoft's data puts contamination, bent fiber and loose transceivers at the top.

The best public evidence on causes comes from a Microsoft study of packet corruption across production data centers, presented at SIGCOMM 2017. It traced packet corruption to a short list of physical root causes, with these shares:

• Connector contamination: 17 to 57% of cases.
• Bent or damaged fiber: 14 to 48%.
• Bad or loose transceiver: 6 to 45%.
• Shared component failure: 10 to 26%, where several links share a failing part.
• Decaying transmitter: under 1%.

The ranges are wide because technicians often took several actions without recording which one fixed the link. The study also predates today's 400G and 800G links, so treat its numbers as direction rather than as current rates; the physical causes have not changed.

Causes the study does not list separately

Two more causes matter in modern halls. Nearby work moves cables, and a cable that is moved can bend, unseat or pick up dust, as section 06 explains. Heat matters too: transceivers run hotter behind blocked airflow or trapped bundles, and cable management in AI halls covers how dressing traps heat at the optics.

The technicians, contractors and commissioning engineers we interviewed agreed on the mix. Most problems were unseated transceivers, fibers and cassettes, plus kinks and dirt, and dirty end faces were the number one fiber defect.

Short answer: CRC errors count frames that arrived damaged; FEC counters show how many bit errors were corrected and how many codewords could not be fixed. The error rate before correction is the early warning, and both its limits and the transceiver's alarm thresholds depend on the link type and the module.

CRC and FCS errors

Every Ethernet frame carries a check value. If the frame arrives and the check does not match, the switch counts a CRC or FCS error and drops the frame. A rising CRC count says data is being damaged on its way in, which on fiber almost always means a physical problem somewhere along the link.

FEC: corrected and uncorrectable

Fast links use forward error correction, which adds extra bits so the receiver can repair some errors on its own. The corrected count tells you how hard FEC is working, and the uncorrectable count tells you how often it failed and a frame was lost.

The most useful number is the error rate before correction, often shown as pre FEC BER. It rises long before frames are lost, so it is the early warning that a link is getting worse. The acceptable limit depends on the link type and speed, so compare a link with others of the same type rather than with one fixed number.

Transceiver telemetry

Each optic also reports its temperature, voltage, laser bias, transmit power and receive power. Every module carries warning and alarm thresholds set by its manufacturer, so check readings against that module's own thresholds and against its history, not against a single number for all optics.

Read both ends of the link. A low receive value on one end means little until you know what the other end is transmitting, and as the guide to what network software cannot detect explains, none of these values says where along the link the problem sits.

Counters tell you a link is sick and roughly how. They never tell you which connector to look at.

Short answer: Specific patterns in receive power, transmit power, temperature, direction and timing point to a likely physical cause. Use them to decide where to look first, then confirm by looking.

Signature to cause

1. 1. Low receive power on one side, normal transmit power on both: points to contamination at a connector on the weak side's path.
2. 2. Low receive power on both sides, normal transmit power: points to bent or damaged fiber somewhere along the link.
3. 3. Transmit power low or falling on one side, receive power low on the other: points to an aging laser in the transmitting optic.
4. 4. Normal power on both sides, but errors anyway: points to a bad or loosely seated transceiver.
5. 5. Several neighboring links erroring together with good power: points to a shared component such as a panel, cassette or trunk.
6. 6. A loss difference between two wavelengths at the same OTDR event: points to a macrobend at that spot.
7. 7. Errors that started right after work in the rack: point to disturbance from that work.
8. 8. High module temperature with normal power: points to heat and airflow around the optic.
9. 9. Errors in one direction only: point to a physical fault rather than congestion.

The first five patterns come from the Microsoft study above, which used receive and transmit power to suggest likely causes. The authors note the method is imperfect, because transceivers do not report reflections, and the study predates current 400G and 800G optics.

The macrobend test comes from OTDR practice: VIAVI explains that longer wavelengths are more sensitive to bends, so an event that loses more light at the longer wavelength is likely a bend. The one direction rule is also from Microsoft's data: only 8.2% of links with packet corruption corrupted in both directions, against most congestion losses.

Every signature narrows the search. None of them replaces looking at the connection.

Short answer: Because the transceiver is often not the problem. In Microsoft's study, the first repair attempt fixed the link only about half the time, each failed attempt added about two days, and field crews report that most pulled optics were healthy.

Swapping the optic is the fastest action a technician can take, so it is usually the first. The Microsoft study found that the link was fixed by the first step only 50% of the time, and that each failed repair attempt added two more days during which the link had to stay disabled.

Following the study's own recommendations raised first attempt success to 80% when technicians followed them. The difference came from matching the action to the likely cause, which is exactly what a signature and a physical look provide.

The technicians, contractors and commissioning engineers we interviewed told us the same story from the floor. Pulled transceivers were rarely faulty: they had overheated behind blocked airflow or trapped bundles, and they worked again after cooling.

Meta engineers saw the same pattern at hyperscale. Their BER Illusion methodology paper, presented at ASME InterPACK 2021, found that a majority of optical link issues were resolved with no failed optic as the root cause, a high no trouble found rate for the parts that were pulled.

A swap can also hide the real cause for a while. Pulling and reinserting an optic disturbs the cord and the connectors, cools the module for a few minutes and can clear a loose seat, so the link recovers and the ticket closes, until the dirt, bend or heat brings it back.

If the link fails again after a new optic, the optic was probably never the problem.

Short answer: Yes. Moving cables during nearby work can cause transient loss in the cables that were touched, and a disturbed cord can be left bent, partly unseated or dusty. Errors that start right after work in a rack are a strong clue.

Microsoft researchers wrote it plainly in a HotNets 2024 paper on datacenter robotics: the movement of the cables can cause transient packet loss in the touched cables. The same paper notes that gray or transient failures are common and that a physical repair usually takes days.

In a dense rack, every change touches more than its own cords. A technician replacing one optic pushes aside a bundle, a door closes on a cord, a cassette is nudged. Most of these do no harm, but some leave a link one bump away from flapping.

The technicians, contractors and commissioning engineers we interviewed knew this well. One interviewee found connections disturbed by other teams only by matching link down times to who had worked in the area the day before.

The same logic applies to any recurring fault. Our guide to recurring HVAC failures shows how a dated record of what changed turns repeat failures into an explained pattern.

Before you replace anything, ask who worked near the link and when.

Short answer: Far more than one link's worth. Synchronous training waits for the slowest exchange, so a single flapping link can stall a whole job, trigger timeouts or force a restart from the last checkpoint, while thousands of GPUs sit idle.

Large training runs move data between every GPU in lockstep. When one link drops or slows, the others wait, and the cluster runs at the speed of its worst connection.

ByteDance and Peking University researchers described exactly this in their MegaScale paper at NSDI 2024. They saw training stalls and slowdowns caused by frequent network interface flapping, found that the default collective timeout fired before the network card came back up, and traced the problem to poor link quality between network card, optical cable and switch.

Meta's published research on a 54 day Llama 3 training run on 16,384 GPUs reported 419 unexpected interruptions, of which 35, or 8.4%, came from network switches and cables, as reported by DCD in July 2024. Those are only the interruptions that stopped the job; links that slowed it without stopping it do not appear in that count.

Even links that never drop can hurt. Google has written that a single straggler can cause a 60 to 70% drop in workload performance, and a link that corrupts and retransmits is one way a node can fall behind. As the cable management guide shows, heat at the optics is a common way that link gets there.

A degraded link that drops does not always stop the job. Traffic shifts to its redundant path and the network stays up, but with less headroom, so the next failure on that path becomes an outage. A link that keeps flapping quietly spends that spare capacity.

In an AI cluster, a flapping link is billed in GPU hours, not in optics.

Short answer: Physical first, parts last: read the counters and telemetry at both ends, check what changed nearby, inspect the physical connection, scope and clean the end faces, test with an OTDR if needed, and only then replace parts, one at a time.

1. 1. Read the counters and DOM at both ends: CRC, corrected and uncorrectable FEC, receive and transmit power, temperature, and when the errors started.
2. 2. Check recent changes: work orders, tickets and access in that row in the days before the first error.
3. 3. Inspect the physical connection: dust caps on neighbors, bends at the panel and manager, seating of the optic and connector, the label against the record, and heat around the module.
4. 4. Scope and clean the end faces: inspect both sides, clean if needed and inspect again, as the guide to fiber connector contamination explains.
5. 5. Test the link if loss persists: use an OTDR to locate a bend, a bad splice or a damaged section.
6. 6. Replace parts one at a time: transceiver, then fiber, then the port or card, checking the link after each step.

The last step matches vendor practice. NVIDIA's InfiniBand cluster operations guide lists cleaning and reinserting the fiber before replacing the transceiver, then trying a new fiber, then the switch or NIC, one fix at a time, and warns that a short error test may not be enough to prove a link is healthy.

The order saves time because the cheap, likely causes come first. Looking takes minutes, cleaning takes minutes, and both preserve the evidence that a swap would destroy.

Record what you found and what fixed it. The Microsoft study's ranges are wide precisely because nobody wrote down which action worked, and your next intermittent link deserves a better record than that.

Be careful with no fault found. Close a ticket as having no physical cause only when every point on the link's path has been seen with good evidence, from the optic at one end to the optic at the other. If any point was not seen, the honest status is unresolved, and the link stays on the watch list.

Short answer: Start from the first error time, list every change and every person in that row in the days before, compare images of the link's ports and its neighbors from before and after, check the physical condition, then decide what to replace.

The change correlation workflow

1. 1. Pull the first error time: the moment CRC, FEC or flap counts first rose on that link.
2. 2. List every change in the prior days: tickets, work orders, deliveries and access records for that row, with the people involved.
3. 3. Compare before and after images: the link's ports and its neighbors, looking for cords moved, caps missing or bundles pushed.
4. 4. Check the physical condition: seating, bends, caps, label and heat at the connection today.
5. 5. Decide what to replace: only after the evidence points to a part, and record which action fixed it.

The workflow depends on one thing most sites lack: images of how the rack looked before the fault. Without them, step three is guesswork; with them, the cause is often visible in a side by side comparison. Our guide to lidar and the digital twin shows what a before and after record can capture.

It also changes the conversation with other teams. A dated photo of a moved cord settles the question of what happened without anyone having to remember.

Short answer: When your switches flag a link, a FacilityOps robot goes to that port and photographs the connector, bend, cap and label, so the technician arrives knowing the likely cause, or knowing the cabling is ruled out. Before and after records from every round show what changed and when.

FacilityOps AI (RoboSkill Inc.) builds an autonomous inspection platform for data centers. Our inspection robots run fully autonomously, and a technician can take manual control at any time.

• Go to the bad link: a FacilityOps robot goes to the flagged port and photographs the connector, the bend, the dust caps on its neighbors and the label, before anyone touches it.
• Before and after: change detection on every round shows what moved, appeared or disappeared near the link, with dated images from the 3D digital twin.
• Disturbance and attribution: after anyone works in a rack, the next round shows what was disturbed and when, so the change correlation workflow starts with evidence.
• Heat at the optics: thermal scans find transceivers running hot against the rack's own baseline, compared with each module's own temperature reading, as thermal imaging does elsewhere in the hall.
• Last known healthy: a record of when the link was last physically healthy, so you know which change to look at.
• Seating confirmed: connectors visibly out of position are flagged, then confirmed against your link error counters.
• No false all clear: we report no physical cause found only after every point on the link's path has been seen with good evidence. Otherwise the finding stays unresolved and names the points that were not seen.

What a bad link visit looks like

For example, a switch flags a link on a leaf in the middle of the night. On its next pass the robot photographs both ends: the cord at the panel is bent tight behind a door that was closed on it the day before, the caps on its neighbors are in place, and the optic reads warm but inside its own limits.

The technician arrives with the photo and the likely cause, fixes the bend on the first visit and leaves the healthy optic where it is. The next round confirms the cord is dressed properly and the errors have stopped.

We never need access to your network. We work from an export of counters you produce on your own schedule, which is the split described in monitoring, inspection and robotic action.

We check the work and never do the work: no plugging, unplugging, reseating, cleaning or part swaps, and nothing in the signal path. Anything the robot cannot read clearly goes to a person, a person reviews every finding, and detection and the 3D model run on the robot's onboard computer, so you own every image and record. The platform is hardware agnostic, with robots and sensor hubs as the most common capture sources, and patent pending.

Predicting which links are likely to fail from heat is on our roadmap. So is a fingerprint of every link at handover, so that later repairs take minutes rather than days.

Where it stands today

Whole facility rounds, the 3D digital twin and change detection run in a production data center today. Since July 28, 2026, a FacilityOps robot has run inspection rounds at the CRSC Data Center, powered by Solutions IS, in Houston, completing 19 autonomous inspection runs and more than 1,600 observations in its first three weeks.

Fiber verification runs on our own test rack, and a trial proves it on your racks, judged on your own data. Run it for a month on the links your switches flag and count the parts you did not have to pull.

• Link flap: a link that goes down and comes back up on its own.
• CRC and FCS: check values that reveal frames damaged in transit; a mismatch is counted as an error.
• FEC: forward error correction, extra bits that let the receiver repair some errors.
• Pre FEC BER: the bit error rate before correction, the early warning of a degrading link.
• Uncorrectable codeword: a block FEC could not repair, so data was lost.
• DOM: digital optical monitoring, the temperature, voltage, bias and power readings an optic reports.
• Receive power: the light arriving at an optic, in dBm.
• Macrobend: a visible bend tight enough to leak light.
• Gray failure: a partial fault that health checks miss while workloads suffer.
• Straggler: a node or link that falls behind and slows a synchronous job.
• OTDR: optical time domain reflectometer, a tester that locates loss events such as bends and bad splices along a fiber.
• Transceiver: the pluggable optic that turns electrical signals into light and back at each end of a link.

What causes a fiber link to flap?

Usually a physical problem: a dirty or damaged end face, a bend, a loose or partly seated transceiver or connector, heat at the optic, or a cable disturbed by nearby work. Read the counters and power at both ends to narrow it down, then inspect the connection before replacing parts.

Why are CRC errors increasing on my fiber port?

Frames are arriving damaged, which on fiber almost always means light is being lost or distorted along the link. Common causes are contamination, bent fiber and loose optics. Compare receive and transmit power at both ends, check recent work in the rack, and inspect the connection.

How do I tell a bad transceiver from a bad cable?

Compare power at both ends. Normal power with errors points to the transceiver or its seating; low receive power on one side points to a dirty connector; low receive power on both sides points to bent or damaged fiber. Confirm by inspecting and scoping before swapping anything.

What is a normal receive power for a transceiver?

There is no single number. It depends on the module type and the link, so check the module's datasheet and the warning and alarm thresholds it reports. Compare the reading with the far end's transmit power and with the link's own history, not with other optics.

What does uncorrectable FEC mean?

Forward error correction repairs some bit errors at the receiver. An uncorrectable codeword is a block with too many errors to repair, so data is lost and must be resent. Rising uncorrectable counts mean the link is past the point FEC can hide, so look for the physical cause.

Can a dirty connector cause intermittent link drops?

Yes. Contamination lowers receive power and can push a link to the edge of working, so small changes in temperature or position tip it over. In Microsoft's study, contamination was the largest cause of packet corruption. Scope the end faces, clean if needed and inspect again.

Can bending a fiber cable cause packet loss?

Yes. A tight bend leaks light and lowers receive power, often on both ends of the link, which raises errors and can cause flaps. Microsoft researchers also found that moving cables can cause transient loss in the cables touched. An OTDR can locate a bend along the link.

Does reseating a transceiver fix flapping?

Sometimes, if the optic was loose, but it often only hides the cause. Pulling an optic disturbs the cord, cools the module and can clear a marginal seat for a while. In Microsoft's study, first repair attempts fixed the link only about half the time.

Can transceiver temperature cause a link to go down?

Yes. Optics running hot behind blocked airflow or trapped bundles can raise errors, flap or shut down, and heat ages lasers faster. Check the module's own temperature reading against its thresholds and against neighboring modules, and look for bundles or blockages around it.

How does a flapping link affect GPU training jobs?

Synchronous training waits for the slowest exchange, so one flapping link can stall a job. Researchers at ByteDance and Peking University reported training stalls from interface flapping, with collective timeouts firing before the link recovered. Restarts from checkpoints then cost GPU hours across the cluster.

Why does a link fail again after the optic was replaced?

Because the optic was probably not the cause. Dirt, a bend, a loose cord or heat around the module can survive a swap, and pulling the optic can briefly mask them. Inspect the connection, scope the end faces and check recent work before replacing more parts.

How do I connect link flaps to maintenance work?

Start from the time of the first error, list every change and person in that row in the days before, and compare before and after images of the link's ports and neighbors. Errors that start right after work in a rack strongly suggest disturbance from that work.

↑ Back to contents

Intermittent fiber links almost always have a physical cause: dirt, a bend, a loose seat, heat or a cable disturbed by nearby work. Counters and telemetry narrow the search, and the signature to cause list points to where to look.

Looking first, with a record of what changed and when, turns a fault that comes and goes into one that gets fixed once.

An intermittent fault is only intermittent between observations.

• Zhuo and colleagues, Microsoft Research: Understanding and Mitigating Packet Corruption in Data Center Networks, ACM SIGCOMM 2017
• Hong and colleagues, Microsoft Research: Self maintaining networked systems: the rise of datacenter robotics, ACM HotNets 2024
• Jiang and colleagues, ByteDance and Peking University: MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs, USENIX NSDI 2024
• DCD: Meta report details hundreds of GPU and HBM3 related interruptions to Llama 3 training run, July 29, 2024
• VIAVI Solutions: Macrobend detection using OTDR, 2021 (vendor)
• Google Cloud: Stragglers in AI: a guide to automated straggler detection, August 12, 2025 (vendor)
• NVIDIA: InfiniBand Cluster Operation and Maintenance Guide, list of scenarios (vendor)
• Rao, Chakravarty and Jani, Meta: BER Illusion Methodology, ASME InterPACK 2021, November 2021
• Field figures: FacilityOps interviews with data center technicians, contractors and commissioning engineers, October 2026

• Why Fiber Cabling Errors Survive Data Center Commissioning, and How to Catch Them Before Power On
• What Network Software Cannot Detect About Physical Cabling, and Why LLDP Is Not Enough
• Fiber Cable Management in AI Data Centers: Tight Bends, Blocked Airflow and Overheating Optics