Reducing Fiber Commissioning Rework With Robotics: Catch Defects Before Turn Up, Not After

• Fiber is usually verified late in commissioning, at installation checks and Tier 1 testing, and the physical port check is often a sample.
• The technicians, contractors and commissioning engineers we interviewed put rework at about 5% of connections during a build and 1 to 2% still wrong at handover.
• Most remaining defects surface at turn up, after power on, when each one costs a troubleshoot, a callback and a retest.
• Late handover damages typically run $20,000 to $50,000 a day, and commissioning itself runs about $25,000 per MW.
• Checking is the bottleneck: senior technicians walk rows by hand, and junior facilities staff are in the shortest supply.
• Every defect type has a cheaper moment to be caught, almost always before power on.
• A robot check after the crew finishes, with port level punch items and photos, turns late rework into same day fixes.

Short answer: Fiber is usually verified at the installation check and equipment test levels, through visual checks and Tier 1 testing, and then again when switches are turned up. Exactly where depends on the owner, because commissioning levels are named and scoped differently from project to project.

Commissioning moves a data center from drawings to a proven, working facility in stages. Most projects use some version of the five levels, often with a design review in front:

• Level 0, design review: the design is checked against the owner's requirements, the cheapest place to find a problem.
• Level 1, factory tests: equipment is tested at the factory before it ships.
• Level 2, installation checks: installed equipment is inspected before power on.
• Level 3, equipment tests: each device is started and tested on its own.
• Level 4, system tests: each system is tested as a whole in place.
• Level 5, integrated systems test: everything runs together under failure scenarios.

The levels were built around power and cooling, and fiber often sits awkwardly inside them. Some owners treat structured cabling as part of installation checks, some hand it to the network team at turn up, and some fold it into the general contractor's closeout.

Tier 1 and Tier 2 testing

Fiber has its own test language. As Belden explains, Tier 1 tests use an optical loss test set to check loss, length and polarity, and TIA treats them as required. Tier 2 tests add an OTDR for troubleshooting and for checking splice, connector and return loss, and are optional.

Both are essential, and both measure the light, not the layout. A link can pass Tier 1 with flying colors and still land in the wrong port, carry the wrong label or sit with a cap missing next door. That physical check is usually a walk with a clipboard, and usually a sample.

Getting Levels 2 and 3 right pays off later. Clean installation checks and equipment tests mean fewer failed system and integrated tests at Levels 4 and 5, and fewer reruns of tests that tie up the whole commissioning team.

Fiber gets tested for light at Level 2 or 3. Whether every port matches the plan is often checked by sample, or at turn up.

Short answer: Because a small share of thousands of connections goes wrong during any build, checking samples rather than covers every port, and most of what slips through stays invisible until switches power on and links fail to come up.

The numbers come from the field. The technicians, contractors and commissioning engineers we interviewed told us about 5% of connections need fixing while a row is built: roughly 50 in an 800 connection row of 10 cabinets with 80 connections each.

Most of those are caught by the crew itself. Even so, a good contractor can hand over with up to 1 to 2% still wrong after its own testing, which is 8 to 16 per row and 100 to 200 on a hall with 10,000 connections. Those are the defects at handover the hub guide in this series walks through.

The mix is consistent: unseated transceivers, fibers and cassettes, plus kinks and dirt. Dirty end faces are the number one fiber defect, and one interviewee put cleaning and label problems at more than half of all punch list items.

Why they surface so late

Physical checks are sampled because checking every port by hand is slow. Scope results are often logged per panel, per link or as a checkbox rather than per connection, so the record cannot show which exact connection was checked.

That leaves turn up as the real test. When switches power on, network software finally compares live links with the plan, and wrong ports and dead links appear all at once. As the guide to what network software cannot detect explains, even then it cannot see passive panels, caps or labels.

Most fiber defects are not found by inspection. They are found by failure, at turn up.

Short answer: Each late defect costs a troubleshoot, a callback and a retest instead of a two minute fix, and late defects push handover back, where liquidated damages typically run $20,000 to $50,000 a day.

A wrong port found by the installer before power on is a cord moved and a box ticked. The same wrong port found at turn up starts a chain: a network engineer traces it, someone calls the contractor back, a technician fixes it under a change window, and the link is retested and recorded again.

Multiply that by 100 to 200 defects on a hall and the schedule moves. The technicians, contractors and commissioning engineers we interviewed put commissioning itself at about $25,000 per MW, and late handover liquidated damages typically at $20,000 to $50,000 a day.

For AI halls the opportunity cost is larger still. By our calculation from published GPU rental rates, an AI rack idle for a day forgoes roughly $18,000 to $47,000, and a 100 rack hall live one day sooner is worth $1.8 million to $4.7 million. A week late costs that same hall $12.6 million to $32.9 million.

Construction as a whole knows this pattern. A study by Autodesk and FMI estimated that decisions made on bad data caused 14% of all construction rework in 2020, about $88.69 billion globally (vendor study, September 2021). A cutsheet that does not match the floor is bad data in exactly that sense.

The defect costs the same to fix either way. Finding it late is what makes it expensive.

Short answer: Because port by port checking needs experienced people, and experienced people are scarce. Checking 1,000 ports by hand takes two people about two days and still only samples, while the industry struggles to hire.

The technicians, contractors and commissioning engineers we interviewed put manual checking of 1,000 ports at two people for about two days, roughly $4 to $5 per port at commissioning billing rates. Even then the result is a sample, because nobody walks every port on every row twice.

The workload keeps growing. Corning says a 72 GPU AI rack carries 576 fibers, 18 times a legacy CPU rack (vendor figure, July 2024), so each new hall multiplies the number of ports a senior technician is asked to walk.

The people are not there

In Uptime Institute's 2025 staffing survey, 37% of operators named hiring new staff a top challenge and 23% named retention, and junior facilities employees remained in the shortest supply. Our guide to growing data center challenges covers the wider labor picture.

The construction side is tighter still. CSIS estimates that AI infrastructure could require roughly 140,000 incremental skilled trades positions beyond current capacity by 2030 in its high case, and more than 100,000 in its mid case, across trades such as electricians and HVAC technicians rather than fiber alone.

Fiber has its own gap. The Fiber Broadband Association estimates the need for 180,000 additional fiber technicians over the next decade for broadband deployment, and large operators are training thousands of new fiber technicians of their own. New crews do good work, but their work gets checked twice, usually by the same few senior people.

The scarcest resource on a fiber build is not cable or optics. It is the senior technician's walk.

Short answer: Before power on, almost every time. Each common fiber defect is introduced during installation, usually found at turn up or later, and could be caught by a full port check after the crew finishes and before the switches light.

The defect to stage map

• Wrong port: introduced while patching; found today at turn up by validation tools, or never on passive panels; catch it with a port by port check against the cutsheet before power on.
• Missing dust cap: introduced during install on unused ports; found today rarely, often only after a contaminated connector fails; catch it before power on and on every round after.
• Wrong label: introduced while labeling; found today the next time someone traces the link, often months later; catch it by judging each label against the port's position before handover.
• Tight bend: introduced while dressing or closing doors; found today when errors rise, or never; catch it with close up checks before power on and after every change.
• Unseated connector: introduced while patching; found today at turn up when a link fails or flaps; catch it by flagging connectors visibly out of position before power on, confirmed against error counters once lit.
• Dirty end face: introduced before mating; found today at turn up as loss or errors; catch it with a scope before mating, with the result tied to the exact port.
• Wrong cable type: introduced at the pull or patch; found today at Tier 1 testing, at turn up or never; catch it with a check of jacket color and print against the design before power on.

The pattern is the point. Every defect on the list is introduced by a person during installation, and every one has a cheaper moment to be caught than the moment it is usually found.

That moment is the window between the crew finishing a row and the switches powering on. The installers are still on site, the racks are open and nothing is in service, so a fix costs minutes.

Every fiber defect has a cheaper moment to be caught, and it is almost always before power on.

Short answer: They work at room level. Site cameras and progress tracking confirm that a tray, a rack or a panel is installed; nothing in them confirms that port 47 on panel P2 holds the connector the cutsheet says it should.

Reality capture and construction progress tracking are valuable. They show what has been built against the schedule, catch clashes and give owners a visual record of the site, room by room and day by day.

They were not built to read a port. A 360 degree photo of a row shows that the panels are full; it cannot show which cord is in which port, whether a label matches its position or whether an unused port has its cap. The resolution and the record are both at the wrong level.

The port level punch list

Compare a typical room level punch item with the same finding written the way a robot record writes it:

• Room level: Hall 2, Row C, fiber patching incomplete, see photos.
• Port level: Hall 2 · Row C · Rack C07 · Panel P2 · Port 47 · Defect: connector found in port 48, cutsheet says port 47 · Photo attached · Found October 14, 10:42 · Closed October 14, 15:10 with photo of the corrected port.

The first tells a crew to go and look. The second tells them exactly what to fix and proves when it was fixed, which is what a commissioning agent needs to close the item.

Short answer: By checking every port after the crew finishes and before power on, sending findings with photos to the crew while they are still on site, and closing each punch item location by location with proof. Senior people then handle the exceptions only.

The logic is about timing and coverage, not about replacing anyone. An inspection robot walks a finished row, checks every port in scope against the cutsheet and photographs each one, which gives full coverage at the moment fixes are cheapest.

• Check before power on: run the walk after the crew finishes a row and before the switches light.
• Fix the same day: findings go to the crew with photos while they are still on site.
• Close with proof: each punch item is closed location by location, with a photo of the corrected port.
• See patterns: defect rates per installer or crew show who needs help, without anyone standing over them.
• Spend senior time well: the senior technician reviews the exceptions instead of walking every row.

The same walks help beyond fiber. Thermal scans across the hall during load bank tests catch hot breakers, busway joints and connections while the load is on, and during liquid cooling pressure tests every leak is recorded with the time, the location and photos from several angles. The commissioning agent still runs and signs off every test; the robot adds the evidence.

Positive confirmation

The most important change is in the record. Most checklists treat silence as a pass: if nothing was written down, the item is assumed fine. The technicians, contractors and commissioning engineers we interviewed told us positive confirmation, where each item is recorded as checked and passed, beats that every time.

A robot round produces positive confirmation by default, because every port in scope has its own entry and photo. It is the same principle as proving every inspection round was actually completed.

A robot does not make fewer mistakes than the crew. It finds the crew's mistakes while they are still cheap.

Short answer: Less than you might expect. Owners already pay for independent verification, and a port level record helps contractors get paid sooner and protects them later, while commissioning firms can use the robot as a tool inside their own scope.

Owners hire independent commissioning agents because the contractor cannot verify its own work, and that principle is already in most contracts. A robot round is one more way to verify contractor work, applied at the port.

The contractor's upside is real. Defects are found while the crew is still on site, punch items close with proof so sign off and final payment come sooner, and fewer late defects mean less exposure to liquidated damages.

There is protection too. If a connection is disturbed after handover by another trade, a dated photo of every port as it was left shows the contractor did it right. One interviewee found connections disturbed by other teams only by matching link down times to who had worked in the area the day before.

For commissioning firms, the robot is a tool rather than a competitor. It can run under their contract and their scope, and it frees their senior people for the judgment calls the job is really about.

Short answer: A FacilityOps robot checks every port in scope against your cable plan after the crew finishes and before power on, matches your crews' scope reports to their exact ports, closes punch items with photos and hands over a verified as built on day one.

FacilityOps AI (RoboSkill Inc.) builds an autonomous inspection platform for data centers. Our inspection robots run fully autonomously, and a technician can take manual control at any time.

• Every port in scope: a FacilityOps robot walks the row and checks that each connector is present in its planned port at both ends, including passive panels and cross connects that LLDP cannot see.
• Labels, caps and bends: ports are identified by their position in our 3D digital twin of the hall, labels are judged against that position, dust caps are checked on unused ports, and tight bends are flagged from close up images.
• Seating: connectors visibly out of position are flagged, then confirmed against link error counters once the link is lit.
• Scope reports: we import your crews' end face scope reports, match each passing result to its exact port, and flag any connection plugged in without one before power on.
• Punch list closure: findings arrive as port level punch items with photos, and can be matched against the punch list you already keep, for example an Autodesk Construction Cloud export.
• Verified as built: the result is a verified record of the row as built on day one, kept current on every later round.

We can run under your commissioning agent's or owner's representative's contract. We check the work and never do the work: no plugging, unplugging, reseating, cleaning or repair, and nothing in the signal path. Anything the robot cannot read clearly goes to a person, and a person reviews every finding before it reaches you.

The one row trial

Pick one row or one fit out and send us the cable plan. We check every port before power on, then compare our findings with your own turn up data in four groups: what both found, what only we found, what we missed and any false alarms.

The trial is measured on the metrics in the next section, including technician hours, which are measured in the trial rather than promised up front. Detection and the 3D model run on the robot's onboard computer, nothing has to leave the building, and you own every record.

Where it stands today

Whole facility rounds, the 3D digital twin and change detection run in a production data center today. Since July 28, 2026, a FacilityOps robot has run inspection rounds at the CRSC Data Center, powered by Solutions IS, in Houston, completing 19 autonomous inspection runs and more than 1,600 observations in its first three weeks.

Fiber verification runs on our own test rack, and the one row trial proves it on your racks, judged on your own data. Inspection robots and sensor hubs are our most common capture sources, but the platform works with any capture source. Patent pending.

Short answer: Measure what the owner cares about: defects caught before power on, days from crew complete to turn up, rework after turn up and days to accepted handover. Compare a checked row with an unchecked one.

• Defects caught before power on per row.
• Days from crew complete to turn up for each row.
• Rework tickets per 1,000 ports after turn up.
• Errors in the first 30 days of service on the checked links.
• Technician hours saved, measured in the trial, not estimated.
• Days from substantial completion to accepted handover.

Run the comparison on your own data, with your own turn up tools as the referee. If the checked row does not turn up cleaner and sooner, you will see it in the numbers.

• Commissioning levels: the staged tests that take a facility from design review to integrated systems test.
• IST: integrated systems test, the final level, where all systems run together under failure scenarios.
• Turn up: powering on network equipment and bringing links into service.
• Punch list: the list of items that must be fixed before handover.
• Tier 1 testing: loss, length and polarity testing with an optical loss test set.
• Tier 2 testing: optional OTDR testing for troubleshooting and connector, splice and return loss.
• Cutsheet: the cable plan listing which port connects to which.
• Liquidated damages: a fixed daily amount the contractor owes for finishing late.
• Substantial completion: the point at which the owner can use the facility for its purpose.

What are the levels of data center commissioning?

Most projects use five levels, often with a design review first: factory tests, installation checks, individual equipment tests, system tests and the integrated systems test, where everything runs together under failure scenarios. Naming and scope vary by owner, so check how your project defines each level before planning fiber checks.

At which commissioning level is fiber cabling verified?

Usually at installation checks and equipment tests, through visual checks and Tier 1 testing, and then again at turn up when network software compares live links with the plan. Where the port by port check sits varies by owner, and it is often a sample rather than every port.

What is the difference between Tier 1 and Tier 2 fiber testing?

Tier 1 uses an optical loss test set to check loss, length and polarity, and is required by TIA. Tier 2 adds an OTDR for troubleshooting and for checking connector, splice and return loss, and is optional. Neither checks that a link lands in the planned port with the right label.

What causes fiber commissioning rework?

Defects made during installation that slip past sampled checks: unseated transceivers, fibers and cassettes, wrong ports, kinks, dirty end faces and label problems. Most surface at turn up, after the crew has moved on, so each one becomes a troubleshoot, a callback and a retest instead of a quick fix.

How many fiber connections fail at turn up?

The technicians, contractors and commissioning engineers we interviewed told us about 5% of connections need fixing during a build, and even a good contractor can hand over with 1 to 2% still wrong after its own testing. On a 10,000 connection hall, that is 100 to 200 defects.

How much does a data center handover delay cost?

Late handover liquidated damages typically run $20,000 to $50,000 a day, according to the people we interviewed. For AI halls the lost use is larger: by our calculation from published GPU rental rates, an idle AI rack forgoes roughly $18,000 to $47,000 a day.

Why is there a shortage of fiber technicians?

Demand has outrun training. The Fiber Broadband Association estimates 180,000 additional fiber technicians are needed over the next decade for broadband alone, AI racks carry many times more fiber than older racks, and operators report junior facilities staff in the shortest supply.

Can robots inspect data center cabling during construction?

Yes, once a row is installed. An inspection robot can walk the finished row before power on, check every port in scope against the cutsheet, photograph each one and flag wrong ports, missing caps, label mismatches and tight bends while the installers are still on site.

What does reality capture miss in data center construction?

The port. Site cameras and progress tracking confirm that trays, racks and panels are installed, but they work at room level. They cannot confirm which cord is in which port, whether a label matches its position or whether an unused port still has its cap.

How do you close a punch list with proof?

Write each item at port level, with hall, row, rack, panel, port, the defect and a photo, then close it with a dated photo of the corrected port. A commissioning agent can then accept the item without walking back to check it in person.

Will contractors accept independent verification?

Most already work under it, because owners hire commissioning agents to verify contractor work. A port level record also helps contractors: defects are fixed while the crew is on site, punch items close sooner, and dated photos show the work was right if something is disturbed later.

How do you run a one row fiber verification trial?

Pick one row, send the cable plan and check every port before power on. Then compare the findings with your own turn up data: what both found, what only the check found, what it missed and any false alarms, plus days to turn up and rework tickets.

↑ Back to contents

Fiber defects are made during installation and usually found at turn up, after power on, when each one becomes a troubleshoot, a callback and a retest. The people who could fix them in minutes have often left the building by then.

Checking every port before power on, while the crew is still on site, moves that discovery to the cheapest possible moment and gives senior technicians their time back.

The cheapest defect is the one fixed before the crew leaves.

• CBRE: North American data center demand continues to outpace supply despite record construction, September 8, 2026
• LPL Research: Hyperscalers plan to live up to their name, February 12, 2026
• Hioki: Data center commissioning and 5 levels of testing (vendor)
• Belden: OLTS or OTDR: which test is acceptable for fiber certification?, February 8, 2024 (vendor)
• Autodesk and FMI: Better data strategies could save the global construction industry $1.85 trillion, September 14, 2021 (vendor study)
• Data Center Knowledge: Corning lifts expectations amid fiber demand in AI data centers, July 8, 2024, citing Corning (vendor)
• Uptime Institute: Staffing Survey 2025: staffing crisis persists, October 2025
• Center for Strategic and International Studies: GenAI's Human Infrastructure Challenge, September 16, 2025
• Fiber Broadband Association: OpTIC Path workforce program expansion, May 18, 2026
• Field figures: FacilityOps interviews with data center technicians, contractors and commissioning engineers, October 2026; idle rack and hall figures are our calculation from published GPU rental rates

• Why Fiber Cabling Errors Survive Data Center Commissioning, and How to Catch Them Before Power On
• Automated Fiber Optic Inspection: How Robots Check Every Port, Connector and Label
• Fiber Infrastructure Digital Twins: Keeping As Built Cabling Records True After Handover