January 17 – February 13, 2025 · Lumen to Allo Fiber
The long weekend we moved YPIC to fiber.
At 5 PM on a Friday, we unplugged the agency from the internet provider it had used for years. Phones, email, websites, ISDS and the tunnels between our sites all had to be rebuilt on a new network. This is how the YPIC IT team brought them back, one service at a time.
5:00 PMFriday, Jan 17: the switch begins, after hours as we asked
4 daysThe MLK Day long weekend, Friday night to Tuesday morning
12+ hoursA normal day in the data center during the first ten days
9Services rebuilt on the new network, in order
27 daysFrom the first fiber cutover to the last site online
Phones first. Then everything else.
We couldn't bring the whole agency back at once, so we brought back what people needed most, first. This is the order services came back on the new network.
- PhonesFirst night
- InternetWeekend
- EmailWeekend · 72 h of held mail delivered
- PrintersWeekend · BizHubs
- Meet & ZoomBy Monday, Jan 20
- WebsitesJan 22–23 · DNS fixed
- ISDS & databasesJan 22–23 · the hardest part
- VPN to MLKJan 31 · Y drive back
- VPN to SomertonFeb 13 · move complete
The full story
- Before · planning the move
Questions for Allo Fiber
We asked how long the switch would take and how much downtime to expect. The answer was four to six weeks to install, and "a few minutes" of downtime at cutover, done after hours. The real job turned out much bigger, because so much of our network was built around the old provider.
Paperwork and planning
We sent Lumen our disconnect notice, filled out the justification forms that Allo needs to hand out blocks of static public addresses, and told the team what was coming. Our own words to the team that month: this would be a learning experience for all of us.
Getting ready
We mapped every server and device to its new address in one spreadsheet. Lumen had always hosted our public DNS, so we asked them for our records, then rewrote the whole zone for Allo with a five-minute refresh time so changes would spread fast. We prepared new router and firewall settings and saved them in a folder named for exactly when to use them: after 5 PM.
Moving the cutover to after hours
Allo scheduled the installs for 8:30 AM and 1 PM on Friday. We asked for 5 PM so no one would lose a working day, and Allo agreed. The new addresses for the main site arrived Thursday, one day before the switch.
- The long weekend
Lights out
Allo crews connected the Main Site and MLK to fiber. The moment Lumen went quiet, so did everything that depended on it: phones, email delivery, websites, ISDS and the tunnels between sites. We worked through the night.
Phones first, and one wrong address
Phones and extensions came back first, then internet. While checking our work we found one wrong address in the DNS file we had sent Allo, sent a corrected file the same evening, and pointed our domain to Allo's DNS servers.
A solid foundation
By the holiday Monday we had the network, email, Google Meet and Zoom, internet, BizHub printers, and phones and extensions working. We split the week into fronts so support requests kept moving while the core work continued.
- The hard week
Staff are back, the databases are not
ISDS, CRS, Program and the client systems were still offline. Our partners told us ISDS wasn't reachable for the public or for administrators. And we learned something no one had told us: unlike Lumen, Allo doesn't host DNS for its customers, so the internet had nowhere reliable to look up ypic.com. Mail sent to us was being held, not lost.
A server crashes
In the middle of all this, one of our servers crashed. Its off-site backup refused to restore. A second backup, made with a different tool and kept locally, worked. We relearned an old rule the hard way: if you really want it, keep more than one copy.
Calling in the person who built it
We proposed a short consulting contract with the engineer who originally built our network, our former IT manager: about ten hours over two days. That afternoon email was flowing again, the 72 hours of held mail started to arrive, and printers and scanners were back. After 5 PM we tested firewalls and VPNs.
ISDS comes back
The answer to DNS was already in our building: our own domain controllers could answer for ypic.com. Together we pointed the domain registrar at them, updated the records on them, fixed the mail records, reinstalled the main firewall with its new settings, and confirmed ISDS and our websites were working again. Firewall, VPN, Gmail and BizHub tests all passed.
Wrap-up, inside the budget
The consulting work finished within the hours we agreed to. What was left, the VPNs for MLK and Somerton, was waiting on new static address blocks from Allo, with written instructions for us to finish it ourselves.
- Finishing the sites
MLK fully operational
With its own address block in place, we rebuilt MLK's connection and tested it at noon. The VPN came back, and with it the shared Y drive. We also tracked down why MLK's new connection felt slow and learned how to test internet speed properly along the way. The same day, the team did MLK's twice-a-year maintenance desk by desk.
Fine-tuning
Some of our outgoing email was being marked as spam, so we asked Allo to set up reverse DNS for our addresses. Small details like this kept surfacing for weeks, and we kept closing them.
Somerton, last
Somerton took the longest. The equipment at the edge of that site belonged to the provider, so we couldn't get into it, and we didn't have all the access we needed for the site's firewall or its tunnels. We learned how to set up a tunnel between sites ourselves, Somerton got its new addresses and its VPN, and the internet side of the move was officially done. Fax lines for the south county offices were the last item, handled with Allo as a separate order.
The hardest part was ISDS.
ISDS runs on our oldest production server, and it had been set up years earlier by our former IT manager. Its network settings had been layered over time, and much of how it fit together was never written down. When its address changed, it stopped talking to the world, and ISDS, CRS, Program and the client systems went with it.
We wrote a plan for every way it could fail. Before touching it, we listed five plans in order, from the simplest check to the deepest, so that whoever was at the keyboard at 2 AM knew the next step.
Then we called the person who built it. It was not easy to ask for help, but it was the right call. Two days with the original architect gave us the missing pieces and a refresher on DNS. The part we are proudest of came after: we configured MLK and Somerton ourselves, using what we learned.
- ACheck the basic network settings: new address, gateway, name servers.
- BCheck the two network ports the server bonds into one connection.
- CCheck the server's own firewall and routing table.
- DRead the logs and capture live traffic to see what leaves and what comes back.
- ECheck the switch and router the server is plugged into.
- Written before we started, so every step had a next step.
Inside the firewalls, it went well. Outside was the problem.
That is how one of us summed it up. Everything we had planned worked. What slowed us down were the things nobody knew we didn't know.
Inside: went as planned
- The Allo Fiber installation
- Switching the internet over to Allo
- Changing every server's address
- Updating our public DNS records
- The new main firewall configuration
- Getting the copiers working with Google again
Outside: the surprises
- Allo doesn't host DNS for customers
- Our own domain controllers could do the job, and we hadn't known
- The equipment at the edge belonged to the provider, so we couldn't get into it
- Access to the Somerton firewall and the site tunnels lived with someone no longer here
- Nobody on the team had built a tunnel between sites before
What we can do now that we couldn't before
- Restore a server from backup, including a full bare-metal recovery
- Restore from a second backup tool when the first one fails
- Get domain controllers to recognize and sync with each other again
- Re-address the ISDS server and rebuild its bonded network ports
- Build and configure a tunnel between sites
- Track down a slow internet connection
- Run the servers' remote management cards on internal addresses
- Push back when a vendor says no
How we worked
We worked as one IT team on four fronts at the same time, so the agency could keep running while the network was rebuilt underneath it.
Front 1
The core
Firewalls, DNS with both providers and on our own servers, the data center, the ISDS server and the VPN tunnels.
Front 2
Keep the agency running
Support requests, an important Tuesday morning meeting on Google Meet, and onboarding a new IT intern in the middle of it all.
Front 3
The sites
A temporary Y drive for MLK, a local network set up like Somerton's, and desk-by-desk checks once each site was back.
Front 4
Keep people informed
We sent staff and partners an update at every milestone, always listing what was already working, not only what was still down.
What changed
Fiber at every site
The Main Site, MLK and Somerton run on Allo Fiber, and the old Lumen account is closed.
Each site has its own addresses
MLK and Somerton each have their own block of static addresses, so each firewall and tunnel stands on its own.
Our DNS, in our own hands
Our own domain controllers answer for ypic.com, and every public record is in a file we own and understand, instead of in a ticket queue at the old provider.
Know-how kept in the team
We can now rebuild a site's firewall and VPN ourselves. That knowledge used to live with one person, and so did some of the access. Now more than one of us has both.
What we learned
Get every address in writing first
The main site's addresses arrived one day before cutover, and the other sites' blocks came weeks later. Next time, no cutover date until every site has its addresses.
Know who hosts your DNS
Our old provider held our public DNS, and the new one doesn't offer it at all. Ask that question first. We now host it ourselves, keep a copy of every record, and lower the refresh time days before any change.
Keep the old service a little longer
We had about three days between the old line going away and needing everything back. Next time, we pay for one extra month of the old provider as a safety net. Hope for the best, plan for the worst.
Keep more than one backup
The off-site backup failed to restore when we needed it. A second copy, made with a different tool, saved the server. Test restores, not only backups.
No single keyholder
Some firewall and management access lived with one person who was no longer here. Every critical system now has more than one person who can get in.
Look for hidden dependencies
A fax line rode on another provider's internet, the phone system had its own router, and some partners used software VPNs. Map these before the day, not during it.
Call the builder, and learn while they're here
Asking the original architect for two days saved us many more. Treat that time as training, so the knowledge stays.
Schedule after hours, in writing
The installs were first booked for the middle of a workday. Agree on the cutover time in the contract.
Tell people what works
Every update started with what was already back. It kept patience high while the hard parts took longer.
What people said
“Everything inside the firewalls went very well; everything outside was the problem.”
From the IT team“If you really want it, you'll have multiple copies.”
From the IT team“Hope for the best, plan for the worst.”
From the IT team“Thank You all for your great work! It does not go unnoticed”
A partner program director, Jan 22“We value your expertise and your team.”
Agency leadership, Jan 24“Your professionalism and dedication is truly appreciated!”
Agency leadership, Feb 14
More voices coming
More of the IT team's memories of that weekend will be added here.