I inherited a mess, now what?

Many of Crossconnect’s long term partnerships have started with us untangling a network “mess” for a new client. These “messes” are typically a result of poor design and patchwork growth with little planning. The cumulative result is an environment that’s completely broken or has intermittent problems and is a liability. 

Crossconnect has built our reputation as the go-to solution for sorting out messes and ensuring they never happen again. Every Crossconnect engineer goes through a rigorous interview and simulated network problem solving tests to ensure they can find creative solutions while under stress. 

We don’t want to just be another company that says, “Hey we’re good. Trust us.” The proof is always in the pudding. We have specific examples of problems and solutions we’ve come up with over the years. 

None of these networks were built by Crossconnect, and none of them had documentation available. They all had to be untangled “the hard way.” With that said, Crossconnect doesn’t require long-term projects. You may have a problem that requires only a few hours to fix, and if that’s the case, we can help with those too.

Case Studies

Some Messes We’ve Untangled Over the Years 

The Problem 

This example is a “triple play” that included voice, video, and data. The data and voice service worked perfectly. However, when multicast TV utilization increased after 5:00 p.m. local time, video began to fail, to the point where it was unwatchable. This only occurred in the new market and not in the existing ones. 

Crossconnect Solution 

Links were methodically eliminated, with no success. Line cards were then changed out on the central network core, until a bad network blade was found. The card was replaced and TV service was restored. 

The Problem 

The client was experiencing voice outages on heavy use days. On rare days when calls queued beyond fifty, all calls had severe audio loss. Tuning QoS had no effect on the problem. Changing network paths to determine how the voice cells came in on the WAN had no initial effect.  

Crossconnect Solution 

We started by finding a way to duplicate the problem using mass API-generated phone calls from an internet service to load up the call queue followed by using network traffic generation to busy out the network. We discovered a router in the network was improperly placed, and was hitting 99% CPU load when in use. The solution was using a different WAN link for an ingress path. 

The Problem 

A large, organically grown, seven-campus layer 2 hospital network was experiencing rolling network outages, taking out vital equipment every time the outage happened. Because deliberately recreating the problem in the production environment was out of the question, a mock-up of the network was built in Crossconnect’s lab, down to the specific IOS versions of the devices in question.

Crossconnect Solution 

A bug was found on the specific code release with Cisco’s uplinkfast and bankbonefast functions, and was able to be reproduced in the lab repeatedly. The features were safely removed from production and the network stopped experiencing the outages. Eventually, Crossconnect migrated the customer to a layer 3 network to avoid any similar future problems entirely.

The Problem 

A customer experienced high CPU load on their edge firewalls, and suspected the problem was related to a specific security application running inside the network. 

Crossconnect Solution 

Within thirty minutes of engaging, Crossconnect discovered a routing loop was likely the issue based on visual inspections of the routing tables, where hundreds of static routes had grown over the last fifteen years. Converting the bulk of the network to dynamic routing protocols eliminated the problem. The edge firewall CPU load dropped almost 50% over the subsequent two weeks.

The Problem 

A customer was having problems establishing external client VPN connections to their various Azure networks. The original network had been built, standardized, and then duplicated by a team unfamiliar with network concepts – twenty times over, with the same IP space. 

Crossconnect Solution 

Crossconnect implemented a “hub & spoke” network topology, assisted with re-IP of 19 of the networks, and then coded the changes into the customer’s Terraform.

The Problem 

A bank was losing internet connection to their Meraki routers, which were running their ATM network. This was the result of a change in their firewall, which was sitting in front of the Meraki network. 

Crossconnect Solution 

Not having seen any portion of the network prior to this, and during a full outage, Crossconnect methodically worked through the blockage, with the customer live on the phone call, and brought the network back online over the course of four hours.