Debate: Should you fail over to backup WAN or let users suffer through a partial outage?

We had a 10G fiber cut in our Denver office last Tuesday around 2pm. The primary link dropped hard but our Meraki SD-WAN didn't fail over to the 1G cable backup because the circuit wasn't technically "dead" - just packet loss above 40%. So users could load some pages but nothing worked right for about 45 minutes before I flipped the failover manually. My boss says that's the right call - let users know something's broken so we push the ISP to fix it faster. I think if we have backup bandwidth, we should use it automatically even if performance sucks. What do you guys do with your setups? Stick with strict failover thresholds or flip over at the first sign of trouble?

3 comments

3 Comments

linda_butler2825d ago

Nah @webb.val, hiding the problem just delays the real fix.

webb.val25d ago

40% packet loss is past what most apps can handle, video calls are dead and file transfers time out constantly. Tyler's right that 30% is already borderline for everything, so waiting for the link to fully die just wastes everyone's time. If the backup is there and functional, I say flip over automatically and let the monitoring tools tell the ISP something's broken instead of making users guess.

tylerw9225d ago

Read a Gartner report saying 30% packet loss is basically downtime for most apps anyway.