For two years now, Digital Operatives has hosted some systems on Verizon FiOS. Largely it has been trouble-free. That is, until about a week ago.
About a week ago, two employees (eventually three) described an issue they were having connecting to our mail server and SSH server residing on the FiOS network. When connected via their home FiOS connections, they would be able to establish a TCP connection, and then the connection would timeout before any meaningful data could be communicated. This happened to several of our employees, but not all employees have FiOS and they could connect from their home networks.
The employees with FiOS could connect with the same computer system—and establish strong connections—while connected via Comcast or AT&T phone tethering. What makes this weirder is we have a chat server that those same employees could connect to and maintain connections despite the problems with SSH and mail.
What accounted for the difference? Why would this happen?
It took a few days for us to not just cough it up to something strange or intermittent. Anybody that has ever dealt with an ISP’s tech support knows it is not going to be a terribly enjoyable experience. Nevertheless, after several days of troubleshooting (getting through the stages of Verizon help), they insisted on sending us a “new” modem/router. This was tremendously frustrating as we are not “run of the mill” users and couldn’t logically explain why it wasn’t something inside Verizon’s internal network as opposed to the router that we’ve had for two years.
Before waiting for the router, we decided to do some old school Internet troubleshooting. Using the ping command with the -s option, you can send packets with different sizes. Below are some results of using this command from our server to different employees’ locations.
| Command | Result | Comment |
|---|---|---|
ping ${NON_VERIZON_FIOS_EMPLOYEE_IP} |
100% connectivity | Expected result |
ping -s 500 ${VERIZON_FIOS_EMPLOYEE_IP} |
100% connectivity | Expected result |
ping -s 1500 ${VERIZON_FIOS_EMPLOYEE_IP} |
0% connectivity | ??? Strange |
ping -s 1499 ${VERIZON_FIOS_EMPLOYEE_IP} |
100% connectivity | Expected result |
What in the hell? Are you telling me that some router on Verizon’s network can’t handle 1500 byte packets?
Sure enough, after we changed the packet MTU size on the SSH server and mail server, connections were established and maintained just fine. No more issues…
Somewhere inside Verizon’s network is likely to be a misconfigured router or switch. Not sure I know how to tell the right person. Even if they are using a protocol or format that doesn’t support 1500 byte MTU, shouldn’t they at least support fragmentation on those links? The world may never know.
Good luck playing Call of Duty GHOSTS with your Verizon peers when the PS4 comes out!