Network level troubleshooting is incredibly difficult, especially for individuals who don't have a networking background. Even showing someone how to read wireshark often isn't enough.
I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.
So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.
And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.
True, but Wireshark will highlight dodgy TCP frames (retransmits, dups, etc) which should give a small clue to look further. I agree that it is necessary to understand how TCP works (or have access to someone who does) in order to run Internet services.
I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.
So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.
And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.