• TCP connections freezing unexplicably

    From Stefan Monnier@110:110/2002 to All on Mon May 12 02:52:51 2014

    I have a problem where TCP connections to a machine of mine tend to
    freeze in a way I can't explain.

    Here are the symptoms:
    - I ssh into the machine, and right in the middle of some output the
    connection just freezes. If I have other simultaneous SSH connections
    multiplexed over the same underlying TCP connection, they freeze
    as well.
    - I connect via imap-over-ssl to that same machine, and similarly the
    connection randomly hangs.
    - As mentioned, those hangs seem to happen when there is activity,
    rather than when the connection is idle.
    - One TCP connection freezing does not prevent other TCP connections
    from working just fine. E.g. if I have two SSH connections (not
    multiplexed over the same TCP connection), one might freeze while the
    other keeps working fine.
    - This happens when connecting from various different places (e.g. from
    home behind a NAT router, from a server with no NAT in sight, ...).
    - I can somewhat reproduce the freeze at will, by connecting over ssh
    and then doing "du /home" (which generates a fair bit of output).
    Maybe I'll have to do "du /home" a second or a third time, but I've
    never needed more to reproduce the problem.
    - There's nothing magical about "du", the freeze also happens in many
    other cases.
    - "dmesg" is desperately silent.

    Some more info about the context:

    This machine is connected to the Internet via its eth0 with a fixed
    routable IPv4 address. It is running Debian stable over Linux-3.4
    (well, the linux-sunxi variant of it, but that shouldn't make much
    difference here, hopefully).

    Other than its eth0 interface it has two more network interfaces:
    - an ethernet USB dongle (name "eth-usb").
    - a tun0 handled by OpenVPN.
    Each of those uses a local non-routable (192.168.x.x) network, so
    it uses NAT to provide Internet access to those machines connected over
    one of those two interfaces.
    The iptables setup is very simple, see below.

    The SSH and IMAP-over-SSL connections I talk about above all come from
    the Internet (i.e. over the eth0 interface).

    Any idea where the problem might be coming from? How could I try to
    track it down?


    Stefan


    # iptables -t nat -vL
    Chain PREROUTING (policy ACCEPT 605K packets, 65M bytes)
    pkts bytes target prot opt in out source destination

    Chain INPUT (policy ACCEPT 595K packets, 65M bytes)
    pkts bytes target prot opt in out source destination

    Chain OUTPUT (policy ACCEPT 2104 packets, 147K bytes)
    pkts bytes target prot opt in out source destination

    Chain POSTROUTING (policy ACCEPT 11 packets, 874 bytes)
    pkts bytes target prot opt in out source destination
    2296 161K MASQUERADE all -- any eth0 anywhere anywhere # iptables -vL
    Chain INPUT (policy ACCEPT 1109K packets, 189M bytes)
    pkts bytes target prot opt in out source destination

    Chain FORWARD (policy ACCEPT 1925 packets, 417K bytes)
    pkts bytes target prot opt in out source destination

    Chain OUTPUT (policy ACCEPT 97455 packets, 29M bytes)
    pkts bytes target prot opt in out source destination
    #

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: A noiseless patient Spider (110:110/2002@linuxnet)
  • From Jorgen Grahn@1:0/0 to All on Mon May 12 10:21:50 2014
    On Mon, 2014-05-12, Stefan Monnier wrote:

    I have a problem where TCP connections to a machine of mine tend to
    freeze in a way I can't explain.

    Two things to start with:
    - 'netstat -s' -- monitor the machine-global counters for TCP and see if
    some unusual one starts increasing. Both on client and host would
    be nice.
    (Even nicer if you could monitor this per socket and avoid the
    background noise, but I don't think you can.)
    - tcpdump. You probably should specify a filter, e.g. 'icmp or tcp
    port XXXXX'.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From Richard Kettlewell@110:110/2002 to All on Mon May 12 19:35:42 2014
    Jorgen Grahn <grahn+nntp@snipabacken.se> writes:
    On Mon, 2014-05-12, Stefan Monnier wrote:

    I have a problem where TCP connections to a machine of mine tend to
    freeze in a way I can't explain.

    Two things to start with:
    - 'netstat -s' -- monitor the machine-global counters for TCP and see if
    some unusual one starts increasing. Both on client and host would
    be nice.
    (Even nicer if you could monitor this per socket and avoid the
    background noise, but I don't think you can.)
    - tcpdump. You probably should specify a filter, e.g. 'icmp or tcp
    port XXXXX'.

    Specifically, tcpdump both endpoints of a connection concurrently and
    see if they both see the same thing.

    My wild stab in the dark is that the problem is MTU-related.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Anjou (110:110/2002@linuxnet)
  • From Stefan Monnier@110:110/2002 to All on Tue May 13 13:26:50 2014
    My wild stab in the dark is that the problem is MTU-related.

    The netstat output didn't help me very much, and before I got to the
    tcpdump approach, I took a look at the MTU aspect, which made me think
    of comparing the setup (including ifconfig and ethtool's output) to the
    one from another machine which used to work in the exact same situation.

    It turns out that it's probably a bug in the ethernet card's driver.
    At least after playing around with "ethtool -s eth0 speed 100 duplex
    full" and "ethtool -s eth0 speed 1000 duplex full", the freezes are
    now gone.

    Thanks for putting me on the right path,


    Stefan

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: A noiseless patient Spider (110:110/2002@linuxnet)