• IPtables flagging packets invalid, no access

    From 86brown@1:0/0 to All on Sat Apr 19 20:13:54 2014
    Hi all,

    Looking to get some help on an issue I'm having, preventing me from getting=
    a certain new location going. New datacenter, connecting the feed they're = providing me to L3 switch. Have not been able to set up a proper iptables f= irewall in this location (everything is blocked, no access). Only way to ac= cess is adding my IP to allow list.

    Packets are being flagged as invalid, no reply to [SYN] packets from the se= rver. Not sure if this is whats blocking or not.

    This is what most of the blockings are saying:
    kernel: [20604.837769] Firewall: *TCP_IN Blocked* IN=3Dvenet0 OUT=3D MA= C=3D SRC=3D*MYIP* DST=3D*SERVERIP* LEN=3D48 TOS=3D0x00 PREC=3D0x00 TTL=3D11=
    2 ID=3D15851 DF PROTO=3DTCP SPT=3D61742 DPT=3D21 WINDOW=3D8192 RES=3D0x00 S=
    YN URGP=3D0

    Here's the invalid packet:
    kernel: [16708.550424] Firewall: *INVALID* IN=3Dvenet0 OUT=3D MAC=3D SR= C=3DMYIP DST=3DSERVERIP LEN=3D48 TOS=3D0x00 PREC=3D0x00 TTL=3D112 ID=3D1228=
    1 DF PROTO=3DTCP SPT=3D60992 DPT=3D80 WINDOW=3D8192 RES=3D0x00 SYN URGP=3D0

    I ran a Wireshark & tcpdump simultaneously to catch client/server side and = here was the result:
    Client side (my PC):
    77 4.434317000 192.168.2.244 SERVER-IP TCP 66 63866 > http [SYN] Seq=3D=
    0 Win=3D8192 Len=3D0 MSS=3D1460 WS=3D4 SACK_PERM=3D1

    Server side (server tcpdump)
    1 0.000000 MY-ISP-IP SERVER-IP TCP 68 61992 > http [SYN] Seq=3D0 Win=3D= 8192 Len=3D0 MSS=3D1452 WS=3D4 SACK_PERM=3D1

    So the only thing I notice is that the server-side MSS is different then wh=
    at the client (my PC) sent out. Is this normal, is this what's flagging as = invalid?

    Basically I'm trying to setup a cPanel server with csf firewall (which uses=
    iptables) but as soon as its active I get no access, have to log onto VPS = node, drop into via 'vzctl enter *' and shutdown iptables.

    For hardware I'm using a Nortel Baystack 5510 L3 switch, and I believe that=
    the DC is using a Cisco ASA but I could be wrong.

    Any suggestions to the solution here would be greatly appreciated!! :)

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From Pascal Hambourg@110:110/2002 to All on Sun Apr 20 09:27:49 2014
    Reply-To: pascal.news@plouf.fr.eu.org

    Hello,

    86brown a ‚crit :
    Hi all,

    This is what most of the blockings are saying:
    kernel: [20604.837769] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC=
    SRC=*MYIP* DST=*SERVERIP* LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=15851 DF PROTO=TCP SPT=61742 DPT=21 WINDOW=8192 RES=0x00 SYN URGP=0

    Here's the invalid packet:
    kernel: [16708.550424] Firewall: *INVALID* IN=venet0 OUT= MAC= SRC=MYIP
    DST=SERVERIP LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=12281 DF PROTO=TCP SPT=60992 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0

    We don't know what means "INVALID" nor the reason for "TCP_IN Blocked".
    This is ruleset-dependent. Examining the output of iptables-save may
    provide clues.

    I ran a Wireshark & tcpdump simultaneously to catch client/server side and
    here was the result:
    Client side (my PC):
    77 4.434317000 192.168.2.244 SERVER-IP TCP 66 63866 > http [SYN] Seq=0
    Win=8192 Len=0 MSS=1460 WS=4 SACK_PERM=1

    Server side (server tcpdump)
    1 0.000000 MY-ISP-IP SERVER-IP TCP 68 61992 > http [SYN] Seq=0 Win=8192
    Len=0 MSS=1452 WS=4 SACK_PERM=1

    So the only thing I notice is that the server-side MSS is different
    then what the client (my PC) sent out. Is this normal

    Clamping the MSS to 1452 is a rather common ISP practice to workaround
    MTU blackhole issues which may affect some DSL lines subscribers using
    PPPoE.

    is this what's flagging as invalid?

    Not if done properly, i.e. recalculating checksums as needed.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From 86brown@1:0/0 to All on Sun Apr 20 16:20:11 2014
    On Saturday, 19 April 2014 17:13:54 UTC-3, 86brown wrote:
    Hi all,
    =20
    =20
    =20
    Looking to get some help on an issue I'm having, preventing me from getti=
    ng a certain new location going. New datacenter, connecting the feed they'r=
    e providing me to L3 switch. Have not been able to set up a proper iptables=
    firewall in this location (everything is blocked, no access). Only way to = access is adding my IP to allow list.
    =20
    =20
    =20
    Packets are being flagged as invalid, no reply to [SYN] packets from the =
    server. Not sure if this is whats blocking or not.
    =20
    =20
    =20
    This is what most of the blockings are saying:
    =20
    kernel: [20604.837769] Firewall: *TCP_IN Blocked* IN=3Dvenet0 OUT=3D =
    MAC=3D SRC=3D*MYIP* DST=3D*SERVERIP* LEN=3D48 TOS=3D0x00 PREC=3D0x00 TTL=3D= 112 ID=3D15851 DF PROTO=3DTCP SPT=3D61742 DPT=3D21 WINDOW=3D8192 RES=3D0x00=
    SYN URGP=3D0
    =20
    =20
    =20
    Here's the invalid packet:
    =20
    kernel: [16708.550424] Firewall: *INVALID* IN=3Dvenet0 OUT=3D MAC=3D =
    SRC=3DMYIP DST=3DSERVERIP LEN=3D48 TOS=3D0x00 PREC=3D0x00 TTL=3D112 ID=3D12= 281 DF PROTO=3DTCP SPT=3D60992 DPT=3D80 WINDOW=3D8192 RES=3D0x00 SYN URGP=
    =3D0
    =20
    =20
    =20
    I ran a Wireshark & tcpdump simultaneously to catch client/server side an=
    d here was the result:
    =20
    Client side (my PC):
    =20
    77 4.434317000 192.168.2.244 SERVER-IP TCP 66 63866 > http [SYN] Seq=
    =3D0 Win=3D8192 Len=3D0 MSS=3D1460 WS=3D4 SACK_PERM=3D1
    =20
    =20
    =20
    Server side (server tcpdump)
    =20
    1 0.000000 MY-ISP-IP SERVER-IP TCP 68 61992 > http [SYN] Seq=3D0 Win=
    =3D8192 Len=3D0 MSS=3D1452 WS=3D4 SACK_PERM=3D1
    =20
    =20
    =20
    So the only thing I notice is that the server-side MSS is different then =
    what the client (my PC) sent out. Is this normal, is this what's flagging a=
    s invalid?
    =20
    =20
    =20
    Basically I'm trying to setup a cPanel server with csf firewall (which us=
    es iptables) but as soon as its active I get no access, have to log onto VP=
    S node, drop into via 'vzctl enter *' and shutdown iptables.
    =20
    =20
    =20
    For hardware I'm using a Nortel Baystack 5510 L3 switch, and I believe th=
    at the DC is using a Cisco ASA but I could be wrong.
    =20
    =20
    =20
    Any suggestions to the solution here would be greatly appreciated!! :)

    Thanks for your reply :)

    Could it be due to TCP window scaling or TCP sequence number randomization =
    on the ASA, or I've also read reference to PIX? Do you have any suggestions=
    of what I should ask my DC?

    Next time I go there (Tuesday), I'm going to bypass my L3 Baystack 5510 swi= tch, connecting the feed directly to the server. Then see if the packets ar=
    e still being flagged invalid and being dropped. This is a standard install=
    and I've setup 30 some servers in a similar fashion in different DC's but = not yet in this one as it's new and trying to get things prepared.

    No good if any new packet marked only [SYN] to initiate a connection is bei=
    ng ignored for some reason, thats the whole basis of a web server is initia= ting new connections to new visitors, so disabling stateful iptables rules =
    is not an option as we want SPI to be available.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From Pascal Hambourg@110:110/2002 to All on Sun Apr 20 16:54:25 2014
    Reply-To: pascal.news@plouf.fr.eu.org

    86brown a ‚crit :

    Could it be due to TCP window scaling or TCP sequence number
    randomization on the ASA, or I've also read reference to PIX?

    This is unlikely if "adding your IP address to the allow list" allows
    you to connect to the server. I think that it is due to the iptables
    ruleset.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From 86brown@1:0/0 to All on Sun Apr 20 17:23:17 2014
    On Sunday, 20 April 2014 13:54:25 UTC-3, Pascal Hambourg wrote:
    86brown a =E9crit :
    =20
    =20
    =20
    Could it be due to TCP window scaling or TCP sequence number
    =20
    randomization on the ASA, or I've also read reference to PIX?
    =20
    =20
    =20
    This is unlikely if "adding your IP address to the allow list" allows
    =20
    you to connect to the server. I think that it is due to the iptables
    =20
    ruleset.

    Yes adding my IP to the allow list does allow me through. The server receiv=
    es the packet but flags it invalid, and I assume it then checks to see if I=
    'm on the allow list whether to allow it or drop it (csf+lfd).

    The problem I have there is that even CentOS default iptables rules have th=
    is issue, if its enabled. Or at least when I switched over to OpenVZ kernel=
    .. I had to drive to the DC to do a 'service iptables stop' and 'chkconfig i= ptables off' to make sure I wouldn't lose access again during my messing wi=
    th it.

    I've compared the rules on a working server to non-working server, and even=
    copied the configuration of csf to the non-working server and there are no=
    changes.

    This is not happening on my other servers in other DC's (only diff here is = I'm using a Layer 3 switch, other DC's I'm on a Layer 2). I'm not sure if i=
    ts the DC traffic or once it goes past my L3 switch, but something is causi=
    ng iptables to flag those packets, which I would assume to be the root caus=
    e of the issue.

    Sure, I could disable the statefulness but this is not an ideal solution, e= specially for our clients that may want these types of features. Really nee=
    d to find out why these are coming in that way causing such an issue.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From Pascal Hambourg@110:110/2002 to All on Sun Apr 20 18:24:01 2014
    Reply-To: pascal.news@plouf.fr.eu.org

    86brown a ‚crit :

    I've compared the rules on a working server to non-working server, and
    even copied the configuration of csf to the non-working server and there
    are no changes.

    You keep talking about invalid packets, but you wrote in your first post
    that most firewall log messages were flagged "TCP_IN Blocked", not
    "INVALID". So what's the bigger problem ?

    Besides, we don't know what do these messages mean exactly. You need to
    check which rules produce them.
    Is it "INVALID" per the connection tracking ? AFAIK a TCP SYN packet
    cannot have the INVALID conntrack state, unless a connection with the
    same characteristics already exists (and the packet would be discarded
    by the TCP layer anyway), or the conntrack table is full. If "INVALID"
    meant "malformed" (e.g. bad checksum), then the packet would also be
    discarded by the TCP/IP stack regardless of iptables.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From 86brown@1:0/0 to All on Sun Apr 20 23:51:13 2014
    On Sunday, 20 April 2014 15:24:01 UTC-3, Pascal Hambourg wrote:
    86brown a =E9crit :
    =20
    =20
    =20
    I've compared the rules on a working server to non-working server, and
    =20
    even copied the configuration of csf to the non-working server and ther=
    e
    =20
    are no changes.
    =20
    =20
    =20
    You keep talking about invalid packets, but you wrote in your first post
    =20
    that most firewall log messages were flagged "TCP_IN Blocked", not
    =20
    "INVALID". So what's the bigger problem ?
    =20
    =20
    =20
    Besides, we don't know what do these messages mean exactly. You need to
    =20
    check which rules produce them.
    =20
    Is it "INVALID" per the connection tracking ? AFAIK a TCP SYN packet
    =20
    cannot have the INVALID conntrack state, unless a connection with the
    =20
    same characteristics already exists (and the packet would be discarded
    =20
    by the TCP layer anyway), or the conntrack table is full. If "INVALID"
    =20
    meant "malformed" (e.g. bad checksum), then the packet would also be
    =20
    discarded by the TCP/IP stack regardless of iptables.
    I honestly am not very much sure what's going on either, hence the reason f=
    or my postings here!

    All I know, is that the ports are opened, but I can't get access to my serv=
    er with iptables running, unless I explicitly add a rule for my IP (in othe=
    r words add it in CSF's allow list). CSF takes control of all that and it i=
    s working perfectly on my other CentOS 6.5 box in a different datacenter. *= Invalid* is only one of the things I saw, a lot of them say TCP_IN or UDP b= locked but I believe that is after the initial packet, upon TCP retransmiss= ion, not sure. In Wireshark, there is no response [ACK] to the initial pack=
    et [SYN] requesting to start a TCP session.

    Are default iptables rules on CentOS 6 to allow SSH connection by default? = Because if so, even that's not working.

    Either way, why would the packet be blocked if the port is opened?? Clearly=
    its hitting the firewall, with the logs I'm getting. I guess the question = is, what can make iptables think it needs to block a connection to an opene=
    d port?? Would it not be malformed or invalid packets of some kind? This is=
    why I keep referring to that, because it's the only oddity I've seen so fa=
    r.

    Any tips or suggestions on how to find out the problem and furthermore solu= tion to the issue would be great! I can go and remove the problem rules fro=
    m iptables, but that's just a band-aid and is not ideal to the resolution o=
    f the root cause of the issue. There must be a good reason that invalid/mal= formed packets are blocked, so me allowing them through purposely I'm sure =
    is not an action plan.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From 86brown@1:0/0 to All on Mon Apr 21 19:26:21 2014
    On Sunday, 20 April 2014 20:51:13 UTC-3, 86brown wrote:
    On Sunday, 20 April 2014 15:24:01 UTC-3, Pascal Hambourg wrote:
    =20
    86brown a =E9crit :
    =20
    =20
    =20
    =20
    =20
    =20
    =20
    I've compared the rules on a working server to non-working server, an=
    d
    =20
    =20
    =20
    even copied the configuration of csf to the non-working server and th=
    ere
    =20
    =20
    =20
    are no changes.
    =20
    =20
    =20
    =20
    =20
    =20
    =20
    You keep talking about invalid packets, but you wrote in your first pos=
    t
    =20
    =20
    =20
    that most firewall log messages were flagged "TCP_IN Blocked", not
    =20
    =20
    =20
    "INVALID". So what's the bigger problem ?
    =20
    =20
    =20
    =20
    =20
    =20
    =20
    Besides, we don't know what do these messages mean exactly. You need to
    =20
    =20
    =20
    check which rules produce them.
    =20
    =20
    =20
    Is it "INVALID" per the connection tracking ? AFAIK a TCP SYN packet
    =20
    =20
    =20
    cannot have the INVALID conntrack state, unless a connection with the
    =20
    =20
    =20
    same characteristics already exists (and the packet would be discarded
    =20
    =20
    =20
    by the TCP layer anyway), or the conntrack table is full. If "INVALID"
    =20
    =20
    =20
    meant "malformed" (e.g. bad checksum), then the packet would also be
    =20
    =20
    =20
    discarded by the TCP/IP stack regardless of iptables.
    =20
    I honestly am not very much sure what's going on either, hence the reason=
    for my postings here!
    =20
    =20
    =20
    All I know, is that the ports are opened, but I can't get access to my se=
    rver with iptables running, unless I explicitly add a rule for my IP (in ot= her words add it in CSF's allow list). CSF takes control of all that and it=
    is working perfectly on my other CentOS 6.5 box in a different datacenter.=
    *Invalid* is only one of the things I saw, a lot of them say TCP_IN or UDP=
    blocked but I believe that is after the initial packet, upon TCP retransmi= ssion, not sure. In Wireshark, there is no response [ACK] to the initial pa= cket [SYN] requesting to start a TCP session.
    =20
    =20
    =20
    Are default iptables rules on CentOS 6 to allow SSH connection by default=
    ? Because if so, even that's not working.
    =20
    =20
    =20
    Either way, why would the packet be blocked if the port is opened?? Clear=
    ly its hitting the firewall, with the logs I'm getting. I guess the questio=
    n is, what can make iptables think it needs to block a connection to an ope= ned port?? Would it not be malformed or invalid packets of some kind? This =
    is why I keep referring to that, because it's the only oddity I've seen so = far.
    =20
    =20
    =20
    Any tips or suggestions on how to find out the problem and furthermore so=
    lution to the issue would be great! I can go and remove the problem rules f= rom iptables, but that's just a band-aid and is not ideal to the resolution=
    of the root cause of the issue. There must be a good reason that invalid/m= alformed packets are blocked, so me allowing them through purposely I'm sur=
    e is not an action plan.

    Could it possibly be due to the fact that I've not configured the L3 switch=
    any further? Have not tagged/untagged any ports on the VLANS or the ISP po= rt, should I?

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From Pascal Hambourg@110:110/2002 to All on Mon Apr 21 23:34:14 2014
    Reply-To: pascal.news@plouf.fr.eu.org

    86brown a ‚crit :

    Could it possibly be due to the fact that I've not configured the L3
    switch any further? Have not tagged/untagged any ports on the VLANS or
    the ISP port, should I?

    No. VLAN tagging is a layer 2 function, iptables does not care about it.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From 86brown@1:0/0 to All on Tue Apr 22 00:15:43 2014
    On Monday, 21 April 2014 20:34:14 UTC-3, Pascal Hambourg wrote:
    86brown a =E9crit :
    =20
    =20
    =20
    Could it possibly be due to the fact that I've not configured the L3
    =20
    switch any further? Have not tagged/untagged any ports on the VLANS or
    =20
    the ISP port, should I?
    =20
    =20
    =20
    No. VLAN tagging is a layer 2 function, iptables does not care about it.

    Ok thank you, I'll just bypass the switch tomorrow and see if it still happ= ens. If it does, I have no idea what I'll ask my colo though.

    Thanks!

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From 86brown@1:0/0 to All on Wed Apr 23 03:10:41 2014
    On Sunday, 20 April 2014 06:27:49 UTC-3, Pascal Hambourg wrote:
    Hello,
    =20
    =20
    =20
    86brown a =E9crit :
    =20
    Hi all,
    =20
    =20
    =20
    This is what most of the blockings are saying:
    =20
    kernel: [20604.837769] Firewall: *TCP_IN Blocked* IN=3Dvenet0 OUT=
    =3D MAC=3D SRC=3D*MYIP* DST=3D*SERVERIP* LEN=3D48 TOS=3D0x00 PREC=3D0x00 TT= L=3D112 ID=3D15851 DF PROTO=3DTCP SPT=3D61742 DPT=3D21 WINDOW=3D8192 RES=3D= 0x00 SYN URGP=3D0
    =20
    =20
    =20
    Here's the invalid packet:
    =20
    kernel: [16708.550424] Firewall: *INVALID* IN=3Dvenet0 OUT=3D MAC=
    =3D SRC=3DMYIP DST=3DSERVERIP LEN=3D48 TOS=3D0x00 PREC=3D0x00 TTL=3D112 ID= =3D12281 DF PROTO=3DTCP SPT=3D60992 DPT=3D80 WINDOW=3D8192 RES=3D0x00 SYN U= RGP=3D0
    =20
    =20
    =20
    We don't know what means "INVALID" nor the reason for "TCP_IN Blocked".
    =20
    This is ruleset-dependent. Examining the output of iptables-save may
    =20
    provide clues.
    =20
    =20
    =20
    I ran a Wireshark & tcpdump simultaneously to catch client/server side =
    and here was the result:
    =20
    Client side (my PC):
    =20
    77 4.434317000 192.168.2.244 SERVER-IP TCP 66 63866 > http [SYN] Se=
    q=3D0 Win=3D8192 Len=3D0 MSS=3D1460 WS=3D4 SACK_PERM=3D1
    =20
    =20
    =20
    Server side (server tcpdump)
    =20
    1 0.000000 MY-ISP-IP SERVER-IP TCP 68 61992 > http [SYN] Seq=3D0 Wi=
    n=3D8192 Len=3D0 MSS=3D1452 WS=3D4 SACK_PERM=3D1
    =20
    =20
    =20
    So the only thing I notice is that the server-side MSS is different
    =20
    then what the client (my PC) sent out. Is this normal
    =20
    =20
    =20
    Clamping the MSS to 1452 is a rather common ISP practice to workaround
    =20
    MTU blackhole issues which may affect some DSL lines subscribers using
    =20
    PPPoE.
    =20
    =20
    =20
    is this what's flagging as invalid?
    =20
    =20
    =20
    Not if done properly, i.e. recalculating checksums as needed.

    Turns out issue was related to recent change in OpenVZ: http://git.openvz.org/?p=3Dvzctl;a=3Dcommit;h=3D9b8afa654945acc6d3bd782f622= aaf9c54e4e87b

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From Pascal Hambourg@110:110/2002 to All on Wed Apr 23 22:30:02 2014
    Reply-To: pascal.news@plouf.fr.eu.org

    86brown a ‚crit :

    Turns out issue was related to recent change in OpenVZ:

    http://git.openvz.org/?p=vzctl;a=commit;h=9b8afa654945acc6d3bd782f622aaf9c54e4e 87b

    Can you please elaborate ? What exactly caused the breakage and how did
    you fix it ?

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)