• iproute2 fwmark doesn't work (or iptables --set-mark doesn't)

    From Adam Wysocki@110:110/2002 to All on Tue Dec 3 13:45:33 2013
    Hi,

    I just subscribed this group. Hello everyone!

    I have a problem with setting up routing. In short, I want to mark certain packets with iptables and then catch these marks with iproute2. However
    Linux seems to ignore these marks.

    I have a 3.11.4 kernel with CONFIG_IP_ADVANCED_ROUTER and CONFIG_IP_MULTIPLE_TABLES
    set. Here's my setup:

    - eth0: LAN
    - IP 192.168.0.1, netmask /24
    - It connects to the LAN

    - eth1: WAN
    - IP 192.168.1.180, netmask /24, gateway 192.168.1.1
    - It connects to the Internet

    - tap0: VPN (created with OpenVPN)
    - Local IP 172.24.25.4, remote IP (and gateway) 172.24.25.2, netmask /24

    I want to pass certain traffic through 192.168.1.1 (eth1) and certain through 172.24.25.2 (tap0) - local traffic as well as routed traffic. Rules will be somewhat complicated (for locally-generated traffic and routed traffic), now just let's assume that I want to mark all packets to go through 192.168.1.1.

    I have added two new routing tables to /etc/iproute2/rt_tables:

    #v+
    2 vpn
    3 wan
    #v-

    I've created two of them, so the main table will not have a default gateway. This way I can avoid errors - if I make a mistake and traffic goes through
    the main table, it won't get out.

    Tables are then populated:

    #v+
    ip route flush table main
    ip route add 192.168.0.0/24 dev eth0 table main
    ip route add 192.168.1.0/24 dev eth1 table main
    ip route add 172.24.25.0/24 dev tap0 table main

    ip route flush table wan
    ip route add 192.168.0.0/24 dev eth0 table wan
    ip route add 192.168.1.0/24 dev eth1 table wan
    ip route add 172.24.25.0/24 dev tap0 table wan
    ip route add default via 192.168.1.1 dev eth1 table wan

    ip route flush table vpn
    ip route add 192.168.0.0/24 dev eth0 table vpn
    ip route add 192.168.1.0/24 dev eth1 table vpn
    ip route add 172.24.25.0/24 dev tap0 table vpn
    ip route add default via 172.24.25.2 dev tap0 table vpn
    #v-

    And now rules. I want packets marked with 2 going through WAN and packets marked with 3 going through VPN.

    #v+
    ip rule flush

    # restore default rules (deleted by flush)
    ip rule add from all lookup main prio 32766
    ip rule add from all lookup default prio 32767

    ip rule add fwmark 2 table wan
    ip rule add fwmark 3 table vpn
    #v-

    Now the cache is reloaded...

    #v+
    ip route flush table cache
    #v-

    And iptables flushed:

    #v+
    iptables -t filter -F
    iptables -t nat -F
    iptables -t mangle -F
    #v-

    And now the tricky part. I can't make iptables mark packets (or can't
    make iproute2 honor fwmark). I've tried different combinations of:

    #v+
    iptables -t mangle -A PREROUTING -j MARK --set-mark 2
    iptables -t mangle -A POSTROUTING -j MARK --set-mark 2
    iptables -t mangle -A OUTPUT -j MARK --set-mark 2
    iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
    iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
    iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
    iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
    iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
    iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2 iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2 iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
    iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
    #v-

    And nothing. All commands succeeed, but traffic still gets through the
    main table. Regarding to this:

    http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.netfilter.html

    it should work. What am I missing?

    Kernel doesn't have CONFIG_IP_ROUTE_FWMARK (it's completely absent), but
    I've read that in new kernels it is enabled by default and not found in config.

    Thanks for your time.

    Cheers.

    AW

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: ATMAN - ATM S.A. (110:110/2002@linuxnet)
  • From Tauno Voipio@110:110/2002 to All on Tue Dec 3 14:46:23 2013
    On 3.12.13 15:45, Adam Wysocki wrote:
    Hi,

    I just subscribed this group. Hello everyone!

    I have a problem with setting up routing. In short, I want to mark certain packets with iptables and then catch these marks with iproute2. However
    Linux seems to ignore these marks.

    I have a 3.11.4 kernel with CONFIG_IP_ADVANCED_ROUTER and
    CONFIG_IP_MULTIPLE_TABLES
    set. Here's my setup:

    - eth0: LAN
    - IP 192.168.0.1, netmask /24
    - It connects to the LAN

    - eth1: WAN
    - IP 192.168.1.180, netmask /24, gateway 192.168.1.1
    - It connects to the Internet

    - tap0: VPN (created with OpenVPN)
    - Local IP 172.24.25.4, remote IP (and gateway) 172.24.25.2, netmask /24

    I want to pass certain traffic through 192.168.1.1 (eth1) and certain
    through
    172.24.25.2 (tap0) - local traffic as well as routed traffic. Rules will be somewhat complicated (for locally-generated traffic and routed traffic), now just let's assume that I want to mark all packets to go through 192.168.1.1.

    I have added two new routing tables to /etc/iproute2/rt_tables:

    #v+
    2 vpn
    3 wan
    #v-

    I've created two of them, so the main table will not have a default gateway. This way I can avoid errors - if I make a mistake and traffic goes through the main table, it won't get out.

    Tables are then populated:

    #v+
    ip route flush table main
    ip route add 192.168.0.0/24 dev eth0 table main
    ip route add 192.168.1.0/24 dev eth1 table main
    ip route add 172.24.25.0/24 dev tap0 table main

    ip route flush table wan
    ip route add 192.168.0.0/24 dev eth0 table wan
    ip route add 192.168.1.0/24 dev eth1 table wan
    ip route add 172.24.25.0/24 dev tap0 table wan
    ip route add default via 192.168.1.1 dev eth1 table wan

    ip route flush table vpn
    ip route add 192.168.0.0/24 dev eth0 table vpn
    ip route add 192.168.1.0/24 dev eth1 table vpn
    ip route add 172.24.25.0/24 dev tap0 table vpn
    ip route add default via 172.24.25.2 dev tap0 table vpn
    #v-

    And now rules. I want packets marked with 2 going through WAN and packets marked with 3 going through VPN.

    #v+
    ip rule flush

    # restore default rules (deleted by flush)
    ip rule add from all lookup main prio 32766
    ip rule add from all lookup default prio 32767

    ip rule add fwmark 2 table wan
    ip rule add fwmark 3 table vpn
    #v-

    Now the cache is reloaded...

    #v+
    ip route flush table cache
    #v-

    And iptables flushed:

    #v+
    iptables -t filter -F
    iptables -t nat -F
    iptables -t mangle -F
    #v-

    And now the tricky part. I can't make iptables mark packets (or can't
    make iproute2 honor fwmark). I've tried different combinations of:

    #v+
    iptables -t mangle -A PREROUTING -j MARK --set-mark 2
    iptables -t mangle -A POSTROUTING -j MARK --set-mark 2
    iptables -t mangle -A OUTPUT -j MARK --set-mark 2
    iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
    iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
    iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
    iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK
    --restore-mark
    iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK
    --restore-mark
    iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2 iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2 iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK
    --restore-mark
    iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
    #v-

    And nothing. All commands succeeed, but traffic still gets through the
    main table. Regarding to this:

    http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.netfilter.html

    it should work. What am I missing?

    Kernel doesn't have CONFIG_IP_ROUTE_FWMARK (it's completely absent), but
    I've read that in new kernels it is enabled by default and not found in config.

    Thanks for your time.

    Cheers.

    AW


    Iptables and routing apply to layer 3 (network).
    Your tunnel interface (tap0) is a layer 2 (data link) device.

    If you're tunneling at data link layer, the firewall does
    not do anything at it.

    I'd start by changing the VPN to use tun0 instead.

    --

    Tauno Voipio


    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: A noiseless patient Spider (110:110/2002@linuxnet)
  • From Adam Wysocki@110:110/2002 to All on Tue Dec 3 18:07:01 2013
    Tauno Voipio <tauno.voipio@notused.fi.invalid> wrote:

    Iptables and routing apply to layer 3 (network).
    Your tunnel interface (tap0) is a layer 2 (data link) device.

    If you're tunneling at data link layer, the firewall does
    not do anything at it.

    I'd start by changing the VPN to use tun0 instead.

    Hi,

    Thanks for the reply, but I don't think it's the case here. Why:

    1. For now I'm dealing only with two tables - "main" and "wan".
    I'll use "vpn" table when I'll be able to correctly mark packets
    to pass through "wan" table instead of "main" table. For now we
    can assume that VPN doesn't exist, it will come into play later.

    2. tap0 is on layer 2, but it's only an interface, I want to
    redirect layer 3 traffic to this tunnel in the same way it
    would be done with other, physical datalink interfaces (e.g.
    Ethernet), basing on their assigned IP addresses.

    Once again thanks for the reply.

    Cheers.

    AW

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: ATMAN - ATM S.A. (110:110/2002@linuxnet)
  • From Pascal Hambourg@110:110/2002 to All on Tue Dec 3 22:58:31 2013
    Reply-To: pascal.news@plouf.fr.eu.org

    Tauno Voipio a ‚crit :

    Iptables and routing apply to layer 3 (network).
    Your tunnel interface (tap0) is a layer 2 (data link) device.

    So what ? eth0 and eth1 are layer 2 devices too.

    If you're tunneling at data link layer, the firewall does
    not do anything at it.

    I'd start by changing the VPN to use tun0 instead.

    Total nonsense. Iptables and ip do not care about the type of interface.
    As you wrote, they work at the layer 3.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From Pascal Hambourg@110:110/2002 to All on Tue Dec 3 23:13:17 2013
    Reply-To: pascal.news@plouf.fr.eu.org

    Adam Wysocki a ‚crit :

    And now the tricky part. I can't make iptables mark packets (or can't
    make iproute2 honor fwmark). I've tried different combinations of:

    Which combinations exactly ? You cannot have all those rules together,
    it just does not make sense.

    #v+
    iptables -t mangle -A PREROUTING -j MARK --set-mark 2

    Fine, that should mark all incoming packets.

    iptables -t mangle -A POSTROUTING -j MARK --set-mark 2

    Useless : POSTROUTING is after the routing decision, so it will have no
    effet.

    iptables -t mangle -A OUTPUT -j MARK --set-mark 2

    Fine, that should mark all locally generated packets.
    I'd start with just the above two rules in PREROUTING and OUTPUT, append
    a couple of LOG rules to print the packet mark in the kernel log and see
    what happens.

    iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
    iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
    iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
    iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK
    --restore-mark
    iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK
    --restore-mark
    iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2 iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2 iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK
    --restore-mark
    iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

    There are a lot of rules with --restore-mark (copy the connection mark
    to the packet mark), but only one rule setting the connection mark.

    And nothing. All commands succeeed,

    Did you check the actual results by printing the routing tables, routing
    rules and iptables ruleset ?

    but traffic still gets through the main table.

    How do you know ?

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From Adam Wysocki@110:110/2002 to All on Sat Dec 7 12:36:10 2013
    Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> wrote:

    Hi,

    Sorry for the late reply, but I finally found some time to fiddle with it...

    And now the tricky part. I can't make iptables mark packets (or can't
    make iproute2 honor fwmark). I've tried different combinations of:

    Which combinations exactly ? You cannot have all those rules together,
    it just does not make sense.

    Ok, I left only these rules that you said.

    I'd start with just the above two rules in PREROUTING and OUTPUT, append
    a couple of LOG rules to print the packet mark in the kernel log and see
    what happens.

    It seems that marking is done properly, but iproute2 does not direct
    matched packets to proper tables.

    iptables -t mangle -A PREROUTING -j MARK --set-mark 2
    iptables -t mangle -A PREROUTING -j LOG --log-prefix "prerouting "

    iptables -t mangle -A OUTPUT -j MARK --set-mark 2
    iptables -t mangle -A OUTPUT -j LOG --log-prefix "output "

    It gave log lines with MARK=0x02:

    Dec 7 13:27:32 somewhere kernel: [71053.449380] output IN= OUT=eth1 SRC=192.168.1.180 DST=89.72.158.60 LEN=107 TOS=0x00 PREC=0x00 TTL=64 ID=3020 DF PROTO=TCP SPT=43104 DPT=443 WINDOW=28723 RES=0x00 ACK PSH URGP=0 MARK=0x2 Dec 7 13:27:35 somewhere kernel: [71056.520793] prerouting IN=eth1 OUT= MAC=00:50:04:30:e1:90:74:ea:3a:bd:5d:10:08:00 SRC=89.72.158.60 DST=192.168.1.180 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=59395 DF PROTO=TCP SPT=443 DPT=44304 WINDOW=12600 RES=0x00 ACK URGP=0 MARK=0x2
    Dec 7 13:27:38 somewhere kernel: [71059.616705] output IN= OUT=eth0 SRC=192.168.0.1 DST=192.168.0.3 LEN=116 TOS=0x10 PREC=0x00 TTL=64 ID=14433 DF PROTO=TCP SPT=22 DPT=33868 WINDOW=330 RES=0x00 ACK PSH URGP=0 MARK=0x2
    Dec 7 13:27:38 somewhere kernel: [71059.617382] prerouting IN=eth0 OUT= MAC=00:0a:5e:60:ce:b0:00:19:7e:19:1e:10:08:00 SRC=192.168.0.3 DST=192.168.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58487 DF PROTO=TCP SPT=33868 DPT=22 WINDOW=1001 RES=0x00 ACK URGP=0 MARK=0x2

    And nothing. All commands succeeed,

    Did you check the actual results by printing the routing tables, routing rules and iptables ruleset ?

    Yes:

    [gof@raid ~]$ ip r s
    172.24.25.0/24 dev tap0 scope link
    192.168.0.0/24 dev eth0 scope link
    192.168.1.0/24 dev eth1 scope link

    [gof@raid ~]$ ip r s t vpn
    default via 172.24.25.2 dev tap0
    172.24.25.0/24 dev tap0 scope link
    192.168.0.0/24 dev eth0 scope link
    192.168.1.0/24 dev eth1 scope link

    [gof@raid ~]$ ip r s t wan
    default via 192.168.1.1 dev eth1
    172.24.25.0/24 dev tap0 scope link
    192.168.0.0/24 dev eth0 scope link
    192.168.1.0/24 dev eth1 scope link

    [gof@raid ~]$ ip ru s
    0: from all lookup local
    32764: from all fwmark 0x3 lookup vpn
    32765: from all fwmark 0x2 lookup wan
    32766: from all lookup main
    32767: from all lookup default

    [gof@raid ~]$ sudo iptables -t mangle -L -n
    Chain PREROUTING (policy ACCEPT)
    target prot opt source destination
    MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK set 0x2
    LOG all -- 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix `prerouting '

    Chain INPUT (policy ACCEPT)
    target prot opt source destination

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination
    MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK set 0x2
    LOG all -- 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix `output '

    Chain POSTROUTING (policy ACCEPT)
    target prot opt source destination

    but traffic still gets through the main table.

    How do you know ?

    Main table doesn't have default route, so:

    [gof@raid ~]$ ping 1.2.3.4
    connect: Network is unreachable

    [gof@raid ~]$ telnet 1.2.3.4 1234
    Trying 1.2.3.4...
    telnet: Unable to connect to remote host: Network is unreachable

    AW

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: ATMAN - ATM S.A. (110:110/2002@linuxnet)
  • From Pascal Hambourg@110:110/2002 to All on Sat Dec 7 15:14:26 2013
    Reply-To: pascal.news@plouf.fr.eu.org

    Adam Wysocki a ‚crit :

    but traffic still gets through the main table.
    How do you know ?

    Main table doesn't have default route, so:

    [gof@raid ~]$ ping 1.2.3.4
    connect: Network is unreachable

    [gof@raid ~]$ telnet 1.2.3.4 1234
    Trying 1.2.3.4...
    telnet: Unable to connect to remote host: Network is unreachable

    That's the result of the first routing decision which happens while the
    packet is being built, before it is passed to iptables' OUTPUT chains
    and marked. Thus the main routing table must have a default route
    (possibly with a nonexistent gateway so that packets won't actually be
    sent out using it) to avoid this error. However routing of incoming
    packets should have worked.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From Pascal Hambourg@110:110/2002 to All on Sat Dec 7 15:21:26 2013
    Reply-To: pascal.news@plouf.fr.eu.org

    Pascal Hambourg a ‚crit :

    That's the result of the first routing decision which happens while the packet is being built, before it is passed to iptables' OUTPUT chains
    and marked.

    One more side note about this : you can see in the OUTPUT chain log that locally generated packets have an initial source address and output
    interface as a result of this routing decision. The rerouting after the
    OUTPUT chain will update the output interface, but not the source
    address. So you may need to update it with SNAT or MASQUERADE.

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)
  • From logsnaath@gmail.com@1:0/0 to All on Sun Aug 3 20:06:28 2014
    I am facing similar issue.

    Does locally generated packets reach the OUTPUT chain after routing decision? Should there be any entries in main table always
    Any good solution.

    =============
    ip route flush cache
    ip route flush table main

    iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
    iptables -t mangle -A OUTPUT -m owner --uid-owner squid -j MARK --set-mark 0x5
    iptables -t mangle -A OUTPUT -j LOG --log-prefix "IPtab: "

    ip route add 192.168.1.0/24 dev wlan0 src 192.168.1.103 table logu
    ip route add default via 192.168.1.1 dev wlan0 table logu
    ip rule add fwmark 0x5 table logu

    ip route flush cache

    ========================
    As squid user

    $ ping 74.125.236.39
    connect: Network is unreachable

    Adding an entry like this in main table works
    $ route add -net 0.0.0.0/0 dev wlan0

    Regards
    Logu

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: The Kofo System II BBS telnet://fido2.kofobb
  • From Pascal Hambourg@110:110/2002 to All on Sun Aug 3 21:22:11 2014
    Reply-To: pascal.news@plouf.fr.eu.org

    logsnaath@gmail.com a ‚crit :
    I am facing similar issue.

    Similar to what ? That thread is so old that my news server does not
    have the messages any more.

    Does locally generated packets reach the OUTPUT chain after routing
    decision?

    Yes. There is always a routing decision before the OUTPUT chains. There
    may be a rerouting decision after the OUTPUT chains.

    Should there be any entries in main table always

    Not necessarily, but the routing rules and routes must be able to route
    the original packets before it is mangled in the OUTPUT chains,
    otherwise it is discarded immediately.

    Anyway, what would be the purpose of flushing the main routing table ?

    --- MBSE BBS v1.0.1 (GNU/Linux-i386)
    * Origin: Plouf ! (110:110/2002@linuxnet)