• wpad.dat attack on Linux Apache server

    From Sandman@1:0/0 to All on Fri May 24 11:22:15 2013
    I don't actually know if it's an attack.

    My Apache server was slow - symptoms being slow responses (after PHP had processed and sent to client) and dropped connections, looked in the log files, found 12GB of accesses to /wpad.dat on my catch-all vhost.

    I got thousands of requests per minute from hundreds of different hosts,
    and a sample of these showed that they all seemed like legit end-user
    hosts, not a tor proxy botnet at least.

    I googled some, and found that wpad is some form of auto-discover proxy settings. Problem is, it should be done to the local network. So if my
    machine is on the "example.com" network, my browser will send a request
    to "wpad.example.com" to find proxy settings. not "wpad.remoteinternetsite.com".

    So, maybe it is an attack after all?

    I added a wpad.dat file to the server, with this content:

    function FindProxyForURL(url, host) { return "PROXY 127.0.0.1:445"; }

    Which just tells these clients to look for a proxy on localhost. Nothing changed. Thousands of thousands of requests.

    In /server-status for Apache, my queue is filled with /wpad.dat requests
    with the "K" status (Keep-Alive), so that sounds like why it's slow.

    Ok, so I'll block it. Blocking it in apache seemd stupid, it would still process the requests, so to iptables:

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
    --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

    Yes, I know this analyzes *every* request, and wastes CPU cycles, I may
    get around to chain this into a seperate iptables chain and only act on correct parts. In about an hour, this has blocked 45k requests, or about
    750 per minute

    Either way, the requests are gone, the log file is clean(er) but the
    server is still slow and still drops connections.

    So, troubleshooting some more. CPU is at 0.8% usage, Memory is 80% free. Asking my hosting ISP, my bandwidth capacity is at 0.62%

    ethtool -S eth0 shows no errors (but a shitload of packets, of course)

    Someone suggested that it could have something to do with my using a
    wildcard DNS. So, my server hosts some 100+ virtual hosts. All my
    clients are told to use a CNAME pointer for their servers. So:

    www.client.com -> CNAME -> cluster.mydomain.com -> A -> 123.123.123.123

    Which means that every visitor to my sites has their web browser first
    look up www.client.com to find cluster.mydomain.com which in turn points
    to my IP.

    No, the mydomain.com had a wildcard setting, so if and when they would
    access "wpad.mydomain.com" my DNS would point that to
    cluster.mydomain.com and then that wold point to the IP. So supposedly,
    all the request could channel to my server this way.

    I have now removed wildcard for mydomain.com, and also added a wpad host
    for all my domains that points to 127.0.0.1. I'm waiting to see that
    propagate and see if it makes any difference. It hasn't so far.


    Do any of you guys have any ideas what this might be? Or rather - how
    do I trouble shoot this some more?

    I have:

    Slow transfer speeds on apache
    Super fast on other ports (SFTP for instance)
    Thousands of requests per minute that are now being blocked
    Super low CPU usage
    Super low RAM usage
    No reported ethernet errors








    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 11:37:40 2013
    Sandman <mr@sandman.net> writes:
    Ok, so I'll block it. Blocking it in apache seemd stupid, it would still process the requests, so to iptables:

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
    --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

    Yes, I know this analyzes *every* request, and wastes CPU cycles, I may
    get around to chain this into a seperate iptables chain and only act on correct parts. In about an hour, this has blocked 45k requests, or about
    750 per minute

    While this will send a TCP reset to the misbehaving client, AIUI it will
    not send anything to your Apache, which by this point will have an open
    TCP connection and will be awaiting the start of the HTTP request. It
    will presumably continue waiting up to some timeout. You should be able
    to use netstat to confirm or refute this.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Fri May 24 11:44:27 2013
    In article <87vc68ra57.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    Sandman <mr@sandman.net> writes:
    Ok, so I'll block it. Blocking it in apache seemd stupid, it would still process the requests, so to iptables:

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
    --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

    Yes, I know this analyzes *every* request, and wastes CPU cycles, I may get around to chain this into a seperate iptables chain and only act on correct parts. In about an hour, this has blocked 45k requests, or about 750 per minute

    While this will send a TCP reset to the misbehaving client, AIUI it will
    not send anything to your Apache, which by this point will have an open
    TCP connection and will be awaiting the start of the HTTP request. It
    will presumably continue waiting up to some timeout. You should be able
    to use netstat to confirm or refute this.

    Thank you for your reply. I thought "reject" just rejected the request
    and nothing came to Apache?

    Using /server-status I see a huge difference in active connections, but
    I do see a tremendous amount of "Total accesses" which doesn't seem to correlate to the actual number of "valid" HTTP requests.

    Current Time: Friday, 24-May-2013 11:39:38 CEST
    Restart Time: Friday, 24-May-2013 11:22:51 CEST
    Parent Server Generation: 0
    Server uptime: 16 minutes 47 seconds
    Total accesses: 14027 - Total Traffic: 144.8 MB
    CPU Usage: u124.78 s5.91 cu136.95 cs0 - 26.6% CPU load
    13.9 requests/sec - 147.2 kB/second - 10.6 kB/request
    29 requests currently being processed, 24 idle workers

    ...._.KK.KK..........._.._...K._.R..._...K_.._...K......._R..K_.
    K_KKKK.._K_KW_WK_K__KK_....K_K_K___K_KK._.......................
    ................................................................
    ................................................................


    As you can see - almost a thousand accesses per minute. Yet, my child processes are free to serve new processes.

    "netstat -lap" shows about 400 lines of this:

    tcp 0 0 www.mydomain.com:www c-83-233-215-17.c:49686 SYN_RECV - tcp 0 0 www.mydomain.com:www h-5-200.a327.priv:50165 SYN_RECV - tcp 0 0 www.mydomain.com:www c-62-220-189-209.:50627 SYN_RECV - tcp 0 0 www.mydomain.com:www 238.77.85.212.bah:58190 SYN_RECV - tcp 0 0 www.mydomain.com:www c-89-160-22-176.c:57315 SYN_RECV -

    So yes, something is still knocking on that door - could this be bogging
    down my server?

    I have also a number of lines with TIME_WAIT status, suggesting that
    some queue is full here... Right?

    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 12:04:12 2013
    Sandman <mr@sandman.net> writes:
    Richard Kettlewell <rjk@greenend.org.uk> wrote:
    Sandman <mr@sandman.net> writes:
    Ok, so I'll block it. Blocking it in apache seemd stupid, it would still >>> process the requests, so to iptables:

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
    --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

    Yes, I know this analyzes *every* request, and wastes CPU cycles, I may >>> get around to chain this into a seperate iptables chain and only act on >>> correct parts. In about an hour, this has blocked 45k requests, or about >>> 750 per minute

    While this will send a TCP reset to the misbehaving client, AIUI it will
    not send anything to your Apache, which by this point will have an open
    TCP connection and will be awaiting the start of the HTTP request. It
    will presumably continue waiting up to some timeout. You should be able
    to use netstat to confirm or refute this.

    Thank you for your reply. I thought "reject" just rejected the request
    and nothing came to Apache?

    You’re rejecting a packet that is part of an already-established TCP connection. iptables cannot go back in time and prevent the TCP
    connection from being established in the first place.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Fri May 24 12:13:40 2013
    In article <87ppwgr8wz.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    While this will send a TCP reset to the misbehaving client, AIUI it will >> not send anything to your Apache, which by this point will have an open
    TCP connection and will be awaiting the start of the HTTP request. It
    will presumably continue waiting up to some timeout. You should be able >> to use netstat to confirm or refute this.

    Thank you for your reply. I thought "reject" just rejected the request
    and nothing came to Apache?

    You’re rejecting a packet that is part of an already-established TCP connection. iptables cannot go back in time and prevent the TCP
    connection from being established in the first place.

    Yes, like I said - I thought nothing came through to Apache. But looking
    at server-status, it seems it does anyway?


    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 12:20:40 2013
    Sandman <mr@sandman.net> writes:
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    While this will send a TCP reset to the misbehaving client, AIUI it will >> >> not send anything to your Apache, which by this point will have an open >> >> TCP connection and will be awaiting the start of the HTTP request. It
    will presumably continue waiting up to some timeout. You should be able >> >> to use netstat to confirm or refute this.

    Thank you for your reply. I thought "reject" just rejected the request
    and nothing came to Apache?

    You’re rejecting a packet that is part of an already-established TCP
    connection. iptables cannot go back in time and prevent the TCP
    connection from being established in the first place.

    Yes, like I said - I thought nothing came through to Apache. But
    looking at server-status, it seems it does anyway?

    I don’t know how to put it any more clearly; I give up.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Fri May 24 12:23:54 2013
    In article <87k3mor85j.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    While this will send a TCP reset to the misbehaving client, AIUI it
    will
    not send anything to your Apache, which by this point will have an
    open
    TCP connection and will be awaiting the start of the HTTP request. It >> >> will presumably continue waiting up to some timeout. You should be
    able
    to use netstat to confirm or refute this.

    Thank you for your reply. I thought "reject" just rejected the request >> > and nothing came to Apache?

    You’re rejecting a packet that is part of an already-established TCP
    connection. iptables cannot go back in time and prevent the TCP
    connection from being established in the first place.

    Yes, like I said - I thought nothing came through to Apache. But
    looking at server-status, it seems it does anyway?

    I don’t know how to put it any more clearly; I give up.

    No, please don't. Maybe I am misunderstanding you? I am not trying to
    argue with you.

    I thought that rejecting the TCP request in iptables blocked the request
    from ever reaching the httpd process. Obviously it isn't blocked from
    the *machine*, and I apologize if you thought that was what I meant.

    Mind you, I don't get any HTTP requests in Apache, but it does increment
    the requests number in a rate faster than the normal requests I see.

    You are free to call me stupid and ignorant about iptables/httpd here,
    of course, but I would still very much like to solve my problem even so
    :)


    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From J G Miller@110:300/1.1 to All on Fri May 24 13:07:54 2013
    On Friday, May 24th, 2013, at 12:23:54h +0200, Sandman pondered:

    I thought that rejecting the TCP request in iptables blocked the request from ever reaching the httpd process.

    You need to re-read the very first reply from R J Kettlwell.

    The sequence of events is, you have an open port 80, so
    some remote host connects on port 80 to your Apache server
    which then is waiting for data on what to do next.

    Now you have suggested a rule

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
    --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

    This will reject any packets containing the string "GET /wpad.dat"
    but it will not close the already opened connection to apache
    and furthermore will not block any packets from the host which do
    not contain the string "GET /wpad.dat".

    DONT' PANIC

    A quick web search reveals that other people have had
    this problem and it is probably not a malicious attack
    but most probably misbehaving Windoze 7 clients.

    Please read the discussion at

    <http://forums.freebsd.ORG/showthread.php?t=27668>

    which after three pages offers a potential solution to the problem.

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: A noiseless patient Spider (110:300/1.1@linuxnet)
  • From Chris Davies@110:300/1.1 to All on Fri May 24 13:31:24 2013
    Reply-To: chris@roaima.co.uk

    In comp.os.linux.networking Sandman <mr@sandman.net> wrote:
    In article <87k3mor85j.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:
    You're rejecting a packet that is part of an already-established TCP
    connection. iptables cannot go back in time and prevent the TCP
    connection from being established in the first place.

    Yes, like I said - I thought nothing came through to Apache. But
    looking at server-status, it seems it does anyway?

    I thought that rejecting the TCP request in iptables blocked the request from ever reaching the httpd process. Obviously it isn't blocked from
    the *machine*, and I apologize if you thought that was what I meant.

    The TCP sequence goes like this:

    1. Remote sends SYN to Webserver
    2. Webserver sends SYN/ACK to Remote
    3. Remote sends ACK to Webserver
    --connection now established--
    4. Remote sends "GET / HTTP/1.0 [..etc..]" to Webserver
    5. Webserver sends ACK to Remote
    6. Webserver sends the HTTP response to Remote
    7. Remote sends ACK
    8. Connection gets reused (from #4) or closed (FIN - FIN/ACK)

    Often #3 and #4 are merged, and potentially #5 and #6 could be, too. Item
    #6 might be spread across several packets, in which case the Remote will
    send an ACK (#7) for each packet.

    Your iptables rule matches #4, but by this stage the Webserver has already
    got a connection established from the Remote, and possibly even an Apache
    child ready to serve it.

    Chris

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Roaima. Harrogate, North Yorkshire, UK (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Fri May 24 14:29:32 2013
    In article <clv37ax9nr.ln2@news.roaima.co.uk>,
    Chris Davies <chris-usenet@roaima.co.uk> wrote:

    I thought that rejecting the TCP request in iptables blocked the request from ever reaching the httpd process. Obviously it isn't blocked from
    the *machine*, and I apologize if you thought that was what I meant.

    The TCP sequence goes like this:

    1. Remote sends SYN to Webserver
    2. Webserver sends SYN/ACK to Remote
    3. Remote sends ACK to Webserver
    --connection now established--
    4. Remote sends "GET / HTTP/1.0 [..etc..]" to Webserver
    5. Webserver sends ACK to Remote
    6. Webserver sends the HTTP response to Remote
    7. Remote sends ACK
    8. Connection gets reused (from #4) or closed (FIN - FIN/ACK)

    Often #3 and #4 are merged, and potentially #5 and #6 could be, too. Item
    #6 might be spread across several packets, in which case the Remote will
    send an ACK (#7) for each packet.

    Your iptables rule matches #4, but by this stage the Webserver has already got a connection established from the Remote, and possibly even an Apache child ready to serve it.

    Ok, thank you for the explanation, I had that backwards.

    So, where would I start at for finding out who has done a /wpad.dat
    request and then add them to a firewall IP block list? Maybe that's the
    best route to go?


    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 14:43:36 2013
    In article <knnhma$50u$4@dont-email.me>, J G Miller <miller@yoyo.ORG>
    wrote:

    On Friday, May 24th, 2013, at 12:23:54h +0200, Sandman pondered:

    I thought that rejecting the TCP request in iptables blocked the request from ever reaching the httpd process.

    You need to re-read the very first reply from R J Kettlwell.

    The sequence of events is, you have an open port 80, so
    some remote host connects on port 80 to your Apache server
    which then is waiting for data on what to do next.

    Now you have suggested a rule

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
    --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

    This will reject any packets containing the string "GET /wpad.dat"
    but it will not close the already opened connection to apache
    and furthermore will not block any packets from the host which do
    not contain the string "GET /wpad.dat".

    DONT' PANIC

    A quick web search reveals that other people have had
    this problem and it is probably not a malicious attack
    but most probably misbehaving Windoze 7 clients.

    Please read the discussion at

    <http://forums.freebsd.ORG/showthread.php?t=27668>

    which after three pages offers a potential solution to the problem.

    Yes, I have seen this thread, and they are talking about wildcard DNS
    being the culprit - I have yet to understand how this applies to me?

    I mean - I *DO* use wildcard DNS for all domains that I have a DNS for.
    The DNS server is on the machine that is currently being flooded on port
    80. Ok. So the DNS is "ns1.mydomain.com" (for example).

    So, for my clients - whose web pages I host on this machine, I tell
    *them* to point their subdomains (I.e. www.client.com) to the domainname "cluster.mydomain.com" as a CNAME record.

    Now, when the visitor types in www.client.com into the web browser,
    their DNS says that that resolvs to cluster.mydomain.com - which in turn
    has an A record for an IP number. So in the end, they surf to cluster.mydomain.com asking for www.client.com

    This works very good and has for more than a year.

    Now, mydomain.com har a wildcard DNS, meaning that
    "lkjkljklj.mydomain.com" points to "cluster.mydomain.com" and then to
    the IP.

    According to that thread, Internet Explorer and/or Windows makes
    assumptions about where to look for "wpad.dat", a javascript file that
    aims to provide the browser/hte OS info about proxy servers.

    So, Windows/IE asks for "http://wpad.client.com:80/wpad.dat" (as far as
    I know) and that's where the problem is.

    Now, to counter this:

    1. I have removed wildcard DNS on mydomain.com
    propagation may take a while though

    2. I am actively pointing wpad.mydomain.com to
    127.0.0.1, also waiting for the TTL there.

    3. I am trying to use iptables to block these accesses

    The open questions seem to be several, which the forum thread doesn't
    seem to have an answer for:

    1. Why would thousands of clients per minute all over Sweden ask for
    a wpad.dat file on *my* machine? According to the standard, they
    should be asking for it on wpad.*client.com*, not wpad.mydomain.com

    2. And why the *excessive* amount of traffic. several hundreds of IPS
    make up thousands of requests per minute, meaning that one IP makes
    several requests often.

    3. Blocking these IP-number, would I also be blocking their normal
    traffic to the server? Meaning, are these flooding some form of
    colleteral traffic from normal surfing?

    Thanks for all your replies, guys. This is a huge problem for me right
    now...


    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 14:49:48 2013
    In article <mr-983E88.11221424052013@News.Individual.NET>,
    Sandman <mr@sandman.net> wrote:

    I don't actually know if it's an attack.

    85.24.167.69 MY_IP - [24/May/2013:14:24:33 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "-"
    83.233.16.50 MY_IP - [24/May/2013:14:24:33 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "-"
    88.83.39.117 cluster.mydomain.com - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Win64; Trident/6.0)"
    88.83.39.117 cluster.mydomain.com - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Win64; Trident/6.0)"
    88.83.39.117 cluster.mydomain.com - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Win64; Trident/6.0)"

    Above are five sample rows (from thousands) on the server.

    The log format I'm using is putting the vhost in the second column, so you see that the two first requests are to my IP, and the second three are to cluster.mydomain.com <- The hostname where my clients point their CNAME subdomains

    The ones that access the IP has no agent string (consistently) and the ones that access cluster.mydomain.com does.

    I also have these:

    46.59.81.183 wpad - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36"


    I.e. accesses to the vhost "wpad", which is odd - becuase that's not the name of a vhost, nor is it a qualified domain name unless you're on a local network (right?) Is that a clue that some network out there thinks I am part of their local network?

    And, as you can see - one IP above made three identical requests (which it got a 200 reponse to) three times in one second. I have more lika that with fvive, six or seven conescutive times and so on.


    How do I block this?? :)



    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Casper H.S. Dik@1:0/0 to All on Fri May 24 15:05:23 2013
    Sandman <mr@sandman.net> writes:

    Yes, I have seen this thread, and they are talking about wildcard DNS
    being the culprit - I have yet to understand how this applies to me?

    All systems shipped these days are configured to search for a
    automatic proxy configuration from "http://wpad/wpad.dat"

    Because of your use of wildcard DNS *everyone* who starts their
    webbrowser will find wpad.their.domain then look for it on
    your webserver.

    Your ip filtering rules block your customers; it also makes their
    web experience interesting (it will take some time before the system
    figures out that there is no wpad.dat and will then connect to
    the internet directly)

    I mean - I *DO* use wildcard DNS for all domains that I have a DNS for.
    The DNS server is on the machine that is currently being flooded on port
    80. Ok. So the DNS is "ns1.mydomain.com" (for example).

    Well, you shouldn't have done that.

    1. Why would thousands of clients per minute all over Sweden ask for
    a wpad.dat file on *my* machine? According to the standard, they
    should be asking for it on wpad.*client.com*, not wpad.mydomain.com

    But you're serving their domains too, right?

    Casper

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 15:14:19 2013
    In article <519f6593$0$15954$e4fe514c@news2.news.xs4all.nl>,
    Casper H.S. Dik <Casper.Dik@OrSPaMcle.COM> wrote:

    Yes, I have seen this thread, and they are talking about wildcard DNS >being the culprit - I have yet to understand how this applies to me?

    All systems shipped these days are configured to search for a
    automatic proxy configuration from "http://wpad/wpad.dat"

    All? I thought it was Windows thing.

    Because of your use of wildcard DNS *everyone* who starts their
    webbrowser will find wpad.their.domain then look for it on
    your webserver.

    Why? I mean - when they go to wpad.their.domain, why would they end up
    with the IP of my server, or the CNAME of cluster.mydomain.com

    That's the part I just can't understand.

    Your ip filtering rules block your customers; it also makes their
    web experience interesting (it will take some time before the system
    figures out that there is no wpad.dat and will then connect to
    the internet directly)

    But the people surfing to my web server wouldn't be asking my server,
    any more than microsoft.com, for information about their own networks
    proxy settings, surely?

    I mean - I *DO* use wildcard DNS for all domains that I have a DNS for. >The DNS server is on the machine that is currently being flooded on port >80. Ok. So the DNS is "ns1.mydomain.com" (for example).

    Well, you shouldn't have done that.

    Fair enough, but I still don't know how that messed this up. I just
    can't wrap my head around it.

    1. Why would thousands of clients per minute all over Sweden ask for
    a wpad.dat file on *my* machine? According to the standard, they
    should be asking for it on wpad.*client.com*, not wpad.mydomain.com

    But you're serving their domains too, right?

    No. Only my own domains. Their IT managers have set up their subdomains
    (i.e. www.) to point to cluster.mydomain.com which points to my IP





    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Joe Beanfish@110:300/1.1 to All on Fri May 24 15:39:41 2013
    On Fri, 24 May 2013 15:14:19 +0200, Sandman wrote:

    In article <519f6593$0$15954$e4fe514c@news2.news.xs4all.nl>,
    Casper H.S. Dik <Casper.Dik@OrSPaMcle.COM> wrote:

    Yes, I have seen this thread, and they are talking about wildcard DNS
    being the culprit - I have yet to understand how this applies to me?

    All systems shipped these days are configured to search for a automatic
    proxy configuration from "http://wpad/wpad.dat"

    All? I thought it was Windows thing.

    Because of your use of wildcard DNS *everyone* who starts their
    webbrowser will find wpad.their.domain then look for it on your
    webserver.

    Why? I mean - when they go to wpad.their.domain, why would they end up
    with the IP of my server, or the CNAME of cluster.mydomain.com

    That's the part I just can't understand.

    Your ip filtering rules block your customers; it also makes their web
    experience interesting (it will take some time before the system
    figures out that there is no wpad.dat and will then connect to the
    internet directly)

    But the people surfing to my web server wouldn't be asking my server,
    any more than microsoft.com, for information about their own networks
    proxy settings, surely?

    I mean - I *DO* use wildcard DNS for all domains that I have a DNS
    for. The DNS server is on the machine that is currently being flooded
    on port 80. Ok. So the DNS is "ns1.mydomain.com" (for example).

    Well, you shouldn't have done that.

    Fair enough, but I still don't know how that messed this up. I just
    can't wrap my head around it.

    Wildcard DNS is asking for issues unless you fully understand all the ramifications. Best not to use it unless you really really need it and
    fully understand it.

    You have no entry for "wpad" so your wildcard is used. Get people off your server by creating a wpad entry in your dns that points to a nonexistent
    host or a host you want to handle that discovery traffic, even if only
    to reject it.

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: A noiseless patient Spider (110:300/1.1@linuxnet)
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 16:13:56 2013
    Sandman <mr@sandman.net> writes:
    Casper H.S. Dik <Casper.Dik@OrSPaMcle.COM> wrote:

    Because of your use of wildcard DNS *everyone* who starts their
    webbrowser will find wpad.their.domain then look for it on
    your webserver.

    Why? I mean - when they go to wpad.their.domain, why would they end up
    with the IP of my server, or the CNAME of cluster.mydomain.com

    That's the part I just can't understand.

    Perhaps quoting some of the domain names involved would clarify matters.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Fri May 24 16:23:13 2013
    In article <knnqit$ib6$1@dont-email.me>,
    Joe Beanfish <joebeanfish@nospam.duh> wrote:

    Well, you shouldn't have done that.

    Fair enough, but I still don't know how that messed this up. I just
    can't wrap my head around it.

    Wildcard DNS is asking for issues unless you fully understand all the ramifications. Best not to use it unless you really really need it and
    fully understand it.

    Fair enough.

    You have no entry for "wpad" so your wildcard is used.

    Yes, that's how wildcards works - but not only do I not understand why thousands of hosts from all over the swedish internet would start to
    request wpad.* on my server, some of the up to thirty times per second -
    per host!

    I am also not hosting any of their domains, so why would would they ever
    come to me to ask for this?

    Get people off your server by creating a wpad entry in your dns that
    points to a nonexistent host or a host you want to handle that
    discovery traffic, even if only to reject it.

    I did that yesterday, didn't change a single thing... :(



    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 16:26:56 2013
    In article <8761y8qxcr.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    Because of your use of wildcard DNS *everyone* who starts their
    webbrowser will find wpad.their.domain then look for it on
    your webserver.

    Why? I mean - when they go to wpad.their.domain, why would they end up with the IP of my server, or the CNAME of cluster.mydomain.com

    That's the part I just can't understand.

    Perhaps quoting some of the domain names involved would clarify matters.

    Yeah, ok.

    So a client to me, for example http://www.stadsnat.se has their DNS set
    up as such:

    host www.stadsnat.se
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Now, atlascms.se WAS a wildcard DNS, but isn't any longer.

    Even so, the requests I get look largely like this:

    94.254.41.78 cluster.atlascms.se - [24/May/2013:16:25:24 +0200] "GET
    /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0;
    Win32; Trident/6.0)"

    I.e. a request to that domain name, not to a wpad subdomain. So the
    wildcard DNS thing doesn't seem to even apply... Or am I mistaken?



    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 18:38:29 2013
    Sandman <mr@sandman.net> writes:
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    Because of your use of wildcard DNS *everyone* who starts their
    webbrowser will find wpad.their.domain then look for it on your
    webserver.

    Why? I mean - when they go to wpad.their.domain, why would they end
    up with the IP of my server, or the CNAME of cluster.mydomain.com

    That's the part I just can't understand.

    Perhaps quoting some of the domain names involved would clarify
    matters.

    Yeah, ok.

    So a client to me, for example http://www.stadsnat.se has their DNS set
    up as such:

    host www.stadsnat.se
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Now, atlascms.se WAS a wildcard DNS, but isn't any longer.

    Even so, the requests I get look largely like this:

    94.254.41.78 cluster.atlascms.se - [24/May/2013:16:25:24 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0;
    Win32; Trident/6.0)"

    I.e. a request to that domain name, not to a wpad subdomain. So the
    wildcard DNS thing doesn't seem to even apply... Or am I mistaken?

    I agree; I think the wildcard was probably a red herring, and the longer
    it gets since you removed it, the more certain that is (although it’s
    worth remembering that not all DNS clients honor TTLs correctly).

    The request doesn’t seem consistent with the way that wpad searching is described as working, but of course it may be that there’s more to the implementation that the various descriptions online imply.

    Do you have any idea how many distinct addresses are involved? Are they
    in fact _all_ Swedish IP addresses or are any of them from further
    afield? Can you tell whether any are associated with any of your
    customers (e.g. if you keep logs of where they upload from, do any of
    the oddly behaving clients appear there)?

    Have you recently annoyed anyone who might have sufficiently poor
    judgement to launch a DDoS attack?

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Casper H.S. Dik@1:0/0 to All on Fri May 24 18:47:27 2013
    Sandman <mr@sandman.net> writes:

    host www.stadsnat.se
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Now, atlascms.se WAS a wildcard DNS, but isn't any longer.

    So when someone looked up wpad.stadsnet.se it was mapped
    to cluster.atlascms.se? That, I think, is the root
    of your problem.

    Even so, the requests I get look largely like this:

    94.254.41.78 cluster.atlascms.se - [24/May/2013:16:25:24 +0200] "GET >/wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0;
    Win32; Trident/6.0)"

    I.e. a request to that domain name, not to a wpad subdomain. So the
    wildcard DNS thing doesn't seem to even apply... Or am I mistaken?

    I wouldn't be too sure about that. Note that wpad/wpad.dat is looked
    with a different algorithm then ordinary websites because it needs
    to sidestep the proxies and such.

    Casper

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 18:53:39 2013
    In article <519f999f$0$15903$e4fe514c@news2.news.xs4all.nl>,
    Casper H.S. Dik <Casper.Dik@OrSPaMcle.COM> wrote:

    host www.stadsnat.se
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Now, atlascms.se WAS a wildcard DNS, but isn't any longer.

    So when someone looked up wpad.stadsnet.se it was mapped
    to cluster.atlascms.se? That, I think, is the root
    of your problem.

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    I only know of one client that has wildcard:ed their DNS to me,
    actually. Maybe I should tell them to exempt wpad...

    Even so, the requests I get look largely like this:

    94.254.41.78 cluster.atlascms.se - [24/May/2013:16:25:24 +0200] "GET >/wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0;
    Win32; Trident/6.0)"

    I.e. a request to that domain name, not to a wpad subdomain. So the >wildcard DNS thing doesn't seem to even apply... Or am I mistaken?

    I wouldn't be too sure about that. Note that wpad/wpad.dat is looked
    with a different algorithm then ordinary websites because it needs
    to sidestep the proxies and such.





    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 19:02:13 2013
    In article <87d2sgl4e2.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    Even so, the requests I get look largely like this:

    94.254.41.78 cluster.atlascms.se - [24/May/2013:16:25:24 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0;
    Win32; Trident/6.0)"

    I.e. a request to that domain name, not to a wpad subdomain. So the wildcard DNS thing doesn't seem to even apply... Or am I mistaken?

    I agree; I think the wildcard was probably a red herring, and the longer
    it gets since you removed it, the more certain that is (although it’s
    worth remembering that not all DNS clients honor TTLs correctly).

    Indeed.

    The request doesn’t seem consistent with the way that wpad searching is described as working, but of course it may be that there’s more to the implementation that the various descriptions online imply.

    Or this seemingly benign request is used to stage a flood attack
    against me or my clients. Since google can't find any more serious
    attacks, especially not current one (there is that one forum post), I
    am starting to wonder why this is.

    Do you have any idea how many distinct addresses are involved?

    I now have a cronjob that reads the access_log file for wpad.dat
    requests and then add them to a blacklist and to iptables. It has been
    in effect for maybe two hours and the list is 4000 IP's long. 4000
    seemingly normal swedish IP's from normal swedish ISP's. All
    bombarding me with millions of wpad.dat requests.

    Some IP's send 30-40 requests per second in a burst.

    With 4000 in two hours, I'm guessing that tomorrow morning it will be
    over 10000, and then using iptables becomes increasingly stupid.

    Are they in fact _all_ Swedish IP addresses or are any of them from
    further afield?

    I have made samples now and then - all have been swedish IP's
    according to various online ip -> location functions.

    Can you tell whether any are associated with any of your
    customers (e.g. if you keep logs of where they upload from, do any of
    the oddly behaving clients appear there)?

    Even so, I don't have anywhere near to 4000 customers so this can't be
    due to one of my clients faulty network either. This seems like a
    targeted attack.

    Have you recently annoyed anyone who might have sufficiently poor
    judgement to launch a DDoS attack?

    I can think of only one person (from here on usenet) but he's from
    America and I doubt he has the ability to muster a botnet of
    Swedish-only clients. He has tried to flood me before, but only from a
    single IP. So no, I have to answer that I know of no one that could do
    this specifically against *me*. Maybe against one of my clients?

    Because, if they were targetting me, they would target my homepage (sandman.net) or some other, these attacks seem to either target the
    IP or my cluster domain name - and the cluster domain is not something
    used for anything but DNS redirection.



    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From J G Miller@110:300/1.1 to All on Fri May 24 19:15:40 2013
    On Friday, May 24th, 2013, at 18:53:39h +0200, Sandman wrote:

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    Did you read this article?

    <https://nodpi.ORG/2013/05/09/wpad-the-internet-explorer-security-flaw-that-exp oses-all-microsoft-users-in-the-uk/>

    According to that article all Microsoft Internet Explorer users in the UKofGB&NI
    are being directed by default towards a site run by a Brazilian, which obviously
    is not their DNS provider.

    QUOTE

    Sadly WPAD has some serious flaws. In particular, if DHCP discovery
    fails… WPAD reverts to a crude search for a source of configuration
    using DNS.

    In Windows, this search appears to be governed by the

    *DNS suffixes used to resolve unqualified domain names*

    (see the Advanced TCP/IP Settings dialog, right).


    UNQUOTE

    Perhaps you appearing as wpad.yourdomain.se because of the above (and maybe repeated combinations thereof because of your wild card setup) and so every MSIE from your clients tries that in its crude attempts to find
    the wpad proxy info?

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: A noiseless patient Spider (110:300/1.1@linuxnet)
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 19:45:15 2013
    Sandman <mr@sandman.net> writes:
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    Can you tell whether any are associated with any of your customers
    (e.g. if you keep logs of where they upload from, do any of the oddly
    behaving clients appear there)?

    Even so, I don't have anywhere near to 4000 customers so this can't be
    due to one of my clients faulty network either. This seems like a
    targeted attack.

    I don’t currently have a better theory.

    Have you recently annoyed anyone who might have sufficiently poor
    judgement to launch a DDoS attack?

    I can think of only one person (from here on usenet) but he's from
    America and I doubt he has the ability to muster a botnet of
    Swedish-only clients. He has tried to flood me before, but only from a
    single IP. So no, I have to answer that I know of no one that could do
    this specifically against *me*. Maybe against one of my clients?

    Because, if they were targetting me, they would target my homepage (sandman.net) or some other, these attacks seem to either target the
    IP or my cluster domain name - and the cluster domain is not something
    used for anything but DNS redirection.

    From what you’ve said (and I may be wrong) it sounds like it could be targetting your source of income. The ability to run a botnet
    personally isn’t necessarily relevant, even if you’re right about that; botnet operators rent them out.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From David W. Hodgins@110:300/1.1 to All on Fri May 24 19:50:38 2013
    On Fri, 24 May 2013 12:53:39 -0400, Sandman <mr@sandman.net> wrote:

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    If stadsnat.se is an isp, and stadsnat.se is a cname for your system,
    and their customers get dhcp addresses of the form ipaddr.stadsnat.se,
    then all of their customers will be going to stadsnat.se/yourssystem,
    to look for the wpad info.

    Workaround would require stadsnat.se to point to one of their own
    servers, and have www.stadsnat.se be the cname to your system, with
    there server redirecting stadsnat.se to www.stadsnat.se.

    By blacklisting the ip addresses, the website for stadsnat.se will
    become unreachable by their customers.

    Regards, Dave Hodgins

    --
    Change nomail.afraid.org to ody.ca to reply by email.
    (nomail.afraid.org has been set up specifically for
    use in usenet. Feel free to use it yourself.)

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: A noiseless patient Spider (110:300/1.1@linuxnet)
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 20:10:20 2013
    "David W. Hodgins" <dwhodgins@nomail.afraid.org> writes:
    On Fri, 24 May 2013 12:53:39 -0400, Sandman <mr@sandman.net> wrote:

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    If stadsnat.se is an isp, and stadsnat.se is a cname for your system,
    and their customers get dhcp addresses of the form ipaddr.stadsnat.se,
    then all of their customers will be going to stadsnat.se/yourssystem,
    to look for the wpad info.

    That wouldn’t explain why the Host: header is cluster.atlascms.se (you can’t reach that name using rdns, for instance).

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Richard Kettlewell@110:300/1.1 to All on Fri May 24 20:15:12 2013
    Richard Kettlewell <rjk@greenend.org.uk> writes:
    "David W. Hodgins" <dwhodgins@nomail.afraid.org> writes:
    On Fri, 24 May 2013 12:53:39 -0400, Sandman <mr@sandman.net> wrote:

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    If stadsnat.se is an isp, and stadsnat.se is a cname for your system,
    and their customers get dhcp addresses of the form ipaddr.stadsnat.se,
    then all of their customers will be going to stadsnat.se/yourssystem,
    to look for the wpad info.

    That wouldn’t explain why the Host: header is cluster.atlascms.se (you can’t reach that name using rdns, for instance).

    ....I wonder if it really is, or if the logfile fragments posted so far
    are misleading. Sandman, can you capture an example request off the
    wire, e.g. with:

    tcpdump -nX port 80 and host <one of the problem IPs>

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Fri May 24 21:12:17 2013
    In article <874ndsl1as.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    I can think of only one person (from here on usenet) but he's from
    America and I doubt he has the ability to muster a botnet of
    Swedish-only clients. He has tried to flood me before, but only from a single IP. So no, I have to answer that I know of no one that could do
    this specifically against *me*. Maybe against one of my clients?

    Because, if they were targetting me, they would target my homepage (sandman.net) or some other, these attacks seem to either target the
    IP or my cluster domain name - and the cluster domain is not something
    used for anything but DNS redirection.

    From what you’ve said (and I may be wrong) it sounds like it could be targetting your source of income. The ability to run a botnet
    personally isn’t necessarily relevant, even if you’re right about that; botnet operators rent them out.

    I didn't know that :)

    Thanks for your comments, it's a possible scenario I suppose.


    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 21:20:29 2013
    In article <kno77s$78m$2@dont-email.me>, J G Miller <miller@yoyo.ORG>
    wrote:

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    Did you read this article?


    <https://nodpi.ORG/2013/05/09/wpad-the-internet-explorer-security-flaw-that-ex
    poses-all-microsoft-users-in-the-uk/>

    Not that one specifically, but I have read about the point he is
    making.

    According to that article all Microsoft Internet Explorer users in
    the UKofGB&NI are being directed by default towards a site run by a Brazilian, which obviously is not their DNS provider.

    Which is due to UK citizens usually having a two part top level domain
    name. WPAD see's the domain "domain.co.uk" and thinks (correctly, one
    might add) that "domain" is a subdomain to "co" which is the local
    domain to the top level "uk". Which means that "wpad.co.uk" is a
    logical assumption for this function.

    This is not relevant to atlascms.se or any of swedish domains, really
    (we did have the pp.se domain thing for private person, but that was a
    decade ago).

    QUOTE

    Sadly WPAD has some serious flaws. In particular, if DHCP discovery
    fails… WPAD reverts to a crude search for a source of configuration
    using DNS.

    In Windows, this search appears to be governed by the

    *DNS suffixes used to resolve unqualified domain names*

    (see the Advanced TCP/IP Settings dialog, right).


    UNQUOTE

    Perhaps you appearing as wpad.yourdomain.se because of the above (and maybe repeated combinations thereof because of your wild card setup) and so every MSIE from your clients tries that in its crude attempts to find
    the wpad proxy info?

    The only one of my clients that have wildcard:ed their domain to me is opennet.se, I'll have a talk to them. As far as I know, they don't
    have almost 5000 users on their LAN though.

    because even if what you postulate is a possible scenario, it would
    under no circumstances generate hundreds of thousands of requests -
    sometimes 30-40 per second from one single host. That's where the
    entire "misconfigured DNS" idea falls slightly apart, don't you agree?

    I mean, if I came here wondering about these wpad.dat requests I see
    now and then, then that would be a logical question. But I get about
    20-30 requests per second, every second. That just can't be due to a misconfigured wildcard DNS.

    Or do you think I am jumping to conclusions?



    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 21:25:26 2013
    In article <op.wxlnuoipa3w0dxdave@hodgins.homeip.net>,
    "David W. Hodgins" <dwhodgins@nomail.afraid.org> wrote:

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    If stadsnat.se is an isp

    They aren't actually. The domain is owned by a client of miune, the
    citynet of rebro, a swedish town. They use "www.stadsnat.se" to
    promote their citynet. None of my clients are ISP's...

    and stadsnat.se is a cname for your system,

    Just to clarify - "stadsnat.se" can never be a CNAME, the top domain
    can only be pointed with an A record to an IP number. Their "www"
    subdomain is, however, CNAME:d to my "cluster.atlascms.se".

    and their customers get dhcp addresses of the form ipaddr.stadsnat.se,
    then all of their customers will be going to stadsnat.se/yourssystem,
    to look for the wpad info.

    Yes, all of what you say is true - if they would have been an ISP

    Workaround would require stadsnat.se to point to one of their own
    servers, and have www.stadsnat.se be the cname to your system, with
    there server redirecting stadsnat.se to www.stadsnat.se.

    Which, coincidentally, is exactly how it is set up right now :)

    By blacklisting the ip addresses, the website for stadsnat.se will
    become unreachable by their customers.

    Yes, this has been a concern for me. I did some checking, and of all
    the hundreds and thousands of IP addresses that are flooding me, I
    couldn't find any that was at the same time surfing the apache server normally.

    So, these IP's seem to be normal user IP's and I am actively blocking
    almost 5000 of them by now, which must include some that sooner or
    later will want to surf one of the many web sites hosted by this
    machine.

    Blocking them in the firewall is just a measure I'm taking in order to
    make my web server responsive again. It's hardly a long-term solution
    (where would it stop? 10k? 100k?)




    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Fri May 24 21:33:22 2013
    In article <871u8wz1lb.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    Richard Kettlewell <rjk@greenend.org.uk> writes:
    "David W. Hodgins" <dwhodgins@nomail.afraid.org> writes:
    On Fri, 24 May 2013 12:53:39 -0400, Sandman <mr@sandman.net> wrote:

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    If stadsnat.se is an isp, and stadsnat.se is a cname for your system,
    and their customers get dhcp addresses of the form ipaddr.stadsnat.se,
    then all of their customers will be going to stadsnat.se/yourssystem,
    to look for the wpad info.

    That wouldn’t explain why the Host: header is cluster.atlascms.se (you can’t reach that name using rdns, for instance).

    ...I wonder if it really is, or if the logfile fragments posted so far
    are misleading. Sandman, can you capture an example request off the
    wire, e.g. with:

    tcpdump -nX port 80 and host <one of the problem IPs>

    Certainly. I'll readily admit to not being sure whether this tells me
    anything or not. It appears to be a normal (?) ACK, followed by the
    HTTP request, with the host set to the IP number of the machine. and
    then followed by my machine giving them the wpad.dat file as I have
    set it up currently.

    21:27:29.055469 IP 83.172.125.62.65391 > 94.247.170.170.80: Flags [.],
    ack 1545, win 16232, length 0
    0x0000: 4500 0028 2866 4000 7606 01de 53ac 7d3e E..((f@.v...S.}>
    0x0010: 5ef7 aaaa ff6f 0050 5c93 2f59 a06e 3313 ^....o.P\./Y.n3.
    0x0020: 5010 3f68 36b2 0000 0000 0000 0000 P.?h6......... 21:27:29.905887 IP 83.172.125.62.65087 > 94.247.170.170.80: Flags
    [P.], seq 340:425, ack 1545, win 16328, length 85
    0x0000: 4500 007d 2877 4000 7606 0178 53ac 7d3e E..}(w@.v..xS.}>
    0x0010: 5ef7 aaaa fe3f 0050 9103 ca53 e213 d0bc ^....?.P...S....
    0x0020: 5018 3fc8 e032 0000 4745 5420 2f77 7061 P.?..2..GET./wpa
    0x0030: 642e 6461 7420 4854 5450 2f31 2e31 0d0a d.dat.HTTP/1.1..
    0x0040: 436f 6e6e 6563 7469 6f6e 3a20 4b65 6570 Connection:.Keep
    0x0050: 2d41 6c69 7665 0d0a 4163 6365 7074 3a20 -Alive..Accept:.
    0x0060: 2a2f 2a0d 0a48 6f73 743a 2039 342e 3234 */*..Host:.94.24
    0x0070: 372e 3137 302e 3137 300d 0a0d 0a 7.170.170.... 21:27:29.906203 IP 94.247.170.170.80 > 83.172.125.62.65087: Flags
    [P.], seq 1545:1931, ack 425, win 63, length 386
    0x0000: 4500 01aa c4f4 4000 4006 99cd 5ef7 aaaa E.....@.@...^...
    0x0010: 53ac 7d3e 0050 fe3f e213 d0bc 9103 caa8 S.}>.P.?........
    0x0020: 5018 003f dc28 0000 4854 5450 2f31 2e31 P..?.(..HTTP/1.1
    0x0030: 2032 3030 204f 4b0d 0a44 6174 653a 2046 .200.OK..Date:.F
    0x0040: 7269 2c20 3234 204d 6179 2032 3031 3320 ri,.24.May.2013.
    0x0050: 3139 3a32 373a 3239 2047 4d54 0d0a 5365 19:27:29.GMT..Se
    0x0060: 7276 6572 3a20 4170 6163 6865 2f32 2e32 rver:.Apache/2.2
    0x0070: 2e31 3620 2844 6562 6961 6e29 0d0a 4c61 .16.(Debian)..La
    0x0080: 7374 2d4d 6f64 6966 6965 643a 2054 6875 st-Modified:.Thu
    0x0090: 2c20 3233 204d 6179 2032 3031 3320 3231 ,.23.May.2013.21
    0x00a0: 3a34 313a 3037 2047 4d54 0d0a 4554 6167 :41:07.GMT..ETag
    0x00b0: 3a20 2233 3135 3430 3039 2d34 362d 3464 :."3154009-46-4d
    0x00c0: 6436 3938 6133 3665 3263 3022 0d0a 4163 d698a36e2c0"..Ac
    0x00d0: 6365 7074 2d52 616e 6765 733a 2062 7974 cept-Ranges:.byt
    0x00e0: 6573 0d0a 436f 6e74 656e 742d 4c65 6e67 es..Content-Leng
    0x00f0: 7468 3a20 3730 0d0a 4b65 6570 2d41 6c69 th:.70..Keep-Ali
    0x0100: 7665 3a20 7469 6d65 6f75 743d 3135 2c20 ve:.timeout=15,.
    0x0110: 6d61 783d 3335 380d 0a43 6f6e 6e65 6374 max=358..Connect
    0x0120: 696f 6e3a 204b 6565 702d 416c 6976 650d ion:.Keep-Alive.
    0x0130: 0a43 6f6e 7465 6e74 2d54 7970 653a 2061 .Content-Type:.a
    0x0140: 7070 6c69 6361 7469 6f6e 2f78 2d6e 732d pplication/x-ns-
    0x0150: 7072 6f78 792d 6175 746f 636f 6e66 6967 proxy-autoconfig
    0x0160: 0d0a 0d0a 6675 6e63 7469 6f6e 2046 696e ....function.Fin
    0x0170: 6450 726f 7879 466f 7255 524c 2875 726c dProxyForURL(url
    0x0180: 2c20 686f 7374 2920 7b20 7265 7475 726e ,.host).{.return
    0x0190: 2022 5052 4f58 5920 3132 372e 302e 302e ."PROXY.127.0.0.
    0x01a0: 313a 3434 3522 3b20 7d0a 1:445";.}.




    And here is the request from a host I specifically looked where
    logging as the "cluster.atlascms.se" vhost:

    1:30:59.331074 IP 85.24.180.196.60901 > 94.247.170.170.80: Flags [P.],
    seq 2476116866:2476116951, ack 4035600534, win 16425, length 85
    0x0000: 4500 007d 05e3 4000 7806 e919 5518 b4c4 E..}..@.x...U...
    0x0010: 5ef7 aaaa ede5 0050 9396 8b82 f08a 6096 ^......P......`.
    0x0020: 5018 4029 5527 0000 4745 5420 2f77 7061 P.@)U'..GET./wpa
    0x0030: 642e 6461 7420 4854 5450 2f31 2e31 0d0a d.dat.HTTP/1.1..
    0x0040: 436f 6e6e 6563 7469 6f6e 3a20 4b65 6570 Connection:.Keep
    0x0050: 2d41 6c69 7665 0d0a 4163 6365 7074 3a20 -Alive..Accept:.
    0x0060: 2a2f 2a0d 0a48 6f73 743a 2039 342e 3234 */*..Host:.94.24
    0x0070: 372e 3137 302e 3137 300d 0a0d 0a 7.170.170....

    As you can see, it also requests the IP as host.

    I hope this shows what you wanted to see.


    I wanted to thank everyone that has participated, all comments and
    theories have been very welcome!
    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Chris Davies@110:300/1.1 to All on Sat May 25 00:45:23 2013
    Reply-To: chris@roaima.co.uk

    In comp.os.linux.networking Sandman <mr@sandman.net> wrote:
    So, where would I start at for finding out who has done a /wpad.dat
    request and then add them to a firewall IP block list? Maybe that's
    the best route to go?

    It should be in your webserver log. Here's an example from mine:

    192.168.130.16 - - [24/May/2013:23:40:02 +0100] "GET /proxy.pac HTTP/1.1" 200 1485 "-" "-"

    and here's the log definition line for this vHost:

    CustomLog "|/usr/bin/cronolog /home/www/wpad/logs/%Y/%m/%d/public-access.log" combined

    If you're going to block by firewall rule I'd suggest you take a close
    look at fail2ban, which dos this kind of process very well indeed.

    For comparison, I've had over 4000 hits in my access log today. And this
    is from a fairly lightly loaded network with just a few PCs and
    servers. Windows 7 appears to be *very* noisy indeed (I think I might
    need to investigate the document expiry time to see if I can persuade
    W7 to cache the answer a little more often).

    Chris

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Roaima. Harrogate, North Yorkshire, UK (110:300/1.1@linuxnet)
  • From Richard Kettlewell@110:300/1.1 to All on Sat May 25 09:38:55 2013
    Sandman <mr@sandman.net> writes:
    [...]
    As you can see, it also requests the IP as host.

    That’s consistent with the IE6 behavior (as described by Wikipedia)
    which might have been copied by other software. Unfortunately, that
    means we still don’t know for sure what hostname the client thought it
    was connecting to.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Richard Kettlewell@110:300/1.1 to All on Sat May 25 09:45:50 2013
    Sandman <mr@sandman.net> writes:
    The only one of my clients that have wildcard:ed their domain to me is opennet.se, I'll have a talk to them. As far as I know, they don't
    have almost 5000 users on their LAN though.

    $ host wpad.opennet.se
    wpad.opennet.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Looks like a smoking gun to me...

    because even if what you postulate is a possible scenario, it would
    under no circumstances generate hundreds of thousands of requests - sometimes 30-40 per second from one single host. That's where the
    entire "misconfigured DNS" idea falls slightly apart, don't you agree?

    I mean, if I came here wondering about these wpad.dat requests I see
    now and then, then that would be a logical question. But I get about
    20-30 requests per second, every second. That just can't be due to a misconfigured wildcard DNS.

    Or do you think I am jumping to conclusions?

    More then one thing can be broken at once. In this case, the easiest
    thing to do is to stop wpad.opennet.se pointing at you. If that fixes
    it, then it’s time to speculate about why so many IP clients were
    involved. If it doesn’t, move on to the next theory. Checking for a ‘wpad’ subdomain for each of your customer domains would seem like the logical next step.

    --
    http://www.greenend.org.uk/rjk/

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: Anjou (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Sat May 25 09:54:19 2013
    In article <87ehcvqz81.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    The only one of my clients that have wildcard:ed their domain to me is opennet.se, I'll have a talk to them. As far as I know, they don't
    have almost 5000 users on their LAN though.

    $ host wpad.opennet.se
    wpad.opennet.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Looks like a smoking gun to me...

    Yes, but they are still no ISP, and all requests comes from swedish
    ISP's (like bahnhof.se, bredband2.se and such) and wpad.bahnhof.se
    doesn't point to me.

    because even if what you postulate is a possible scenario, it would
    under no circumstances generate hundreds of thousands of requests - sometimes 30-40 per second from one single host. That's where the
    entire "misconfigured DNS" idea falls slightly apart, don't you agree?

    I mean, if I came here wondering about these wpad.dat requests I see
    now and then, then that would be a logical question. But I get about
    20-30 requests per second, every second. That just can't be due to a misconfigured wildcard DNS.

    Or do you think I am jumping to conclusions?

    More then one thing can be broken at once. In this case, the easiest
    thing to do is to stop wpad.opennet.se pointing at you. If that fixes
    it, then it’s time to speculate about why so many IP clients were
    involved. If it doesn’t, move on to the next theory. Checking for a ‘wpad’ subdomain for each of your customer domains would seem like the logical next step.

    Yes, I have found none. I keep blocking them (over night, they didn't
    grow to more than about 5000 actually, I'm at 5163 right now)


    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Sandman@1:0/0 to All on Sat May 25 10:04:07 2013
    In article <87ehcvqz81.fsf@araminta.anjou.terraraq.org.uk>,
    Richard Kettlewell <rjk@greenend.org.uk> wrote:

    Sandman <mr@sandman.net> writes:
    The only one of my clients that have wildcard:ed their domain to me is opennet.se, I'll have a talk to them. As far as I know, they don't
    have almost 5000 users on their LAN though.

    $ host wpad.opennet.se
    wpad.opennet.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Looks like a smoking gun to me...

    Like I said in an earlier reply, Opennet is not an ISP they are a communication operator, which means they own citynets, so they don't
    have end customers on their opennet.se host.

    But looking at the domain names of all the hosts that I am currently
    blocking (5000+), there are some references to opennet, like <random>.opennet.bredband2.se" or "<random>.karlstad.bredband2.se" anD Karlstad is a Opennet city.

    So yes, this may very well be a smoking gun as you say.


    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Whiskers@110:300/1.1 to All on Sat May 25 16:52:32 2013
    On 2013-05-24, Sandman <mr@sandman.net> wrote:
    In article <519f999f$0$15903$e4fe514c@news2.news.xs4all.nl>,
    Casper H.S. Dik <Casper.Dik@OrSPaMcle.COM> wrote:

    host www.stadsnat.se
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Now, atlascms.se WAS a wildcard DNS, but isn't any longer.

    So when someone looked up wpad.stadsnet.se it was mapped
    to cluster.atlascms.se? That, I think, is the root
    of your problem.

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).
    actually. Maybe I should tell them to exempt wpad...

    [...]

    But

    $ host stadsnet.se
    stadsnet.se has address 46.30.211.54
    stadsnet.se mail is handled by 10 mx-cluster-a1.one.com.
    stadsnet.se mail is handled by 10 mx-cluster-a2.one.com.

    $ host wpad.stadsnet.se
    wpad.stadsnet.se has address 46.30.211.54

    $ host qwerty.stadsnet.se
    qwerty.stadsnet.se has address 46.30.211.54

    .... looks like a wildcard setting somewhere, to me.

    --
    -- ^^^^^^^^^^
    -- Whiskers
    -- ~~~~~~~~~~

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: is an alien concept (110:300/1.1@linuxnet)
  • From Roger@110:300/1.1 to All on Sat May 25 18:19:26 2013
    On Sat, 25 May 2013 15:52:32 +0100, Whiskers
    <catwheezel@operamail.com> wrote:

    host www.stadsnat.se

    $ host stadsnet.se

    It's stadsnAt.se, not stadsnEt.se.
    ^ ^
    --
    Roger

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: A noiseless patient Spider (110:300/1.1@linuxnet)
  • From J G Miller@110:300/1.1 to All on Sat May 25 19:22:02 2013
    On Saturday, May 25th, 2013, at 17:19:26h +0100, Roger explained:

    It's stadsnAt.se, not stadsnEt.se.

    So maybe Mr Sandman should, just to be certain,
    check his DNS setup does not have a typographical
    error of net rather than nat in his configuration files?

    find with a grep --color would do the job ...

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: A noiseless patient Spider (110:300/1.1@linuxnet)
  • From Sandman@1:0/0 to All on Sat May 25 19:41:15 2013
    In article <slrnkq1k1g.5go.catwheezel@ID-107770.user.individual.net>,
    Whiskers <catwheezel@operamail.com> wrote:

    On 2013-05-24, Sandman <mr@sandman.net> wrote:
    In article <519f999f$0$15903$e4fe514c@news2.news.xs4all.nl>,
    Casper H.S. Dik <Casper.Dik@OrSPaMcle.COM> wrote:

    host www.stadsnat.se
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Now, atlascms.se WAS a wildcard DNS, but isn't any longer.

    So when someone looked up wpad.stadsnet.se it was mapped
    to cluster.atlascms.se? That, I think, is the root
    of your problem.

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).
    actually. Maybe I should tell them to exempt wpad...

    [...]

    But

    $ host stadsnet.se
    stadsnet.se has address 46.30.211.54
    stadsnet.se mail is handled by 10 mx-cluster-a1.one.com.
    stadsnet.se mail is handled by 10 mx-cluster-a2.one.com.

    $ host wpad.stadsnet.se
    wpad.stadsnet.se has address 46.30.211.54

    $ host qwerty.stadsnet.se
    qwerty.stadsnet.se has address 46.30.211.54

    ... looks like a wildcard setting somewhere, to me.

    Only, you accidentally looked up "stadsnet" instead of "stadsnat" :)

    host stadsnat.se
    stadsnat.se has address 94.247.170.170

    That's my IP

    host www.stadsnat.se
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    host wpad.stadsnat.se
    Host wpad.stadsnat.se not found: 3(NXDOMAIN)


    And again - stadsnat.se is NOT an ISP.






    --
    Sandman[.net]

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne
  • From Casper H.S. Dik@1:0/0 to All on Sun May 26 12:12:41 2013
    Richard Kettlewell <rjk@greenend.org.uk> writes:

    More then one thing can be broken at once. In this case, the easiest
    thing to do is to stop wpad.opennet.se pointing at you. If that fixes
    it, then it’s time to speculate about why so many IP clients were
    involved. If it doesn’t, move on to the next theory. Checking for a >‘wpad’ subdomain for each of your customer domains would seem like the >logical next step.

    Right; by filtering wpad queries from all these clients also is a
    denial of service attack on all those clients, especially if you
    drop the packet but don't give them an answer.

    Casper

    --- MBSE BBS v0.95.15 (GNU/Linux-x86_64)
    * Origin: The Kofo BBS MBSE - telnet://fido1.kofobbs.ne