A cricket bat!
Twelve years, and four psychiatrists!
Four?
I kept biting them!
Why?
They said you weren't real.

Saturday, December 07

Rant

Dealing With Assholes Running Web Spiders On The Internet

Your site can deal with 20 simultaneous threads crawling the entire history of all your content right?  Right.  I mean, if your server crashes it's not our problem.  Also, fuck sending a clear identifier, we're just a Mac running Safari that just happens to be downloading the contents of 70,000 blogs.

Update: We were - and indeed, still are - getting hit by a torrent of garbage requests coming through a group of proxy servers run by an SEO company.  I don't know why, but they are now all getting 404.

Update 2: I killed that, and then I discovered we were also getting scanned by a total of 175 different servers at various hosting companies.  Those are all blocked at the firewall now, and don't even get the courtesy of a 404.

If you have a problem reaching the site - say it works on mobile but not at home, or vice versa - please let me know.  I will double-check the list of blocked IPs tomorrow and convert it to a smaller number of IP ranges.

Posted by: Pixy Misa at 08:38 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 187 words, total size 1 kb.

<< Page 1 of 1 >>
36kb generated in CPU 0.19, elapsed 0.9855 seconds.
49 queries taking 0.9109 seconds, 196 records returned.
Powered by Minx 1.1.6c-pink.
Using https / https://ai.mee.nu / 194