It was a bad day. A lot of bad stuff happened. And I'd love to forget it all. But I don't. Not ever. Because this is what I do. Every time, every day, every second, this: On five, we're bringing down the government.

Saturday, December 07

Rant

Dealing With Assholes Running Web Spiders On The Internet

Your site can deal with 20 simultaneous threads crawling the entire history of all your content right?  Right.  I mean, if your server crashes it's not our problem.  Also, fuck sending a clear identifier, we're just a Mac running Safari that just happens to be downloading the contents of 70,000 blogs.

Update: We were - and indeed, still are - getting hit by a torrent of garbage requests coming through a group of proxy servers run by an SEO company.  I don't know why, but they are now all getting 404.

Update 2: I killed that, and then I discovered we were also getting scanned by a total of 175 different servers at various hosting companies.  Those are all blocked at the firewall now, and don't even get the courtesy of a 404.

If you have a problem reaching the site - say it works on mobile but not at home, or vice versa - please let me know.  I will double-check the list of blocked IPs tomorrow and convert it to a smaller number of IP ranges.

Posted by: Pixy Misa at 08:38 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 187 words, total size 1 kb.

<< Page 1 of 1 >>
37kb generated in CPU 0.0162, elapsed 0.4022 seconds.
49 queries taking 0.3926 seconds, 204 records returned.
Powered by Minx 1.1.6c-pink.
Using https / https://ai.mee.nu / 202