Saturday, December 07

Rant

Dealing With Assholes Running Web Spiders On The Internet

Your site can deal with 20 simultaneous threads crawling the entire history of all your content right?  Right.  I mean, if your server crashes it's not our problem.  Also, fuck sending a clear identifier, we're just a Mac running Safari that just happens to be downloading the contents of 70,000 blogs.

Update: We were - and indeed, still are - getting hit by a torrent of garbage requests coming through a group of proxy servers run by an SEO company.  I don't know why, but they are now all getting 404.

Update 2: I killed that, and then I discovered we were also getting scanned by a total of 175 different servers at various hosting companies.  Those are all blocked at the firewall now, and don't even get the courtesy of a 404.

If you have a problem reaching the site - say it works on mobile but not at home, or vice versa - please let me know.  I will double-check the list of blocked IPs tomorrow and convert it to a smaller number of IP ranges.

Posted by: Pixy Misa at 08:38 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 187 words, total size 1 kb.




Apple pies are delicious. But never mind apple pies. What colour is a green orange?




44kb generated in CPU 0.02, elapsed 0.1844 seconds.
56 queries taking 0.166 seconds, 287 records returned.
Powered by Minx 1.1.6c-pink.