Comments on: abusive crawler http://cleverhack.com/2006/08/27/abusive-crawler/ A Blog About Technology, Search Engine Optimization (SEO), Internet Marketing And More. Sat, 10 Jan 2009 00:32:39 +0000 http://wordpress.org/?v=2.0.2 by: joy http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24545 Thu, 31 Aug 2006 03:51:31 +0000 http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24545 Thad, Well, if you think about it this way, every server has finite resources. So, if you're running a Webserver and a database and a mail MTA, server processes add up. In addition, I'm on a shared server, so I'm sure my hosting provider would rather have me ban a bad bot than have it not only stress my site, but potentially stressing the rest of the server. <a href="http://cleverhack.com/2006/08/30/my-wp-corrupted/" rel="nofollow">Read here for what can happen in a case like that</a>. Thad,

Well, if you think about it this way, every server has finite resources. So, if you’re running a Webserver and a database and a mail MTA, server processes add up.

In addition, I’m on a shared server, so I’m sure my hosting provider would rather have me ban a bad bot than have it not only stress my site, but potentially stressing the rest of the server. Read here for what can happen in a case like that.

]]> by: Thad http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24539 Wed, 30 Aug 2006 03:20:47 +0000 http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24539 OK. The only web content I've ever done has been static, so I don't have any feel for how much load a database-driven site can create. I'll believe it when you say an incessant request every 4 seconds could contribute to hosing a database-driven site. I mean, after all, there were web servers for 8-bit embedded processors back in '99, devices that are fast only when compared to geological measures. I looked into this for my company at that time. (In that application, the content would have been dynamically generated, though not from a database.) Given what an eternity 4 seconds is for even a lowly 1GHz processor these days.... So, note to self: re-write spam-bot so it pauses for 5 seconds.... beta test on cleverhack to see if it passes muster. ;-) OK. The only web content I’ve ever done has been static, so I don’t have any feel for how much load a database-driven site can create. I’ll believe it when you say an incessant request every 4 seconds could contribute to hosing a database-driven site.

I mean, after all, there were web servers for 8-bit embedded processors back in ‘99, devices that are fast only when compared to geological measures. I looked into this for my company at that time. (In that application, the content would have been dynamically generated, though not from a database.) Given what an eternity 4 seconds is for even a lowly 1GHz processor these days….

So, note to self: re-write spam-bot so it pauses for 5 seconds…. beta test on cleverhack to see if it passes muster.

;-)

]]> by: joy http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24536 Wed, 30 Aug 2006 01:23:58 +0000 http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24536 Thad, Because too many requests per minute will take down the site. This is especially true of database driven sites like this one. I don't need a non-legit source of traffic to add to the load. Thad, Because too many requests per minute will take down the site. This is especially true of database driven sites like this one.

I don’t need a non-legit source of traffic to add to the load.

]]> by: Thad http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24535 Wed, 30 Aug 2006 01:15:27 +0000 http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24535 If I may ask, what's so abusive about once every 4 seconds? In all seriousness, that sounds like a request rate that a Commodore 64 ought to be able to keep up with. I mean, sure, spam harvesting = teh suck, but I'm referring only the crawler's behavior itself. If I may ask, what’s so abusive about once every 4 seconds? In all seriousness, that sounds like a request rate that a Commodore 64 ought to be able to keep up with.

I mean, sure, spam harvesting = teh suck, but I’m referring only the crawler’s behavior itself.

]]> by: Don McArthur http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24509 Sun, 27 Aug 2006 22:54:02 +0000 http://cleverhack.com/2006/08/27/abusive-crawler/#comment-24509 Thanks for sharing. In exchange, here's my current /etc/hosts.deny sshd collection: http://www.mcarthurweb.com/archive.php?item=224 Thanks for sharing. In exchange, here’s my current /etc/hosts.deny sshd collection:

http://www.mcarthurweb.com/archive.php?item=224

]]>