Netpond ™ - View Single Post

Fuckin Bill · 03-27-2007, 01:52 PM

I'm going to try to come back and post more things in this thread as I find/remember/update them. If you want to trade traffic, you need to know how to protect and work the traffic to your advantage. Hopefully this will grow into a nice list of useful information and tools to keep your streets clean.

I'm going to keep this thread closed and sticky. If you have any comments or questions, feel free to start a new thread. This one will be used just to add new information.

To start things off, here's some mod-rewrite code for your .htaccess files that will block most known bad bots and site ripper programs that identify themselves to your web server.

Code:

RewriteEngine  On
RewriteCond %{HTTP_USER_AGENT} Ants [NC,OR]
RewriteCond %{HTTP_USER_AGENT} attach [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Backweb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Bandit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} BlackWidow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Bot.mailto:craftbot@yahoo.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Buddy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CherryPicker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ChinaClaw [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Collector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Copier [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Crescent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Custo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DA.4.0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DA.5.0 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DA.5.3 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DIIbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DISCo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Download.Demon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Download.Ninja [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Downloader [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Drip [NC,OR]
RewriteCond %{HTTP_USER_AGENT} eCatch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EirGrabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EmailCollector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EmailWolf [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Express.WebPictures [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ExtractorPro [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EyeNetIE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} FileHound [NC,OR]
RewriteCond %{HTTP_USER_AGENT} FlashGet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GetLeft [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GetRight [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GetWeb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Gets [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Go.Zilla [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Go-Ahead-Got-It [NC,OR]
RewriteCond %{HTTP_USER_AGENT} gotit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Grabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GrabNet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Grafula [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HMView [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ia_archiver [NC,OR]
RewriteCond %{HTTP_USER_AGENT} IBrowse [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Image.Stripper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Image.Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Indy.Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} InterGET [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Internet.Ninja [NC,OR]
RewriteCond %{HTTP_USER_AGENT} JetCar [NC,OR]
RewriteCond %{HTTP_USER_AGENT} JOC [NC,OR]
RewriteCond %{HTTP_USER_AGENT} JustView [NC,OR]
RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} leech [NC,OR]
RewriteCond %{HTTP_USER_AGENT} LeechFTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} lftp [NC,OR]
RewriteCond %{HTTP_USER_AGENT} likse [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Lickity [NC,OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Link.Sleuth [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Magnet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Mag-Net [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Mass.Downloader [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Memo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Microsoft.URL [NC,OR]
RewriteCond %{HTTP_USER_AGENT} MIDown.tool [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Mirror [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Mister.Pix [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Navroad [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NearSite [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Net.Vampire [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NetAnts [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NetCaptor [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NetSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NetZIP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NICErsPRO [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Octopus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Offline [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PageGrabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Papa.Foto [NC,OR]
RewriteCond %{HTTP_USER_AGENT} pcBrowser [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PicaLoader [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PictureRipper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PicHunter [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Pockey [NC,OR]
RewriteCond %{HTTP_USER_AGENT} psbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Pump [NC,OR]
RewriteCond %{HTTP_USER_AGENT} RealDownload [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Reaper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Recorder [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ReGet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Siphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sitecheck.internetseer.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SiteSnagger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SmartDownload [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Snagger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Snake [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SpaceBison [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Stripper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SuperBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SuperHTTP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Surfbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} tAkeOut [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Teleport [NC,OR]
RewriteCond %{HTTP_USER_AGENT} IUPUI [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Vacuum [NC,OR]
RewriteCond %{HTTP_USER_AGENT} VoidEYE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web.Bandit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web.Image.Collector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web.Sucker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebAuto [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebCopier [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web.Downloader [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebEMailExtrac [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebGo.IS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebFetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebLeacher [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebReaper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebRipper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebSauger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Website.eXtractor [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Website.Quester [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Webster [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebStripper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebWhacker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebZIP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Whacker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Widow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WWWOFFLE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Xaldon.WebSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Zeus [NC] 
RewriteRule .*$ - [F]

This list has been checked and tested using http://www.wannabrowser.com/index.php

The list was compiled and tweaked from my own list of bots, and another that was posted recently by GTP Dave.

It is recommended that you check your own site with WannaBrowser as well to be sure it is functioning. I've seen several of these types of lists floating around that simply do nothing at all because coding is incorrect.

There are also a few that block the "wget" user agent. Doing this can be dangerous since many linux based scripts (Like just about all TGP scripts) use wget to download remote files. If you block wget from your gallery server, TGP's won't be able to grab your pics for thumbnailing, for example.