Block Bots by User Agent String - Revision history

Brian: /* Via BrowserMatch */

2020-10-27T01:26:19Z

‎Via BrowserMatch

Brian: /* Via BrowserMatch */

2020-10-27T01:24:10Z

‎Via BrowserMatch

Brian: /* Via mod_rewrite */

2020-10-27T01:23:40Z

‎Via mod_rewrite

Brian: /* Block the BOT */

2018-04-13T11:17:01Z

‎Block the BOT

Brian: /* Block the BOT */

2016-01-29T17:14:14Z

‎Block the BOT

Brian: Created page with "==How to block a bot by User Agent Sting== Do you have those bandwidth hogging bots as much as Phil and I do? Did you know you can block them in your .htaccess file? ===B..."

2014-01-06T19:19:35Z

Created page with "==How to block a bot by User Agent Sting== Do you have those bandwidth hogging bots as much as Phil and I do? Did you know you can block them in your .htaccess file? ===B..."

New page

==How to block a bot by User Agent Sting==

Do you have those bandwidth hogging bots as much as Phil and I do? Did you know you can block them in your .htaccess file?

===Block the BOT===
Let block the most annoying bot on the internet - Baidu spider

vim .htaccess

Now Add the following to block

RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Baidu [NC]
RewriteRule ^.*$ - [F,L]

Thats it save the file, and you are now blocking the Baidu spider

===Testing to see if its blocked===

One way to do this is to use curl

curl -I http://www.briansnelson.com -A "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

Now you will get a 403 Forbidden

HTTP/1.1 403 Forbidden
Date: Mon, 06 Jan 2014 19:18:11 GMT

@@ Line 31: / Line 31: @@
   BrowserMatchNoCase "Facebot" bots
   BrowserMatchNoCase "DotBot" bots
   Order Allow,Deny

@@ Line 20: / Line 20: @@
 ===Via BrowserMatch===
-−
   BrowserMatchNoCase "Baiduspider" bots
   BrowserMatchNoCase "HTTrack" bots
@@ Line 34: / Line 35: @@
   Allow from ALL
   Deny from env=bots
 That's it save the file, and you are now blocking the Baidu spider

@@ Line 12: / Line 12: @@
 ===Via mod_rewrite===
   RewriteEngine On
   RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR]
   RewriteCond %{HTTP_USER_AGENT} Baidu [NC]
   RewriteRule ^.*$ - [F,L]
 ===Via BrowserMatch===

@@ Line 8: / Line 8: @@
   vim .htaccess
-Now Add the following to block
+Now Add the following to block bad bots
   RewriteEngine On
@@ Line 14: / Line 16: @@
   RewriteCond %{HTTP_USER_AGENT} Baidu [NC]
   RewriteRule ^.*$ - [F,L]
 That's it save the file, and you are now blocking the Baidu spider