robots.txt usage and examples, lets compile a robots.txt example thread |
robots.txt usage and examples, lets compile a robots.txt example thread |
askapache |
Feb 7 2007, 04:58 AM
Post
#1
|
Newbie Group: Members Posts: 18 Joined: 29-December 06 From: USA Member No.: 1,407 |
I am curious to see some example uses and implementations of robots.txt.. specifically implementations and the reasons behind them for increasing SEO.
The robots.txt I am using for WordPress 2.1 is based on the example at SEO with robots.txt: CODE User-agent: * # disallow files in /cgi-bin Disallow: /cgi-bin/ Disallow: /comments/ Disallow: /z/j/ Disallow: /z/c/ # disallow all files ending in .php Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.txt$ #disallow all files in /wp- directorys Disallow: /wp-*/ # disallow all files with ? in url Disallow: /*? Disallow: /stats* Disallow: /dh_ Disallow: /about/legal-notice/ Disallow: /about/copyright-policy/ Disallow: /about/terms-and-conditions/ Disallow: /about/feed/ Disallow: /about/trackback/ Disallow: /contact/ Disallow: /tag Disallow: /docs* Disallow: /manual* Disallow: /category/uncategorized* Basically this helps get rid of duplicate content, low-quality content, css, javascript, php, etc.. but does allow search engines to read the articles, find images, find pdfs, etc. Also, I know the wildcard in my robots.txt works for googles bots, but do you konw if it works for other bots? Anyone else have improvements or other robots.txt examples? |
askapache |
Mar 30 2007, 05:14 PM
Post
#2
|
Newbie Group: Members Posts: 18 Joined: 29-December 06 From: USA Member No.: 1,407 |
There has been a lot of talk on the net lately about robots.txt for seo, heres a couple more robots.txt articles.
Create a robots.txt file and increase your search engine rankings WordPress optimized robots.txt |
Lo-Fi Version | Time is now: 26th April 2024 - 08:01 PM |