The Web Design Group

... Making the Web accessible to all.

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
> Robots.txt, Robots.txt
Jesica
post Jul 13 2017, 03:56 PM
Post #1





Group: Members
Posts: 1
Joined: 13-July 17
From: India
Member No.: 26,462



I want to know that robots.txt is necessary for website or not, please share your view ?
User is offlinePM
Go to the top of the page
Toggle Multi-post QuotingQuote Post
CharlesEF
post Jul 13 2017, 05:54 PM
Post #2


This is My Life
*******

Group: Members
Posts: 1,436
Joined: 27-April 13
From: Edinburg, Texas
Member No.: 19,088



As I understand it, robots.txt was designed as a file robots (crawlers) can read and NOT index any paths found in that file. The problem is that a lot of robot don't follow that rule.
User is offlinePM
Go to the top of the page
Toggle Multi-post QuotingQuote Post

Reply to this topicStart new topic
2 User(s) are reading this topic (2 Guests and 0 Anonymous Users)
0 Members:

 



- Lo-Fi Version Time is now: 21st November 2017 - 10:58 AM