Help - Search - Members - Calendar
Full Version: All My Sites Display x-robots-tag: noindex/nofollow
HTMLHelp Forums > Web Authoring > Web Site Functionality
mrspock
Hello all. I need some help. I'm not sure when or why it happened but all of my websites are displaying an x-robots-tag with noindex/nofollow attributes. This happens on every domain no matter if it is a wordpress site or a simple index.html file with one word. Somehow, somewhere, this tag is being inserted into every page and consequently seriously affecting my chances of having these pages indexed. In fact, many of these sites used to be in google's index but are not anymore. Here is an example of a site that used to be indexed but is not anymore:

http://www.jasonjonesdrummer.com/

What the heck is putting these tags into my pages? I looked at the source code and it isn't in there! I am with hostgator but their support was basically useless even though I suspect it is something they are doing. The SEO forums weren't interested in this problem so here I am. If I can't get an answer I guess I will start over with another host but I don't really want to go through that.

Thanks for any help!
pandy
How are you hosted?

Do you see the x-robots tag in the page you linked to and the pages linked from that page? Because I don't. If you see it it must be your browser doing it. But I have never heard of any malware injecting that Meta tag and I don't understand why it would. ohmy.gif
mrspock
I use hostgator on a shared plan. I have checked this using online tools as well and they all show these tags so it's not my browser. It is very frustrating and seemingly way beyond my technical knowledge. I guess I may need to start thinking about changing hosts.
pandy
View source at some other site that isn't your own to check if you see that Meta tag there too.

Try for example https://htmlhelp.com/ . I checked to be sure the tag isn't there.


pandy
QUOTE(mrspock @ Feb 28 2020, 09:55 PM) *

I use hostgator on a shared plan. I have checked this using online tools as well and they all show these tags so it's not my browser. It is very frustrating and seemingly way beyond my technical knowledge. I guess I may need to start thinking about changing hosts.


But they don't for me. The source shoud be printed below the validation result here. No fishy Meta tag.
https://htmlhelp.com/cgi-bin/validate.cgi?u...s&input=yes
https://validator.w3.org/check?uri=http%3A%...roup=0&ss=1

A real mystery, it seems!


Christian J
QUOTE(mrspock @ Feb 28 2020, 07:17 PM) *

I looked at the source code and it isn't in there!

I don't understand, how do you know there's an x-robots-tag if you can't see it? unsure.gif

pandy
Oops! My eyes must have slipped over that last paragraph. I thought you saw it in the source. Then I wonder the same thing as Christian.
pandy
I like mysteries. But I hate when they don't get a solution. rolleyes.gif

But it dawned on me that my first inkling probably was correct after all. It's there. In the HTTP headers.


CODE
HTTP Response

HTTP/1.1                   200 OK
Date:                      Sun, 01 Mar 2020 08:36:00 GMT
Server:                    Apache
Upgrade:                   h2,h2c
Connection:                Upgrade, close
Last-Modified:             Fri, 16 Mar 2018 16:25:24 GMT
Accept-Ranges:             bytes
Content-Length:            1747
Vary:                      Accept-Encoding
X-Robots-Tag:              noindex, nofollow
Content-Type:              text/html



If you are sure you haven't put it in an .htaccess file yourself you should contact your host and ask them to remove it. If your host doesn't want to play I assume you could override that header with a directive in your .htaccess file. I'm not sure about the syntax, but that should be easy to find.
mrspock
Thanks everyone for the help, much appreciated! I purposely deleted every line from my htacess file to test this and it is still showing up. Adding a robots.txt file didn't help either, so maybe I could try and override with htaccess but I don't know how that would go. I will search later to see if I can find the code I need to try it.

Update: I just now found one of these tags in another add-on domain that has nothing to do with any of the other domains affected. I don't have time now but will go through every add-on domain and check every single htaccess file to see if that fixes it. But this really shouldn't happen right? These domains and their respective htaccess files should operate independently of one another right?
pandy
You don't need to do that since .htaccess only works for the directory it's in and directories below that. It doesn't affect other directories. It must come from your host's config files.

I searched a little, and I *think* this would be enough, placed in a the top directory for each domain, provided you want everything to be indexed. Note that I've never tried this, so do your own googling also. It was hard find anything other than examples of how to exclude certain files of file types.

Either this
CODE
Header set X-Robots-Tag "index, follow"

or this, that should be the same as the above. At least I figured out that all is shorthand for index and follow.
CODE
Header set X-Robots-Tag "all"
Christian J
Could it be your host's Control Panel that somehow lets you enable this for all your sites?
pandy
That's an option, I guess.

Doesn't HostGator have a good rep? Send them an email or use support ticket or ask at their forum, whatever method they use do communicate. They should answer.
mrspock
Hello. I seem to have fixed it! Deleting those lines from that other domain's htaccess file fixed the problem for all my other domains as far as I can see. I know it doesn't make sense but shortly after I did that the tags disappeared.

As I said, I'm not a developer, just a guy with some websites but I think I have a theory. When connecting via ftp, it has always been the case that add-on domains have been placed within the public_html/www folders. However, I noticed a couple of domains I added more recently showed up in the root folder, technically placing them ABOVE the public_html folder, and this was the location of the offending domain (which was a personal site I didn't want indexed, hence the tags). So maybe any rules placed in these htaccess files would trickle down to the other domains?

I'm just so glad to have it fixed that I don't have the time or energy to test this right now. But I did notice that any new add-on domains are placed in the root, not in public_html so maybe somewhere along the way hostgator changed where they put these folders and it messed with my domains downstream (I've been with them since 2008). It's pretty annoying but I'm just so glad I don't have to change hosts right this minute. Though I will consider making a move eventually.

Their support person had no idea how to fix this and was searching the web for articles (all of which I had already read). And they basically refused to escalate this to a more knowledgeable tech so I am definitely looking to move my stuff whenever it is more convenient for me.

Anyway, thanks for the help, I appreciate your responses for such an odd problem from a new member. Take care everyone!
mrspock
One more related question. Do I need to do anything to get google interested again? I've registered these domains with them and verified ownership. Just not sure if I need to officially tell them I want these sites indexed once more or if ditching those tags is good enough.
pandy
Don't know. I've never done anything SEO-related. If there is a way to tell them, maybe that would speed up the process. But I don't know if there is.

I don't get the .htaccess thing. Did you have such a .htaccess file in a folder that was common for all your domains, sort of a top folder with all domains in it? Otherwise it doesn't make sense to me.

Oh well. Good it's fixed anyway.
Christian J
QUOTE(mrspock @ Mar 2 2020, 09:56 PM) *

One more related question. Do I need to do anything to get google interested again?

Just check in your web stats that googlebot is still visiting your sites. If it doesn't you could alway re-register your sites.

QUOTE
Just not sure if I need to officially tell them I want these sites indexed once more or if ditching those tags is good enough.

My guess is that once Google notices the changed robots directive it will act on it automatically.

mrspock
The pages are already being indexed so it is all fixed. Thanks again!
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2020 Invision Power Services, Inc.