Printable Version of Topic
Click here to view this topic in its original format
HTMLHelp Forums _ Web Site Functionality _ All My Sites Display x-robots-tag: noindex/nofollow
Posted by: mrspock Feb 28 2020, 01:17 PM
Hello all. I need some help. I'm not sure when or why it happened but all of my websites are displaying an x-robots-tag with noindex/nofollow attributes. This happens on every domain no matter if it is a wordpress site or a simple index.html file with one word. Somehow, somewhere, this tag is being inserted into every page and consequently seriously affecting my chances of having these pages indexed. In fact, many of these sites used to be in google's index but are not anymore. Here is an example of a site that used to be indexed but is not anymore:
http://www.jasonjonesdrummer.com/
What the heck is putting these tags into my pages? I looked at the source code and it isn't in there! I am with hostgator but their support was basically useless even though I suspect it is something they are doing. The SEO forums weren't interested in this problem so here I am. If I can't get an answer I guess I will start over with another host but I don't really want to go through that.
Thanks for any help!
Posted by: pandy Feb 28 2020, 03:51 PM
How are you hosted?
Do you see the x-robots tag in the page you linked to and the pages linked from that page? Because I don't. If you see it it must be your browser doing it. But I have never heard of any malware injecting that Meta tag and I don't understand why it would. 
Posted by: mrspock Feb 28 2020, 03:55 PM
I use hostgator on a shared plan. I have checked this using online tools as well and they all show these tags so it's not my browser. It is very frustrating and seemingly way beyond my technical knowledge. I guess I may need to start thinking about changing hosts.
Posted by: pandy Feb 28 2020, 03:57 PM
View source at some other site that isn't your own to check if you see that Meta tag there too.
Try for example https://htmlhelp.com/ . I checked to be sure the tag isn't there.
Posted by: pandy Feb 28 2020, 04:00 PM
I use hostgator on a shared plan. I have checked this using online tools as well and they all show these tags so it's not my browser. It is very frustrating and seemingly way beyond my technical knowledge. I guess I may need to start thinking about changing hosts.
But they don't for me. The source shoud be printed below the validation result here. No fishy Meta tag.
https://htmlhelp.com/cgi-bin/validate.cgi?url=http%3A%2F%2Fwww.jasonjonesdrummer.com%2F&warnings=yes&input=yes
https://validator.w3.org/check?uri=http%3A%2F%2Fwww.jasonjonesdrummer.com%2F&charset=%28detect+automatically%29&doctype=Inline&group=0&ss=1
A real mystery, it seems!
Posted by: Christian J Feb 28 2020, 04:25 PM
I looked at the source code and it isn't in there!
I don't understand, how do you know there's an x-robots-tag if you can't see it?
Posted by: pandy Feb 28 2020, 04:41 PM
Oops! My eyes must have slipped over that last paragraph. I thought you saw it in the source. Then I wonder the same thing as Christian.
Posted by: pandy Mar 1 2020, 03:31 AM
I like mysteries. But I hate when they don't get a solution. 
But it dawned on me that my first inkling probably was correct after all. It's there. In the HTTP headers.
HTTP/1.1 200 OK
Date: Sun, 01 Mar 2020 08:36:00 GMT
Server: Apache
Upgrade: h2,h2c
Connection: Upgrade, close
Last-Modified: Fri, 16 Mar 2018 16:25:24 GMT
Accept-Ranges: bytes
Content-Length: 1747
Vary: Accept-Encoding
X-Robots-Tag: noindex, nofollow
Content-Type: text/html
If you are sure you haven't put it in an .htaccess file yourself you should contact your host and ask them to remove it. If your host doesn't want to play I assume you could override that header with a directive in your .htaccess file. I'm not sure about the syntax, but that should be easy to find.
Posted by: mrspock Mar 1 2020, 11:56 AM
Thanks everyone for the help, much appreciated! I purposely deleted every line from my htacess file to test this and it is still showing up. Adding a robots.txt file didn't help either, so maybe I could try and override with htaccess but I don't know how that would go. I will search later to see if I can find the code I need to try it.
Update: I just now found one of these tags in another add-on domain that has nothing to do with any of the other domains affected. I don't have time now but will go through every add-on domain and check every single htaccess file to see if that fixes it. But this really shouldn't happen right? These domains and their respective htaccess files should operate independently of one another right?
Posted by: pandy Mar 1 2020, 12:43 PM
You don't need to do that since .htaccess only works for the directory it's in and directories below that. It doesn't affect other directories. It must come from your host's config files.
I searched a little, and I *think* this would be enough, placed in a the top directory for each domain, provided you want everything to be indexed. Note that I've never tried this, so do your own googling also. It was hard find anything other than examples of how to exclude certain files of file types.
Either this
or this, that should be the same as the above. At least I figured out that all is shorthand for index and follow.
Posted by: Christian J Mar 1 2020, 12:49 PM
Could it be your host's Control Panel that somehow lets you enable this for all your sites?
Posted by: pandy Mar 1 2020, 12:58 PM
That's an option, I guess.
Doesn't HostGator have a good rep? Send them an email or use support ticket or ask at their forum, whatever method they use do communicate. They should answer.
Posted by: mrspock Mar 2 2020, 02:24 PM
Hello. I seem to have fixed it! Deleting those lines from that other domain's htaccess file fixed the problem for all my other domains as far as I can see. I know it doesn't make sense but shortly after I did that the tags disappeared.
As I said, I'm not a developer, just a guy with some websites but I think I have a theory. When connecting via ftp, it has always been the case that add-on domains have been placed within the public_html/www folders. However, I noticed a couple of domains I added more recently showed up in the root folder, technically placing them ABOVE the public_html folder, and this was the location of the offending domain (which was a personal site I didn't want indexed, hence the tags). So maybe any rules placed in these htaccess files would trickle down to the other domains?
I'm just so glad to have it fixed that I don't have the time or energy to test this right now. But I did notice that any new add-on domains are placed in the root, not in public_html so maybe somewhere along the way hostgator changed where they put these folders and it messed with my domains downstream (I've been with them since 2008). It's pretty annoying but I'm just so glad I don't have to change hosts right this minute. Though I will consider making a move eventually.
Their support person had no idea how to fix this and was searching the web for articles (all of which I had already read). And they basically refused to escalate this to a more knowledgeable tech so I am definitely looking to move my stuff whenever it is more convenient for me.
Anyway, thanks for the help, I appreciate your responses for such an odd problem from a new member. Take care everyone!
Posted by: mrspock Mar 2 2020, 03:56 PM
One more related question. Do I need to do anything to get google interested again? I've registered these domains with them and verified ownership. Just not sure if I need to officially tell them I want these sites indexed once more or if ditching those tags is good enough.
Posted by: pandy Mar 2 2020, 04:43 PM
Don't know. I've never done anything SEO-related. If there is a way to tell them, maybe that would speed up the process. But I don't know if there is.
I don't get the .htaccess thing. Did you have such a .htaccess file in a folder that was common for all your domains, sort of a top folder with all domains in it? Otherwise it doesn't make sense to me.
Oh well. Good it's fixed anyway.
Posted by: Christian J Mar 2 2020, 05:42 PM
One more related question. Do I need to do anything to get google interested again?
Just check in your web stats that googlebot is still visiting your sites. If it doesn't you could alway re-register your sites.
My guess is that once Google notices the changed robots directive it will act on it automatically.
Posted by: mrspock Mar 11 2020, 01:13 PM
The pages are already being indexed so it is all fixed. Thanks again!
Powered by Invision Power Board (http://www.invisionboard.com)
© Invision Power Services (http://www.invisionpower.com)