I have an old site up, that I'm assuming can't be indexed. I've registered with Google and Yahoo (and one other that I forget). Here's the link to it:

http://headbodyandsoul.110mb.com/index.htm

I don't understand the problem. It's got low key-word density, good DOCTYPE (I think). It does have a lot of links on one page. I've read up on
SEO and searched everywhere. Can someone give me a hint? It won't even show up when I google the URL except through another site (cubestat or something like that). Please help Thanks in advance

-Stubbs

How long has the site been up, and how long ago did you register? Did you keep other content on it before? Have you seen any visit from e.g. Googlebot in the web stats?


I don't see anything obviously wrong with it. A couple of validator errors but nothing serious AFAIK.

Links from other sites are usually good (not the ones from this forum though, SEs are instructed to not follow those).

Hey, thanks for the response and for taking the time to look at this. I registered my site about 5-6 months ago, right after I put it up. I don't use web stats, so I don't know about the bots. My web host 110MB refers me to another site for stats, and at the time I was weary of letting them access my site. I should probably sign up for something, though, right? I briefly looked into Google Analytics, but want to revamp my site first (new look, valid XHTML and CSS, etc.).

In your opinion, if I trade links with similar sites and signed up for some kind of stats program, would I be in better shape? Thanks again for the quick response. This is my third site, and none of them show up, which is very frustrating.

Maybe but probably not. Do you know if other sites link to you?

BTW you don't have to register with real search engines like Google, Yahoo! and so on. Directories require submission though and search engines spider (some) diectories...

Trying to submit your site to dmoz may be a good idea, since it's definitely spidered by Google et al. Take care to follow their guidelines and to choose the right section. Your submission will be reviewed by humans and they can be grumpy.

That's odd. Are they all hosted by 100mb?

The only thing I notice is that http://headbodyandsoul.110mb.com/robots.txt redirects to a site that looks pretty spammy.

Jesus. Seems to be a persistent problem on 100mb...
http://bowlstaradale.110mb.com/robots.txt
http://rendaextra10.110mb.com/robots.txt

Obviously those sites were indexed by Google though. I also found sites hosted by them with a normal robots.txt, but there are more with the spammy one (judging from Google's first result page).

Well, that's quite a big Thing, isn't it? Presumably this would stop any honest bot in its tracks. Can't quite see the point, though, since only nerd humans go looking at robots.txt anyway. Hmm, but 00mb.com redirects to http://100mb.ru/ ... Perhaps their business model somehow depends on sites not being in search engines. Presumably this is "free", so what do you expect?


*** Sorry! Ignore the last bit - I typed 100 instead of 110. But I don't believe there is ever going to be a "deal" where some company just provides what you need for no reason other than feeling warm about it. They make some obviously false claims, such as "And it's further supported by a rock-solid company with unlimited funding." This is unlimited, meaning until it reaches the limit, right?

I'd expect less features and reliability for free, but not to have my site more or less killed.

My first two were with Lycos. Only this one is with 110MB. And I'm pretty sure that NO other sites link to mine. I'll work on that (find similar sites and trade maybe). Do you ever get in trouble by linking to a site that may not want you linking to it? I can't imaging that being a problem (they want traffic too, right?). I do have my address on a couple of profiles (like deviant art).

On a side note, Google analytics isn't reading my content either, but they were able to read an xml sitemap I uploaded, as well as a meta data verification line (have no idea what that does). I did read a few posts in Google's analytics forum that it may take a few hours (a few had checked the next morning and found it working).

So, is 110MB maybe not the safest of choices? I had Lycos before, but they put ads everywhere. I guess you get what you pay for

I've learned a lot today. Thanks for looking at this guys. I'm going to read about robot.txt now. If I come up with something I'll post it here.

If it is as we think, the spam link in your robots.txt may stop SEs from indexing your site. Free hosts may be unreliable in many ways, they may go down often, be slow, go away overnight... but I've never heard of something like this before. I wouldn't stay there.

OK. I'll see what I can do about it. I'm going to contact them first, but I'll probably be moving soon. Thanks for the heads up.

It does send you to http://www.110mb.com/404.php (despite returning a 302 header, shouldn't it return a 404?) which in turn redirects to spam pages (same thing happens with other missing pages).

Will this really stop spidering bots? I imagine it should work if the 404 page really sent a 404 header?

I didn't notice that. Then it should be enough to upload a real robots.txt. Fact remains that with that setup, the host should have done that.

http://www.robotstxt.org/

The requested document (which is http://headbodyandsoul.110mb.com/robots.txt in this case) should return 404 if it is Not Found.

A lot of sites are misconfigured, and instead redirect to a 404.html or 404.php or 404.whatever document, and that "404" document returns 200 0K.

Redirecting to a "404" document that then redirects to another site's spam page is a new one to me.

OK, I have an update on this. I'm now indexed with Google, and these are the steps I took:

1. ) I added a robots.txt file , 2. ) I added an XML sitemap, 3. ) I registered with Google Analytics, 4. ) I resubmitted my site (google and yahoo) for indexing. 4-6 weeks later it was indexed. I didn't get any sites linking to mine, other than this forum thread (which shouldn't count). I'm assuming that one of the steps above is what did it.

(edit) I did register with dmoz as well: http://www.dmoz.org/ (/edit)

However, instead of hitting a 404 error page, though, it does still redirect. I found out that you have to pay ($9) to use an .htaccess file. This is quite intentional. The moderators say in the forums that this is how they make money. I'm not trying to complain (the site is free after all), but this is how it works. Hopefully this takes some of the mystery out of for anyone reading this. Thanks again to all the people who looked at this and gave me advice. Much appreciated!

This is the robots.txt file I used (I don't have anything private or any real scripting that I want to hide or anything like that):

User-Agent:
*Allow: /

Good it works. But what redirects? Not the robots.txt, I tried that.

I found it by typing an address in wrong (example): http://headbodyandsoul.110mb.com/missing.html

Ah, yes. But that won't hurt you much. Your reputation maybe, but that's another thing.

Yeah, no worries. That went down the drain years ago!

I imagined that much.