It’s an interesting story that I first came across over 2 years ago.
I once wrote an article about SEO for Magento 2. It was one of the first guides for the Magento 2.0 platform, and it became very popular with merchants, marketers and SEOs.
Even now, the post is being ranked in a featured snippet above the official Magento website:
One day I needed to send a link to this post to my client. I googled it… and the post didn’t show up on the 1st page. Or on any other page. It didn’t even show up for my name + topic.
It was weird so I started digging into it.
All this has led to this story and the experiment I did back then and revised again before writing this post.
Let’s start, shall we?
Table of Contents
Why a Page Is Not In Google Index?
There are many possible reasons why a page might not be indexed by Google, including:
- A page is set to noindex
- A page is not indexed as it has thin or duplicate content
- The whole website is de-indexed because of technical or quality issues
In my case, the website was healthy, all other pages were ranking as usual.
The page I was looking for was of very high quality and used to rank in the top 3.
So the only possible reason was… the page being noindexed?
But why? Why would someone noindex just one blog post which is performing really well?
That’s what I started looking into.
The Page is Indexed… Right? Right?!
As it turned out, the answer was ‘“yes” and “no” (maybe it’s just Schrödinger’s page!).
The meta robots tag in the <head> section of the page was set to “index,follow”.
But something caught my eye: there was this phase on the page saying:
When I checked this part in the page source, it read like this:
So META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW” was missing from the copy seen visually on the page but available in the page source.
It could only mean one thing:
Google picked up the meta robots tag added to the <body> of the page instead of the <head> section of this page.
So weird and interesting!
The formatting used to be correct. But it seems that after some re-design changes, the text got reformatted and instead of displaying META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW” visually, this part became part of the source code of this page.
I reached out to the blog’s team and they fixed it quickly. After that, the page was re-indexed by Google and went back to the top rankings.
The Experiment with Noindex in the Page <body>
I would not be a tech SEO geek if I didn’t try to replicate this finding with my experiment.
I did it a while ago. It worked.
And I did it one more time before writing this post. It worked again.
Here is what I did and the results it yielded.
Become an Advanced Technical SEO Professional
(even if you don’t have coding skills or a development background). Join Tech SEO Pro today>>
Before doing the experiment, I also asked people on my social media one question:
Most people answered that the noindex meta robots tag in the page <body> won’t work. Let’s check!
Stage 1: indexing
I created a page on my website called “A page to test noindex” (creative, right?).
The page got indexed:
Stage 2: Adding noindex to the <body>
Then I added the noindex meta robots tag to the <body> of the page in one of the paragraphs using an HTML editor:
If you look at the source code of the page (and rendered HTML), you will see that the meta robots tag in the <head> section allows indexing of this page:
But then the tag in the body looks like this:
Stage 3: Checking how if noindex placed in the <body> seen by tools
Since the experiment page got noindexed after I added the meta robots noindex tag to the body, it’s evident that Google picked it up.
But what did the tools show?
Google Search Console URL Inspection tool picks up the noindex in the <body>:
The Mobile-friendly testing tool also picks up the noindex tag:
The noindex tag is also seen in the rendered HTML:
Screaming Frog noticed the noindex as well:
Stage 4: Re-indexing the page
To finilize the experiment, I decided to make the last step: remove the noindex tag from the body and see if the page will be added to Google index.
To do that, I replaced the HTML-edited copy with the visually edited copy. So that on the frontend it looks this way:
And on the backend, it looks like this:
The result? See for yourself:
My Conclusions after The Noindex Experiment
Based on the evidence, Googlebot reads and respects the meta robots tag found in the page <body>. It goes against the official Google guidelines that say:
What it means for SEOs:
- A page can be noindexed even if the meta robots noindex tag is placed outside the <head> section.
When you see that a page is noindexed but can’t find the tag in the head section, look into the rest of the page.
I don’t think that it’s a frequent scenario. But it’s good to be well-equipped if you come across something like this in the future.
- You can potentially add the noindex tag in the page body to quickly noindex the page
I do not recommend using the meta robots noindex tag in the page body regularly or as a replacement for the recommended tag placement in the page <head>.
But if there’s a rare situation when you need to get a page noindexed ASAP and you rely on someone else to implement it and have to wait, you can use this method by adding the tag to the page <body> in HTML.
Once the tag is implemented properly – in the <head> section – remove yours from the <body>.
Again, I don’t recommend it in general, so use your common sense (or you can just use the Remove URL tool in GSC).
Become an Advanced Technical SEO Professional
(even if you don’t have coding skills or a development background). Join Tech SEO Pro today>>
Back to You
Technical SEO is fascinating. And as you can see, it can it’s important to know it even if you want to be a well-versed SEO professional.
P.S. This page is allowed for indexing 🙂
After 10+ years in SEO, I founded MarketingSyrup Academy where I teach smart SEOs. Over 500+ people have gone through my courses, including SEO Challenge and Tech SEO Pro.
I’m also a creator of the SEO Pro extension with 30K active users worldwide.
