Issues with indexing can affect individual pages, categories, or sections, or the entire website as a whole. Each case is unique and requires a detailed analysis of the reasons behind the problems. In this article, we will explore common reasons for pages not being indexed by Google and methods for addressing them

1. Site indexing prohibition

A common mistake that causes Google to not see or index a website is the prohibition of indexing the site or a section in the Robots.txt file or meta tag.

You can view the contents of the file by entering the following in your browser's address bar:
https://site.ru/robots.txt , where instead of 'site.ru,' you should specify your domain address. If the file contains:
User-agent: *
Disallow: /
In this case, your site is completely closed to search engines, and it won't be indexed because Google can't see it. In this scenario, you should fix the Robots.txt file and only block technical pages from indexing.

You can check the correctness of your file settings using the Search Console tool
https://www.google.com/webmasters/tools/robots-testing-tool

It's also important to check for the absence of the Noindex meta tag on website pages. To do this, you can open the page's source code through your browser and search for the phrase:
<meta name=”robots” content=”noindex”>
If this tag is present, the page will be excluded from the index.

For WordPress websites, it's common to forget to remove the 'visibility for search engines' option in the Settings → Reading section

For Tilda websites, you need to check the indexing status in the Settings → SEO → Noindex section.

You can check the indexing status of individual pages in the page settings → SEO tab

2. No access to pages

Often, a website is not indexed by Google because the search engine robot cannot access the site's pages, and therefore cannot scan them and send them for indexing. This typically happens when a website has a complex menu and lacks internal linking between pages, or when the menu is implemented using a script that the search engine robot cannot recognize.

You can programmatically check this by using free services like XENU , Site Analyzer, or paid services like Screaming Frog SEO Spider or Netpeak Spider.

The program will show the number of scanned pages, which you need to compare with the actual number of pages on your site. If you notice that the number of scanned pages is significantly less than the actual number of pages on your site (excluding sites with more than 100,000 pages, which may have a higher margin of error during scanning), you need to address this issue.

Steps to resolve this issue:

1. Adjust the website's structure.
2. Implement a menu with links to URL pages without using JavaScript.
3. Create a correct Sitemap file that includes all website pages.
4. Expedite the indexing of website pages using Fast Indexing

3. Incorrect redirection

Another reason why Google may not index website pages is incorrect redirection. This error is often encountered when moving a website from one CMS to another. Incorrectly configured 301 redirects can make some content inaccessible to both users and search engines. Chain redirections from one URL to another and circular redirection are considered serious errors.

You can perform a mass check of response codes and the presence of redirection on website pages using the service Converseo

4. Floating Server Access Error

For correct website indexing, the server should return a 200 response code. You can check the response code using services like be1 or Converseo.

If there are issues with the website's accessibility, causing 5xx errors, the pages of such a site will not be displayed in Google. In this case, you need to address the server administrators or investigate the causes of the error on the CMS side.

5. Incorrect Canonical Page Settings

Page canonicality is indicated through the "rel=canonical" attribute, and you can check which page is set as canonical in the website's source code.

Issues with canonicalization often arise in CMS platforms where product cards are duplicated in different categories, for example, in Bitrix. In some cases, a non-existent page is specified, or conflicts occur due to cyclic canonicalization rules.

To avoid indexing problems in this scenario, the correct use of the canonical attribute will be beneficial.

6. Indexing Errors in Google Search Console

After checking and resolving the major critical errors, it's important to go to the Google Search Console interface, navigate to the "Index" section, and look at the errors reported by the service. Typically, the service will show which pages Google is not indexing and the reasons why.

For analyzing pages with errors, there is an option to export up to 1000 URLs.

Conclusions

In this article, we've described only the most common indexing errors we encounter when working with websites, but this is by no means an exhaustive list. In each case, it's necessary to approach the issue of indexing and related problems individually.

Here's a brief checklist for assessing a website with poor indexing:

1. Indexing restrictions in the robots.txt file, meta name "robots," or CMS settings.
2. Site structure, page accessibility, and the sitemap.xml file.
3. Server response codes and proper redirection.
4. Server accessibility.
5. Canonicalization settings for pages.
6. Other errors indicated by Google Webmaster.

If you don't have time to conduct a thorough analysis of your website and identify the reasons for indexing issues in Google, you can contact us at any time, and we'll assist you in resolving this problem.