An important aspect of website indexing is the discrepancy between the nominal number of pages and the actual number of indexed pages. Search engines, for various reasons, do not index all pages of a website or, conversely, add unwanted ones to the index. As a result, it's practically impossible to find a site with 100% correctly indexed static pages.

For e-commerce websites, service sites, and other commercial websites, the absence of pages in search results can result in lost revenue.

Therefore, it's crucial to know how many pages are indexed, continuously analyze the indexing of website pages in Google, identify missing pages, errors, and redundant pages, and take action on them.

The Stages of Indexing Websites and Pages in Google
Methods for Checking Indexing in Google
Verification of the Actual Number of Pages on the Site
Submitting Pages for Indexing

The Stages of Indexing Websites and Pages in Google

1. Crawling: Google's search robots continuously scan the pages of all websites and store them in their database.
Each website is assigned a maximum limit of pages for indexing, known as the "crawling budget." This limit depends on various factors, such as site loading speed, the ratio of valuable to junk pages, the presence of duplicate content, the correctness of site navigation and internal linking, and so on.

2. Downloading: A separate Google robot downloads pages and analyzes their elements and document areas, including meta tags, images, text, videos, markup, and other data.

3. Verification: Each page is checked for compliance with various requirements:

• including accessibility and absence of indexing restrictions,
• content quality and structure,
• the absence of prohibited SEO techniques on the page,
• mobile-friendliness,
• and so on.

4. Indexing: If a page successfully passes the verification process, it is added to Google's index.


When is it necessary to check indexing?

• After creating a new project
• After migrating to a new CMS or a new domain
• After changing the site's structure
• After adding new pages and sections
• In any case when new URLs are modified or added to the site

Methods for Checking Indexing in Google

1. Using the "SITE:" Operator
If you don't have access to the website or webmaster tools, the simplest method is to use the "site" operator. In the search bar, enter "site:https://website-address/".
This will show you whether your site is indexed in Google and the number of indexed pages. Keep in mind that if you have many pages, the information may not be entirely accurate.

To check the indexing of a specific page, you can use the "site:https://page-url" operator, and to check the indexing of a category or section, you can use "site:https://website-url/section/*".

2. Using Google Search Console
Google Search Console is a webmaster tool provided by Google. To access this resource, you need to register in the system using a Gmail account and verify your ownership of the website. The number of indexed pages can be found in the "Index" section under "Pages." You can view the list of pages by clicking on the "See Index Coverage" link.

If you need to determine whether a specific page is indexed in Google, you should click on the "URL Inspection" menu item and enter the URL of the page. After processing, you will see all the information about that particular page, and there will be a "Request Indexing" button available for submission to the indexing queue.

Verification of the Actual Number of Pages on the Site

For conducting a thorough analysis of website indexing, it's important to understand the actual number of pages present on the site. Of course, the simplest way is to access the website's administrative panel and check the number of static pages, product cards, categories, and service pages. However, in some cases, this approach may not be suitable, and that's when you can use data scraping tools.

You can use free programs like XENU , Site Analyzer, or paid ones like Screaming Frog SEO Spider or Netpeak Spider. These programs operate on a common principle. You input the website's URL and wait for the scanning to finish. As a result, the program will display the total number and a list of all available URLs on the website

Checking indexing in Screaming Frog through connected Search Console

The advantage of this method is that with a single scan, you can see both the actual list of website pages and whether a page is indexed in Google. After exporting the results, you can easily obtain a list of non-indexed pages in one step.

For websites with up to 500 URLs, you can use the free version of the program, but for sites with a larger number of pages, a license is required.

To work with it, you need to set up integration between Screaming Frog (version 18 and above) and a Google Search Console account with the appropriate access rights for the selected website.
In the menu, go to Configuration -> API Access -> Google Search Console and configure the following:

1. Account Information - Add your Google account.
2. Search Analytics - Check the 'Crawl New URLs Discovered In Google Search Console' checkbox.
3. URL Inspection - Check the 'Enable URL Inspection' checkbox.

After running the parsing process, the report regarding the presence of pages in Google's index will be available on the Search Console tab under the Summary column.

"URL is on Google" - the page is present in Google's index.
"URL is not on Google" - the page is absent from Google's index, and it is these pages that require indexing. You can export them and start the indexing process

Submitting Pages for Indexing

We analyze the number of pages on the site with the status 'URL is not on Google.' If there are not many pages, you can manually submit them for indexing through Google Search Console. If there are more than 100 pages, then it's more effective to use the Fast Indexing service.

Conclusions

"Google has limited resources, and when it encounters the nearly infinite amount of content available on the Internet, the Googlebot robot can find and scan only a portion of this content. Then, from the scanned content, we can index only a portion of it."

Google visits only a fraction of all pages on the Internet, and from those, only a fraction is indexed. It's important to keep this in mind and continuously analyze the indexing of website pages in Google, even if the site seems perfect at the moment.