Every person who has ever encountered the concepts of SEO website promotion, search engine optimization, Google search results, etc., has asked himself the question of how to check the results of his efforts, and if an SEO agency is working on promotion, then, of course, you want to see the results of their work first of all, if it is not possible to track the results immediately from organic search results.

In this article, I want to talk about the most common methods used in search engine optimization of a website.

Introductory remarks

In my opinion, it is more expedient to start the article by considering how the process of crawling the site/pages of the site by a search robot takes place in general.

The process of visiting the site by a search robot (bot, crawler) is carried out in three stages:

1. Scan.
2. Indexing.
3. Formation of relevant search results.

Scan

So, a search robot* got to the site page, it read the information, transmitted it further to the search engine (PS) servers and went on to crawl the information on other pages of the site. Therefore, it is important to make a variety of linking on the site, because the robot goes from page to page only by links located on the page.

*Search robot is a program code that reads content information from a site page.

Indexing

At this stage, the information that was crawled on the page goes into the database of the search engine, in which a certain index is assigned. Assigning an index is necessary purely for the convenience of searching for information in the PS database. I would like to draw your attention right away that crawler crawling site pages does not always begin with scanning. If the robot got to the page from a third-party resource, it may not scan the page, but enter into the database information from the article from which it got to the site.

SERPs

At this stage, the PS analyzes the information in the index (this is a complex internal process based on the principles of machine learning and neural networks). As a result of this process, we get search engine results.

How to check whether there are pages in the index or not?

We will not torment you for a long time with complex data, algorithms for the operation of the PS and proceed directly to the consideration of the topic of our article. You can check if a page is in the index like this:

• Check the information in Search Console. To do this, go to the project’s Search Consultant page, then go to the “Pages” tab. In this section, you can see complete information with page statuses, a description of errors that were detected at the scanning stage.

You can see more information about the status of page indexing in the “Overview” tab.

You can also check a specific page separately. To do this, you need to enter the URL of the page you are interested in in the “Check all URLs on the resource” field.

After that, you will see one of the messages:

The first message indicates that everything is fine with the page, it is in the index, the second – unfortunately, is not encouraging, since the page did not get into the index.

• With the help of search operators. The most common search operator is the “site:” operator. This is not the most reliable and reliable source of data, with its help you can see only the approximate number of pages in the index, besides, it often shows inaccurate data. With this operator, you can check not the entire domain, but only the page you are interested in.

• With the help of special services. These tools will allow you to crawl the site, simulating the crawling of a search robot. These services include: Netpeak Spider, Screaming Frog, LinkChecker, SiteAnalyzer, etc. There are a huge number of such services on the Internet. With the help of such services, you can not only check whether the pages are in the index, but also get information about response codes, collect metadata, see the Canonical list, read Robots meta tags, etc. These services are quite good in operation, but they also have disadvantages: usually their use is paid, it takes a lot of time, PC power to check the site with their help

• Using the Web Archive service. Web Archive is a free service that stores archives of site pages. With its help, you can track changes on the page, in case of data loss, they can be restored from the web archive (if of course there is a cached version of the page in the archive).

This service is easy to use:

1. Go to the main page – https://web.archive.org/
2. Enter the name of the domain/site/keyword.

• Next, a window opens with information about how many copies were made. The information is displayed in the calendar by year. The dates when the copies were created are highlighted in blue.

• After selecting the date of interest and clicking on it, you will be redirected to the archive page.

Conclusion

In the article, we considered the most common options for checking the presence of a page in the index:

1. Using Search Сonsole.
2. Using search operators.
3. With the help of special services.
4. Using the Web Archive service.

Memorize

The goal of the site owner is to ensure that all pages open for indexing are in the search engine index. This is difficult to achieve. In addition, it is important to monitor the process of entering pages into the index: sudden changes in a positive or negative direction are a signal of a problem. We have described four ways to check the indexing of site pages:

1. In Google and Yandex webmaster panels.
2. Using the search operator site:.
3. With the help of plugins, such as the RDS bar, and bookmarklets.
4. In special services, for example, Netpeak Spider.

The site may be poorly indexed due to errors on the site, lack of a sitemap file. It is also important to check if the site is closed from indexing using the robots meta tag, x-robots tag, or in robots.txt.

To improve indexation, the following will help:

• optimization and correct configuration of robots.txt sitemaps;
• optimization of site loading speed;
• deleting duplicate pages and content;
• competent linking (between pages and when previews of new pages are displayed on the main page);
• promotion of new pages in social networks;
• regular updating and improvement of the site’s content.

Often it’s not about indexing, but about the approach to optimization. Respond to a user’s request better than anyone else. In this case, all the tips will be useful to you only to fix a good result.