HTML Meta robots help search engine crawlers to easily navigate the content on the web-page and how to treat particular aspects within the web page.
On the web page, aspects such as content, links require attention and by giving these aspects robot tags search engine crawlers understand what the owner of the website wants to do with them.
<meta name="robots" content="instructions to be followed">
As shown above, we can see that we can change the instructions depending on the need of the user where it varies from tag to tag such as index, noindex, follow, nofollow, all or none-thing.
No index, No Follow robot meta tags tells search engine crawlers not to index and follow the links on the webpage. This step takes place when the web page is in its development phase.
This robot meta tag indexes the webpage but it doesn't follow the links on the webpage to other pages.
This robot meta tag doesn't index the web page instead it follows the links on the webpage to other pages.
This robot meta tag indexes the web page in the search index and also follows the links on the web page to other pages.