In SEO, search engines (For example: Google, Bing, Yahoo,Baidu etc.) analyze the content of a web page to determine if it would be relevant for any given search query.
Search Engines have primarily three functions:
1. Crawling: It's a process in which team of robots find new and updated content. Here content means a web page, a video, an image, a PDF etc.
Google bot starts out by fetching a few web pages, and then follows the links on those web pages to find new URLs. Hence a massive database of URL's is collected called Caffeine.Crawling is the first part of having a search engine recognize your page and show it in search results.
2. Indexing: It's a process in which the search engine stores and organize the data collected in Index (After crawling). The indexing of your content by Google is determined by system algorithms that take into account user demand and quality checks.
3. Ranking: To show the results in such a relevant order in SERP for any user's query is called Ranking. In general, you can assume that the higher a website is ranked, the more relevant the search engine believes that site is to the query.
It’s possible to block search engine crawlers from part or all of your site, or instruct search engines to avoid storing certain pages in their index. While there can be reasons for doing this, if you want your content found by searchers, you have to first make sure it’s accessible to crawlers and is indexable