Search engines find URLs, request their files with crawlers and render pages so they can understand the content available to users.
What you will learn
By the end, you will understand:
Separate discovery, crawling and rendering
Explain how links and sitemaps help discovery
Identify common reasons a crawler cannot read a page
Visual explainer
See the idea clearly.
trusted local service
1
A lesson diagram showing URL discovery, crawler request, downloaded files and a rendered page before indexing.
Discovery begins with a URL
A search engine first needs to learn that a web address exists. It can discover a URL through links from pages it already knows, an XML sitemap, previous crawls, redirects and other signals. Google’s crawler is commonly called Googlebot.
A sitemap is a file listing URLs a site considers important. It can help discovery, especially on large or new sites, but it does not force crawling or indexing. Links remain important because they help people and crawlers move through the site and understand relationships.
Crawling is the request
A crawler requests a URL much like a browser requests it. The server responds with a status code and content. A successful response is commonly 200. A permanent redirect is commonly 301 or 308. A missing page is commonly 404.
Search engines decide how often and how deeply to crawl. They may avoid URLs blocked by robots.txt, slow down when a server struggles, or spend time on duplicate and low-value URLs if a site creates too many of them.
Rendering builds the page
Modern pages may need CSS, images and JavaScript before their important content appears. Rendering is the process of turning downloaded resources into the page a browser or search system can interpret.
Google can render JavaScript, but that does not mean every implementation is equally reliable. Important content and links should be available clearly, required files should not be blocked, and the page should work even when a script is slow or fails.
Crawled does not mean indexed
Crawling only means the search engine requested the URL. After processing and evaluating it, the engine may index the page, choose another canonical version or decide not to keep it. This distinction prevents a common mistake: seeing a crawler visit and assuming the page must appear in results.
Real-world example
Example: a new service page with no links
Example
A company publishes `/services/roof-repair/` but never links to it and omits it from the sitemap. A person who knows the exact URL can open it, but discovery may be slow. Adding it to the real Services navigation and sitemap creates clear discovery paths; neither action guarantees indexing.
Try this
Trace one page
Choose an important page and find every internal page that links to it. Then check whether it appears in the XML sitemap. If it is important but isolated, improve the real navigation rather than creating dozens of artificial links.
Common questions
Questions beginners ask.
What is a crawler?
Software that automatically requests web pages and follows discoverable links. Google’s main search crawler is commonly called Googlebot.
Does submitting a sitemap guarantee indexing?
No. A sitemap can help discovery and provide information, but the search engine still decides whether to crawl and index each URL.
Can Google read JavaScript?
Google can render many JavaScript pages, but blocked resources, errors, delayed content or complex interactions can still cause problems.
What is crawl budget?
It is industry shorthand for the amount and pattern of crawling a search engine is willing and able to perform on a site. It usually matters most for very large or rapidly changing sites.
How can I see what Google received?
Search Console’s URL Inspection tool can show indexing information and a tested view of a URL. Server logs can also show crawler requests.
