Technical SEO checklist: crawling, indexing and site speed

What technical SEO is and when it matters most

Technical SEO is the work of making a site easy for search engines to crawl, render and index, so that good content can actually compete in the results. It deals with the infrastructure behind your pages rather than the words on them.

On a small site built on a mainstream CMS, technical SEO is often a short list of checks. On large e-commerce stores, multilingual sites and JavaScript applications, it can be the main thing holding rankings back. A single wrong setting can remove thousands of pages from Google.

Use the checklist below in order. Each section builds on the previous one: there is no point tuning page speed on pages Google cannot index.

Crawling: can search engines reach your pages?

Crawling is the first step, and the question is simple: can Googlebot request your important URLs and get a working response? If it cannot, nothing else on this list matters.

Crawling checklist

• Run a crawl of the site with a tool such as Screaming Frog or Sitebulb and compare the number of URLs found with the number you expect.
• Check that important pages return a 200 status code, not 404, 5xx or soft 404s.
• Make sure key pages are linked through normal HTML <a href> links. Google does not reliably follow links that only work through JavaScript click events.
• Look for orphan pages, which exist but have no internal links pointing to them.
• Review the Crawl stats report in Search Console for spikes in server errors or slow response times.

Crawl budget

Crawl budget only becomes a real concern on very large sites or sites that generate many URLs automatically, such as through filters and parameters. If that is you, the aim is to stop Google spending time on low-value URL variations so it gets to your important pages faster.

Robots.txt

Robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may or may not crawl. It is a crawl control tool, not an indexing control tool, and confusing the two is one of the most common technical mistakes.

• Check that the file sits at /robots.txt on each host and protocol, for example on https://www.example.com/robots.txt.
• Make sure it does not block important sections, CSS or JavaScript files Google needs to render pages.
• Add a Sitemap: line pointing to your XML sitemap.
• Keep it under Google's 500 KiB size limit. Content beyond that limit is ignored.
• Remember that a URL blocked in robots.txt can still be indexed if other pages link to it, usually without a description. To keep a page out of the index, allow crawling and use noindex.

Watch out for a leftover Disallow: / from a staging environment. It is a surprisingly common reason for a new site launch to lose its traffic.

Indexing and XML sitemaps

Indexing determines which of your crawled pages Google stores and can show in search. The goal is for every page you want to rank to be indexed and for low-value pages to stay out.

Indexing checklist

• Open the Pages report in Search Console and review each reason listed under "Why pages aren't indexed".
• Use the URL Inspection tool on key pages to see whether they are indexed and which canonical Google chose.
• Search the code for noindex in meta robots tags and in X-Robots-Tag HTTP headers, and confirm each one is intentional.
• Keep thin, duplicate or utility pages, such as internal search results, out of the index.

XML sitemaps

An XML sitemap lists the URLs you want search engines to know about. It helps discovery, especially for large or new sites, but it does not force Google to index anything.

• Include only canonical, indexable URLs that return a 200 status.
• Keep each sitemap file within the protocol limits of 50,000 URLs and 50 MB uncompressed. Use a sitemap index file to group several sitemaps on larger sites.
• Use accurate lastmod dates. Google uses this value when it is consistently accurate, and ignores priority and changefreq.
• Submit the sitemap in Search Console and check the number of discovered URLs.

Canonicals and duplicate content

A canonical tag tells Google which URL is the main version when several URLs show the same or very similar content. Clear canonical signals stop duplicates from splitting your ranking signals.

Duplicates appear for many ordinary reasons: HTTP and HTTPS versions, www and non-www, trailing slash variations, tracking parameters, sorting and filter parameters, and print versions.

• Add a self-referencing rel="canonical" tag to every indexable page, using the full absolute URL.
• Make sure canonical tags, internal links, sitemaps and redirects all point to the same preferred version.
• Do not point canonicals to URLs that redirect, return errors or carry noindex.
• Remember that canonicals are a strong hint, not a directive. If your other signals contradict them, Google may choose a different URL.

On multilingual sites, each language version should canonicalize to itself and be linked with hreflang, not canonicalized to the main language. Our guide to international SEO and hreflang covers this in more detail.

Redirects and status codes

Redirects send users and search engines from an old URL to a new one, and getting them right protects the rankings and links that old URLs have built up. Use permanent redirects for permanent moves.

• Avoid redirect chains, where URL A redirects to B, which redirects to C. Point A directly to C.
• Update internal links to point to final URLs instead of relying on redirects.
• Do not redirect all removed pages to the homepage. Google tends to treat that as a soft 404.
• During migrations, map every old URL to its closest new equivalent before launch.

JavaScript rendering

Google can render JavaScript, but content that depends on it is processed in an extra step and can fail in ways that plain HTML does not. The safest approach is to make sure important content and links are present in the HTML the server sends.

• Compare the raw HTML with the rendered HTML in the URL Inspection tool. Key text, links, titles and canonical tags should appear in both.
• Prefer server-side rendering or static generation for pages that need to rank, especially on React, Vue or Angular sites.
• Do not block JavaScript or CSS files that are needed to render content.
• Avoid content that loads only after a user action, such as clicking a tab or scrolling, if you need it indexed.
• Use real URLs for different views rather than fragments after a #, which Google generally ignores.

Also keep in mind that many AI crawlers do not run JavaScript at all. If your content only exists after rendering, it may be invisible to them.

Site speed and Core Web Vitals

Site speed affects both user experience and search, and Google measures it through Core Web Vitals: Largest Contentful Paint, Interaction to Next Paint and Cumulative Layout Shift. A page passes when at least 75% of real visits meet the "good" threshold for each metric.

• LCP: the main content should load within 2.5 seconds.
• INP: the page should respond to interactions within 200 milliseconds.
• CLS: layout shift should stay at 0.1 or below.

Check field data in the Core Web Vitals report in Search Console and in PageSpeed Insights. Common fixes include compressing and resizing images, reducing render-blocking scripts, using a CDN, cutting third-party tags and reserving space for images and ads. Our full guide to Core Web Vitals explains how to diagnose each metric.

Other basics belong here too: the whole site should run on HTTPS, and pages should work well on mobile, since Google indexes the mobile version of your site.

How to prioritise technical fixes

Fix whatever stops important pages from being crawled or indexed first, then deal with signals that split or waste authority, and only then optimize speed. That order matches how search engines process a site.

1. Blockers: accidental noindex, robots.txt blocks, server errors and broken rendering on key templates.
2. Consolidation: canonicals, redirect chains, duplicate URL variants and sitemap clean-up.
3. Performance: Core Web Vitals and mobile usability on your highest-traffic templates.

Work at template level where possible. Fixing the product page template once fixes thousands of URLs. If you want this checklist applied to your own site with priorities set for you, our technical SEO service starts with exactly this kind of review.

Frequently asked questions