Sitemaps, Robots.txt & Canonicalization
Sitemaps, robots.txt, and canonical tags are the quiet infrastructure that tells search engines what to crawl, what to skip, and which version of a page counts. Get them wrong and you can accidentally block your own content, split ranking signals across duplicate URLs, or waste crawl budget on pages that don't matter. These five guides cover how to build and maintain each of these files correctly, with real examples of the mistakes that quietly cost sites their rankings.
In this section
How to Build and Optimize an XML Sitemap That Actually Helps Crawling
Learn how to structure, generate, and submit an XML sitemap correctly, including common mistakes that quietly waste your site's crawl budget.
Robots.txt Syntax Explained: Directives, Wildcards, and the Mistakes That Cost Traffic
Master robots.txt syntax and directives, and learn the common mistakes — like accidentally blocking your whole site — that quietly tank rankings.
Canonical Tags Explained: How to Consolidate Duplicate Content and Avoid Common Errors
Learn how rel=canonical tags work, when to use them, and the self-referencing, cross-domain, and conflicting-signal mistakes that undermine them.
Handling URL Parameters to Protect Crawl Budget and Prevent Duplicate Indexing
Filter, sort, and tracking parameters can multiply your URLs and drain crawl budget. Learn how to handle parameters correctly for SEO.
WWW vs. Non-WWW and HTTP vs. HTTPS: Getting Domain-Level Canonicalization Right
Learn how to permanently resolve www vs non-www and HTTP vs HTTPS duplication with correct 301 redirects, canonicals, and Search Console setup.