Technical SEO: advanced crawl control

Robots.txt

• Purpose: controls which pages search engines are allowed to crawl.
• Location: always in the root directory (https://example.com/robots.txt).
• Usage: allow/disallow specific directories or pages; block duplicate content from being crawled.
• In WordPress, the Rank Math plugin can manage robots.txt settings directly.

Robots meta tags and X-Robots-Tag

• Meta robots tag: placed in the HTML <head> to control indexing per page: index/noindex, follow/nofollow.
• X-Robots-Tag: the same control, added via HTTP headers instead, for non-HTML files (PDFs, images).
• Typical use: blocking indexing of thank-you pages, admin areas, or staging sites.

llms.txt

An emerging standard, positioned as a guide for AI crawlers: it sits in the root directory like robots.txt and is used to control how AI systems use and crawl the site's content.

Sitemap.xml

• Purpose: tells search engines which URLs on the site matter.
• Best practices: include only indexable pages; update it automatically as new content is added; submit it in Google Search Console.

Crawl budget optimization

Reduce unnecessary crawling by blocking irrelevant pages in robots.txt, fixing redirect chains, and eliminating duplicate URLs. The payoff: search engines spend their limited crawl budget on the pages that actually matter.

Canonical tags

Prevent duplicate-content issues by naming the preferred version of a page:

<link rel="canonical" href="https://example.com/main-page">

Typical uses: HTTP/HTTPS duplicates, www/non-www duplicates, and pagination.

.htaccess

A server configuration file used for 301 redirects, forcing HTTPS, blocking specific IPs, and custom error pages.