How to use
- 1Start from a preset: allow all, block all, WordPress, Shopify-style or block AI crawlers.
- 2Add or edit rules per bot (Allow/Disallow paths) and an optional crawl-delay.
- 3Add your sitemap URLs.
- 4Copy or download robots.txt and upload it to your site’s root folder.
Robots.txt syntax in 60 seconds
A file is made of groups. Each starts with one or more User-agent lines followed by Allow and Disallow rules: “User-agent: *” plus “Disallow: /cart/” blocks every crawler from /cart/. An empty Disallow allows everything. * is a wildcard and $ marks the end of a URL. Sitemap lines can go anywhere and must be absolute URLs. The file must be plain text, UTF-8, at /robots.txt, and Google reads at most the first 500 KiB.
Blocking AI crawlers
The AI preset adds Disallow: / groups for GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Gemini training), PerplexityBot and Bytespider (ByteDance). Google-Extended doesn’t affect Google Search crawling or ranking. Reputable AI companies honor robots.txt, but it is a request, not enforcement — use server or CDN bot rules if you need hard blocking.
Platform tips
WordPress: block /wp-admin/ but allow /wp-admin/admin-ajax.php, and don’t block /wp-content/ (themes and plugins hold CSS and JS Google needs). Online stores: block cart, checkout, account and internal search URLs, plus sorting and filter parameters that create endless duplicate URLs. Crawl-delay is ignored by Google; Bing and Yandex respect it.
Frequently asked questions
Can robots.txt remove a page from Google?
No. It only stops crawling. To deindex a page, allow crawling and add a noindex meta tag, or remove the page and return 404/410.
Where do I upload robots.txt?
To the root of your domain, so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
Does blocking AI crawlers affect my Google rankings?
No. Blocking GPTBot, ClaudeBot or Google-Extended doesn’t change how Googlebot crawls or ranks your site.
How can I check my robots.txt after uploading?
Use our robots.txt tester to fetch the live file and test specific URLs against any crawler.
Not happy with the results?
Talk to Webin Agency about fast, SEO-friendly websites, e-commerce and Google Ads management.