Build a perfect robots.txt file in seconds — no coding needed
Quick Presets
Use * for all crawlers, or a specific bot name like Googlebot.
Allow paths
Disallow paths
Sitemap URLs
Used by some crawlers (e.g. Yandex) to specify the preferred domain version.
robots.txt output
6 lines
How to deploy
Save the output as robots.txt and upload it to your site root so it is served at https://yourdomain.com/robots.txt.
A robots.txt generator lets you visually build the crawl-control file that search engines read before indexing your site. Configure user-agent rules, allow and disallow paths, set a crawl-delay to manage server load, and declare your sitemap URLs — then copy the ready-to-deploy output straight to your web root. No syntax memorization required.
A robots.txt file sits at the root of your website and tells search engine crawlers which pages or directories they are allowed — or not allowed — to visit. It helps prevent crawlers from indexing admin panels, duplicate content, or private sections, which can protect sensitive areas and focus your crawl budget on important pages.
Robots.txt is a voluntary standard. Well-behaved crawlers like Googlebot and Bingbot respect it, but malicious bots or scrapers can ignore it entirely. For pages that must stay private, use server-side authentication or a noindex meta tag in addition to robots.txt rules.
Disallow prevents a crawler from accessing a path, while Allow explicitly grants access to a sub-path inside a disallowed directory. For example, you can disallow /wp-content/ but Allow /wp-content/uploads/ so product images are still indexed while plugin files are not.
The file must live at the root of your domain and be accessible at https://yourdomain.com/robots.txt. If you host on a subdomain, each subdomain needs its own robots.txt. For WordPress or Next.js sites, copy the generated file into the public/ directory so it is served automatically.