URL parser
Split a URL into protocol, host, path and query.
Robots.txt generator
Create a robots.txt that blocks the paths you choose and points to your sitemap.
Bing and some others obey it. Google ignores it.
Block specific AI crawlersSave it as robots.txt in the root of your site, so it opens at https://yoursite.com/robots.txt. It asks well-behaved crawlers to stay out, but does not protect private pages.
Your files stay on your device
How it works
- 1Choose whether to allow crawling by default or block the whole site.
- 2List the paths to disallow, one per line, each starting with a slash.
- 3Add your sitemap address and tick any crawlers you want to block.
- 4Download robots.txt and upload it to the root of your site.
What the options do
- Disallow paths
- Crawlers are asked not to fetch these paths. A path ending in a slash covers everything inside the folder.
- Allow exceptions
- Lets a crawler fetch a path that sits inside a disallowed folder.
- Crawl delay
- Seconds between requests. Respected by some crawlers, ignored by Google.
- AI crawlers
- Adds a block for the named bot. It works only for bots that obey robots.txt.
Privacy
Everything runs in your browser. Nothing you type is sent to a server.
Questions people ask
Does robots.txt hide a page from search?
No. It stops well-behaved crawlers from fetching the page, but a blocked URL can still appear in results if other sites link to it. To keep a page out of search, use a noindex tag and let it be crawled.
Is robots.txt a security measure?
No. The file is public and bad bots ignore it. Never list secret paths in it, and protect private areas with a login.
Where does the file go?
At the root of the domain: https://example.com/robots.txt. Each subdomain needs its own file.
Will blocking AI crawlers remove my content from AI tools?
It asks those crawlers not to collect your pages in future. It does not remove what was collected earlier, and bots that ignore the file are not stopped.
What if I block CSS or JavaScript by mistake?
Google needs them to render your pages. Do not disallow folders that hold your scripts and styles.