Robots.txt Generator
Build robots.txt rules with allow, disallow and sitemap entries.
The Robots.txt Generator assembles a robots.txt file from allow and disallow path lists plus an optional sitemap URL, ready to deploy with your site.
Please enter some input.
How to use
- Enter allowed and disallowed paths, one per line.
- Add a sitemap URL if you have one.
- Copy the generated robots.txt content.
Frequently asked questions
What is included by default?
A User-agent: * rule followed by the Allow and Disallow lines you provide.
Can I allow the whole site?
Enter / in the Allow field.
Related tools
IP Address Lookup
Look up IP address location and network details.
User Agent Parser
Parse browser, OS and device from a user agent string.
HTTP Headers Checker
Inspect response headers for any web page.
DNS Lookup
Resolve DNS records for a domain.
URL Parser
Break a URL into protocol, host, path, query, fragment and decoded parameters.
HTTP Status Code Lookup
Look up HTTP status codes by number or description.