Quick answer: A robots.txt file tells search-engine crawlers which parts of your site they may or may not request, and an XML sitemap lists the URLs you want indexed. Set your rules below to generate a valid
robots.txt, then paste your page URLs to build a sitemap.xml. Upload both to the root of your domain.Robots.txt generator
Use * for all bots, or e.g. Googlebot.
Google ignores this; Bing honours it.
Paths crawlers should not request. Leave blank to allow everything.
XML sitemap generator
Focused on growth, not plumbing? Bluey Email is all-inclusive email marketing with send-based pricing — automations, CRM and landing pages on every plan. See how it compares.
How the robots.txt and sitemap generator works
Robots.txt uses a simple grammar: a User-agent line naming the crawler, followed by Disallow and Allow rules that match URL paths, and an optional Sitemap line. This tool assembles those lines in the correct order and adds your sitemap reference. The sitemap generator wraps each URL you paste in the standard <url><loc> elements defined by the sitemaps.org protocol, optionally adding a lastmod date and changefreq hint. Save the outputs as robots.txt and sitemap.xml and upload them to your domain root.
Frequently asked questions
Does Disallow remove a page from Google?No. Disallow stops crawling, but a blocked URL can still be indexed if other pages link to it. To keep a page out of search, use a noindex meta tag and let it stay crawlable.
Where does robots.txt go?At the root of your domain, at example.com/robots.txt. Crawlers only read it at that exact location, not in subfolders.
Do I need an XML sitemap?It is not required, but it helps search engines discover pages faster, especially on large or new sites with few inbound links. Submit it in Google Search Console.
Is this generator free?Yes, completely free and private — everything runs in your browser and nothing is sent anywhere.
Keep exploring
Related tools: Schema markup generator, SERP snippet preview and Heading structure checker.