FREE TOOL · LLMS.TXT GENERATOR
Enter your URL and get a clean, ready-to-use llms.txt file in seconds, with a preview of which AI crawlers like GPTBot and ClaudeBot are allowed.
Allow or block AI crawlers
GPTBot, ClaudeBot and more
Preview before you publish
Download in one click
Free
No sign-up
Any URL
5M+
Visitor Sessions
100M+
Context Signals
1B+
Experience Decisions
200K+
Adaptive Experiences
$50M+
Incremental Revenue
+25%
Avg. Conversion Lift
50K+
AI Visitors Assisted
10K+
AI Journeys Recovered
WHAT YOU GET
An llms.txt file built from a scan of your site’s key pages
Settings to allow all, block specific AI crawlers or disallow all
The latest AI crawlers, like GPTBot, ClaudeBot and PerplexityBot
A visual preview of what’s allowed or blocked
A one-click download for your site root, next to robots.txt
Generate my llms.txt
Free, no sign-up needed.
What is an llms.txt file?
Websites are built for people and browsers, so much of what an AI model sees when it reads a page is noise: menus, scripts, cookie banners, footers. An llms.txt file cuts through it. It’s a short Markdown file published on your site that tells AI systems what your site is about and which pages matter most, with a line on each.
The format was proposed by Jeremy Howard in September 2024 and is a voluntary convention, not an official standard. Think of it as a curated table of contents written for machines.
What goes in an llms.txt file?
The structure is deliberately simple: your name as a heading, a one- or two-line summary, then grouped lists of key links with a short note on each. A final, optional group holds lower-priority links that can be dropped when a shorter version is needed.
# Your Company
> What you do, in one sentence.
## Products
- [Pricing](https://site.com/pricing): Plans
## Optional
- [Blog](https://site.com/blog): Guides
The value is in what you leave out. A file that lists every page is just a worse sitemap.
How Fibr’s llms.txt generator builds your file
Writing llms.txt by hand means deciding which pages matter, writing a description for each and getting the Markdown right. Fibr’s generator does that first pass for you.
It scans your domain’s structure to find the pages that best explain your business, then writes them into a correctly formatted file with sensible default crawler settings. You review the result before downloading, so you stay in control of what AI tools are pointed to.
Customizing AI crawler access in Fibr
Alongside the file itself, Fibr previews which AI crawlers your settings allow, including GPTBot, ClaudeBot and PerplexityBot. From there you can choose one of three approaches:
Allow all. Suits most sites that want to be found and cited by AI tools.
Block specific crawlers. Useful if you’re comfortable with some AI uses of your content but not others.
Restrict everything. For sites that don’t want their content used by AI at all.
The preview updates as you change settings, so you can see exactly what you’re publishing before you download.
Read more
FROM THE BLOG
View all articles
FAQ
Answers to common questions about Fibr’s free llms.txt generator and llms.txt files.
How do I use the Fibr llms.txt generator?
Enter your full website URL, for example https://example.com, and run the scan. Review the file, download it with one click and upload it to your site.
Is the Fibr llms.txt generator free?
What else makes a website ready for AI agents?
Where do I put my llms.txt file?
What is the difference between llms.txt and robots.txt?
What is the difference between llms.txt and sitemap.xml?
Does llms.txt help SEO or AI visibility?
Do ChatGPT, Claude and Google actually read llms.txt?
Is your website starting every visit from zero?
Fibr gives your website the intelligence it needs, right from the start.