llms.txt for Next.js: Implementation Checklist for AI Crawler Readiness

Next.js can be excellent for AI crawler readiness, but only if public content is server-visible. A crawler should be able to request your homepage, feature pages, docs, case studies, and blog posts and receive useful HTML without running a browser.

GEO Scout helps teams treat this as an implementation workflow, not a theory. After shipping the checklist below, track prompts and sources in geoscout.pro to see whether AI systems cite the pages you exposed.

File Placement

For static content, use:

public/llms.txt
public/robots.txt
public/sitemap.xml

For generated content in the App Router, use route handlers:

app/llms.txt/route.ts
app/robots.txt/route.ts
app/sitemap.ts

The root URLs should resolve directly:

https://example.com/llms.txt
https://example.com/robots.txt
https://example.com/sitemap.xml

Minimal llms.txt

Keep it short and operational:

# Example SaaS
 
> Product documentation, pricing, case studies, and implementation resources for Example SaaS.
 
## Core pages
- https://example.com/
- https://example.com/features
- https://example.com/pricing
- https://example.com/customers
 
## Documentation
- https://example.com/docs/getting-started
- https://example.com/docs/api
 
## Policies
- https://example.com/security
- https://example.com/privacy

Do not turn llms.txt into a keyword dump. It should help crawlers find canonical, useful public information.

robots.txt

Allow the AI crawlers you want to access public content, block private or duplicate paths, and include the sitemap:

User-agent: GPTBot
Allow: /
 
User-agent: ClaudeBot
Allow: /
 
User-agent: PerplexityBot
Allow: /
 
User-agent: *
Disallow: /app/
Disallow: /api/
Disallow: /checkout/
 
Sitemap: https://example.com/sitemap.xml

Blocking /api/ is usually correct. Blocking /docs/, /blog/, or /customers/ is usually harmful for GEO.

Sitemap

In app/sitemap.ts, include all canonical public pages:

import type { MetadataRoute } from 'next'
 
export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://example.com/',
      lastModified: new Date('2026-04-24'),
      changeFrequency: 'weekly',
      priority: 1,
    },
    {
      url: 'https://example.com/features',
      lastModified: new Date('2026-04-24'),
      changeFrequency: 'weekly',
      priority: 0.8,
    },
  ]
}

Use stable canonical URLs. If localized pages exist, connect them with alternates.languages in metadata and hreflang in sitemaps where your stack supports it.

Rendering Checklist

• Use SSG or ISR for blog, docs, feature pages, pricing, comparisons, and case studies.
• Use SSR only where public content depends on request-time data.
• Keep private dashboards behind auth and out of crawler paths.
• Render H1, body copy, FAQ, tables, and JSON-LD before hydration.
• Avoid hiding critical content behind tabs that are empty in the initial HTML.
• Validate with curl -A "GPTBot/1.0" https://example.com/features.

Structured Data

For SaaS pages, add Organization, SoftwareApplication, FAQPage, BreadcrumbList, Article, and Product where relevant. Put JSON-LD in server-rendered components or metadata helpers so crawlers receive it immediately.

Logs

Use Vercel logs, CDN logs, or server logs to verify bot access:

GPTBot
ClaudeBot
PerplexityBot
Google-Extended
Googlebot
Bingbot

Look for status codes, blocked paths, excessive redirects, and pages returning thin HTML.

Release Checklist

1. Publish /llms.txt.
2. Confirm robots.txt allows the AI crawlers you want.
3. Include sitemap links and canonical URLs.
4. Render public pages with SSG, ISR, or SSR.
5. Add server-visible schema.
6. Inspect raw HTML with bot user agents.
7. Monitor AI mentions and source citations in GEO Scout.

The goal is not to create a special website for bots. The goal is to make your best public information easy to retrieve, understand, and cite.