Developer-Native Marketing

File Generation

Automated generation of robots.txt, sitemap.xml, and SEO metadata files.

Live Generation Results from raz-kaplan-website

Real files generated from your actual content

1,424

robots.txt bytes

6,520

sitemap.xml bytes

URLs mapped

100%

AI bot ready

Quick Generation Commands

# Generate all SEO files

gtm-toolkit generate --all

# Generate specific files

gtm-toolkit generate robots

gtm-toolkit generate sitemap

gtm-toolkit generate meta

Robots.txt Generation

Control search engine and AI bot access to your site

Generated Output Example

# Robots.txt generated by GTM Toolkit
# Site: raz-kaplan-website
# Generated: 2024-12-25T10:30:00Z

User-agent: *
Allow: /

# SEO-friendly crawler permissions
User-agent: Googlebot
Allow: /
Crawl-delay: 1

User-agent: Bingbot
Allow: /
Crawl-delay: 1

# AI Bot Permissions - Configured for GTM Toolkit
User-agent: Claude-Web
Allow: /
Crawl-delay: 2

User-agent: ChatGPT-User
Allow: /
Crawl-delay: 2

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /blog/
Disallow: /admin/
Disallow: /private/

# Sitemap location
Sitemap: https://raz-kaplan-website.vercel.app/sitemap.xml

# Security restrictions
Disallow: /admin/
Disallow: /api/
Disallow: /.env
Disallow: /config/

✅ Allowed by Default

• All standard search engines (Google, Bing, etc.)

• Claude AI (Claude-Web, Anthropic crawlers)

• ChatGPT and OpenAI bots

• Public content and blog posts

• Static assets and images

🚫 Blocked by Default

• Admin panels and private areas

• API endpoints and configuration files

• Environment variables and secrets

• Development and testing directories

• User-defined restricted paths

AI Bot Strategy

GTM Toolkit implements an “AI-friendly” robots.txt strategy, allowing AI bots to access content for training and analysis while respecting rate limits. This helps your content become part of AI training datasets, potentially increasing discoverability.

Generation Commands Reference

Complete command-line interface for file generation

# Generate all files

npx gtm-toolkit generate --all

# Generate specific files

npx gtm-toolkit generate --robots

npx gtm-toolkit generate --sitemap

npx gtm-toolkit generate --meta

# Custom output locations

npx gtm-toolkit generate --robots && mv public/robots.txt custom/robots.txt

npx gtm-toolkit generate --sitemap && mv public/sitemap.xml static/sitemap.xml

# Configuration overrides

Edit robots.customRules and content paths inside gtm.config.js.

Use a custom script (see advanced tab) to supply alternate static routes.

# Development tip

Re-run the command after content updates or add it to a pre-deploy script.