Build a crawler policy in WordPress

Better Robots.txt Pro extends a guided workflow for configuring robots.txt. Choose a preset, adjust plain-language settings, and inspect the generated directives in the Review & Save step before saving. Core WordPress protection rules remain visible and editable, and you can revisit settings to regenerate the file.

Set rules by crawler category

Manage search engine visibility, AI training and search or discovery crawlers, SEO utilities, bad bots, archive services, feed crawlers, crawl traps, social preview bots, and other automated agents. AI-related agents are grouped as training crawlers, search and discovery crawlers, and user-action fetchers.

• Pro and Premium add per-bot AI crawler rules and custom user agents.
• Configure AI usage signals such as search, ai-input, and ai-train, and generate llms.txt and other machine-readable governance documents.
• Manage archive and Wayback access, feeds, and crawl traps.
• Review WooCommerce crawl cleanup settings and rules for CSS, JavaScript, and picture crawling.
• Configure social preview crawlers and ads.txt or app-ads.txt allowance.
• Use advanced directives such as crawl-delay and custom rules.

Robots.txt policies depend on crawler compliance. User-action fetchers may not follow robots.txt in the same way as automated crawlers. Where stronger enforcement is needed, server logs, WAF rules, provider verification, or IP-based controls may help, depending on availability.

Choose a starting preset

Essential delivers a practical starting point for general websites. AI-First supports AI-related governance while keeping discovery in view. Custom lets you configure modules manually. Fortress is a Premium preset for sites seeking more restrictive protection against scraping, archive capture, and unnecessary crawl activity. You can adjust settings after choosing a preset, and edition labels identify which options require Pro or Premium.

Pro and Premium options

Pro adds advanced governance and protection modules, including additional AI, crawler, and cleanup controls. Premium includes the most restrictive protection options, the Fortress preset, and additional high-control modules.

Configure, review, and save

1. Install and activate Better Robots.txt in WordPress, then open its settings.
2. Select a preset and configure the available crawler and governance controls.
3. Review the generated robots.txt in the Review & Save step.
4. Save when the directives match your intended policy. Return to the settings later to make changes and regenerate the file.