Granular Bot Policies & Custom Profiles (Pro)

Configure tailored content delivery policies for individual AI crawlers. Choose whether specific AI agents receive structured Markdown payloads or fall back to standard WordPress HTML.

1. Understanding Bot Delivery Policies

Not all AI bots operate identically. Some specialized web scrapers benefit from immediate Markdown delivery, while others might require standard HTML or specific content filtering rules.

Delivery Format Policy (Not 403 Blocking)

Bot Policies control whether a recognized crawler receives Markdown vs HTML. This is a content formatting and delivery router, not a firewall or 403/network blocking tool.

2. Supported AI Bot Families

The Pro plugin includes built-in recognition profiles for major AI crawler signatures:

OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
Anthropic: ClaudeBot, anthropic-ai
Perplexity: PerplexityBot
Google AI: Google-Extended (Note: traditional Googlebot always receives HTML)
Other Engines: CCBot, Bytespider, Cohere-training, Diffbot

3. Configuring Policies in WP-Admin

Navigate to Bot Settings

Go to Settings → AI Markdown → Bot Policies in your WordPress dashboard.

Select Delivery Action

For each recognized crawler, choose between Always Serve Markdown, Serve HTML (Fallback), or Inherit Global Settings.

Add Custom User-Agent Rules

Add custom crawler regex strings or User-Agent substrings for private LLM crawlers or custom retrieval pipelines.

4. Simulating Bot Requests

Test individual bot policies from your terminal by specifying custom User-Agent headers:

Terminal / Bash Simulate GPTBot

# Simulating GPTBot with Accept header
curl -i -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  -H "Accept: text/markdown" https://your-site.com/my-post/

Expected Result

Bot configured for Markdown receives text/markdown payload.
Bot configured for HTML fallback receives text/html response.

Common Mistakes

Attempting to route traditional search engines (e.g. Googlebot) to Markdown (safeguards will enforce HTML).
Overriding standard browser User-Agents unintentionally.