Content Negotiation & SEO Safeguards

Learn how standard HTTP Accept header content negotiation delivers clean Markdown representations of your posts to AI agents from the exact same canonical URL without risking duplicate content or search cloaking penalties.

1. How Content Negotiation Works

Content negotiation is a core mechanism of the HTTP/1.1 specification (RFC 7231). When an AI client (such as ChatGPT, Claude, or Perplexity) requests a webpage, it can send an Accept header indicating its preferred response format:

Standard Browser / Googlebot: Sends Accept: text/html,application/xhtml+xml... → WordPress serves standard rendered HTML.
AI Agent / LLM Ingestion: Sends Accept: text/markdown → AI Markdown Generator intercepts the request and outputs high-density Markdown from the database cache.

Same Canonical URL

Both representations exist at the exact same permalink (e.g. https://your-site.com/seo-guide/). There are no secondary URLs, separate subdomains, or redirects required.

2. Why Direct .md URLs Return 404 Not Found

Some third-party tools create static .md files accessible via direct URLs like https://your-site.com/my-post.md. In production environments, this introduces significant SEO risks:

Duplicate Content Indexing: Search engines discover and index both /my-post/ and /my-post.md, splitting link equity and creating canonical ambiguities.
Canonical Fragmentation: Different web scrapers may cite inconsistent URL paths.

AI Markdown Generator strictly avoids this anti-pattern. Direct requests to .md file extension URLs are intentionally rejected with an HTTP 404 Not Found status to protect your search index integrity.

3. Search Engine Cloaking Safeguards

Major search engines (including Google and Bing) strictly penalize websites that attempt to "cloak"—serving different content to search engine crawlers than what is presented to human visitors.

To ensure strict safety for supported search crawler flows, AI Markdown Generator includes hardcoded User-Agent evaluation logic:

✓

Googlebot, Bingbot & Yandex Always Receive HTML

Traditional search indexers are forced to receive the full, original WordPress HTML theme output, preserving your exact rankings, structured schema markup, and metadata.

✓

Vary: Accept Response Header

The plugin automatically appends Vary: Accept, User-Agent to all responses, instructing downstream CDN edge servers, reverse proxies, and browser caches to store and serve the HTML and Markdown versions in separate cache buckets.

4. Practical Testing Commands

You can verify both flows using the command line:

Terminal / Bash 1. AI Agent Simulation

# Simulating an AI Agent requesting Markdown
curl -i -H "Accept: text/markdown" https://your-site.com/sample-page/

# Response: HTTP/1.1 200 OK
# Content-Type: text/markdown; charset=UTF-8
# Vary: Accept, User-Agent

Terminal / Bash 2. Googlebot Simulation

# Simulating Googlebot (forced to HTML regardless of Accept header)
curl -i -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -H "Accept: text/markdown" https://your-site.com/sample-page/

# Response: HTTP/1.1 200 OK
# Content-Type: text/html; charset=UTF-8

Expected Result

AI agent requests receive text/markdown.
Googlebot / Bingbot requests receive text/html.
Direct .md URL requests return 404 Not Found.

Common Mistakes

Configuring aggressive edge caching without forwarding the Vary: Accept header.
Assuming Markdown delivery guarantees specific rankings (rankings depend on content quality and authority).