AI crawlers, Markdown and content signals

How AI crawlers can read our pages: Markdown on request, Content Signals in robots.txt and what we allow.

AI search tools and assistants read the web on your behalf. We make our public pages easy for them to read and cite, and we state clearly what they may not do with them.

Markdown on request#

Send Accept: text/markdown to any public page (home, pricing, guides, blog, these docs) and you get a clean Markdown version instead of HTML. Browsers never ask for Markdown, so people always see the normal page.

Shell

curl -H "Accept: text/markdown" https://linkinseconds.com/pricing

Response header	Meaning
`Content-Type: text/markdown`	The body is Markdown.
`x-markdown-tokens`	A rough token count, so an agent can budget its context.
`Vary: Accept`	Caches keep the HTML and Markdown versions apart.

The home page answers with llms.txt, our curated summary of the site. Private pages (the dashboard, sign-in) are never converted: they answer 406 Not Acceptable. Files people share on /p/… links are never converted either.

Finding the Markdown version without negotiating

Every public page also has a direct Markdown address at /agent-markdown/ followed by the page's own path, and every public HTML response advertises it twice, so a crawler that only follows links finds it too:

An HTTP Link response header on the HTML page, for example on /pricing:

Response header on https://linkinseconds.com/pricing

Link: <https://linkinseconds.com/agent-markdown/pricing>; rel="alternate"; type="text/markdown"

The same relation in the page <head>:

HTML

<link rel="alternate" type="text/markdown" href="/agent-markdown/pricing">

The home page's alternate is /agent-markdown (which serves llms.txt); a localized page keeps its prefix, so /es/pricing points at /agent-markdown/es/pricing. The direct address answers text/markdown whatever Accept header you send, is marked noindex, and gives 406 for any page that has no Markdown version. Pages without an alternate (the dashboard, sign-in, shared files) carry neither the header nor the tag.

llms.txt and llms-full.txt#

File	What it is	When to fetch it
/llms.txt	A short, curated index: what the product does, the authoritative limits and prices, and one line per public page.	To decide which page answers a question, in one small fetch.
/llms-full.txt	The full text: the same summary and facts, then the English copy of every landing page (hero, answer, limits, sections, comparison tables), then the docs and guides as described links. Around 300 KB, capped near 400 KB.	To read the whole marketing site at once instead of crawling 50 pages.

Both are generated from the same registries and plan limits as the pages themselves, so they can never carry a stale price or a page that no longer exists. Both are plain UTF-8 text, cached for an hour.

robots.txt and Content Signals#

robots.txt (excerpt)

User-Agent: *
Allow: /
Disallow: /dashboard
Disallow: /api/
Content-Signal: search=yes, ai-input=yes, ai-train=no

Signal	Our choice	Meaning
`search`	yes	Search engines may index and link to our pages.
`ai-input`	yes	AI assistants may read our pages to answer a question and cite us.
`ai-train`	no	Nothing here, including files our users share, may be used to train AI models.

Your shared files

Links you share stay readable by anyone you send them to. The ai-train=no signal tells AI companies that respect Content Signals not to train on them. Signals are a stated preference, not a technical lock: for files that must stay private, use a password or an email gate (see Link settings).

Crawlers we allow

Our robots.txt has one rule set, User-Agent: *, so every well-behaved crawler gets the same access: all public pages, never the dashboard, sign-in, or the API. The Content-Signal line above applies to all of them. The crawlers we see most, and what each may do here:

User agent	Who	Allowed to
`OAI-SearchBot`	OpenAI, ChatGPT search index	Index public pages so ChatGPT search can link to and cite them.
`ChatGPT-User`	OpenAI, on a user's request	Fetch a public page live to answer that person's question (ai-input).
`GPTBot`	OpenAI, model training crawler	Fetch public pages; the ai-train=no signal says they may not be used for training.
`Claude-SearchBot`	Anthropic, search index	Index public pages so Claude can cite them in search answers.
`Claude-User`	Anthropic, on a user's request	Fetch a public page live to answer that person's question (ai-input).
`ClaudeBot`	Anthropic, general crawler	Fetch public pages; the ai-train=no signal says they may not be used for training.
`PerplexityBot`	Perplexity, search index	Index public pages so Perplexity can cite them.
`Perplexity-User`	Perplexity, on a user's request	Fetch a public page live to answer that person's question (ai-input).
`Googlebot`	Google Search	Index public pages for Search, AI Overviews and AI Mode (search, ai-input).
`Google-Extended`	Google, Gemini training control	Fetch is allowed like everyone else; the ai-train=no signal says our content is not for model training.
`Bingbot`	Microsoft Bing and Copilot	Index public pages for Bing search and Copilot answers (search, ai-input).

All of them get the Markdown alternates above, so an agent that reads this page can fetch the Markdown instead of parsing HTML.

What crawlers see on shared links#

Shared file pages (/p/…) and albums (/b/…) are marked noindex, so they don't show up in search results.
Password-protected links show a generic title and no preview image to crawlers.

For agents#

Agents that want to do something, not just read, should use the MCP server or the API, and can find both through the discovery files.

PreviousAgent discovery

Last updated 4 October 2026. Something unclear or missing? Tell us.