AI crawlers, Markdown and content signals

How AI crawlers can read our pages: Markdown on request, Content Signals in robots.txt and what we allow.

AI search tools and assistants read the web on your behalf. We make our public pages easy for them to read and cite, and we state clearly what they may not do with them.

Markdown on request#

Send Accept: text/markdown to any public page (home, pricing, guides, blog, these docs) and you get a clean Markdown version instead of HTML. Browsers never ask for Markdown, so people always see the normal page.

Shell

curl -H "Accept: text/markdown" https://linkinseconds.com/pricing

Response header	Meaning
`Content-Type: text/markdown`	The body is Markdown.
`x-markdown-tokens`	A rough token count, so an agent can budget its context.
`Vary: Accept`	Caches keep the HTML and Markdown versions apart.

The home page answers with llms.txt, our curated summary of the site. Private pages (the dashboard, sign-in) are never converted: they answer 406 Not Acceptable. Files people share on /p/… links are never converted either.

robots.txt and Content Signals#

robots.txt (excerpt)

User-Agent: *
Allow: /
Disallow: /dashboard
Disallow: /api/
Content-Signal: search=yes, ai-input=yes, ai-train=no

Signal	Our choice	Meaning
`search`	yes	Search engines may index and link to our pages.
`ai-input`	yes	AI assistants may read our pages to answer a question and cite us.
`ai-train`	no	Nothing here, including files our users share, may be used to train AI models.

Your shared files

Links you share stay readable by anyone you send them to. The ai-train=no signal tells AI companies that respect Content Signals not to train on them. Signals are a stated preference, not a technical lock: for files that must stay private, use a password or an email gate (see Link settings).

What crawlers see on shared links#

Shared file pages (/p/…) and albums (/b/…) are marked noindex, so they don't show up in search results.
Password-protected links show a generic title and no preview image to crawlers.

For agents#

Agents that want to do something, not just read, should use the MCP server or the API, and can find both through the discovery files.

PreviousAgent discovery

Last updated 4 October 2026. Something unclear or missing? Tell us.