[Docs](https://linkinseconds.com/docs)AI agents and MCP # AI crawlers, Markdown and content signals How AI crawlers can read our pages: Markdown on request, Content Signals in robots.txt and what we allow. AI search tools and assistants read the web on your behalf. We make our public pages easy for them to read and cite, and we state clearly what they may not do with them. ## Markdown on request Send `Accept: text/markdown` to any public page (home, pricing, guides, blog, these docs) and you get a clean Markdown version instead of HTML. Browsers never ask for Markdown, so people always see the normal page. Shell ``` curl -H "Accept: text/markdown" https://linkinseconds.com/pricing ``` | Response header | Meaning | | `Content-Type: text/markdown` | The body is Markdown. | | `x-markdown-tokens` | A rough token count, so an agent can budget its context. | | `Vary: Accept` | Caches keep the HTML and Markdown versions apart. | The home page answers with [llms.txt](https://linkinseconds.com/llms.txt), our curated summary of the site. Private pages (the dashboard, sign-in) are never converted: they answer `406 Not Acceptable`. Files people share on `lisview.com` links are never converted either. ### Finding the Markdown version without negotiating Every public page also has a direct Markdown address at `/agent-markdown/` followed by the page's own path, and every public HTML response advertises it twice, so a crawler that only follows links finds it too: - An HTTP `Link` response header on the HTML page, for example on `/pricing`: Response header on https://linkinseconds.com/pricing ``` Link: ; rel="alternate"; type="text/markdown" ``` - The same relation in the page ``: HTML ``` ``` The home page's alternate is `/agent-markdown` (which serves llms.txt); a localized page keeps its prefix, so `/es/pricing` points at `/agent-markdown/es/pricing`. The direct address answers `text/markdown` whatever `Accept` header you send, is marked `noindex`, and gives `406` for any page that has no Markdown version. Pages without an alternate (the dashboard, sign-in, shared files) carry neither the header nor the tag. ## llms.txt and llms-full.txt | File | What it is | When to fetch it | | [/llms.txt](https://linkinseconds.com/llms.txt) | A short, curated index: what the product does, the authoritative limits and prices, and one line per public page. | To decide which page answers a question, in one small fetch. | | [/llms-full.txt](https://linkinseconds.com/llms-full.txt) | The full text: the same summary and facts, then the English copy of every landing page (hero, answer, limits, sections, comparison tables), then the docs and guides as described links. Around 300 KB, capped near 400 KB. | To read the whole marketing site at once instead of crawling 50 pages. | Both are generated from the same registries and plan limits as the pages themselves, so they can never carry a stale price or a page that no longer exists. Both are plain UTF-8 text, cached for an hour. ## robots.txt and Content Signals robots.txt (excerpt) ``` User-Agent: * Allow: / Disallow: /dashboard Disallow: /api/ Content-Signal: search=yes, ai-input=yes, ai-train=no ``` | Signal | Our choice | Meaning | | `search` | yes | Search engines may index and link to our pages. | | `ai-input` | yes | AI assistants may read our pages to answer a question and cite us. | | `ai-train` | **no** | Nothing here, including files our users share, may be used to train AI models. | Your shared files Links you share stay readable by anyone you send them to. The `ai-train=no` signal tells AI companies that respect Content Signals not to train on them. Signals are a stated preference, not a technical lock: for files that must stay private, use a password or an email gate (see [Link settings](https://linkinseconds.com/docs/link-settings)). ### Crawlers we allow Our robots.txt has one rule set, `User-Agent: *`, so every well-behaved crawler gets the same access: all public pages, never the dashboard, sign-in, or the API. The Content-Signal line above applies to all of them. The crawlers we see most, and what each may do here: | User agent | Who | Allowed to | | `OAI-SearchBot` | OpenAI, ChatGPT search index | Index public pages so ChatGPT search can link to and cite them. | | `ChatGPT-User` | OpenAI, on a user's request | Fetch a public page live to answer that person's question (ai-input). | | `GPTBot` | OpenAI, model training crawler | Fetch public pages; the ai-train=no signal says they may not be used for training. | | `Claude-SearchBot` | Anthropic, search index | Index public pages so Claude can cite them in search answers. | | `Claude-User` | Anthropic, on a user's request | Fetch a public page live to answer that person's question (ai-input). | | `ClaudeBot` | Anthropic, general crawler | Fetch public pages; the ai-train=no signal says they may not be used for training. | | `PerplexityBot` | Perplexity, search index | Index public pages so Perplexity can cite them. | | `Perplexity-User` | Perplexity, on a user's request | Fetch a public page live to answer that person's question (ai-input). | | `Googlebot` | Google Search | Index public pages for Search, AI Overviews and AI Mode (search, ai-input). | | `Google-Extended` | Google, Gemini training control | Fetch is allowed like everyone else; the ai-train=no signal says our content is not for model training. | | `Bingbot` | Microsoft Bing and Copilot | Index public pages for Bing search and Copilot answers (search, ai-input). | All of them get the Markdown alternates above, so an agent that reads this page can fetch the Markdown instead of parsing HTML. ## What crawlers see on shared links - Shared file pages (`lisview.com/…`) and albums (`lisview.com/b/…`) are marked `noindex`, so they don't show up in search results. - Password-protected links show a generic title and no preview image to crawlers. ## For agents Agents that want to **do** something, not just read, should use the [MCP server](https://linkinseconds.com/docs/mcp) or the [API](https://linkinseconds.com/docs/api/uploads), and can find both through the [discovery files](https://linkinseconds.com/docs/agents). [PreviousAgent discovery](https://linkinseconds.com/docs/agents) Last updated 4 October 2026. Something unclear or missing? [Tell us](https://linkinseconds.com/contact). Source: https://linkinseconds.com/docs/agents/crawling