Robots, LLMs, & OpenTerms

AI Permission Sources Overview

Publishers expose several kinds of machine-readable evidence, but they do not all carry the same meaning. Robots.txt is a crawler-control standard, llms.txt is primarily a content map, and OpenTerms can carry explicit permission values. AgentPermission evaluates the available signals without treating them as a universal legal answer.

1. Robots.txt

Created in 1994, robots.txt is the original standard for managing web crawlers. Located in a website's root (for example, https://example.com/robots.txt), it expresses path-access preferences for named user-agents. It is not account authorization or legal advice.

Robots.txt in the AI Era

Today, robots.txt is used to block AI scrapers specifically. Site administrators block user-agents like GPTBot, ClaudeBot, and Google-Extended to protect content from being used in AI training datasets.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

2. Llms.txt

The llms.txt proposal is primarily a Markdown map of important content for language-model readers. Located at https://example.com/llms.txt, it can provide a title, summary, and curated links. That discovery content does not imply permission and is not, by itself, a formal permission protocol.

Format and Purpose

AgentPermission records a normal title, summary, and link map as discovery evidence, but it does not turn that content into an allow decision. Only explicit permission-like key/value lines inside a clearly labeled policy section can affect a decision.

# Example Documentation
> A curated map for machine readers.

## API Documentation
- [API Reference](/docs): Request and response formats.

## Permissions
- summarize: allow
- train: deny

3. OpenTerms

OpenTerms is a machine-readable JSON term-sheet format. AgentPermission looks for it at https://example.com/.well-known/openterms.json and evaluates explicit action permissions declared by the publisher.

OpenTerms JSON Structure

The policy engine recognizes read, scrape, train, transact, post, write, and API-access values expressed as allow, deny, or escalate (with booleans normalized where supported).

{
  "openterms_version": "1.0",
  "permissions": {
    "read_content": "allow",
    "scrape_data": "escalate",
    "train_data": "deny",
    "post": "escalate",
    "transact": "escalate"
  }
}

Synthesizing Rules with AgentPermission

The AgentPermission API evaluates robots.txt, llms.txt, and /.well-known/openterms.json in one preflight query. The result records which sources were available, how they were interpreted for the requested action, and whether your integration should continue, stop, or ask a human.