RAG Ingestion Rules

RAG Ingestion & Training-Permission Checks

Retrieval-Augmented Generation (RAG) and model fine-tuning pipelines constantly ingest content from public and private sources. A preflight lets the pipeline record and enforce the machine-readable signals available before content enters durable storage.

RAG Ingestion Preflight

Before feeding target pages or PDF files into vector databases (like Pinecone, pgvector, or Chroma), run a preflight check using the summarize or train actions.

import { Document } from "langchain/document";

async function ingestUrlToVectorStore(url: string, store) {
  // 1. Run permission check with the "train" / "summarize" action
  const preflightRes = await fetch("https://agent-permission-api-788656703262.us-central1.run.app/v1/check", {
    method: "POST",
    headers: {
      "Authorization": "Bearer apk_live_your_key",
      "Content-Type": "application/json"
    },
    body: JSON.stringify({ url, action: "train" })
  });
  
  const { decision, reasons } = await preflightRes.json();
  
  if (decision !== "allow") {
    console.warn(`Ingestion skipped for ${url}. Reason: ${reasons.join(', ')}`);
    return;
  }
  
  // 2. Fetch and split content since action is allowed
  const content = await fetchPageContent(url);
  const doc = new Document({ pageContent: content, metadata: { source: url } });
  await store.addDocuments([doc]);
}

AI Training Consent Signals

The AgentPermission engine synthesizes multiple training consent formats:

• robots.txt User-Agent rules: Evaluates blocks like Disallow: / for user-agents like Google-Extended, GPTBot, Claude-Web, etc.
• llms.txt evidence: Records the content map and uses only explicit recognized permission directives. A normal title, summary, or link does not grant permission.
• OpenTerms block metadata: Analyzes structured term sheets defining machine data rights and model training consent.

Clean Data Pipelines

Adding a permission preflight gives the pipeline:

• Controlled Vector Stores: Your loader can exclude content when available signals deny the action or when strict mode returns escalate.
• Training evidence: Signed receipts record which machine-readable signals supported or blocked each training decision.
• Conservative interpretation: Informational discovery files stay evidence rather than being promoted into implicit permission grants.