Web-Ingestion Evidence Receipts

Crawler logs often show what was fetched without showing which publisher signals were considered first. Evidence receipts capture that pre-ingestion decision.

Why are crawler logs not enough?

Useful ingestion evidence is tied to a URL hash, a declared purpose, a timestamp, and hashes of the policy sources used by the system.

Tamper-evident evidence, not marketing claims

A receipt is useful because it is specific. It records a target URL hash, purpose, decision, timestamp, source hashes, and service signature material your team can store with the ingestion record.

Receipt storage example
async function recordPermissionReceipt(ingestionJobId: string, checkResult: any) {
  await ingestionJobs.update(ingestionJobId, {
    permissionDecision: checkResult.decision,
    permissionReasons: checkResult.reasons,
    permissionReceiptId: checkResult.receipt?.receipt_id,
    permissionReceiptIssuedAt: checkResult.receipt?.issued_at
  });
}

Attach receipts to the ingestion job

The receipt should travel with source metadata so data governance and engineering teams can reconstruct why the pipeline fetched or skipped a URL.

  • Run a check for the exact target URL and ingestion purpose before fetching content.
  • Persist the decision, reasons, source evidence summary, and receipt ID with the ingestion job.
  • Keep secret-bearing query parameters out of submitted URLs and internal review tools.
  • Use the receipt as the starting point when a source is skipped, escalated, or admitted into a corpus.

Why receipts matter

  • Data review starts from the pre-ingestion decision instead of an incomplete crawler log.
  • Teams can distinguish policy skips from downstream fetch failures.
  • Governance reviewers get a concrete artifact tied to the source and declared use.