Skip to content
← All work

Self-healing regulatory pipeline

193 legal sources across 130 jurisdictions, scraped by fetchers an AI writes and repairs behind a deterministic harness.

Role
Senior engineer — architecture, fetcher toolchain, LLM stages
Stack
Python 3.12 Airflow 3 Kubernetes Anthropic LLMs PostgreSQL MongoDB Docling

Problem

Regulation lives on a catalogue of roughly 290 official and legal portals spread over 130 jurisdictions. None of them have an API, all of them change their markup without warning, and the failure is silent by construction: the site is redesigned, the selectors match nothing, and the run reports success with zero items found. Three fetchers existed, hand-written at about 170 lines each. Writing two hundred more by hand does not scale — and every one written is one more thing that will break.

Approach

The observation the whole design rests on is that healing a broken fetcher and generating a new one are the same operation with different inputs: a source row, plus failure evidence when there is any, in — a module, its tests, its registration and a pull request out. So there is one machine with two modes, and the trust boundary is not the model's self-assessment but a deterministic harness: seven stages, cheap to expensive — wiring, an AST whitelist, lockfile untouched, lint and types, the unit suite, protocol conformance checked by signature, and finally one budgeted live fetch judged by the same health rules the monitoring uses. Mode choice is measured rather than guessed by a probe that spends at most eight requests on robots.txt, anchor metrics, SPA markers and feed autodiscovery, then decides against a written table. Every pull request is reviewed by a human; nothing auto-merges, not even a green heal.

Outcome

193 of the catalogued sources are covered, 153 fetcher modules live in the repository, and report processing takes the legal team 70% less time than before. A daily health job marks a fetcher broken after three consecutive failures, or degraded after three zero-yield runs against its own positive baseline, and hands the worst two to the healer — which opens a reviewed pull request instead of paging anybody.

Discovery Source catalogue ≈290 sources · 130 jurisdictions 193 covered · none has an API 153 fetcher modules http · api · browser generated and healed by AI Deterministic harness 7 stages · human review · no auto-merge the agent holds no credentials Extraction · daily 00:00 scrape every due source PDF extraction Docling ESG gate is it relevant? Insights · hourly ingest Mongo → process English translation · summary · ESG tags bundles: intensity, scope, key dates, anti-ESG, SICS industries every value carries reasoning + evidence quote publish → Mongo Around it classify_document_type every 2h · 3-tier resolver rules → principles → LLM compare_live_regulations daily 05:00 · the evaluation harness scores against the reg team’s truth discovery_health daily 06:00 · marks broken/degraded triggers at most 2 heals
The regulatory pipeline’s five DAGs and what each one feeds. Discovery and health close the loop over the same fetcher layer.
1 · Detection 3 failures in a row → broken 3 zero-yield runs vs. its own baseline → degraded at most 2 per run 2 · Evidence the source row the last 10 runs the newest positive success (the yield baseline) 3 · Caged agent no credentials, no git write, no curl, no free fetch cannot edit its own cage diff vs. mechanical allowlist 4 · Human review the PR carries the harness JSON, probe facts and cost no auto-merge, ever The harness — cheap to expensive, between step 3 and step 4 1. wiring 2. AST whitelist 3. lockfile 4. lint + types 5. unit suite 6. protocol 7. live fetch a bad generation dies in seconds, not after hitting a site that hosts laws
Generating a fetcher and healing a broken one are the same operation. The agent proposes, the harness decides, a human approves.

The claim people ask about

“Two hundred scrapers in a week with AI” is the line that makes experienced engineers suspicious, and they are right to be. It is not two hundred bespoke parsers written by a chatbot. It is a generator — a repository skill invoked by an agent, interactively on a laptop or headless in CI — producing modules against a deliberately narrow port interface, each one verified by a harness that the agent cannot talk its way past. The leverage is not the model. The leverage is that the acceptance criteria are executable.

The harness is the product

Seven stages, ordered cheap to expensive so a bad generation dies in seconds rather than after a live request:

  1. Wiring — is the module registered where the loader will find it?
  2. AST whitelist — imports checked, no async, no writes outside its lane.
  3. Lockfile untouched — a fetcher may not add a dependency.
  4. Lint and types.
  5. Unit suite against an HTML snapshot captured during probing.
  6. Protocol conformance by inspect.signature — a structural check that only compares names is not a check.
  7. One budgeted live fetch, judged by the same rules the health monitor uses, with a per-source cooldown so iteration cannot hammer a site that hosts laws.

robots.txt is parsed properly, per RFC 9309, and a disallow is a refusal the agent has to accept — it is never argued around.

The agent runs in a cage

In CI the agent holds no credentials: the bot token, the AWS credentials and the OIDC token are stripped from its environment before it starts, and it gets an explicit tool allowlist with no git write, no curl, no arbitrary fetch. The cage is checked twice — a pre-execution hook refuses to run anything once a forbidden path is dirty, and the publish step compares the final diff against a mechanical allowlist, so an agent that got prompt-injected still cannot edit the file that defines its own limits.

Detection, and knowing when not to try

A daily job reads each fetcher’s run history: three consecutive failures make it broken; three successful runs that return nothing, measured against the newest positive result however old, make it degraded. At most two candidates are healed per run, deduplicated statelessly against Bitbucket so a source with a heal pull request touched in the last week backs off. Sources already carrying a human verdict that no code change can satisfy — needs credentials, needs a browser, not actually a source — are still monitored but never offered to the healer. With only two slots, one unfixable source retried forever would starve every fixable one.

What I can and cannot show

Under NDA: the shape and the reasoning, never customer data, prompts or internal diagrams.