Skip to content
← All work

News intelligence pipeline

20,000 news items a day, embedded, clustered into stories and enriched by LLMs — at 15% of the bill it used to run on.

Role
Senior engineer — design, implementation and operation
Stack
Python 3.12 Airflow 3 Kubernetes pgvector PostgreSQL MongoDB LLMs AWS

Problem

Enterprise customers need to know which news actually touches them. The raw feed is around twenty thousand items a day and heavily duplicated: the same story rewritten by fifteen outlets, none of them adding anything. Volume was the product's biggest liability — more input meant more noise, and running an LLM over every item meant paying fifteen times for one story.

Approach

A staged pipeline on Airflow 3 over Kubernetes, ordered so the cheap deterministic work happens before the expensive probabilistic work. Every article headline is embedded into a 1536-dimension vector stored in Postgres with pgvector; articles are grouped by cosine distance under a tuned threshold, and each cluster elects a representative by source importance. Only then do the LLM insight stages run — over stories, not articles. A second clustering pass merges stories across days, so a topic that runs for a week stays one thread instead of seven. Model and prompt choices are decided by evaluation against the values the analyst team validated, with cost sitting in the column next to accuracy.

Outcome

Twenty thousand items a day flow through unattended, and the pipeline's annual LLM bill came down by 85% — from a six-figure sum to a five-figure one. Clustering is what did most of it: paying once per story instead of once per article, then letting evaluation move stages onto smaller models wherever the numbers said quality held.

Daily DAG · Airflow 3 over Kubernetes scan ≈20,000 articles/day adapt to the domain embed headline → 1536-d daily cluster cosine · pgvector insights LLM per story multi-day merge still the same story? → export Why the order matters fifteen outlets rewrite the same story embedding a headline: cents per thousand classifying it with a frontier model: not −85% annual bill six figures down to five
Cheap and deterministic before expensive and probabilistic: you pay once per story, not once per article.

Cheap work before expensive work

The single decision that paid for everything else was ordering the stages by cost. Embedding a headline is cents per thousand; classifying an article with a frontier model is not. Collapsing fifteen near-duplicate articles into one story before the LLM stage means the expensive step runs on a fraction of the input — and it improves the output too, because the analyst cares about the story, not about which outlet filed it first.

Clustering, concretely

Headlines are embedded rather than full bodies: a 1536-dimension vector per article, written in batches to keep the database from becoming the bottleneck, and queried with pgvector’s cosine distance operator under a threshold tuned to sit between “splits one story in two” and “merges two stories into one”. A cluster’s representative article is chosen by source importance, so the story that surfaces is the one from the outlet that matters.

Then the same machinery runs a second time over a multi-day window. Daily clustering answers “what happened today”; the multi-day pass answers “is this still the same story as yesterday”, which is the question a customer tracking a regulation actually has.

Evaluation, not vibes

Model choice is the least interesting decision in an LLM pipeline. What makes a stage shippable is being able to answer “is this better than what we had?” before a customer sees it — which means a labelled set drawn from decisions the analyst team already validated, and a score per candidate. Once that exists, moving a stage to a smaller model is a one-line experiment with a number attached, and that is where most of the 85% came from.

What I can and cannot show

Under NDA: shape and outcome, never customer data, prompts or internal diagrams.