August 16, 2026
A Research Pipeline That Filters Vendor Marketing From Real Architecture Patterns
A source-registry-driven pipeline that watches Snowflake, Databricks, dbt, Spark, Qlik, Redshift, MS Fabric, and n8n for genuine architecture changes -- not marketing -- and surfaces them at /research, through the chat widget's RAG, and via a public MCP tool.
Problem
Vendor changelogs and engineering blogs for the modern data stack are mostly marketing wrapped around the occasional genuine architectural change -- pricing and GA announcements, customer case studies, partnership posts crowding out the incremental-modeling techniques, semantic layer designs, and orchestration strategies actually worth tracking. Reading all of it by hand across eight vendors doesn't scale, and a generic AI news summarizer would just launder the marketing faster, not filter it out.
Approach
A source-registry table (vendor, feed URL, poll cadence -- config, not code) feeds three scoped fetchers (RSS, GitHub releases, a changelog-scraping fallback) into a raw items table. Each new item runs through a Claude classification call with an explicit reject list -- pricing, GA, case studies, partnerships -- before anything gets summarized, so the model judges substance over tone. Survivors are condensed into a structured finding (what changed, why it matters for pipeline design, which stack components it touches) and deduplicated by embedding similarity on the summarized text, not the raw item -- vendors reuse near-identical titles across a changelog entry, a blog recap, and a GitHub release in ways title-matching alone misses. Findings land in Postgres for review, not auto-published. Approval is the only publish step -- the content is already the model's structured output being vouched for or rejected, so there's no separate drafting stage. Approved findings go live immediately at /research, join the site's existing RAG index, and are queryable through a new search_pipeline_patterns MCP tool with structured vendor/category/date filters plus an optional semantic re-rank. A weekly cron drafts a "Note of the week" post from the week's approved findings, still unpublished until reviewed. One deliberate schema call: research_findings is application-owned, not a dbt materialization -- unlike the GitHub-activity pipeline below, whose marts tables get dropped and rebuilt every run safely because nothing references them by id. Findings get cited by UUID from the RAG index, the audit log, and any digest post the moment they're approved, so that pattern would silently orphan every reference. The classify/summarize pipeline itself runs as a second repo on its own GitHub Actions schedule, same shape as that sibling pipeline, so it can't take this site's own build down with it.
Outcome
The site-side half of this is live in production right now -- schema, /research, admin review and source-registry management, the RAG hook, the MCP tool, the digest cron. The ingestion agent that actually populates it is written and tested locally but not yet deployed, so /research is honestly empty until it's running on a real schedule against real sources.