Feeding a Delta Sync Vector Index into Genie: Closing the Structured/Unstructured Gap

The gap nobody mentions in the Genie demos

Every Genie demo looks the same: point it at a clean sales table, ask "what were top products last quarter," get a chart back. What the demos skip is that Genie only reasons over structured tables. If your business question needs a contract clause, a support ticket transcript, or a PDF spec sheet, Genie has nothing to say — someone in the Databricks community hit this directly while trying to build combined search across PDFs and Delta tables, and the answer came back plainly: Genie only operates with structured data, so it's not suitable for unified search that includes PDF-extracted text.

That's not a knock on Genie. Text-to-SQL and RAG are different problems with different failure modes. But most real business questions blend both — "show me revenue by account, and pull the relevant clauses from their contract" — so if you're building on Genie you eventually need a second retrieval path for the unstructured half.

What a Delta Sync index actually gives you

A Delta Sync Index is the vector-search side of this. It computes embeddings from a text column using a hosted model and keeps itself in sync with table changes via the Change Data Feed, so as source rows are added, updated, or deleted, the corresponding vector index is automatically updated to match. Under the hood, Vector Search manages failures, handles retries, and optimizes batch sizes without any input from the engineering team.

Two things matter if you're setting one up. First, Change Data Feed has to be turned on for the source table — ALTER TABLE ... SET TBLPROPERTIES (delta.enableChangeDataFeed = true) — because the sync mechanism reads the CDF stream rather than rescanning the whole table each time. Second, write pattern changes the sync behavior: an OVERWRITE replaces the table wholesale, so the index typically treats it as a full refresh, while a MERGE only recalculates the rows that actually changed. If your ingestion job does periodic full overwrites instead of incremental merges, expect the sync to be slower and more expensive than it needs to be.

Wiring the two paths together

The practical pattern is two lanes feeding one conversational surface:

• Structured lane: a Genie space pointed at curated Delta tables or, better, a metric view rather than raw tables. Community guidance on this is consistent — pointing Genie directly to a raw table is fine for a proof of concept, but as the schema grows, Genie is left guessing what columns mean and making mistakes, which is exactly the job a metric view or well-documented table is meant to remove.
• Unstructured lane: a Delta Sync index built from your parsed documents, wrapped in a retriever tool. Databricks' own agent framework docs show the pattern with VectorSearchRetrieverTool, pointing at an index name in catalog.schema.index form and returning the columns you need for grounding.

An orchestrator agent — or a simple router that classifies the incoming question — decides which lane to hit, or hits both and merges the results before answering. This mirrors what Databricks itself has been shipping externally: Genie exposed over MCP so external orchestrators (their Adobe integration is one public example) can call it as a structured-data tool alongside other retrieval tools in the same agent loop.

Where this actually breaks

The failure mode isn't the wiring, it's curation debt on both sides. Genie's quality is bounded by the room curator's discipline, and without someone who owns the semantic layer, natural-language quality drifts over time. The vector side has the mirror problem — stale embeddings from a full-overwrite job that should have been an incremental merge, or a text column doing double duty as both display data and the embedding source. Neither Delta Sync nor Genie fixes bad source data; they both just make it faster to be confidently wrong. Budget the same ongoing attention for the metric view and the chunking strategy as you would for the model choice.