Looker's Semantic Layer is a Prison. Here's The Escape Key.

TL;DR

• Looker's core abstraction, the semantic layer (LookML), creates a static, manually-maintained bottleneck that slows down data access and burns senior engineering cycles.
• Persistent Derived Tables (PDTs) are a brittle, expensive workaround for Looker's inability to generate efficient queries against modern data warehouses. They're a materialized view with a marketing name and a massive price tag on your Snowflake bill.
• The future isn't another monolithic BI tool. It's AI coding agents (Claude, Cursor) generating clean SQL against a database that has a self-optimizing "brain," eliminating the need for a manual semantic layer.

---

I once watched a team spend three days trying to add a single customer_segment column to a critical revenue dashboard. The DDL change took five minutes. The remaining 71 hours were a death march through a maze of LookML files, untangling dependencies, kicking off PDT rebuilds that took hours, and finally deploying the change in a nail-biting release window. The dashboard updated. The cost? Around $15,000 in engineering time and lost business momentum. For one column.

This isn't a rare horror story; it's the daily reality of working with Looker at scale. Looker, and its paradigm of the centralized semantic layer, was a smart answer to a 2014 problem. The problem was protecting a production Postgres or MySQL instance from analysts running SELECT * without a WHERE clause, triggering a seq scan, blowing out the buffer cache, and taking down the application.

In that world, abstracting the database behind a curated set of 'Explores' and 'Views' made sense. But that world is gone. Your data lives in Snowflake, BigQuery, or Redshift now. These are columnar beasts designed for analytical workloads. Coddling them with a semantic layer isn't protecting them; it's putting a governor on a race car.

The Original Sin: LookML is a Schema Prison

LookML is a parallel, manually-curated representation of your database schema. It’s a git repo that must stay perfectly in sync with your actual DDL, but without any of the database's internal knowledge. Every time a data engineer alters a table, they (or more likely, someone else) must translate that change into LookML.

This creates two sources of truth, a recipe for guaranteed model drift. A BIGINT is changed to a VARCHAR in the database. What happens in Looker? The LookML model, unaware of the change, breaks. Dashboards fail with cryptic errors. An analyst files a ticket. A data engineer investigates. Time is burned. This isn't a graceful failure; it's a brittle system cracking under the pressure of reality.

You now have to hire "LookML Developers." This isn't a transferable skill. You're paying a skilled engineer to become a specialist in a proprietary DSL. You’ve taken a high-leverage engineer and made their primary job fighting a tool, not modeling data.

The semantic layer is a prison for your schema. It locks you into a static view of your data, stifles exploration, and funnels every single change through a human bottleneck.

PDTs: The Million-Dollar CREATE TABLE AS SELECT

If the semantic layer is the prison, Persistent Derived Tables (PDTs) are the solitary confinement cell. PDTs exist for one reason: Looker's generated SQL is often horrifically inefficient. Because it's working from a dumbed-down model of your data (LookML), it can’t generate optimal queries for complex joins. So, it punts. It forces you to pre-aggregate and pre-join your data into a physical table.

A PDT is just a CREATE TABLE AS SELECT ... statement wrapped in proprietary YAML and a flaky trigger mechanism. It's a materialized view without the robustness of a database-native solution. And the cost is staggering.

I’ve seen clients where PDT rebuilds accounted for over 40% of their Snowflake bill for BI-dedicated warehouses. A multi-hour job spinning up a large warehouse to run a massive MERGE statement, all because the BI tool couldn't be trusted to execute a multi-table JOIN efficiently at query time.

Looker's abstractions leak, and the result is often a query plan that no sane DBA would ever approve. The semantic layer obscures the database's own cost-based optimizer, which has access to table statistics (pg_stats in Postgres, similar internals elsewhere) that LookML can only dream of. For instance, the BI tool might generate a query that tricks the planner into a bad decision:

-- A query plan typical of bad semantic-layer-generated SQL
EXPLAIN ANALYZE SELECT ...

                                           QUERY PLAN
-------------------------------------------------------------------------------------------------
 HashAggregate  (cost=194520.82..194520.92 rows=10 width=40) (actual time=2105.41..2105.80 rows=10)
   Group Key: u.region
   ->  Nested Loop  (cost=0.56..193270.82 rows=500000 width=36) (actual time=0.04..1881.51 rows=500000)
         ->  Seq Scan on users u (cost=0.00..24500.00 rows=100000 width=36) (actual time=0.01..82.34 rows=100000)
               Filter: (u.signup_date > '2023-01-01'::date)
         ->  Index Scan using orders_user_id_idx on orders o (cost=0.56..1.60 rows=5 width=8) (actual time=0.01..0.015 rows=5)
               Index Cond: (user_id = u.id)

A senior engineer would spot this instantly. The planner chose a Nested Loop because its estimate for the inner side of the join (rows=5) was catastrophically wrong. It's iterating 100,000 times and doing an index scan each time. A Hash Join would be orders of magnitude faster. This kind of mistake is common when the query is generated by a layer that is divorced from the database's live statistics. You're paying six figures for a tool that writes worse SQL than a junior analyst.

The 2026 Stack: AI Agents + A Database Brain

The fundamental workflow is broken. The solution isn't a better semantic layer or a faster BI tool. The solution is to remove the semantic layer entirely.

Here’s the new workflow. Your CFO Slacks the head of data: "What was our quarterly YoY revenue growth for customers in EMEA on the enterprise plan?"

An LLM coding agent—Claude 3.5 Sonnet, a custom-trained model, whatever—gets the prompt. It has two things Looker never did:

1. True Natural Language Understanding: It grasps the intent behind the user's question.
2. Dynamic, Rich Context: It doesn't query a stale LookML repo. It queries a live system that understands the database schema as it exists right now.

This second piece is the "database brain." It's a system that has ingested the database's DDL, information_schema, foreign key constraints, and—most importantly—the query history (pg_stat_statements, Snowflake's query_history, etc.). It knows which tables are joined most often. It knows that sales.transactions.amount_cents is the canonical column for "revenue." It knows this not because a developer wrote it in YAML, but because it has seen hundreds of successful queries written by your best engineers that use it that way.

The AI agent asks the brain, "How do I calculate 'revenue' for 'enterprise customers'?" The brain responds with the exact tables, columns, join paths, and filter conditions, derived from historical precedent. The agent then generates clean, readable SQL.

An analyst reviews the SQL—a five-second sanity check—and runs it. The whole process takes less than a minute. No tickets, no LookML developers, no PDT builds, no multi-day waits for a one-column change.

This isn't sci-fi. The LLM agents are already here and getting terrifyingly good at code generation. The missing piece is the database brain that provides them with high-fidelity, dynamic context. That's what replaces the static, brittle semantic layer.

What DeepSQL does about this

DeepSQL is built to be this database brain. By continuously analyzing query logs like pg_stat_statements or Snowflake's query_history alongside your schema, it reverse-engineers the semantic concepts latent in your warehouse. It builds a dynamic knowledge graph of your data, automatically identifying core business concepts like 'revenue', 'active users', and 'churn' based on how your team already queries them. When an LLM agent needs to write a query, DeepSQL provides it with this rich, query-proven context—the exact tables, columns, and join logic derived from thousands of past successful queries. This turns a generic LLM into a specialized data analyst, eliminating the need for a manual semantic layer entirely.