Engineering

Home

Enzyme@

Databricks

SQL Query Accelerator

Incremental Updates In Materialized Views

LIQUID

@

LinkedIn

Query Evaluation Engine

I led the design of the LIquid Distributed Declarative Query Evaluation Layer. The goal was to let customers write what they want without specifying how, while preserving speed, correctness, and scalability across billions of data points.

The system architecture includes:

  • Hydra: a graph-structured in-memory engine for intermediate query states

  • Planner: transforms declarative logic into optimized low-level operations

  • Constraints: enforce semantic correctness and path consistency

Each shard plans and executes locally using a hill-climbing cost-based planner, enabling high-throughput, distributed query evaluation.

Key innovations: Hydra for Path Queries, Budgeted Planning Strategy
Impact: Powered LinkedIn’s transition to expressive, high-performance declarative queries that boosts scalability and developer velocity across core data systems.
Patent: US11704309B2

Storage Engine

I co-designed LIquid Storage Layer, a scalable, memory-efficient backend for LinkedIn’s distributed graph query engine. The system supports real-time ingestion via Kafka and fast, concurrent reads through layered indexes over a persistent graph log.

The architecture supports branching for snapshot isolation, multi-level hashmaps and VList stores for fast lookups, and compound indexing for managing complex relationships (like Endorsements or Connections). The system also enables one-hop and two-hop graph traversals, supporting both imperative and declarative query models.

Key Innovations:

  • Compound Indexing with High Cardinality Stores — optimized for real-world graph relationships with frequent mutations

  • Branching for Snapshot Queries — allows isolation of queries without exposing future writes

  • Parallel and Lazy Compaction — improves read efficiency without blocking writers

  • Read API Tiering — flexible lookup/materialization strategy for fast, selective data access

Impact: This system became the foundation for LIquid's scalable query evaluation layer, serving production traffic with strong guarantees around performance, consistency, and data isolation. It significantly boosted developer velocity, supported expressive relationship modeling, and powered core LinkedIn features dependent on graph traversal.
Patent: US11567995B2
Paper: Nanosecond Indexing of Graph Data with Hash Maps and VLists