How DataLynxr works under the hood.
Stateless compute. Decoupled storage. One catalog to enforce ACID across all workload types. No shared mutable state between SQL, streaming, and ML nodes.
The full-stack architecture
What each layer does
Compute Layer — Stateless nodes
SQL, streaming, and ML compute nodes are stateless containers. They autoscale independently — more SQL queries spin more SQL nodes; a Kafka backlog scales streaming nodes. They never share in-memory state with each other.
Catalog + Transaction Coordinator — One ACID source of truth
Every read and write — from any workload type — passes through the catalog transaction coordinator before touching storage. Snapshot isolation is enforced here. This is what makes ACID semantics possible across SQL, streaming, and ML simultaneously.
Open Table Format Layer — Iceberg, Delta, Hudi
DataLynxr reads and writes all three major open table formats. The format stores metadata (partition layout, schema, column statistics) that the catalog uses for query planning and data skipping.
Your Cloud Object Storage — Nothing leaves your account
All data lives in your S3, GCS, or ADLS bucket under your access controls and bucket policy. DataLynxr uses your cloud identity (IAM role / service account) for all storage operations. We never see credentials, never copy data, never charge for storage.
Why we built it this way
Why stateless compute?
Stateful compute nodes create coordination overhead when multiple workload types access the same table. Stateless nodes eliminate node-to-node coordination — the catalog is the single coordination point.
Why one catalog, not three?
Separate catalogs for SQL, streaming, and ML would require a sync layer to keep them consistent. That sync layer is the ETL problem recreated inside the platform. One catalog, one truth.
Why open table formats?
Proprietary formats create lock-in. Iceberg, Delta, and Hudi are readable by dozens of other tools — Spark, Trino, Flink, DuckDB. You're not locked into DataLynxr as your only reader.
Read the technical docs.
Architecture explainers, catalog internals, and storage connector design in our documentation.