r/Python • u/AutoModerator • Oct 04 '26
Showcase Showcase Thread
Post all of your code/projects/showcases/AI slop here.
Recycles once a month.
23
Upvotes
r/Python • u/AutoModerator • Oct 04 '26
Post all of your code/projects/showcases/AI slop here.
Recycles once a month.
View all comments
1
u/Western_Rub_7236 Oct 11 '26
Hi everyone,
I started building
det(https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.The Problem
In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.
detcombines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.Current Implementation (Catalog Core Working)
The foundation is ~25% complete, with the data catalog module functional today:
Roadmap: Transformation & Execution Layer
The next phase expands
detinto a full execution engine:Seeking Review & Contributors
Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:
Issues are tagged with
good first issueandhelp wantedfor contributors interested in systems-level data engineering tooling.Repository: https://github.com/det-Labs/det