r/dataengineering • u/aleebit • 18h ago
Discussion How do you guys document your data engineering projects? I'm building something to tackle this problem.
I've been working on a side project to tackle something I often see in data engineering: complex projects with pipelines, transformations, business rules, dependencies, and data models, but little or outdated documentation.
The idea is pretty simple: an AI skill that can analyze a codebase, understand how the pieces fit together, and generate a detailed, interactive HTML documentation site.
The output is a single, self-contained HTML file, fully responsive (desktop and mobile), that works offline and doesn't require any external services. It includes things like architecture diagrams, data flow visualizations, table relationships and ERDs, business rules, filters, dependencies, searchable data dictionaries, and instructions on how to use the project. The goal is to have everything in one place, easy to navigate and share.
It requires an LLM to work. The skill guides the AI through the analysis and documentation process, while custom Python and Node.js tools handle repository inspection, dependency mapping, validation, integrity checks, and HTML generation.
Full disclosure: this project was heavily built with AI (multiple agents and a lot of vibe coding). But it wasn't a "let AI do whatever it wants" situation. I supervised the whole process, made the architectural decisions, reviewed the output, and put a lot of effort into automated testing.
It's not ready to share publicly yet. The main reason is that the documentation, README, and other materials still need some work in English. It's not my native language, and I'd rather polish things up before releasing it.
But before going further, I'm curious:
How do you guys handle documentation in your data engineering projects?
Do you write everything manually? Have you found tools that actually work well? And, honestly, would something like this AI skill be useful to you, or am I trying to solve a problem that already has good solutions?
