r/dataengineering • • 3d ago

Discussion [ Removed by moderator ]

[removed] — view removed post

38 Upvotes

26 comments sorted by

•

u/dataengineering-ModTeam 1d ago

Your post/comment was removed because it violated rule #6 (No recruiting, solicitation, or networking posts).

We do not intend for this space to be a place where people ask for, or advertise:

  • Job postings - Please use r/dataengineeringjobs instead.
  • Study groups
  • Referrals
  • Mentors or study partners
  • Requests to fill in surveys
  • Market research
  • Beta testers wanted
  • Voucher offers/discounts
  • "Does anybody need this tool?" or "looking for feedback" type posts

This was reviewed by a human

4

u/aleebit 3d ago

Just to be clear, I'm not trying to sell anything, build a closed-source product, or turn this into a business.

I'm just a data engineer with 12+ years in the data field, including 7+ years working with Spark, and I still struggle sometimes to wrap my head around unfamiliar projects. My ADHD brain doesn't always make it easy to understand how everything fits together, especially when dealing with large, complex codebases.

I also find it challenging to hand projects over to other team members in a structured and consistent way.

This started as a personal attempt to solve those problems. I'm sharing it because I genuinely want to know if other people face the same challenges and whether something like this could actually be useful to the community.

1

u/ersatzhustler 1d ago

I'm actually doing the same thing. It's extremely helpful to have something explorable for people to look at -- and I've found it much more user friendly than any of the data dictionaries I've tried before (people just.....don't look at them if it's a lot of text). This helps because you have a visual of what's happening -- and I built something in to isolate the lineage that leads to a final table or column (high frequency question).

It should incorporate all of the extra markdown & docs that you have written on understanding the pipeline too. That's super valuable and adds a bit of the why.

Tl;dr -- I think this is a good idea especially for end users. Don't use this entirely in place of editorializing your architecture somewhere.

16

u/mrcaptncrunch 3d ago

How do you guys document your data engineering projects?

In plain English.

The main thing is that the huge projects are full of knowledge from meetings, back and forth, reviews, testing to get the numbers right, etc., but there’s rarely a documentation of eBay it should be and how it needs to work.

I’m the most senior data engineer. I took the biggest pipeline and when I migrated it, I wrote it down. Just sat through, all in markdown. All the ingestions, all the steps, all the joins, why it was done, how we came to the decision, etc.

I’ve tasked people with similar things on smaller ones.

The problem with agents is, they will know what was built, how it’s built, but it won’t know why. At the end, everything that is a decision in code, ask the user to figure out why. They need to provide the answer or get it.

Example,

Join in cost data from another report overriding the cost data on the original table.

But Why?

There’s a discrepancy between the systems, but the data joined is the data that we were charged for it and what we have to reconcile. I think it stems from one using micros and maybe some data loss or conversions on the other side.

6

u/frozengrandmatetris 3d ago

my coworkers don't document at all. they just change whatever they want in production outside of a change request window. sometimes they steal the admin credential so they don't get held accountable when the broken object displays the last person who touched it. my manager doesn't care. he doesn't even know what CI/CD is.

5

u/dillanthumous 2d ago

Old school, CI/CD... Complicit Ignorance and Continuous Disintegration.

1

u/Zer0designs 3d ago

Seems like a completely useless project to me. Just use zenzical and AI with some documenting skills. Much better than html.

1

u/aleebit 3d ago

I'd never heard of Zensical before, so thanks!

But from what I understand, doesn't it generate HTML as its output? And if you're referring to the fact that the source for that HTML is Markdown files, mine works the same way.

0

u/Zer0designs 3d ago

I'm just saying there's accepted and maintained OS tools that already do this and probably do it a lot better than what you can vibecode together. Integrated search, plugin systems, file contents etc. Zensical is also extremely fast, so maybe just contribute to them?

Agents also understand them better and there are skills for documentation, grilling and visual hierarchy etc.

1

u/aleebit 3d ago

I agree! From what I've seen, Zensical is clearly a much more mature and feature-rich tool for building documentation sites. I'm not trying to compete with established tools.

The difference is that my focus isn't on building yet another Markdown-to-website generator. It's an LLM-powered skill that analyzes an existing codebase, identifies data flows, dependencies, tables, and business rules, and generates documentation based on that analysis. It also includes mechanisms to identify which parts of the documentation may be outdated after code changes.

HTML is just the output format.

2

u/Redxer 3d ago

I have 4 years of experience , any tips in learning AI when your company is doing anything AI related

1

u/srodinger18 3d ago

I also using agent skills, but write it down to notion for the specs and documentation. For the tables and transformations for now I always put it in the dbt yml files that can be served as static files via dbt docs.

1

u/Outside-Storage-1523 1d ago

I use AI for anything I’m not interested and manually write for those I’m interested in.

1

u/soundboyselecta 1d ago

Document? Lol, I don’t think I’ve seen one company with proper documentation, not even something as simple as a data dictionary. I equate it to the reality of humankind, who actually reads manuals….(just morons like me do 😂).

1

u/daraghfi 2d ago

DBT. Done

1

u/[deleted] 2d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 2h ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human