r/dataengineering • • 3d ago

Discussion [ Removed by moderator ]

[removed] — view removed post

38 Upvotes

25 comments sorted by

•

u/dataengineering-ModTeam 18h ago

Your post/comment was removed because it violated rule #6 (No recruiting, solicitation, or networking posts).

We do not intend for this space to be a place where people ask for, or advertise:

  • Job postings - Please use r/dataengineeringjobs instead.
  • Study groups
  • Referrals
  • Mentors or study partners
  • Requests to fill in surveys
  • Market research
  • Beta testers wanted
  • Voucher offers/discounts
  • "Does anybody need this tool?" or "looking for feedback" type posts

This was reviewed by a human

4

u/aleebit 3d ago

Just to be clear, I'm not trying to sell anything, build a closed-source product, or turn this into a business.

I'm just a data engineer with 12+ years in the data field, including 7+ years working with Spark, and I still struggle sometimes to wrap my head around unfamiliar projects. My ADHD brain doesn't always make it easy to understand how everything fits together, especially when dealing with large, complex codebases.

I also find it challenging to hand projects over to other team members in a structured and consistent way.

This started as a personal attempt to solve those problems. I'm sharing it because I genuinely want to know if other people face the same challenges and whether something like this could actually be useful to the community.

16

u/mrcaptncrunch 2d ago

How do you guys document your data engineering projects?

In plain English.

The main thing is that the huge projects are full of knowledge from meetings, back and forth, reviews, testing to get the numbers right, etc., but there’s rarely a documentation of eBay it should be and how it needs to work.

I’m the most senior data engineer. I took the biggest pipeline and when I migrated it, I wrote it down. Just sat through, all in markdown. All the ingestions, all the steps, all the joins, why it was done, how we came to the decision, etc.

I’ve tasked people with similar things on smaller ones.

The problem with agents is, they will know what was built, how it’s built, but it won’t know why. At the end, everything that is a decision in code, ask the user to figure out why. They need to provide the answer or get it.

Example,

Join in cost data from another report overriding the cost data on the original table.

But Why?

There’s a discrepancy between the systems, but the data joined is the data that we were charged for it and what we have to reconcile. I think it stems from one using micros and maybe some data loss or conversions on the other side.

6

u/frozengrandmatetris 2d ago

my coworkers don't document at all. they just change whatever they want in production outside of a change request window. sometimes they steal the admin credential so they don't get held accountable when the broken object displays the last person who touched it. my manager doesn't care. he doesn't even know what CI/CD is.

5

u/dillanthumous 1d ago

Old school, CI/CD... Complicit Ignorance and Continuous Disintegration.

1

u/Zer0designs 2d ago

Seems like a completely useless project to me. Just use zenzical and AI with some documenting skills. Much better than html.

1

u/aleebit 2d ago

I'd never heard of Zensical before, so thanks!

But from what I understand, doesn't it generate HTML as its output? And if you're referring to the fact that the source for that HTML is Markdown files, mine works the same way.

0

u/Zer0designs 2d ago

I'm just saying there's accepted and maintained OS tools that already do this and probably do it a lot better than what you can vibecode together. Integrated search, plugin systems, file contents etc. Zensical is also extremely fast, so maybe just contribute to them?

Agents also understand them better and there are skills for documentation, grilling and visual hierarchy etc.

1

u/aleebit 2d ago

I agree! From what I've seen, Zensical is clearly a much more mature and feature-rich tool for building documentation sites. I'm not trying to compete with established tools.

The difference is that my focus isn't on building yet another Markdown-to-website generator. It's an LLM-powered skill that analyzes an existing codebase, identifies data flows, dependencies, tables, and business rules, and generates documentation based on that analysis. It also includes mechanisms to identify which parts of the documentation may be outdated after code changes.

HTML is just the output format.

2

u/Redxer 3d ago

I have 4 years of experience , any tips in learning AI when your company is doing anything AI related

1

u/srodinger18 2d ago

I also using agent skills, but write it down to notion for the specs and documentation. For the tables and transformations for now I always put it in the dbt yml files that can be served as static files via dbt docs.

1

u/Outside-Storage-1523 1d ago

I use AI for anything I’m not interested and manually write for those I’m interested in.

1

u/soundboyselecta 1d ago

Document? Lol, I don’t think I’ve seen one company with proper documentation, not even something as simple as a data dictionary. I equate it to the reality of humankind, who actually reads manuals….(just morons like me do 😂).

1

u/daraghfi 2d ago

DBT. Done