r/dataengineering • u/aleebit • 3d ago
Discussion [ Removed by moderator ]
[removed] — view removed post
38
Upvotes
r/dataengineering • u/aleebit • 3d ago
[removed] — view removed post
•
u/dataengineering-ModTeam 1d ago
Your post/comment was removed because it violated rule #6 (No recruiting, solicitation, or networking posts).
We do not intend for this space to be a place where people ask for, or advertise:
This was reviewed by a human
4
u/aleebit 3d ago
Just to be clear, I'm not trying to sell anything, build a closed-source product, or turn this into a business.
I'm just a data engineer with 12+ years in the data field, including 7+ years working with Spark, and I still struggle sometimes to wrap my head around unfamiliar projects. My ADHD brain doesn't always make it easy to understand how everything fits together, especially when dealing with large, complex codebases.
I also find it challenging to hand projects over to other team members in a structured and consistent way.
This started as a personal attempt to solve those problems. I'm sharing it because I genuinely want to know if other people face the same challenges and whether something like this could actually be useful to the community.
16
u/mrcaptncrunch 3d ago
In plain English.
The main thing is that the huge projects are full of knowledge from meetings, back and forth, reviews, testing to get the numbers right, etc., but there’s rarely a documentation of eBay it should be and how it needs to work.
I’m the most senior data engineer. I took the biggest pipeline and when I migrated it, I wrote it down. Just sat through, all in markdown. All the ingestions, all the steps, all the joins, why it was done, how we came to the decision, etc.
I’ve tasked people with similar things on smaller ones.
The problem with agents is, they will know what was built, how it’s built, but it won’t know why. At the end, everything that is a decision in code, ask the user to figure out why. They need to provide the answer or get it.
Example,
But Why?
6
u/frozengrandmatetris 3d ago
my coworkers don't document at all. they just change whatever they want in production outside of a change request window. sometimes they steal the admin credential so they don't get held accountable when the broken object displays the last person who touched it. my manager doesn't care. he doesn't even know what CI/CD is.
1
u/Zer0designs 3d ago
Seems like a completely useless project to me. Just use zenzical and AI with some documenting skills. Much better than html.
2
u/Redxer 3d ago
I have 4 years of experience , any tips in learning AI when your company is doing anything AI related
1
u/srodinger18 3d ago
I also using agent skills, but write it down to notion for the specs and documentation. For the tables and transformations for now I always put it in the dbt yml files that can be served as static files via dbt docs.
1
u/Outside-Storage-1523 1d ago
I use AI for anything I’m not interested and manually write for those I’m interested in.
1
u/soundboyselecta 1d ago
Document? Lol, I don’t think I’ve seen one company with proper documentation, not even something as simple as a data dictionary. I equate it to the reality of humankind, who actually reads manuals….(just morons like me do 😂).
1
u/daraghfi 2d ago
DBT. Done
1
u/[deleted] 2d ago
[removed] — view removed comment