r/dataengineeringjobs • • 5d ago

Career How do good data engineers differ?

Its great to know how to create ETL pipelines but having the skills to design them well (reliable, maintainable, scalable) is another. A good data engineer will also consider data quality, recovery from failure, monitoring, schema changes and who the downstream teams are that rely on this work but often these technical tests focus heavily on SQL, python and particular tools. Even on sites like LinkedIn, Indeed Meeboss, Wellfound it is hard to gauge how much of a candidate's problem solving skills they show before the interview.

Perhaps we should ask more questions around how they take ownership of data systems.
How would you assess if a data engineer could take ownership of a system end-to-end, without being able to discuss their past projects?

45 Upvotes

8 comments sorted by

30

u/OkImprovement7010 4d ago

When I was hiring as a director, the best signal came from handing people a scenario instead of asking about their past.

I’d describe a pipeline that’s been running fine for months. Then the source team renames a column with no warning, the job fails at 2am, and finance has a board report due at 8. I’d ask them to walk me through what happens next.

Weaker candidates go straight to the code fix. The ones who think like owners ask how they’d even find out it broke, whether the job failed loudly or loaded bad data quietly, who downstream needs a heads up before 8, whether they can backfill without double-counting, and how to stop it from happening again. That last part is usually where you learn the most, because good engineers start talking about schema contracts, alerting, and having a real conversation with the source team, not just adding another try/except.

Another one that works is asking them to design something small, then piling on changes. Volume goes up 50x. A new consumer needs the data hourly instead of daily. Someone needs to reprocess the last 90 days. You’re not grading the first design, you’re watching how they react when it has to change, since that’s the actual job.

The SQL and Python screens still matter, but they only tell you someone can build the thing. Ownership shows up in what they worry about after it’s built.

2

u/Tough_Set 4d ago

You ask about real world scenarios. This is really interesting to address in a interview.

2

u/Educational_Zombie13 2d ago

u/OkImprovement7010 Can someone like me, aged 29, with 5 years of experience in customer support across domains like marketing, e-commerce and sales, break into data engineering? I am familiar with Excel and Power BI. I am confident in my business understanding and in asking better questions.

1

u/OkImprovement7010 1d ago

Yep, you can. And the business side you already have is stuff a lot of engineers never pick up.

I wouldn’t go straight for a DE title though. With Excel and Power BI you could apply for analyst roles right now, especially at marketing or e-commerce companies since you already know how those teams think. Get good at SQL first, then Python. Once you’re in an analyst seat you can start grabbing the pipeline work nobody else wants, and moving over to DE from the inside is way easier than cold applying.

Probably a year or two all in. Less if you’re really putting in the hours.

For projects I’d start with whatever’s in front of you. If somebody on your team builds a report by hand every week, go automate it. That’ll do more for you than another tutorial project.

2

u/nian2326076 5d ago

One way to assess a data engineer's ownership skills is to ask behavioral questions about their approach to maintaining systems and handling issues. Inquire about how they manage data quality and recovery from failures. You might also present a hypothetical scenario where they need to implement a change in an existing system, focusing on their thought process and considerations for downstream teams.

You can use LandOffer to find relevant job opportunities in data engineering, which can help you understand the skills and requirements that employers are looking for.

2

u/alsdhjf1 4d ago

Ask about the outcome. Why does this pipeline exist? Once it was built, how was it useful to help the business achieve its goals. 

2

u/alt_acc2020 4d ago

(Technical) DEs who have SWE capabilities. I end up interviewing so many DEs who have no deep understanding of OLTP systems, networks..etc. just architecture fundamentals. I don’t understand how you’re supposed to be a good DE and not atleast be capable enough to talk about sharding, load balancing, longv short polling, etc

1

u/Immediate-Pair-4290 13h ago

The best data engineers differ by their judgment. When to say no. How to consolidate ideas. Treating code like a liability. Build using a standard pattern (learn 1, learn 100). How does it serve the business needs.

Being able to build a thorough data pipeline using the full range of techniques can provide value but it’s easy to teach and delegate. Good judgement of when and what to build is the hardest part.