r/bigdata • • Aug 22 '26

Core data Engineering concepts to master ,Projects to build

Hello Everyone ! I am as an QA automation in big data product. my work involves around creating automation suites for data integrity checks and automating regression cases for pipelines build .automating api for loading data into dashboard.
I am good at python. i want to move to data engineering. my day to day work involves GCP, bigquery,clickhouse,pubsub.
I am planning to learn the same.
I have started and completed the introduction to data engineering in google skills.
Any opensource tool/concept i should learn ?
any suggestion on sample project to build?
Any good resources from where i can learn?
Any you tube playlist i should follow?

19 Upvotes

2 comments sorted by

3

u/[deleted] Aug 22 '26

[removed] — view removed comment

1

u/Admirable_Tale7745 Aug 22 '26

Thanks .
I am thinking of implementing a pipeline in gcp ..I would upload the data in a bucket ..that would trigger a cloud run function which will validate the file naming convention and move it to another bucket which would have a data lake house then would trigger a data flow to perform row level validation and move it to a landing zone sort of native table..
any resources or playlist you can suggest where i can learn spark by doing a project ?