Data Engineer (HA-34243378) Rotterdam, Netherlands
| Salary: | competitive |
Data Engineer
Currently we are looking for a Data Engineer for one of our Retail clients. The scope of the role is more pipelines, data validation and CI/CD.
Tasks
- Ingestion of data in multiple formats like CSV, JSON and Delta tables
- Improve efficiency to reduce running times and costs
- Analyse and refactor legacy code
- Implement data cleansing and transformations
- Implement data quality checks and validations
- Publish datasets and maintain metadata
- Write unit and integration tests
- Automate the development process using CI/CD
- Monitor the run of the data pipelines and troubleshoot in case or errors
- Align with FE developers
Requirements
- Deep understanding of Azure Cloud and experience with Azure Data Factory
- Deep understanding of Spark and distributed computation
- Understanding of the Data mesh architecture. Knowledge about the medallion architecture (or any other similar flavour) and data products and publishers
- Proficient in Python and PySpark
- Good communication skills
- Pro-active attitude to improve as we go
- Understanding of one of the 3 major clouds (AWS. GCP, Azure)
- Understanding of the different stages of data processing. Ingestion/Clean/Publish/Transformations, etc.
- Experience building data pipelines and able to produce complex queries.
- Pro-active and can-do attitude
Contract
- Start ASAP
- 32 - 40 hours a week
- Working at the office 1 - 2 times a week, in the Rotterdam area
- 4 - 6 months, with option to extend
*Please note that it is mandatory to live in the Netherlands for this position