Senior Data Engineer (DataOps) (INBW23163) Amsterdam, Nederland
As a Senior DataOps Engineer, you will design, build and maintain scalable and efficient data pipelines. You will ensure the smooth flow of data from various sources into the data lake or warehouse and drive efficient data processing and analytics.
You will warrant the reliability, scalability, and performance of our data infrastructure.
To gather requirements and implement data integration and transformation processes, you will collaborate with data engineers, data scientists, and other stakeholders. You will work with DevOps teams to automate deployment, monitoring, and scaling of Databricks clusters and associated infrastructure.
You will provide technical guidance and support to team members. You promote best practices in data engineering and ensure adherence to coding standards and documentation.
Among your responsibilities:
- Optimizing and fine-tuning data pipelines to enhance performance, scalability, and reliability;
- Developing and maintaining data models and schemas.
- Supporting data processing and analytics, ensuring data consistency and adherence to best practices;
- Monitoring and troubleshooting data pipeline issues, performance bottlenecks, and data quality problems;
- Implementing data governance and data lineage processes, ensuring the traceability and documentation of data movement and transformations;
- Implementing and maintaining data security and access controls within Databricks;
- Ensuring compliance with data privacy regulations and company policies.
- Evaluating and recommending tools and techniques to improve data processing efficiency and scalability.