This position is no longer open for applications

Lead Data Engineer
Start Date: January 2024
Contract Length: 6 Months
Location/ Remote Working: 2 days onsite kings cross (London) 3 days remote
Pay Rate: 525
inside IR35

Job Summary: Seeking an experienced individual with a strong background in designing and implementing Data Lake solutions, particularly in the Azure ecosystem. The ideal candidate should possess extensive knowledge in data engineering practices and principles, coupled with technical proficiency in relevant Azure services and associated technologies.
Key Responsibilities:
 General:
   - Create low-level designs for Data Lake solutions in Azure or similar platforms.
   - Implement Data Lake solutions in Azure.
   - Demonstrate in-depth understanding of data engineering practices and principles.

Technical Skills:
   - Expertise in Azure services like Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure SQL Database, and Azure Stream Analytics.
   - Proficient in Python or Scala with Spark, PySpark.
   - Knowledgeable in Terraform.
   - Experience with both relational and NoSQL databases.
   - Significant experience in designing data pipelines, data modelling concepts, ETL procedures, and data quality frameworks.
   - Familiarity with data schemas (Avro Schema, JSON Schema) and data serialization (Parquet, ORC, Avro).
   - Exposure to middleware such as Apache Kafka, AWS Kinesis, Azure Event Bridge, or GCP PubSub.
   - Hands-on experience with continuous integration and ingestion frameworks.

3. Azure Services Knowledge:
   - Advanced knowledge of Azure Data Factory for orchestrating and automating data workflows.
   - Expertise in SQL and T-SQL for data querying and manipulation in Azure SQL Database.
   - Ability to design and implement data warehousing solutions using Azure Synapse Analytics.
   - Proficiency in using Azure Databricks for data processing and analytics.
   - Deep understanding of Apache Spark for large-scale data processing.
   - Expertise in setting up and optimizing Azure Data Lake Storage.

Cloud Experience:
   - Understanding of Azure Resource Manager (ARM) for deploying, managing, and monitoring Azure resources.
   - Proficiency in using Azure Portal and Command-Line Interface (CLI) for resource management.
   - Knowledge of Azure security features and compliance standards.
   - Familiarity with Azure Monitor and Azure Log Analytics for monitoring and logging.

Ways of Working:
   - Experience in Data Migration, Data Profiling, and Data Quality enhancement.
   - Proficient in Agile (Scrum) way of working.
   - Understanding of CI/CD processes and tools, including GitCI.
   - Strong understanding of software development life cycle and best practices.