Data engineer

Опис проекту

Responsibilities

• Work closely with your colleagues within a product team to design scalable, efficient and stable products
• Work closely with product owners to build and deliver features designed to support PANDORA to drive commercial business cases and efficiencies within reporting and analytics and other data driven use cases
• Create and maintain high quality data pipeline architecture
• Obtain relevant business and system knowledge to convert requirements efficiently to technical design documentation and conceptual data modelling
• Ingest large, complex data sets, automate manual processes, optimize data delivery, improve data quality, etc.
• Own your data products and track continuous stability and benefits
• Work with stakeholders and teams to assist with data-related technical challenges
• Follow best practice and existing guidelines but also push the limits
• Thrive in a fast paced environment and you get motivated by challenges as a true problem solver.

Skills Required

• 2+ years of experience as a Data Engineer or in Software engineering role with focus on data
• Proven experience with ADF, Data Lake, Synapse, T/U-SQL, DataBricks, MDS, DAX, PowerBI and SQL Server
• Stream-processing systems: Kafka or Spark Streaming
• DevOps and CI/CD focus
• Hands-on with git and coding best practice
• Unit testing
• Experience in working with relational data sets using SQL
• Experience with datawarehousing, dimensional modelling and cubes
• Experience in building, and optimizing data pipelines, architectures, and datasets
• Experience in working with Azure, AWS or other cloud providers
• Designed, built and delivered production ready data products at an enterprise level
• Highly proficient in English and able to communicate with all levels
• Agile mindset and experience of Scrum

• Big data: Spark, Hadoop
• Data pipeline and workflow tools: Airflow
• Infrastructure: Kubernetes and Helm; we’re using Airflow running on Kubernetes