Python

remote

Let’s breathe life into great tech ideas! With 3,000 people globally, Intellias is a company where benchmark technological solutions are born. Join in and take your part in digitalizing the world.

What project we have for you

We are building a large real-time data pipeline to move data from an RDBMS to several destinations – datalake, search engine (e.g. OpenSearch). 300M+ records, 10Tb volume. Real-Time + Historical data load. Transformation and aggregations on the stream.

What you will do

Collaborate with business stakeholders and technical teams to understand and analyze data requirements
Lead the design and implementation of data models and database structures that meet business needs
Profile, refactor, and tune performance in the database
Design and implement complex ETL processes to extract, transform, and load data from various source systems into the data warehouse
Ensure data integrity, consistency, and accuracy through robust data quality assurance measures
Review and support team members, providing guidance and mentorship
Supervise and contribute to the data-driven strategy for the project, aligning it with business objectives

What you need for this

Tech Stack:

Python
Kafka
Apache Flink / Apache Spark (Streaming)
Apache Hudi / Apache Iceberg

Required skills:

5+ years of experience as a Data Engineer or similar role, with hands-on expertise in large-scale, production-grade data pipelines.
3+ years of experience designing and running real-time data streaming systems (Kafka + Flink / Spark Streaming).
3+ years of proficiency in Python for data engineering (data processing, orchestration, automation).
Solid understanding of distributed systems, data partitioning, checkpointing, and fault-tolerant stream processing.
Practical experience with Apache Hudi or Apache Iceberg for incremental data storage and schema evolution.
Experience with RDBMS sources (PostgreSQL, MySQL, etc.) and data lakes / object storage (S3, GCS, etc.).
Deep understanding of ETL / ELT design patterns, data modeling, and data quality principles.
Experience deploying and maintaining data pipelines in AWS is preferred.
Excellent analytical and problem-solving skills, with the ability to design robust, scalable, and efficient architectures.
Strong communication skills and ability to collaborate with cross-functional teams.

What it’s like to work at Intellias

At Intellias, where technology takes center stage, people always come before processes. By creating a comfortable atmosphere in our team, we empower individuals to unlock their true potential and achieve extraordinary results. That’s why we offer a range of benefits that support your well-being and charge your professional growth.We are committed to fostering equity, diversity, and inclusion as an equal opportunity employer. All applicants will be considered for employment without discrimination based on race, color, religion, age, gender, nationality, disability, sexual orientation, gender identity or expression, veteran status, or any other characteristic protected by applicable law.We welcome and celebrate the uniqueness of every individual. Join Intellias for a career where your perspectives and contributions are vital to our shared success.