Job Description

:

--------------------


We are seeking a highly skilled and experienced Data Engineer with 6+ years of proven expertise in building, delivering, and maintaining data pipelines. The ideal candidate should be capable of independently architecting and implementing scalable data solutions in cloud environments, primarily AWS and Databricks, while closely collaborating with onshore teams and stakeholders.



Key Responsibilities:



Design, build, and deploy end-to-end scalable data pipelines using AWS and Databricks platforms.



Implement and manage data lake, data warehousing, and real-time streaming solutions.



Collaborate with architects and data scientists to translate business requirements into data solutions and models.



Develop ETL/ELT jobs using Python, SQL, Apache Spark, and other industry-standard tools.



Establish and enforce data governance and data quality frameworks.



Automate data preparation, ingestion, transformation, and monitoring processes.



Ensure data security, compliance, and privacy best practices are embedded in all solutions.



Perform automated testing, validation, and continuous monitoring of data pipelines.



Work in Agile/DevOps environments using CI/CD for data pipeline development and deployment.



Mandatory Skills & Experience:



Strong experience in AWS data and analytics architecture (Glue, Redshift, S3, Lambda, etc.)



Hands-on expertise in Databricks and Apache Spark



Deep understanding of data integration, data lakes, and data warehousing concepts



Proficiency in Python, SQL, and scripting for data engineering tasks



Experience with real-time data streaming technologies (Kafka, Kinesis, etc.)



Good knowledge of data modeling (dimensional, normalized, denormalized)



Ability to implement and automate data quality checks and governance processes



Exposure to continuous monitoring and test-driven development in data engineering