Job Description

MAIN OBJECTIVE OF ROLE: To design, build, and maintain scalable data architectures and pipelines using AWS services, focusing on efficiently ingesting, processing, and transforming data to ensure high quality and accessibility for analytics and reporting. KEY RESPONSIBILITIES:

• Designs, builds and maintains scalable data pipelines to ingest and process data from various sources.
• Leverages Amazon Web Services (AWS) tools like AWS Glue, Amazon S3, Amazon Redshift, and Amazon RDS to manage data pipeline, storage, transformation, and analytics.
• Builds and maintains Cloud framework for orchestrating data-pipelines, continuously optimizing and enhancing it to integrate new data sources effectively.
• Develops Extract, Transform, Load (ETL) processes to ensure data is cleaned, transformed, and loaded into data warehouses. Use appropriate scripting such as PySpark, Python, Scala to perform data transformations.
• Automates data integration and processing tasks to enhance efficiency and reduce manual intervention.
• Sets up monitoring tools to track data pipeline performance and troubleshoot issues as they arise.
• Maintains comprehensive documentation of data architectures, processes, and workflows, ensuring updates are made consistently with every release.
• Follows defined Data Governance policies and guidelines and enforces team to align with standard procedures.
• Ensures data security and compliance with relevant regulations, including GDPR or industry-specific guidelines.
• Leads the junior data engineers, provide technical guidance, manage work allocation and delivery.
• Leads and mentors junior data analysts, collaborate with cross-functional teams to meet data needs, implements data governance practices, and optimizes performance to support informed decision-making within the organization.

QUALIFICATIONS:

• Bachelor's Degree (3+ years)
• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field
• Fluent in English
• Must have 8+ years of industry experience, with 5 years working with design and development of data pipelines at enterprise scale. Hands on experience in building streaming, batch pipelines for structured, semi-structured data Must have, solid experience in handling large volume of data and performing transformation using scripts such as PySpark, Scala, Python. Must have experience working with AWS cloud services including EMR, Lambda, MKS, Redshift, GLue, Preference for working experience in airline industry
• AWS EMR, MKS, Glue, PySpark, Python, Scala, OpenSearch, Redshift, Dynamo DB, RDS
• Years with qualifications: 8 - 10 years

COMPETENCIES:

• Customer Focus
• Teamwork
• Effective Communication
• Personal Accountability & Commitment to achieve
• Resilience & Flexibility (Can do attitude)

ISR Requirements: Reads and complies with the ISR policies of the Company and diligently reports any weakness or incidents to the respective Line Manager or the Information Security team. Completes all required ISR awareness sessions and follows associated guidelines in the day-to-day business operations.

Skills Required