Role : Data Engineer

Work Location : Dubai,UAE

Salary : AED 18000-AED 20,000+Standard UAE Benefits

NOTE: Immediate Joiners will be preferred.

Key Responsibilities:

• Data Engineering

These are the below objectives:

1 Data Acquisition

\xc2\xb7 Vendor should manage the existing Data pipelines built for data ingestion.

\xc2\xb7 Create and manage new data pipelines following the best practises for the new ingestion of data.

\xc2\xb7 Continuously monitor the data ingestion through Change Data Capture for the incremental load

\xc2\xb7 Any failed batch job schedule to be analysed and fixed to capture the data

\xc2\xb7 Maintaining and continuously updating on the technical documentation of the ingested data and maintaining the centralized data dictionary, with necessary data classifications.

2 Data Extraction and Cleaning

\xc2\xb7 Extraction of data from the data sources to be cleaned and ingested into big data platform

\xc2\xb7 Automation of data cleaning has to be defined before ingestions

\xc2\xb7 Data cleaning to handle the missing data and remove any outliers and resolve any inconsistencies

\xc2\xb7 Data quality check has to be performed in terms of accuracy, completeness, consistency, timeliness, believability, and interpretability

3 Data Integration, Aggregation and Representation

\xc2\xb7 Exposing of Data views or Data models to Reporting and source systems using Hive or Impala, or similar tools provided by us

\xc2\xb7 Exposing of cleansed data to Artificial Intelligence team for building data science models

Below are skills which are required but not limited to

Skill Set:

Strong Experience in Spark and Storm.

Expertise in Monitoring Tools like Grafana,Prometheous,Zabbix.

Hands on experience in Management of Hadoop Pentaho.

Expertise in storage layer/DB like HDFS,Cassandra,Opentsdb.

Expertise in Event-driven Architecture.

Caliberly