Vivek Shukla

Data Engineer

Dubai, United Arab Emirates4 yrs 10 mos experience

Highly Stable

Key Highlights

Expert in Big Data Ecosystem and Data Engineering.
Proficient in developing scalable ETL pipelines.
Strong experience with Apache Spark and AWS Glue.

Stackforce AI infers this person is a Data Engineering expert with a focus on Big Data and ETL solutions.

Contact

Skills

Core Skills

Data EngineeringApache SparkEtlAndroid Development

Other Skills

ADFAWS GlueAngularApache OozieAurora DBDB2Data StructuresGCPGoogle Maps APIJavaPostgreSQLPower BIProblem SolvingPySparkSQL

About

Dedicated IT professional with Hands on experience developing and utilizing Big Data Ecosystem , data analytics and application development experience seeking assignments in Software Development/ R&D with an organization of high repute. Skill Summary • Develop, refine and scale data management and analytics procedures, systems, workflows, and best practices. • Strong data modelling skills and knowledge of MySQL • Experience in Hadoop Technology stack which includes ecosystems like SPARK, HDFS. • In-depth knowledge in Handling Bigdata using spark. • Used various data formats like csv, Json, parquet, orc. • Hands on experience in writing ETL scripts using Scala and spark. • Hands on experienced in Agile Methodology system projects. • In-depth knowledge of developing web applications using Angular 11 and Java/Springboot.

Experience

Emirates

Senior Data Engineer

Sep 2024 – Present · 1 yr 6 mos · Dubai, United Arab Emirates

Developed a scalable feedback data product using ADF, Scala, and Apache Spark on Cloudera, with Snowflake as the target database.
Created optimized views on top of Snowflake tables, enabling seamless integration with Power BI for business reporting.
Built a generic pipeline to process 10+ daily surveys up to the enrichment layer, delivering actionable metrics for decision-making.
Designed a contactability pipeline (handling around 30 million records daily) to monitor passenger contactability per flight, with insights like region-wise, channel-wise, and 30-day trend metrics to support disruption management.
Orchestrated and scheduled workflows using Apache Oozie for consistent and timely data delivery.

ADFScalaApache SparkSnowflakePower BIApache Oozie+1

Hashedin by deloitte

3 roles

Software Engineer-II

Mar 2023 – Jul 2024 · 1 yr 4 mos · Bengaluru, Karnataka, India

Designed and implemented pipeline to process structured data by integrating more than 10 million records from different sources using pyspark.
Designed an ETL script with AWS Glue, extracting structured records from DB2, applying sophisticated business transformations, and loading refined data into Aurora DB for a seamless and efficient data flow.
Developed a concise Low-Level Design (LLD) for an ETL script leveraging AWS Glue to extract structured data from DB2, execute intricate transformations, and load into Aurora DB.
Developed an AWS Glue-based ETL script to reconcile record disparities between DB2 and Aurora databases, seamlessly generating and storing detailed reports in an S3 repository.

PySparkAWS GlueDB2Aurora DBData EngineeringETL

Software Engineer

Jun 2021 – Mar 2023 · 1 yr 9 mos · Bengaluru, Karnataka, India

Spearheaded schema change adaptations in ETL jobs using Scala and Spark, ensuring resilient data processing.
Used Talend Data Studio tool to migrate data present in csv format to different destination such as Salesforce after complex transformations and joins.
Pioneered the development of a comprehensive ETL script from scratch using Scala and Spark, orchestrating the extraction and complex transformation of data from GCP.
Implemented strategic modifications to optimize query costs by introducing pivotal partitions in an ongoing Data Engineering project for a high-profile client.

ScalaSparkTalend Data StudioGCPData EngineeringETL