Apr 2021 – Aug 2023 · 2 yrs 4 mos · Hybrid

• Architected and optimized real-time Spark data pipelines to process event data at 100K RPS from Kafka, storing outputs in GCS (Parquet) and Apache Kudu, reducing ETL lag from 2 hours to under 5 seconds and integrating with BigQuery.
• Led the initiative to develop a process for generating real-time user locations to enhance targeted delivery of ad campaigns, resulting in a 15% increase in revenue.
• Implemented a process leveraging Prometheus, ETCD and creating RESTful API in Flask to compute near real-time RPS for ad campaigns on Dailyhunt and Josh apps; improved campaign delivery accuracy by 20% .
• Analysed and optimised ad campaigns delivery by introducing new factors for pacing and rebooking.
• Revamped and fine-tuned multi-processing Python processes to parallelise data processing of millions of events, resulting in a 50% decrease in processing time.
• Created Superset dashboards on GCP, Kudu and Hive tables for data visualisation and analysis for business stakeholders.
• Orchestrated the migration of 50+ jobs on Airflow and automated monitoring by setting up alerts and reporting.
• Led technical onboarding for new data engineers, delivering comprehensive training sessions on team projects and coding best practices.
• Collaborated effectively within Agile environments and Scrum practises, contributing to iterative development and timely achievement of project milestones.
• Developed and implemented Bash scripts to automate tasks and streamline workflows.
• Extensive hands-on experience working in Unix and Linux environments, adept at utilising command-line interfaces.

Oct 2019 – Apr 2021 · 1 yr 6 mos · Hybrid

• Analysed and optimised ETL processes by developing real time data pipeline sourcing data from Kafka topics and pushing to Apache Kudu tables using Spark and Scala, thereby reducing the lag from 2 hours to less than 5 seconds.
• Collaborated with Ad Operations team to develop User Monetisability module for revenue computation from impressions, clicks, installs, and leads.
• Enhanced the computation process for installs numbers by extracting data from Kudu and Hive, and facilitating seamless data loading into Mongo.
• Developed Impala view for integration of data from Kudu and Hive.

Jul 2018 – Oct 2019 · 1 yr 3 mos · Bengaluru, Karnataka, India · On-site

• Owned, developed and maintained Hadoop based ETL Jobs in Cascading (Java) for data transformation in horizontally scalable architecture to achieve a multi-tenant based Data Warehousing architecture.
• Innovated and created Django server to publish DAGs to show ETL jobs’ dependencies on one another and current status by fetching states from MySQL; improving team’s productivity.
• Migrated data from Hive to Amazon S3, created build and deployment plans using Bamboo and scheduled the scripts using Oozie.

Jan 2017 – Jun 2017 · 5 mos · Gurgaon, Haryana, India

• Prepared bench-marking document comparing Big Data technologies based on functionalities.
• Created a use case to mine Twitter data using Spark Streaming and perform sentiment analysis on tweets using Natural Language Processing.
• Created charts for data visualization in Tableau.