Nov 2023 – Aug 2025 · 1 yr 9 mos · Greater Toronto Area, Canada · Remote

• AWS Expertise – Hands-on with MWAA (Managed Airflow), Glue, S3, Lambda, Athena, Redshift, EC2, CloudFormation, and SNS.
• Data Engineering & ETL – Designed batch & real-time pipelines using Databricks, Fabric, Data Factory, PySpark, and Glue, cutting processing time by 25%.
• Cloud Migrations – Led migration of 50+ data pipelines from Azure to AWS with Glue, Redshift, and Airflow orchestration.
• Security & Compliance – Strengthened pipelines with IAM, Secrets Manager, and Systems Manager for credential & access control.
• Workflow Orchestration – Managed 50+ Airflow workflows for large-scale data processing in Databricks.
• Database & APIs – Built Python-based RESTful APIs for ingesting and querying financial data in Redshift.
• Infra & DevOps – Automated CI/CD with Code Pipeline & Terraform, streamlining deployments.
• SQL & DB Migration – Migrated SQL Server to AWS Postgres using SCT & DMS.
• Automation & Monitoring – Integrated SNS alerts for Glue jobs, ensuring proactive failure notifications.
• Analytics Enablement – Built curated datasets in Fabric Lakehouse (Delta), exposed them via SQL Endpoints & APIs, and delivered real-time Power BI dashboards for operational insights.

Feb 2023 – Sep 2023 · 7 mos · Greater Toronto Area, Canada · Remote

• Tools: Python, Airflow, Spark, Kubernetes, Hadoop, Hive, Presto/Trino, MinIO, DB2, SAS, Rancher, Docker, Jenkins, Google Cloud Platform, JFrog Artifactory, and Black Duck Scan.
• Implement and maintain data pipelines that enable large amounts of data to be transformed and
• managed.
• Monitor and ensure the accuracy of production data by implementing processes and systems available
• to key stakeholders and the business processes that depend on them.
• Orchestrate data pipelines and ETL processes using Apache Airflow and monitor them for accuracy,
• reliability, and timeliness
• Enable the migration of data stores and analytical datasets from the data warehouse to the enterprise
• Data lake to the global data & and analytics platform.
• Migrate SAS-based data processing workflows to Apache Spark and Python.
• Implement the data pipelines using CI/CD with Docker.
• Develop new reusable managed datasets for CBA stakeholders by researching, planning, and
• implementing them.
• Work with internal and external teams to uncover business requirements, analyze root causes, and
• develop performance-effective solutions.

Jan 2022 – Nov 2022 · 10 mos · Greater Chennai Area

Jul 2020 – Jan 2022 · 1 yr 6 mos · Greater Chennai Area

Apr 2020 – Jul 2020 · 3 mos

• Branch powers deep linking and mobile attribution for 35000+ of the most advanced apps in the world, including household names like Airbnb, Pinterest, Slack, Ticketmaster, Tinder, Starbucks, and Yelp. Every day, our customers generate more than 100 million new URLs, and we process 2.5 TB of raw data from over 10 billion events.
• I designed and developed a Drop wizard service using Spark to export raw event logs from our HDFS warehouse to customer S3 buckets. Some customers generated over 1 million daily events and built big data pipelines and warehousing, handling more than 10 billion events daily.
• I implemented a Flink streaming pipeline to archive over 4 billion attributed and organic daily events from Kafka to S3.
• Created automated data workflows using Python, improving operational efficiency and reducing manual errors.
• As part of Tune Attribution's acquisition, I led the export migration project with zero downtime.
• We developed a Kubernetes Spark operator to migrate all Spark services from Mesos to Kubernetes as part of cost-saving efforts.
• Implemented RESTful API endpoints in Python, facilitating seamless communication between different components of data systems.
• Generated cohort aggregated analytics using batch pipelines for Druid.
• We implemented database, table, and crawler configurations for AWS Glue.
• We set up Kinesis Firehose to convert raw data to parquet files and store them in AWS S3.

Mar 2019 – Apr 2020 · 1 yr 1 mo

• Branch powers deep linking and mobile attribution for 35000+ of the most advanced apps in the world, including household names like Airbnb, Pinterest, Slack, Ticketmaster, Tinder, Starbucks, and Yelp. Every day, our customers generate more than 100 million new URLs, and we process 2.5 TB of raw data from over 10 billion events.

Jun 2016 – Jan 2019 · 2 yrs 7 mos · Chennai, Tamil Nadu, India · On-site

• Assisted in the analysis, cleaning, and transformation of raw data.
• I have Imported and exported data into HDFS, S3 Buckets, and Hive using SQOOP, Flume, and NIFI.
• To analyze the processed results, I created Hive tables on top of them.
• Used the Oozie Scheduler system to automate the pipeline workflow, orchestrate the map, and reduce jobs that extract the data on time.
• RDD transformations were used for enrichment, cleansing, and shared data aggregation.
• Interactive analysis of Hive tables through various data frame operations using Spark SQL.
• I was loading fact tables into Elasticsearch for visualization through Kibana.
• Supplied on-call support to ensure the platform was up and running 24/7.
• I have created ETL IBM-Trex (DataStage) parallel jobs to extract and reformat the source data so I could load it into the new data warehouse schema.
• Using Oozie Scheduler, we automated the pipeline workflow, orchestrated map reductions, and retrieved data as soon as possible.