May 2023 – Nov 2024 · 1 yr 6 mos

• Part of Credit Card and Loan Team

Jul 2022 – Apr 2023 · 9 mos · Bengaluru

• Part of Amazon Prime Subscription Team.
• Architected and implemented scalable data analytics pipelines processing terabytes of Prime customer data to deliver insights on customer acquisition, retention, and engagement for Product Managers, leveraging Python, SQL, AWS Glue, Lambda, Step Functions, API Gateway, DynamoDB, and QuickSight.
• Conducted statistical analysis on large datasets to evaluate business metrics and experiment results using Z-test, Kolmogorov–Smirnov test, and Mann–Whitney U test, enabling data-driven product decisions.
• Optimized distributed ETL workflows at TB scale, achieving a 50% reduction in overall runtime by applying broadcast joins, dynamic DPU allocation, and hash-based aggregation techniques.
• Developed production-grade dashboards and reporting solutions to monitor sales performance and customer behavior during peak traffic events such as Prime Day and Black Friday, supporting real-time decision-making.
• Collaborated cross-functionally with Product Managers and Senior PMs to gather ad-hoc analytical requirements, translate business needs into efficient data models, and deliver high-performance data solutions.

Apr 2021 – Jul 2022 · 1 yr 3 mos · Gurugram, Haryana, India

• Designed and implemented end-to-end data pipelines using Python, PySpark, and SQL to process 100+ GB of historical and incremental SharePoint data, integrating Axway APIs and Databricks, and delivering data to Amazon S3 via AWS Lambda triggered by API Gateway.
• Refactored and optimized incremental ingestion logic, achieving a 200% improvement in data load performance by leveraging multi-processing and multi-threading techniques.
• Developed a Databricks log monitoring and observability solution using Postman and DBFS, improving pipeline reliability, failure detection, and operational visibility.
• Built scalable ingestion and processing pipelines for Axway MFT and Salesforce data, utilizing REST APIs, Databricks, Apache Airflow, Amazon RDS, SQS, S3, AWS AppFlow, Snowflake, and StreamSets to support enterprise-grade data integration.

Mar 2019 – Mar 2021 · 2 yrs · Noida Area, India

• Key Responsibilities
• Refactored and optimized legacy big data pipelines to enable reliable data acquisition and forecasting, leveraging Python, Scala, Apache Spark, Spark SQL, Hive, Sqoop, and Hadoop.
• Designed and implemented automated Talend-based ETL workflows at scale, significantly reducing operational overhead, infrastructure costs, and support effort across multiple accounts.
• Modernized and orchestrated job monitoring, dependency management, and pipeline execution on the Azure cloud, improving reliability and observability.
• Performed job validation and performance tuning using YARN and Ambari, ensuring stable execution of large-scale distributed workloads and reducing production incidents.
• Key Achievements
• Built and deployed Azure Data Factory (ADF) pipelines integrated with Azure Databricks, resulting in a 35% reduction in overall data processing costs.
• Scaled existing systems to handle 300% higher data volumes within defined SLAs through pipeline optimization and distributed processing enhancements.
• Architected and optimized MapReduce- and Spark-based workflows using SQL and Spark SQL, achieving up to 250% performance improvement.
• Managed ingestion and extraction of up to 10 TB datasets using Hive and Sqoop into RDBMS systems; successfully migrated production ETL jobs from Ab Initio to Talend with zero data loss and minimal downtime.

Jun 2018 – Feb 2019 · 8 mos · New Delhi Area, India

• Worked as web & android application developer.