Jan 2021 – Jul 2024 · 3 yrs 6 mos · Bengaluru, Karnataka, India

• Led the migration of Petabyte-scale data lake from Redshift to open data lakehouse platform using Spark engine and Delta, Iceberg file formats, enhancing availability and significantly reducing operational costs.
• Achieved annual cost savings of over $1 million through strategic optimization of storage, infrastructure layers, and ETL queries, resulting in reduced AWS cloud and Databricks cluster expenses.
• Architected and developed self-serve batch and streaming data ingestion and ETL platforms utilizing Airflow, Spark, EMR, Kafka, Flink, and Databricks, achieving streamlined configuration and deployment.
• Boosted Data Ingestion platform efficiency through Delta and Iceberg file format tuning, leveraging technologies such as Apache Spark, DuckDB, RocksDB, and Scala, and employing partition-to-partition joins.
• Designed and developed a reconciliation platform featuring robust data models and ETL pipelines to process millions of transactions from diverse sources.
• Established Data Quality Platform using Great Expectations and Airflow for ensuring data integrity.
• Enforced best data and security practices, ensuring compliance with industry regulations.
• Mentored Data Engineering team, offering technical leadership and guidance.

May 2020 – Jan 2021 · 8 mos · Bengaluru, Karnataka, India

• Designed and implemented end-to-end data solutions, including data warehouses, data lakes, batch and streaming ETL processes, and self-service BI tools for the retail and digital media sectors.
• Developed dimensional models and ETL pipelines for fulfilment, inventory, and user journey analysis for retail and digital media clients.

Apr 2017 – Apr 2020 · 3 yrs · Bengaluru, Karnataka, India

• Established AWS cloud based Data Lake using Apache Spark, Databricks, Delta, and Athena, ensuring robust data storage and processing capabilities.
• Developed Python-based data pipelines leveraging Apache Airflow and Databricks to extract, clean, process, and ingest data from SQL Server, Postgres, SFTP, and REST APIs into the Data Lake.
• Collaborated with the Data Science team to enhance feature calculations, suppression, and machine learning model scoring processes, transitioning from Pandas to PySpark on Databricks for improved parallel processing efficiency.
• Migrated and optimized 5k lines of code for fraud monitoring from SQL Server to Spark SQL, enhancing performance and scalability.
• Engineered schema finder job using Spark to extract complex nested JSON raw data, transforming and writing it to Parquet format for efficient storage and retrieval.
• Contributed to the development of a streaming platform for event processing from Kafka and Spark Streaming, ensuring real-time data processing capabilities.
• Conducted POC and engineered Apache Airflow Infrastructure setup using Bamboo, Docker, DC/OS, and Kubernetes, enhancing workflow automation and management.
• Actively participated in Agile and Scrum meetings, facilitating Sprint Planning, Retrospective, and Daily Standups for efficient project execution and collaboration.
• Involved in on-call rotations for production ETL Data pipelines (~10% of work)

Nov 2011 – Mar 2017 · 5 yrs 4 mos · Mumbai, Maharashtra, India

• Managed a portfolio of 50+ MySQL and SQL Servers housing extensive OLTP and OLAP databases, ensuring optimal performance and reliability.
• Engineered robust and scalable ETL data pipelines using Python, SSIS, SQL Server, and MySQL to handle billions of monthly page views.
• Developed and optimized critical processes including spam detection, revenue auditing, and Affiliate referral using MySQL, SQL Server procedures, Apache Spark, Redis, and Kafka.
• Designed and implemented interactive reports such as Drill Down, Drill through, and Cross-tab reports using SQL and MDX for comprehensive publisher and admin reporting.
• Conducted performance tuning of databases, queries, and stored procedures, enhancing overall system efficiency.
• Constructed data warehouse and OLAP cube with SSAS attribute relationships and hierarchies to optimize MDX query performance.
• Automated routine maintenance and monitoring tasks using Python, Shell Script, and Linux Crontab, improving operational reliability and efficiency.
• Facilitated setup and migration of database instances from on-premise to AWS Cloud platform, ensuring seamless transition and optimal resource utilization.

Aug 2007 – Oct 2011 · 4 yrs 2 mos · Mumbai, Maharashtra, India

• Engineered scalable distributed data processing framework utilizing replication and Sharding to process 1 TB of data daily across 40+ MySQL database servers, handling 500M queries/day.
• Developed data models for managing customer payouts, ensuring data integrity and accuracy.
• Conducted fine performance tuning of MySQL servers and optimized Ad serving processes to accommodate optimal concurrent requests.
• Engineered and optimized data pipelines for efficient data flow, implementing data aggregations for processing raw Ad click and Impression data.
• Enhanced business-critical Ad Serving and Revenue Auditing processes for high throughput and reliability.