Jul 2023 – Aug 2024 · 1 yr 1 mo · Mumbai, Maharashtra, India · Hybrid

• Worked on developing a Multi-Tenant Data Platform in GCP providing capabilities such as Data Collection – Rest APIs built using Java/Python publishing data to Kafka. Streaming Ingestion – Apache Spark consuming from Kafka and writing to Iceberg/BigQuery with exactly once guarantees. Data Processing – Implemented Dataform for batch data processing, improving data pipeline reliability and developer productivity. Data Governance – Leveraged Schema Registry and Centralized Config Management to ensure data quality and consistency across the platform.

Sep 2021 – Jun 2023 · 1 yr 9 mos · Bengaluru, Karnataka, India

• Designed and architected an automated End to End Regression Framework for batch(scheduled and event based) pipelines for the data platform which resulted in accelerating Time to Market for new product features/fixing existing defects. This solution also resulted in savings of $2 million/year in terms of direct and indirect cost for organization.

Feb 2019 – Sep 2021 · 2 yrs 7 mos · Mumbai, Maharashtra, India · Hybrid

• Led the Data and Application team. Introduced many development practices to the organization such as DataOps, CICD pipelines, GitOps.
• Owner for building centralized data platform on GCP using Airflow/Etlflow for ETL job scheduling, Google BigQuery as primary data warehouse, Kylin for handling low latency queries (< 2s) and DataProc as managed Spark service.
• Led the frontend team for building custom web application dashboards for business using various JavaScript frameworks and backend team for building scalable Rest and GraphQL APIs in Scala.

Jul 2015 – Feb 2019 · 3 yrs 7 mos · Mumbai, Maharashtra, India

• Worked with Domain Monetization team for Media.Net Contextual Advertising platform.
• Engineered efficient data pipelines for capturing data from different sources such as Kafka (Traffic Data), Relational Database (Master Data) etc and saving it in optimized Hadoop formats for faster and consistent analytics, also achieved Near Real Time delay for data processing in some specific use cases using HBase as lookup datastore for huge datasets.
• Developed a parallel data loading framework which captured data from 10 different OLTP systems into single relational database using Bulk Load and Partitioning.
• Designed memory efficient architecture using in memory cache (REDIS) to find DISTINCT VISITORS COUNT which facilitated faster processing of SQL queries and increased overall concurrency of the SQL Server system.
• Introduced automated cross-database deadlock and error detection, while establishing disaster recovery solutions including Backup, Log Shipping, and Replication for relational databases.

Dec 2014 – Jul 2015 · 7 mos · Mumbai, Maharashtra, India

• Worked as Data Analyst.
• Engaged in diverse data analytics projects encompassing demographic and accounting data, assuming duties such as data validation, database normalization, and deriving insights through proficient utilization of Tableau and Excel. Additionally, contributed to the creation and deployment of a scalable spark application targeting the identification and elimination of duplicate personnel/vendors, executed across a robust cluster of 10 AWS EMR instances.

Jul 2011 – Dec 2014 · 3 yrs 5 mos · Mumbai, Maharashtra, India

• Worked as a data warehouse developer and ETL developer to create BI Solution for the clients.
• Engaged in extracting data from diverse transactional databases, orchestrating its progression through Staging, EDW, and Datamart utilizing ETL tools such as SSIS and Informatica. Proficiently crafted optimized SQL, stored procedures, functions, and views, along with generating insightful SSRS reports via SSRS and SSAS.