Feb 2021 – Dec 2022 · 1 yr 10 mos · Singapore

• → Technical lead for Growth pillar responsible for driving multiple work streams around building data pipelines and datasets for analytics, experimentation and BI.
• → Role encompassed building data pipelines, quarterly project planning, technical leadership, stakeholder management and translating business requirements into actionable technical specifications.
• → Facilitated the adoption of Google cloud platform for building data pipelines such as Dataflow, Bigquery and Looker..
• → Developed experiment metrics calculation pipelines and dashboards on GCP stack that were used by 8 different teams to track experiment performances.
• → Developed a complex data pipeline that calculated various diversity and nicheness metrics on user’s follow graphs and favorite graphs.
• → Developed looker models on top of BigQuery datasets for building dashboards for data scientists.
• → Tech Stack: Google Cloud, Airflow, Scio, Spark, Dataflow, BigQuery, Looker, Scala, Java, Python.

Apr 2018 – Feb 2021 · 2 yrs 10 mos · Singapore · On-site

• → Primary contributor in developing the real-time analytics pipeline by adoption of open source Redhat Debezium tool for CDC ingestion of 60+ source databases.
• → Deployed and managed 60+ Debezium connectors on 4 kafka connect nodes that ingested 1500+ topics into Kafka.
• → Datalake : Primary developer in developing spark-based batch data loader to ingest data from various data sources like MySQL RDS, postgres, MSSQL Server, DynamoDB, ElasticSearch etc to build a datalake on AWS S3. The data loader supported data partitioning, versioning and parallelism.
• → The S3 datalake had 2200+ tables from various data sources and was available for querying via presto engine. The datalake also supported downstream data pipelines to build a data warehouse.
• → Contributed 2 bug fixes on Redhat Debezium open source project - [DBZ-927, DBZ-935].
• → Tech Stack: AWS Cloud, Spark, Kafka, Debezium, Presto, Scala, Airflow, Python, Java.

Aug 2016 – Mar 2018 · 1 yr 7 mos · Bengaluru, Karnataka, India

• → Developed a kafka-based ingestion module that ingested CDC events from MSSQL database and propagated the change event through the real-time processing data pipeline using Kafka.
• → Developed batch ingestion pipeline with a distributed rate limiter that read data from HDFS/Vertica and published to an API. Implemented a critical feature i.e. Distributed Rate Limiter which throttled the POST request to REST API.
• → Received 3 spotlight awards for the contributions in various projects.
• → Tech stack: Kafka, Hadoop, Spark Streaming, Kafka Connect, Zookeeper.

Oct 2015 – Apr 2018 · 2 yrs 6 mos

Feb 2015 – Aug 2016 · 1 yr 6 mos · Greater Bengaluru Area

• → Primary contributor in developing Query as a service(QAAS) platform using Apache lens. QAAS was a scalable query service that provided abstraction over data sources such as Hive, Vertica, ElasticSearch and support query caching using Redis.
• → Became open source committer and PMC for Apache Lens by contributing on multiple features and bug fixes to the project; Contributed a patch to Apache Hive as well [HIVE-11485].
• → Tech stack: Apache Hive, Hadoop, Apache Lens, Vertica, ElasticSearch, Java

Jan 2014 – Jan 2015 · 1 yr

• → Primary developer in building a forecast platform which was an E2E solution from data ingestion to data aggregation, producing time series data, generated weekly forecasts for ad requests, clicks, impressions, revenue etc and made them available on publisher web UI interface via REST API.
• → Developed the java-based forecasting library that consisted of various preprocessing modules like outlier detection, filters, seasonality calculator ; various time series forecasting models like Holt winters method, double exponential smoothing, post-processing modules for biasness handling etc.
• → The forecast platform also computed various error metrics like MAPE, RMSE, directional error, model win ratio etc. that we used to further fine-tune the forecasting models.
• → Tech stack: Java, Hadoop, Mapreduce, postgres, MATLAB

Apr 2012 – Dec 2013 · 1 yr 8 mos

Aug 2010 – Mar 2012 · 1 yr 7 mos · Gurugram, Haryana, India

• → Developed custom ETL pipelines using informatica, SQL procedures, custom java hooks and bash scripts for client implementations.
• → Engaged with customers for implementation, deployment and support on different client projects.