Aug 2018 – Oct 2020 · 2 yrs 2 mos · Bengaluru Area, India

• AWS Migration
• AWS migration is to migrate data & ingestion pipelines to AWS infrastructure. I was responsible for
• 1. Designing strategies for the data & pipelines migration
• 2. Capacity/Infrastructure planning
• 3. Security review
• 4. Leading the development effort for migration
• Migrated around 8000+ tables, 15PB+ data without any downtime.
• Unified Ingestion Platform
• Unified Ingestion platform is a data ingestion platform that replicates data from any source type (DB, File, API, etc) to Data lake (AWS) in near real-time. I was responsible for
• 1. Designing and developing Kafka connect based collator service
• 2. Designed and developed an efficient data reconciliation platform. Some of the features include data snapshotting, backfill, replayability, etc
• 3. Designed and led the development of Self-serve onboarding of datasets/tables. It supports provisioning of hardware resources/components for datasets, efficient scheduling of pipelines along with pipeline setup.
• 4. Infrastructure/resources/cost optimization

Feb 2016 – Jul 2018 · 2 yrs 5 mos · Bengaluru Area, India

• R3D3
• I was part of the R3D3 team. It collects CDC (change data capture) events (events from Redo logs/Bin logs) in real-time from RDBMS and replicates them to Hive in batches. I was responsible for
• 1. Designing and developing collator service, which replicates data from Kafka and stores it in HDFS.
• 2. Designed and contributed to efficient Data Merging/reconciliation algorithms.
• 3. Self serve platform for onboard datasets
• 4. Capacity planning and resources optimization
• EventTrackingSystem
• Event tracking system is a data pipeline auditing framework. I have designed and led the development of this system. This framework detects if there is any event loss in the pipeline in near real-time.

Apr 2014 – Jan 2016 · 1 yr 9 mos · Bengaluru Area, India

• Profile Matching Framework
• Entity/Profile matching framework is a scalable matching framework that does the entity/profile matching in real-time and develops relationships between entities in a Graph Store. Developed matching algorithms based on fuzzy matching techniques.
• Profile Attributes Aggregator
• This framework collects attributes of a profile from various different sources, unions the different attributes to create a unified profile of an entity, and stores it in Profile Store.

Oct 2014 – Dec 2018 · 4 yrs 2 mos

• Apache Drill is distributed query engine on top of any data store.
• 1. Developed Mongo Storage Plugin for Apache Drill. This capability allows Apache Drill users to query MongoDB using ANSI SQL.
• 2. Developed Kafka Storage Plugin for Apache Drill. This capability allows Apache Drill users to query Kafka using ANSI SQL.
• 3. Enhanced Avro file format support for Apache Drill. Added support for Schema validation, support for nested data structures.

Aug 2012 – Mar 2014 · 1 yr 7 mos · Bengaluru Area, India

• Clickstream Ingestion Platform
• Designed and developed a Clickstream ingestion Platform. This platform
• 1. Ingests clickstream and transaction logs from different App servers to Hadoop.
• 2. Standardizes and cleanses clickstream and transaction logs and stores them in Hive Warehouse
• Analytics
• Developed applications for
• 1. Finding the Clickthrough and Conversion rate of each web page.
• 2. Cart abandonment.
• 3. Top Selling products/queries
• Search & Autosuggestion
• I was part of the search team developing/analyzing users' queries.
• 1. Developed applications to normalize user queries,
• 2. Mining frequently searched queries in a user session
• 3. Developing query recommendation using the collaborative technique
• Recommender System
• Designed and developed Recommender systems
• 1. Product recommendation using content & collaborative filtering
• 2. Personal recommendation using Matrix Factorization Techniques.
• 3. Developed scalable Recommendentation API

Jul 2011 – Jul 2012 · 1 yr · Hyderabad Area, India

• Developed Video analytics applications for TubeMogul client using Hadoop.

Jul 2010 – Jun 2013 · 2 yrs 11 mos

• Contributed around 16 patches, all of which are accepted/committed.
• Contributions link - tinyurl.com/ypeackdz

Jan 2010 – Jun 2011 · 1 yr 5 mos · Bengaluru Area, India

• I was part of the Platform team, worked on open source technologies like Apache Hadoop MapReduce/Yarn and Restlet. As part of MapReduce/Yarn enhancement & stabilization, fixed many critical bugs and contributed many new features. Actively participated in Apache Open Source community and contributed some of the enhancements done back to open source. I have contributed 16 patches and all of which are accepted.
• Contributions - tinyurl.com/ypeackdz