Apr 2022 – Aug 2025 · 3 yrs 4 mos · Bengaluru, Karnataka, India · On-site

• Leading the Development and Growth for Flink, Hadoop and Trino offerings for AWS EMR on EC2/EKS and Athena.
• [0] https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-runtime-for-apache-spark-with-emr-s3a/
• [1] https://docs.aws.amazon.com/emr/latest/ReleaseGuide/trino-ft.html
• [2] https://docs.aws.amazon.com/emr/latest/ReleaseGuide/presto-spot-loss.html
• [3] https://docs.aws.amazon.com/emr/latest/ReleaseGuide/presto-strict-mode.html
• [4] https://aws.amazon.com/about-aws/whats-new/2023/11/data-lake-queries-amazon-athena-s3-express-one-zone/
• [5] https://aws.amazon.com/blogs/big-data/run-trino-queries-2-7-times-faster-with-amazon-emr-6-15-0/
• [6] https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-express-one-zone-storage-class-emr/

Apr 2021 – Apr 2022 · 1 yr

• Azure HDInsight

Aug 2020 – Mar 2021 · 7 mos

• Azure HDInsight

Apr 2018 – Aug 2020 · 2 yrs 4 mos

• Initially was part of Cosmos Resource Management Team, where most of the work is mainly focused with Apache YARN running on 100k Nodes.
• Developed a framework to support Azure CosmosDB and any Document Store vendor as a backend for Apache YARN ATSv2, also contributed this work to OSS as YARN-9016.
• https://issues.apache.org/jira/browse/YARN-9016
• Built the Infrastructure to support autoscale, based on load or schedule for Azure HDInsight Clusters and later led the project for GA.
• https://azure.microsoft.com/en-gb/updates/autoscale-for-azure-hdinsight-is-now-general-available/
• Actively contributing patches to OSS Hadoop.

Apr 2019 – Apr 2020 · 1 yr · Bengaluru, Karnataka, India

Apr 2017 – Apr 2018 · 1 yr

• ◘ Built a real-time stream upserts framework end to end, some key features are
• A continuous running pipeline listening to a topic on Apache Kafka and can perform UPSERTS in real-time on Distributed FileSystem i.e ObjectStore (S3) or BlockStore(HDFS)
• Users can parallelly query the data from FileSystem as Hive External Tables via Presto / Tez / SparkSQL
• All the UPSERTS are eventual consistent while reading and no locks are applied i.e READ and UPSERTS are Parallel.
• Technologies/Components Used:
• Apache YARN with Node Labels for compute.
• Apache Beam for building the continuous running pipeline framework.
• Apache Spark / Flink on Yarn for pipeline orchestration and Disaster recovery.
• In-house Data Governance Warehouse MetaData Service to sanitize the messages being read from Kafka.
• ◘ Part of Building KAAS(Kafa As A Service) with AuthN, AuthZ, Audits and Multitenancy for entire OLA Engg. I developed a ConfigSVC end to end for orchestrating and managing Kafka Clusters and Clients.

Apr 2016 – Mar 2017 · 11 mos

• Built the Data Lake for Big Data from scratch for democratising and discovering of data for analytics, starting form Preparation -> Ingestion -> Cleaning -> Transformation/Processing -> Consumption
• Preparation - All the transactional(Entities) or non transactional(Events) systems data in OLA are prepared by registering the metadata in the warehouse with additional PIE(Processing , Ingestion and Execution times) semantics in each payload.
• Ingestion :
• Push Based - All the data payloads would be directly pushed to a messaging system via ingestion
• library.
• Pull Based - For MySQL by reading binlogs and taking care of the schema management automatically. For NoSQL DB, I wrote a custom sqoop from scratch for bulk pulling the data which syncs the schema automatically with the warehouse metadata and stores the data in the FileSystem by skipping the messaging queue. This was also used for backup and disaster recovery.
• Cleaning - This is aka journalling, where all the entities/events would be deduplicated and partitioned by date and hour on a particular datetime field to answer the question as of then data for entities.
• Transformation - Reconciling(snapshotting) of the data periodically by fetching the changed data from Journal Store, to answer the question as of now data for entities.
• Consumption - The consumption of the journal and snapshot related data were exposed as external hive tables and were queried via Hive/Tez/Spark/Presto engines.
• Technologies/Components Used:
• HDP Distro Hadoop 2.7 cluster managed by Ambari
• Compute via Yarn and Storage via S3A
• Hive , Tez, Presto and Spark for querying
• Dropwizard for creating microservices
• Customised version of Maxwell binlog parser project from github
• Customised version of Kafka to S3 project called Secor from github
• In house config service for resolving application config for microservices on marathon
• Mesos, Docker, Marathon for microservices orchestration
• Oozie/Azkaban for scheduling

Jul 2014 – Mar 2016 · 1 yr 8 mos · Greater Bengaluru Area

• Was part of developing a batch system to process system generated logs which was around 6-7TB of data getting ingested on normal days and 50+TB on sale days as part of the Data Governance practice and enable processing pipelines to run on them by the data analysts.
• Generated Canned Reports by writing processing pipelines for funnel analysis by mining the logs that was not available from Omniture Software (Site catalyst) for Analysis's .
• Later served as part of the Infrastructure and Systems Engineering team of Flipkart's Data Platform where we developed in house Cloud Platform Products through open source Big Data technologies. Some of the key features are like porting compute systems(i.e Spark, Storm etc) on YARN, metering ,billing, auditing , security etc similar to a private cloud. I developed org wide monitoring tool end to end for all the components, contributed to DC Migration for transferring PB’s of data across DC, benchmarking 1000+ node hadoop cluster by writing my own custom benchmarks to handle the Big Billion Day sales load. Open Sourced Project : https://github.com/flipkart-incubator/BlueShift

Jul 2013 – Mar 2014 · 8 mos · Bangalore

• Worked as a research assistant under Prof.T Satish Kumar in my final year undergrad.The research was based on applying compiler optimisation for programs running on distributed/multicore systems and also published two Journals on it.
• 1 - Optimizing Code by Selecting Compiler Flags using Parallel Genetic Algorithm on Multicore CPUs
• 2 - Compiler Phase ordering and Optimizing MPI runtime parameters using Heuristic Algorithms on
• SMPs

Jan 2006 – Apr 2008 · 2 yrs 3 mos · Bangalore Urban, Karnataka, India · On-site

• Freelance Tabla Player for various artists and events in Bangalore.