Dec 2019 – Dec 2022 · 3 yrs

• Datacoral was acquired by Cloudera in 2021, and over the past year, I've been working under the Cloudera brand
• Part of the founding R&D team that developed a secure, end-to-end, AWS-native serverless ETL platform, enabling the ingestion, transformation, and loading of data from 60+ sources into data warehouses such as Snowflake, Redshift, and Apache Hive.
• Led the connector team, expanding the data ingestion framework using AWS Lambda, and designed/developed 20+ connectors to address unique big data challenges, enabling the processing of billions of rows.
• Engineered a split and batch framework to streamline parent-child API interactions, enhancing efficiency across connectors.
• Built Change Data Capture (CDC) connectors to extract billions of rows from Write-Ahead Logs (WAL) like PostgreSQL and MySQL, supporting Big Data operations and insight generation with integrated data quality checks.
• Developed Python UDFs optimized for AWS Batch, enabling historical data syncs for over 10 years of data.
• Created scalable and robust REST APIs and a CLI interface to enhance operational efficiency and user experience.
• Designed the warehouse framework for Cloudera EDH driver, ensuring seamless data integration.
• Collaborated extensively with AWS services, including CloudFormation, DynamoDB, Elastic Beanstalk, and Lambda, to build and maintain scalable and reliable infrastructure.

Jul 2019 – Nov 2019 · 4 mos

Sep 2018 – Apr 2019 · 7 mos · India

• Led multiple projects in the Qubole Hive team to enhance the Big Data Platform built on Apache Hadoop for advanced analytics. Optimized Apache Hadoop Map-Reduce, Tez job schedulers, and Apache Hive performance. Improved the 'Cloudman' interface to manage engines like Spark and Presto on AWS EC2 (on-demand and spot-block) instances

Aug 2015 – Aug 2018 · 3 yrs · Greater Bengaluru Area

• Developed database modules for ASE, ADS, and HANA, focusing on security and query performance using C/C++ on Linux. Migrated databases to SAP cryptographic libraries for secure data handling and addressed memory vulnerabilities to protect sensitive data. Led projects on secure auditing, rolling upgrades for reduced downtime, and optimized distributed data on AWS EC2 clusters.