Dec 2020 – Aug 2022 · 1 yr 8 mos · Bangalore Urban, Karnataka, India

• Collaborated closely with data scientists on multiple use cases, contributing to the development of data-driven solutions that enhanced decision-making processes.
• Built a fraud detection network using Apache Spark, leveraging calculated metrics to identify potential fraud before loan disbursals, which significantly reduced fraudulent activities and improved the loan approval process for merchants.
• Created a data mart for business analysts, enabling easy exploration of data, and built multiple OLAP cubes based on user requests to enhance data accessibility and analysis.
• Conducted code reviews to ensure code quality, consistency, and adherence to best practices, fostering a culture of excellence within the team.
• Developed a scaffolding framework for data pipelines, improving pipeline maintenance, scalability, and reliability for long-term operational efficiency.

Nov 2019 – Nov 2020 · 1 yr · Belfast, United Kingdom

• Worked on a compliance project where all company communications were monitored to detect misconduct and issues related to trading, ensuring adherence to regulations.
• Applied Natural Language Processing (NLP) rules in Apache Spark to enhance the processing and analysis of unstructured data.
• Optimized existing Spark applications, resulting in significant performance improvements and more efficient data processing.

Jul 2018 – Oct 2019 · 1 yr 3 mos · Pune, Maharashtra, India · On-site

• Worked with a client to migrate data from IoT devices, such as smart TVs, smart refrigerators, and smart doorbells, into S3 using Apache Spark, enabling efficient storage and analysis.
• Developed data pipelines to support machine learning models, ensuring smooth data flow and readiness for model training and deployment.
• Contributed as an individual contributor (IC) to the entire project lifecycle, from setting up the scaffolding to deploying production pipelines, ensuring seamless operations.
• Worked on an in-house Configuration Management tool, streamlining configuration processes and improving overall system management efficiency.

Jun 2017 – Jul 2018 · 1 yr 1 mo · Pune Area, India

• Worked on a UI-based product that simplified the creation of data pipelines, enhancing user accessibility and operational efficiency.
• Contributed to multiple aspects of the product, including developing an import/export utility to streamline data integration processes.
• Implemented encryption for data at rest on HDFS, ensuring compliance with security standards and safeguarding sensitive information.
• Enabled security within the Hadoop cluster using Ranger, providing robust access control and data protection.
• Set up and configured a Spark standalone cluster, optimizing data processing and performance for the team.
• Explored the Apache Spark source code to understand its internal workings and successfully implemented a feature for the product.

Mar 2014 – Mar 2017 · 3 yrs · Pune Area, India

• Built a data pipeline for Tableau reports, integrating clickstream and transactional data to deliver valuable business insights.
• Collaborated with the community to resolve issues related to ORC corrupt blocks, ensuring data integrity and reliability.
• Developed a utility to audit job execution, improving monitoring and tracking of data pipeline processes.
• Worked on multiple use cases involving clickstream data, enhancing analytics capabilities and providing actionable insights.
• Developed a web crawler to capture data from two websites and compare product prices, enabling competitive analysis and market insights.
• Contributed to the development of an in-house product for metadata management and data governance, streamlining data organization and compliance.