Sep 2020 – Dec 2021 · 1 yr 3 mos · Bengaluru, Karnataka, India · Remote

• Led a team of 5 engineers to automate data pipelines using AWS Glue and Redshift, reducing data processing times by 30% and enabling real-time data ingestion across multiple data sources.
• Designed and implemented data warehouses and optimized data workflows, improving query performance by 25% and reducing storage costs by 15% through better data compression and partitioning strategies.
• Utilized AWS Glue to automate data extraction from S3 to Redshift, reducing manual ETL efforts by 50% and improving data validation accuracy through automated checks, ensuring 99% data integrity in production systems.
• Developed and maintained ETL pipelines for structured and semi-structured data across cloud (AWS) and on-premise databases, streamlining data flows and reducing pipeline failures by 20% through robust error handling and monitoring mechanisms.
• Created data models and optimized Redshift clusters, reducing query run times by 30% and achieving a 20% cost reduction by right-sizing instances and employing optimized partitioning and indexing strategies.
• Collaborated with business teams to define data requirements and designed ETL workflows that improved reporting accuracy by 20% and reduced report generation times from hours to minutes, meeting critical business objectives efficiently.

May 2017 – Aug 2020 · 3 yrs 3 mos · Chennai, Tamil Nadu, India · On-site

• Developed ETL pipelines using Spark (SQL and Core) in Scala, Hive, and Unix, processing over 10TB of data daily, improving pipeline efficiency by 30% and reducing manual interventions through automation, which directly contributed to a 40% increase in online sales for the Ad-tech division.
• Implemented a pluggable and reusable ETL framework with built-in monitoring and error logging, improving development time for new use cases by 20% and increasing operational visibility across all ETL pipelines.
• Developed Avro utilities for dynamic schema generation of daily refreshed tables, reducing development time by 20% and improving data consistency across multiple reporting systems.
• Implemented performance optimizations in Spark applications, including tuning memory executors and cores, reducing job completion times by 35% and improving overall system resource utilization.
• 15+ Data Scientists from 4 agile autonomous teams got the data that they need to forecast sales and demand.
• Developed a pipeline for efficient and intelligent storing of archival data in google cloud and Amazon s3.
• Developed an auditing and orchestration tool in Java to execute Hive and spark scripts in parallel and putting more granular control and error checks for each step. Using Simba JDBC to execute big-queries from the java application.
• Making the application as step level and monitoring and logging the status of each step for better job management.

Nov 2014 – May 2017 · 2 yrs 6 mos · Chennai Area, India · On-site

• Automated ETL processes using Spark and Hive, reducing data wrangling time by 40% and increasing pipeline reliability by implementing robust error handling and job monitoring.
• Designed and implemented reusable generic Hadoop systems, cutting development time by 30% and enabling the team to scale data processing across multiple projects, increasing operational efficiency by 20%.
• Developed and delivered custom reports for CEC management and QA teams, improving performance management and reducing report generation time by 25%, leading to better decision-making and resource allocation within the team.
• Provided in-depth analysis comparing customer and internal evaluations, leading to a 20% improvement in customer service quality by identifying key areas for training and process improvements.
• Utilized Sqoop to ingest and retrieve data from RDBMS systems like MSSQL and Oracle, reducing data extraction time by 15% and ensuring seamless integration with the Hadoop ecosystem for large-scale data processing and analytics.
• Developed a Spark and Scala application to automate seller information extraction, reducing manual data retrieval by 30% and providing real-time reports in XML format, enhancing the business team's decision-making process.
• Tuned Hive and Spark query performance, reducing data retrieval times by 25% and optimizing resource utilization for processing large datasets, leading to a 15% cost reduction in operational expenditures.

Sep 2011 – Nov 2014 · 3 yrs 2 mos · Bengaluru, Karnataka, India · On-site

• Built with Java, Spring and Hibernate the application provides end-to end support for insurance quote to claim processing.
• Creating the operational report for the business team in SQL and xml and involves in bug resolving and enhancement of the previous ops report.
• Bug resolving with respect to System testing and enhancement according to client demand.
• Creating modules for database and web-page communication.
• Bug resolving with respect to System testing and enhancement according to client requirements.
• Created and worked in an application on java with JDBC for the advisor team.
• Creating the operational report for the business team in SQL and xml and involves in bug resolving and enhancement of the previous ops report.