Nov 2024 – Mar 2026 · 1 yr 4 mos · Bangalore Urban, Karnataka, India · Remote

• As a Software Architect at Confluent, I design and implement enterprise-grade data streaming solutions for major financial institutions. My focus lies in building high-performance, secure, and fault-tolerant Kafka-based systems that enable real-time data processing.
• I work closely with clients to design monitoring frameworks, integrate advanced security measures, and ensure robust data flow across critical banking platforms.
• Deployed low-latency, high-throughput data streaming applications for ICICI and DBS, supporting petabyte-scale workloads.
• Built a custom monitoring solution using Spring Boot to track Kafka lag, under-replicated partitions, and cluster health across Confluent Platform and Confluent Cloud
• Implemented advanced security integrations, including LDAP and mTLS, ensuring encrypted communication between Kafka brokers and client systems
• Designed Proofs of Concept using Apache Flink, Kafka, and Spring Boot to enable real-time analytics and decision-making for banking clients
• Improved Kafka producer-consumer performance through fine-tuning configurations, reducing latency, and enhancing system reliability in production setups
• Apache Kafka, Confluent Platform, Apache Flink, Spring Boot, Data Streaming Architecture, mTLS & LDAP Security, Real-time Data Processing

Dec 2020 – Oct 2024 · 3 yrs 10 mos · Bengaluru, Karnataka, India

• As a Senior AWS Data Engineer at J.P. Morgan, I focused on designing and building high-performance data pipelines and cloud-native solutions.
• My role involved developing streaming and batch data workflows, optimizing data movement across AWS services, and enabling secure and efficient analytics delivery. I worked extensively with Spark, Glue, Redshift, Lambda, and Terraform to support large-scale financial data operations.
• Engineered multiple Spark Streaming applications on EMR to process and analyze large data streams in real time
• Automated nightly ETL workflows using AWS Glue, transforming data and loading it efficiently into Redshift for analytics
• Designed a Lambda-based data viewing pipeline, improving query efficiency and user access to transformed data
• Architected an EKS cluster to handle high-volume API requests exceeding API Gateway payload limits
• Built infrastructure as code using Terraform (HCL) to provision AWS components, ensuring consistency and scalability
• Developed Python and Scala utilities to join, load, and register data across Redshift, S3, and DynamoDB, improving data accessibility and reliability
• Recognized for innovation and technical excellence by winning the internal Hack-AI-Thon competition
• AWS EMR, AWS Glue, Redshift, Apache Spark, AWS Lambda, Terraform (HCL), DynamoDB, Amazon EKS, Python, Scala

Nov 2019 – Dec 2020 · 1 yr 1 mo

• As a Senior Data Engineer at Hewlett Packard Enterprise, I developed near-real-time data streaming and analytics pipelines for large-scale storage systems.
• My work involved building Spark-based solutions, managing distributed data stores, and ensuring secure, efficient data access across diverse platforms. I focused on improving data reliability, scalability, and system responsiveness through robust engineering practices.
• Built Spark DStream applications to efficiently combine and process heterogeneous file formats in real time
• Developed a responsive, high-performance streaming application supporting near real-time analytics and monitoring
• Designed maintenance strategies for Cassandra tables to ensure stable performance and data consistency
• Engineered a near real-time analytics pipeline for parsing and analyzing data from multiple storage array types
• Integrated user-level access control services, enhancing data security and compliance across streaming systems
• Apache Spark, Spark DStream, Cassandra, Real-time Data Processing, Data Pipeline Development, Access Control Management, Distributed Systems

Jun 2015 – Jan 2019 · 3 yrs 7 mos

• As a Data Engineer at Hewlett Packard Enterprise, I built data processing pipelines and analytics models to enhance visibility into storage array performance and utilization.
• I contributed to improving operational efficiency and data-driven decision-making within storage analytics systems.
• Developed Spark batch jobs to parse and process large volumes of storage array data for analytics and reporting
• Created a time-series model to monitor and forecast storage capacity utilization, improving resource planning accuracy
• Implemented a new data pipeline for parsing multiple storage array types with near real-time insights
• Built an AI-powered chatbot to analyze and respond to emails with an 80% accuracy rate, reducing manual workload
• Enhanced data accessibility and timeliness through optimized data ingestion and transformation workflows
• Apache Spark, Data Pipeline Development, Time-Series Forecasting, Python, Real-time Analytics, Natural Language Processing (NLP), Data Modelling

Jan 2019 – Nov 2019 · 10 mos · Bangalore

• As a Senior Consultant at Allstate Solutions, I managed end-to-end data engineering workflows using modern big data technologies.
• I developed Spark-based data products, implemented automated pipelines, and ensured data accuracy and reliability across multiple environments. My work contributed to improving data quality and accelerating analytics delivery for enterprise use cases.
• Designed PySpark and Sqoop-based ingestion frameworks to transform raw data into meaningful, analytics-ready datasets
• Automated workflow orchestration using Apache Oozie, enhancing data pipeline efficiency and visibility
• Built PySpark-driven data quality monitoring jobs to detect and resolve ingestion inconsistencies
• Implemented CI/CD pipelines in Jenkins for seamless job deployment and version control across environments
• Optimized Spark jobs for better execution speed and reduced cluster resource utilization
• Executed large-scale data cleansing and transformation processes to improve dataset consistency and usability
• PySpark, Apache Sqoop, Apache Oozie, Jenkins CI/CD, Spark Performance Tuning, Data Ingestion & ETL, Data Quality Management

Jun 2013 – May 2015 · 1 yr 11 mos · Bangalore

• As an Engineer at Brillio Solutions, I worked on backend development, data processing, and early-stage analytics initiatives.
• My responsibilities included building APIs, creating database procedures, developing web scraping tools, and applying sentiment analysis techniques to unstructured data. I focused on enabling efficient data delivery and generating actionable insights from external sources.
• Developed stored procedures to load and manage data in intermediate staging layers for downstream applications
• Built multiple Spring Boot APIs to deliver data seamlessly from Oracle databases to client-facing services
• Created a web scraping application to collect Disney review data for analytics and reporting
• Performed sentiment analysis on collected reviews, extracting valuable customer sentiment insights
• Improved data ingestion speed and reliability by automating collection and storage workflows
• Spring Boot, Oracle Database, SQL Stored Procedures, Web Scraping, Sentiment Analysis, Python, API Development