Jul 2023 – Apr 2024 · 9 mos · Gurugram, Haryana, India · Remote

• Architected and optimized high-throughput streaming infrastructure serving millions of users, leveraging Spark, Scala, Kafka, and Nifi to reduce data processing latency by 75% and achieve 4x pipeline acceleration through advanced multithreading optimization at RDD, Kafka, and API levels.
• Led critical system modernization initiative migrating legacy batch processes from Drools to real-time Spark Streaming, delivering 90%+ performance improvement and reducing processing time from 20 minutes to near real-time execution, enabling 3-4x revenue growth for 8+ enterprise clients through enhanced real-time decision-making.
• Designed scalable, reusable Rule Engine framework with trigger logic for dynamic rule activation/deactivation, reducing development and deployment time by 100% for new rules, eliminating manual development overhead and cutting deployment cycles to integration testing only.
• Built production-grade streaming pipelines for mission-critical modules including Reporting, Rule Engine, Notification, Notification Retry, and API Calling with comprehensive testing coverage (unit/integration), ensuring enterprise-grade reliability and seamless deployment.
• Delivered customer-centric solutions through direct stakeholder engagement, translating complex business requirements into technical implementations while maintaining high customer satisfaction through proactive bug resolution.
• Drove security and infrastructure excellence by upgrading Log4j across 10+ microservices (Spark, Scala, Kafka, MySQL stack), eliminating critical vulnerabilities while maintaining zero downtime and reducing system downtime by 70% through optimized multi-server deployments.
• Enhanced data platform capabilities through end-to-end pipeline management, Kafka cluster optimization, and comprehensive Nifi automation (setup, version control, CLI), improving network data ingestion efficiency by 20% and overall data flow by 75% while conducting advanced R&D on Kafka rebalancing solutions.

Oct 2021 – Mar 2023 · 1 yr 5 mos · Bengaluru, Karnataka, India · Hybrid

• Team: Innovation and Insights Team (CTO’s Office)
• About Company: Data observability/actionability platform providing complete visibility and AI-powered optimization for data pipelines across multi-cloud environments. Trusted by Fortune 500 companies including Adobe, Mastercard, and others.
• As part of this team, responsible for developing innovative prototypes to delight customers and fuel growth.
• Implemented the Spark insights feature 'gap analysis' within Unity-One and Unravel apps, identifying runtime delays and delivering actionable optimization steps that enhanced Spark job performance analysis. This feature continues to be used by many customers to this day, indirectly contributing to revenue growth through improved customer satisfaction and platform value.
• Designed and developed an ETL pipeline to collect cluster event data from Databricks using the Clusters API, ingesting into ElasticSearch to generate real-time insights that reduced critical events by over 75%.
• Refactored and enhanced the Comparator App to support multiple Unravel app versions by accommodating data source and schema changes, increasing customer accessibility by 30%.
• Automated application deployment across instances using Jenkins pipelines, achieving a 90% reduction in deployment time.

Jul 2019 – Oct 2021 · 2 yrs 3 mos · Navi Mumbai, Maharashtra, India · On-site

• Team: Analytics Center of Excellence (CoE)
• About Company: Jio, India’s leading telecommunications and digital services provider, serving over 500 million subscribers and processing multiple petabytes of data daily through vast, scalable data infrastructure and analytics platforms.
• Data Engineer (Job Role) (Official Title: Manager) at Analytics CoE, driving big data platform development on JBDL (Jio Big Data Lake), contributing to up to 30% improvement in data processing efficiency.
• Planned, designed, and managed scalable big data platforms processing petabytes of data from multiple sources by implementing optimized end-to-end data pipelines.
• Built robust in-house big data capabilities, replacing external vendors and saving 80% of resources.
• Contributed to the development and use of in-house ingestion, business, and platform frameworks, building multiple data use cases that leveraged these tools for optimized processing and analytics.
• Developed KPIs using HDFS and Hive, visualized on dashboards to deliver actionable insights that improved network latency of Jio’s cellular and fiber networks by 40%.
• Deployed and scaled data solutions involving ingestion, preprocessing, aggregation, and visualization across batch, mini-batch, and streaming using Spark, NiFi, Kafka, Scala, Hive, and visualization tools.
• Scaled infrastructure to handle increased data volumes and pipeline complexity while ensuring high reliability and performance, and applied engineering and data analytics expertise to drive business strategy and data-driven decision-making.
• Contributed to the upgrade from HDP 2.0 to HDP 3.0, followed by a successful proof of concept (POC) and migration from HDP 3.0 to Cloudera Data Platform (CDP), and supported a hybrid cloud strategy by migrating select workloads from on-premises HDP/CDP clusters to Azure, evaluating performance, scalability, integration, and cloud interoperability to modernize the data platform.

Dec 2018 – Apr 2019 · 4 mos · Sendai, Miyagi, Japan · On-site

• Completed Master’s Thesis in Computer Vision (Grade: 9/10) on “Detection of components of the building structure (bridges) from their images using deep learning”
• Researched and implemented convolutional neural networks (CNNs) for edge detection and object detection of bridge components, with the objective of identifying areas prone to cracks.
• Encountered two main challenges: generation of training data and improvement of detection accuracy. To address the first, generated synthetic images from a 3D bridge model, applied data augmentation, and validated results using real bridge images. For the second challenge, examined the differences between synthetic and real data, identified variations in noise and illumination, and developed methods to minimize their impact, thereby making both domains more consistent for better detection performance.
• Discovered that incorporating edge maps into detection tasks produced more accurate results than relying solely on original images.
• Concluded that leveraging synthetic training data greatly reduces time and effort compared to collecting large volumes of real-world data, while still ensuring reliable performance.

May 2018 – Aug 2018 · 3 mos · Tromsø, Troms og Finnmark, Norway · On-site

• Computer Vision Intern in the Department of Engineering Science and Safety (Automation)
• Project: "Investigating the role of asymmetry and symmetry operators in visual attention"
• Applied fundamental mathematical principles to understand underlying mechanisms of visual attention.

Jul 2017 – May 2018 · 10 mos · Roorkee, Uttarakhand, India

• HEC is IIT Roorkee's most famous cultural club. It is responsible for organizing various events such as treks, rafting, paragliding, CAT course, cycle race, adventure activities, etc.
• Around 1500 students participate in all these events held throughout the year. HEC has a committee of approximately 30 students, which manages the club's activities.
• Secretary of Fitness Conditioning and Training.

Jul 2016 – Apr 2017 · 9 mos · Roorkee, Uttarakhand, India

• - Joint Secretary of Database Management and Design.

Aug 2016 – Sep 2017 · 1 yr 1 mo · Mumbai

• Achieved Gold level during Quantitative Research Summer Program.
• Ranked within ALL INDIA TOP 40 on the leaderboard.
• Involved in building high sharpe ratio alphas to determine the risk-return profile of equities.
• Analysed the financial market, worked with basic stock pricing and volume data, performed simulations, interpreted results, and converted ideas to algorithms.
• Implemented trading strategies in Python using scipy, numpy, scikit-learn, etc to place bets on instruments which are expected to be profitable in long run.

Jul 2014 – Apr 2019 · 4 yrs 9 mos · Roorkee, India

• Master of Thesis Grade: 9.0 (Computer Vision)
• Some of the courses taken:
• C++, Data Structures, Design and Analysis of Algorithms, DBMS, Graph Theory, Discrete Mathematics, Linear Algebra, Linear and Non-Linear Programming (Operations Research), Mathematical Statistics, Computer Vision, Digital Image Processing, Statistical Inference, Soft Computing, Numerical Methods, Number Theory, Theory of Computation, Evolutionary Algorithms.