Jan 2020 – Oct 2022 · 2 yrs 9 mos

• At NetApp, I led several key initiatives aimed at enhancing data reliability, platform efficiency, and operational excellence. My role encompassed the management of diverse teams, the implementation of cutting-edge technologies, and the adoption of AGILE methodologies to deliver high-value projects.
• Key Projects:
• ZeroOps: Developed an in-house proactive monitoring platform using open-source technologies to ensure data and platform reliability across Kafka, Kubernetes, and Spark jobs. The platform, built on the principles of Detect - Notify - Act, leverages ML algorithms for issue detection and automates resolutions, significantly improving system uptime and efficiency.
• Data Lake Modernization: Spearheaded the modernization of a multi-PB data lake to enhance query performance and reduce operating costs. By introducing Dremio on StorageGRID object storage, we achieved a 4x improvement in query performance and a 60% reduction in costs, ensuring scalability and reliability for critical business insights.
• Active IQ Data Engineering SRE: Managed a dedicated SRE team focused on automating monitoring and incident resolution to minimize critical incidents and maximize platform availability. Additionally, oversaw a team of L2 engineers for first-line support across various data platform components, driving continuous improvement and operational excellence.
• Technologies & Methodologies:
• Expertise in Prometheus, Victoria Metric, Python, MLFlow, Anomaly Detection Algorithms, Dremio, Hive, Kubernetes, Airflow, and Dynatrace.
• Championed AGILE practices, set up high-performance teams, and fostered a culture of continuous learning and improvement.
• In each of these roles, my focus was on driving innovation, ensuring data integrity, and enhancing the operational efficiency of our platforms, contributing significantly to NetApp's data-driven success.

Nov 2018 – Jan 2020 · 1 yr 2 mos

• 1. Zero Ops
• AI-driven platform to Proactively Detect, Notify and Auto Resolve (Self heal) different components of complex distributed computing systems.
• Tech Stack - Python, Docker, Time Series Analytics, NLP, TensorFlow(below Keras).
• 2. Vital Signs
• AI-driven platform to Analyze, Identify and provide actionable insights to customers for the best utilization of NetApp storage systems.
• Tech Stack - PySpark, Python, Numpy, Pandas, Keras, TensorFlow(below Keras).

Nov 2015 – Nov 2018 · 3 yrs

• 1. Big Data Platform (ACE DeepIQ)
• Big Data processing Platform.
• Tech Stack - Spark Streaming, Kafka, Scala, Python.
• 2. NDU
• NDU is an initiative to modularise Core business processing engine from external interactions.
• Processing speed ~ 10K message per min.
• Tech Stack - Spark Streaming, Kafka, Scala, Python.

Aug 2015 – Nov 2015 · 3 mos · Bengaluru Area, India

• At Nexti, I worked on an advanced data analytics platform designed to streamline the processing and analysis of high-volume data feeds. Leveraging a robust architecture, we efficiently managed data flow from ingestion to storage and analysis:
• Data Ingestion: Integrated IBM MQ with Nexti for reliable message subscription, ensuring seamless data capture from the central distribution system.
• Stream Processing: Utilized Kafka as the central message hub, where messages were efficiently queued for processing.
• ETL & Analytics: Implemented Spark Streaming to perform real-time ETL operations, transforming raw data into actionable insights.
• Data Storage & Reporting: Stored processed data in Cassandra, enabling the generation of comprehensive reports and supporting ad-hoc queries for in-depth analysis.
• This role involved working closely with cutting-edge technologies to enhance data reliability, scalability, and accessibility, contributing to more informed decision-making processes.

Oct 2014 – Aug 2015 · 10 mos

• Contributed to the development and enhancement of a comprehensive Big Data Analytics Platform, focused on optimizing the storage, retrieval, and analysis of extensive log files from various devices. Key responsibilities and achievements include:
• Centralized Data Repository: Spearheaded the consolidation of log data into a unified repository, enabling efficient data management and accessibility.
• Data Access & Reporting: Implemented dynamic data access mechanisms, catering to diverse request types and facilitating the generation of timely reports and ad-hoc queries.
• Real-Time Data Streaming: Enhanced the platform with a real-time data streaming feature, allowing end-users to access and analyze data instantaneously, supporting informed decision-making.
• Performance Optimization: Achieved a significant milestone by optimizing query response times to 0.5 seconds on a massive dataset of 30TB, demonstrating exceptional skill in performance tuning and data optimization.

Jan 2014 – Sep 2014 · 8 mos

• Played a pivotal role in the evolution of ASUP .Next, an advanced framework designed for the efficient storage and retrieval of data snapshots (ASUPs) from storage units worldwide. My contributions to this innovative project included:
• Data Integration & Storage: Led the integration of ASUP data into the HDFS storage system, ensuring timely and accurate data dumps from global storage units after thorough parsing.
• Optimized Data Retrieval: Developed a sophisticated multi-stage data retrieval process, starting with Solr to leverage its multi-index capabilities, followed by HBase for path details, and culminating in HDFS for final data access.
• Performance Enhancements: Implemented a specialized retrieval path through NFS for specific requests requiring lower response times, significantly improving SLA adherence and system efficiency.

Jun 2011 – Dec 2013 · 2 yrs 6 mos

• Worked as Developer in EDW space.
• As part of project, worked on Hadoop, Map Reduce, Hive, Java and Talend

Nov 2010 – Dec 2013 · 3 yrs 1 mo

• I have been primarily working on EWD projects that involve both ETL and migration.
• Technologies worked on : Big Data - Hadoop, Map reduce, Hive , Java , talend
• In-depth knowledge about the latest Teradata utility TPT.
• Closely worked on architecture and design of project.