sooraj s

DevOps Engineer

Bangalore, Karnataka, India15 yrs 5 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Stackforce AI infers this person is a Data Engineering specialist with expertise in cloud and big data technologies.

HadoopCloud ComputingPythonKafkaPerformance TuningData PipelinesAws

Amazon Web Services (AWS)AnsibleApache KafkaApache SparkApache StormAutomationDjangoDockerFalconFlaskFlumeGitHBaseJenkinsLinux

Data Systems SRE

2 roles

Dec 2023 – Present · 2 yrs 3 mos · Hybrid

Jul 2018 – Nov 2023 · 5 yrs 4 mos · Hybrid

Managing Walmart Production and Development Hadoop clusters and related ecosystem.
Developed an Inventory tracking portal for the datacenter hosted Hadoop clusters using Python and Flask.
Cloud migration strategy and implementation project - automations and support for Walmart tech teams' cloud onboarding.
Cloudera cloudbreak clusters deployment and production support for internal customers.
Automated Custom Google dataproc clusters provisioning for internal projects.
Developed Kerberos keytab download portal for customers using Python and Django.

HadoopPythonFlaskCloud ComputingAutomation

3 roles

Promoted

Jan 2017 – Jul 2018 · 1 yr 6 mos

GRID & DATA Systems
Managing Production Clusters
Hadoop - 5 Clusters [4 LocalDataCenter; 1 GlobalDataCenter]; Version : HDP243 (hadoop 2.7.1);
TotalStorage : 9.5PB; TotalCompute : 48TB memory + 17.5k cores
Kafka - 4 Clusters; Version : (0.10.2.1); TotalNodes : 61; Messages/Day : 95-105 billion
Flume - TotalNodes : 35; Messages/Day : 70-80 billion
Storm – 4 Clusters; Version : 1.1.0; TotalNodes : 20; Mem : 2.6TB; Cores : 800
Storage and Compute Capacity Planning
Collecting the critical metrics / monitor / maintain production clusters healthy / Identify the degradations
Debugging and fixing production issues related to infrastructure and data pipelines / Root Cause Analysis and implementing long term remediation.
Mentoring junior members in the team.
Coordinating the team efforts to ensure timely completion of projects.

HadoopKafkaFlumeStormPerformance Tuning

Jan 2015 – Dec 2016 · 1 yr 11 mos

Hadoop / DataSystem Engineer : Setting up Production infrastructure for Streaming and Batch Processing.
Performance Tuning, Troubleshooting - Hadoop, Flume/Kafka, Storm and Spark Infrastructure.
Production Configuration Management using : puppet, chef, ambari(for hadoop)
Scripting for Monitoring and Infrastructure Automation using bash and Python.
Setting up New Hadoop Clusters. Migrating petabytes of data to new cluster.

HadoopFlumeKafkaStormSparkPerformance Tuning

Feb 2012 – Dec 2014 · 2 yrs 10 mos

Data Systems Engineer
Hadoop Administrator : Deploying hadoop clusters and related services like HBase, Oozie, Falcon. Adding/decommissioning hadoop nodes. MapReduce queue management.
Owning the DataPipelines with >30k jobs per day. Onboarding new projects/jobs in Falcon and Oozie . Troubleshooting hadoop job issues.
Elastic Mapreduce in Cloud.
Data Archival.

HadoopHBaseOozieFalconData Pipelines

Aug 2010 – Feb 2012 · 1 yr 6 mos · Kochi, Kerala, India

AWS / Linux / Hadoop / NoSQL Systems Engineer
Deployment Automation / Monitoring Linux(RedHat, Centos, Ubuntu),Apache,Php,MySql,Qmail/Postfix, Hadoop+Cassandra Systems.
Troubleshooting and Performance Tuning Systems and Applications.
Securing the systems, managing SSL.
POC for Cloud platform - CloudSigma.

AWSLinuxHadoopNoSQL