Mar 2020 – Sep 2024 · 4 yrs 6 mos · Hyderabad, Telangana, India · Hybrid

Jan 2015 – Jan 2020 · 5 yrs · Sydney, New South Wales, Australia · On-site

• ● Migrated ETL/ELT workflows from RDBMS to Google cloud using NiFi and airflow thereby reducing the cost of operation for clients.
• ● Spearheaded the development of a comprehensive data processing platform from the ground up. Utilized Scala, Python, Google Dataproc, Apache Beam, and Big Query to manage the seamless migration of 800 databases from SQL Server to Big Query
• ● Optimized processing speed with Pyspark distributed computing, improving processing time by 80%.
• ● Managed and supervised a team of 5 and implemented the Ringelmann Effect rule that resulted in a 30% increase in productivity.
• ● Conducted successful negotiations with stakeholders and successfully onboarded 3 new projects.
• ● Played a crucial role in building and analyzing data samples used for training machine learning models. Leveraged Google Cloud AutoML for streamlined data preparation, contributing to more accurate and efficient machine learning outcomes.

Nov 2012 – Jan 2015 · 2 yrs 2 mos · Pune Area, India · On-site

• Responsible for a new cluster set up, load balancing and setting up new users in Hadoop
• Responsible for migrating historical data of transactions from RDBMS to HDFS on an ad-hoc basis and vice versa using Sqoop and oozie Workflow Scheduler/client request.
• Developing hive ad pig queries to be run in Cloudera for data extraction and filtering of data.
• Creating Custom alerts for jobs monitoring and mail trigger
• Involves designing, capacity arrangement, cluster set up, performance fine-tuning, monitoring, structure planning, scaling, and administration.
• Responsible for installing, administering, Cloudera distribution in Linux and window environment.
• Accountable for storage, performance tuning and volume management of Hadoop clusters and MapReduce routines.
• Implementing the data profiling techniques for certain file formats. • Developed oozie workflow for importing fixed length and flat file data in the hive.
• Created cron jobs and Oozie workflow for data archival (greater than 3 months) to Hadoop Backup cluster.
• Created Stream sets workflows to consume the data from Mysql and mssql to dump in Hive. • Testing and QA coordination and defect closure.
• Manage and analyze Hadoop log files. • File system management and monitoring.
• Migrated transactional Data to Hadoop on-Prem cluster. Built automated flow for data archival in Apache NiFi.
• Worked on integrating SAP HANA into Cloudera Distribution using Spark HANA connector. Ingesting data from RDBMS like Oracle and MySQL.
• Built automation framework using shell scripting for Ingestion. Developed web UI for clients using Bootstrap and HTML 5.
• Developed an application in Java to create XSDs and WSDLs as input in an SOA framework.
• Created RESTful and microservices in Java.
• Developed highly maintainable Hadoop code and followed all best practices regarding coding.