Jun 2021 – Nov 2022 · 1 yr 5 mos · Hyderabad, Telangana, India · Remote

• Finance on Cloud (FOTC) Program Development: Spearheaded the development of core components for HSBC's FOTC program, automating data processing and report generation using Big Data frameworks like Spark, Scala, and Hadoop Ecosystem tools
• Improved operational efficiency and enhanced data accessibility for stakeholders through modern ETL and orchestration workflows
• Big Data and Spark Optimization: Designed and optimized large-scale data pipelines using Apache Spark with Scala, achieving a 25% increase in data processing efficiency
• Utilized HDFS, YARN, and Hive for distributed data storage and query optimization, ensuring faster analytics on structured and semi-structured datasets
• ETL Development with AWS Glue: Built and deployed highly efficient ETL pipelines using AWS Glue with Python and Scala
• Leveraged Glue's dynamic frames and data frames APIs to enable seamless integration with services like S3, Redshift, RDS, and DynamoDB
• Designed visual workflows in Glue Studio, streamlining data extraction, transformation, and loading
• Data Lake Creation and Monitoring: Developed and maintained centralized data lakes on AWS Glue to facilitate scalable data storage and processing
• Configured monitoring and logging mechanisms using AWS CloudWatch and Glue logs, ensuring quick resolution of errors and proactive job management for enhanced system reliability
• Data Workflow Automation with Azure Data Factory: Automated and orchestrated data workflows across cloud platforms using Azure Data Factory (ADF)
• Designed robust data pipelines to extract, transform, and load data into Azure Data Lake Storage, Synapse Analytics, and Azure SQL Database, reducing manual efforts and accelerating analytics readiness

Jan 2020 – May 2021 · 1 yr 4 mos · Singapore · On-site

• Managed and optimized the MAS Enterprise Data Lake (EDL) on Cloudera Hadoop (HDFS, Hive, HBase), improving system throughput by 35% while achieving 100% compliance with MAS regulatory standards and enforcing RBAC and encryption policies.
• Designed and implemented distributed data pipelines using Apache Spark (Scala, PySpark) for both batch and real-time processing, resulting in a 40% improvement in structured and semi-structured data transformation efficiency.
• Built scalable ETL frameworks on HDFS, Hive, and Apache Oozie, enabling parallelized and compressed ingestion pipelines that reduced ingestion time by 50%.
• Enhanced Hive data modeling using advanced partitioning, indexing, and bucketing techniques to support OLAP workloads, cutting query execution latency by 30%.
• Automated batch data workflows using Apache Oozie, improving reliability and reducing manual operations by 60%.
• Collaborated across business, analytics, and compliance teams to deliver trusted, high-availability datasets for BI and regulatory reporting, accelerating insight delivery timelines by 40%.
• Contributed to data infrastructure readiness for future Lakehouse expansion by aligning with Apache Iceberg and Delta Lake adoption paths and introducing data contracts and schema evolution strategies.
• Developed STRE (Stress Testing & Risk Engine) applications on Cloudera Hadoop, integrating regulatory models (Basel III/IV, CCR) and reducing end-to-end risk calculation latency by 30% through Spark-based parallel processing.
• Engineered Spark ETL pipelines (Scala, PySpark) for processing structured and semi-structured datasets from banking and trade systems, achieving 40% processing efficiency gains and enhancing financial model scalability.
• Designed microservice-based ingestion systems using Spring Boot and REST APIs, improving modularity and reducing dependency bottlenecks across compute layers.

Apr 2016 – Nov 2019 · 3 yrs 7 mos · Hyderabad Area, India · On-site

• Enterprise Hadoop Environment Implementation and Management: Designed and supported a robust enterprise Hadoop ecosystem with expertise in Core Java, Hadoop Distributed File System (HDFS), and cluster administration
• Led capacity planning, performance tuning, and high-availability configurations while collaborating with cross-functional teams, including infrastructure, network, database, and BI teams
• Applied Microservices architecture principles to enhance integration and ensure seamless performance across data engineering systems
• Comprehensive Utilization of Hadoop Ecosystem Components: Extensively worked with a variety of Hadoop ecosystem tools, including Apache Spark, PySpark, YARN, MapReduce, Hive, HBase, Zookeeper, Pig, and HDFS, to enable end-to-end data processing, analytics, and governance
• Enhanced processing pipelines by integrating Scala and Python, driving efficiency within big data workflows
• AWS Glue Integration for Data Lake Solutions: Designed and implemented serverless ETL pipelines with AWS Glue, integrating Glue Crawlers for automated schema discovery and updates
• Enhanced pipeline performance by leveraging AWS Lambda for automation, implementing data quality checks, and enabling schema evolution in a scalable, cost-effective manner
• Applied Microservices principles to design modular, reliable data ingestion pipelines
• ETL Pipeline Design and Development with AWS Glue: Built and managed efficient ETL pipelines in AWS Glue, extracting, transforming, and loading data from diverse sources into centralized data lakes on S3
• Leveraged Glue's features, such as DynamicFrames and DataFrames, to improve data transformation capabilities and analytics readiness, supporting near real-time data availability

Sep 2014 – Nov 2015 · 1 yr 2 mos · Bengaluru Area, India · On-site

• Administration and Management of Big Data and AWS Environments: Administered and managed projects such as UCP, EWS, and Web Activity with a strong focus on AWS cloud environments
• Leveraged advanced knowledge of AWS Redshift, S3, and EC2 to design, manage, and maintain scalable data warehousing solutions
• Ensured high performance, data quality, and availability by implementing performance tuning techniques, enabling efficient analytics and reporting
• Optimization and Deployment of Hadoop Environments: Proposed and implemented optimized configurations for Hadoop environments, including HDFS, YARN, and MapReduce
• Conducted capacity planning and performance optimization for seamless big data processing
• Deployed new hardware and software solutions to enhance scalability and system responsiveness, ensuring the efficient handling of growing datasets
• User Management and Cluster Maintenance in Hadoop: Managed Hadoop user accounts and permissions, ensuring secure access and efficient data workflows
• Administered Hadoop clusters by adding or removing nodes and monitoring cluster health with tools like Ambari, Cloudera Manager, and Core Java for creating custom management solutions
• Maintained data integrity and optimized resource utilization for improved performance
• CI/CD and Build Operations for Data Engineering Projects: Orchestrated CI/CD pipelines using tools like Jenkins, Git, GitHub Actions, and Bamboo to streamline automated builds, testing, and deployments
• Monitored and optimized build processes to ensure efficiency and consistency across data engineering projects
• Enhanced collaboration through version control and automated workflows
• Data Quality, Governance, and Availability in HDFS: Ensured the availability and quality of data stored in HDFS by implementing stringent data governance practices and conducting regular validations

Jan 2010 – May 2013 · 3 yrs 4 mos · Hyderabad, Telangana, India · On-site

• Designing web templates including logos, banner ads, buttons, flash interactions, icons, flash intros, JavaScript and flash animated menus, style sheets etc. Structured and developed full web sites including HTML, XHTML, CSS and JavaScript, JQuery coding from raw level to Dream weaver.
• CMS Administration, CRM Maintenance, Website Backend support and content management.
• SEO On page and off page Optimization. Design and Basic HTML Conversion of Web Pages and Coordinating with the Programmers for further changes and adjustments in designing Web Pages. Coordinating with the Business Development Team in making presentations and creating visuals for the business proposals, SEO On page and off page Optimization. Developing and maintenance of the company website. CMS Administration, CRM Maintenance with content management.