Jul 2021 – Jun 2022 · 11 mos

• Main Duties:
• 1) Define and own end-to-end Software Architecture from definition phase to go-live phase for large and complex systems based on client requirements.
• 2) Build scalable data architectures for data ingestion, storage, transformation, and analysis.
• 3) Design platforms as consumable data services across the organization using the Big Data tech stack.
• 4) Work with Client Product Management and Operations in an agile environment to accomplish team goals.
• 5) Assemble large, complex data sets that meet functional / non-functional business requirements.
• 6) Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• 7) Drive innovation with technologies and solutions in the large-scale distributed cloud services and petabyte-scale data processing space.
• 8) Create solutions as scalable generic reusable organization-wide platforms.
• 9) Collaborate to deliver rapid, iterative technology proof of concepts and support client's request for proposal response by providing cost-effective solutions.
• 10) Provide technical insights, delivering recommendations, tutorials, blog articles and technical presentations, adapting communication to different levels of technical partners.

Jul 2020 – Jul 2021 · 1 yr

• Platform and Skills: Lambda, SNS, SQS, python, Java, REST APIs, ML Algorithms, MSK, Aurora (Postgres), Fargate, Redshift, DMS, Terraform, Cloudwatch, DynamoDB, DocumentDB, S3, Jenkins
• Project Description: Due to COVID-19, Customers are not visiting client's physical stores but using website to purchase product items more via online, and as per predictive analysis there will be a huge increase in online order sales during holidays in 2020. So this is an initiative to optimize the margin associated with shipments per online order and also to minimize split in shipments. It's an inventory operation focused project which has a direct impact on profit per online order and making order fulfilment lifecycle more intelligent via Machine Learning models.

Jan 2020 – Jun 2020 · 5 mos

• Platform and Skills: Spark, Scala, Python, EMR, S3, Redshift, Lambda, Glue, Athena, Redshift Spectrum, SQS, Cloudwatch, DataPipeline, EC2
• Project Description: The Consumer Data Hub (CDH) serves as the technology foundation and Data Platform to connect disparate data from a single, accessible source. Main objective is to understand the 360-degree view of consumer behaviour collected from data across multiple channels (Online, Email, In-Store, Social, others) to make available: a unified consumer profile, cross-channel consumer
• interactions, consumer preferences, rich consumer attribution, and consumer suggestions.
• Key responsibilities include:
• 1) Customers' trusted advisor: collaborate with customer/AWS partner account to deliver highly efficient and cost-effective Data solutions.
• 2) Responsible for creating and optimizing Spark Jobs in production environment both infrastructure and code wise.
• 3) Being a Big Data subject matter expert, key role is to present best Data solutions and Design from
• ideation to execution along with enterprise best practices in Coding for scalable solutions.
• 4) Heavily engaged in Customer Data Platform Development on AWS in Design, Development, Infrastructure, Integration, Automation, Testing, Design Patterns and Orchestration.
• 5) Presented Data Migration Approach for various data sources along with Architecture and associated costs.
• 6) Algorithmic Implementation for a module and it's automated deployment along with orchestration.

Mar 2019 – Dec 2019 · 9 mos

• Platform & Skills: Spark, Hive, Scala, Azure, Cloudera, Azure Data Lake Storage, Azure DevOps, Splunk, Elasticsearch, Flume, Kafka, Oozie, Jenkins
• Brief: Purpose of this project is to migrate Data Platform (named Helix 2.0) Tier-3 environment (Hadoop storage, Batch and Stream Processing) from on-premise to Cloudera on Azure (Iaas followed by ADLS Gen 2), implement security enhancements, build a common ingestion layer capability for Helix cloud platform. Improve product stability with separation of Test, UAT and Production environments, automate the deployment of compute and storage resources in the cloud.
• Key responsibilities include:
• 1) Leading an onshore team in all key milestones, collaboration with client teams for requirement gathering and freezing, demoing deliverables on a weekly basis and key performer to represent the team at client location in front of various stakeholders including Product Owners, Senior Solution Architects to include 360-degree feedback and it’s implementation using Agile methodologies.
• 2) Setup of clusters for all environments and creation of deployment packages for deploying the EDH/ Helix foundational components.
• 3) Implement Physical Design in Azure for EDH requirements
• 4) Implement CI/CD Pipeline for Batch and Streaming Applications in the cloud for EDH.
• 5) Complete the data migration for the project with monitoring and data reconciliation for every component of the data platform.
• 6) Carry out performance tests for Application and Infrastructure Benchmarking.
• 7) Implementation of services for Monitoring and Logging Architecture on Azure.

Oct 2018 – Mar 2019 · 5 mos

• Platform & Skills: Spark, Hive, Scala, Azure, PowerBI, Structured Streaming, Event Hub, Azure Data Lake Storage, Azure Data Factory, STRIIM, MySQL, Campaign Management Tools like Oracle Eloqua
• Brief: The purpose of this product is to build an enterprise Data Platform on the cloud using Big Data Stack for building ETL solution along with real-time dashboarding using PowerBI. The client is having property and entertainment business and wants to leverage the power of Big Data Stack on the cloud for building a data platform. This solution is built on data lake which has different layers like - Ingestion Layer, Processing Layer, Storage Layer and finally KPI is calculated for a specific use case on PowerBI, using both batch and streaming applications.
• Key Responsibilities include:
• 1) Provides architecture for an end to end Big Data Development on Azure.
• 2) Responsible for designing high level and low-level design of Data Platform, as well as its implementation and deployment.
• 3) Collaboration with different stakeholders and teams in an agile manner.
• 4) Optimization of Spark jobs and troubleshooting in different environments.
• 5) Responsible for active development and enhancement of various layers like Data Migration, Data Ingestion, Speed layer which includes PB and TB amount of customer property and entertainment data onto Azure cluster.
• 6) Efficiently leading a complete data development life cycle including Design, Development, Deployment, and Testing.

Feb 2018 – Oct 2018 · 8 mos

• Platform & Skills: Hadoop, Sqoop, Hive, Spark, Python, pyspark, Bash Scripting, Airflow, Redis, GCP
• Brief: The purpose of this product is to build a retail analytics solution on the cloud using a pyspark for building an ETL solution. Some goals are to deliver market-based segmentation (retail analytics algorithm) based on customer buying patterns. This ETL solution is built on data lake which has different layers like - Ingestion Layer, Caching Layer, Processing Layer and various downstream systems leverage information from this data lake based on their requirements.
• Key responsibilities include:
• 1) Provides approach and write retail analytics ETL solution using Big Data Technologies.
• 2) Creating a high- and low-level design of the assigned module.
• 3) Behavior Driven Development of scalable architecture using Big Data technologies.
• 4) Automate deployment process using GitLab CI/CD
• 5) Migrate existing retail domain data to the cloud (Google Cloud Platform).

Mar 2017 – Feb 2018 · 11 mos

• Platform & Skills: Hadoop, Sqoop, Hive, Spark, HBase, Bash Scripting, Java, Python, Scala, Kafka, Flume, MongoDB
• Brief: Data Modernization- A data platform whose objective is to generate insights from the data of different categories of Investment bank. It takes data in EBSDIC format, performs cleaning of data, and push it in data lake where all business logic is there to deal with different use cases. The main objective is to transfer data to Hadoop, create an analytics layer on top of it using Hive and provide insights to Business users and different other platforms when needed using various Big Data technologies.
• Roles and Responsibilities:
•  Worked at the client location, handling investment bank data by leveraging power of Big Data technologies.
•  Worked on the Development of Real-time streaming architecture from scratch.
•  Responsible for designing high-level and low-level design of Data platform, as well as it's implementation and deployment in development and QA environment.
•  Build complex and non-interactive systems batch, distributed, etc in the project.
•  Developed efficient shell scripting jobs that could minimize the code work and increase the
• efficiency of the whole process.
•  Responsible for active development and enhancement of various layers like Data Migration, Data Ingestion which includes PB and TB amount of data of various monetary credit card transactions and customer details into the Hadoop cluster.
•  Generating insights from the huge amount of customer data which involves use case like Fraud Detection, Customer 360 degree, Benefits and Values, etc
•  Developed and Contributed in existing Java, python and scala applications which facilitated the process to ingest data in Hadoop ecosystem in an efficient manner

Oct 2015 – Feb 2017 · 1 yr 4 mos · Gurugram, Haryana, India

• Current role involves following:
• Design and Development of robust data processing pipelines for enterprise-level business analytics.
• Using Big Data Technologies like Apache Spark, Elasticsearch, MongoDB etc to implement analytics engine.
• Learning Predictive Analytics, Content Analytics and Machine Learning.
• Recently worked on creating ETL tool for solving one of the analytics problem.
• Provide solution of various analytics engine problems and then implement it.
• Actively involved and implement various phases of analytics engine development from scratch like: Data Discovery, Data Exploration, Data Prediction, Data Visualization and Dashboards.
• Working with smart and innovative team for solving various complex parallel data processing problems.

Jul 2014 – Oct 2015 · 1 yr 3 mos

• The role involves:
• Writing efficient code and documenting it as per the design specification of module / component
• Deploying components and/ or applications
• Unit Testing, debugging and maintaining code in ones own development environment
• Preparing the low level design document if required
• Low level scheduling of the Module
• Interacting with Senior Software Engineers to get the requirements/design
• Reviewing the code
• Learning release cycle and performing in various phases including packaging and deployment

Jul 2013 – Jul 2014 · 1 yr

• Writing efficient code and documenting it as per the design specification of module/component.
• Deploying components and/ or applications.
• Unit testing, debugging and maintaining code in project development environment.
• Problem-solving and thinking laterally as part of a team, or individually, to meet the needs of the project.
• Participating in design discussions and analyze the problem in detail to understand the key points.