SHIWANGI BHATIA

Software Engineer

Gurgaon, Haryana, India9 yrs 5 mos experience

Most Likely To Switch

Key Highlights

Over 9 years of experience in data engineering.
Expertise in Python and NoSQL technologies.
Proven track record in optimizing data solutions.

Stackforce AI infers this person is a Data Engineer specializing in Fintech and Healthcare data solutions.

Contact

Skills

Core Skills

Apache PysparkAwsData EngineeringHive

Other Skills

AWS EMRAWS(EMR cluster)Amazon Web Services (AWS)Apache GlueApache KafkaApache NiFiApache PigApache SparkApache Spark StreamingApache ZooKeeperAthenaBig DataBitbucketData AnalysisDatabases

About

As a Data Engineer at NatWest Group, I leverage my expertise in Python and NoSQL to design, develop, and maintain scalable data solutions. With over 9 years of experience in the field, I have worked with various technologies and frameworks, including Apache Spark, Hadoop, and AWS. I am passionate about solving complex problems and delivering high-quality results. I hold certifications in Business Analysis and Consulting from Aspiring Minds, and a Python Certification from HackerRank. I am always eager to learn new skills and explore new opportunities in the data engineering domain.

Experience

Natwest group

Big Data Engineer

Aug 2023 – Present · 2 yrs 7 mos · Gurugram, Haryana, India · On-site

American express

Senior Data Engineer

Apr 2021 – Jan 2024 · 2 yrs 9 mos · Gurugram, Haryana, India

Financial Data Engineering

Nagarro

Senior Associate

Sep 2019 – Apr 2021 · 1 yr 7 mos · Gurgaon, India

Senior Data Engineer

Accenture

Pyspark Developer

Mar 2019 – Sep 2019 · 6 mos · gurgaon

Environment:
AWS(EMR cluster), S3 for Storage, Apache Pyspark for ETL processing(Pyspark Dataframes and Spark SQL)
Experience on Converting Existing Data Model from SAS to Pyspark and loading SAS extracts to S3 Buckets
and using EMR cluster on AWS .Using Athena for faster querying and Apache Glue for metadata storage
Responsibilities:
● Responsible for coding existing SAS logic for 200 trench 1 tables in Pyspark and Spark SQL.
● Ingested SAS extracts from SAS server to AWS.
● Dynamic creation of column names using data frame logics using latest year month values
● Implementing merge logic of SAS using joins in Pyspark
● Implementation of clustered tables in Pyspark using Indexing .
● Loading month wise tables into Hive using partitions on yaer/month values.
● Using Thread abstraction to create concurrent threads of execution.

AWS(EMR cluster)S3Apache PysparkETL processingSpark SQLAthena+2

Tata consultancy services

Data Engineer

Oct 2016 – Feb 2019 · 2 yrs 4 mos · New Delhi Area, India

Pharmacy Project
Tasks Involved:-
It was a project involving migration of data from Terdata to Pyspark .
To load data from Terdata to Hive parquet tables , used Sqoop to load the data
Then Created Hive tables where I used and Wrote Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive for optimized performance.
and to Solved performance issues in Hive scripts with use of Joins, Group and
aggregation and how does it translate to MapReduce jobs was analyzed
Then processing of data in Hive for rejection of duplicates and error records
Performed data analysis with HBase using Hive external tables to HBase.
Collecting data using Pyspark and staging data in HDFS for further analysis.
Created Hbase tables to store various data.
worked on loading data from LINUX file system to HDFS.
Importing and exporting data into HDFS and Hive using Sqoop.
Experience working on processing flat files data using Pig and Hive.
Supported MapReduce Programs those are running on the cluster.