Shivangi Singh — Data Engineer

As an accomplished Data Engineer with over 6+ years of hands-on experience in Big Data technologies, I specialize in designing, developing, and optimizing scalable data pipelines and ETL processes across various cloud platforms including AWS, Azure, and Google Cloud Platform (GCP). My expertise spans a wide range of tools and technologies, from Hadoop HDFS, Apache Spark, and Hive to advanced data transformation and aggregation using Python ,PySpark and Hive. I am proficient in creating robust data models and managing large-scale data storage solutions, ensuring data quality, integrity, and security at every step. My work has involved collaborating with cross-functional teams to translate business requirements into effective data solutions, applying advanced query tuning techniques, and leveraging cloud-enabled platforms to streamline data workflows and generate meaningful insights. In my current role, I have successfully led projects that required extensive data cleaning, preprocessing, and real-time data processing using AWS Kinesis, Glue, and Redshift. Additionally, my experience with data visualization tools like GCP Looker Studio and AWS QuickSight allows me to present complex data in an accessible and actionable format. Driven by a passion for continuous learning and innovation, I am always looking to expand my knowledge of emerging data technologies and methodologies to deliver high-impact solutions. My skill set also includes proficiency in Python, NLP, Flask, FastAPI, Unit-testing, asynchronous functions, Airflow, Postman, Git, and extensive experience in data science POCs. I excel in the development and thorough debugging of end-to-end data pipelines, ensuring the smooth flow of production release of the project. My background in Computer Science, combined with a strong foundation in programming, data analytics, and cloud computing, equips me to tackle the most challenging data engineering tasks. Let's connect if you're interested in data-driven solutions, cloud architecture, or simply exploring how data can drive business success!

Stackforce AI infers this person is a Data Engineer specializing in cloud-based data solutions and big data technologies.

Location: Noida, Uttar Pradesh, India

Experience: 6 yrs 9 mos

Skills

Google Cloud Platform (gcp)
Amazon Web Services (aws)
Natural Language Processing (nlp)

Career Highlights

Over 6 years of experience in Big Data technologies.
Expertise in designing scalable data pipelines across cloud platforms.
Proficient in advanced data transformation using Python and PySpark.

Work Experience

CLOUDSUFI

Senior Data Engineer (3 mos)

CBTS

Senior Data Engineer (1 yr 3 mos)

Capgemini

Senior Consultant (1 yr)

Consultant (2 yrs 3 mos)

Cognizant

Programmar Analyst (1 yr 1 mo)

Programmer Analyst Trainee (11 mos)

Intern (6 mos)

Education

Engineer’s Degree at Lovely Professional University

Senior Secondary School at Kendriya Vidyalaya

Higher Secondary School at Army Public School

Shivangi Singh

Data Engineer

Noida, Uttar Pradesh, India6 yrs 9 mos experience

AI EnabledHighly Stable

Key Highlights

Over 6 years of experience in Big Data technologies.
Expertise in designing scalable data pipelines across cloud platforms.
Proficient in advanced data transformation using Python and PySpark.

Stackforce AI infers this person is a Data Engineer specializing in cloud-based data solutions and big data technologies.

Contact

Skills

Core Skills

Google Cloud Platform (gcp)Amazon Web Services (aws)Natural Language Processing (nlp)

Other Skills

PythonBigQueryCloud StorageDataprocComposerAirflowPySparkNLPFlaskDockerJenkinsShell ScriptingJavaComputer VisionMachine Learning

About

Experience

6 yrs 9 mos

Total Experience

2 yrs 2 mos

Average Tenure

3 mos

Current Experience

Cloudsufi

Senior Data Engineer

Feb 2026 – Present · 3 mos · Noida, Uttar Pradesh, India

Cbts

Senior Data Engineer

Nov 2024 – Feb 2026 · 1 yr 3 mos

Working on a American Express Project named Sales Incentive Project for Data Analytics team whose role is to migrate the system to Google Cloud Platform.
Designed end-to-end GCP architecture using BigQuery, Cloud Storage (GCS), Dataproc, Composer (Airflow), and SFTP.
Designed the end-to-end architecture and led the migration of sales incentive workflows to Google Cloud Platform (GCP) using services like GCS, Composer, Dataproc, and SFTP.
Migrated and automated key data feeds including client signings, bulk volume, and hurdle.
Built and orchestrated scalable ETL pipelines using BigQuery SQL and Airflow for data transformation and delivery.
Generated merged .tar files with processed incentive data and automated secure delivery to clients.
Collaborated with data engineering, QA, and product teams, participating in sprint ceremonies and stakeholder meetings.
Ensured high performance, data quality, and secure operations across Gold, Silver, and Platinum layers.

Google Cloud Platform (GCP)Python

Capgemini

2 roles

Senior Consultant

Nov 2023 – Nov 2024 · 1 yr · Gurugram, Haryana, India · Hybrid

Storing the data into Data refinery and Data Lake with multiple file format and perform partition.
Worked on cloud-enabled platforms such as Google Cloud Platform (GCP) to store the data in GCS bucket using DataFlow ,Dataproc,BigQuery to generate reports in Looker Studio.
Developed and maintained automated data pipelines using GCP Cloud Composer, streamlining data workflows using GCP Workflow and improving efficiency.
Demonstrated expertise in big data technologies, specializing in Apache Spark, to process and analyse vast datasets efficiently and extract meaningful insights using BigQuery.
Conducted comprehensive data cleaning and prepossessing, Optimizing data quality for accurate analysis and modelling.
Collaborate with cross-functional teams to understand business requirements and translate them into data solutions.
Ability to manage multiple tasks and projects effectively.

Google Cloud Platform (GCP)PySpark

Consultant

Aug 2021 – Nov 2023 · 2 yrs 3 mos · Gurugram, Haryana, India · Hybrid

Developed and maintained ETL pipelines using AWS Glue and PySpark to efficiently load data into AWS Redshift. Implemented real-time data ingestion and processing with AWS Kinesis, seamlessly integrating it with S3, Glue, and Redshift for streaming analytics. Managed data storage in AWS S3, optimizing organization, security, and retrieval with partitioning and compression. Leveraged AWS Athena for ad-hoc querying and AWS QuickSight for interactive dashboards. Collaborated with stakeholders to understand requirements, integrate Databricks for data cleaning and transformation, and ensure smooth CI/CD for data pipeline solutions.

PySparkAmazon Web Services (AWS)

Cognizant

3 roles

Programmar Analyst

Jul 2020 – Aug 2021 · 1 yr 1 mo

Designed and implemented an MVP using NLP techniques to identify potential risks for the client, utilizing transfer learning methods such as pre-trained word embeddings, entity detection, and sentiment analysis. Enhanced word search and topic analysis performance through Elasticsearch optimizations. Seamlessly integrated the solution with the client’s ecosystem using Flask, Docker, Jenkins, and other tools, ensuring a robust CI/CD pipeline. Played a key role in the project’s planning, development, deployment, and testing phases, and developed comprehensive documentation to support project approval and stakeholder communication.

Natural Language Processing (NLP)Python