Palak Agarwal — Data Scientist

While working on numerous projects in the field of data engineering, my interest to explore new technologies and big data has increased exponentially. Seeking work in fields of data engineering, Data Pipelines, ETL, Big data and, Cloud Technologies • Experience in building pipelines using python, scala and the subsequent trigger notifications for Cadenz Customer Profiling. • Majorly skilled in different Cloud Stack, Big data Technologies, Python, ETL, Building pipelines, Customer data analytics, including customer segmentation, behavioral and direct marketing strategies. • Demonstrated business intelligence and attention to detail. • Developed the feature to automatically on-board the new attributes in Cadenz User Profiling and the subsequent trigger notifications using SNS. • Successfully architectured and independently created the campaign management system via AWS Step Functions, Kinesis and json parser. • Created AWS Data Ingestion Pipelines using pagination to query DynamoDB using AWS lambda for serving the subsequent API calls. • Implemented the Flink framework for calculating the attributes for upto 180 days by maintaining their states on different realtime Kafka Streams. • Worked on re-orchestrating and migrated the entire system from airflow to AWS managed airflow to make it more manageable and effective. Specialties: • Proficient in handling various business scenarios and converting them into technical code and offering business solutions/recommendations or insights • Strong expertise in SQL concepts, Kafka, Python, Scala, Spark and AWS. • Good ETL development experience to load and unload the data in data warehouse. Experience working with tool named Abinitio. • Expert in creating AWS data ingestion pipelines using pagination. • Implemented the stateful Flink framework and efficiently optimized the redshift queries. • Knowledge of Parsing json format for various APIs and sdks.

Stackforce AI infers this person is a Data Engineering expert with a strong focus on AWS and ETL processes.

Location: San Jose, California, United States

Experience: 6 yrs 6 mos

Skills

Data Engineering
Aws
Data Analysis
Etl Development
Data Visualization

Career Highlights

Expert in building AWS data ingestion pipelines.
Proficient in ETL development and data visualization.
Strong background in data engineering and big data technologies.

Work Experience

Abbott

Senior Data Engineer (10 mos)

Intuit

Data Engineer (6 mos)

Autodesk

Data Engineer (1 yr)

TheDataTeam

Data Engineer (2 yrs 1 mo)

FICO

Solution Integration Consultant (2 yrs 1 mo)

Education

Master of Science - MS at San José State University

Masters of Technology at Jaypee Institute Of Information Technology

Bachelor of Technology at Jaypee Institute Of Information Technology

Palak Agarwal

Data Scientist

San Jose, California, United States6 yrs 6 mos experience

AI EnabledAI ML Practitioner

Key Highlights

Expert in building AWS data ingestion pipelines.
Proficient in ETL development and data visualization.
Strong background in data engineering and big data technologies.

Stackforce AI infers this person is a Data Engineering expert with a strong focus on AWS and ETL processes.

Contact

Skills

Core Skills

Data EngineeringAwsData AnalysisEtl DevelopmentData Visualization

Other Skills

PySparkDynamoDBSQLHiveAWS LambdaSnowflakePythonMySQLGitScalaAirflowSparkHadoopAbinitioTableau

About

Experience

6 yrs 6 mos

Total Experience

1 yr 5 mos

Average Tenure

10 mos

Current Experience

Abbott

Senior Data Engineer

Jul 2025 – Present · 10 mos · California, United States

Intuit

Data Engineer

Jan 2025 – Jul 2025 · 6 mos · California, United States

Autodesk

Data Engineer

May 2023 – May 2024 · 1 yr · San Francisco Bay Area · Hybrid

· Designed batch ETL PySpark scripts to process weekly user revenue data uploads from SharePoint API, leveraging Spark transformations.
· Developed intermediate tables and temporary views using UPSERT transactions for incremental data loading in Hive, optimizing revenue tracking dashboards by reducing processing time by 25% and ensuring idempotency.
· Created testing scripts for Mock APIs and developed common utility scripts to retrieve data from APIs using AWS Secrets Manager.
· Conducted comparative analysis between Snowflake and Hive finance tables to architect data pipelines, reducing manual efforts by 40%.
· Built AWS data ingestion pipelines with pagination to efficiently query DynamoDB and utilized AWS Lambda for handling subsequent API calls.

PySparkAWSDynamoDBSQLHiveAWS Lambda+1

Thedatateam

Data Engineer

Mar 2021 – Apr 2023 · 2 yrs 1 mo

· Designed Spark data pipeline architecture to onboard new attributes for User Profiling and triggered notifications via SNS, Slack, and Airflow.
· Worked on optimizing and performance tuning the SQL queries for achieving joins on 10 tables reducing execution time to 40%
· Implemented a Spark framework to ingest telecom attributes, validate data configurations, transform, and load the processed data into PostgreSQL and S3 in Parquet format. The data was leveraged to send alerts to telecom users about remaining data quotas.
· Re-architected and migrated the entire system from Airflow to AWS services using Python, achieving a 60% reduction in MapReduce processing time. Implemented auto-scaling for EMR clusters and EKS pods to execute Ruby scripts within Docker images, following Agile methodologies.
· Developed Airflow orchestration using Python and DAG operators to manage upstream and downstream data flows in the Spark pipeline for specific dates. Configured retry and resume functions using CRON-based date and time settings.
Skills: AWS, Python, MySQL, Git, Scala, Airflow, Scala, Spark, Curl, Hadoop, SQL, Mysql, Data Analysis.

AWSPythonMySQLGitScalaAirflow+4

Fico

Solution Integration Consultant

Feb 2019 – Mar 2021 · 2 yrs 1 mo

Working on Creating the Abinitio ETL workflows for different clients and further using Tableau for visualization.
Entirely responsible for delivering the brand campaign flows with numerous segmentation techniques.
Automating the data extraction and uploads via effective scripts in graphs for data formatting and transforming.
Working on managing and logically suppressing the database tables using the ETL graphs for filtering the target customers.
Analyzing and creating the visualizations for operational data extracted from campaign executions.
Tools Used: Abinitio, Putty, WinScp, WinSQL, Customer Dialogue Manager CDM, Db Visualizer, MS Excel, Tableau, AWS

AbinitioTableauAWSETL DevelopmentData Visualization