Shivam Pandey — Data Engineer
Passionate about turning raw data into actionable insights and always ready to solve complex data challenges that create real business impact. Data Engineer with hands-on experience designing and optimizing data pipelines and architectures that turn complex datasets into meaningful insights. I work across the full data lifecycle, from ingestion and cleaning to modeling, warehousing, and delivery, ensuring that data is well-structured, accurate, and ready for analysis. With a strong foundation in data processing, transformation, and orchestration, I specialize in building scalable ETL workflows and solving data quality issues such as inconsistencies, missing values, and schema mismatches. My work helps ensure that analytics teams receive clean, analysis-ready data at the right time and scale. Key Skills & Expertise: • Data Engineering: Experienced in end-to-end pipeline development, from business requirement analysis to implementation, with a focus on efficiency, reliability, and reusability. • Apache Spark: Skilled in building and optimizing Spark jobs for large-scale distributed data processing, reducing cost and improving runtime performance. • PySpark & Spark SQL: Strong command over large-scale data manipulation and transformation using PySpark and Spark SQL. • Cloud Technologies: Azure: Databricks, Data Factory, ADLS Gen2, Synapse Analytics AWS: S3, RDS, Redshift, EMR, EC2, Lambda GCP: BigQuery, Cloud Storage (GCS), Dataflow (Apache Beam), Pub/Sub, Composer (Airflow) • Automation & Orchestration: Proficient in designing automated workflows and managing data pipelines using tools like Azkaban, Tidal, Argo, and ADF. Interpersonal Skills: Effective in time management, team collaboration, mentoring, communication, and handling end-to-end project ownership across cross-functional teams. Career Highlights: Industry experience in the Supply Chain domain through my work at Körber Supply Chain. Skilled at resolving complex data quality issues and ensuring data consistency across systems Built and maintained high-performing pipelines for structured and semi-structured data at scale Contributed to team-wide initiatives focused on data governance, cost optimization, and reliability Outside of work, I actively build hobby-level data engineering projects to experiment with new technologies and stay aligned with evolving industry trends. I’m always exploring better ways to handle data using tools like Python, SQL, Apache Spark, and modern cloud platforms.
Stackforce AI infers this person is a Data Engineer specializing in scalable ETL workflows and cloud-based data solutions.
Experience: 4 yrs 4 mos
Skills
- Data Engineering
- Pyspark
Career Highlights
- Expert in building scalable ETL workflows.
- Proficient in resolving complex data quality issues.
- Strong foundation in cloud technologies and data processing.
Work Experience
EXL
Assistant Manager - Data Engineering (3 mos)
Fractal
Data Engineer (6 mos)
BNP Paribas
Software Engineer (1 yr)
NamasteDev.com
Namaste React Bootcamp (3 mos)
Körber Supply Chain
Software Engineer (2 yrs 7 mos)
CodeQuotient
Supercoder (5 mos)
United Group of Institutions
Frontend Developer (7 mos)
Education
Bachelor of Technology - BTech at Dr. A.P.J. Abdul Kalam Technical University
Big Data Masters at Trendytech