Manish Mishra — Associate Consultant

Lead Data Engineer | Architecting Real-Time & Scalable Data Platforms on GCP & AWS Results-driven Lead Data Engineer with 9+ years of experience in designing, building, and managing high-scale data infrastructure. Proven expertise in leading the development of cloud-native data solutions that drive business intelligence, machine learning, and operational excellence. Core Technical Leadership & Expertise: · Real-Time Streaming Architectures: Designed and deployed low-latency streaming pipelines using Apache Beam, Google Cloud Dataflow, and Apache Kafka for event-driven systems like Order Management (OMS), implementing advanced windowing and stateful processing. · Cloud-Native Platform Engineering: Spearheaded full platform ownership, including workload migration to Google Kubernetes Engine (GKE) for improved scalability and cost efficiency. Built secure foundations using GCP Secrets Manager and cross-cloud integrations (AWS S3, Kinesis, Lambda). · End-to-End Data Pipeline Development: Expert in building batch and streaming pipelines from diverse sources (RDBMS, Cloud Storage) to Google BigQuery, utilizing Apache Airflow for orchestration and dbt for modern transformation layers. · Data Governance & Observability: Established robust data governance with PII management, DLP, and Data Catalog. Built an observability platform to monitor pipeline health, ensure data reliability, and meet strict SLAs. · Mentorship & Best Practices: Actively mentor engineers on Kubernetes, cloud-native principles, and DataOps practices, fostering a culture of quality and innovation within the team. Strategic Impact: · Enabled real-time business decisions by reducing data latency in critical OMS applications. · Delivered substantial cost savings and operational resilience through platform modernization and centralized secret management. · Empowered business and marketing teams by developing internal data products and managing high-stakes partner integrations (Swiggy, Amazon, Ola). Passionate about transforming complex data challenges into scalable, secure, and value-driven engineering solutions. Eager to lead teams in building the next generation of data platforms. Technologies: GCP (Dataflow, BigQuery, GKE, Pub/Sub), Apache Beam, Kafka, Airflow, Kubernetes, dbt, Python, SQL, AWS.

Stackforce AI infers this person is a Data Engineering expert in Fintech and SaaS industries.

Location: Pune, Maharashtra, India

Experience: 14 yrs 9 mos

Skills

Google Cloud Platform (gcp)
Real-time Streaming Architectures
Cloud-native Platform Engineering
End-to-end Data Pipeline Development
Real-time Streaming Solutions

Career Highlights

Expert in real-time data pipeline architecture.
Proven track record in cloud-native platform engineering.
Strong mentorship and leadership in data engineering.

Work Experience

HSBC

Senior consultant specialist (1 yr 3 mos)

Acko

Data Engineer-3 (3 yrs 10 mos)

Data Engineer-2 (1 yr 5 mos)

NDTV

Sr Software Engineer (2 yrs 5 mos)

CARS24

Senior Software Engineer (10 mos)

CarDekho

Sr Software Engineer (1 yr)

Tavant Technologies

Sr Software Eng (1 yr 4 mos)

Brain Technosys Pvt Ltd

Software Engineer (2 yrs 4 mos)

Incite Software Pvt. Ltd.

SEO EXECUTIVE (4 mos)

Education

Bachelor's degree at Motivational Pathway

Intermediate at Kendriya Vidyalaya

High School at Kendriya Vidyalaya

Manish Mishra

Associate Consultant

Pune, Maharashtra, India14 yrs 9 mos experience

Highly Stable

Key Highlights

Expert in real-time data pipeline architecture.
Proven track record in cloud-native platform engineering.
Strong mentorship and leadership in data engineering.

Stackforce AI infers this person is a Data Engineering expert in Fintech and SaaS industries.

Contact

Skills

Core Skills

Google Cloud Platform (gcp)Real-time Streaming ArchitecturesCloud-native Platform EngineeringEnd-to-end Data Pipeline DevelopmentReal-time Streaming Solutions

Other Skills

Apache BeamGoogle Cloud DataflowApache KafkaGoogle BigQueryGoogle Kubernetes Engine (GKE)TerraformPython (Programming Language)Google Cloud Secrets ManagerApache AirflowCloud DataflowAWS S3Google Cloud StoragePythonPostgreSQLMySQL

About

Experience

14 yrs 9 mos

Total Experience

1 yr 11 mos

Average Tenure

1 yr 3 mos

Current Experience

Hsbc

Senior consultant specialist

Mar 2025 – Present · 1 yr 3 mos · Pune, Maharashtra, India · On-site

🔹 Designing and developing real-time streaming data pipelines using Google Cloud Dataflow, Apache Beam, and Apache Kafka, enabling low-latency data ingestion and processing for Order Management Systems (OMS).
🔹 Exploring and implementing advanced Beam constructs such as windowing strategies, side inputs, and stateful processing to optimize performance and resource efficiency in streaming applications.
🔹 Led research initiatives on Apache Beam and Dataflow to enhance low-latency, event-driven architectures — sharing findings with the broader team to accelerate project delivery and innovation.
🔹 Spearheaded the successful onboarding of workloads to Google Kubernetes Engine (GKE) — from cluster provisioning to application migration — achieving improved scalability and substantial cost savings.
🔹 Enabled secure operations by building a centralized credential management system using Google Cloud Secrets Manager, ensuring strong protection of sensitive data and seamless integration across services.
🔹 Actively mentored peers on Kubernetes best practices, promoting cloud-native thinking and fostering a culture of operational excellence within the team.

Google Cloud Platform (GCP)Apache BeamGoogle Cloud DataflowApache KafkaGoogle BigQueryGoogle Kubernetes Engine (GKE)+3

Acko

2 roles

Data Engineer-3

Promoted

Apr 2021 – Feb 2025 · 3 yrs 10 mos

🔹 End-to-End Data Pipeline Ownership:
Built and maintained scalable data pipelines across multiple sources including RDBMS (PostgreSQL/MySQL), file systems like AWS S3 and Google Cloud Storage (GCS), enabling efficient data ingestion and transformation.
🔹 Batch Processing at Scale:
Developed robust batch pipelines using Apache Airflow, Apache Beam, Cloud Dataflow, and other GCP services such as Cloud Functions, Data Catalog, DLP, and Secret Manager.
🔹 Real-Time Streaming Solutions:
Engineered streaming pipelines to move data from S3, DynamoDB, AWS Kinesis, etc., into Google BigQuery, leveraging Pub/Sub, Beam, Dataflow, and cross-cloud orchestration with Cloud Functions and AWS Lambda.
🔹 Data Privacy & Governance:
Managed PII data modeling across the data lake and data marts, ensuring compliance with internal security standards and external regulatory requirements.
🔹 Advanced Data Mart Segmentation:
Implemented segmentation pipelines for targeted marketing and analytics using Airflow, Python, and BigQuery.
🔹 Internal Data Products:
Led backend development for data.acko.com, a centralized reporting platform for Marketing and Business teams, built with Node.js, Angular, Python, and PostgreSQL.
🔹 Partner Integrations at Scale:
Managed and integrated data flows for leading partners including Swiggy, Amazon, Rapido, Ola, and others, ensuring high data quality and reliability.
🔹 Modern Data Stack Enablement:
Built a mature data transformation layer using DBT, Airflow, and BigQuery, enabling reusable, testable, and version-controlled transformations.
🔹 Feature Engineering for MLOps:
Developed a reusable and scalable feature engineering layer to support machine learning pipelines in production.
🔹 Observability for Data Engineering:
Designed and implemented an observability platform to monitor and track the health of the entire data engineering ecosystem — ensuring data reliability, latency tracking, and SLA adherence.