Shubham Tomar

Data Engineer

Bengaluru, Karnataka, India6 yrs 4 mos experience

Highly Stable

Key Highlights

Stackforce AI infers this person is a Data Engineering expert with a strong focus on scalable data solutions in Fintech.

Data EngineeringData ArchitectureData AnalyticsCloud Engineering

Go (Programming Language)icebergApache KafkaApache FlinkKubernetesPythonMongoDBPostgresETLClickHouseKafkaCassandraBigQueryMySQLRedis Streams

Software engineer with expertise in Data Engineering field

6 yrs 4 mos

Total Experience

2 yrs 6 mos

Average Tenure

1 yr 4 mos

Current Experience

Jan 2025 – Present · 1 yr 4 mos · Bengaluru, Karnataka, India · On-site

Leading development of lakehouse architecture using Iceberg, nessie, warpstream(Kafka alternative), spark, prefect, Doris/veloDB, and other modern data engineering tools and frameworks, ingesting +500 Million events per day.
Helped Pixis build Interact, an AI powered conversational analytics interface for marketing teams to query ad performance and receive actionable insights. Leveraged MongoDB to store dynamic queries, responses, and campaign metadata for personalized recommendations and traceability.
Developed Nessie go-client library for seamless interaction with Nessie catalog.
Developed icebridge microservice which acts as a bridge between Nessie catalog and other applications with Iceberg.
Contributed to open source apache/iceberg-go library by adding fix for createTable API.
Migrated 4TB of Postgres data from non-partitioned to partitioned schema using custom built pgx-copy pipeline for faster data transfer. Improved query performance by 4x.

Go (Programming Language)icebergApache KafkaApache FlinkKubernetesPython+2

Dec 2020 – Dec 2024 · 4 yrs · Bengaluru, Karnataka, India · On-site

Developed in house data platform to handle batch as well as real-time data processing using Kafka, cassandra, Clickhouse, and other modern data engineering tools and frameworks.
Spearheaded the development of a robust ETL Data Framework for handling over 200 million transactions daily across diverse data sources (MySQL, Postgres, Kafka, Clickhouse, BigQuery), ensuring optimal monitoring, alerting, and data availability.
Engineered a high-throughput Real-Time Streaming pipeline, achieving ~350k logs/sec, integrating technologies such as Redis Streams, Cassandra, Kafka, and Clickhouse.
Achieved a significant reduction in data pipeline costs by approximately 80% and enhanced analytical query performance by 10x, leveraging Clickhouse optimizations.
Pioneered the Autopilot platform for seamless cloud deployments (GCP/AWS), incorporating features like config control, staggered release, and autoscaling, enhancing deployment efficiency and cloud resource management.

Go (Programming Language)ClickHouseKafkaCassandraBigQueryETL+2