SHIVAM SINGH

Software Engineer

Bengaluru, Karnataka, India6 yrs 8 mos experience
Highly Stable

Key Highlights

  • Expert in designing scalable data pipelines.
  • Proven track record in real-time data processing.
  • Strong background in mentorship and project planning.
Stackforce AI infers this person is a SaaS-focused software engineer with expertise in data processing and real-time analytics.

Contact

Skills

Core Skills

Apache KafkaSpark Structured StreamingMap MatchingSpring BootPysparkRest ApisFlask

Other Skills

AWS EBAlfrescoAmazon EC2Amazon RedshiftAmazon Relational Database Service (RDS)Amazon S3Amazon SQSApache SparkBarefoot libraryBigQueryC (Programming Language)C++Data pipelineDatabasesDocumentDB

About

An experienced software engineer seeking to learn new technologies to enhance skills. My specialities include quickly learning new skills and programming languages, problem solving, low level design, high level design, mentorship, creating project plans and effort estimates. So far I have Data structures and algorithms, OOPs, Python, java, c, c++, Pyspark, Spark structured streaming, Kafka, Spring Boot, Flask, AWS, K8s, Jenkins, OSM, Shapely, Geopandas, Barefoot, rTree, SAML2.0, Elasticsearch, mongodb, RDBMS, Operating System, Kibana, Hostedgraphite, JIRA, Confluence, GIT, GERRIT, GITHUB, Pagerduty, PyCharm, Alfresco, Slack under my belt. I like to contribute to Open Source Projects also.

Experience

6 yrs 8 mos
Total Experience
4 yrs 10 mos
Average Tenure
1 yr 10 mos
Current Experience

Credit karma

2 roles

Software engineer IV

Promoted

Apr 2025Present · 1 yr 2 mos · Bengaluru, Karnataka, India · Hybrid

Software engineer III

Jul 2024Mar 2025 · 8 mos · Bengaluru, Karnataka, India · Hybrid

Zendrive

3 roles

Software Engineer III

Promoted

Jan 2022Jun 2024 · 2 yrs 5 mos

  • As a software developer in Lead Generation Platform team
  • IQL(Insurance Qualified Leads)
  • 1. Designed and developed a scalable and event driven platform using Spark Structured Streaming and Apache Kafka which evaluates the driver for offer at the end of each trip. The same design was adopted by multiple services to support the project IQL.
  • 2. Designed and developed the Kafka Consumer Java Application using confluent’s Parallel Consumer library improving the processing by 20x integrating it with Mongo Database. Reduced the DB Reads by 10x per user per day while aggregating the trip data with cumulative aggregations. Reconfigured the application Concurrency and Batch Size increasing the QPS and making optimal usage of resources.
  • 3. Designed a reporting framework using Spark Structured Streaming and Apache Kafka to make the data available in near real time into centralized databases like BigQuery and Redshift for analytics and reporting purposes.
  • 4. Helped the product team to improve the business funnel identifying the missing features in one of crucial metrics used in business logic.
  • Barefoot Map Matching
  • 1. Deployed event driven async Map Matching architecture in EU region. This takes GPS points as input and returns the best possible route which might be taken by the driver.
  • iSaaS (Insurance SaaS)
  • 1. Proposed the idea of improving the Insurance Data Pipeline and REST APIs making them capable of serving the driver score on the basis of real time driving behavior. Got the approval from the product team and designed it using Apache Kafka for data consumption, DocumentDB(MongoDB) to store the data and Spring Boot for insurance score REST APIs.
Spark Structured StreamingApache KafkaMongoDBJavaBigQueryRedshift+1

Software Engineer II

Promoted

Jul 2020Dec 2021 · 1 yr 5 mos

  • As a software developer in Driving Behavior and Scoring team
  • Route Risk Analysis
  • 1. Developed the PySpark data pipeline for spatial data processing using the Barefoot library to get the best possible route based on GPS input taken by a driver.
  • 2. Scaled PySpark data pipeline to process more than 3M records (1K gps points in each) within 2 hours where map matching was being done on gps points.
  • 3. Collaborated with a team from CUEBiC for business opportunities based on Route Risk Analysis.
  • IQL(Insurance Qualified Leads)
  • 1. Designed the data pipeline and REST apis for MVP and created the road map for the project involving 2 team members. Planned the EU region deployment as well.
  • Fairmatic (AI Powered Insurance for fleets)
  • 1. Project was transitioned from Zendrive as a new company. I owned the whole transition activity of the backend tech stack and helped the people in Fairmatic understand the Project from scratch. At present, continuing on any requirement from Fairmatic.
  • REST APIs
  • 1. Designed and developed Flask Application hosting trip type feedback apis and driver insurance score apis on AWS EB.

Software Development Engineer I

Jul 2019Jul 2020 · 1 yr

  • As a software developer in the customer products team
  • 1. Scaled the PySpark data pipelines moving data dependencies from internal REST APIs to AWS S3.
  • As part of SDK Validation Tool, developed REST apis and PySpark data pipelines.
  • 2. Researched on Alfresco ECM and BPM to create a customer onboarding automation tool called Policy Admin Tool.

Education

Motilal Nehru National Institute Of Technology

Bachelor of Technology - BTech — Computer Science

May 2015May 2019

Stackforce found 100+ more professionals with Apache Kafka & Spark Structured Streaming

Explore similar profiles based on matching skills and experience