Soumya Banerjee

Director of Engineering

Bengaluru, Karnataka, India19 yrs 10 mos experience
Highly StableAI Enabled

Key Highlights

  • Over 20 years of experience in engineering leadership.
  • Expertise in building innovative data products.
  • Proven track record in scaling high-performance organizations.
Stackforce AI infers this person is a SaaS expert with a strong focus on data engineering and cloud-native solutions.

Contact

Skills

Core Skills

Engineering ManagementDistributed SystemsStream ProcessingData EngineeringApache SparkCloud ComputingApache Spark Streaming

Other Skills

Large Language Models (LLM)Big DataMicroservicesKubernetesSoftware as a Service (SaaS)Research and Development (R&D)Cross-functional Team LeadershipApache KafkaSpring BootDirector levelGenerative AISoftware DevelopmentRequirements AnalysisSOAAlgorithms

About

With over 20 years of experience, I am an engineering leader dedicated to building innovative data products and scaling high-performance organizations. My career spans the entire lifecycle of data management—from seeding engineering teams from scratch to leading global departments of 100+ members at Fortune 500 companies. Core Expertise: Technical Leadership: Specialized in event processing, real-time query engines, ETL pipelines, and fully managed cloud-native services. Organizational Growth: Proven track record of bootstrapping initiatives and scaling engineering charters across diverse domains.

Experience

19 yrs 10 mos
Total Experience
3 yrs 2 mos
Average Tenure
1 yr 3 mos
Current Experience

Linkedin

Director of Engineering

Jan 2025Present · 1 yr 3 mos · Bengaluru · On-site

  • Building a unified Data Platform which caters to all data movement within LinkedIn at its scale. Managing multiple products and teams catering to various aspects of data ingestion, storage and operationalisation.
Engineering ManagementDistributed SystemsLarge Language Models (LLM)Big DataStream Processing

Primary venture partners

Expert

Jul 2024Present · 1 yr 9 mos · Bengaluru, Karnataka, India · Remote

  • As an Expert, I provide strategic advice and guidance to the investment team and founders within the Primary portfolio and broader network.
  • Primary is New York City’s premier early-stage venture firm working alongside founders to build unicorns like Alloy, Alma, Chief, Dandy Electric, Latch, K Health, Stellar Health, Slice, and many more.

Confluent

Head of Engineering for Confluent Platform, KSQLDB, Audit Logs and Audit Trail Insights.

Aug 2021Nov 2024 · 3 yrs 3 mos · Bangalore Urban, Karnataka, India · Hybrid

  • One of the first leadership hires in Confluent India site.
  • Bootstrapped, led, and transitioned multiple product groups.
  • 1. Confluent Audit Logs
  • 2. KSQL
  • 3. Confluent Platform
  • 4. Audit Trail Insights
Engineering ManagementStream ProcessingMicroservicesKubernetesSoftware as a Service (SaaS)Cloud Computing+3

Uber

Engineering Leader - Competitive Intelligence and Android App Bundles

Jan 2020Jan 2021 · 1 yr · Bengaluru, Karnataka, India · On-site

  • Lead multiple initiatives/teams
  • 1. Data engineering for generating Competitive Intelligence metrics
  • 2. Android Play feature delivery and dynamic modules to optimise Uber app size across verticals
  • 3. Uber Beta app for employee testing.
  • 4. Uber app rating improvement in iOS.
  • 5. Discovery, to enable users to discover places of interest within the Uber app.
  • Leading a geographically distributed team of data engineers, senior architects and tech leads to generate insights around Uber's competitiveness with respect to its competitors worldwide across Rides, Eats, Grocery and Alcohol Delivery. Metrics are business critical to drive Uber's strategic decisions.
  • Tech stack : Java, Python, Scala, MachineLearning, Microservices, Apache Spark, Airflow, Hive, Hadoop, Presto, Kafka
  • Leading the foundational work on optimising the Uber Android app size across all the verticals by leveraging the Google Dynamic Play Feature and Delivery. Supported the team to contribute Play feature delivery into Facebook's open source Buck build system.
  • Successfully completed the 'Discovery' proof of concept a new greenfield initiative which lets Uber users discover places of interest such as restaurants, malls, museums and subsequently take a ride increasing the GB.
  • Led the Uber Android Beta app and beta test initiative which is used by Uber employees for new feature dog-fooding before being rolled out to general public.
  • Led the initiative to improve Uber app ratings by prompting the user to rate the app on certain favourable conditions.
Engineering ManagementData EngineeringResearch and Development (R&D)Cross-functional Team Leadership

Informatica

4 roles

Director Of Engineering

Promoted

Apr 2018Nov 2020 · 2 yrs 7 mos

  • https://www.informatica.com/blogs/informatica-ranked-2-in-the-gartner-market-share-analysis.html
  • Headed all data streaming products and the connector SDK framework of Informatica. Grew a team from 5 to 75 comprising of Software Developers, QA, DevOps, Operations across US and India. Managed and mentored 5 senior managers and 4 senior architects. Built a portfolio of products generating multi million dollar revenue.
  • Built the following products from 0-1
  • 1. Cloud Mass Ingestion : A multi-tenant managed service on Informatica Cloud to ingest real time data (Database CDC, IOT, ClickStream, Social Media feeds) at scale from diverse sources to Cloud data lakes, data warehouses and messaging hubs. Managed a full stack team consisting of UI engineers, backend engineers, QE and DevOps folks.
  • 2. Data Engineering Streaming : A real time data integration product with a visual designer to author data pipelines leveraging Apache Spark, Kafka and big data to provide a data integration platform to process streaming data at massive scale.
  • 3. RulePoint : A highly available, distributed complex event processing system with a custom domain specific language similar to SQL to express queries on real time data as its flowing through to generate business insights. Was the Principal Engineer for the core Cluster Manager, Scheduler and Metrics Manager component.
  • 4. Proactive Monitoring for Power Center : A solution to monitor 'Power Center' in real time to predict anomalies and alert the administrator. Built on RulePoint and deployed across Power Center nodes.
  • Inherited the following and extended it
  • 5. Connector SDK : Connector SDK is the development kit to rapidly build connectors to diverse systems to ingress/egress data and metadata at scale in a consistent fashion across all Informatica products.
  • 6. Informatica Edge Data Streaming : A product to continuously ingest data from IOT devices with edge data processing capabilities remote deployment and configuration management.
Apache SparkApache Spark StreamingMicroservicesKubernetesSoftware as a Service (SaaS)Cloud Computing+4

Senior Manager

Apr 2015Mar 2018 · 2 yrs 11 mos

  • Architecture and Development of Informatica's 'Data Engineering Streaming' product
  • Lead a team of developers and qa engineers for the development of the product using Spark Streaming, Kafka and Big Data Technologies.
  • Showcased the first version in Strata Hadoop NY Conference in 2016.
  • Increased customer adoption from 0 to 150+ customers
  • Worked closely with the customer support, professional services and sales team to help customers onboard and become successful.
  • Drove the strategic direction of the product working along with the product management organisation.
  • Managed the 'Edge Data Streaming' product. Invested significantly in stabilisation of the product and developed many new features.
Apache Spark StreamingResearch and Development (R&D)

Principal Software Engineer

Oct 2012Mar 2015 · 2 yrs 5 mos

  • Core design and development of ‘Grid Manager’ and ‘Activity Manager’ component in ‘RulePoint’ which is a highly distributed and scalable complex event processing system.
  • Implemented cluster management module to bring up, destroy and maintain the cluster with high availability, resiliency and scalability of the processing units.
  • Implemented the scheduler module which schedules the task subcomponents onto the worker nodes in a multi-threaded incremental fashion across the worker nodes.
  • Implemented 2PC with a WAL to achieve atomicity for the task deployment.
  • Implemented a highly scalable datastore backed by a relational database to store and index all events in a timeseries fashion.
  • Mentored junior members to ramp up and guide them on the core product architecture.
  • Worked with the QA team to design test plan and methodologies.
Apache Spark StreamingResearch and Development (R&D)

Lead Engineer

Jun 2011Sep 2012 · 1 yr 3 mos

  • Worked on core product development of highly distributed Complex Event Processing system RulePoint. The product provides real time analytics based on event correlation across diverse systems.
  • Worked on ‘Proactive Monitoring for PowerCenter (PMPC)’ which is a solution built on RulePoint which proactively monitors PowerCenter at real time by monitoring PowerCenter repository and processes to raise alerts based on rules.
  • Involved in the design and development of the SERVER module which manages the Lifecycle, HA, Failover and Task assignment of the worker processes across the network.
  • Technologies Involved : Core JAVA, Spring, Hibernate, REST, Hadoop.
Research and Development (R&D)

Jp morgan

Associate

Mar 2010Jun 2011 · 1 yr 3 mos · Bengaluru, Karnataka, India

  • JP Morgan Chase (JP), the second largest financial services company in the US, is exposed to credit risk through its lending, trading and capital market activities. JP''s credit risk management practices are designed to preserve the independence and integrity of the risk assessment process.
  • Worked in the Credit Risk organisation of the Investment Banking unit, building software for analysing credit risk. Tech stack: Java, J2EE, Spring, Databases.

Nokia networks

Senior R&D Engineer

Feb 2008Mar 2010 · 2 yrs 1 mo · Bengaluru, Karnataka, India

  • Worked on SOA architecture using JBI and Apache Servicemix to build a framework where new mediations can be developed plug and play.
  • Worked along with colleagues from China in Chengdu to help them develop 3GPP Corba mediation on the mediation framework.

Wipro technologies

Project Engineer

Aug 2005Feb 2008 · 2 yrs 6 mos · Bengaluru, Karnataka, India,

  • • Worked on building Operational Risk Assessment software for a leading investment bank using Java, EJB and JSP/Servlet technologies.

Education

Visvesvaraya Technological University

BE — Computer Science

Jan 2001Jan 2005

Stackforce found 100+ more professionals with Engineering Management & Distributed Systems

Explore similar profiles based on matching skills and experience