Dibyendu Karmakar — Senior Software Engineer

Working on Amazon's JWO tech. Prior to this I have worked on Swiggy's Data Intelligent Platform Team. This team is responsible for building platforms for Big data & Analytics use cases, Data science use cases(Serving layer for low latency models and feature engineering) My major responsibilities were Design and Architect, Development, and Maintaining platforms. Contributed to following projects - 1. Building Data Lake with Change Data Capture. https://bytes.swiggy.com/introduction-to-cdc-system-an-efficient-way-to-replicate-transactional-data-into-data-lake-c10f99c7a3fd https://bytes.swiggy.com/architecture-of-cdc-system-a975a081691f 2. Designing end-to-end fully automated and self-serve Data Lake with CDC pipeline, which creates a near real-time replica of transactional database(MySql or DynamoDB) in S3. The data is queryable using different query engines like Hive, Presto, Snowflake. This system empowers analytics to generate reports, dashboards, etc. based on near real-time transactional data. Used Apache Hudi, Spark, Kafka, Snowflake, Databricks Delta, Databricks runtime, EMR, S3, etc. to build the system. 3. ETL pipeline design using Spark. 4. Building machine learning platform to serve high throughput and low latency model prediction requests. 5. Building feature engineering platform to extract feature data from various sources and enriching feature stores. 6. I have also led multiple initiatives to optimize costs, performances, etc. Prior to this, I have worked on Huawei's Hadoop kernel team. Worked on HDFS kernel development, HDFS router-based federation. Open source contributions- https://issues.apache.org/jira/browse/HDFS-13289 https://issues.apache.org/jira/browse/HDFS-13386 https://issues.apache.org/jira/browse/HDFS-13453 https://issues.apache.org/jira/browse/HDFS-13506 https://issues.apache.org/jira/browse/HDFS-13528 https://issues.apache.org/jira/browse/HDFS-13583 https://issues.apache.org/jira/browse/HDFS-13776

Stackforce AI infers this person is a Data Engineering expert with a focus on Big Data and MLOps.

Location: Bengaluru, Karnataka, India

Experience: 10 yrs 11 mos