Aug 2020 – Present · 5 yrs 7 mos · Hyderabad, Telangana, India

• Major contribution on successful migration of two projects: market data in PySpark from on-premise system to cloud (Databricks & AWS) and investment framework python app to AWS cloud.
• 1. For market data project, set up various checks for the batch process that were key to Production sign off and go-live. Following are some of the key components:
• Wrote code to perform sanity checks on data generated that included a statistical model which was the core component to ensure data quality
• Developed file watcher code to monitor and track arrival of critical files in AWS S3 bucket from upstream system
• Set up other critical notifications and PagerDuty to support daily batch runs and maintain SLA apart from regular functional tasks.
• Key contributor for building a standalone scheduler for scheduling hundreds of databricks jobs in cloud using AWS lambda, step function, SNS and SQS deployed using Terraform code (Infrastructure as Code).
• 2. TPIF app migration to cloud using a variety of AWS services - DMS for data migration, lambda, SQS, SNS, S3, aurora RDS, ECS, fargate and ALB, infrastructures are created using Terraform code.
• Key contributor to the success of an enterprise wide project, to create a top down return and exposure attribution for the total fund.
• Set up Jenkins Pipeline to build and deploy database queries to Amazon Aurora using liquidbase between environments in different AWS accounts.
• Working with Total Portfolio Management on Portfolio Investment Framework and Balancing process.
• Exposed to other technologies like Python (Flask, SQLAlchemy, Pandas), Database - Oracle, PostgreSQL, Versioning - Github, Scheduling tools - Tidal and Jams, Project management tools - Jira, ServiceNow, Reporting tool - Tableau.
• Tools: AWS S3, Athena, Glue, lambda, step function, cloud watch, airflow, databricks, EMR, EC2, JIRA, Confluence