Shubham

WORK EXPERIENCE Koantek (Former Apisero company), Pune Sept 2022-Present Data Engineer

● Developed and tested the internal framework Ascend for Automated, fast development and monitoring of the data pipelines which reduced deployment time by 90%.

● Designed and developed the Unity Catalog Upgrade Tool to sync the hive-metastore table to the Unity catalog which provided a unified data governance solution and helped manage fine-grained access controls in Databricks lakehouse on any cloud.

● Built, configured, and managed the data infrastructure using Databricks and Cloud Services to provide meaningful insights for the business. Tata Consultancy Services, Pune April 2021-Sept 2022 Data Engineer

● Designed, developed high throughput and reliable data and reporting from the ground up using SQL, Python, and Power BI to provide real-time insights to the logistic company handling 400GB of data every 15 minutes.

● Investigated, Optimized, architected the legacy Data Warehouse which reduced the query runtime from 390 Minutes to 17 Minutes and helped to save cost by 95% associated.

● Implemented ETL, orchestrated the data pipelines using AWS Lambda, and visualized the data using the Power BI dashboard to produce real-time insights into the Business which monitored and increased the efficiency by 70%.

● Analyzed the client requirements and led a team of 4 team members which provided data as a service.