Data Engineering
Location: Bengaluru, Chennai
Job code: 100873
Posted on: Apr 04, 2024
As a key member of the analytics function, work closely with the technology team to strategize the data workflows as required.
Constructing infrastructure for efficient ETL processes from various sources and storage systems
Create a consistent pipeline for analytical consumption of data.
Drive automation of BI reports using job schedulers.
Implement methods to improve data reliability and quality. Reformulating existing frameworks to maximize their functioning.
Design and test architecture for data extraction and transformation from xml or json files for predictive or prescriptive modeling.
Interpreting data trends and patterns to establish operational alerts.
Continuously exploring opportunities to enhance data quality and reliability.
Experience and skills:
Experience in Data Engineering/ML Engineering. Experience in NBFCs, financial services, viz. commercial banking or investment banking is desired.
Knowledge of architectural best practices for building data lakes.
Familiarity with agile methodologies.
High proficiency in Python/Scala/Java and Spark for applied large-scale data processing.
Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.
Solid understanding of batch and streaming data processing techniques.
Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.
Expert-level ability to write complex, optimized SQL queries across extensive data volumes.
Experience with RDBMS and OLAP databases like MySQL, Redshift.