My tasks

As a Data Engineer, my primary responsibility was to design, develop and implement a batch data ingestion pipeline, which processes large volumes of data and delivers near-real-time analytics for a leading banking company. Here are the tasks and responsibilities I undertook during the project:

• Gathered requirements from the client and collaborated with cross-functional teams to understand the business goals and data sources.
• Designed and architected the data pipeline using modern technologies and best practices, such as Apache Kafka, Apache Spark, and Apache Avro, to ensure high throughput, low latency, and fault-tolerance.
• Implemented automated testing, monitoring, and alerting mechanisms to ensure data accuracy, quality, and availability, and maintained the pipeline performance and uptime using tools like Jenkins, Linux..
• Deployed the pipeline in a distributed environment, and integrated it with other systems, such as a data lake in Hadoop, a dashboarding tool in Azure, and a machine learning platform.
• Implemented data governance policies and procedures, including GDPR compliance, and ensured data security and privacy.
• Designed and implemented a workflow orchestration system to manage the pipeline's dependencies with tools similar to AWS Glue and Azure Data Factory, schedule, and execution, and track its progress and performance metrics.
• Documented the pipeline architecture, data flow, and technical specifications, and provided training and support to the client's data team and stakeholders using Agile and SAFe methodologies.

Through my technical expertise and experience in Data Engineering, I delivered a robust and scalable data ingestion pipeline that enabled the client to gain valuable insights into customer behavior, product trends, and marketing campaigns, and improved their business decisions and outcomes.