Data Flow Engineer

Company Description

Arηs Group, Part of Accenture, specializes in the management of complex public sector IT projects, including systems integration, informatics and analytics, solution implementation and program management. Our team helps lead clients through digital and information systems design, bringing expertise in a variety of areas ranging from software development, data science and security management to machine learning, cloud, and mobile development.
Arηs Group was acquired by Accenture in July 2024.

Job Description

• Design, develop, and maintain complex data flows within Cloudera DataFlow (Apache NiFi), ensuring scalable, reliable, and high-performance data movement across systems.
• Develop and optimize real-time and near real-time data pipelines leveraging NiFi, Kafka, and CDC technologies (e.g., Debezium, SQL-based connectors).
• Implement integrations with internal and external systems using REST APIs, JDBC, Kafka, and other communication protocols, ensuring secure and resilient data exchange.
• Design and manage data schemas (Avro), metadata, and lineage using Apache Atlas, ensuring full traceability and governance of data flows.
• Define and enforce data security and access control policies using Apache Ranger in alignment with enterprise governance frameworks.
• Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and scalability, including proactive alerting and issue resolution.
• Collaborate with data engineers, architects, and business stakeholders to define requirements, design architectures, and deliver robust data flow solutions.
• Create and maintain technical documentation, SOPs, and runbooks for operational support and knowledge sharing.
• Support platform lifecycle activities, including upgrades, migrations, and enhancements across CDP, NiFi, and Kafka environments.
• Perform other related duties as assigned by the team leader.

Qualifications

• Advanced university degree (Master’s or equivalent) in computer science, information systems, data engineering, or a related field; a first-level degree combined with additional experience may be accepted in lieu of the advanced degree.
• At least one of the following certifications:

1. Cloudera Certified Developer for Apache NiFi (or equivalent)
2. Cloudera DataFlow (CFM) certification (or equivalent)
   Equivalent certifications must be internationally recognized and accepted as valid credentials.

• Minimum 2–3 years of hands-on experience working with Apache NiFi, preferably within the Cloudera Data Platform (CDP) environment, including flow design, deployment, monitoring, and troubleshooting.
• Proven experience delivering at least one large-scale integration project using NiFi as a core technology (API integrations, database connectivity, transformation, routing, and delivery).
• Expert knowledge in designing, implementing, and maintaining complex data flows using Apache NiFi / Cloudera DataFlow.
• Advanced Python programming skills for data processing, automation, and custom flow development.
• Strong experience in building and integrating REST APIs, including authentication (OAuth/JWT), rate limiting, and error handling strategies.
• Hands-on experience with CDC (Change Data Capture) approaches, using NiFi processors/connectors and SQL-based methods.
• Practical experience with Apache Iceberg, including table design, schema evolution, partitioning, and integration with processing engines (e.g., Spark, Flink).
• Solid knowledge of data governance and catalog tools within CDP, including Apache Atlas (metadata, lineage, tagging) and Apache Ranger (security policies, authorization).
• Experience working with Apache Kafka as a messaging platform, including topics, producers/consumers, schema management, and NiFi integration.
• Good understanding of data serialization using Apache Avro, including schema evolution and compatibility principles.
• Strong analytical and problem-solving skills, with the ability to diagnose and resolve complex data pipeline issues.
• Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.
• Fluency in written and spoken English.

Additional Information