Lead Hadoop Data Engineer

This role is for one of the Weekday's clients

Min Experience: 6 years

Location: Hyderabad

JobType: full-time

We are seeking a highly skilled and experienced Lead Hadoop Data Engineer to join our growing data engineering team. The ideal candidate will have strong expertise in Big Data technologies, Hadoop ecosystem components, Python programming, SQL development, and hands-on experience working in on-premise data environments. This role requires a strong technical leader who can design scalable data solutions, optimize large-scale data pipelines, and support enterprise-level analytics and data processing initiatives.

As a Lead Hadoop Data Engineer, you will play a critical role in building and maintaining robust data infrastructure capable of handling high-volume, high-velocity, and high-variety datasets. You will collaborate closely with cross-functional teams including Data Science, Analytics, DevOps, and Business stakeholders to deliver reliable and scalable data solutions aligned with organizational goals.

Key Responsibilities

• Design, develop, and maintain scalable Big Data solutions using Hadoop ecosystem technologies.
• Build and optimize batch and real-time data pipelines for large-scale data ingestion and processing.
• Develop robust ETL workflows using Python, SQL, and Hadoop-based frameworks.
• Work extensively on distributed data processing platforms such as HDFS, Hive, Spark, MapReduce, YARN, and related technologies.
• Manage and support on-premise Big Data infrastructure and ensure high availability, performance, and scalability.
• Analyze complex data requirements and translate them into efficient technical solutions.
• Optimize query performance and data processing jobs for large datasets.
• Collaborate with architecture and infrastructure teams to improve system reliability and performance.
• Lead code reviews, technical discussions, and best practice implementation across the data engineering team.
• Troubleshoot production issues, identify bottlenecks, and implement performance tuning strategies.
• Ensure data quality, governance, security, and compliance standards are maintained across platforms.
• Mentor junior engineers and provide technical guidance on Big Data technologies and development standards.
• Participate in capacity planning, cluster management, and Hadoop administration activities when required.

Required Skills & Qualifications

• 6 to 13 years of experience in Data Engineering or Big Data development.
• Strong hands-on expertise in Hadoop and Big Data ecosystem technologies.
• Proficiency in Python scripting and advanced SQL programming.
• Experience working with Hadoop components such as HDFS, Hive, Spark, Sqoop, Kafka, MapReduce, Pig, or HBase.
• Solid understanding of distributed computing and large-scale data processing concepts.
• Strong experience in building ETL/data pipeline solutions in enterprise environments.
• Hands-on experience managing or working within on-premise infrastructure environments.
• Good understanding of data warehousing concepts and relational databases.
• Experience with Linux/Unix environments and shell scripting.
• Strong analytical, troubleshooting, and performance optimization skills.
• Familiarity with workflow orchestration tools such as Airflow or Oozie is preferred.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications

• Experience leading technical teams or managing enterprise-scale Big Data projects.
• Exposure to cloud migration projects or hybrid data environments is an added advantage.
• Knowledge of DevOps practices, CI/CD pipelines, and containerization tools is desirable.

Experience

6 – 13 Years

Must-Have Skills

• Big Data
• Hadoop
• Python
• SQL
• On-Premise Data Environment Experience