Data Platform Manager- Hybrid (Applicants must reside

accoy· Information Technology
Apply Now ↗
📍 Pittsburgh, PA, USFULL TIME

About this role

Overview

Location: Hybrid, Pittsburgh, PAJob Type: Full Time / PermanentWork Authorization: No C2C or Sponsorship

 

The A.C.Coy company has an immediate opening for a Data Platform Manager. This role will be responsible for designing, building, and optimizing enterprise wide data platforms within the Data Warehouse.

Responsibilities

  • Lead and mentor a team of data engineers, conducting code reviews and ensuring development standards
  • Support troubleshooting and incident management for data-related issues in production
  • Collaborate with business stakeholders, data scientists, and other team members to gather requirements and translate them into technical specifications
  • Lead the design, development and deployment of scalable and high-performance data pipelines using Azure Databricks; ensuring the data integrity, availability, efficient extraction, transformation, and loading of data from various sources into the Azure Databricks Data Warehouse
  • Collaborate with data scientists, analysts, and other engineering teams to deliver business-critical insights. Optimize pipeline performance, cost, and scalability in the Azure cloud environment
  • Define best practices for data ingestion, processing, storage, and governance. Implement data quality checks and validation procedures to ensure the accuracy and integrity of data between various sources, including API’s, databases and streaming platforms
  • Collaborate with data scientists and analysts to operationalize and deploy machine learning models
  • Architecture Design:
    • Define the end-to-end Lakehouse architecture using Delta Lake, implementing medallion architecture (Bronze, Silver, Gold layers) for robust data processing
    • Familiarity with data modeling and schema design principles
  • Pipeline Engineering:
    • Oversee the development of robust, scalable batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL and with minimal latency
    • Implement data transformations, enrichment, and quality checks using PySpark/Scala within the Databricks environment
    • Integrate real-time and batch data sources using Apache Kafka and ADF
    • Support large-scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory (ADF)
  • Data Governance & Security:
    • Implement Unity Catalog for unified governance, data security, fine-grained access control (RBAC), privacy measures, and data lineage tracking
  • Performance Optimization & Tuning:
    • Tune Spark jobs and Databricks clusters to maximize throughput while maintaining cost efficiency through auto-scaling and cluster policies
    • Expertise in indexing strategies, query optimization, execution plans, and partitioning/sharding
  • Platform Integration:
    • Orchestrate workflows by integrating Databricks with other Azure services like Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Azure DevOps for CI/CD pipelines

Qualifications

Required Education

  • Bachelor's degree in Computer Science, Engineering, or a related field

Required Experience

  • 5-7+ years hands-on data engineering or architecture, with at least 2-4 years specifically focused on Azure Databricks, including Azure cloud technologies
  • 2-5 years experience in managing a team of data engineers, data scientists and/or analysts
  • Certifications (Preferred): Microsoft Certified: Azure Data Engineer Associate (DP-203), Databricks Certified Data Engineer Professional, or Azure Solutions Architect Expert
  • Database Architecture: Proficiency in both Relational (SQL) and NoSQL (Document, Key-Value, Graph, Columnar) databases. Develop and maintain data models and schemas to support data analysis and reporting requirements
  • Distributed Systems: Knowledge of frameworks like Apache Hadoop, Spark, or Presto/Trino for optimizing and handling massive data volumes and retrieval mechanisms, ensuring the efficient processing of large datasets
  • Storage Optimization: Understanding file formats like Parquet, Avro, or ORC and compression techniques
  • Deep proficiency in programming languages: Python (specifically PySpark), SQL, PowerShell, and Scala
  • Infrastructure: Hands-on experience with Azure Cloud infrastructure, including Networking (VNETs), Key Vault, and Identity Management
  • Big Data Tools: Deep knowledge of Apache Spark runtime internals, MLflow for MLOps, and orchestration tools like Airflow

Frequently Asked Questions

Is the salary disclosed for the Data Platform Manager- Hybrid (Applicants must reside position at accoy?
The salary for this Data Platform Manager- Hybrid (Applicants must reside role at accoy is not publicly listed. Click "Apply Now" to learn more about the compensation package on their official careers page.
Where is the Data Platform Manager- Hybrid (Applicants must reside position at accoy located?
This Data Platform Manager- Hybrid (Applicants must reside role at accoy is based in Pittsburgh, PA, US. The position is listed as on-site or hybrid. Check the full job description or apply directly to confirm the work arrangement.
Is the Data Platform Manager- Hybrid (Applicants must reside role at accoy full-time or part-time?
This is listed as a FULL TIME position. It is posted as a Data Platform Manager- Hybrid (Applicants must reside role in the Information Technology department at accoy.
Which team or department does the Data Platform Manager- Hybrid (Applicants must reside at accoy belong to?
This Data Platform Manager- Hybrid (Applicants must reside position is part of the Information Technology department at accoy. See the full job description for more information about the team structure and responsibilities.
How do I apply for the Data Platform Manager- Hybrid (Applicants must reside position at accoy?
Click the "Apply Now" button on this page. You will be redirected to accoy's official application portal hosted on icims where you can submit your application directly.
When was the Data Platform Manager- Hybrid (Applicants must reside job at accoy posted?
This Data Platform Manager- Hybrid (Applicants must reside position at accoy was posted on Jun 11, 2024. Apply as soon as possible — early applications are often reviewed first.
Data Platform Manager- Hybrid (Applicants must reside
accoy
Apply for this role ↗

You'll be redirected to accoy's official application page on icims.