Senior AI Engineer - AI Systems Evaluation Team

Real Dev Incยท Engineering
Apply Now โ†—
๐Ÿ“ Tel Aviv-Yafo, Tel Aviv District, IsraelFull time

About this role

REAL is building an AI Execution Platform for real estate organizations.

Today, the data required to run real estate is scattered across fragmented systems, leading to missed insights and preventable financial leakage.

REAL transforms this complexity into connected intelligence and automated execution, enabling enterprises to operate with greater precision and confidence.

REAL Values

  • Ownership: We take responsibility and move decisively.
  • Clarity: We simplify complexity to deliver meaningful impact.
  • Accuracy: Precision matters in everything we build.
  • Velocity: We work with urgency and intent.
  • Partnership: We collaborate closely with customers and teammates.

Role Overview

  • Own the systems that define, measure, and enforce AI quality at REAL.
  • Translate ambiguous model behavior into measurable signals, automated tests, and release gates.
  • Operate across evaluation design, tooling, and production integration.

What You'll Do

  • Design evaluation architectures (benchmarks, regression suites, coverage)
  • Build automated pipelines to run and score evals across models and prompts
  • Implement scoring systems (LLM-as-judge, rubrics, hybrid approaches)
  • Create and maintain golden datasets + edge-case suites
  • Develop internal tools for prompt testing, dataset generation, experiment tracking
  • Instrument systems for traces, outputs, and debugging
  • Detect regressions and enforce quality gates in CI/CD
  • Monitor model performance in production
  • Close the loop between eval insights and product improvements

What We're Looking For

  • 3-6 years building production software, internal platforms, ML/data infrastructure, experimentation systems, or AI tooling
  • Strong backend and systems engineering fundamentals with hands-on applied AI experience
  • Strong Python, production-level systems experience
  • Built testing frameworks or validation systems end-to-end
  • Hands-on with LLMs / RAG / agent workflows
  • Understands eval methods (benchmarking, A/B, LLM-as-judge, HITL)
  • Experience with observability / logging / experiment tracking
  • Strong systems thinking (coverage, reliability, reproducibility)
  • Comfort with non-deterministic systems

Nice to Have

  • Experience with eval, tracing, observability, or experimentation tooling (one or more of the following: LangSmith, Braintrust, Phoenix, MLflow, OpenTelemetry, PostHog, custom eval stacks)
  • Familiarity with dataset/versioning workflows, HITL systems, and production AI observability systems
  • CI/CD integration for model evaluation
  • Background in search, retrieval, or document systems
  • Built internal platforms or developer tools
  • Experience working in startups and business driven environments

Frequently Asked Questions

Is the salary disclosed for the Senior AI Engineer - AI Systems Evaluation Team position at Real Dev Inc?
The salary for this Senior AI Engineer - AI Systems Evaluation Team role at Real Dev Inc is not publicly listed. Click "Apply Now" to learn more about the compensation package on their official careers page.
Where is the Senior AI Engineer - AI Systems Evaluation Team position at Real Dev Inc located?
This Senior AI Engineer - AI Systems Evaluation Team role at Real Dev Inc is based in Tel Aviv-Yafo, Tel Aviv District, Israel. The position is listed as on-site or hybrid. Check the full job description or apply directly to confirm the work arrangement.
Is the Senior AI Engineer - AI Systems Evaluation Team role at Real Dev Inc full-time or part-time?
This is listed as a Full time position. It is posted as a Senior AI Engineer - AI Systems Evaluation Team role in the Engineering department at Real Dev Inc.
Which team or department does the Senior AI Engineer - AI Systems Evaluation Team at Real Dev Inc belong to?
This Senior AI Engineer - AI Systems Evaluation Team position is part of the Engineering department at Real Dev Inc. See the full job description for more information about the team structure and responsibilities.
How do I apply for the Senior AI Engineer - AI Systems Evaluation Team position at Real Dev Inc?
Click the "Apply Now" button on this page. You will be redirected to Real Dev Inc's official application portal hosted on workable where you can submit your application directly.
When was the Senior AI Engineer - AI Systems Evaluation Team job at Real Dev Inc posted?
This Senior AI Engineer - AI Systems Evaluation Team position at Real Dev Inc was posted on May 12, 2026. Apply as soon as possible โ€” early applications are often reviewed first.
Senior AI Engineer - AI Systems Evaluation Team
Real Dev Inc
Apply for this role โ†—

You'll be redirected to Real Dev Inc's official application page on workable.