Back to results

AlgoShack

AI Test Engineer

Location not specified

Experience
5+ years
Experience level
Senior
Job type
Full-Time
Workplace
Onsite

About the role

AI Test Engineer

The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change, and who can build that testing setup alone.

What this person will actually do

•

  • Build automated accuracy checks for the AI
  • Make those checks a gate in the CI pipeline,]
  • Build and maintain test datasets
  • Test the whole pipeline end to end:
  • Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
  • Run safety and robustness testing: prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
  • Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).

Must-have skills (screen hard on these)

•

  • Strong Python test automation: pytest, API testing, CI pipelines (GitLab).
  • LLM evaluation hands-on: has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
  • Experience with at least one eval or tracing tool: LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
  • Understands non-deterministic testing: how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
  • Document or vision AI exposure: OCR or extraction from PDFs and images is a big plus for us.
  • Solo-capable: has set up a test approach from scratch, not only executed someone else’s.

Good-to-have

•

  • LangChain / LangGraph. Useful, but not essential: our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
  • Google Cloud / Vertex AI, Django APIs, Playwright.
  • Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
  • Performance and cost testing of LLM calls (latency, tokens).

Job details

Make your next move

You can preview 30 jobs. Sign in to search, filter, save and browse the full collection.

Continue to sign in

We request your basic Google profile, without the email permission. No marketing emails. Privacy policy