Eurisko is looking for an AI QA Engineer to test and validate applications powered by large language models (LLMs) and generative AI.
The role combines traditional quality assurance with prompt testing, AI response evaluation, RAG validation, and automated quality assessments. You will work closely with AI engineers, developers, and product teams to identify failure patterns, establish measurable quality standards, and prevent regressions across AI-powered applications.
Key Responsibilities
- Test prompts and conversational behavior: Design and execute test scenarios covering system instructions, prompt variations, ambiguous inputs, edge cases, multilingual interactions, and multi-turn conversations. Validate instruction-following, context retention, clarification behavior, and response consistency.
- Evaluate AI-generated responses: Assess accuracy, relevance, completeness, tone, and adherence to business requirements. Identify hallucinations, unsupported claims, inappropriate responses, and failures to acknowledge uncertainty.
- Validate RAG applications: Test retrieval quality, answer grounding, citation accuracy, and behavior when information is missing, outdated, contradictory, or restricted by user permissions.
- Build automated evaluation and regression suites: Create representative test datasets, scoring criteria, and repeatable evaluations. Compare results across prompt, model, knowledge-base, and application changes, using automated checks alongside human review.
- Perform safety and adversarial testing: Test prompt injection, attempts to bypass instructions, sensitive-data exposure, and unauthorized tool use. Validate guardrails, refusal behavior, and safe fallback handling.
- Test end-to-end AI workflows and report quality: Validate APIs, agent actions, tool calls, conversation memory, error handling, latency, and integration behavior. Document reproducible defects, track evaluation results, and collaborate with engineering teams on fixes and release readiness.
Required Qualifications
- Professional experience in software QA, including test planning, test-case design, defect management, API testing, and test automation.
- Hands-on experience testing LLM-powered or generative AI applications, with a strong understanding of prompt behavior and AI-specific failure modes.
- Working knowledge of RAG, embeddings, vector stores, context management, and tool calling, sufficient to design meaningful tests and investigate failures.
- Practical Python skills for test automation, dataset preparation, API integration, and evaluation scripts.
- Ability to define measurable evaluation criteria and test systems whose outputs may vary between runs, rather than relying solely on exact-text assertions.
- Experience working in Agile/Scrum teams, with strong analytical, documentation, and communication skills.
Preferred Qualifications
- Experience in banking or fintech, particularly applications involving sensitive information and controlled business workflows.
- Experience evaluating Arabic and English AI interactions, including dialects, mixed-language inputs, and localized terminology.
- Familiarity with AI evaluation and observability tools, CI/CD quality gates, microservice architectures, and agent-based applications.
