About the company
Konecta
Outsourcing and Offshoring Consulting
About the role
We are looking for a Senior QA Engineer with proven expertise in testing agentic systems
to establish and coordinate quality assurance across the platform and all Agentic
Solutions published on it.
Based in Egypt, you will work with developers, engineering leads, the Proxy Product Owner
and business stakeholders across the entire project. You will combine hands-on testing
and automation with the development of consistent quality standards, evaluation
methods and release criteria.
Your scope will cover both the reliability of the underlying platform and the ability of
individual Agentic Solutions to complete business tasks accurately, securely and within
their authorised boundaries.
Key responsibilities
• Define and implement a QA strategy covering functional, integration, end-to-end,
regression, performance and AI evaluation testing.
• Establish consistent testing standards across teams and workstreams.
• Prioritise coverage according to business impact and risk, automating critical journeys
first.
• Work with developers to embed testing throughout delivery and improve the testability
of new features.
• Maintain a consolidated view of quality, coverage gaps and release risks.
• Design and execute tests for platform APIs, user interfaces, tenant provisioning,
authentication, roles and permissions.
• Validate knowledge-base management, agent and crew orchestration, tracing,
configuration and billing capabilities.
• Test integrations between platform components and external services.
• Verify tenant isolation, access boundaries and correct handling of sensitive data,
working with security specialists.
• Test failure handling and recovery, including timeouts, retries, unavailable
dependencies and interrupted workflows.
• Ensure platform changes do not introduce regressions in existing capabilities or
published Agentic Solutions.
• Define and execute a repeatable validation process for every Agentic Solution before
publication and following significant changes.
• Test complete business journeys against agreed requirements and expected
outcomes.
• Validate tool selection, tool inputs, execution sequences, API interactions, knowledge
retrieval, memory and human handoffs where applicable.
• Inspect execution traces to assess intermediate actions and identify the source of
failures.
• Verify that agents handle ambiguous requests, missing information, unexpected
inputs and dependency failures appropriately.
• Confirm that agents respect authorisation boundaries, approval requirements and
limits on their actions.
• Test safeguards against prompt injection, sensitive-data disclosure, unauthorised
actions and unintended or duplicate execution.
• Use controlled test environments to validate actions without affecting live customers
or production data.
• Build and maintain versioned evaluation datasets covering representative business
scenarios, edge cases and adversarial inputs.
• Define measurable acceptance thresholds with the Proxy Product Owner and relevant
business owners.
• Evaluate task completion, factual accuracy, grounding, tool-use correctness and
compliance with solution-specific requirements.
• Combine automated scoring with human review where appropriate, accounting for
variability in AI outputs.
• Run regression evaluations following changes to models, prompts, tools, knowledge
sources or platform components.
• Maintain traceability between evaluation results and the versions of the solution,
model and configuration tested.
• Build and maintain automated API, integration, browser-based and end-to-end test
suites.
• Integrate tests and AI evaluations into GitHub-based CI/CD workflows.
• Implement agreed quality gates for pull requests, releases and solution publication.
• Maintain reliable test data, fixtures and environments.
• Investigate unstable tests and keep automated results actionable and trustworthy.
• Test platform and solution behaviour under expected load and concurrent usage.
• Measure latency, throughput, failure rates and recovery behaviour.
• Assess response times and cost per completed agentic journey against agreed
targets.
• Identify bottlenecks and work with engineers to verify improvements.
• Record defects in GitHub with reproducible steps, expected and actual results, severity,
evidence and affected versions.
• Work with the Proxy Product Owner and engineering leads to assess business impact
and prioritise fixes.
• Track defects through resolution, retesting and closure.
• Provide evidence-based release and publication recommendations.
• Document outstanding defects, untested areas and residual risks so designated
release owners can make informed decisions.
• Translate production incidents and recurring defects into regression tests and process
improvements.
• Provide a formal weekly quality update to the Global Head of Platform, Proxy Product
Owner and engineering leads.
• Report test results, critical defects, coverage gaps, AI evaluation trends and release
readiness.
• Escalate significant quality risks promptly, with their impact and recommended
actions.
• Participate in refinement and planning to challenge unclear requirements and
improve acceptance criteria.
• Present relevant findings and demonstrations during team reviews and Show & Tell
sessions.
Required experience and skills
• At least five years of experience in software QA or test engineering, including
substantial hands-on automation experience.
• Demonstrated experience establishing or improving QA practices across software
delivery teams.
• Proven hands-on expertise testing tool-using AI agents and multi-step agentic
workflows, including intermediate actions, failure recovery and business-outcome
completion.
• Practical experience building AI evaluation datasets, automated evaluation pipelines
and regression suites.
• Strong Python skills for test automation, using pytest or equivalent tooling.
• Strong API testing skills and experience with browser automation frameworks such as
Playwright, Cypress or equivalent.
• Experience integrating automated tests into CI/CD pipelines, preferably GitHub
Actions.
• Experience with performance and load testing of API-based services.
• Working understanding of RAG, agent orchestration, tool calling, memory and
execution tracing.
• Familiarity with security and privacy testing relevant to enterprise AI systems.
• SQL and data-validation skills.
• Excellent written and spoken English, with the ability to explain quality risks clearly to
technical and business stakeholders.
• A proactive approach: you identify gaps, investigate issues independently and bring
practical recommendations.
Nice to have
• Experience with AI evaluation tools such as promptfoo, Ragas, DeepEval or equivalent.
• Experience testing enterprise platforms with multiple tenants.
• Background in conversational AI, contact-centre technology or customer experience.
• Experience testing voice agents, including interruptions, speech recognition, latency
and human transfer.
• Exposure to GCP, containerised environments and cloud-native applications.
• Experience with production monitoring and AI observability.
• ISTQB, ISTQB AI Testing or equivalent certification.