QA Engineer (AI & Feature Testing)
Solvd · Full-time · Posted 2w ago
Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.
Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery—from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.
We are looking for a Senior QA Engineer to join a team building and validating AI-powered and traditional features for a leading personal finance platform. You’ll own quality across web, mobile, and API — with a particular focus on testing AI/LLM-integrated components where standard deterministic methods don’t apply.
This role requires someone who understands the fundamental difference between testing software that has a right answer and evaluating AI outputs that have a probability distribution — and who has done both in production.
Plan, design, and execute test cases for AI-powered components and traditional features, applying the appropriate methodology — eval-based or deterministic — based on the target behavior.
Analyze traditional functional requirements as well as AI model behavior specifications, prompt outputs, and edge-case scenarios.
Turn business requirements, user flows, and business process procedures into comprehensive test plans, test cases, and test scripts.
Perform manual feature testing across platforms and verify bug fixes.
Create clear, detailed defect reports and document unexpected AI model responses, hallucinations, and performance regressions.
Assist in developing QA practices for emerging technologies including AI frameworks, and disseminate best practices across the team.
Become a subject matter expert in core systems, interfaces, business processes, and reporting tools.
Partner closely with functional teams, engineers, and IT in a global environment with aligned LATAM working hours.
5–7 years of experience in software testing across web, mobile, and API.
Hands-on experience with Playwright in a production testing environment — this is a primary requirement for this role.
Hands-on AI feature testing experience — validating AI/LLM-integrated products for accuracy, relevance, robustness against adversarial prompts, edge cases, and safety/toxicity.
Clear understanding of eval vs. deterministic testing — the difference between pass/fail based on exact input/output and evaluating probabilistic AI outputs using human-in-the-loop evaluations, score metrics, benchmark datasets, and semantic evaluation.
Strong understanding of web and mobile application development lifecycles including iOS and Android specifics.
Proven experience in integration, regression, performance/load, and UAT.
Hands-on knowledge of browser DevTools and log monitoring.
Advanced proficiency in Jira and Confluence.
Experience with test management systems.
Experience with HTML proxy tools such as Charles Proxy or Fiddler.
Hands-on knowledge of Xcode logs and ADB logs.
Experience in the financial services or fintech domain.
Bachelor’s degree in Computer Science or equivalent.
When you join Solvd, you'll…
Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.
Be part of a global team with equal opportunities for collaboration across continents and cultures.
Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.
Ready to make an impact?
If you're excited to build things that matter, champion responsible AI, and grow with some of the industry’s sharpest minds. Apply today and let’s innovate together.
Solvd is an equal opportunity employer.
New Zealand