This role is for one of our clients
Role Type: Contract
Engagement Duration: 3 months
Location: Remote – North America preferred
Preferred Geographies: United States (approved regions) and Canada
Start Date: Immediate
Experience Required: 5+ years
Compensation: $100–$150 per hour
Availability: Minimum 4 hours per day and 20 hours per week, with at least 4 hours of overlap with Pacific Standard Time (PST)
We are looking for a highly skilled Senior Software Engineer with expertise in Python, TypeScript, JavaScript, or Go to support advanced AI development and evaluation initiatives. The ideal candidate combines strong software engineering fundamentals with exceptional attention to quality, hands-on experience in preference data collection or rubric creation, and the ability to translate complex technical work into clear, actionable insights.
This role is suited to engineers who can operate effectively in ambiguous, fast-paced environments, exercise sound technical judgment, and prioritize practical execution. Candidates should be able to distinguish high-quality examples from mediocre ones, articulate their reasoning, and help turn qualitative assessments into scalable evaluation processes.
Prior experience working with large, complex codebases, such as Visual Studio Code (VS Code), is an advantage.
Requirements
Key Responsibilities
- Rubric Development: Design, refine, and iterate on rubrics, evaluation frameworks, and quality criteria for preference data.
- Data Review and Quality Assurance: Review, assess, and label data while identifying subtle inconsistencies, quality issues, and opportunities for improvement.
- Engineering Infrastructure: Build, maintain, and optimize pipelines and infrastructure supporting data generation, collection, processing, and evaluation workflows.
- Technical Documentation: Synthesize findings from data analysis and evaluation activities into concise reports, updates, recommendations, and actionable summaries.
- Cross-Functional Collaboration: Work closely with researchers and engineers to translate qualitative judgments into structured, repeatable, and scalable processes.
- Codebase Navigation: Understand, contribute to, and maintain complex software repositories while ensuring code quality, reliability, and performance.
- Continuous Improvement: Identify workflow inefficiencies and implement practical improvements to data quality, evaluation consistency, and engineering processes.
Required Skills and Qualifications
- At least 5 years of professional software engineering experience with Python, particularly in building and maintaining pipelines and infrastructure, or with TypeScript, JavaScript, or Go.
- Hands-on experience with preference data collection, rubric creation, or developing evaluation criteria.
- Strong understanding of software engineering principles, code quality, debugging, and maintainable system design.
- Excellent written and verbal communication skills, with the ability to explain complex technical concepts clearly and concisely.
- Strong analytical skills and the ability to make consistent, well-reasoned quality judgments.
- Demonstrated ability to explain why a particular solution, example, or implementation meets a high quality standard.
- Comfort working independently in ambiguous environments while prioritizing execution and timely delivery.
Preferred Qualifications
- Experience contributing to or navigating large-scale, complex open-source codebases, such as VS Code.
- Experience building data pipelines, developer infrastructure, or automated evaluation systems.
- Background in AI model evaluation, preference data workflows, data annotation, or quality assessment.
- Experience collaborating with AI researchers, machine learning engineers, or multidisciplinary technical teams.
- Education from a leading university or professional experience at a highly regarded technology company.
Evaluation and Selection Process
Candidates will undergo the following evaluation stages:
- Application and Screening: Submit an application and complete the required screening questions.
- AI Interview: Complete an AI-based interview.
- Coding Challenge: Demonstrate technical proficiency through a coding assessment.
- Technical and Cultural Interview: Participate in an approximately 30-minute discussion covering technical expertise, communication, judgment, and collaboration.
- Final Review: Complete the final candidate evaluation and selection process.
Additional Requirements
- A LinkedIn profile is mandatory and must be included in the candidate's resume.
- Candidates from leading technology companies and top-tier universities are strongly preferred.
- Candidates must not currently be employed by Microsoft.
- Applicants should be able to commit to the required weekly hours and PST overlap for the full contract duration.
New Zealand