Position: AI Evaluation Specialist
Type: Contract
Compensation: $20 - $35/hour
Location: Remote
Commitment: 10-40 hrs/week
Role Responsibilities
- Design and implement self-contained evaluation tasks, including prompts, supporting files, and detailed grading rubrics to assess AI performance on practical computer-based workflows.
- Define clear, unambiguous written criteria for successful and unsuccessful task completion across diverse administrative and workflow scenarios.
- Meticulously observe and document AI agent behaviors, producing crisp, precise summaries and reports in high-quality English.
- Iterate and refine evaluation tasks and rubrics based on feedback and team collaboration to ensure robust benchmarking methodologies.
- Collaborate with the customer's team to share insights and help drive continuous improvement in AI evaluation techniques.
Requirements
- Have a minimum of of experience in roles emphasizing written precision and structured thinking, such as paralegal, executive assistant, junior analyst, librarian, document archival specialist, research assistant, technical writer, or QA analyst.
- Be native or fluent in English writing, with a demonstrated ability to produce observations that are succinct, specific, and unambiguous.
- Proven skill in designing or applying rubric-based evaluation, grading against set criteria, or building structured scoring frameworks is required.
- Possess high attention to detail and the ability to notice subtle patterns or inconsistencies others might miss.
- Exceptional written and verbal communication skills are necessary, especially for documenting nuanced observations and feedback.
Application Process
- Easy Apply on LinkedIn
- Check email for next steps
- Participate in resume evaluation & interview stage
