About The RoleRex.zone is hiring for remote data labeling jobs in Amsterdam focused on building high-quality training data for AI/ML systems. You will follow annotation guidelines, perform QA evaluation, and support LLM training pipelines through RLHF, prompt evaluation, and content safety labeling. Work is fully remote with asynchronous collaboration and defined quality checkpoints.
What You Will Do
- * Perform data labeling and data annotation across NLP and computer vision tasks (classification, spans, bounding boxes, segmentation, ranking)
- Execute RLHF-style preference judgments and prompt evaluation to support LLM training pipelines and evaluation suites
- Run QA evaluation checks: consistency audits, spot checks, rework loops, and error taxonomy reporting
- Maintain annotation guidelines compliance and propose clarifications for ambiguous cases
- Measure and improve training data quality using agreed metrics (accuracy, agreement, defect rates)
- Label and review content safety categories (toxicity, self-harm, sexual content, violence, regulated goods)
- Document edge cases and rubric updates to improve repeatability and inter-annotator agreement
- Support dataset versioning and change management to connect data changes to model performance improvements
Required Qualifications
- * Mid-Senior experience in data labeling, data annotation, QA evaluation, or related data operations roles
- Familiarity with NLP tasks such as named entity recognition, classification, and summarization evaluation
- Comfort with computer vision annotation concepts and review workflows
- Experience with LLM/prompt evaluation rubrics (helpfulness, correctness, harmlessness, groundedness)
- Strong written communication and disciplined remote work habits
How To ApplyApply via Rex.zone with your resume and a brief summary of relevant labeling and QA experience. Selected candidates may complete a short calibration task to verify rubric understanding and consistency.
