About this role
Description
You'll evaluate prompt-response pairs generated by large language models, scoring them across dimensions like factual accuracy, instruction-following, coherence, and appropriate refusal behavior. Each session involves reviewing batches of 20–50 pairs using a structured rubric inside a web-based annotation platform, flagging edge cases, and writing brief justifications for non-obvious scores.
Beyond surface-level ratings, you'll identify subtle failure modes — responses that are technically accurate but misleading, answers that follow the letter of a prompt while violating its intent, and outputs that pass a quick read but contain embedded errors. You'll work asynchronously, with a typical batch taking 60–90 minutes, and you're expected to maintain inter-annotator agreement scores above 0.75 kappa.
Feedback you submit feeds directly into preference datasets used to fine-tune and align production models. Calibration sessions run bi-weekly, where you'll review disagreements with a senior evaluator and update your scoring approach. Strong performance can lead to specialized tracks covering domain-specific evaluation (legal, medical, code).
Skills & requirements
Prompt evaluationRubric-based scoringLLM output analysisInstruction-following assessmentFactual accuracy verificationAnnotation platform proficiencyApply NowFrequently asked questions
What does a typical batch look like?How is inter-annotator agreement measured?Can I specialize in a domain like legal or medical?// QUICK INFOType
Contractor
Compensation
$35–55/hr
Location
Remote
Category
RLHF
Posted
August 3, 2026
Apply Now Back to all positions// RELATED POSITIONS
More RLHF rolesRLHFMath Reasoning RLHF Annotator$35–55/hrViewRLHFLegal Reasoning AI Response Evaluator$55–100/hrViewRLHFPhysics Problem Evaluator$55–100/hrViewRLHFHuman Feedback Annotator$35–40/hrViewView all open positions// READY TO GET STARTED?
Apply in minutesCreate your profile, select your areas of expertise, and start working on frontier AI projects.
Apply NowBacked by AI company Reka, with clear flat rates published on every role. Video-capture pay depends heavily on region, and expert roles are few.
AITrainerGigs aggregates this listing from Claru.