Legal Reasoning AI Response Evaluator

Description

You'll evaluate AI-generated responses to legal questions — covering case analysis, statutory interpretation, procedural questions, and litigation strategy. Your primary task is assessing whether a model's answer reflects sound legal reasoning, accurate doctrinal knowledge, and appropriate caveats. This isn't about copy-editing; you're judging whether the legal analysis would hold up in front of a practitioner, and flagging anything that could mislead a non-expert relying on the output.

Day-to-day work involves reading a legal prompt, reviewing the model's response, and scoring it across dimensions like analytical rigor, jurisdictional accuracy, and practical usefulness. For litigation-focused tasks, you'll also evaluate whether the model correctly identifies procedural posture, burden of proof, and viable arguments. Feedback is written in structured rubric fields, with an optional free-text section for nuanced critique. Tasks are asynchronous — you work batches on your own schedule through a browser-based annotation platform.

A minimum of 3 years of litigation or substantive legal practice is required. Familiarity with U.S. federal and state court procedure is expected; experience in additional jurisdictions is a bonus. Law professors and clerks are also strong candidates. You do not need to be currently practicing, but you must be able to credibly evaluate whether legal reasoning is sound, not just grammatically correct.

Skills & requirements

Legal reasoning and case analysisLitigation and civil procedureStatutory and regulatory interpretationAnalytical writingJurisdictional knowledge (US federal/state)Apply Now

Frequently asked questions

Do I need to be a currently licensed attorney?Is the work focused on a specific area of law?How long does a typical task take?// QUICK INFO

Type

Contractor

Compensation

$55–100/hr

Location

Remote

Category

RLHF

Posted

June 29, 2026

Apply Now Back to all positions

// RELATED POSITIONS

More RLHF rolesRLHFAI Prompt Quality Rater$35–55/hrViewRLHFMath Reasoning RLHF Annotator$35–55/hrViewRLHFPhysics Problem Evaluator$55–100/hrViewRLHFHuman Feedback Annotator$35–40/hrViewView all open positions

// READY TO GET STARTED?

Apply in minutes

Create your profile, select your areas of expertise, and start working on frontier AI projects.

Apply Now