Domain Expert, AI Evaluation Task Writer: Legal (Contract, Remote) at Patronus AI, Inc.

United States

$0.05k - $0.15k

<div class="content-intro"><h2 style="text-align: justify;"><span style="font-family: arial, helvetica, sans-serif;"><strong>About Patronus AI</strong></span></h2> <p style="text-align: justify;">&nbsp;</p> <p style="text-align: justify;"><span style="font-family: arial, helvetica, sans-serif;">Patronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.</span></p> <p style="text-align: justify;">&nbsp;</p> <p style="text-align: justify;"><span style="font-family: arial, helvetica, sans-serif;">We are the team behind some of the earliest and most influential research in AI evaluation like<a class="css-173makr-linkStyle" href="https://arxiv.org/pdf/2311.11944" target="_blank"> <u>FinanceBench</u></a>,<a class="css-173makr-linkStyle" href="https://www.forbes.com/sites/rashishrivastava/2024/07/11/this-ai-powered-coach-catches-hallucinations-in-other-ai-models/" target="_blank"> Lynx</a>,<a class="css-173makr-linkStyle" href="https://arxiv.org/abs/2311.08370" target="_blank"> <u>SimpleSafetyTests</u></a>,<a class="css-173makr-linkStyle" href="https://www.cnbc.com/2024/03/06/gpt-4-researchers-tested-leading-ai-models-for-copyright-infringement.html" target="_blank"> <u>CopyrightCatcher</u></a>,<a class="css-173makr-linkStyle" href="https://arxiv.org/abs/2501.14249" target="_blank"> <u>Humanity’s Last Exam</u></a>, and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.</span></p></div><h3>&nbsp;</h3> <h3><strong>At a glance</strong></h3> <ul> <li><strong>Engagement:</strong> Independent contractor, fully remote, asynchronous</li> <li><strong>Pay:</strong> $50-150/hr for task authoring, paid when the full task is submitted, plus a potential bonus for each accepted task. Strong performers move up within that range or onto productivity-based pay. The trial task is paid.</li> <li><strong>Commitment:</strong> 10+ hours per week</li> <li><strong>Start:</strong> Rolling, as soon as you pass the trial task</li> <li><strong>Location:</strong> Remote</li> </ul> <p>&nbsp;</p> <h3><strong>The role</strong></h3> <p>You'll design realistic legal tasks that a qualified professional could solve with the right knowledge and tools, and that today's AI models get wrong. Each task is built around a scenario we provide. You create the working documents someone in your role would actually handle, write the request, solve it yourself and define how the answer gets graded. Your work becomes evaluation and training data for frontier models.</p> <p>You don't need an AI background. You do need to be the kind of legal professional who checks the amendment before relying on the agreement, and notices when a deadline has moved.</p> <h3>&nbsp;</h3> <h3><strong>Areas we're hiring for</strong></h3> <ul> <li><strong>Civil litigation:</strong> paralegal and litigation support work across discovery and productions, privilege review and logs, depositions, pleadings and motions, subpoenas, dockets and deadlines under federal and local rules</li> <li><strong>Corporate and M&amp;A transactions:</strong> transactional paralegal work on due diligence, data rooms, disclosure schedules, signature pages and closings, entity records and cap tables, and purchase and merger agreements</li> <li><strong>Commercial contracts:</strong> contract managers, specialists and administrators who review and redline against playbooks and handle MSAs, amendments, renewals and notice periods, obligation tracking and procurement or vendor contracts</li> <li><strong>Privacy and data subject requests:</strong> privacy and data protection analysts who fulfill access and deletion requests end to end, maintain data maps, ROPAs and retention schedules, and apply GDPR and CCPA in practice, not only in policy</li> </ul> <h3>&nbsp;</h3> <h3><strong>What you'll do</strong></h3> <ul> <li>Build realistic evidence files from a provided scenario: services agreements and amendments, email threads, pleadings and discovery requests, privilege logs, docket entries, closing checklists and disclosure schedules, signature packages, cap tables, contract playbooks and redlines, request intake records, data maps and retention schedules</li> <li>Write clear task requests that state what the deliverable is and how uncertainty and justified assumptions should be handled</li> <li>Solve each task end to end and document a reference solution with the reasoning behind every step</li> <li>Write explicit scoring criteria that let two independent graders reach the same verdict, including acceptable alternative answers and the limits of what the solver is authorized to decide</li> <li>Run tasks against current models, find where and why they fail, and adjust difficulty</li> <li>Check your own and peers' tasks for legal accuracy, ambiguity, answers exposed in the materials and shortcuts that skip the intended reasoning</li> <li>Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases you run into</li> </ul> <h3>&nbsp;</h3> <h3><strong>Who we're looking for</strong></h3> <p><strong>Required</strong></p> <ul> <li>3+ years of hands-on experience in one or more of the four areas above, as a paralegal, legal assistant, contracts manager or specialist, privacy or data protection analyst, or attorney. You need to have done the work yourself, not only supervised it.</li> <li>Working knowledge of the rules, procedures and documentation standards that govern your area (for example federal and local court rules and discovery practice, closing mechanics, contract playbooks and CLM workflows, GDPR and CCPA request handling)</li> <li>Careful reading, and the habit of citing the clause, message or docket entry behind every conclusion</li> <li>Clear, precise writing in professional English</li> <li>Ability to work independently and asynchronously against deadlines</li> </ul> <p><strong>Nice to have</strong></p> <ul> <li>Paralegal certification (for example NALA CP or NFPA RP), a law degree, CIPP/US or CIPP/E, or a contract management credential</li> <li>Experience with the systems used in your area, such as Relativity, Everlaw or DISCO, a virtual data room, Ironclad, Icertis, Agiloft or Conga, or OneTrust, TrustArc or Transcend</li> <li>Law firm, in-house and legal operations experience across more than one setting</li> <li>Prior experience writing or reviewing AI evaluation tasks, exam questions, case studies or training material</li> <li>Familiarity with how AI models fail (fabricated citations, misread clauses, miscounted deadlines, overconfident conclusions)</li> </ul> <h3>&nbsp;</h3> <h3><strong>Our standards</strong></h3> <ul> <li>All tasks, documents and solutions are human-authored. LLM-generated content is not permitted.</li> <li>Tasks and solutions must be original. Nothing may be copied from published or easily searchable sources.</li> <li>Never use real client, matter, employer or personal data, or anything covered by privilege or a confidentiality obligation. All parties and scenarios are fictional.</li> <li>Quality is judged on accuracy, rubric validity, difficulty and reviewer agreement, not volume.</li> </ul> <h3>&nbsp;</h3> <h3><strong>How hiring works</strong></h3> <ul> <li>Short AI interview and resume review</li> <li>One paid trial task, scored against a fixed quality checklist</li> <li>Onboarding and your first project</li> </ul> <h3>&nbsp;</h3> <p>We aim to make a decision within two weeks of your application.</p> <p>&nbsp;</p> <p>&nbsp;</p><div class="content-conclusion"><p>&nbsp;</p> <p><span style="font-family: arial, helvetica, sans-serif;"><em>Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected</em></span><span style="font-family: arial, helvetica, sans-serif;"><em> characteristics.</em></span></p> <p>&nbsp;</p> <p><span style="font-family: arial, helvetica, sans-serif;">By clicking ‘Apply’, you agree to Greenhouse's <a href="https://www.greenhouse.com/legal">Terms of Service&nbsp;</a>and&nbsp;<a class="css-1d2k4kf e10ya71h0" href="https://www.greenhouse.com/privacy-policy" target="_blank">Privacy Policy</a><a class="css-1d2k4kf e10ya71h0" href="https://app.rippling.com/legal/privacy" target="_blank">.</a></span></p> <p><span style="font-family: arial, helvetica, sans-serif;">By clicking 'Apply', you agree to Patronus AI, Inc. <a href="https://www.patronus.ai/privacy-policy">Privacy Policy</a>.</span></p></div>