Domain Expert, AI Evaluation Task Writer: Unreal Game Development (Contract, Remote) at Patronus AI, Inc.

United States

$0.05k - $0.15k

<div class="content-intro"><h2 style="text-align: justify;"><span style="font-family: arial, helvetica, sans-serif;"><strong>About Patronus AI</strong></span></h2> <p style="text-align: justify;">&nbsp;</p> <p style="text-align: justify;"><span style="font-family: arial, helvetica, sans-serif;">Patronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.</span></p> <p style="text-align: justify;">&nbsp;</p> <p style="text-align: justify;"><span style="font-family: arial, helvetica, sans-serif;">We are the team behind some of the earliest and most influential research in AI evaluation like<a class="css-173makr-linkStyle" href="https://arxiv.org/pdf/2311.11944" target="_blank"> <u>FinanceBench</u></a>,<a class="css-173makr-linkStyle" href="https://www.forbes.com/sites/rashishrivastava/2024/07/11/this-ai-powered-coach-catches-hallucinations-in-other-ai-models/" target="_blank"> Lynx</a>,<a class="css-173makr-linkStyle" href="https://arxiv.org/abs/2311.08370" target="_blank"> <u>SimpleSafetyTests</u></a>,<a class="css-173makr-linkStyle" href="https://www.cnbc.com/2024/03/06/gpt-4-researchers-tested-leading-ai-models-for-copyright-infringement.html" target="_blank"> <u>CopyrightCatcher</u></a>,<a class="css-173makr-linkStyle" href="https://arxiv.org/abs/2501.14249" target="_blank"> <u>Humanity’s Last Exam</u></a>, and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.</span></p></div><h3>&nbsp;</h3> <h3><strong>At a glance</strong></h3> <ul> <li><strong>Engagement:</strong> Independent contractor, fully remote, asynchronous</li> <li><strong>Pay:</strong> $50-150/hr for task authoring, paid when the full task is submitted, plus a potential bonus for each accepted task. Strong performers move up within that range or onto productivity-based pay. The trial task is paid.</li> <li><strong>Commitment:</strong> 10+ hours per week</li> <li><strong>Start:</strong> Rolling, as soon as you pass the trial task</li> <li><strong>Location:</strong> Remote<br><br></li> </ul> <h3><strong>The role</strong></h3> <p>You'll design realistic Unreal Engine 5 tasks that an experienced developer, designer or artist could complete with the right project and tools, and that today's AI models get wrong. Each task is built around a brief we provide for a project area. You write the task the way you would hand it to a mid-level colleague, list the assets involved, build or document the reference solution and write acceptance checks a tester could verify without opinion. One task is about a working day of developer effort. Your work becomes evaluation and training data for frontier models.</p> <p>You don't need an AI background. You do need to be the kind of Unreal developer who knows what makes a task real work rather than busywork, and can tell when a requirement can be checked objectively and when it can't.</p> <h3>&nbsp;</h3> <h3><strong>Areas we're hiring for</strong></h3> <ul> <li><strong>Gameplay programming:</strong> C++ against engine source with Blueprint interop, and deep Gameplay Ability System work, including abilities, effects, attribute sets, gameplay cues and replicated abilities</li> <li><strong>Technical design:</strong> Blueprint-first mechanics, the game framework and game state, data tables and data assets, Enhanced Input, and prototyping a mechanic from a one-page concept</li> <li><strong>Technical art:</strong> the material editor and material instances, Niagara, hooking effects into gameplay cues, and performance budgets, with a clear sense of which visual requirements can be checked objectively</li> <li><strong>Level design:</strong> greybox to a brief, spawn and cover placement, nav mesh, and encounter scripting in Blueprint</li> </ul> <h3>&nbsp;</h3> <h3><strong>What you'll do</strong></h3> <ul> <li>Author tasks from a short brief for a project area: write the task the way you would hand it to a mid-level colleague, list the assets, Blueprints and source files involved, and write acceptance checks a tester could verify without opinion</li> <li>Build the project materials a task needs: Blueprints, C++ classes, data assets, materials, Niagara systems, levels and test maps, along with the build, cook and log output that shows the starting state</li> <li>Solve each task end to end: implement the reference solution in UE 5.8, deliver the changed files, and record each step and how you verified every check</li> <li>Write explicit scoring criteria that let two independent graders reach the same verdict, including acceptable alternative implementations</li> <li>Run tasks against current models, find where and why they fail, and adjust difficulty</li> <li>Review tasks written by others: open the project, read the task, score each part on whether it is realistic and whether the checks test the right thing, and cite what you looked at for every score</li> <li>Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases you run into</li> </ul> <h3>&nbsp;</h3> <h3><strong>Who we're looking for</strong></h3> <p><strong>Required</strong></p> <ul> <li>3+ years of hands-on Unreal Engine development in one or more of the four areas above, as a gameplay programmer, technical designer, technical artist or level designer. You need to have built the work yourself, not only directed it.</li> <li>Working command of the engine in your area (for example the gameplay framework and Gameplay Ability System, the material and Niagara systems, or greybox, nav mesh and encounter scripting), and the ability to open a project in UE 5.8, rebuild it, and work in both C++ and Blueprint</li> <li>The habit of verifying behavior in the editor and in a packaged build, and of citing the log line, callstack, asset or setting behind every conclusion</li> <li>Clear, precise writing in professional English</li> <li>Ability to work independently and asynchronously against deadlines</li> <li>A Windows machine capable of running Unreal Engine 5.8</li> </ul> <p><strong>Nice to have</strong></p> <ul> <li>At least one shipped UE5 title, or several years on a single UE5 project in your role, ideally a shooter, action or multiplayer game</li> <li>Experience with UE5 systems beyond your core area, such as Nanite, Lumen, World Partition, PCG, MetaSounds, Mass or Control Rig, and with shipping on more than one platform</li> <li>Experience with the surrounding toolchain, such as Perforce or Plastic SCM, Unreal Build Tool and Unreal Automation Tool, Unreal Insights, RenderDoc and a CI system such as Horde or Jenkins</li> <li>Prior experience writing or reviewing AI evaluation tasks, technical interview questions, tutorials, courses or engine documentation</li> <li>Familiarity with how AI models fail on engine work (APIs that don't exist or changed between versions, Blueprint logic that can't be built as described, fixes that compile but break replication or packaged builds, confident answers about settings that don't apply)</li> <li>Some overlap with US working hours</li> </ul> <h3>&nbsp;</h3> <h3><strong>Our standards</strong></h3> <ul> <li>All tasks, project files and solutions are human-authored. LLM-generated code, content or text is not permitted.</li> <li>Tasks and solutions must be original. Nothing may be copied from published or easily searchable sources, including tutorials and marketplace content.</li> <li>Never include code, assets or project files from a current or former employer, client or studio, or anything covered by an NDA. All studios, projects and scenarios are fictional, and all code and content is your own.</li> <li>Quality is judged on accuracy, rubric validity, difficulty and reviewer agreement, not volume.</li> </ul> <h3>&nbsp;</h3> <h3><strong>How hiring works</strong></h3> <ol> <li>Short AI interview and resume review</li> <li>One paid trial task, scored against a fixed quality checklist</li> <li>Onboarding and your first project</li> </ol> <p>&nbsp;</p> <p>We aim to make a decision within two weeks of your application.</p> <p>&nbsp;</p> <p>&nbsp;</p><div class="content-conclusion"><p>&nbsp;</p> <p><span style="font-family: arial, helvetica, sans-serif;"><em>Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected</em></span><span style="font-family: arial, helvetica, sans-serif;"><em> characteristics.</em></span></p> <p>&nbsp;</p> <p><span style="font-family: arial, helvetica, sans-serif;">By clicking ‘Apply’, you agree to Greenhouse's <a href="https://www.greenhouse.com/legal">Terms of Service&nbsp;</a>and&nbsp;<a class="css-1d2k4kf e10ya71h0" href="https://www.greenhouse.com/privacy-policy" target="_blank">Privacy Policy</a><a class="css-1d2k4kf e10ya71h0" href="https://app.rippling.com/legal/privacy" target="_blank">.</a></span></p> <p><span style="font-family: arial, helvetica, sans-serif;">By clicking 'Apply', you agree to Patronus AI, Inc. <a href="https://www.patronus.ai/privacy-policy">Privacy Policy</a>.</span></p></div>