OpenAI asks contractors to upload work from past jobs to evaluate AI agent performance

OpenAI asks third-party contractors to upload real-world assignments and tasks from their current or previous workplaces so they can use the data to evaluate their next generation’s performance AI modelsaccording to records from OpenAI and training data company Handshake AI obtained by WIRED.

The project appears to be part of OpenAI’s efforts to establish a human baseline for different tasks that can then be compared to AI models. In September, the company launched a new assessment process to measure the performance of its AI models against human professionals in a variety of industries. OpenAI claims this is a key indicator of its progress toward achieving AGI, or an AI system that outperforms humans at the most profitable tasks.

“We hired people from all professions to help us collect real-world tasks modeled after those you did in your full-time jobs, so that we can measure the effectiveness of AI models on these tasks,” reads a confidential document from OpenAI. “Take existing long-term or complex work tasks (hours or days+) that you have done in your profession and turn each one into a task.”

OpenAI asks contractors to describe tasks they have performed in their current or past jobs and to upload real-world examples of work they have performed, according to an OpenAI presentation on the project viewed by WIRED. Each of the examples must be “a concrete result (not a summary of the file, but the file itself), for example a Word document, PDF, Powerpoint, Excel, image, deposit,” notes the presentation. OpenAI says people can also share fabricated working examples to demonstrate how they would realistically react in specific scenarios.

OpenAI and Handshake AI declined to comment.

Real-world tasks have two components, according to OpenAI’s presentation. There is the task request (what a person’s manager or co-worker told them to do) and the task deliverable (the actual work they produced in response to that request). The company repeatedly emphasizes in its instructions that examples shared by contractors should reflect “actual on-the-ground work” that the person has done.In fact do.”

An example in OpenAI’s presentation describes a task of a “Senior Lifestyle Manager at a luxury concierge company for ultra-high net worth individuals.” The goal is to “prepare a short 2-page PDF draft of an outline of a 7-day yacht trip to the Bahamas for a family traveling there for the first time.” It includes additional details regarding the family’s interests and what the itinerary should look like. The “Experienced Human Deliverable” then shows what the contractor in this case would upload: a real Bahamas itinerary created for a client.

OpenAI requires contractors to remove company intellectual property and personally identifiable information from the work files they upload. In a section titled “Important Reminders,” OpenAI asks employees to “delete or anonymize: personal information, proprietary or confidential data, important non-public information (e.g., internal strategy, unpublished product details). »

One of the files viewed by the WIRED document mentions a ChatGPT tool called “Superstar Scrub» which provides advice on how to delete confidential information.

Evan Brown, an intellectual property attorney at Neal & McDevitt, told WIRED that AI labs that receive confidential information from contractors on this scale could face trade secret misappropriation claims. Contractors who offer documents from their former workplaces to an AI company, even if redacted, could risk violating their former employers’ nondisclosure agreements or revealing trade secrets.

“The AI ​​lab places a lot of trust in its contractors to decide what is confidential and what is not,” Brown says. “If they let something slide, are the AI ​​labs really taking the time to determine what is and isn’t a trade secret? It seems to me that the AI ​​lab is putting itself at great risk.”