It’s important for all parties to have their incentives aligned and not feel like they’re on opposing sides.

Let’s begin by evaluating pricing options for your project. Most data vendors and clients will go for flat rate per task pricing based on the task complexity and requirements. While this makes the cost very predictable for the research team client, it can make the quality more unpredictable.

We recommend working with vendors on an hourly basis and auditing the hours of annotators based on averages for each project. Your vendor should also have tools to assist in these efforts, which will be covered in a later section. Paying annotators on an hourly basis encourages perfection and double-checking for adherence to instructions.

Paying annotators on a per-task basis encourages rushing and disincentivizes asking for clarification or re-reading project instructions, especially since annotators can usually only work on one task at a time.

There are many cases when you, the research team, pay the data vendor on a per task basis, but the vendor has to pay the annotators on an hourly basis. This creates a strong incentive for the data vendors to instruct their annotators to get the tasks done as fast as possible and stretch the bounds of your service level agreement for task quality. As such if you pay a data vendor a flat rate per task, you should inquire as to how the annotators get paid. Everyone’s incentives can align when the data vendors charge a flat percentage fee on top of the annotator’s hourly rate.

Paying hourly instead of per-task may sound more expensive, but consider again that vendors billing you per-task may take shortcuts to preserve their margins. While you have a fiscal responsibility to your stakeholders, you also have a responsibility to get high quality training data.

micro1 works with AI Data Providers and Foundational model companies on an hourly basis. We assign experts with a minimum of 10h/week per annotator in most cases, and charge for the total hours they put in, once a month. We also handle global payroll, benefits, and compliance so that research teams don’t have to worry about the tedious HR aspects of scaling up a large global team.

PipelinesTransparency with Clients