Projects should begin with a preparation phase, which may also be your pilot phase with the data vendor. Instructions should be carefully made with a lot of communication between the project manager and research team. A small team of experienced annotators should use these instructions to create the first set of completed tasks, which are then reviewed by the PM & research team.

Use this first batch of tasks to iterate on the instructions. These tasks then become the first set of benchmark tasks, which you’ll test future onboarded annotators on. You’d give new annotators these tasks and expect the answers to be nearly the same, this way you can immediately score an annotator’s performance on the benchmark before allowing them to work on real tasks. Additionally you should periodically give benchmarks to annotators to test for continued adherence to the instructions.

Some data vendors operate mostly flat hierarchies for their projects. At micro1 we’ve learned that placing annotators and their reviewers into a single team or “pod” has several benefits:

  • People know who their manager is.
  • Pod Leads feel responsible for upskilling their annotators.
  • Annotators feel more comfortable asking for clarification within their pods.
  • Since everyone in the pod has the same specialization, pods can stay together between projects.
  • Annotators can take feedback from their pod’s reviewers more constructively.

At micro1, pods are composed of 10 annotators, 5 reviewers, and a pod lead. This pod lead may either be a senior reviewer or a member of micro1’s Client Success team. Pod sizes can also vary, but a good rule of thumb is that the pod lead should be able to recall the name of every annotator in their pod. If the pod is too large, the pod lead may not be able to efficiently manage and track the progress of their annotators.

While research teams don’t need to be familiar with their data vendor’s internal team structures, they should be aware that different structures can lead to different incentives and project outcomes and they should feel comfortable inquiring about this when selecting a data vendor.

Project InstructionsPipelines