Project Instructions

After sourcing and vetting annotators, providing clear instructions is crucial for project success, impacting cost, speed, and data quality. At micro1, we develop project instructions before production and rigorously test and refine them with each new added person to the pilot, avoiding changes but allowing clarifications. Updating instructions requires reviewing all previous tasks for compliance. When paying vendors per task, they typically include a mark-up for potential changes, while hourly plus percentage fees reveal the direct costs of instruction updates. We highly recommend maintaining a dated changelog to ensure annotators and teams stay informed of any changes.

Research teams should be specific on what is unsafe or inappropriate language rather than relying on annotators to make these decisions. Annotator workforces come from many countries with very different cultures and cultural norms. What may be considered taboo or inappropriate to talk about in one culture is completely fine or even encouraged in another. As such you need to make sure everyone is aligned on guidelines and opinions. You may see an example of this below.

A strong review pipeline is one way to ensure consensus amongst your annotators, but we found two other cost-effective ways to accomplish this:

1. If you are fine-tuning a model for consumption in the US & Canada, then you should staff the project with talent from the US & Canada or culturally similar locations (your data vendor, like micro1, can help with this)

2. You need to be very explicit about what constitutes a taboo topic or undesired behavior for an AI model. This could include defining sensitive or off-limits subjects, such as violence, illegal activities, or explicit content. Additionally, you should clarify behavioral boundaries, such as avoiding biased or discriminatory language, refraining from providing harmful advice, or steering clear of promoting unethical practices. By explicitly outlining these constraints, you ensure the AI operates within the desired ethical and operational parameters.

Otherwise, you’ll have poor post-training results from inconsistent human data, or even worse, your AI model will have ethics that are very different from the end users of your AI model which can severely impact the utility of your model.

At micro1 in order to help annotators get familiar with the platform, we build an FAQ for non-project specific matters. We found that on our biggest project, with 800+ micro1 sourced developers working everyday, our Client Success Managers became overwhelmed with the number of queries from our developers.

To solve this, we developed an evolution of question-answer bots we dubbed RAG 2.0, which features better information retrieval techniques and it only replies when it’s confident in the response. We then deployed this AI bot in our slack team channel to provide immediate support to our annotators. We wouldn’t recommend having a bot answer project instruction questions as a model hallucination could be costly and all annotators should read the instructions line-by-line. This is a good balance between automation and increased productivity while minimizing potential mistakes.

If you’d like to read more about our AI bot which made each dev success manager 2x faster, you can view our blog post linked below.