Amongst many data vendors there are periods of crunch time as project deadlines approach. Unfortunately you can expect the task quality during these time periods to potentially go down as there may be extra resources pulled into the project, who will be at the bottom of the learning curve. If you wish to avoid this happening to your project, we encourage you to set up a dashboard with your data vendor on overall project progress and man-hours on the project to see steady progress.

Additionally, you should have access to the demographics of the annotators working on your project. While data vendors run background checks on their annotators, there are some regions that experience elevated rates of fraud through identity theft or account sharing. If your team‘s nature of work requires elevated steps towards protecting the confidentiality of the tasks, then it would be prudent to restrict your project from being worked on from within these regions.