Statistics and Fraud

It’s a given that in any system with large amounts of global contractors working asynchronously there is the potential for some form of fraud.

In this section we will introduce you to the features that micro1 has implemented in order to minimize all forms of fraud.

We will refer to a task within a certain stage in a pipeline as just a task. Tasks typically go through multiple layers and annotators. Annotators that review another annotator’s work are called reviewers. Depending on project requirements, there are senior reviewers that review the reviews.

Every task in a batch/project should take around the same time to complete. You can find the mean time to complete for a batch, and from there calculate the Z score for a given task done by an annotator. Finding an annotator’s mean Z score for tasks completed by them provides very accurate insight into how fast they work across projects and disciplines. Annotators with a mean Z score above 1.5 are usually either perfectionists, multi-tasking, or committing time theft. Data vendors should provide tools to project managers to automatically calculate these statistics so the project manager can compare their task quality to their time spent on tasks. If you identify a perfectionist through these means, from our experience we strongly encourage teams to promote these annotators to help in the Quality Assurance portions of the project. These people are uncommon but make an outsized impact in moving projects forward. This system of directly comparing time and performance is completely non-intrusive.

Some data vendors use platforms that track annotator’s actions on the platform such as mouse clicks, cursor movement, keyboard presses. While this is not visibly intrusive, it is not entirely reliable as this would be unable to capture activity in messaging applications or conference calls. Additionally depending on the project, you might need annotators to access resources hosted outside the platform. At micro1 we recommend using time tracking software such as HubStaff to track hours and activity throughout all resources annotators might use, and to not rely solely on client-side logged activity on the data platform. HubStaff would be able to track activity on third party websites and tools such as Slack.

As mentioned previously when discussing starting a new project, instructions should be used by experienced annotators to complete the first set of tasks. These tasks are then referred to as benchmark tasks or assessments. If project instructions are clear enough, then most non-open ended questions in a task should be deterministic. You can then give these tasks to annotators new to the project to see how far they deviate from these benchmarks. As these tasks represent the golden standard for high quality highly adherent work, your research team should cooperate with the data vendor to ensure the benchmarks exceed expectations.

Your data vendor’s platform should offer continuous vetting with discreet benchmarks. This ensures annotators adhere to project instructions not just in the beginning but during any moment of the project This can instantly detect a drop in quality. If there is account sharing going on, this can be a significant way to detect and combat fraud early.