We adopt a hybrid crowdsourced labeling strategy using heterogeneous annotators with varying expertise and cost, from laypersons to MDs. Annotators were first tasked with marking text spans that described the respective PICO elements. Identified spans were subsequently annotated in greater detail: this entailed finer-grained labeling of PICO elements and mapping these onto a normalized vocabulary, and indicating redundancy in the mentions of PICO elements.
We hired three medical professionals via Upwork, an online platform for hiring skilled freelancers. After reviewing several dozen suggested profiles, we selected three workers that had the following characteristics: Advanced medical training (the majority of hired workers were Medical Doctors, the one exception being a fourth-year medical student); Strong technical reading and writing skills; And an interest in medical research.
This is an example document annotated by our experts with all three PICO phases. To browse the complete set of test annotations, you can view the document collection here.
In the second phase of annotation, the previous spans were further annotated with more detailed labels as shown below. Again, all annotations are browsable here. If you are interested in one PICO element specifically, you can view only participants, interventions, or outcomes.