The Entity Ranking track results need assessments of wikipedia articles (corresponding to entities) at the document level. The evaluation of both the list completion and entity ranking tasks will be based on the same set of judgments. Assessments are binary, i.e., an entity is either relevant or not to a topic. Each entity is considered independently, i.e., the relevancy is independent from all other entities.
Consistency while judging is very very important. One person, preferably the topic author, should assess all candidates for one topic.
The judging interface opens with the list of topics to be assessed.
Select a topic by clicking on its JUDGE link, and the list of entities is opened.
The topic title is shown above the list of entities (with its expected entity type). The complete topic text is shown at the bottom right. Only a subset of all documents to assess is shown. You use the prev and next buttons at top and bottom of the list to see more documents.
When you click on an entity identifier, the bottom right pane shows the contents of the corresponding document as it is stored in the collection, at the top right you make the decision for the document's relevance. Instead of directly selecting an entity identifier, you can click FIRST link at the top of the list to go to the first unassessed entity.
Enter your decision after reading the document, by selecting the appropriate radio-button.
The pools will contain documents that do not correspond to an entity but that are on topic. Of course, such documents should be labelled non-relevant (we are not evaluating document retrieval!).
Click NEXT (a link below the radio-buttons) to proceed to the next document, and repeat this until you reach the final document. You may stop your session at any time and resume later, all assessments are stored at all times.
Keep in mind that each entity should be judged independently from all others. As a consequence, in the (not so likely) case that an entity would have more than one corresponding page in the collection, both are equally relevant. Documents that have been assessed already will be highlighted in the document list, a green background indicates marked relevant, a red background indicates marked not relevant. If you change your mind on the relevance of the document, the assessment can be changed using the radio-buttons.
You may use the comment boxes for your convenience, e.g., to register a particular interpretation of the topic. There is a comment box for comments relating to the topic in general (below the document list) and a comment box for a particular topic-document pair (or judgement) to the right of the judgement radio buttons.
Internet explorer does not support the absolute positioning of elements that is used in the judging interface. We recommend to use Firefox or Opera. The judging interface has been tested with firefox 2.0 under Linux and Windows Vista.