[Elsnet-list] ResPubliQA 2009:Call for Participation and Final
forner at celct.it
Tue Feb 3 11:45:48 CET 2009
Apologies for multiple posting
ResPubliQA exercise at the
MULTILINGUAL QUESTION ANSWERING TRACK AT CLEF 2009
NEW: The Final Guidelines of the ResPubliQA exercize are now available on the website!
Call for Participation
We are glad to announce that a NEW EXERCISE will be proposed this year within the Question Answering track at the Cross Language Evaluation Forum (CLEF).
For more information and updates visit the new ResPubliQA website at:
We invite participation, both from academic institutions and industrial organizations, on this new task.
Guidelines describing the task are distributed among the participants and are downloadable from the ResPubliQA website. http://celct.isti.cnr.it/ResPubliQA/Documents
Training data have also been made available to participants in order to test the systems with procedures to be used in the formal evaluation campaign. http://celct.isti.cnr.it/ResPubliQA/Downloads
The results of the evaluation will be disseminated at the final workshop which will be organized in Corfu in conjunction with ECDL 2009.
ResPubliQA 2009: TASK OVERVIEW
TASK DESCRIPTION: Systems receive natural language questions as input, and must return one paragraph containing the answer from the document collection. No exact answer is required neither multiple responses.
DOCUMENT COLLECTION: The subset of JRC-Acquis documents that have parallel aligned translations into all the languages involved will be used, namely Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian and Spanish. The sub-collection is available at the ResPubliQA website http://celct.isti.cnr.it/ResPubliQA/Downloads
QUESTIONS: a pool of 500 independent questions (factoid, definition, reason, purpose and procedure) is provided
o NO LIST questions
o NO topic related questions (questions linked to the same topic)
o NO NIL questions
ANSWERS: one of the following two responses must be returned
a) one single paragraph containing the candidate answer. Multi-paragraph answers are not considered in this task
b) the string NOA to indicate that the system prefers not to answer the question.
Systems that give no answers (NOA) instead of wrong answers will be rewarded by the evaluation measure. Answer Validation techniques (including Machine Learning) are expected to be used for taking this final decision.
LANGUAGES INVOLVED: Basque (EU), Bulgarian (BG), Dutch (NL), English (EN), French (FR), German (DE), Italian (IT), Portuguese (PT), Romanian (RO) and Spanish (ES).
A monolingual English (EN) task will also be activated this year, as both the exercize and the collection are different from TREC.
Basque has been included exclusively as a source language, as there is no Basque collection available.
Registration Open: February 4, 2009
Final Track Guidelines: February 2, 2009
Test Sets Release: May 25, 2009
Submissions of Runs by Participants: June 5, 2009
Release of Individual Results: from July 15, 2009
Submission of Papers for Working Notes: August 14, 2009
CLEF Workshop (in Corfu, Greece): 30 September - 02 October 2009
The participants will have 5 DAYS to upload their submissions, starting from the moment when the questions are downloaded, and not later than June 5, 2009.
TRACK COORDINATORS AND ORGANIZERS
- UNED (coordinator)
Spanish Distance Learning University, Spain
- CELCT (coordinator)
Center for the Evaluation of Language and Communication Technology, Italy
Pamela Forner and Danilo Giampiccolo
Evaluations and Language Resources Distribution Agency, France
- University of Limerick, Ireland
Bulgarian Academy of Science, Bulgaria
- UAIC and RACAI, Romania
Alexandru Ioan Cuza University and Romanian Academy Research Institute for Artificial Intelligence, Romania
University of Basque Country, Spain
- Donna Harman (National Institute for Standards and Technology (NIST), USA)
- Maarten de Rijke (University of Amsterdam, The Netherlands)
- Dominique Laurent (Synapse Développement, France.)
CELCT (web: www.celct.it<http://www.celct.it/>)
Center for the Evaluation of Language and Communication Technologies
Via alla Cascata 56/c
38100 Povo - TRENTO -Italy
email: forner at celct.it<mailto:forner at celct.it>
tel.: +39 0461 314 804
fax: +39 0461 314 846
Secretary Phone: +39 0461 314 870
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Elsnet-list