[Elsnet-list] GREC Shared Tasks 2009 (NLG/Summarisation)

Gatt, A. a.gatt at abdn.ac.uk
Mon Sep 15 16:17:34 CEST 2008



Part of Generation Challenges 2009, in conjunction with ENLG 2009.

Generation Challenges 2009 is being organised to provide a common forum for a number of different NLG Shared Tasks (see http://www.nltg.brighton.ac.uk/research/genchal09/).

As part of Generation Challenges 2009, we are organising two GREC Shared Task Competitions.  The first is the GREC-MSR (Main Subject
References) Task which uses the GREC-2.0 Corpus of 2,000 Wikipedia introduction sections where references to the main subject of the Wikipedia article have been annotated, and the task is to develop a system that can select (from a given list) an MSR that is appropriate in the context.  The second is the GREC-NEG (Named Entity Generation) Task which uses the new GREC-People Corpus of 1,000 Wikipedia introduction sections about people in which single and plural references to all people mentioned in the text have been annotated. The task in GREC-NEG is to select appropriate referential expressions for all mentions (singular and plural) of people.

Submissions to both tasks will be evaluated using a range of intrinsic and extrinsic measures, some assessed automatically, some manually.
Submitted systems and evaluation results will be presented in a special session at ENLG'09, and published in the ENLG'09 proceedings.

1. Background

There has been increasing interest recently among text summarisation researchers in postprocessing techniques to improve the referential clarity and coherence of extractive summaries, and among language generation researchers in generating referential expressions in context.  The GREC tasks are aimed at researchers in both of these groups, and the objective is the development of methods for generating chains of referential expressions for discourse entities in the context of a written discourse, as is useful for postprocessing extractive summaries and repeatedly edited texts (such as Wikipedia articles).

2. Data

The GREC data resources consist of introduction sections collected from Wikipedia articles in which three broad syntactic categories of overt reference to named entities have been annotated: subject NPs, object NPs and genitive subject-determiners (such as "Faraday's" in "Faraday's law of induction"). The annotations include features encoding basic syntactic and semantic information.

The GREC-2.0 corpus consists of 2,000 texts in five different domains (cities, countries, rivers, people and mountains).  In this corpus, only references to the single entity that is the main subject of a Wikipedia article (e.g. "Michael Faraday") have been annotated.

The new GREC-People corpus consists of 1,000 texts in just one domain, people. Here, all references to all people mentioned in a text have been annotated.  GREC-People therefore includes explicit coreference annotation for one or more coreference chains (whereas in GREC-2.0 texts there is always just one annotated coreference chain).

For GREC-2.0 and GREC-People we have test sets of 200 and 100 texts, respectively, where referential expressions have been selected by participants in an elicitation experiment. In these test sets, there are three versions for each corpus text, in each of which the referential expressions have been manually selected by a single participant in the experiment.

3. The GREC'09 Tasks

The GREC-MSR Task has the same task definition as the GREC shared task at REG'08.  Participating systems need to select the referential expression (RE) from a given set of alternatives that is most appropriate in the given context, which may involve e.g. ensuring that pronouns can be resolved.  Systems will be evaluated both against the REs in the corpus and against human-selected topline solutions for this task.  Results and descriptions of participating systems from the
REG'08 run of this task can be found here:

The new GREC-NEG Task is an extension of GREC-MSR in that it requires participating systems to select appropriate referential expressions for all discourse entities of the same type (people in this round) as the main subject of the article.

4. Evaluation

For both tasks, the data will be randomly divided into training, development and test data.  Participants will compute evaluation scores on the development set (using code provided by the organisers), and the organisers will perform evaluations on the test data set.

We will use a range of different evaluation methods, including intrinsic and extrinsic, automatically assessed and human-evaluated.
The intrinsic methods will include string-accuracy, feature-accuracy and string-similarity measures, as well as human-produced quality assessments.  The extrinsic methods will include a reading/comprehension experiment and measuring coreference resolver success (for details about the previous edition, see http://www.aclweb.org/anthology/W/W08/W08-1127.pdf).

Full details of the evaluation methods for GREC'09 will be given in the Participants' Pack that will be distributed to registered participants.

6. Participation

Registration is now open at the GREC'09 homepage (http://www.nltg.brighton.ac.uk/research/genchal09/grec).  Once registered, participants in the GREC-MSR Task will receive the complete training and development set, evaluation software and detailed documentation (collectively known as the Participants' Pack) for this task.  Participants in GREC-NEG will first receive a sample of the training and development data, to enable them to start building systems; they will receive the complete Participants' Pack for GREC-NEG by the end of September 2008.

7. Proceedings and Presentations

The Generation Challenges 2009 meeting will be held as a special session at ENLG 2009. The session will include overviews of all the shared tasks, including the GREC'09 Tasks. The participating systems will additionally be presented as papers in the ENLG'09 proceedings, and as posters during the ENLG'09 poster session.

GREC'09 papers will not undergo a selection procedure with multiple reviews, but the organisers reserve the right to reject material which is not appropriate given the participation guidelines.

8. Important Dates

Sep 05, 2008   First Call for Participation in GREC'09 Tasks; GREC'09
               sample data sets available Jan 01-31, 08  GREC'09 test data
               1. submit system report;
               2. download test data;
               3. submit outputs within 48h.
Jan 31, 2009   Final deadline for submission of GREC'09 test data outputs
Feb 01-28, 09  GREC'09 Evaluation period Mar 30-31, 09  Generation                         Challenges 2009 meeting at ENLG'09 (date to be

9. Organisation

Albert Gatt, Computing Science, University of Aberdeen, UK
Anja Belz, NLTG, University of Brighton, UK
Eric Kow, NLTG, University of Brighton, UK
Jette Viethen, Macquarie University, Australia

GREC'09 homepage: http://www.nltg.brighton.ac.uk/research/genchal09/grec
Generation Challenges homepage: http://www.nltg.brighton.ac.uk/research/genchal09
Generation Challenges email: nlg-stec at itri.brighton.ac.uk

The University of Aberdeen is a charity registered in Scotland, No SC013683.

More information about the Elsnet-list mailing list