[Elsnet-list] GIVE-2: The Second NLG Challenge on Generating Instructions in Virtual Environments

Alexander Koller koller at mmci.uni-saarland.de
Fri Aug 14 01:57:20 CEST 2009

Second NLG Challenge on Generating Instructions in Virtual Environments

                Second Call for NLG System Submissions

		  Part of Generation Challenges 2010
	       Endorsed by SIGGEN, SIGDIAL, and SIGSEM


We invite contributions to the second round of the Challenge on
Generating Instructions in Virtual Environments (GIVE-2). See below
for details of the challenge and a timeline, or check our website.

You can try out a prototype of the GIVE-2 client with a
proof-of-concept NLG system here: http://www.give-challenge.org/give2


The Challenge on Generating Instructions in Virtual Environments
(GIVE) is a novel approach to the notoriously hard problem of
evaluating NLG systems. In this scenario, a human user performs a
"treasure hunt" task in a virtual 3D environment. The NLG system's job
is to generate, in real time, a sequence of natural-language
instructions that will help the user perform this task. The crucial
thing is that users connect to the generation systems over the
Internet. By logging how well they were able to follow the system's
instructions, we can evaluate the quality of these instructions in
terms of task completion rates and times, subjective measures such as
helpfulness and friendliness, and runtime performance. Because the
user and the system don't need to be physically in the same place,
access to experimental subjects over the Internet becomes easy.

GIVE is a theory-neutral, end-to-end evaluation effort for NLG
systems. It involves research opportunities in text planning, sentence
planning, realization, and situated communication. One particularly
interesting aspect of situating the generation problem in a virtual
environment is that spatial and relational expressions play a bigger
role than in other NLG tasks. Beyond NLG, GIVE can be interesting as a
testbed for improving the NLG components of dialogue systems, and for
computational semanticists working on spatial language.


In the GIVE-1 Challenge, which we ran last year, five NLG systems were
evaluated using data from almost 1200 game runs. To our knowledge,
this made GIVE-1 the largest ever NLG evaluation effort in terms of
the number of experimental subjects. We presented the results of the
evaluation at the ENLG Workshop, and have verified that these results
are consistent with (but more detailed than) the results that could be
obtained from a traditional lab-based evaluation.

Now we invite you to develop a GIVE NLG system and participate in
GIVE-2. The main novelty in GIVE-2 is that where GIVE-1 used discrete
worlds (which were based on square tiles, and the user could only jump
from the center of one tile to the center of the next, and turn in 90
degree steps), GIVE-2 will permit free, continuous movements in the
worlds. This will make the generation task more challenging because
simple instructions of the form "walk three steps forward" are no
longer possible.

Anyone is invited to submit an NLG system to participate in the GIVE-2
Challenge. We particularly invite contributions from students and
student teams. All participating systems will be evaluated, and the
results will be presented at INLG 2010. To get an idea of what this
involves, you may want to go to the GIVE website mentioned above and
take a look at our EACL 2009 demo paper describing the software
architecture, or download the GIVE-1 software and look at it in more


We plan to release a first version of the GIVE-2 software package by
the end of this month. You will then have time until February to
develop your systems. From February to April, we will run the
evaluation; at this time, users will connect to the NLG systems over
the Internet, through a central matchmaking service which we will
provide. You will then have the opportunity to write a research report
about your system, which will be presented together with the
evaluation results at the Generation Challenges session of the INLG
conference in July 2010.

Important dates:

August 2009            materials distributed to participants
February-April 2010    evaluation period
July 2010              presentation of results at INLG

If you are interested in participating, please let us know as soon as
you can at koller at mmci.uni-saarland.de. We look forward to hearing
from you!


We plan to repeat GIVE-2 with minor modifications in 2010-11. If you
happen not to have time or available students this summer, this means
you get a second chance to participate. In addition, GIVE-2
participants will have an opportunity to improve their systems based
on their experiences in the GIVE-2 evaluation.

Organizing committee

Donna Byron, Northeastern University
Justine Cassell, Northwestern University
Robert Dale, Macquarie University
Alexander Koller, Saarland University
Johanna Moore, University of Edinburgh
Jon Oberlander, University of Edinburgh
Kristina Striegnitz, Union College

More information about the Elsnet-list mailing list