[Elsnet-list] Recognising Textual Entailment (RTE) Challenge: 2nd Call for Participation and Schedule Change

Oren Glickman oren at glickman.com
Thu Sep 2 17:20:42 CEST 2004


***************************************************************
           Recognising Textual Entailment (RTE) Challenge
           A Common Benchmark for Semantic Inference

         2nd Call for Participation and Schedule Change

          http://www.pascal-network.org/Challenges/RTE/
***************************************************************

Note: The RTE Challenge schedule has been shifted by a few months
due to schedule changes in the general PASCAL Challenges Program.

Recent  years  have  seen  a  surge  in  research  of   text
processing   applications  that  perform   semantic-oriented
inference   about   concrete   text   meanings   and   their
relationships.  Even though many applications  face  similar
underlying  semantic  problems, these problems  are  usually
addressed  in  an application-oriented manner.  Consequently
it  is  difficult  to  compare, under a  generic  evaluation
framework,  semantic  methods  that  were  developed  within
different  applications. The PASCAL RTE Challenge introduces
textual entailment as a common task and evaluation framework
for  Natural Language Processing, Information Retrieval  and
Machine Learning researchers.

Textual  entailment  recognition is the  task  of  deciding,
given two text fragments, whether the meaning of one text is
entailed (can be inferred) from the other text. Participants
in  the  evaluation exercise will be provided with pairs  of
small  text  snippets,  which we term Text-Hypothesis  (T-H)
pairs.   The entire data set includes over 1000 English  T-H
pairs  from  the news domain (political, economical,  etc.),
split  into  development  and test subsets.   Examples  were
manually tagged for entailment (i.e. whether T entails H  or
not) by human annotators. Participating systems will have to
decide for each T-H pair whether T indeed entails H or  not,
and results will be compared to the manual gold standard.

The  dataset  was collected with respect to  different  text
processing application settings, such as question answering,
information   extraction,  information   retrieval,   multi-
document summarization, paraphrase acquisition, and  machine
translation. Each portion of the dataset includes typical T-
H  examples that correspond to success and failure cases  of
such  applications.  The examples represent different levels
of   entailment  reasoning,  such  as  lexical,   syntactic,
morphological and logical.

The  goal of the challenge is to provide a first opportunity
for   presenting  and  comparing  possible  approaches   for
modeling  textual entailment. In this spirit, we aim  at  an
explorative   rather  than  a  competitive  setting.   While
participant results will be reported there will  not  be  an
official  ranking of systems. Partial coverage  submissions,
for  systems  that handle only specific parts  of  the  test
examples,  are allowed. The development subset  is  released
first  to give an impression of the different types of  test
examples,     but    systems    are    likely     to     use
external/unsupervised knowledge resources.


Examples
-----------------------------------------------------------
TRUE

C&D Technologies announced that it has closed the acquisition
of Datel, Inc.

C&D Technologies acquired Datel Inc.
-----------------------------------------------------------
FALSE

The National Institute for Psychobiology in Israel was 
established in May 1971 as the Israel Center for 
Psychobiology by Prof. Joel.

Israel was established in May 1971.
-----------------------------------------------------------
FALSE

Reagan announced he had Alzheimer's disease, an incurable 
brain affliction.

Reagan died of pneumonia complicated by Alzheimer's disease
-----------------------------------------------------------
TRUE 

Eyeing the huge market potential, currently led by Google, 
Yahoo took over search company Overture Services Inc last year.

Yahoo bought Overture.
-----------------------------------------------------------

Important Dates (may be subject to change)
Full Development Set Available   August 30, 2004
Release of Test Set              Dec 15, 2004
Deadline  for Participants'      
Results Submissions              February 15, 2005
Release of individual Results	   February 25, 2005

Submission of Participants' 
Reports                          March 11, 2005
PASCAL Challenges Workshop       
(in Southampton, U.K.)           11-13  April  2005

Organizing Committee
Ido Dagan, Oren Glickman (Bar Ilan University, Israel)
Bernardo   Magnini   (ITC-irst,  Istituto   per   la   Ricerca
Scientifica e Tecnologica, Italy)

For  registration, further information and inquiries  -  visit
the challenge web site
(http://www.pascal-network.org/Challenges/RTE/)   or   contact
Oren Glickman <glikmao at cs.biu.ac.il>.



More information about the Elsnet-list mailing list