[Elsnet-list] Companions Project Newsletter

Yorick Wilks Yorick at dcs.shef.ac.uk
Thu Jan 31 11:19:13 CET 2008

Dear Colleague:
Welcome to the first newsletter of the COMPANIONS project: a 12.88m Euro
interdisciplinary research project which focuses on combining advanced
technologies to create personal, persistent 'agents' or 'Companions'.  
will be an agent or 'presence' that communicates and develops a  
with its user primarily by using and understanding speech.

This version is in text, with appropriate web links but you can  
download the Newsletter as pdf (316 kb):

You can go to our website to find out more:

Please forgive any duplication of this in your mailbox.
Best wishes,
Professor Yorick Wilks

1. What are Companions?
2. Demonstrators: The Health and Fitness Companion
3. Demonstrators: Introducing the PhotoPal WoZ Platform
4. Fourth Bellagio International Workshop on Human-Computer Conversation
5. Scientists Discuss Artificial Companions
6. Feature: Nabaztag as a first Companion
7. Our Industrial Partners
8. Scientific Governing Council
9. Contact

1. What are Companions?

COMPANIONS is a new paradigm for the way people deal with the  
Internet, as
computational agents that are personalized and persistent, sensitive  
to the
needs of, and relationship with, the single owner. The embodiment of a
Companion is relatively unimportant: it could be a screen head, a mobile
phone, or some simple object, easy to carry about, like a handbag,  
The key
thing is talk: the Companion, whatever its size or shape, will be a
conversational entity, interacting with its owner over long periods.  
It is
an ECA (Embodied Conversational Agent) but with an emphasis on the
conversation rather than the graphical forms. That is what the project's
first trial demonstrators are aiming at, however simplified they turn  
out to

The Companion vision is that a Companion becomes, in a precise sense,  
of the user's memory on the web, essentially their memory of  
themselves and
their life events. The originality here is the use of conversation as  
a tool
of reminiscence for users who will already have much of their life's  
data in
digital form, such as images, texts and videos. The Companion is  
there to
give that data a narrative form, a life story, for the benefit of the  
and their successors. In the Senior Companion of course the aim is  
also to
divert, entertain, instruct and help.

A crucial component in the Companion vision is the relationship of the
Companion, not only to its owner, but to the Companions of other  
users; as
well as the relationship of users to each other mediated by their
Companions. Users will be likely to have revealed details of themselves,
their financial intentions, their health etc. to their Companion they  
not want shared or gossiped about. Companions will need to show trust by
understanding different levels and types of information entrusted to  
and what can be done with it: how far, when making a date with  
another user
via his or her Companion, it can reveal relevant and useful things  
('I think
my owner likes yours!'). Companions must therefore know how they are
affecting the moods and emotions of their users and how far they are  
what they were intended to do, and not more; they must be able to assess
both their owner's state and their own capacities.


2. Demonstrators: The Health and Fitness Companion

The Companions Demonstrators will show off the capabilities of the
technology, and a number of components or modules that future  
research and
products can build on to create a variety of Companions.

The Health and Fitness Companion supports a healthy lifestyle by  
records of a user's health-related information (eating, physical  
over a long time.

In the first prototype we have focused on dialogues where the user  
gives the
Companion background information, such as current exercise and eating
habits. Based on this, the Companion will provide the user  
information on
healthier eating habits (including how to cook healthier food), and how
these relate to various fitness activities. Typically, the user and the
Companion make an overall plan for the whole day, and there can be more
detailed conversations during the day. The first  prototype is  
capable of
having a dialogue with the user covering all daily meals. The following
dialogue excerpt gives an example of the current system:

Markku: <Touches the Companion with the RFID-tag>
Companion: Hi Markku! How are you doing today?
Markku: I'm fine, thank you.
Companion: How about taking a look at your eating habits?
Markku: Sure, why not.
Companion: Would you say yesterday was an average day?
Markku: Yes, I guess so.
Companion: What time did you wake up?
Markku: At seven o'clock
Companion: Do you usually wake up around seven in the morning?
Markku: No, usually around 8 in the morning.
Companion: What did you eat and drink between getting up and going to  
Markku: Some coffee and juice
Companion: Did you eat something between getting up and going to work?
Markku: Yes, I had toast
Companion: Did you have milk or sugar in your coffee?
Markku: Yes, both milk and sugar

The current implementation uses a unique combination of spoken dialogue
system technology and physical Nabaztag/tag rabbits. In future  
cooking and physical exercises will be incorporated, including a mobile
Companion that provides feedback and advice during physical  
activities, such
as jogging exercises.


Watch the demonstrator on YouTube:

3. Demonstrators: Introducing the PhotoPal WoZ Platform

In order to explore how people talk about their digital photos, the  
of events, of places, of memories, the Companions project have  
developed a
series of Wizard of Oz (WoZ) platforms.

The notion of the WoZ approach is that to explore advanced  
technologies that
are beyond the reach of the researcher (through barriers of cost,  
feasibility etc) we can replace the technology with a human 'wizard'.  
wizard then allows the unknowing participant to experience, describe and
interact with the technological functionality with which the  
researcher is

The Companions WoZ examples allow a participant to discuss their  
photos with
an on screen avatar, their PhotoPal. The set up enables people to  
engage in
an experience which is not yet technically possible, namely to talk  
and naturally about their photos to what they believe to be an  
computer system. The value of this approach is twofold:

1. On an interaction design level it is possible to investigate how  
best to
develop elements such as interface, avatar design, aesthetics and  
2. The sessions provide a hugely rich source of data for a variety of
dialogue corpora, thus aiding the speech work which will make such an
interaction a reality.


4. Fourth Bellagio International Workshop on Human-Computer Conversation

This workshop brings together academic researchers and industrialists
concerned with all aspects of human-computer conversation and the  
research issues of emotion, relationships, companionship, embodiment,  
memory, evaluation etc. that persistent, personalised computational  
will need solutions for if they are to exist, as they will, in the
reasonably near future.

Earlier workshops in this series were highly successful at getting  
groups (as well as university and independent researchers) to display  
interests and prototypes.

Since 2000, there has been a great upsurge in research funding, and  
aim of the meeting will be to bring together members of currently active
research consortia in the EU and US, such as CALLAS, CALO, COSY, AMIDA,
INDIGO, SOPRANO and others.

Calls for papers, lists of invited speakers and the program  
committee, will
be sent out nearer the time.

Location: Grand Hotel Villa Serbelloni, Bellagio, Italy
Date: 6-7 October, 2008
Register your interest: events at oii.ox.ac.uk
Details: http://www.companions-project.org/events/

5. Scientists Discuss Artificial Companions

The Oxford Internet Institute, a Companions project partner, hosted a
workshop meeting of scientists from Europe and the USA to discuss the  
of 'Artificial Companions in Society' in Oxford on October 26th, 2007.
Research into such characters is fast expanding and the general  
is one of a future in which people will become emotionally attached  
to these
companions and will rely on them in various ways, such as storing and
accessing their personal memories on the Internet.

The workshop was preceded by a lecture by Sherry Turkle, Abby  
Mauze Professor of the Social Studies of Science at MIT and Technology
Director, MIT Initiative on Technology and Self. Her lecture, entitled
'Cyberintimacies/Cybersolitudes', asked questions such as 'What kinds of
relationships are appropriate to have with machines?'

Papers presented at the workshop, on speakers' perspectives on the  
and future of this emerging science,  included titles such as:
'Conversationalists, maybe - But Confidants?' (by Margaret Boden);  
in Love with a Companion' (David Levy); 'Are Artificial Companions  
Than Real Ones?' (Joanie Gillispie); 'A Victorian Companion?' (Yorick  
and  'Robots Should be Slaves' (Joanna Bryson) (the Position papers are
available on the Companions website).

The event was organized by Professor Yorick Wilks, Director of the EU- 
Companions project, on behalf of the e-Horizons Institute, a unit of the
James Martin School of the 21st Century; the workshop was also  
supported by
Microsoft Research.

Companions events:

6. Feature: Nabaztag as a first Companion

The Nabaztag rabbit is being used as the first Companion avatar in two
initial demonstrators in the project. In the Health and Fitness  
being integrated at the University of Tampere, Nabaztag is the free- 
rabbit available commercially from Violet in Paris, adapted with an  
API so
as to receive speech input as well as output. In the Senior  
Companion, being
integrated at the University of Sheffield, Nabaztag is appearing as a
talking 2D screen rabbit, discussing the content of personal  
is the brain-child of Rafi Halidjan (the name is the Armenian for  
It is a plastic rabbit that can move its ears, flash colours and  
speak. Its
input is entirely wifi, driven from a remote website. The standard  
use is to
convey messages and emotions, all input at a website, to a remote  
friend or
lover: it can flash blue with ears down and say or sing 'I miss you', or
indeed anything else you type in. Rafi Halidjan is on the Companions
Governing Council and we hope for a closer relationship in the future as
Nabaztag has much of the attractive, distinctive and expressive  
quality one
wants in a physical embodiment of the Companion idea: especially as the
Tampere version can now listen as well as talk!

7. Our Industrial Partners

The automatic speech recognizer (ASR) and the text-to-speech (TTS)  
integrated in the present Companions demonstrators have been provided by
Loquendo. While themes related to improving acoustic quality,  
and expressivity of speech generation are subjects of research done by
Loquendo in Companions, the ASR integrated in the demonstrators is the
Loquendo commercial product. Loquendo ASR can support speech  
that need accurate recognition of broad vocabularies, up to 1,000,000  
of words. The speech recognition engine is based on algorithms that
integrate neural networks and continuous density hidden Markov  
models. The
TTS pronunciation lexicon ensures that specialized vocabularies,
abbreviations, acronyms, and even regional pronunciation differences,  
sound as the speech application developer intends them to. The acoustic
characteristics of the voices (speech, speaking rate, and volume) can be
fine tuned and controlled at the application level.

One of the today's buzzwords among telco companies and a technology  
that can
significantly enable some of the future Companions features is IMS (IP
Multimedia Subsystem). This acts as the glue that will bring today's  
fixed and IP telephony into a convergent, unified entity. IMS will offer
significant advantages on concepts such as multidevice situations,
usability, session transferring, identity tracking, personalization,
security and others, all of which will undoubtedly be crucial in the  
Companions prototypes. The same Telefonica I+D team working on  
has recently collaborated in the development of scenarios and  
for one of these projects, involving many of the above concepts. Our  
presence and identity tracking demonstrator performs transference of
sessions between different devices such as PCs, smartphones and touch
panels, enabling users to perform complex tasks such as reading their
incoming mail and SMS/MMS, accessing the internet and granting access to
their homes. We hope to bring some of this technology in later stages  
of the
project to make the Companion not only able to recognise each user's
presence, but to follow her or him along using different devices.

As An Angel
As An Angel is a European SME, founded in 2001, based in Paris,  
in human-computer dialogue and agent-based multimodal interfaces. It has
produced a dozen conversational agents, prototypes and demonstrators for
advertising agencies, banks, and other advertisers. As An Angel is  
about to
launch an online automated generator of personified avatars, named
AngelStudio. This generator allows the user to develop his or her  
double in just a few minutes. This personified and learning avatar  
will be
able to achieve the following delegated actions such as representing the
user on his/her personal site or blog, meeting sites and forums, online
communities or virtual worlds and simulation games. The first  
application of
AngelStudio technology is the game SimDate, currently in its final  
stage of
development and due to launch online in January 2008. In this  
simulation of
a seduction dialogue, the players purpose is to seduce a character  
from a real person) through an instant messaging-like dialogue with a

The major undertaking of TeliaSonera has been the development of a  
management method suitable for automatic troubleshooting and other
problem-solving applications. The method has a theorem-proving  
flavour, in
that it recursively decomposes tasks into sequences of subtasks and  
actions. An explicit objective when designing the method has been  
that it
should be tailored for use by other people than the designers  
themselves.  A
pilot implementation in the domain of over-the-phone broadband  
support has
been developed and has been used for internal data collection in  
order to
validate the basic ideas and fine-tune the design. We are particularly
interested in trying to apply unsupervised machine learning methods,  
such as
reinforcement learning, to enable this kind of system to learn to  
its behaviour by itself.

The consortium:

8. Companions Scientific Governing Council

Chair: Professor Ricardo Baeza-Yates (Yahoo! Research, Barcelona,  
Members: Professor James Allen (University of Rochester, USA);  
Professor Dr Harry C.
Bunt (University of Tilburg, Netherlands); Dr Gregory Grefenstette (CEA,
France); Dr Rafi Haladjian (Violet, France); Dr David Levy (Intelligent
Systems, UK).

9. Contact

Professor Yorick Wilks
Computer Science Dept, Univ. of Sheffield, 211 Portobello Rd, Regent  
Sheffield S1 4DP, UK
Tel: +44 (0)114 2221804
Email: mailto:yorick at dcs.shef.ac.uk
Web: http://www.companions-project.org/

Companions is a European Commission Sixth Framework Programme  
Society Technologies Integrated Project (IST-34434)

More information about the Elsnet-list mailing list