\documentclass[letterpaper,twoside,11pt]{article}
\usepackage{jair}
\usepackage{theapa}
\input epsf
\usepackage[dvips]{graphicx}
\usepackage[latin1]{inputenc}

%\jairheading{15}{2001}{263-287}{3/01}{10/01}
\jairheading{18}{2003}{117-147}{7/02}{2/03}

\ShortHeadings{Translation of Pronominal Anaphora between English
and Spanish} {Peral \& Ferr\'andez}
\firstpageno{117}


\begin{document}

\title{Translation of Pronominal Anaphora between English and Spanish: Discrepancies and Evaluation}

\author{\name Jes\'us Peral \email jperal@dlsi.ua.es \\
        \addr Dept. Lenguajes y Sistemas Inform\'aticos\\
       University of Alicante\\
       Alicante, SPAIN
       \AND
       \name Antonio Ferr\'andez \email antonio@dlsi.ua.es \\
       \addr Dept. Lenguajes y Sistemas Inform\'aticos\\
       University of Alicante\\
       Alicante, SPAIN}


\maketitle



\begin{abstract}

This paper evaluates the different tasks carried
out in the translation of pronominal anaphora in a machine
translation (MT) system. The MT interlingua approach named AGIR
(\emph{Anaphora Generation with an Interlingua Representation})
improves upon other proposals presented to date because
it is able to translate intersentential anaphors, detect
co-reference chains, and translate Spanish zero pronouns into
English---issues hardly considered by other systems. The paper
presents the resolution and evaluation of these anaphora problems
in AGIR with the use of different kinds of knowledge (lexical,
morphological, syntactic, and semantic). The
translation of English and Spanish anaphoric third-person
personal pronouns (including Spanish zero pronouns) into the
target language has been evaluated on unrestricted corpora. We
have obtained a precision of 80.4\% and
84.8\% in the translation of Spanish and English pronouns,
respectively. Although we have only studied the Spanish and
English languages, our approach can be easily extended to other
languages such as Portuguese, Italian, or Japanese.

\end{abstract}


\section{Introduction}
The anaphora phenomenon can be considered one of the most
difficult problems in natural language processing
(NLP). The etymology of the term \emph{anaphora} originates with the Ancient
Greek word ``anaphora'' ($\alpha \nu \alpha \varphi o \rho \alpha$), which
is made up of the separate words, $\alpha \nu
\alpha$ (``back, upstream, back in an upward direction'') and
$\varphi o \rho \alpha$ (``the act of carrying''), and which denotes the
act of carrying back upstream.

Presently, various definitions of the term \emph{anaphora} exist,
but the same concept underlies all of them. Halliday \&
Hassan \citeyear{Halliday76} defined anaphora as ``the cohesion
(presupposition) which points back to some previous item.'' A
more formal definition was proposed by Hirst \citeyear{Hirst81}, which
defined anaphora as ``a device for making an abbreviated reference
(containing fewer bits of disambiguating information, rather than
being lexically or phonetically shorter) to some entity (or
entities) in the expectation that the receiver of the discourse
will be able to disabbreviate the reference and, thereby,
determine the identity of the entity.'' Hirst refers to the
entity as an \emph{anaphor}, and the entity to which it
refers is its \emph{antecedent}:



\begin{itemize}
\item \emph{[Mary]$_i$} went to the cinema on Thursday. \emph{She$_i$} didn't like the
film.
\end{itemize}

In this example, the pronoun \emph{she} is the anaphor and
the noun phrase \emph{Mary} is the antecedent. This type of anaphora is the
most common type, the so-called \emph{pronominal
anaphora}.

The anaphora phenomenon can be further broken down into two processes:
that of resolution and generation. ``Resolution'' refers to the process of
determining the antecedent of an anaphor; ``generation'' is the
process of creating references over a discourse entity.

In the context of machine translation, the resolution of
anaphoric expressions is of crucial importance in order to
translate/generate them correctly into the target language
\cite{Mitkov98}. Solving the anaphora and extracting the
antecedent are key issues for correct translation into the target
language. For instance, when translating into languages which
mark the gender of pronouns, resolution of the
anaphoric relation is essential. Unfortunately, the majority of MT systems do
not deal with anaphora resolution, and their successful operation
usually does not go beyond the sentence level.

We have employed a computational system that focuses on anaphora
resolution in order to improve MT quality and have then measured
the improvements. The SUPAR (\emph{Slot Unification Parser for
Anaphora Resolution}) system is presented in the work of
Ferrández, Palomar, \& Moreno \citeyear{Ferrandez99}. This system
can deal with several kinds of anaphora with good results. For
example, the system resolves pronominal anaphora in Spanish with
a precision rate of 76.8\% (Palomar et al., 2001); it resolves
one-anaphora in Spanish dialogues with a precision rate of 81.5\%
\cite{Palomar01b}, and it resolves definite descriptions in
Spanish direct anaphora and bridging references with precision
rates of 83.4\% and 63.3\%, respectively \cite{Munoz2000}. In the
work presented here, we have used an MT system exclusively for
pronominal anaphora resolution and translation. This kind of
anaphora is not usually taken into account by most of the MT
systems, and therefore pronouns are usually translated
incorrectly into the target language. Although we have focused on
pronominal anaphora, our approach can be easily extended to other
kinds of anaphora, such as one-anaphora or definite descriptions
previously resolved by the SUPAR system.

It is important to emphasize that in this work we only
resolve and translate personal pronouns in the third person whose
antecedents appear before the anaphor---that is, an anaphoric
relation between the pronoun and the antecedent is established,
and cataphoric relations (in which the antecedent appears after the anaphor)
are not taken into account.

This paper focuses on the evaluation of the different tasks
carried out in our approach that lead to the final task: the
translation of the pronominal anaphora into the target language.
The main contributions of this work are a presentation and
evaluation of the multilingual anaphora resolution module
(English and Spanish) and an exhaustive evaluation of the
pronominal anaphora translation between these languages.

The paper is organized as follows: Section 2 shows the
anaphora-resolution needs in MT and the deficiencies of
traditional MT systems to resolve this phenomenon conveniently.
Section 3 presents the analysis module of our approach. In Section
4, we identify and evaluate the NLP problems related to pronominal
anaphora resolved in our system. Section 5 presents the generation
module of the system. In Section 6, the generation module is
evaluated in order to measure the efficiency of our proposal.
Finally, we present our conclusions.

\section{Anaphora Resolution and its Importance in MT}
As noted earlier, anaphora resolution is of crucial importance in
order to translate anaphoric expressions correctly into a target
language. Let us consider the sentences \cite{Hutchins92}:

\begin{enumerate}
\item \emph{[The monkey]$_i$} ate the banana because \emph{it$_i$} was hungry.
\item The monkey ate \emph{[the banana]$_i$} because \emph{it$_i$} was ripe.
\item The monkey ate the banana because \emph{it} was tea-time.
\end{enumerate}

In each sentence the pronoun \emph{it} refers to something
different: in sentence (1), it refers to \emph{the monkey}, in
sentence (2) to \emph{the banana}, and in sentence (3), to the
abstract notion of time. If we wish to translate these sentences
into Spanish or German (languages which mark the gender of
pronouns), anaphora resolution will be absolutely essential
since, in these languages, pronouns take the gender of their
antecedents. Therefore, in Spanish, we would obtain the following
pronouns: (1) \emph{\'este} (in the masculine form since the
antecedent---\emph{the monkey}---is masculine), (2) \emph{\'esta}
(feminine---\emph{the banana}), and (3) an omitted pronoun (since
the second clause of the sentence is impersonal in Spanish and
does not need any subject). On the other hand, in German we would
obtain: (1) \emph{er} (masculine antecedent), (2) \emph{sie}
(feminine antecedent), and (3) \emph{es} (neutral).

Besides these problems, originated by the gender of anaphoric
expressions in different languages, there are other differences
(that we have named discrepancies) which influence the process of
translation of these expressions. These discrepancies have been
previously studied by other authors. Mitkov \& Schmidt
\citeyear{Mitkov98} present several problems to be taken into
account in the translation of pronominal anaphors between
different languages (German, French, English, Malay, and Korean);
Nakaiwa \& Ikeara \citeyear{Nakaiwa92} treat the problem of the
translation of elliptical constructions in a Japanese--English MT
system; and Mitkov et al. \citeyear{Mitkov94a} and Geldbach
\citeyear{Geldbach99} present the discrepancies in an
English--Korean MT system and a Russian--German MT system,
respectively.


Another difference between languages is that of number discrepancies,
in which certain nouns are referred to by a singular
pronoun in one language and by a plural noun in the
other. For example, the word \emph{people} is plural in English,
whereas in Spanish or German it is singular. Hence, in translations
from English to Spanish, or from English to German, the plural pronoun will become
a singular pronoun.

On the other hand, although in the majority of cases language-pairs
pronouns in the source language are translated by target-language
pronouns that correspond to the antecedent of the anaphor, there
are some exceptions. In most of these cases, pronominal anaphors are simply
omitted in the target language. For instance, in translations
from English to Spanish, pronouns are frequently not translated because
of the typical Spanish elliptical zero-subject construction.
Other languages with typical zero constructions are Japanese,
Italian, Thai, or Chinese.

In some languages, however, the pronoun is directly translated by its
antecedent. For example, in English--Malay translations there is a
tendency to replace the \emph{it} pronoun with its antecedent, in which case
the translator must first be able to identify the antecedent.

Some languages translate pronouns into different expressions,
depending on the syntactic and semantic information of the
antecedent. For example, in English--Korean translation pronouns
can be elliptically omitted, or they can be translated into definite noun
phrases, into their antecedent, or into different Korean pronouns.

All the above-mentioned problems in the translation of
anaphoric expressions into a target language show that it is
very important to carry out a detailed analysis of these
expressions (including their resolution and the identification of the
antecedent).

Because the majority of MT systems only handle
one-sentence input, they usually cannot deal with anaphora resolution, and if
they do, their successful operation usually does not go beyond
the sentence level. In order to assess the deficiencies of MT
systems, we analyzed the characteristics of different
MT systems, with an emphasis on those characteristics
related to anaphora resolution and translation into a target
language. An overview of our analysis can be seen in Table
\ref{tabla:sistemasTA}.


\begin{table}[htpb]
\begin{minipage}{\textwidth}
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|c|c|} \hline MT system& Strategy\footnote{Strategy
of translation: direct, transfer, or interlingua.}&
Restrict\footnote{Restricted domain.}& Partial\footnote{Partial
parsing.}& Anaphor\footnote{Resolution of intersentential
anaphora.}& Corefer\footnote{Identification of co-reference
chains.}&
Zero\footnote{Translation of zero pronouns into the target language.}\\
\hline Systran & Direct& No& Yes& Yes& No& Yes \\
\hline M\'{e}t\'{e}o & Direct& Yes& No& No& No& No \\
\hline SUSY & Transfer& No& No& Yes& No& No \\
\hline Ariane & Transfer& No& No& Yes& No& Yes \\
\hline Eurotra & Transfer& No& No& No& No& No \\
\hline METAL & Transfer& No& Yes& Yes& No& Yes \\
\hline Candide & Transfer& Yes& No& No& No& No \\
\hline Inter-Nostrum & Transfer& Yes& Yes& No& No& No \\
\hline IXA & Transfer& No& Yes& No& No& No \\
\hline Episteme& Transfer& No& No& No& No& No \\
\hline KANT & Interlingua& Yes& No& Yes& No& Yes \\
\hline DLT & Interlingua& No& No& No& No& No \\
\hline DLT (BKB) & Interlingua& No& No& Yes& No& No \\
\hline Rosetta & Interlingua& No& No& No& No& No \\
\hline CREST & Interlingua& Yes& No& Yes& Yes& Yes \\
\hline $\mu $kosmos & Interlingua& Yes& No& No& No& No \\
\hline
\end{tabular}
\end{center}
\end{minipage}
\caption{\label{tabla:sistemasTA}Characteristics of main MT
systems}
\end{table}





The table reflects a number of different system characteristics.

\begin{enumerate}
%\usecounter{enumiv}
%\setcounter{enumiv}{1} \alph{enumiv}

\item MT system. The MT systems studied included Systran
\cite{Toma77,Wheeler87}; M\'{e}t\'{e}o
\cite{Chandioux76,Chandioux89}; SUSY \cite{Maas77,Maas87}; Ariane
\cite{Boitet81,Boitet89}; Eurotra \cite{Varile88,Allegranza91};
METAL \cite{Bennet85,Thurmair90}; Candide (Berger et al., 1994);
Inter-Nostrum (Canals-Marote et al., 2001a, 2001b); IXA
\cite{Diaz2000,Diaz2001}; Episteme \cite{Amores97,Quesada2000};
KANT \cite{Goodman89,Nirenburg89,Mitamura91}; DLT
\cite{Witkam83,Schubert86}; DLT with Bilingual Knowledge Bank
(BKB) \cite{Sadler89}; Rosetta \cite{Appelo86,Landsbergen87b};
CREST \cite{Farwell00}; and $\mu $kosmos
\cite{Mahesh95a,Mahesh95b}.

\item Strategy of translation. This characteristic indicates the strategy used by
the MT system in accordance with the existence of intermediate
representations: direct, transfer, or interlingua.

\item Restricted domain. This characteristic tells if the texts of the source language
are of a specific domain (restricted domain).

\item Partial parsing. This characteristic indicates if the MT system
carries out a partial parsing of the source text by identifying only
some constituents (noun phrases, prepositional phrases, etc.) and
some relations between them.

\item Resolution of intersentential anaphora. This characteristic indicates whether the MT
system resolves intersentential anaphora. If it does not, then
the anaphoric expressions that have their antecedents in previous
sentences will be incorrectly translated into the target language,
in most of cases.

\item Identification of co-reference chains. This characteristic
tells us if the co-reference chains of the source text are
identified after resolving intersentential anaphora.

\item Translation of zero pronouns. This characteristic indicates
if the MT system detects and resolves omitted pronouns (zero
pronouns\footnote{This kind of pronouns will be presented in
detail in Section \ref{section:ZeroPronouns}.}) in the source
language that are compulsory in the target language.
\end{enumerate}

%\vspace{-0.6cm}



After analyzing the characteristics of the primary commercial MT
systems, we concluded that there is no MT system that
can work on unrestricted texts, resolve intersentential anaphora,
identify the co-reference chains of the text, and translate zero
pronouns into the target language after carrying out a partial
parsing of the source text.

Unlike other systems, such as the KANT interlingua system, the
M\'{e}t\'{e}o system, and the Candide system, among others, that are designed
for well-defined domains, our interlingua MT approach, called AGIR (\emph{Anaphora Generation
with an Interlingua Representation}), works on unrestricted texts.
Although we could have applied full parsing to these texts,
we have instead utilized partial
parsing, due to the unavoidable incompleteness of the
grammar. This is a main difference between our system and other
interlingua systems, such as the DLT system (which is based on a
modification of Esperanto), the Rosetta system (which
experiments with Montague semantics as the basis for an
interlingua), the KANT system, and others.

After parsing and solving pronominal anaphora, an interlingua
re\-pre\-sen\-ta\-tion of the entire text is obtained. To do this,
sentences are split into clauses, and a complex feature structure
based on semantic roles (agent, theme, etc.) is generated for each
one. For each clause, the different semantic roles that appear
will be identified and linked with one entity of
the text. If there is an anaphor in the text, it will be linked
with the entity that represents its antecedent. The AGIR's interlingua representation has been presented in more detail in \cite{Peral99a,Peral00}.

From the interlingua representation, the translation of the
anaphor (including the intersentential anaphor), the detection of
co-reference chains of the whole text, and the translation of
Spanish zero pronouns into English have been carried out. AGIR
has been designed to deal with all these issues which are hardly
considered by most of the real MT systems. Furthermore, our
approach can be used for other applications, for example, for
cross-language information retrieval, summarization, etc.


It is important to note that although some of the above-mentioned
MT systems resolve different problems, such as zero pronouns or
pronominal anaphora, their results are not very satisfactory.
Furthermore, we present some examples (extracted from the corpora
used in the evaluation of our approach---see Section 4) of
incorrect Spanish--English--Spanish translations of pronouns done
by Systran\footnote{A free trial of the commercial product
SYSTRANLinks (copyright 2002 by SYSTRAN S.A.) has been used to
translate between the English and Spanish languages all the
corpora used in the evaluation of our approach. (URL =
\texttt{http://w4.systranlinks.com/config}, visited on
06/22/2002).} that AGIR does correctly\footnote{In this paper, we
have used the symbols (S) and (E) to represent Spanish and English
texts, respectively. The symbol ``\emph{\O}'' indicates the
presence of the omitted pronoun. In the examples, the pronoun and
the antecedent have an index; co-indexing indicates co-reference
between them.}:


\begin{itemize}
\item (\textbf{S}) Siempre creí que a lo que yo aspiraba era a la
comunicación perfecta con un hombre, o, mejor dicho, con el
hombre, con ese príncipe azul de los sueños de infancia, un ser
que sabría adivinarme hasta en los más menudos pliegues
interiores. Ahora he aprendido no sólo que \emph{[esa
fusión]$_i$} es imposible, sino además que \emph{\O$_i$} es
probablemente indeseable.

\item (\textbf{E}) I always thought that to which I aspired I was to
the perfect communication with a man, or, rather, with the man,
that blue prince of the childhood dreams, a being who would know
to guess to me until in slightest you fold interiors. Now I have
learned not only that that fusion is impossible, but in addition
that \emph{he} is probably undesirable.
\end{itemize}

In this example, Systran incorrectly translates into English the
zero pronoun of the last sentence of the paragraph,
proposing the pronoun \emph{he} instead of the pronoun \emph{it}
(the antecedent is the noun phrase \emph{esa fusión}---\emph{that fusion}). Our system proposed the correct pronoun. It is
important to note that although the zero pronoun is identified by Systran, it is
incorrectly solved and subsequently incorrectly translated.

\begin{itemize}
\item (\textbf{S}) Al pasar de \emph{[la luminosidad]$_i$} de la calle, y tal vez
por contraste con \emph{ella$_i$}, impresionaba la oscuridad
interior y el vacío de la Catedral, en la que apenas había a la
vista cuatro o cinco personas.

\item (\textbf{E}) When happening of the luminosity of the street, and
perhaps in contrast with \emph{her}, it impressed the inner dark
and the emptiness of the Cathedral, in which as soon as there
were at sight four or five people.
\end{itemize}


In this case, Systran incorrectly translates into English the
pronoun \emph{ella} (with antecedent \emph{la luminosidad}---\emph{the luminosity}), by proposing the pronoun \emph{her} instead of
\emph{it}.

\begin{itemize}
\item (\textbf{E}) If you have not already done so, unpack \emph{[your
printer]$_i$} and the accessory kit that came with \emph{it$_i$}.
\item (\textbf{S}) Si usted no ha hecho ya así
pues, desempaquete su impresora y el kit de accesorios eso vino
con \emph{él}.
\end{itemize}


This example shows an incorrect English--Spanish translation of
the pronoun \emph{it}. In this case, the pronoun is incorrectly
solved (the antecedent is the noun phrase \emph{your printer},
feminine) and then it is incorrectly translated (pronoun \emph{él}---masculine---instead of pronoun \emph{ésta}---feminine).

All the above examples illustrate that the translation of
pronouns could be notably improved if their antecedents were
correctly identified and, subsequently, pronouns
were translated into the target language.




\section{AGIR's Analysis Module}
AGIR system architecture is based on the general architecture of an
MT system that uses an interlingua strategy. Translation is
carried out in two stages: (1) from the source language to the
interlingua, and (2) from the interlingua into the target
language. Modules for analysis are independent from modules for
generation. Although our present work has only studied the
Spanish and English languages, our approach can be easily
extended to other languages, for exampe, to multilingual system, in the
sense that any analysis module can be linked to any generation
module.

In AGIR the analysis is carried out using SUPAR (\emph{slot
unification parser for anaphora resolution}) \cite{Ferrandez99}.
SUPAR is a computational system that focuses on anaphora
resolution. It can deal with several kinds of anaphora, such as
pronominal anaphora, one-anaphora, or definite
descriptions\footnote{\emph{One-anaphora} has the following
structure in English: a determiner and the pronoun \emph{one} with
some premodifiers or postmodifiers (\emph{the red one}; \emph{the
one with the blue bow}). This kind of anaphors in Spanish consists
of noun phrases in which the noun has been omitted (\emph{el
rojo}; \emph{el que tiene el lazo azul}). In \emph{definite
descriptions}, anaphors are formed by definite noun phrases that
refer to objects that are usually uniquely determined in the
context.}. The SUPAR's input is a grammar defined by means of the
grammatical formalism SUG (\emph{slot unification grammar}). A
translator that transforms SUG rules into Prolog clauses has been
developed. This translator provides a Prolog program that will
parse each sentence. SUPAR can perform either a full or a partial
parsing of the text with the same parser and grammar. In this
study, partial-parsing techniques have been utilized due to the
unavoidable incompleteness of the grammar and the use of
unrestricted texts (corpora) as input.

The analysis of the source text is carried out in several steps.
The first step of the analysis module is the lexical and
morphological analysis of the input text. Because of the use of
unrestricted texts as input, the system obtains the lexical and
morphological information of the texts' lexical units from the
output of a part-of-speech (POS) tagger. The word as it appears
in the corpus, its lemma, and its POS tag (with morphological
information) is supplied for each lexical unit in the corpus.

The next step is the parsing of the text (which includes the
lexical and mor\-pho\-lo\-gi\-cal information extracted in the
previous stage). Before applying the parsing, the text is split
into sentences. The output will be the slot structure (SS) that
stores the necessary information\footnote{The SS stores the
following information for each constituent: constituent name (NP,
PP, etc.), semantic and morphological information, discourse
marker (identifier of the entity or discourse object), and the SS
of its subconstituents.}   for the subsequent stages.

In the third step, a module of word-sense disambiguation (WSD) is
used to obtain a single sense for the different texts' lexical
units. The lexical resources, WordNet \cite{Miller90} and
EuroWordNet \cite{Vossen98}, have been used in this
stage\footnote{In the evaluation of our approach, we have only
used an English corpus (SemCor) where all content words are
annotated with their WordNet sense; this sense has been used to
identify the semantic category of the word. The remaining corpora
do not have information about the senses of the content words;
therefore, a set of heuristics has been used to identify their
semantic categories. Currently, a WSD module \cite{Montoyo2000a}
is being developed in our Research Group, which will be incorporated into our system in the future.}.

The SS, enriched with the information from the previous steps, will be
the input for the next step, in which NLP problems (anaphora,
extraposition, ellipsis, etc.) will be treated and solved. In
this work, we have focused on the resolution of NLP problems
related to pronominal anaphora. After this step, a new slot
structure (SS') is obtained. In this new structure, the correct
antecedent---chosen from the possible candidates after applying a
method based on constraints and preferences \cite{Ferrandez99}---for each anaphoric expression will be stored along with its
morphological and semantic information. The new structure SS'
will be the input for the final step of the analysis module.


In the last step, AGIR generates the interlingua representation
of the entire text. This is the main difference between AGIR and
other MT systems, which process the input
text sentence by sentence. The interlingua representation will
allow the correct translation of the intersentential and
in\-tra\-sen\-ten\-tial pronominal anaphora into the target
language. Moreover, AGIR allows the identification of co-reference
chains of the text and their subsequent translation into the
target language.

The interlingua representation of the input text is based on the
clause as the main unit of this representation. Once the text has
been split into clauses, AGIR uses a complex feature structure for
each clause. This structure is composed of semantic roles and
features extracted from the SS of the clause. The notation we
have used is based on the one used in KANT interlingua.

It is important to emphasize that the interlingua lexical unit has
been re\-pre\-sen\-ted in AGIR using the word and its correct
sense in WordNet. After accessing the ILI (inter-lingual-index)
module of EuroWordNet, we will be able to generate the lexical
unit into the target language.

Once the semantic roles have been identified, the interlingua
representation will store the clauses with their features, the
different entities that have appeared in the text and the
relations between them (such as anaphoric relations). This
representation will be the input for the generation module.


\section{Resolution of NLP Problems in AGIR}
The fourth stage of the AGIR's analysis module allows for the
resolution of NLP problems. Our present work focuses on the resolution
of NLP problems related to pronominal anaphora in the source
language so as to translate these anaphoric
expressions correctly into the target language. We are only describing the
translation of anaphoric, third-person, personal pronouns into the
target language. Therefore, we have only focused on the
discrepancies between Spanish and English in the
treatment of these pronouns. In the next two subsections, we will
describe the syntactic discrepancies treated and solved in AGIR
(Spanish zero pronouns) and the anaphora resolution module of the
system.


\subsection{Elliptical Zero-Subject Constructions
(Zero Pronouns)} \label{section:ZeroPronouns}

The Spanish language allows for the omission of the pronominal subject of the
sentences. These omitted pronouns are usually called zero
pronouns. Whereas in other languages (e.g., in
Japanese), zero pronouns may appear in
either the subject's or the object's grammatical position, in Spanish texts zero pronouns only appear in the
position of the subject.

In MT systems, the correct detection and resolution of zero
pronouns in the source language is of crucial importance if
these pronouns are compulsory in the target language. In the
following example, a Spanish sentence that contains a zero pronoun
and its translation into English with the equivalent compulsory
pronoun are shown.

%\vspace{1cm}

\begin{itemize}
\item (\textbf{S}) \emph{[Ese hombre]$_i$} era un boxeador profesional. \emph{\O$_i$} Perdi\'{o}
\'{u}nicamente dos combates.
\item (\textbf{E}) \emph{[That man]$_i$} was a professional boxer. \emph{He$_i$}
only lost two fights.
\end{itemize}



We should remark that zero pronouns can also occur in English,
although they appear less frequently, since they usually are used
in coordinated sentences in which the zero pronoun usually refers
to the subject of the sentence. Although zero pronouns have
already been studied in other languages, such as Japanese---with a
resolution percentage of 78\% in the work of \cite{Okumura96},
they have not yet been studied in Spanish texts.
\cite{Ferrandez00} has presented the first algorithm for Spanish
zero-pronoun resolution. Basically, in order to translate Spanish
zero pronouns into English, they must first be located in the text
(ellipsis detection) and then resolved (anaphora resolution)
\cite{Peral00b}:

\begin{itemize}
\item Zero-pronoun detection. In order to detect zero pronouns,
sentences should be divided into clauses, since the subject can
only appear between the clause constituents. After that, a
noun-phrase (NP) or a pronoun is sought, for each clause, through
the clause constituents on the lefthand side of the verb, unless
it is imperative or impersonal. Such an NP or pronoun must agree
in person and number with the verb of the clause.

\item Zero-pronoun resolution. After the zero pronoun has been
detected, our computational system inserts the pronoun in the
position in which it has been omitted. This pronoun will be
detected and resolved in the following module of anaphora
resolution. Person and number information is obtained from the
clause verb. Sometimes, in Spanish, the gender information of the
pronoun can be obtained from the object when the verb is
copulative. In these cases, the subject must agree in gender and
number with its object whenever the object can have either a
masculine or feminine linguistic form.
\end{itemize}


\subsubsection{Evaluation}

To evaluate this task, two experiments were performed: an
evaluation of zero-pronoun detection and an evaluation of
zero-pronoun resolution. In both experiments the method was tested
on two kinds of corpora. In the first instance, we used a portion
of the LEXESP\footnote{The LEXESP corpus belongs to the project of
the same name, carried out by the Psychology Department of the
University of Oviedo and developed by the Computational
Linguistics Group of the University of Barcelona, with the
collaboration of the Language Processing Group of the Catalonia
University of Technology, Spain.} corpus that contains a set of
thirty-one documents (38,999 words) from different genres and
written by different authors. The LEXESP corpus contains texts of
different styles and on different topics (newspaper articles about
politics, sports, etc.; narratives about specific topics; novel
fragments; etc.). In the second instance, the method was tested on
a fragment of the Spanish version of Blue Book (BB) corpus (15,571
words), a technical manual that contains the handbook of the
International Telecommunications Union (CCITT) published in
English, French, and Spanish. Both corpora are automatically
tagged by different taggers.


We randomly selected a subset of the LEXESP corpus (three documents
---6,457 words) and a fragment of the Blue Book corpus (4,723
words) as training corpora. The remaining fragments of the corpora
were reserved for test data.

It is important to emphasize that all the tasks presented in this
paper were automatically evaluated after the annotation of each
pronoun (including zero pronouns). To do so, each anaphoric,
third-person, personal pronoun was annotated with the information
about its antecedent and its translation into the target language.
Furthermore, co-reference chains were identified. The annotation
phase was accomplished in the following manner: (1) two
annotators (native speakers) were selected for each language, (2)
an agreement was reached between the annotators with regard to the
annotation scheme, (3) each annotator annotated the corpora, and
 (4) a reliability test (Carletta et al., 1997) was done
on the annotation in order to guarantee the results. The
reliability test used the \emph{kappa} statistic that measures
agreement between the annotations of two annotators in making
judgments about categories. In this way, the annotation is
considered a classification task consisting of defining an
adequate solution among the candidate list. According to Carletta
et al. \citeyear{Carletta97}, a \emph{k} measurement such as
$0.68 < k < 0.80$ allows us to draw encouraging conclusions, and a
measurement $k > 0.80$ means there is total reliability between
the results of the two annotators. In our tests, we obtained a
\emph{kappa} measurement of 0.83. Therefore, we consider
the annotation obtained for the evaluation to be totally reliable.


\subsubsection{Evaluation of Zero-Pronoun Detection}

In the evaluation of zero-pronoun detection, the training phase
was used to carry out modifications in the grammar in order to
improve the processes of partial parsing and clause splitting.
After this training, we conducted a blind test over the entire
test corpus. To achieve this sort of evaluation, several different
subtasks may be considered. First, each verb must be detected.
This task is easily accomplished since both corpora have been
previously tagged. The second task is to classify the verbs into
two categories: (a) verbs whose subjects have been omitted, and
(b) verbs whose subjects have not. The obtained results with
the LEXESP and Blue Book corpora appear in Table
\ref{tabla:deteccionOmitidosEvaluacion}.


\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|c|c|c|c|c|c|c|} \hline
\raisebox{-1.50ex}[0cm][0cm]{}& \multicolumn{6}{|c|}{Verbs with
subject omitted} &
\multicolumn{6}{|c|}{Verbs with subject \emph{not} omitted}  \\
\cline{2-13}
 &
1$^{st}$&P({\%})&2$^{nd}$&P({\%})&3$^{rd}$&P({\%})&1$^{st}$&
P({\%})&2$^{nd}$&P({\%})&3$^{rd}$&P({\%})\\
\hline \raisebox{-1.50ex}[0cm][0cm]{LX}& 240& 96.7& 54& 98.1&
1,227& 97.1& 31& 71& 17& 94.1& 1,085&
83.3 \\
\cline{2-13}
 &
\multicolumn{6}{|c|}{PRECISION = 97.1{\%}} &
\multicolumn{6}{|c|}{PRECISION = 83.1{\%}}  \\
\hline \raisebox{-1.50ex}[0cm][0cm]{BB}& 0& 0& 0& 0& 121& 97.5&
0& 0& 0& 0& 351&
82 \\
\cline{2-13}
 &
\multicolumn{6}{|c|}{PRECISION = 97.5{\%}} &
\multicolumn{6}{|c|}{PRECISION = 82.0{\%}}  \\
\hline
\multicolumn{13}{|c|}{\textbf{GLOBAL PRECISION = 90.4{\%}}}  \\
\hline
\end{tabular}
\end{center} \caption{\label{tabla:deteccionOmitidosEvaluacion}Zero-pronoun detection, evaluation phase}
\end{table}



%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=15cm, height=5.28cm, clip]{deteccionOmitidosEvaluacion.eps}
%  \caption{Blue Book corpus: Zero-pronoun detection, evaluation phase}
%  \label{tabla:deteccionOmitidosEvaluacion}
%\end{table}


The table is divided into two parts, corresponding to categories
(a) and (b) previously mentioned. For each category, the number
of verbs in first, second, and third person, together with their
precision (\textbf{P}), are represented. Precision was
defined as the number of verbs correctly classified (subject
omitted or not) divided by the total number of verb classifications attempted for
each type. For example, in the LEXESP corpus 1,227
verbs in the third person with their subjects omitted were classified,
and the precision obtained was 97.1\%.

\textbf{Discussion.} In the detection of zero pronouns the
following results were obtained: for the LEXESP corpus, precisions of 97.1\%
and 83.1\%  were obtained for verbs whose subjects were
omitted or were not omitted, respectively; for the BB corpus, precisions of
97.5\% and 82\% were obtained. For both
corpora, an overall precision of 90.4\%
(2,825 out of a total of 3,126) was obtained for this task.

From these results, we have extracted the following conclusions:

\begin{itemize}
\item There are no meaningful differences between the results
obtained with each corpus.

\item The BB corpus has no verbs in either the first or second person.
This is explained by considering the style of the corpus: it is a technical
manual which usually consists of a series of isolated definitions
done by the writer.

\item The rate of precision for the detection of verbs whose
subjects are not omitted is lower (approximately 15\%) than for
the detection of verbs whose subjects are omitted. There are
several reasons for this:

\begin{itemize}
\item The POS tagger does not identify impersonal verbs. This
problem has been partly resolved heuristically, by the choice of
impersonal verbs (e.g., \emph{llover}---\emph{to rain}), but it
cannot be resolved for all impersonal verbs. For example,
the verb \emph{ser} (\emph{to be}) is not usually impersonal, but
it can be in certain constructions (e.g., \emph{Es hora de
desayunar}---\emph{It is time to have breakfast}).

\item The ambiguity and the unavoidable incompleteness of the
grammar affects the process of clause splitting, and therefore
affects the detection of the possible subject for the clause on
the lefthand side of the verb.
\end{itemize}

\end{itemize}


Since ours is the first study done specifically on Spanish texts
and since the design of the detection stage mainly depends upon
the structure of the language in question, we have not compared
our results with those of other published works. Such comparisons
would prove to be insignificant\footnote{In order to compare our
system with other systems, in Section 6.2 we evaluate pronoun
translation (including zero pronouns) between Spanish and English
using the commercial product SYSTRANLinks and the Spanish LEXESP
corpus. The evaluation highlights the deficiencies of zero-pronoun
detection, resolution, and translation (out of 559 anaphoric,
third-person, zero pronouns in the LEXESP corpus, only 266 were
correctly translated into English---a precision of only 47.6\%).}.

Finally, it is important to emphasize the importance of this
phenomenon in Spanish. Specifically, in both corpora, the subject is omitted in 52.5\%
(1,642 out of 3,126) of the verbs.
Furthermore, this phenomenon is even more important in narrative texts (57.3\%
in the LEXESP corpus) than in the technical manuals (25.6\% in the BB corpus).
These percentages show the importance of correctly detecting
these kinds of pronouns in an MT system so as to conveniently
translate them into the target language.


\subsubsection{Evaluation of Zero-Pronoun Resolution}
After zero pronouns have been detected, they are then resolved in
the subsequent module of anaphora resolution (explained in the
following subsection). Basically, an algorithm that combines
different kinds of knowledge by distinguishing between
constraints and preferences is used (Ferr\'andez et al., 1999;
Palomar et al., 2001).

The set of constraints and preferences presents two basic
differences between zero-pronoun and pronominal anaphora
resolution:

\begin{enumerate}
\item Zero-pronoun resolution has the constraint of agreement
only in person and number, whereas pronominal anaphora resolution
also requires gender agreement.

\item Two new preferences to solve zero pronouns are used: (a) preference is given to
candidates in the same sentence as the anaphor that have also been the solution of a zero pronoun in the same
sentence as the anaphor, and (b) in the case where the zero pronoun has gender
information, preference is given to those candidates that agree in gender.


\end{enumerate}

In evaluating zero-pronoun resolution so as to
obtain the best order of preferences (one that produces the best
performance), we used the training phase
to identify the importance of each kind of knowledge. To do this, we
analyzed the antecedent for each pronoun in the training
corpora, and we identified their configurational
characteristics with reference to the pronoun (e.g.,
\emph{if the antecedent was a proper noun}, \emph{if the
antecedent was an indefinite NP}, \emph{if the antecedent occupied
the same position with reference to the verb as the anaphor
---before or after}, etc.). Subsequently, we constructed a
table that showed how often each configurational characteristic
was valid for the solution of a particular pronoun (e.g.,
the solution of a zero pronoun was a proper noun 63\% of the time,
for a reflexive pronoun, it was a proper noun 53\% of the time, etc.).
In this way, we were able to define the different patterns of Spanish
pronoun resolution and apply them in order to obtain the
evaluation results that are presented in this paper. The order of
importance was determined by first sorting the preferences
according to the percentage of each configurational
characteristic; that is, preferences with higher percentages were
considered more important than those with lower percentages. After
several experiments on the training corpora, an optimal order for
each type of anaphora was obtained. Since in this phase we processed
texts from different genres and by different authors, we can
state that the final set of preferences obtained and their order
of application can be used with confidence on any Spanish text.





After the training, we conducted a blind test over the entire
test corpus, the results for which are shown in Table
\ref{tabla:ceroPronombresEvaluacion}.


\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|} \hline & Cataphoric & Exophoric &
\multicolumn{3}{|c|}{Anaphoric}  \\
\hline & & & Correct& Total &
P({\%})  \\
\hline LEXESP& 640& 28& 455& 559&
81.4 \\
\hline BB& 76& 8& 30& 37&
81.1 \\
\hline \textbf{TOTAL}& \textbf{716}& \textbf{36}& \textbf{485}&
\textbf{596}&
\textbf{81.4} \\
\hline
\end{tabular}
\end{center}
\caption{\label{tabla:ceroPronombresEvaluacion}Zero-pronoun
resolution, evaluation phase}
\end{table}


%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=10.5cm, height=3.5cm, clip]{ceroPronombresEvaluacion.eps}
%  \caption{Zero-pronoun resolution, evaluation phase}
%  \label{tabla:ceroPronombresEvaluacion}
%\end{table}


It is important to mention here that out of 3,126 verbs in these corpora, 1,348
 (Table
\ref{tabla:deteccionOmitidosEvaluacion}) are zero pronouns in
the third person and will be resolved. In Table
\ref{tabla:ceroPronombresEvaluacion} we present a classification
of these third-person zero pronouns, which has been conveniently
divided into three categories:

\begin{enumerate}
\item Cataphoric. This category is comprised of those zero
pronouns whose antecedents, that is, the clause subjects, come after
the verb. For instance, in the following Spanish sentence
\emph{\emph{\O$_i$} Compr\'o \emph{[un niño]$_i$} en el
supermercado} (\emph{\emph{[A boy]$_i$} bought in the
supermarket}), the subject, \emph{un niño} (\emph{a boy}), appears
after the verb, \emph{compr\'o} (\emph{bought}). These kinds of
verbs are quite common in Spanish (\textbf{P} = 53.1\%, 716 out of 1,348), as can be seen in
Table \ref{tabla:ceroPronombresEvaluacion}, and represents
one of the main difficulties in resolving anaphora in
Spanish: the structure of a sentence is more flexible than in
English. These represent intonationally marked sentences, where
the subject does not occupy its usual position in the sentence,
that is, before the verb. Cataphoric zero pronouns will not be
resolved in AGIR, since semantic information is needed to be able
to discard all of their antecedents and \emph{to give preference to} those
that appear within the same sentence and clause after the verb.

For example, the sentence \emph{\emph{\O} Compr\'o un regalo en el
supermercado} (\emph{\emph{[He]$_{\O}$} bought a present in the
supermarket}) has the same syntactic structure as the previous
sentence, i.e., verb, NP, and PP, where the object function of the
NP can only be distinguished from the subject by means of semantic
knowledge.


\item Exophoric. This category consists of those zero pronouns
whose antecedents do not appear linguistically in the text (they
refer to items in the external world rather than things referred
to in the text). Exophoric zero pronouns will not be resolved by
the system.

\item Anaphoric. This category is comprised of those zero pronouns
whose antecedents are found before the verb. These kinds of
pronouns will be resolved by our system.

\end{enumerate}

In Table \ref{tabla:ceroPronombresEvaluacion} the numbers of
cataphoric, exophoric, and anaphoric zero pronouns for each corpus
are shown. For anaphoric pronouns, the number of pronouns
correctly solved as well as the obtained precision, \textbf{P} (number of
pronouns correctly solved divided by the number of solved
pronouns) is presented. For example, in the LEXESP corpus, there
are 640 cataphoric, 28 exophoric, and 559 anaphoric zero pronouns.
From these anaphoric pronouns, only 455 were correctly
solved, giving a precision of 81.4\%.





\textbf{Discussion.} In zero-pronoun resolution, the following
results have been obtained: LEXESP corpus, \textbf{P} = 81.4\%;
BB corpus, \textbf{P}  = 81.1\%. For the combined corpora, an overall
precision for this task of 81.4\% (485 out of 596) was
obtained. The overall recall, \textbf{R} (the number of pronouns
correctly solved divided by the number of real pronouns) obtained
was 79.1\% (485 out of 613).

From these results, we have extracted the following conclusions:

\begin{itemize}
\item There are no meaningful differences between the results
obtained from each corpus.

\item Errors in the zero-pronoun-resolution stage are originated by
different causes:

\begin{itemize}
\item exceptions in the application of preferences that imply the
selection of an incorrect antecedent as solution of the zero
pronoun (64\% of the global mistakes)

\item the lack of semantic information\footnote{It is important to
mention here that semantic information was not available for
the Spanish corpora.}, causing an error rate of 32.4\%

\item mistakes in the POS tagging (3.6\%)
\end{itemize}

\end{itemize}


Since the results provided by other works have been obtained for
different languages (English), texts, and sorts of knowledge
(e.g., Hobbs and Lappin full parse the text), direct comparisons
are not possible. Therefore, in order to accomplish this
comparison, we have implemented some of these approaches in
SUPAR\footnote{A detailed study of these implementations in SUPAR
is presented in Palomar et al. \citeyear{Palomar2001}.}, adapting
them for partial parsing and Spanish texts. Although these
approaches were not proposed for zero pronouns and the comparison
will not be fully fair, we have implemented them since that is the
only way to compare our proposal directly with some well-known
anaphora-resolution algorithms.

We have also compared our system with the typical baseline of
proximity preference (i.e., the antecedent that appears closest to
the anaphora is chosen from among those that satisfy the
constraints---morphological agreement and syntactic conditions).
We have also compared our system with the baseline presented by
Hobbs \citeyear{Hobbs78}\footnote{Hobbs's baseline is frequently
used to compare most of the work accomplished on anaphora
resolution. Hobbs's algorithm does not work as well as ours
because it carries out a full parsing of the text. Furthermore,
the manner in which the syntactic tree is explored using Hobbs's
algorithm is not the best one for Spanish, since it is nearly a
free-word-order language.} and Lappin \& Leass' method
\cite{Lappin94}. Moreover, we also compared our proposal with
centering approach by implementing functional centering
\cite{Strube99}. The precisions obtained with these different
approaches and AGIR are shown in Table
\ref{tabla:ceroPronombresOtrosAutores}. As can be seen, the
precision obtained in AGIR is better than those obtained using the
other proposals.

\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|} \hline & Proximity& Hobbs& Lappin& Strube&
AGIR \\
\hline LEXESP& 54.9& 60.4& 66.0& 59.7&
\textbf{81.4} \\
\hline BB& 48.6& 62.2& 67.6& 59.5&
\textbf{81.1} \\
\hline
\end{tabular}
\end{center}
\caption{\label{tabla:ceroPronombresOtrosAutores}Zero-pronoun
resolution in Spanish, comparison of AGIR with other approaches}
\end{table}


%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=10.3cm, height=3.2cm, clip]{ceroPronombresOtrosAutores.eps}
%  \caption{Zero-pronoun resolution in Spanish, comparison of AGIR with other approaches }
%  \label{tabla:ceroPronombresOtrosAutores}
%\end{table}




\subsection{The Anaphora-Resolution Module}

The anaphora-resolution module used in AGIR is based on the module
presented in (Fe\-rr\'an\-dez et al., 1999; Palomar et al., 2001)
for the SUPAR system. The algorithm identifies noun phrase (NP)
antecedents of personal, demonstrative, reflexive, and zero
pronouns in Spanish. It identifies both intrasentential and
intersentential antecedents and is applied to the syntactic
analysis generated by SUPAR. It also combines different forms of
knowledge by distinguishing between constraints and preferences.
Whereas constraints are used as combinations of several kinds of
knowledge (lexical, morphological, and syntactic), preferences are
defined as a combination of heuristic rules extracted from a
study of different corpora.



A constraint defines a property that must be satisfied in order
for any candidate to be considered as a possible solution of the
anaphor. The constraints used in the algorithm are the following:
morphological agreement (person, gender, and number) and
syntactic conditions on NP-pronoun non-co-reference.

A preference is a characteristic that is not always satisfied by
the solution of an anaphor. The application of preferences
usually involves the use of heuristic rules in order to obtain a
ranked list of candidates. Some examples of preferences used in
our system are the following: (a) antecedents that
are in the same sentence as the anaphor, (b) antecedents that have
been repeated more than once in the text, (c) antecedents that
appear before their verbs (i.e., the verb of the clause in which
the antecedent appears), (d) antecedents that are proper nouns,
(e) antecedents that are an indefinite NP, and so on.


In order to solve pronominal anaphors, they must be first
located in the text (anaphora detection) and then resolved
(anaphora resolution):

\begin{itemize}
\item Anaphora detection. In the algorithm, all the types of
anaphors are identified from left to right as they appear in the
sentence's slot structure obtained after the partial parsing.
To identify each type of pronoun, the information stored in the
POS tags has been used. In the particular case of zero pronouns,
they have been detected in a previous stage, as previously
described.

\item Anaphora resolution. After the anaphor has been detected,
the corresponding method, based on constraints and preferences, is applied to
solve it. Each type of anaphor has its own set of
constraints and preferences, although they all follow the same
general algorithm: constraints are applied first, followed by
preferences. Constraints discard some of the candidates, whereas
preferences simply sort the remaining candidates.

\end{itemize}


\subsubsection{Evaluation}
In evaluating the algorithm for anaphora resolution\footnote{As
previously mentioned, only anaphoric, third-person, personal
pronouns will be resolved in order to translate them into the
target language.}, we looked at pronominal anaphora resolution in
Spanish and English, respectively. For the Spanish evaluation,
the method was tested on the portion of the LEXESP corpus
previously used to evaluate zero-pronoun detection and
resolution. For English, we tested the method on two kinds of
corpora. In the first instance, we used a portion of the SemCor
collection---presented in \cite{Landes98}---which contains a set
of eleven documents (23,788 words) in which all content words are
annotated with the most appropriate WordNet sense. The SemCor
corpus contains texts about different topics (law, sports,
religion, nature, etc.) and was written by different authors. In
the second instance, the method was tested on a portion of the
MTI\footnote{This corpus was provided by the Computational
Linguistics Research Group of the School of Humanities, Languages
and Social Studies, University of Wolverhampton, England. The
corpus is anaphorically annotated indicating the anaphors and
their correct antecedents.} corpus, which contains seven
documents (101,843 words). The MTI corpus contains computer
science manuals on different topics (commercial applications,
word processing applications, device instructions, etc.). Both
English corpora are automatically tagged by different taggers.

We randomly selected a subset of the SemCor corpus (three
documents---6,473 words) and another subset of the MTI corpus
(two documents---24,264 words) as training corpus. The
remaining fragments of the corpora were reserved for test data.

In the two tasks, the training phase was used to identify the
importance of each kind of knowledge to obtain the optimal order
of the preferences.

\subsubsection{Evaluation of Anaphora Resolution in Spanish}
An evaluation of the algorithm for anaphora resolution in Spanish
has been given in detail in the work of Palomar et al.
\citeyear{Palomar2001}. In this paper, we present the obtained
results of the evaluation of this task in AGIR over a different
portion of the LEXESP corpus. Furthermore, non-anaphoric
complement pronouns, that is, complement pronouns that appear next
to the previous indirect object when it has been moved from its
theoretical place after the verb (\emph{A Pedro$_i$} le$_i$
\emph{vi ayer}---\emph{I saw Pedro yesterday}), were not resolved
because this kind of pronoun does not appear in the English
translation. For these reasons, the results of the two works are
slightly different.

After the training phase, the algorithm was evaluated over the
test corpus. In this evaluation, only lexical, morphological, and
syntactic information was used. Table
\ref{tabla:anaforaEspanolEvaluacion} shows the results of this
evaluation.

\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|c|c|c|c|c|} \hline & Comp&P({\%})& Ref&P({\%})&
PP&P({\%})&PP&P({\%})&Total&
P({\%})\\
& & & & & notPP& & inPP& & &Total\\
\hline LEXESP  & 98& 82.6& 105& 92.4& 71& 70.4& 46& 76.1& 320&
82.2\\
\hline
\end{tabular}
\end{center}
\caption{\label{tabla:anaforaEspanolEvaluacion}Anaphora resolution
in Spanish, evaluation phase}
\end{table}



%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=14.8cm, height=2.6cm, clip]{anaforaEspanolEvaluacion.eps}
%  \caption{Anaphora resolution in Spanish, evaluation phase}
%  \label{tabla:anaforaEspanolEvaluacion}
%\end{table}

In Table \ref{tabla:anaforaEspanolEvaluacion} the occurrences of
personal pronouns in the LEXESP corpus are shown. The different types
are: \emph{Comp} (complement personal pronouns), \emph{Ref}
(reflexive pronouns), \emph{PPnotPP} (personal pronouns not
included in a prepositional phrase), and \emph{PPinPP} (personal
pronouns included in a prepositional phrase). For each type, the
obtained precision, \textbf{P} (the number of pronouns correctly solved
divided by the number of solved pronouns), is shown. The last two
columns represent the total number of personal pronouns and the obtained
precision.

\textbf{Discussion.} In pronominal anaphora resolution in
Spanish, we obtained a precision of 82.2\% (263 out of 320). The
recall, \textbf{R} (number of pronouns correctly solved divided by
the number of real pronouns), obtained was of 79\% (263 out of
333).

After analyzing the results, the following conclusions were
extracted:

\begin{itemize}
\item In the resolution of reflexive pronouns, a high precision
(92.4\%) was obtained. This higher percentage is
because the antecedent of these pronouns is usually the closest
NP to the pronoun and it is in the same sentence. Therefore,
after applying preferences, few errors are produced.

\item Analyzing the errors in the remaining pronouns, it is
important to mention the complexity of the LEXESP corpus itself. It
consists of several narrative documents, sometimes with a very
complex style, with long sentences (with an average of 24.6 words
per sentence). This implies a large number of candidates per
anaphor after applying constraints (an average of 16.6).

\item Errors were originated by different causes:
\begin{itemize}
\item exceptions in the application of preferences (66.7\% of the
global mistakes)

\item a lack of semantic information (29.8\%)

\item mistakes in the POS tagging (3.5\%)
\end{itemize}

\end{itemize}

We compared our proposal with the approaches previously
presented in the evaluation of zero-pronoun resolution. As
shown in Table \ref{tabla:anaforaEspanolOtrosAutores}, the
precision obtained using AGIR is better than those for the other proposals.

\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|} \hline & Proximity& Hobbs& Lappin& Strube&
AGIR \\
\hline LEXESP& 52.5& 65.3& 73.3& 68.3&
\textbf{82.2} \\
\hline
\end{tabular}
\end{center}
\caption{\label{tabla:anaforaEspanolOtrosAutores}Anaphora
resolution in Spanish, comparison of AGIR with other approaches}
\end{table}



%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=10.3cm, height=2.8cm, clip]{anaforaEspanolOtrosAutores.eps}
%  \caption{Anaphora resolution in Spanish, comparison of AGIR with other approaches }
%  \label{tabla:anaforaEspanolOtrosAutores}
%\end{table}




\subsubsection{Evaluation of Anaphora Resolution in English}
The algorithm for a\-na\-pho\-ra resolution in English is based on
the one developed for Spanish, and it has been conveniently adapted for
English. The main difference between the two algorithms consists
in a different order of the preferences obtained after the
training phase. After this phase, we extracted the following
conclusions:


\begin{itemize}
\item Spanish has more morphological information than
English. As a consequence, morphological constraints in Spanish
discard more candidates than constraints in English.

\item Spanish is a nearly free-order language, in which the
different constituents of a sentence (subject, object, etc.) can
appear almost at any position. For this reason, the preference of
\emph{syntactic parallelism} has a more important role in the
anaphora-resolution method in English than in Spanish.

\item Spanish sentences are usually longer than English ones. This
fact implies more candidates for Spanish anaphors than for
English ones.
\end{itemize}


After the training phase, the algorithm was evaluated over the
test corpus. In the evaluation phase, two experiments were
carried out. In the first experiment, only lexical, morphological, and
syntactic information was used. The obtained results with
the SemCor and MTI corpora appear in Table
\ref{tabla:anaforaInglesEvaluacionUno}.


\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|c|c|c|c|c|} \hline & He& She& It& They& Him& Her&
Them& Corr& Total&
P({\%}) \\
\hline SEMCOR& 116& 10& 38& 50& 34& 0& 6& 175& 254&
68.9 \\
\hline MTI& 1& 0& 347& 56& 0& 0& 66& 361& 470&
76.8 \\
\hline
\end{tabular}
\end{center}
\caption{\label{tabla:anaforaInglesEvaluacionUno}Anaphora
resolution in English, evaluation phase: experiment 1}
\end{table}


%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=12.6cm, height=2.5cm, clip]{anaforaInglesEvaluacionUno.eps}
%  \caption{Anaphora resolution in English, evaluation phase: experiment 1}
%  \label{tabla:anaforaInglesEvaluacionUno}
%\end{table}


The table shows the number of pronouns (classified by type) for
the different corpora. The last three columns represent the number
of correctly solved pronouns, the total number of pronouns, and the
obtained precision, respectively. For instance, in the MTI
corpus a precision of 76.8\% was obtained.

\textbf{Discussion.} In pronominal anaphora resolution in English,
the following results were obtained in the first experiment:
SemCor corpus, \textbf{P} = 68.9\%, \textbf{R} = 66\%; MTI corpus, \textbf{P} = 76.8\%,
\textbf{R} = 72.9\%.

From these results, we have extracted the following conclusions:

\begin{itemize}
\item The types of pronouns vary considerably according to the
corpus. In the SemCor corpus, 15\% of the pronouns are occurrences
of the \emph{it} pronoun, whereas in the MTI corpus this percentage is
73.8\%. This fact is explained by the kind and domain of each
corpus. The SemCor is a corpus with a narrative style which
contains a lot of \emph{person} entities\footnote{If we use a
basic ontology based on semantic features, at the top level,
entities could be classified into three main categories:
\emph{person}, \emph{animal}, and \emph{object}.} that are
referred to in the text with the use of personal pronouns (\emph{he},
\emph{she}, and \emph{they}). On the other hand, the MTI corpus is a
collection of technical manuals that contains almost no
\emph{person} entities. Rather, most references are to \emph{object}
entities, using \emph{it} pronouns.

\item In the SemCor corpus, errors originated from different
causes:

\begin{itemize}
\item The lack of semantic information caused 57\% of the global
mistakes. There were seventeen mistakes in the resolution of \emph{it}
pronouns, in which the system proposed a \emph{person} entity as
solutions for these pronouns. On the other hand, twenty-eight occurrences of
the pronouns \emph{he}, \emph{she}, \emph{him}, and \emph{her} were
incorrectly solved due to the system proposing an \emph{object} or
\emph{animal} entity as the solution.

\item There were exceptions in the applications of preferences (38\%), mainly
due to the existence of a large number of candidates compatible with the
anaphor\footnote{The sentences of the SemCor corpus are very long
(with an average of 24.3 words per sentence). This fact implies a
large number of candidates per anaphor (an average of 15.2) after applying constraints.}.

\item There were mistakes in the POS tagging (5\%).
\end{itemize}


\item In the MTI corpus, errors were mainly produced in the resolution of
\emph{it} pronouns (73.4\% of the global mistakes). The \emph{it}
pronoun lacks gender information (it is valid for masculine and
feminine) and subsequently there are a lot of candidates per
anaphor\footnote{The sentences of the MTI corpus are not very long
(with an average of 15.5 words per sentence). However, the
candidates per anaphor, after applying constraints, are high (an
average of 13.6).}. This fact originates errors in the application
of preferences. The remaining errors are originated by the lack of
semantic information.


\end{itemize}

After analyzing the results, it was observed that the precision
of the SemCor corpus was approximately 8\% lower than that for the
MTI corpus. The errors in the SemCor corpus mainly originated
with the lack of semantic information. Therefore, in order to
improve the obtained results, a second experiment was carried out with the
addition of semantic information.

The modifications to the second experiment were the following:
\begin{itemize}
\item Two new semantic constraints---presented in \cite{Saiz2000a}---were
added to the morphological and syntactic constraints:

\begin{enumerate}
\item The pronouns \emph{he}, \emph{she}, \emph{him}, and \emph{her}
must have as the antecedents \emph{person} entities.

\item The pronoun \emph{it} must have as its antecedent a \emph{non-person}
entity.
\end{enumerate}

To apply these new constraints, the twenty-five \emph{top concepts} of
WordNet (the concepts at the top level in the ontology) were
grouped into three categories: \emph{person}, \emph{animal}, and
\emph{object}. Subsequently, WordNet was consulted with the head
of each candidate, and thus the semantic category of the
antecedent was obtained.

\item This experiment was exclusively carried out with the SemCor corpus because
it is the only one in which content words are
annotated with their WordNet sense.
\end{itemize}

Table
\ref{tabla:anaforaInglesEvaluacionDos} shows the number of pronouns (classified by type) for
the different corpora after these changes were incorporated.

\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|c|c|c|c|c|} \hline & He& She& It& They& Him& Her&
Them& Corr& Total&
P({\%}) \\
\hline SEMCOR& 116& 10& 38& 50& 34& 0& 6& 220& 254&
86.6 \\
\hline MTI& 1& 0& 347& 56& 0& 0& 66& 361& 470&
76.8 \\
\hline
\end{tabular}
\end{center}
\caption{\label{tabla:anaforaInglesEvaluacionDos}Anaphora
resolution in English, evaluation phase: experiment 2}
\end{table}


%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=12.6cm, height=2.5cm, clip]{anaforaInglesEvaluacionDos.eps}
%  \caption{Anaphora resolution in English, evaluation phase: experiment 2}
%  \label{tabla:anaforaInglesEvaluacionDos}
%\end{table}

As shown in Table \ref{tabla:anaforaInglesEvaluacionDos}, the
addition of the two simple semantic constraints resulted in considerable
improvement in the obtained precision (approximately 18\%) for the SemCor
corpus. We concluded that the use of semantic information
(such as new constraints and preferences) in the process of anaphora
resolution will improve the results obtained.

Finally, Table \ref{tabla:anaforaInglesOtrosAutores} compares anaphora resolution using AGIR with the
other approaches
pre\-vious\-ly pre\-sen\-ted\footnote{As mentioned earlier, all the results presented here were automatically
obtained after the anaphoric annotation of each pronoun. After
the tagging and the partial parsing of the source text,
pronominal anaphora were resolved and translated into the
target language. None of the intermediate outputs needed to be
adjusted manually in order to be processed subsequently.}. It is important to emphasize the high percentages
obtained using our system and Hobbs's method in the SemCor corpus; both
systems incorporate semantic information\footnote{Hobbs proposed the use
of semantic information using selectional restrictions as a
straightforward extension of his method in order to improve the
obtained results in anaphora resolution.} into their methods
using semantic constraints (selectional restrictions), whereas
none of the other authors incorporate semantics in their approaches.

\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|} \hline & Proximity& Hobbs& Lappin& Strube&
AGIR \\
\hline SEMCOR& 37.0& 81.9& 59.4& 59.4&
\textbf{86.6} \\
\hline MTI& 54.9& 66.0& 75.1& 63.2&
\textbf{76.8} \\
\hline
\end{tabular}
\caption{\label{tabla:anaforaInglesOtrosAutores}Anaphora
resolution in English, comparison of AGIR with other approaches}
\end{center}
\end{table}


%\begin{table}[htbp]
% \centering
%  \includegraphics[width=10.3cm, height=3.2cm, clip]{anaforaInglesOtrosAutores.eps}
%  \caption{Anaphora resolution in English, comparison of AGIR with other approaches}
%  \label{tabla:anaforaInglesOtrosAutores}
%\end{table}



\section{AGIR's Generation Module}
The interlingua representation of the source text is taken as
input for the ge\-ne\-ra\-tion module. The output of this module is
the target text, that is, the representation of the source text's
meaning with words of the target language.

The generation phase is split into two modules: syntactic
generation and morphological generation. Although the approach
presented here is multilingual, we have focused on the generation
into the Spanish and English languages.

\subsection{Syntactic Generation}
In syntactic generation, the interlingua representation is
converted by \emph{transformational rules} into an ordered
surface-structure tree, with appropriate labeling of the leaves
with target language grammatical functions and features. The
basic task of syntactic generation is to order constituents in
the correct sequence for the target language. However, the aim of
this work is only the translation of pronominal anaphora into the
target language, so we have only focused on the discrepancies
between the Spanish and English languages in the translation of
the pronoun.

In syntactic generation, Spanish elliptical zero-subject
constructions were studied. This phenomenon was
conveniently treated and solved in the analysis module.
Therefore, all the necessary information to translate these
constructions has been stored in the interlingua representation.


\subsection{Morphological Generation}
The final stage of the generation module is morphological
generation, in which we mainly have to treat and solve number and
gender discrepancies in the translation of pronouns.

\subsubsection{Number Discrepancies} This problem is generated by
the discrepancy between words of different lan\-gua\-ges that
express the same concept. These words can be referred to a
singular pronoun in the source language and to a plural pronoun in
the target language. In order to take into account number
discrepancies in the translation of the pronoun into English or
Spanish, a set of morphological (number) rules is constructed. The
lefthand side of the number rule contains the interlingua
representation of the pronoun, whereas the righthand side contains
the pronoun in the target language.



\subsubsection{Gender Discrepancies} Gender discrepancies come from
existing mor\-pho\-lo\-gi\-cal differences between different
languages. For instance, English has less morphological
information than Spanish. The English plural personal pronoun
\emph{they} can be translated into the Spanish pronouns
\emph{ellos} (masculine) or \emph{ellas} (feminine); the singular
personal pronoun \emph{it} can be translated into
\emph{\'el/\'este} (mas\-cu\-li\-ne) or \emph{ella/\'esta}
(feminine), etc. In order to take into account such gender
dis\-cre\-pan\-cies in the translation of the pronoun into English
or Spanish, a set of morphological (gender) rules was constructed.


\section{Evaluation of the Generation Module}

In this step, we tested the AGIR's generation module by evaluating the translation of English pronouns into Spanish,
and the translation of Spanish pronouns
into English.

As mentioned earlier, the generation module takes the interlingua
representation as input. Previously, Spanish zero pronouns were
detected (90.4\% \textbf{P}) and resolved (81.4\%
\textbf{P}), and anaphoric third-person personal pronouns were
resolved in Spanish (82.2\% \textbf{P}) and English (86.6\%
\textbf{P} in the SemCor corpus with semantic information, and 76.8\%
\textbf{P} in the MTI corpus without semantic information).
Non-referential uses of \emph{it} pronouns were
automatically detected, obtaining an 88.7\% \textbf{P} on
unrestricted texts\footnote{In order to detect pleonastic
\emph{it} pronouns in AGIR, a set of rules, based on pattern
recognition, that allow for the identification of this type of
pronoun is constructed. These rules were based on the work
of \cite{Lappin94,Paice87,Denber98}, which
dealt with this problem in a similar way. We have used the
information provided by the POS tagger in order to improve the
detection of the different patterns. We have evaluated the
method using journalistic texts for a portion of the Federal
Register corpus that contains a set of 313 documents (156,831
words). In the detection of pleonastic \emph{it} pronouns a
88.7\% \textbf{P} (568 out of 640) was obtained. Finally, it is very
important to point out the high percentage of \emph{it} pronouns
in the test corpus that are pleonastic (32.9\%). This fact
demonstrates the importance of the correct detection of this kind
of pronoun in any MT system.}.


Once the interlingua representation was obtained, the method
proposed for pronominal anaphora translation into the target
language was based on the treatment of number and gender
discrepancies.

\subsection{Pronominal Anaphora Translation into Spanish}
In this experiment, the translation of English, third-person,
personal pronouns into Spanish was evaluated.

We tested the method on the portions of the SemCor and MTI
corpora used previously in the process of anaphora resolution.
The training corpus was used for improving the number and gender
rules. The remaining fragments of the corpora were reserved for
test data.

We needed to know the semantic category (\emph{person},
\emph{animal}, or \emph{object}) and the grammatical gender
(\emph{masculine} or \emph{feminine}) of the pronoun's antecedent
in order to apply the number and gender rules. In the SemCor corpus,
the WordNet sense was used to identify the antecedent's
semantic category. In the MTI corpus, due to the lack of semantic
information, a set of heuristics was used to determine the
antecedent's semantic category.

With regard to information about the antecedent's gender, an
English--Spanish electronic dictionary was used since the POS
tag does not usually provide gender and number information. The
dictionary was incorporated into the system as a database.
For each English word, the dictionary provides a translation
into Spanish, and  the word's gender and number in Spanish.

The number and gender rules were applied using this
morphological and semantic information. We conducted a blind
test over the entire test corpus, and the obtained results appear
in Table \ref{tabla:generacionINGESPEvaluacion}.

\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|} \hline & Subject& Compl& Correct& Total&
P({\%})  \\
\hline SEMCOR& 197& 47& 229& 254&
90.2 \\
\hline MTI& 239& 231& 353& 470&
75.1 \\
\hline \textbf{TOTAL}& \textbf{436}& \textbf{288}& \textbf{582}&
\textbf{724}&
\textbf{80.4} \\
\hline
\end{tabular}
\caption{\label{tabla:generacionINGESPEvaluacion}Translation of
pronominal anaphora into Spanish, evaluation phase}
\end{center}
\end{table}


%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=12cm, height=9.03cm, clip]{generacionINGESPEvaluacion.eps}
%  \caption{Generation of pronominal anaphora into Spanish, evaluation phase}
%  \label{tabla:generacionINGESPEvaluacion}
%\end{table}

The evaluation of this task was automatically carried out after
the anaphoric annotation of each pronoun. This annotation
included information about the antecedent and the translation
into the target language of the anaphor. To do so, the human
annotators translated the anaphors according to the criteria
established by the morphological rules. For example, the pronoun
\emph{it} with subject function was translated into the
Spanish pronoun \emph{él} if its antecedent was of the \emph{animal} type
and \emph{masculine}; on the other hand, if its antecedent was of the
\emph{object} type and \emph{masculine}, it was translated into
the Spanish pronoun \emph{éste}; and so on. In the Spanish--English
translation, the pronoun \emph{él} with subject function was
translated into the English pronoun \emph{he} if its antecedent
was a \emph{person} type and \emph{masculine}; on the other hand, if
its antecedent was an \emph{object/animal} type and
\emph{masculine/feminine}, it was translated into the English
pronoun \emph{it}; and so on\footnote{In the automatic
evaluation, a pronoun was considered as correctly translated when
the pronoun proposed by the system was the same as that proposed by
the human annotator. With this criterion, we evaluated the correct
application of the corresponding morphological rule.}.

Table \ref{tabla:generacionINGESPEvaluacion} shows the anaphoric
pronouns of each corpus classified by grammatical function:
\emph{subject} and \emph{complement} (direct or indirect object).
The last three columns represent the number of pronouns
successfully solved, the total number of solved pronouns, and the
obtained precision, respectively. For instance, the SemCor
corpus contains 197 pronouns with subject function and 47
complement pronouns. The precision obtained in this
corpus was of 90.2\% (229 out of 254).


\textbf{Discussion}. In the translation of English personal
pronouns in the third person into Spanish, an overall
precision of 80.4\% (582 out of 724) was obtained.
Specifically, 90.2\% \textbf{P} and 75.1\% \textbf{P} were
obtained in the SemCor and MTI corpora, respectively.


From these results, we have extracted the following conclusions:

\begin{itemize}
\item In the SemCor corpus, all of the instances of the English pronouns
\textit{he}, \textit{she}, \textit{him}, and \textit{her} were
correctly translated into Spanish. There are two
reasons for this:

\begin{itemize}
\item The semantic roles of these pronouns were correctly
identified in all of the cases.
\item These pronouns contain the necessary grammatical information
(gender and number) that allows the correct translation into
Spanish, independent of the antecedent proposed as a solution by
the AGIR system.
\end{itemize}

The errors in the translation of the pronouns \textit{it},
\textit{they}, and \textit{them} were o\-ri\-gi\-na\-ted by
the following different causes:

\begin{itemize}
\item There were mistakes in the anaphora-resolution stage, that is, the
antecedent proposed by the system was not the correct one (44.4\%
of the global mistakes). This caused an incorrect translation into
Spanish mainly due to the fact that the proposed antecedent and
the correct one had different grammatical genders.

\item There were mistakes in the identification of the semantic role of the
pronouns that caused the application of an incorrect morphological
rule (44.4\%). These mistakes mainly originated in an
incorrect process of clause splitting.

\item There were mistakes originated by the English--Spanish electronic dictionary
(11.2\%). Two circumstances could occur: (a) the word did not
appear in the dictionary; and (b) the word's gender in the
dictionary was different from the real word's gender, since the word
had different meanings.
\end{itemize}

\item In the MTI corpus, nearly all the pronouns were
\emph{it}, \emph{they}, and \emph{them} (96.2\% of the
total pronouns). The errors in the translation of these pronouns
originated in the same causes as those in the SemCor corpus, although the
percentages were different:

\begin{itemize}
\item There were mistakes in the anaphora-resolution stage (22.9\% of the
mistakes).

\item There were mistakes in the identification of the pronouns' semantic role
(62.9\%).

\item There were mistakes that originated in the English--Spanish dictionary
(14.2\%). In this corpus, there were a large number of technical words that
did not appear in the electronic dictionary.

\end{itemize}

\item After analyzing the results, we observed that the precision
of the SemCor corpus was approximately 15\% higher than that
obtained by the MTI corpus. The lower percentage obtained by the MTI corpus were
the result of the corpus itself (most of the pronouns
in this corpus are \emph{it}, \emph{they}, and
\emph{them}), and of the lack of semantic information.

\end{itemize}

In order to measure the efficiency of our proposal, we compared
our system with one of the most representative MT systems of the
moment: Systran. Systran was designed and built more than thirty
years ago, and it is being continually modified in order to
improve its translation quality. Moreover, it is easily accessible
to Internet users through the service of MT on the
web---BABELFISH\footnote{URL =
\texttt{http://www.babelfish.altavista.com} (visited on
03/11/2002).}---which provides free translations between different
languages. With regard to the problem of pronominal anaphora
resolution and translation, Systran is one of the best MT systems
studied (see Section 2) because, like our own system, it treats
the problems of intersentential pronominal anaphora and Spanish
zero pronouns on unrestricted texts after carrying out a partial
parsing of the source text. As was mentioned in Section 2, a free
trial of the commercial product SYSTRANLinks\footnote{URL =
\texttt{http://w4.systranlinks.com/config} (visited on
06/22/2002).} was used to translate between the English and
Spanish languages the evaluation corpora. The results appear in
Table \ref{tabla:SystranGeneracionINGESP}.


\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|} \hline & SYSTRANLinks& AGIR \\
\hline SEMCOR& 75.4& \textbf{82.5} \\
\hline MTI& 58.1& \textbf{69.3} \\
\hline
\end{tabular}
\caption{\label{tabla:SystranGeneracionINGESP}Translation of
pronominal anaphora (complement pronouns only) into Spanish, SYSTRANLinks and AGIR}
\end{center}
\end{table}

The evaluation of the SYSTRANLinks output was carried out by a
human translator by hand. Pronouns judged as acceptable by the
translator were considered correctly translated; otherwise, they
were considered incorrectly translated.

Table \ref{tabla:SystranGeneracionINGESP} only shows the
evaluation of English complement pronoun translation into Spanish
because Systran did not translate all the subject
pronouns into Spanish. By analyzing the Systran outputs of both
corpora, we extracted the following conclusions:

\begin{itemize}
\item All the instances of the English pronouns \emph{he} and
\emph{she} (always with subject function) were correctly
translated into their Spanish equivalents \emph{él} and \emph{ella}.

\item All the instances of the English pronouns
\emph{it} and \emph{they} with subject function were omitted in
Spanish---zero pronouns. These pronouns were not resolved in
English, and subsequently were not translated into Spanish.
\end{itemize}

On the other hand, in our AGIR system, we have evaluated the
correct application of the morphological rule to translate all
source pronouns into target pronouns. A subsequent task must
decide if the pronoun in the target language (a) must be
generated as our system proposes, (b) must be substituted by
another kind of pronoun (e.g., a possessive pronoun), or (c) must
be eliminated (i.e., Spanish zero pronouns). Therefore, we have
only taken into account the complement pronoun translation in
order to make a fair comparison between the two systems.

As shown in Table \ref{tabla:SystranGeneracionINGESP}, the
precision obtained using AGIR is approximately 7--11\% higher
(depending on the corpus) than the one obtained using Systran. The
errors in Systran originated in mistakes in the
anaphora-resolution stage that caused incorrect translations,
since the proposed antecedents and the correct ones have different
grammatical gender. These errors can occur in intrasentential
anaphors (as presented in Section 2) or in intersentential
anaphors, as in the following example extracted from the corpora:


\begin{itemize}
\item(\textbf{E}) \emph{[This information]$_i$} is only valid for Linux
on the Intel platform. Much of \emph{it$_i$} should be applicable
to Linux on other processor architectures, but I have no first hand experience or information.
\item (\textbf{S}) Esta información es solamente válida
para Linux en la plataforma de Intel. Mucho de \emph{él} debe ser
aplicable a Linux en otras configuraciones del procesador, pero
no tengo ninguna experiencia o información de primera mano.
\end{itemize}

This example shows an incorrect English--Spanish translation of
the pronoun \emph{it} done by Systran. In this case, the
antecedent (\emph{this information}, feminine) is in the previous
sentence to the anaphor. It is incorrectly solved, and then it is
incorrectly translated (the pronoun \emph{él}---masculine---instead
of the pronoun \emph{ésta}---feminine).


\subsection{Pronominal Anaphora Translation into English}
In this experiment, the translation of Spanish, third-person,
personal pronouns and zero pronouns (excluding reflexive
pronouns) into English was evaluated. We
tested the method on the portion of the LEXESP corpus that was previously used
in the process of anaphora resolution.

We needed to know the semantic category and the grammatical gender
of the pronoun's antecedent in order to apply the number and
gender rules. In the LEXESP corpus, due to the lack of semantic
information, a set of heuristics was used to determine the
antecedent's semantic category. On the other hand, the information
about the antecedent's gender was provided by the POS tag of the
antecedent's head. We conducted a blind test over the entire test
corpus, and the results appear in Table
\ref{tabla:generacionESPINGEvaluacion}.

\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|c|c|c|} \hline & Subject& Compl& Correct& Total&
P({\%})  \\
\hline LEXESP& 630& 145& 657& 775&
84.8 \\
\hline
\end{tabular}
\end{center}
\caption{\label{tabla:generacionESPINGEvaluacion}Translation of
pronominal anaphora into English, evaluation phase}
\end{table}


%\begin{table}[htbp]
%  \centering
%  \includegraphics[width=11.3cm, height=12.35cm, clip]{generacionESPINGEvaluacion.eps}
%  \caption{Generation of pronominal anaphora into English, evaluation phase}
%  \label{tabla:generacionESPINGEvaluacion}
%\end{table}


\textbf{Discussion}. In the translation of Spanish personal
pronouns in the third person into English, an overall
precision of 84.8\% (657 out of 775) was obtained. From
these results, we extracted the following conclusions:

\begin{itemize}
\item All the instances of the Spanish plural pronouns
(\textit{ellos}, \textit{ellas}, \textit{les}, \emph{los},
\emph{las}, and the zero pronouns in plural corresponding to the
English pronouns \emph{they} and \emph{them}), were correctly
translated into English. There are two reasons for this:

\begin{itemize}
\item The semantic roles of these pronouns were correctly
identified in all of the cases.
\item The equivalent English pronouns (\emph{they} and
\emph{them}) lack gender in\-for\-ma\-tion, that is, they are valid for
masculine and feminine. Therefore, the antecedent's gender did
not influence the translation of these pronouns.

\end{itemize}
\end{itemize}


\begin{itemize}
\item The errors occurred in the translation of the Spanish singular
pronouns (\emph{\'el}, \emph{ella}, \emph{le}, \emph{lo},
\emph{la}, and in zero pronouns in singular corresponding to the
English pronouns \emph{he}, \emph{she}, \emph{it}, \emph{him}, and
\emph{her}). There were different causes for these errors:

\begin{itemize}
\item There were mistakes in the anaphora-resolution stage (79.7\% of the global
mistakes), which caused an incorrect translation into Spanish,
mainly due to the proposed antecedent and the correct one having
different grammatical gender. Sometimes both had the same
gender, but they had different semantic categories.

\item There were mistakes in the application of the heuristic used to identify the
an\-te\-ce\-dent's semantic category (20.3\%). This involved
the application of an incorrect morphological rule.


\end{itemize}

\end{itemize}


Our proposal was compared with the SYSTRANLinks output. As
shown in Table \ref{tabla:SystranGeneracionESPING}, the precision
obtained by the AGIR system was approximately 28\% higher than that
obtained by Systran.



\begin{table}[htpb]
\begin{center}
\begin{tabular}
{|c|c|c|} \hline & SYSTRANLinks& AGIR \\
\hline LEXESP& 56.9& \textbf{84.8} \\
\hline
\end{tabular}
\caption{\label{tabla:SystranGeneracionESPING}Translation of
pronominal anaphora into English, SYSTRANLinks and AGIR}
\end{center}
\end{table}


The low results obtained in Systran are mainly the result of
errors that occurred in the translation of Spanish zero pronouns.
Specifically, out of 775 Spanish pronouns, 334 errors occurred, and 293 of them (87.7\% of the global
errors) originated in the translation of zero pronouns, whereas
the remainder (12.3\%) originated in the translation
of the remaining not-omitted pronouns. The errors in the
translation of zero pronouns mainly originated in their
incorrect resolution.

\vspace{1cm}

\section{Conclusion}
In this paper we have evaluated the different
tasks carried out in our MT approach (for Spanish and English
languages) that allowed the correct pronominal anaphora translation
into the target language. We have shown the importance of the
resolution of anaphoric expressions in any MT system for
correct translations into the target language, and how the main MT
systems do not conveniently resolve this phenomenon.

Our approach, called AGIR, works on unrestricted texts to which
partial-parsing techniques have been applied. After parsing and
solving NLP problems, an interlingua representation of the entire
text is obtained. This fact is one of the main advantages of our
system since several problems (hardly solved by the majority of MT
systems) can be treated and solved. These problems are the
translation of intersentential anaphora, the detection of
co-reference chains, and the translation of Spanish zero pronouns
into English.

In the evaluation, we obtained a
precision of 80.4\% and 84.8\% in the translation of Spanish and
English pronominal anaphora, respectively. Previously, Spanish zero
pronouns had been resolved (with a precision of 81.4\%) and
anaphoric personal pronouns had been resolved in English (with
precisions of 86.6\% and 76.8\% in the SemCor corpus with semantic
information and in the MTI corpus without it, respectively) and in Spanish
(with a precision of 82.2\%).

In addition, we carried out an exhaustive comparison with some
well-known anaphora-resolution algorithms. Finally, we also
compared pronoun translation with one of the most representative
MT systems at the moment: Systran. In all of these comparisons,
AGIR was shown to perform better.

A very important conclusion was extracted during the evaluation
phase: the adding of semantic information improves the precision
of the anaphora-resolution process considerably, and therefore the
corresponding precision of the anaphora-translation process.
Currently, the addition of this kind of information in the
different stages of the AGIR system is being studied in order to
improve the overall performance of the system.

The resolution and translation of new types of references, such as
definite descriptions or anaphora originated by demonstrative
pronouns, will be studied in the future. Moreover, the
addition of new languages to the interlingua approach will be
taken into account.


\section*{Acknowledgments}
The authors wish to thank Manuel Palomar for his helpful
revisions of this paper; Ferrán Pla, Ruslan Mitkov, and Richard
Evans for having contributed their corpora; and Rafael Mu\~noz,
Maximiliano Saiz-Noeda, Patricio Mart\'{\i}nez-Barco, and Juan
Carlos Trujillo for their suggestions and willingness to help in any task related
to this paper. We are also grateful for the helpful comments of the anonymous
reviewers of several conference papers in which we presented our preliminary work.

This research has been supported by the Spanish Government, under
projects TIC2000-0664-C02-02 and FIT-150500-2002-416.

\nocite{Berger94}
\nocite{Canals2001a}
\nocite{Canals2001b}
%------------------------------------------------------------------


\bibliographystyle{theapa}
\bibliography{peral03a}

\end{document}
