METEOR


METEOR

METEOR (Metric for Evaluation of Translation with Explicit ORdering) is a metric for the evaluation of machine translation output. The metric is based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision. It also has several features that are not found in other metrics, such as stemming and synonymy matching, along with the standard exact word matching. The metric was designed to fix some of the problems found in the more popular BLEU metric, and also produce good correlation with human judgement at the sentence or segment level This differs from the BLEU metric in that BLEU seeks correlation at the corpus level.Results have been presented which give correlation of up to 0.964 with human judgement at the corpus level, compared to BLEU's achievement of 0.817 on the same data set. At the sentence level, the maximum correlation with human judgement achieved was 0.403.ref|Banerjee2005

Algorithm

As with BLEU, the basic unit of evaluation is the sentence, the algorithm first creates an "alignment" (see illustrations) between two sentences, the candidate translation string, and the reference translation string. The "alignment" is a set of "mappings" between unigrams. A mapping can be thought of as a line between a unigram in one string, and a unigram in another string. The constraints are as follows; every unigram in the candidate translation must map to zero or one unigram in the reference translation and vice versa. In any alignment, a unigram in one string cannot map to more than one unigram in another string.

An alignment is created incrementally through a series of "stages", which are controlled by "modules". A module is simply a matching algorithm, for example the "wn_synonymy" module maps synonyms using WordNet, while the "exact" module matches exact words. Examples are given as follows:

Each stage is split up into two "phases". In the first phase, all possible unigram mappings are collected for the module being used in this stage. In the second phase, the largest subset of these mappings is selected to produce an "alignment" as defined above. If there are two alignments with the same number of mappings, the alignment is chosen with the fewest "crosses", that is, with fewer intersections of two mappings. From the two alignments shown, alignment (a) would be selected at this point. Stages are run consecutively and each stage only adds to the alignment those unigrams which have not been matched in previous stages. Once the final alignment is computed, the score is computed as follows: Unigram precision P is calculated as:

Score: 0.9977 = Fmean: 1.0000 * (1 - Penalty: 0.0023)Fmean: 1.0000 = 10 * Precision: 1.0000 * Recall: 1.0000 / Recall: 1.0000 + 9 * Precision: 1.0000Penalty: 0.0023 = 0.5 * (Fragmentation: 0.1667 ^3) Fragmentation: 0.1667 = Chunks: 1.0000 / Matches: 6.0000

Score: 0.9654 = Fmean: 0.9836 * (1 - Penalty: 0.0185)Fmean: 0.9836 = 10 * Precision: 0.8571 * Recall: 1.0000 / Recall: 1.0000 + 9 * Precision: 0.8571Penalty: 0.0185 = 0.5 * (Fragmentation: 0.3333 ^3)Fragmentation: 0.3333 = Chunks: 2.0000 / Matches: 6.0000

Notes

# Banerjee, S. and Lavie, A. (2005)

References

* Banerjee, S. and Lavie, A. (2005) "METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments" in "Proceedings of Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43rd Annual Meeting of the Association of Computational Linguistics (ACL-2005), Ann Arbor, Michigan, June 2005"
* Lavie, A., Sagae, K. and Jayaraman, S. (2004) "The Significance of Recall in Automatic Metrics for MT Evaluation" in "Proceedings of AMTA 2004, Washington DC. September 2004"

External links

* [http://www.cs.cmu.edu/~alavie/METEOR/ The METEOR Automatic Machine Translation Evaluation System] (including link for download)


Wikimedia Foundation. 2010.

Synonyms:

Look at other dictionaries:

  • Meteor FL.54 — Meteor FL.53 Les Meteor FL.53, FL.54 et FL.55 sont des avions monoplans de tourisme italiens dérivés de l Avia FL.3. Origine En 1953 Meteor S.p.A. achète l entreprise Francis Lombardi Cie et se lance dans la construction d’avions de tourisme. A… …   Wikipédia en Français

  • Meteor FL.55 — Meteor FL.53 Les Meteor FL.53, FL.54 et FL.55 sont des avions monoplans de tourisme italiens dérivés de l Avia FL.3. Origine En 1953 Meteor S.p.A. achète l entreprise Francis Lombardi Cie et se lance dans la construction d’avions de tourisme. A… …   Wikipédia en Français

  • meteor — METEÓR, meteori, s.m. 1. Fenomen atmosferic de natură apoasă, electrică sau optică. 2. Fenomen care constă în apariţia, pe bolta cerească, a unui punct luminos (reprezentând corpuscule cosmice pătrunse în atmosfera terestră) care se mişcă cu mare …   Dicționar Român

  • METEOR — Cette page d’homonymie répertorie les différents sujets et articles partageant un même nom. Meteor, ligne du métro mise en service en 1998 à Paris. Meteor, magazine de science fiction. Meteor, brasseur implanté à Hochfelden, en Alsace. Meteor, un …   Wikipédia en Français

  • Météor — Meteor Cette page d’homonymie répertorie les différents sujets et articles partageant un même nom. Meteor, ligne du métro mise en service en 1998 à Paris. Meteor, magazine de science fiction. Meteor, brasseur implanté à Hochfelden, en Alsace.… …   Wikipédia en Français

  • Meteor! — is a 1987 children s picture book by author Patricia Polacco. Polacco is well known for writing and illustrating novels depicting events from her childhood in Michigan. Meteor! was published in 1987 by The Trumpet Club, commonly known for… …   Wikipedia

  • Meteor S.p.A. — Meteor S.p.A. Costruzioni Aeronautiche est une entreprise italienne de construction aéronautique, créée en 1947, basée dans la région Frioul Vénétie julienne. Historique La société Meteor S.p.A. a été fondée à Monfalcone le 20 mars 1947 …   Wikipédia en Français

  • Meteor — Sm erw. fach. (17. Jh.) Entlehnung. Entlehnt aus gr. metéōron n. Himmelserscheinung , einer Substantivierung von gr. metéōros in der Luft schwebend , zu gr. metaírein wegheben , zu gr. aírein heben und gr. metá .    Ebenso nndl. meteoor, ne.… …   Etymologisches Wörterbuch der deutschen sprache

  • Meteor — Me te*or, n. [F. m[ e]t[ e]ore, Gr. ?, pl. ? things in the air, fr. ? high in air, raised off the ground; ? beyond + ?, ?, a suspension or hovering in the air, fr. ? to lift, raise up.] [1913 Webster] 1. Any phenomenon or appearance in the… …   The Collaborative International Dictionary of English

  • meteor — metèōr m <G meteóra> DEFINICIJA 1. zast. svi događaji u atmosferi 2. astron. dio svemirskog tijela koje, padajući kroz atmosferu, ostavlja svijetleći trag FRAZEOLOGIJA kao meteor blistavo i kratko ETIMOLOGIJA grč. metéōros: uzdignut u zrak …   Hrvatski jezični portal


Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”

We are using cookies for the best presentation of our site. Continuing to use this site, you agree with this.