The BLEU (Bilingual Evaluation Understudy) score is an algorithm used for evaluating the quality of text that has been machine-translated from one natural language to another. It was invented at IBM in 2001 and is one of the first metrics to claim a high correlation with human judgments of quality.
The BLEU score is calculated by comparing the machine-translated text (candidate) with one or more professionally human-translated texts (references). The quality is considered to be the correspondence between a machine's output and that of a human. The central idea behind BLEU is that "the closer a machine translation is to a professional human translation, the better it is".
The BLEU score assesses machine translation quality by measuring how closely it matches human translation. It combines n-gram precision and a brevity penalty for a comprehensive evaluation. N-gram precision counts the shared n-grams (1 to 4 words in sequence) between the machine and human translations, divided by the total n-grams in the machine translation. The brevity penalty applies when the machine translation is shorter than the reference, with a penalty factor of 1 if the machine translation is equal to or longer than the reference.
The score is calculated for individual translated segments—generally sentences—and then averaged over the whole corpus to reach an estimate of the translation's quality. Intelligibility or grammatical correctness are not taken into account.
The BLEU score is always a number between 0 and 1. This value indicates how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. A BLEU score of 1 means that the candidate sentence perfectly matches one of the reference sentences. However, even human translators do not achieve a perfect score of 1.0.
The BLEU score is calculated by comparing the n-grams of machine-translated sentences to the n-grams of human-translated sentences. An n-gram is a contiguous sequence of n items from a given sample of text or speech. The BLEU score also incorporates a brevity penalty to prevent the system from favoring shorter translations.
A good BLEU score varies based on the dataset, language pair, and preprocessing choices. Scores are most meaningful for comparing systems on the same test set under the same evaluation setup. Higher is generally better, but even strong BLEU scores can miss semantic errors or reward n-gram overlap that does not reflect true translation quality.
There is no universal threshold for a good BLEU score. Interpretation depends on domain, language pair, tokenization, smoothing choices, and the number of reference translations. Teams often pair BLEU with human review or metrics like COMET or chrF because BLEU does not directly measure meaning or factual accuracy.
The BLEU Score (Bilingual Evaluation Understudy) is a key metric in machine translation, assessing the quality of text by measuring its similarity to human reference translations. It evaluates the precision of n-grams—sequences of n items from the text—and incorporates a brevity penalty to discourage overly short translations. Despite its utility, the BLEU score has limitations; it does not account for semantic meaning and may inaccurately rate grammatically correct but contextually flawed translations. Nevertheless, its efficiency and simplicity make it a widely used standard in natural language processing for tasks like translation and text summarization.
The BLEU score evaluates machine translation by calculating the precision of n-grams—word sequences that appear in both the machine-generated and human reference translations. It adjusts for length by applying a brevity penalty to shorter translations, ensuring that neither length nor brevity is unfairly rewarded.
A geometric mean of the precision scores across different n-gram lengths is computed and then multiplied by the brevity penalty to produce the final BLEU score.
The BLEU score offers several advantages, such as providing a quantitative measure for comparing machine and human translations, being computationally efficient, and incorporating n-gram precision to reflect certain linguistic structures.
However, it has limitations, including a lack of semantic analysis, potential overrating of grammatically correct but contextually inaccurate translations, dependency on reference translations that may not cover all correct possibilities, and not always being the most suitable metric for every application.
Additionally, the selection of n-gram length and the method for applying the brevity penalty can introduce subjectivity into the scoring process.
More terms
Glossary term
Glossary term
Collaborate with your team on reliable Generative AI features.
Want expert guidance? Book a 1:1 onboarding session from your dashboard.