Text corpus

From Wikipedia, the free encyclopedia

In linguistics, a corpus (plural corpora) or text corpus is a large and structured set of texts (now usually electronically stored and processed). They are used to do statistical analysis, checking occurrences or validating linguistic rules on a specific universe.

A corpus may contain texts in a single language (monolingual corpus) or text data in multiple languages (multilingual corpus). Multilingual corpora that have been specially formatted for side-by-side comparison are called aligned parallel corpora.

In order to make the corpora more useful for doing linguistic research, they are often subjected to a process known as annotation. An example of annotating a corpus is part-of-speech tagging, or POS-tagging, in which information about each word's part of speech (verb, noun, adjective, etc.) is added to the corpus in the form of tags. Another example is indicating the lemma (base) form of each word. When the language of the corpus is not a working language of the researchers who use it, interlinear glossing is used to make the annotation bilingual.

Corpora are the main knowledge base in corpus linguistics. The analysis and processing of various types of corpora are also the subject of much work in computational linguistics, speech recognition and machine translation, where they are often used to create hidden Markov models for POS-tagging and other purposes. Corpora and frequency lists derived from them are useful for language teaching.

Text corpora are also used in the study of historical documents, for example in attempts to decipher ancient scripts, or in Biblical scholarship.

[edit] Some notable text corpora

English language:

American National Corpus
Bank of English
British National Corpus
Brown Corpus
Helsinki Corpus
Longman-Lancaster Corpus
North American News Text corpus
Oxford English Corpus
Scottish Corpus of Texts & Speech

Historical languages:

Thesaurus Linguae Graecae (Ancient Greek)
Electronic Text Corpus of Sumerian Literature
Neo-Assyrian Text Corpus Project

Other languages:

Bilingual corpora:

Evrokorpus English-Slovene parallel corpus
COMPARA Portuguese-English parallel corpus
EuroParl Parallel corpora including 11 European languages: Romanic (French, Italian, Spanish, Portuguese), Germanic (English, Dutch, German, Danish, Swedish), Greek and Finnish. One of the most used corpora on Natural Language Processing.
JRC-Acquis The JRC-Acquis Multilingual Parallel Corpus, includes the languages: Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Slovak, Slovene and Swedish.