English Gigaword Fifth Edition
| Item Name: | English Gigaword Fifth Edition |
|---|---|
| Author(s): | Robert Parker, David Graff, Junbo Kong, Ke Chen, Kazuaki Maeda |
| LDC Catalog No.: | LDC2011T07 |
| ISBN: | 1-58563-581-2 |
| ISLRN: | 911-942-430-413-0 |
| DOI: | |
| Release Date: | June 17, 2011 |
| Member Year(s): | 2011 |
| DCMI Type(s): | Text |
| Data Source(s): | newswire |
| Project(s): | GALE |
| Application(s): | language modeling, information retrieval |
| Language(s): | English |
| Language ID(s): | eng |
| License(s): | LDC User Agreement for Non-Members |
| Online Documentation: | LDC2011T07 Documents |
| Licensing Instructions: | Subscription & Standard Members, and Non-Members |
| Citation: | Parker, Robert, et al. English Gigaword Fifth Edition LDC2011T07. Web Download. Philadelphia: Linguistic Data Consortium, 2011. |
| Related Works:Hide | isVersionOfLDC2003T05 English GigawordLDC2005T12 English Gigaword Second EditionLDC2007T07 English Gigaword Third EditionLDC2009T13 English Gigaword Fourth EditionhasAnnotationLDC2012T21 Annotated English GigawordLDC2016T07 DEFT Narrative TextLDC2017T09 The EventStatus CorpusLDC2018T20 Concretely Annotated English GigawordisOutcomeOfLDC95T21 North American News Text CorpusLDC98T30 North American News Text Supplement |
Introduction
English Gigaword Fifth Edition is a comprehensive archive of newswire text data that has been acquired over several years by the Linguistic Data Consortiume (LDC). The fifth edition includes all of the contents in English Gigaword Fourth Edition (LDC2009T13) plus new data covering the 24-month period January 2009 through December 2010.
The seven distinct international sources of English newswire included in this edition are the following:
- Agence France-Presse, English Service (afp_eng)
- Associated Press Worldstream, English Service (apw_eng)
- Central News Agency of Taiwan, English Service (cna_eng)
- Los Angeles Times/Washington Post Newswire Service (ltw_eng)
- Washington Post/Bloomberg Newswire Service (wpb_eng)
- New York Times Newswire Service (nyt_eng)
- Xinhua News Agency, English Service (xin_eng)
The seven letter codes in the parentheses above include the three-character source name abbreviations and the three-character langauge code (eng) separated by an underscore (_) character. The three-letter language code conforms to LDC's internal convention based on the ISO 639-3 standard.
Data
The following table sets forth the overall totals for each source. Note that Total-MB refers to the quantity of date when unzipped (approximately 26 gigabytes), Gzip-MB refers to compressed file sizes as stored on the DVD-ROMs and K-wrds refers to the number of whitespace-separated tokens (of all types) after all SGML tags are eliminated:
| Source | #Files | Gzip-MB | Totl-MB | K-wrds | #DOCs |
|---|---|---|---|---|---|
| afp_eng | 146 | 1732 | 4937 | 738322 | 2479624 |
| apw_eng | 193 | 2700 | 7889 | 1186955 | 3107777 |
| cna_eng | 144 | 86 | 261 | 38491 | 145317 |
| ltw_eng | 127 | 651 | 1694 | 268088 | 411032 |
| nyt_eng | 197 | 3280 | 8938 | 1422670 | 1962178 |
| wpb_eng | 12 | 42 | 111 | 17462 | 26143 |
| xin_eng | 191 | 834 | 2518 | 360714 | 1744025 |
| TOTAL | 1010 | 9325 | 26348 | 4032686 | 9876086 |
