Wikilinks: A Large-scale Cross-Document Coreference Corpus Labeled via Links to Wikipedia (Original Dataset)
Sameer Singh and Amarnag Subramanya and Fernando Pereira and Andrew McCallum

wiki-links (10 files)
data-00000-of-00010.gz 183.50MB
data-00001-of-00010.gz 183.74MB
data-00002-of-00010.gz 183.72MB
data-00003-of-00010.gz 184.04MB
data-00004-of-00010.gz 183.46MB
data-00005-of-00010.gz 183.76MB
data-00006-of-00010.gz 183.77MB
data-00007-of-00010.gz 183.87MB
data-00008-of-00010.gz 184.39MB
data-00009-of-00010.gz 183.68MB
Type: Dataset

title= {Wikilinks: A Large-scale Cross-Document Coreference Corpus Labeled via Links to Wikipedia (Original Dataset)},
author= {Sameer Singh and Amarnag Subramanya and Fernando Pereira and Andrew McCallum},
abstract= {Cross-document coreference resolution is the task of grouping the entity mentions in a collection of documents into sets that each represent a distinct entity. It is central to knowledge base construction and also useful for joint inference with other NLP components. Obtaining large, organic labeled datasets for training and testing cross-document coreference has previously been difficult. We use a method for automatically gathering massive amounts of naturally-occurring cross-document reference data to create the Wikilinks dataset comprising of 40 million mentions over 3 million entities. Our method is based on finding hyperlinks to Wikipedia from a web crawl and using anchor text as mentions. In addition to providing large-scale labeled data without human effort, we are able to include many styles of text beyond newswire and many entity types beyond people.

### Introduction

 The Wikipedia links (WikiLinks) data consists of web pages that
 satisfy the following two constraints:

a. contain at least one hyperlink that points to Wikipedia, and
b. the anchor text of that hyperlink closely matches the title of the target Wikipedia page.

  We treat each page on Wikipedia as representing an entity
  (or concept or idea), and the anchor text as a mention of the
  entity. The WikiLinks data set was obtained by iterating 
  over Google's web index. 

####  Content

  This dataset is accompanied by the following tech report:

  Please cite the above report if you use this data.

  The dataset is divided over 10 gzipped text files
  data-0000[0-9]-of-00010.gz. Each of these files can be viewed
  without uncompressing them using zcat. For example:

  zcat data-00001-of-00010.gz | head 



  MENTION	vacuum tube	421

  MENTION	vacuum tubes	10838

  MENTION	electron gun	598

  MENTION	fluorescent	790

  MENTION	oscilloscope	1307

  MENTION	computer monitor	1503

  MENTION	computer monitors	3066

  MENTION	radar	1657

  MENTION	plasma screens	2162

  Each file is in the following format:














  where each web-page is identified by its url (annotated
  by "URL"). For every mention (denoted by "MENTION"), we provide the
  actual mention string, the byte offset of the mention from the start
  of the page and the target url all separated by a tab. It is
  possible (and in many cases very likely) that the contents of a
  web-page may change over time. The dataset also contains information
  about the top 10 least frequent tokens on that page at the time
  it was crawled. These line started with a "TOKEN" and contain
  the string of the token and the byte offset from the start of the page.
  These token strings can be used as fingerprints
  to verify if the page used to generate the data has changed. Finally,
  pages are separated from each other by two blank lines.

####  Basic Statistics

  Number of Document: 11 million
  Number of entities:  3 million
  Number of mentions: 40 million

  Finally please note that this dataset was created automatically
  from the web and therefore contains some amount of noise.


  Amar Subramanya (

  Sameer Singh (

  Fernando Pereira (

  Andrew McCallum (
keywords= {},
terms= {},
license= {Attribution 3.0 Unported (CC BY 3.0) Human-Readable Summary}

Hosted by users:

10 day statistics (1 downloads)

Average Time 1 mins, 05 secs
Average Speed 28.28MB/s
Best Time 1 mins, 05 secs
Best Speed 28.28MB/s
Worst Time 1 mins, 05 secs
Worst Speed 28.28MB/s