Text - Academic Torrents

Synthetic Data for Text Localisation in Natural Images (Dataset)

This is a synthetically generated dataset, in which word instances are placed in natural scene images, while taking into account the scene layout. The dataset consists of *800 thousand* images with approximately *8 million* synthetic word instances. Each text instance is annotated with its text-string, word-level and character-level bounding-boxes.

Reading Text in the Wild with Convolutional Neural Networks (Dataset)

The exact data used to train our deep convolutional neural networks (see our [research page]()) is included in this torrent. This is synthetically generated dataset which we found sufficient for training text recognition on real-world images ![Synthetic Data Engine processt]() This dataset consists of *9 million images* covering *90k English words*, and includes the training, validation and test splits used in our work.

PMC Open Access Subset (Dataset)

mirror of ftp.ncbi.nlm.nih.gov:/pub/pmc/oa_bulk PubMed Central® (PMC) is a free full-text archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health s National Library of Medicine (NIH/NLM). The PMC Open Access Subset some or all openaccess content is a part of the total collection of articles in PMC. The articles in the OA Subset are made available under a Creative Commons or similar license that generally allows more liberal redistribution and reuse than a traditional copyrighted work.

r/WritingPrompts, Text (2018) (Dataset)

r/WritingPrompts data, formatted for GPT-2 training.

OpenWebText (Gokaslan's distribution, 2019), GPT-2 Tokenized (Dataset)

Code by eukaryote31 and Joshua Peterson: and Scraped by Aaron Gokaslan and Vanya Cohen: Tokenized by eukaryote31

Flickr8k Dataset (Dataset)

8,000 photos and up to 5 captions for each photo. We introduce a new benchmark collection for sentence-based image description and search, consisting of 8,000 images that are each paired with five different captions which provide clear descriptions of the salient entities and events. … The images were chosen from six different Flickr groups, and tend not to contain any well-known people or locations, but were manually selected to depict a variety of scenes and situations ## Citation Hodosh, Micah, Peter Young, and Julia Hockenmaier. "Framing image description as a ranking task: Data, models and evaluation metrics." Journal of Artificial Intelligence Research 47 (2013): 853-899.

Common Crawl corpus - training-parallel-commoncrawl.tgz (CS-EN, DE-EN, ES-EN, FR-EN, RU-EN) (Dataset)

@article{, title= {Common Crawl corpus - training-parallel-commoncrawl.tgz (CS-EN, DE-EN, ES-EN, FR-EN, RU-EN)}, keywords= {}, author= {}, abstract= {}, terms= {}, license= {}, superseded= {}, url= {} }

UN corpus - training-parallel-un.tgz (ES-EN, FR-EN) (Dataset)

@article{, title= {UN corpus - training-parallel-un.tgz (ES-EN, FR-EN)}, keywords= {}, author= {WMT13}, abstract= {}, terms= {}, license= {}, superseded= {}, url= {} }

Europarl v7 - training-parallel-europarl-v7.tgz (CS-EN, DE-EN, ES-EN, FR-EN) (Dataset)

@article{, title= {Europarl v7 - training-parallel-europarl-v7.tgz (CS-EN, DE-EN, ES-EN, FR-EN)}, keywords= {}, author= {WMT16}, abstract= {}, terms= {}, license= {}, superseded= {}, url= {} }

Phishing corpus (Dataset)

Downloaded at (2015-02-01)

30M Factoid Question-Answer Corpus (30MQA) (Dataset)

The 30M Factoid Question-Answer Corpus consists of 30M natural language questions in English and their corresponding facts in the knowledge base Freebase. The dataset is formatted as a text file, where each line contains: \t \t \t natural language question, where , and are the subject, relationship and object identifier in Freebase corresponding to the natural language question. For a more detailed description, have a look at our paper: Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus Sample: <>www.freebase.com/book/written_work/subjects<>what is the book e about ? <>www.freebase.com/music/release_track/release<>in what release does the release track cardiac arrest come from ? <>www.freebase.com/film/film/country<>what country is the debt from ? <>www.freebase.com/music/producer/tracks_produced<>what songs have nobuo uematsu produced ? <>www.freebase.com/music/release/producers<>who produced eve-olution ? <>www.freebase.com/music/recording/artist<>which artist recorded most of us are sad ?

Indiana University - Chest X-Rays (XML Reports) (Dataset)

1000 radiology reports for the chest x-ray images from the Indiana University hospital network.

Yelp reviews - Polarity (Dataset)

1,569,264 samples from the Yelp Dataset Challenge 2015. This subset has 280,000 training samples and 19,000 test samples in each polarity.

Yelp reviews - Full (Dataset)

1,569,264 samples from the Yelp Dataset Challenge 2015. This full dataset has 130,000 training samples and 10,000 testing samples in each star.

Sogou news (Dataset)

2,909,551 news articles from the SogouCA and SogouCS news corpora, in 5 categories. The number of training samples selected for each class is 90,000 and testing 12,000. Note that the Chinese characters have been converted to Pinyin.

DBPedia ontology (Dataset)

40,000 training samples and 5,000 testing samples from 14 nonoverlapping classes from DBpedia 2014.

Amazon reviews - Polarity (Dataset)

34,686,770 Amazon reviews from 6,643,669 users on 2,441,053 products, from the Stanford Network Analysis Project (SNAP). This subset contains 1,800,000 training samples and 200,000 testing samples in each polarity sentiment.

Amazon reviews - Full (Dataset)

34,686,770 Amazon reviews from 6,643,669 users on 2,441,053 products, from the Stanford Network Analysis Project (SNAP). This full dataset contains 600,000 training samples and 130,000 testing samples in each class.

AG News (Dataset)

496,835 categorized news articles from >2000 news sources from the 4 largest classes from AG’s corpus of news articles, using only the title and description fields. The number of training samples for each class is 30,000 and testing 1900.

WMT 2015 French/English parallel texts (Dataset)

French/English parallel texts for training translation models. Over 20 million sentences in French and English. Dataset created by Chris Callison-Burch, who crawled millions of web pages and then used a set of simple heuristics to transform French URLs onto English URLs, and assumed that these documents are translations of each other.

Wikitext-2 (Dataset)

A subset of Wikitext-103; useful for testing language model training on smaller datasets.

Wikitext-103 (Dataset)

A collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. Widely used for language modeling, including the pretrained models used in the fastai library and ULMFiT algorithm.

IMDb Large Movie Review Dataset (Dataset)

A dataset for binary sentiment classification containing 25,000 highly polarized movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well.

Microsoft Academic Graph - 2016/02/05 (Dataset)

The Microsoft Academic Graph is a heterogeneous graph containing scientific publication records, citation relationships between those publications, as well as authors, institutions, journals, conferences, and fields of study. This graph is used to power experiences in Bing, Cortana, and in Microsoft Academic.

MovieLens 20M Dataset (Dataset)

Stable benchmark dataset. 20 million ratings and 465,000 tag applications applied to 27,000 movies by 138,000 users. Includes tag genome data with 12 million relevance scores across 1,100 tags. Released 4/2015; updated 10/2016 to update links.csv and add tag genome data. ### Summary This dataset (ml-20m) describes 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 20000263 ratings and 465564 tag applications across 27278 movies. These data were created by 138493 users between January 09, 1995 and March 31, 2015. This dataset was generated on October 17, 2016. Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided. The data are contained in six files, genome-scores.csv, genome-tags.csv, links.csv, movies.csv, ratings.csv and tags.csv. More details about the contents and use of all these files follows. This and other GroupLens data sets are publicly available for download at . #### Further Information About GroupLens GroupLens is a research group in the Department of Computer Science and Engineering at the University of Minnesota. Since its inception in 1992, GroupLens s research projects have explored a variety of fields including: recommender systems online communities mobile and ubiquitious technologies digital libraries local geographic information systems GroupLens Research operates a movie recommender based on collaborative filtering, MovieLens, which is the source of these data. We encourage you to visit to try it out! If you have exciting ideas for experimental work to conduct on MovieLens, send us an email at grouplens-info@cs.umn.edu - we are always interested in working with external collaborators. #### Content and Use of Files #### Verifying the Dataset Contents We encourage you to verify that the dataset you have on your computer is identical to the ones hosted at grouplens.org. This is an important step if you downloaded the dataset from a location other than grouplens.org, or if you wish to publish research results based on analysis of the MovieLens dataset. We provide a MD5 checksum with the same name as the downloadable .zip file, but with a .md5 file extension. To verify the dataset: on linux md5sum ml-20m.zip; cat ml-20m.zip.md5 on OSX md5 ml-20m.zip; cat ml-20m.zip.md5 windows users can download a tool from Microsoft (or elsewhere) that verifies MD5 checksums Check that the two lines of output contain the same hash value. #### Formatting and Encoding The dataset files are written as comma-separated values files with a single header row. Columns that contain commas (,) are escaped using double-quotes ("). These files are encoded as UTF-8. If accented characters in movie titles or tag values (e.g. Misérables, Les (1995)) display incorrectly, make sure that any program reading the data, such as a text editor, terminal, or script, is configured for UTF-8. #### User Ids MovieLens users were selected at random for inclusion. Their ids have been anonymized. User ids are consistent between ratings.csv and tags.csv (i.e., the same id refers to the same user across the two files). #### Movie Ids Only movies with at least one rating or tag are included in the dataset. These movie ids are consistent with those used on the MovieLens web site (e.g., id 1 corresponds to the URL ). Movie ids are consistent between ratings.csv, tags.csv, movies.csv, and links.csv (i.e., the same id refers to the same movie across these four data files). #### Ratings Data File Structure (ratings.csv) All ratings are contained in the file ratings.csv. Each line of this file after the header row represents one rating of one movie by one user, and has the following format: userId,movieId,rating,timestamp The lines within this file are ordered first by userId, then, within user, by movieId. Ratings are made on a 5-star scale, with half-star increments (0.5 stars - 5.0 stars). Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970. Tags Data File Structure (tags.csv) All tags are contained in the file tags.csv. Each line of this file after the header row represents one tag applied to one movie by one user, and has the following format: userId,movieId,tag,timestamp The lines within this file are ordered first by userId, then, within user, by movieId. Tags are user-generated metadata about movies. Each tag is typically a single word or short phrase. The meaning, value, and purpose of a particular tag is determined by each user. Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970. Movies Data File Structure (movies.csv) Movie information is contained in the file movies.csv. Each line of this file after the header row represents one movie, and has the following format: movieId,title,genres Movie titles are entered manually or imported from , and include the year of release in parentheses. Errors and inconsistencies may exist in these titles. Genres are a pipe-separated list, and are selected from the following: Action Adventure Animation Children s Comedy Crime Documentary Drama Fantasy Film-Noir Horror Musical Mystery Romance Sci-Fi Thriller War Western (no genres listed) Links Data File Structure (links.csv) Identifiers that can be used to link to other sources of movie data are contained in the file links.csv. Each line of this file after the header row represents one movie, and has the following format: movieId,imdbId,tmdbId movieId is an identifier for movies used by . E.g., the movie Toy Story has the link . imdbId is an identifier for movies used by . E.g., the movie Toy Story has the link . tmdbId is an identifier for movies used by . E.g., the movie Toy Story has the link . Use of the resources listed above is subject to the terms of each provider. Tag Genome (genome-scores.csv and genome-tags.csv) This data set includes a current copy of the Tag Genome. The tag genome is a data structure that contains tag relevance scores for movies. The structure is a dense matrix: each movie in the genome has a value for every tag in the genome. As described in this article, the tag genome encodes how strongly movies exhibit particular properties represented by tags (atmospheric, thought-provoking, realistic, etc.). The tag genome was computed using a machine learning algorithm on user-contributed content including tags, ratings, and textual reviews. The genome is split into two files. The file genome-scores.csv contains movie-tag relevance data in the following format: movieId,tagId,relevance The second file, genome-tags.csv, provides the tag descriptions for the tag IDs in the genome file, in the following format: tagId,tag The tagId values are generated when the data set is exported, so they may vary from version to version of the MovieLens data sets. #### Cross-Validation Prior versions of the MovieLens dataset included either pre-computed cross-folds or scripts to perform this computation. We no longer bundle either of these features with the dataset, since most modern toolkits provide this as a built-in feature. If you wish to learn about standard approaches to cross-fold computation in the context of recommender systems evaluation, see LensKit for tools, documentation, and open-source code examples.

Sentiment Labelled Sentences Data Set (Dataset)

This dataset was created for the Paper From Group to Individual Labels using Deep Features , Kotzias et. al,. KDD 2015 Please cite the paper if you want to use it :) It contains sentences labelled with positive or negative sentiment. ### Format: sentence score ### Details: Score is either 1 (for positive) or 0 (for negative) The sentences come from three different websites/fields: imdb.com amazon.com yelp.com For each website, there exist 500 positive and 500 negative sentences. Those were selected randomly for larger datasets of reviews. We attempted to select sentences that have a clearly positive or negative connotaton, the goal was for no neutral sentences to be selected. ### Attribute Information: The attributes are text sentences, extracted from reviews of products, movies, and restaurants ### Relevant Papers: From Group to Individual Labels using Deep Features , Kotzias et. al,. KDD 2015

Online News Popularity Data Set (Dataset)

##Data Set Information: * The articles were published by Mashable (www.mashable.com) and their content as the rights to reproduce it belongs to them. Hence, this dataset does not share the original content but some statistics associated with it. The original content be publicly accessed and retrieved using the provided urls. * Acquisition date: January 8, 2015 * The estimated relative performance values were estimated by the authors using a Random Forest classifier and a rolling windows as assessment method. See their article for more details on how the relative performance values were set. ##Attribute Information: Number of Attributes: 61 (58 predictive attributes, 2 non-predictive, 1 goal field) 0. url: URL of the article (non-predictive) 1. timedelta: Days between the article publication and the dataset acquisition (non-predictive) 2. n_tokens_title: Number of words in the title 3. n_tokens_content: Number of words in the content 4. n_unique_tokens: Rate of unique words in the content 5. n_non_stop_words: Rate of non-stop words in the content 6. n_non_stop_unique_tokens: Rate of unique non-stop words in the content 7. num_hrefs: Number of links 8. num_self_hrefs: Number of links to other articles published by Mashable 9. num_imgs: Number of images 10. num_videos: Number of videos 11. average_token_length: Average length of the words in the content 12. num_keywords: Number of keywords in the metadata 13. data_channel_is_lifestyle: Is data channel Lifestyle ? 14. data_channel_is_entertainment: Is data channel Entertainment ? 15. data_channel_is_bus: Is data channel Business ? 16. data_channel_is_socmed: Is data channel Social Media ? 17. data_channel_is_tech: Is data channel Tech ? 18. data_channel_is_world: Is data channel World ? 19. kw_min_min: Worst keyword (min. shares) 20. kw_max_min: Worst keyword (max. shares) 21. kw_avg_min: Worst keyword (avg. shares) 22. kw_min_max: Best keyword (min. shares) 23. kw_max_max: Best keyword (max. shares) 24. kw_avg_max: Best keyword (avg. shares) 25. kw_min_avg: Avg. keyword (min. shares) 26. kw_max_avg: Avg. keyword (max. shares) 27. kw_avg_avg: Avg. keyword (avg. shares) 28. self_reference_min_shares: Min. shares of referenced articles in Mashable 29. self_reference_max_shares: Max. shares of referenced articles in Mashable 30. self_reference_avg_sharess: Avg. shares of referenced articles in Mashable 31. weekday_is_monday: Was the article published on a Monday? 32. weekday_is_tuesday: Was the article published on a Tuesday? 33. weekday_is_wednesday: Was the article published on a Wednesday? 34. weekday_is_thursday: Was the article published on a Thursday? 35. weekday_is_friday: Was the article published on a Friday? 36. weekday_is_saturday: Was the article published on a Saturday? 37. weekday_is_sunday: Was the article published on a Sunday? 38. is_weekend: Was the article published on the weekend? 39. LDA_00: Closeness to LDA topic 0 40. LDA_01: Closeness to LDA topic 1 41. LDA_02: Closeness to LDA topic 2 42. LDA_03: Closeness to LDA topic 3 43. LDA_04: Closeness to LDA topic 4 44. global_subjectivity: Text subjectivity 45. global_sentiment_polarity: Text sentiment polarity 46. global_rate_positive_words: Rate of positive words in the content 47. global_rate_negative_words: Rate of negative words in the content 48. rate_positive_words: Rate of positive words among non-neutral tokens 49. rate_negative_words: Rate of negative words among non-neutral tokens 50. avg_positive_polarity: Avg. polarity of positive words 51. min_positive_polarity: Min. polarity of positive words 52. max_positive_polarity: Max. polarity of positive words 53. avg_negative_polarity: Avg. polarity of negative words 54. min_negative_polarity: Min. polarity of negative words 55. max_negative_polarity: Max. polarity of negative words 56. title_subjectivity: Title subjectivity 57. title_sentiment_polarity: Title polarity 58. abs_title_subjectivity: Absolute subjectivity level 59. abs_title_sentiment_polarity: Absolute polarity level 60. shares: Number of shares (target) ##Relevant Papers: K. Fernandes, P. Vinagre and P. Cortez. A Proactive Intelligent Decision Support System for Predicting the Popularity of Online News. Proceedings of the 17th EPIA 2015 - Portuguese Conference on Artificial Intelligence, September, Coimbra, Portugal. ##Citation Request: K. Fernandes, P. Vinagre and P. Cortez. A Proactive Intelligent Decision Support System for Predicting the Popularity of Online News. Proceedings of the 17th EPIA 2015 - Portuguese Conference on Artificial Intelligence, September, Coimbra, Portugal. ##Source: Kelwin Fernandes (kafc â€˜@â€™ inesctec.pt, kelwinfc â€™@â€™ gmail.com) - INESC TEC, Porto, Portugal/Universidade do Porto, Portugal. Pedro Vinagre (pedro.vinagre.sousa â€™@â€™ gmail.com) - ALGORITMI Research Centre, Universidade do Minho, Portugal Paulo Cortez - ALGORITMI Research Centre, Universidade do Minho, Portugal Pedro Sernadela - Universidade de Aveiro

Structured Web Data Extraction Dataset (SWDE) (Dataset)

## Motivation This dataset is a real-world web page collection used for research on the automatic extraction of structured data (e.g., attribute-value pairs of entities) from the Web. We hope it could serve as a useful benchmark for evaluating and comparing different methods for structured web data extraction. ## Contents of the Dataset Currently the dataset involves: 8 verticals with diverse semantics; 80 web sites (10 per vertical); 124,291 web pages (200 ~ 2,000 per web site), each containing a single data record with detailed information of an entity; 32 attributes (3 ~ 5 per vertical) associated with carefully labeled ground-truth of corresponding values in each web page. The goal of structured data extraction is to automatically identify the values of these attributes from web pages. The involved verticals are summarized as follows: |Vertical |#Sites|#Pages|#Attributes|Attributes| |—————-|———|—————|————————|————————-| |Auto|10|17,923|4|model, price, engine, fuel_economy| |Book|10|20,000|5|title, author, isbn_13, publisher, publication_date| |Camera|10 |5,258|3|model, price, manufacturer| |Job|10|20,000|4|title, company, location, date_posted| |Movie|10|20,000|4|title, director, genre, mpaa_rating| |NBA Player|10| 4,405|4|name, team, height, weight| |Restaurant|10|20,000|4|name, address, phone, cuisine| |University|10|16,705|4|name, phone, website, type| # Format of Web Pages Each web page in the dataset is stored as one .htm file (in UTF-8 encoding) where the first tag encodes the source URL of the page. # Format of Ground-truth Files For each web site, the page-level ground-truth of attribute values has been labeled using handcrafted regular expressions and stored in .txt files (in UTF-8 encoding) named as: "--.txt". # In each such file: The first line stores the names of vertical, site, and attribute, separated by TAB characters ( \t ). The second line stores some statistics (separated by TABs) w.r.t. the corresponding site and attribute, including: * the total number of pages, * the number of pages containing attribute values, * the total number of attribute values contained in the pages, * the number of unique attribute values. Each remaining line stores the ground-truth information (separated by TABs) of one page, in sequence of: *page ID, *the number of attribute values in the page, *attribute values ("" in case of non-existence). Notes on Ground-truth Labeling The ground-truth labeling was conducted in the DOM-node level. More specifically, the candidate attribute values in a web page are the non-empty strings contained in text nodes in the corresponding DOM tree. One page (although containing a single data record) may contain multiple distinct values that correspond to an attribute (e.g., multiple authors of a book, multiple granularity levels of addresses). Currently, when a text node presents a mixture of multiple attributes, its string value is labeled with each of these attributes, if no substitute is available. Before being stored in .txt files, the raw attribute values were refined by removing redundant separators (e.g., , \t , \n ). ## Reference We would appreciate it if you cite the following paper when using the dataset: Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. "From One Tree to a Forest: a Uniﬁed Solution for Structured Web Data Extraction". in Proc. of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011), pp.775-784, Beijing, China. July 24-28, 2011. ## Contact If you have questions about this dataset, please contact Qiang Hao (haoq@live.com, Homepage).

SMS Spam Collection Data Set (Dataset)

==Data Set Information: This corpus has been collected from free or free for research sources at the Internet: -> A collection of 425 SMS spam messages was manually extracted from the Grumbletext Web site. This is a UK forum in which cell phone users make public claims about SMS spam messages, most of them without reporting the very spam message received. The identification of the text of spam messages in the claims is a very hard and time-consuming task, and it involved carefully scanning hundreds of web pages. The Grumbletext Web site is: [Web Link]. -> A subset of 3,375 SMS randomly chosen ham messages of the NUS SMS Corpus (NSC), which is a dataset of about 10,000 legitimate messages collected for research at the Department of Computer Science at the National University of Singapore. The messages largely originate from Singaporeans and mostly from students attending the University. These messages were collected from volunteers who were made aware that their contributions were going to be made publicly available. The NUS SMS Corpus is avalaible at: [Web Link]. -> A list of 450 SMS ham messages collected from Caroline Tag s PhD Thesis available at [Web Link]. -> Finally, we have incorporated the SMS Spam Corpus v.0.1 Big. It has 1,002 SMS ham messages and 322 spam messages and it is public available at: [Web Link]. This corpus has been used in the following academic researches:

Enwiki Word2vec model 1000 Dimensions (Dataset)

Gensim Word2vec model built on the english wikipedia, 1000dimensions, 10cbow, no stemming

Yale YouTube Video Text (Dataset)

YouTube Video Text (YVT) contains 30 videos. Each video has 15-second length, 30 frames per second, HD 720p quality and was collected from YouTube. The text content in the dataset can be divided into two categories, overlay text (e.g., captions, songs title, logos) and scene text (e.g. street signs, business signs, words on shirt).

Lerman Twitter 2010 Dataset (Dataset)

Twitter_2010 data set contains tweets containing URLs that have been posted on Twitter during October 2010. In addition to tweets, we also the followee links of tweeting users, allowing us to reconstruct the follower graph of active (tweeting) users. URLs66,059 tweets2,859,764 users736,930 links36,743,448 Tweets Table (in csv format) link_status_search_with_ordering_real_csv contains tweets with the following information link: URL within the text of the tweet id: tweet id create_at: date added to the db create_at_long inreplyto_screen_name: screen name of user this tweet is replying to inreplyto_user_id: user id of user this tweet is replying to source: device from which the tweet originated bad_user_id: alternate user id user_screen_name: tweeting user screen name order_of_users: tweet s index within sequence of tweets of the same URL user_id: user id Table (in csv format) distinct_users_from_search_table_real_map contains names of tweeting users, and the following information for each user: user_id: user id user_screen_name: user name indegree: number of followers outdegree: number of friends/followees bad_user_id: alternate user id Follower graph File active_follower_real_sql contains zipped SQL dump of links between tweeting users in the form: user_id: user id follower_id: user id of the follower Empirical characterization of this data is described in Kristina Lerman, Rumi Ghosh, Tawan Surachawala (2012) "Social Contagion: An Empirical Study of Information Spread on Digg and Twitter Follower Graphs." This data is made available to the community for research purposes only. If you use the data in a publication, please cite the above paper.