Darknet Market Archives 2013-2015 (dnmarchives)
Gwern Branwen and Nicolas Christin and David Décary-Hétu and Rasmus Munksgaard Andersen and StExo and El Presidente and Anonymous and Daryl Lau and Sohhlz, Delyan Kratunov and Vince Cakic and Van Buskirk and Whom a

@article{,
title= {Darknet Market Archives 2013-2015 (dnmarchives) },
journal= {},
author= {Gwern Branwen and Nicolas Christin and David Décary-Hétu and              Rasmus Munksgaard Andersen and StExo and El Presidente and Anonymous              and Daryl Lau and Sohhlz, Delyan Kratunov and Vince Cakic and Van Buskirk              and Whom and Michael McKenna and Sigi Goode},
url= {https://www.gwern.net/DNM-archives},
type= {dataset},
year= {2015},
month= {July},
abstract= {Dark Net Markets (DNM) are online markets typically hosted as Tor hidden services providing escrow services between buyers & sellers transacting in Bitcoin or other cryptocoins, usually for drugs or other illegal/regulated goods; the most famous DNM was Silk Road 1, which pioneered the business model in 2011.

From 2013–2015, I scraped/mirrored on a weekly or daily basis all existing English-language DNMs as part of my research into their usage, lifetimes/​characteristics, & legal riskiness; these scrapes covered vendor pages, feedback, images, etc. In addition, I made or obtained copies of as many other datasets & documents related to the DNMs as I could.

This uniquely comprehensive collection is now publicly released as a 50GB (~1.6TB uncompressed) collection covering 89 DNMs & 37+ related forums, representing <4,438 mirrors, and is available for any research.

This page documents the download, contents, interpretation, and technical methods behind the scrapes.

There are ~89 markets, >37 forums and ~5 other sites, representing <4,438 mirrors of >43,596,420 files in ~49.4GB of 163 compressed files, unpacking to >1548GB; the largest single archive decompresses to <250GB. (It can be burned to 3 25GB BDs or 2 50GB BDs; if the former, it may be worth generating additional FEC.)

These archives are xz-compressed tarballs (optimized with the sort-key trick); typically each subfolder is a single date-stamped (YYYY-MM-DD) crawl using wget, with the default directory/file layout. The majority of the content is HTML, CSS, and images (typically photos of item listings); images are space-intensive & omitted from many crawls, but I feel that images are useful to allow browsing the markets as they were and may be highly valuable in their own right as research material, so I tried to collect images where applicable. (Child porn is not a concern as all DNMs & DNM forums ban that content.) Archives sourced from other people follow their own particular conventions. Mac & Windows users may be able to uncompress using their built-in OS archiver, 7zip, Stuffit, or WinRAR; the PAR2 error-checking can be done using par2, QuickPar, Par Buddy, MultiPar or others depending on one’s OS.

If you don’t want to uncompress all of a particular archive, as they can be large, you can try extracting specific files using archiver-specific options; for example, a SR2F command targeting a particular old forum thread:

```
tar --verbose --extract --xz --file='silkroad2-forums.tar.xz' --no-anchored --wildcards '*topic=49187*'
```

## Citation

Gwern Branwen, Nicolas Christin, David Décary-Hétu, Rasmus Munksgaard Andersen, StExo, El Presidente, Anonymous, Daryl Lau, Sohhlz, Delyan Kratunov, Vince Cakic, Van Buskirk, Whom, Michael McKenna, Sigi Goode. “Dark Net Market archives, 2011–2015”, 12 July 2015. Web.

```
@misc{dnmArchives,
    author = {Gwern Branwen and Nicolas Christin and David Décary-Hétu and
              Rasmus Munksgaard Andersen and StExo and El Presidente and Anonymous
              and Daryl Lau and Sohhlz, Delyan Kratunov and Vince Cakic and Van Buskirk
              and Whom and Michael McKenna and Sigi Goode},
title = {Dark Net Market archives, 2011-2015},
howpublished=  {\url{https://www.gwern.net/DNM-archives}},
url = {https://www.gwern.net/DNM-archives},
type = {dataset},
year = {2015},
month = {July},
timestamp = {2015-07-12},
note = {Accessed: DATE} }
```},
keywords= {},
terms= {},
license= {https://creativecommons.org/about/cc0},
superseded= {}
}