I am currently doing a massive analysis of Reddit's entire publicly available comment dataset.  The dataset is ~1.7 billion JSON objects complete with the comment, score, author, subreddit, position in comment tree and other fields that are available through Reddit's API.  

I'm currently doing NLP analysis and also putting the entire dataset into a large searchable database using Sphinxsearch (also testing ElasticSearch).  

This dataset is over 1 terabyte uncompressed, so this would be best for larger research projects.  If you're interested in a sample month of comments, that can be arranged as well.  I am trying to find a place to host this large dataset -- I'm reaching out to Amazon since they have open data initiatives.

**EDIT:  ~~I'm putting up a Digital Ocean box with 2 TB of bandwidth and will throw an entire months worth of comments up (~ 5 gigs compressed)~~  It's now a torrent.  This will give you guys an opportunity to examine the data.  The file is structured with JSON blocks delimited by new lines (\n).**

**____________________________________________________**

One month of comments is now available here:

**Download Link:** [Torrent](https://mega.nz/#!ysBWXRqK!yPXLr25PgJi184pbJU3GtnqUY4wG7YvuPpxJjEmnb9A)

**Direct Magnet File:** magnet:?xt=urn:btih:32916ad30ce4c90ee4c47a95bd0075e44ac15dd2&amp;dn=RC%5F2015-01.bz2&amp;tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&amp;tr=udp%3A%2F%2Fopen.demonii.com%3A1337&amp;tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&amp;tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969

**Tracker:** udp://tracker.openbittorrent.com:80

**Total Comments:** 53,851,542

**Compression Type:** bzip2 *(5,452,413,560 bytes compressed | 31,648,374,104 bytes uncompressed)*

**md5:** a3fc3d9db18786e4486381a7f37d08e2  RC_2015-01.bz2

**____________________________________________________**

**Example JSON Block:**

    {"gilded":0,"author_flair_text":"Male","author_flair_css_class":"male","retrieved_on":1425124228,"ups":3,"subreddit_id":"t5_2s30g","edited":false,"controversiality":0,"parent_id":"t1_cnapn0k","subreddit":"AskMen","body":"I can't agree with passing the blame, but I'm glad to hear it's at least helping you with the anxiety. I went the other direction and started taking responsibility for everything. I had to realize that people make mistakes including myself and it's gonna be alright. I don't have to be shackled to my mistakes and I don't have to be afraid of making them. ","created_utc":"1420070668","downs":0,"score":3,"author":"TheDukeofEtown","archived":false,"distinguished":null,"id":"cnasd6x","score_hidden":false,"name":"t1_cnasd6x","link_id":"t3_2qyhmp"}


**UPDATE (Saturday 2015-07-03 13:26 ET)**

I'm getting a huge response from this and won't be able to immediately reply to everyone.  I am pinging some people who are helping.  There are two major issues at this point.  Getting the data from my local system to wherever and figuring out bandwidth (since this is a very large dataset).  Please keep checking for new updates.  I am working to make this data publicly available ASAP.  If you're a larger organization or university and have the ability to help seed this initially (will probably require 100 TB of bandwidth to get it rolling), please let me know.  If you can agree to do this, I'll give your organization priority over the data first.

**UPDATE 2 (15:18)**

I've purchased a seedbox.  I'll be updating the link above to the sample file.  Once I can get the full dataset to the seedbox, I'll post the torrent and magnet link to that as well.  I want to thank /u/hak8or for all his help during this process.  It's been a while since I've created torrents and he has been a huge help with explaining how it all works.  Thanks man!

**UPDATE 3 (21:09)**

I'm creating the complete torrent.  There was an issue with my seedbox not allowing public trackers for uploads, so I had to create a private tracker.  I should have a link up shortly to the massive torrent.  I would really appreciate it if people at least seed at 1:1 ratio -- and if you can do more, that's even better!  The size looks to be around ~160 GB -- a bit less than I thought.

**UPDATE 4 (00:49 July 4)**

I'm retiring for the evening.  I'm currently seeding the entire archive to two seedboxes plus two other people.  I'll post the link tomorrow evening once the seedboxes are at 100%.  This will help prevent choking the upload from my home connection if too many people jump on at once.  The seedboxes upload at around 35MB a second in the best case scenario.  We should be good tomorrow evening when I post it.  Happy July 4'th to my American friends!

**UPDATE 5 (14:44)**

Send more beer!  The seedboxes are around 75% and should be finishing up within the next 8 hours.  My next update before I retire for the night will be a magnet link to the main archive.  Thanks!

**UPDATE 6 (20:17)**

**This is the update you've been waiting for!**

The **entire** archive:  

    magnet:?xt=urn:btih:7690f71ea949b868080401c749e878f98de34d3d&amp;dn=reddit%5Fdata&amp;tr=http%3A%2F%2Ftracker.pushshift.io%3A6969%2Fannounce&amp;tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80

Please seed!

**UPDATE 7 (July 11 14:19)**

User /u/fhoffa has done a lot of great work making this data available within Google's BigQuery.   Please check out this link for more information:   /r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/

Awesome work!

