{"id":1002,"date":"2013-10-15T11:12:45","date_gmt":"2013-10-15T15:12:45","guid":{"rendered":"http:\/\/josephpcohen.com\/w\/?p=1002"},"modified":"2022-04-10T21:56:37","modified_gmt":"2022-04-11T01:56:37","slug":"academic-torrents","status":"publish","type":"post","link":"https:\/\/josephpcohen.com\/w\/academic-torrents\/","title":{"rendered":"Academic Torrents &#8211; A Platform for Sharing Data"},"content":{"rendered":"<p><iframe loading=\"lazy\" title=\"What is Academic Torrents?\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/ZYXALE7pdWE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p><a href=\"http:\/\/academictorrents.com\/\">Academic Torrents<\/a> is a platform for researchers to share data. It consists of two pieces: a site where users can search for datasets, and a BitTorrent backbone which makes sharing data scalable and fast. The goal is to facilitate the sharing of datasets amongst researchers. The site provides access to over 32TB of data and delivers over 3TB per day of data to researchers all over the world. I started this project with another grad student in 2013. Since then we have founded the Institute for Reproducible Research (a U.S. 501(c)3 non-profit) in order to continue operation of the of the site.<\/p>\n<p>The site provides access to data including popular machine learning datasets such as all of UCI, ImageNet, and Wikipedia. Though some of these datasets are available elsewhere, Academic Torrents stitches multiple hosting locations together so downloading is much faster and also fault-tolerant. For downloaders there are no sign-up or verification processes in the way, and the collection is more comprehensive than anywhere else. Many datasets such as Netflix, where the original hosting location is no longer available, are made available using Academic Torrents.<\/p>\n<p>As data gets bigger, peer-to-peer file transfer becomes increasingly attractive, since it is the only way distribution scales with the number of users. Academic Torrents currently facilitates the transfer of over 3TB\/day and over 30000 users\/monthly.<\/p>\n<p>The guiding principle of Academic Torrents is to ensure that the data the community needs is always available and can be obtained quickly. In order to ensure that data is always available it needs to be stored in more than one location in case the initial location is not available. Typically, when a user downloads data from a secondary website it is unclear if they they found the correct data. BitTorrent allows data to be mirrored transparently in a peer to peer fashion while maintaining the correctness and authenticity of the data. A speed increase is gained because a user can download from all the mirrors at once.<\/p>\n<p>&nbsp;<\/p>\n<h1>Publications<\/h1>\n<ul>\n<li>Lo, Henry Z. and Cohen, Joseph P., (2015). Academic Torrents: Scalable Data Distribution. Neural Information Processing Systems 2015 Challenges in Machine Learning (CiML) Workshop. NIPS 2015. <a href=\"http:\/\/arxiv.org\/abs\/1603.04395\">http:\/\/arxiv.org\/abs\/1603.04395<\/a><\/li>\n<li>Cohen, Joseph P. and Lo, Henry Z., (2014). Academic Torrents: A Community-Maintained Distributed Repository,&#8221; in Annual Conference of the Extreme Science and Engineering Discovery Environment, XSEDE 2014. <a href=\"http:\/\/doi.org\/10.1145\/2616498.2616528\">http:\/\/doi.org\/10.1145\/2616498.2616528<\/a><\/li>\n<\/ul>\n<h1>Use in research<\/h1>\n<p>To determine the impact of the project we can look for research papers which reference a torrent on Academic Torrents. By googling &#8220;inurl:pdf academictorrents.com\/details&#8221; a list of pdfs which contain a reference to a torrent is shown.<\/p>\n<p><a href=\"http:\/\/josephpcohen.com\/w\/wp-content\/uploads\/Screen-Shot-2018-11-13-at-11.46.16-AM.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-large wp-image-1249\" src=\"http:\/\/josephpcohen.com\/w\/wp-content\/uploads\/Screen-Shot-2018-11-13-at-11.46.16-AM-1024x498.png\" alt=\"\" width=\"1024\" height=\"498\" srcset=\"https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/Screen-Shot-2018-11-13-at-11.46.16-AM-1024x498.png 1024w, https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/Screen-Shot-2018-11-13-at-11.46.16-AM-300x146.png 300w, https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/Screen-Shot-2018-11-13-at-11.46.16-AM-768x373.png 768w, https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/Screen-Shot-2018-11-13-at-11.46.16-AM.png 1996w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/p>\n<p>Here each spike is after the project was discussed on the front page of Hacker News.<\/p>\n<h1>Press<\/h1>\n<p>2018<\/p>\n<ul>\n<li><a href=\"https:\/\/www.techleer.com\/articles\/545-academic-torrents-a-distributed-system-for-sharing-enormous-datasets\/\">TechLeer article on Academic Torrents <\/a><\/li>\n<li><a href=\"https:\/\/news.ycombinator.com\/item?id=17744150\">Hacker News Front Page Post 2018 &#8211;\u00a0Academic Torrents: A distributed system for sharing enormous datasets<\/a><\/li>\n<li><a href=\"https:\/\/motherboard.vice.com\/en_us\/article\/a3yxwb\/ai-researchers-are-boycotting-natures-new-machine-intelligence-journal \">Motherboard, Vice News &#8211; AI Researchers Are Boycotting Nature\u2019s New Machine Intelligence Journal<\/a><\/li>\n<\/ul>\n<p>2017<\/p>\n<ul>\n<li><a href=\"https:\/\/torrentfreak.com\/torrents-help-researchers-worldwide-to-study-babies-brains-170603\/\">Torrent Freak &#8211;\u00a0Torrents Help Researchers Worldwide to Study Babies\u2019 Brains<\/a><\/li>\n<\/ul>\n<p>2016<\/p>\n<ul>\n<li><a href=\"http:\/\/www.kdnuggets.com\/2016\/10\/academic-torrents-data-sharing.html\">KD Nuggets &#8211; What is Academic Torrents and Where is Data Sharing Going?<\/a><\/li>\n<li><a href=\"https:\/\/news.ycombinator.com\/item?id=12381791\">Hacker News Front Page Post 2016 &#8211;\u00a0Academic Torrents: A distributed system for sharing enormous datasets<\/a><\/li>\n<li><a href=\"http:\/\/www.rce-cast.com\/Podcast\/rce-108-academic-torrents.html\">Research Computing and Engineering Podcast Interview<\/a><\/li>\n<\/ul>\n<p>2014<\/p>\n<ul>\n<li><a href=\"http:\/\/sites.uci.edu\/learninglibrary\/2014\/02\/03\/game-changer-academic-torrents\/\">The Learning Library &#8211; Game Changer: Academic Torrents?<\/a><\/li>\n<li><a href=\"http:\/\/lj.libraryjournal.com\/2014\/02\/technology\/academic-torrents-offers-new-means-of-storing-distributing-scholarly-content\/\">Library Journal &#8211; Academic Torrents Offers New Means of Storing, Distributing Scholarly Content<\/a><\/li>\n<li><a href=\"https:\/\/www.mysciencework.com\/omniscience\/academic-torrents-bringing-p2p-technology-to-the-academic-world\">MyScienceWork &#8211; Academic Torrents: Bringing P2P Technology to the Academic World<\/a><\/li>\n<li><a href=\"https:\/\/torrentfreak.com\/academics-launch-torrent-site-to-share-papers-and-datasets-140131\/\">Torrent Freak &#8211; Academics Launch Torrent Site to Share Papers and Datasets<\/a><\/li>\n<li><a href=\"https:\/\/motherboard.vice.com\/en_us\/article\/vvbmax\/a-torrent-site-wants-to-be-the-new-academic-library\">Motherboard, Vice News &#8211; A Torrent Site Wants to Be the New Academic Library<\/a><\/li>\n<li><a href=\"https:\/\/news.ycombinator.com\/item?id=7149006\">Hacker News Front Page Post 2014 &#8211;\u00a0Academic Torrents<\/a><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"mh-excerpt\"><p>Academic Torrents is a platform for researchers to share data. It consists of two pieces: a site where users can search for datasets, and a <a class=\"mh-excerpt-more\" href=\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\" title=\"Academic Torrents &#8211; A Platform for Sharing Data\">[&#8230;]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":1003,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[16],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Academic Torrents - A Platform for Sharing Data - Joseph Paul Cohen PhD<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Academic Torrents - A Platform for Sharing Data - Joseph Paul Cohen PhD\" \/>\n<meta property=\"og:description\" content=\"Academic Torrents is a platform for researchers to share data. It consists of two pieces: a site where users can search for datasets, and a [...]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\" \/>\n<meta property=\"og:site_name\" content=\"Joseph Paul Cohen PhD\" \/>\n<meta property=\"article:published_time\" content=\"2013-10-15T15:12:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-11T01:56:37+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/logo-1200x630.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Joseph Paul Cohen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Joseph Paul Cohen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/josephpcohen.com\/w\/academic-torrents\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\"},\"author\":{\"name\":\"Joseph Paul Cohen\",\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\"},\"headline\":\"Academic Torrents &#8211; A Platform for Sharing Data\",\"datePublished\":\"2013-10-15T15:12:45+00:00\",\"dateModified\":\"2022-04-11T01:56:37+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\"},\"wordCount\":609,\"publisher\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\"},\"articleSection\":[\"Featured\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\",\"url\":\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\",\"name\":\"Academic Torrents - A Platform for Sharing Data - Joseph Paul Cohen PhD\",\"isPartOf\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#website\"},\"datePublished\":\"2013-10-15T15:12:45+00:00\",\"dateModified\":\"2022-04-11T01:56:37+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/academic-torrents\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/josephpcohen.com\/w\/academic-torrents\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/josephpcohen.com\/w\/academic-torrents\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/josephpcohen.com\/w\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Academic Torrents &#8211; A Platform for Sharing Data\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/josephpcohen.com\/w\/#website\",\"url\":\"https:\/\/josephpcohen.com\/w\/\",\"name\":\"Joseph Paul Cohen PhD\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/josephpcohen.com\/w\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684\",\"name\":\"Joseph Paul Cohen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g\",\"caption\":\"Joseph Paul Cohen\"},\"logo\":{\"@id\":\"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Academic Torrents - A Platform for Sharing Data - Joseph Paul Cohen PhD","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/josephpcohen.com\/w\/academic-torrents\/","og_locale":"en_US","og_type":"article","og_title":"Academic Torrents - A Platform for Sharing Data - Joseph Paul Cohen PhD","og_description":"Academic Torrents is a platform for researchers to share data. It consists of two pieces: a site where users can search for datasets, and a [...]","og_url":"https:\/\/josephpcohen.com\/w\/academic-torrents\/","og_site_name":"Joseph Paul Cohen PhD","article_published_time":"2013-10-15T15:12:45+00:00","article_modified_time":"2022-04-11T01:56:37+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/josephpcohen.com\/w\/wp-content\/uploads\/logo-1200x630.jpg","type":"image\/jpeg"}],"author":"Joseph Paul Cohen","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Joseph Paul Cohen","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/josephpcohen.com\/w\/academic-torrents\/#article","isPartOf":{"@id":"https:\/\/josephpcohen.com\/w\/academic-torrents\/"},"author":{"name":"Joseph Paul Cohen","@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684"},"headline":"Academic Torrents &#8211; A Platform for Sharing Data","datePublished":"2013-10-15T15:12:45+00:00","dateModified":"2022-04-11T01:56:37+00:00","mainEntityOfPage":{"@id":"https:\/\/josephpcohen.com\/w\/academic-torrents\/"},"wordCount":609,"publisher":{"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684"},"articleSection":["Featured"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/josephpcohen.com\/w\/academic-torrents\/","url":"https:\/\/josephpcohen.com\/w\/academic-torrents\/","name":"Academic Torrents - A Platform for Sharing Data - Joseph Paul Cohen PhD","isPartOf":{"@id":"https:\/\/josephpcohen.com\/w\/#website"},"datePublished":"2013-10-15T15:12:45+00:00","dateModified":"2022-04-11T01:56:37+00:00","breadcrumb":{"@id":"https:\/\/josephpcohen.com\/w\/academic-torrents\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/josephpcohen.com\/w\/academic-torrents\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/josephpcohen.com\/w\/academic-torrents\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/josephpcohen.com\/w\/"},{"@type":"ListItem","position":2,"name":"Academic Torrents &#8211; A Platform for Sharing Data"}]},{"@type":"WebSite","@id":"https:\/\/josephpcohen.com\/w\/#website","url":"https:\/\/josephpcohen.com\/w\/","name":"Joseph Paul Cohen PhD","description":"","publisher":{"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/josephpcohen.com\/w\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/e25d0d5746952220f35d182ca7aa8684","name":"Joseph Paul Cohen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a810b57939e75247f570c9094e7bd16e?s=96&d=mm&r=g","caption":"Joseph Paul Cohen"},"logo":{"@id":"https:\/\/josephpcohen.com\/w\/#\/schema\/person\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts\/1002"}],"collection":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/comments?post=1002"}],"version-history":[{"count":28,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts\/1002\/revisions"}],"predecessor-version":[{"id":1676,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/posts\/1002\/revisions\/1676"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/media\/1003"}],"wp:attachment":[{"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/media?parent=1002"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/categories?post=1002"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/josephpcohen.com\/w\/wp-json\/wp\/v2\/tags?post=1002"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}