What has Academic Torrents been working on? Tools! We are making a bunch of tools! Here is the first of many documents describing our suite of utilities.
One major problem faced while working with data is moving it onto shared computing systems in order to process it. Tools like scp, rsync, and ftp work to move data from point to point and have facilities for partial transmissions. Using the BitTorrent (BT) protocol to transfer data has many advantages that currently are hard to leverage of on shared computing systems. The tool atdown addresses these shortcomings by:
- Downloading from many locations at once. Cache nodes can be set up close to the system to increase speed further.
- Partial transmissions are part of the protocol which makes restarting a download trivial.
- Validation of existing data to ensure integrity and completeness.