Monthly Archives: January 2018

Use s3cmd to Download Requester Pays Buckets on S3

List files under pdf:

$ s3cmd ls --requester-pays s3://arxiv/pdf
                       DIR   s3://arxiv/pdf/

List files under pdf:

$ s3cmd ls --requester-pays s3://arxiv/pdf/\*
2010-07-29 19:56 526202880   s3://arxiv/pdf/arXiv_pdf_0001_001.tar
2010-07-29 20:08 138854400   s3://arxiv/pdf/arXiv_pdf_0001_002.tar
2010-07-29 20:14 525742080   s3://arxiv/pdf/arXiv_pdf_0002_001.tar
2010-07-29 20:33 156743680   s3://arxiv/pdf/arXiv_pdf_0002_002.tar
2010-07-29 20:38 525731840   s3://arxiv/pdf/arXiv_pdf_0003_001.tar
2010-07-29 20:52 187607040   s3://arxiv/pdf/arXiv_pdf_0003_002.tar
2010-07-29 20:58 525731840   s3://arxiv/pdf/arXiv_pdf_0004_001.tar
2010-07-29 21:11  44851200   s3://arxiv/pdf/arXiv_pdf_0004_002.tar
2010-07-29 21:14 526305280   s3://arxiv/pdf/arXiv_pdf_0005_001.tar
2010-07-29 21:27 234711040   s3://arxiv/pdf/arXiv_pdf_0005_002.tar
...

Get all files under pdf:

$ s3cmd get --requester-pays s3://arxiv/pdf/\*

List all content to text file:

$ s3cmd ls --requester-pays s3://arxiv/src/\* > all_files.txt

Calculate file size:

$ awk '{s += $3} END { print "sum is", s/1000000000, "GB, average is", s/NR }' all_files.txt
sum is 844.626 GB, average is 4.80447e+08

Install Tsunami UDP on CentOS 7

Install dependencies:

$ yum install cvs git gcc automake autoconf libtool -y

Download Tsunami UDP:

$ cd /tmp
$ cvs -z3 -d:pserver:[email protected]:/cvsroot/tsunami-udp co -P tsunami-udp
$ cd tsunami-udp
$ ./recompile.sh
$ make install

Then on the server side:

$ tsunamid --port 46224 * # (Serves all files from current directory for copy)

On the client side:

$ tsunami connect <server_ip> get *

Transfer dataset back to S3:

aws s3 cp --recursive /mnt/bigephemeral s3://<your-new-bucket>/

Limitations:

  • Tsunami UDP transfers only files and doesn’t do directories/ subdirectories, we need to tar them all up as one single tar file (additional storage capacity needs to be taken into consideration).
  • Multi-threading is not supported.
  • Multi session not supported. Client supports only one connection to the server at a time. No parallel file transfer.
  • No resume or retry for file transfer.
  • Does not support Native encryption.

Refs: