s3cmd + debian8 で実現する
事前準備
S3を操作できるユーザをIAMで作成しておく
インストールとセットアップ
root@hostname:/home/shimizu# aptitude install s3cmd
以下の新規パッケージがインストールされます:
python-dateutil{a} python-magic{a} s3cmd
...
root@hostname:/home/shimizu# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: *****
Secret Key: *****
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: *****
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:*****
...
Configuration saved to '/root/.s3cfg'
root@hostname:/home/shimizu# s3cmd --version
s3cmd version 1.5.0-rc1
s3cmd操作
すべてはmanコマンドに
COMMANDS
s3cmd can do several actions specified by the following commands.
s3cmd mb s3://BUCKET
Make bucket
s3cmd rb s3://BUCKET
Remove bucket
s3cmd ls [s3://BUCKET[/PREFIX]]
List objects or buckets
s3cmd la
List all object in all buckets
s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
Put file into bucket
s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Get file from bucket
s3cmd del s3://BUCKET/OBJECT
Delete file from bucket
s3cmd rm s3://BUCKET/OBJECT
Delete file from bucket (alias for del)
s3cmd restore s3://BUCKET/OBJECT
Restore file from Glacier storage
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overriden bns, se e below)
s3cmd du [s3://BUCKET[/PREFIX]]
Disk usage by buckets
s3cmd info s3://BUCKET[/OBJECT]
Get various information about Buckets or Files
s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Copy object
s3cmd modify s3://BUCKET1/OBJECT
Modify object metadata
s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Move object
s3cmd setacl s3://BUCKET[/OBJECT]
Modify Access control list for Bucket or Files
s3cmd setpolicy FILE s3://BUCKET
Modify Bucket Policy
s3cmd delpolicy s3://BUCKET
Delete Bucket Policy
s3cmd multipart s3://BUCKET [Id]
show multipart uploads
s3cmd abortmp s3://BUCKET/OBJECT Id
abort a multipart upload
s3cmd listmp s3://BUCKET/OBJECT Id
list parts of a multipart upload
s3cmd accesslog s3://BUCKET
Enable/disable bucket access logging
s3cmd sign STRING-TO-SIGN
Sign arbitrary string using the secret key
s3cmd signurl s3://BUCKET/OBJECT expiry_epoch
Sign an S3 URL to provide limited public access with expiry
s3cmd fixbucket s3://BUCKET[/PREFIX]
Fix invalid file names in a bucket
s3cmd expire s3://BUCKET
Set or delete expiration rule for the bucket
s3cmd setlifecycle s3://BUCKET
Upload a lifecycle policy for the bucket
s3cmd dellifecycle s3://BUCKET
Remove a lifecycle policy for the bucket
よく使いそうなコマンドを試してみる
### bucket作成 ### root@hostname:/home/shimizu# s3cmd mb s3://www.akat.info.bk Bucket 's3://www.akat.info.bk/' created ### bucket表示 ### root@hostname:/home/shimizu# s3cmd ls 2014-07-02 15:22 s3://www.akat.info ### ファイルを送る ### root@hostname:/home/shimizu# s3cmd put test.txt s3://www.akat.info test.txt -> s3://www.akat.info/test.txt [1 of 1] 25623 of 25623 100% in 0s 279.38 kB/s done ### 確認する ### root@hostname:/home/shimizu# s3cmd la 2015-10-31 13:59 25623 s3://www.akat.info/test.txt ### ファイルを修得する ### root@hostname:/home/shimizu# s3cmd get s3://www.akat.info/test.txt ### 容量が大きいとリトライが何回もかかってエラー ### root@hostname:/home/shimizu# s3cmd put 5G.file s3://www.akat.info ... WARNING: Upload failed: /5G.file?partNumber=23&uploadId=... ([Errno 104] Connection reset by peer) WARNING: Retrying on lower speed (throttle=0.00) ... ### 容量が大きいファイルを送る場合(--multipart-chunk-size-mbというオプションもあり) ### root@hostname:/home/shimizu# s3cmd --multipart-chunk-size-mb=5 put 5G.file s3://www.akat.info ### 容量算出 ### root@hostname:/home/shimizu# s3cmd du s3://www.akat.info 5368709120 s3://www.akat.info/ ### s3間でコピー(移動も可能) ### root@hostname:/home/shimizu# s3cmd cp s3://www.akat.info/test.txt s3://www.akat.info.bk/ File s3://www.akat.info/test.txt copied to s3://www.akat.info.bk/test.txt root@hostname:/home/shimizu# s3cmd la 2015-10-31 13:59 25623 s3://www.akat.info/test.txt 2015-10-31 14:07 25623 s3://www.akat.info.bk/test.txt ### 期限付きアクセス ### root@hostname:/home/shimizu# s3cmd signurl s3://www.akat.info/test.txt 300 http://www.akat.info.s3.amazonaws.com/test.txt?AWSAccessKeyId=AKIAIKWXKI6T6S3VPCPA&Expires=300&Signature=5MEpFuEGrOzTgCNVDpAfsLMn02Q%3D …URLはできたが、アクセスするとなぜかうまくいかなかった ### syncする ### root@hostname:/home/shimizu# s3cmd sync testdir s3://www.akat.info testdir/test -> s3://www.akat.info/testdir/test [1 of 2] 4 of 4 100% in 0s 43.85 B/s done testdir/test2 -> s3://www.akat.info/testdir/test2 [2 of 2] 13 of 13 100% in 0s 276.01 B/s done Done. Uploaded 17 bytes in 0.2 seconds, 104.52 B/s. Copied 0 files saving 0 bytes transfer. ### 完全同期する ### root@hostname:/home/shimizu# s3cmd --delete-removed sync test3.txt s3://www.akat.info File s3://www.akat.info/delete.txt deleted File s3://www.akat.info/testdir/test deleted File s3://www.akat.info/testdir/test2 deleted test3.txt -> s3://www.akat.info/test3.txt [1 of 1] 0 of 0 0% in 0s 0.00 B/s done ### restoreする ### root@hostname:/home/shimizu# s3cmd sync s3://www.akat.info/test3.txt testdir/