S3へのバックアップ取得

s3cmd + debian8 で実現する

事前準備

S3を操作できるユーザをIAMで作成しておく
2015-10-31_202237

インストールとセットアップ

root@hostname:/home/shimizu# aptitude install s3cmd
以下の新規パッケージがインストールされます:
  python-dateutil{a} python-magic{a} s3cmd
...

root@hostname:/home/shimizu# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: *****
Secret Key: *****

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: *****
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:*****

...
Configuration saved to '/root/.s3cfg'

root@hostname:/home/shimizu# s3cmd --version
s3cmd version 1.5.0-rc1

s3cmd操作

すべてはmanコマンドに

COMMANDS
 s3cmd can do several actions specified by the following commands.

 s3cmd mb s3://BUCKET
        Make bucket

 s3cmd rb s3://BUCKET
        Remove bucket

 s3cmd ls [s3://BUCKET[/PREFIX]]
        List objects or buckets

 s3cmd la
        List all object in all buckets

 s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
        Put file into bucket

 s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
        Get file from bucket

 s3cmd del s3://BUCKET/OBJECT
        Delete file from bucket

 s3cmd rm s3://BUCKET/OBJECT
        Delete file from bucket (alias for del)
 s3cmd restore s3://BUCKET/OBJECT
        Restore file from Glacier storage

 s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
        Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overriden bns, se e below)

 s3cmd du [s3://BUCKET[/PREFIX]]
        Disk usage by buckets

 s3cmd info s3://BUCKET[/OBJECT]
        Get various information about Buckets or Files

 s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
        Copy object

 s3cmd modify s3://BUCKET1/OBJECT
        Modify object metadata

 s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
        Move object

 s3cmd setacl s3://BUCKET[/OBJECT]
        Modify Access control list for Bucket or Files

 s3cmd setpolicy FILE s3://BUCKET
        Modify Bucket Policy

 s3cmd delpolicy s3://BUCKET
        Delete Bucket Policy

 s3cmd multipart s3://BUCKET [Id]
        show multipart uploads

 s3cmd abortmp s3://BUCKET/OBJECT Id
        abort a multipart upload

 s3cmd listmp s3://BUCKET/OBJECT Id
        list parts of a multipart upload

 s3cmd accesslog s3://BUCKET
        Enable/disable bucket access logging

 s3cmd sign STRING-TO-SIGN
        Sign arbitrary string using the secret key

 s3cmd signurl s3://BUCKET/OBJECT expiry_epoch
        Sign an S3 URL to provide limited public access with expiry

 s3cmd fixbucket s3://BUCKET[/PREFIX]
        Fix invalid file names in a bucket

 s3cmd expire s3://BUCKET
        Set or delete expiration rule for the bucket

 s3cmd setlifecycle s3://BUCKET
        Upload a lifecycle policy for the bucket

 s3cmd dellifecycle s3://BUCKET
        Remove a lifecycle policy for the bucket

よく使いそうなコマンドを試してみる

### bucket作成 ###
root@hostname:/home/shimizu# s3cmd mb s3://www.akat.info.bk
Bucket 's3://www.akat.info.bk/' created

### bucket表示 ###
root@hostname:/home/shimizu# s3cmd ls
2014-07-02 15:22  s3://www.akat.info

### ファイルを送る ###
root@hostname:/home/shimizu# s3cmd put test.txt s3://www.akat.info
test.txt -> s3://www.akat.info/test.txt  [1 of 1]
 25623 of 25623   100% in    0s   279.38 kB/s  done

### 確認する ###
root@hostname:/home/shimizu# s3cmd la

2015-10-31 13:59     25623   s3://www.akat.info/test.txt

### ファイルを修得する ###
root@hostname:/home/shimizu# s3cmd get s3://www.akat.info/test.txt

### 容量が大きいとリトライが何回もかかってエラー ###
root@hostname:/home/shimizu# s3cmd put 5G.file s3://www.akat.info
...
WARNING: Upload failed: /5G.file?partNumber=23&uploadId=... ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.00)
...

### 容量が大きいファイルを送る場合(--multipart-chunk-size-mbというオプションもあり) ###
root@hostname:/home/shimizu# s3cmd  --multipart-chunk-size-mb=5 put 5G.file s3://www.akat.info

### 容量算出 ###
root@hostname:/home/shimizu# s3cmd du s3://www.akat.info
5368709120   s3://www.akat.info/

### s3間でコピー(移動も可能) ###
root@hostname:/home/shimizu# s3cmd cp s3://www.akat.info/test.txt s3://www.akat.info.bk/
File s3://www.akat.info/test.txt copied to s3://www.akat.info.bk/test.txt
root@hostname:/home/shimizu# s3cmd la
2015-10-31 13:59     25623   s3://www.akat.info/test.txt

2015-10-31 14:07     25623   s3://www.akat.info.bk/test.txt

### 期限付きアクセス ###
root@hostname:/home/shimizu# s3cmd signurl s3://www.akat.info/test.txt 300
http://www.akat.info.s3.amazonaws.com/test.txt?AWSAccessKeyId=AKIAIKWXKI6T6S3VPCPA&Expires=300&Signature=5MEpFuEGrOzTgCNVDpAfsLMn02Q%3D
…URLはできたが、アクセスするとなぜかうまくいかなかった

### syncする ###
root@hostname:/home/shimizu# s3cmd sync testdir s3://www.akat.info
testdir/test -> s3://www.akat.info/testdir/test  [1 of 2]
 4 of 4   100% in    0s    43.85 B/s  done
testdir/test2 -> s3://www.akat.info/testdir/test2  [2 of 2]
 13 of 13   100% in    0s   276.01 B/s  done
Done. Uploaded 17 bytes in 0.2 seconds, 104.52 B/s.  Copied 0 files saving 0 bytes transfer.

### 完全同期する ###
root@hostname:/home/shimizu# s3cmd --delete-removed sync test3.txt s3://www.akat.info
File s3://www.akat.info/delete.txt deleted
File s3://www.akat.info/testdir/test deleted
File s3://www.akat.info/testdir/test2 deleted
test3.txt -> s3://www.akat.info/test3.txt  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

### restoreする ###
root@hostname:/home/shimizu# s3cmd sync s3://www.akat.info/test3.txt testdir/

参考URL

サーバのデータバックアップをAmazon S3に置く

メニューを閉じる