Archive for the ‘MinIO S3 Storage’ Category

FreeNAS – self hosted S3 Object Storage with MinIO for e.g. Duplicati 2.0 a backup software to store encrypted backups online

Mittwoch, April 14th, 2021

MinIO – Distributed Erasure Code

Sonntag, April 11th, 2021

MinIO – a review of the hardware that can be deployed to get the most from your MinIO Object Storage Server

Mittwoch, März 31st, 2021

MinIO – how to enable the Object Store Supervisor Service on VMware vSphere with Tanzu / VCF using vSAN Data Persistence platform (DPp)

Montag, März 22nd, 2021

   MinIO – a provider of on-premises S3 Object Stores we will see how to enable the Service and then how to provision a dedicated S3 Object Store to a particular Namespace with vSphere with Tanzu. This means that a developer or a team of developers using a particular namespace can have their own dedicated S3 Object Store instance. Note that at the time of writing this feature requires a VCF with Tanzu deployment or a vSphere with Tanzu deployment that leverages NSX-T. This is because the Supervisor services leverages PodVMs and the PodVM feature is only available with NSX-T currently

Amazon Web Services (AWS) Command Line Interface (AWS CLI) – essential examples to manage buckets and objects

Freitag, März 19th, 2021

This tutorial explains the basics of how to manage S3 buckets and its objects using aws s3 cli using the following examples

# s3 make bucket (create bucket)
aws s3 mb s3://tgsbucket –region us-west-2
# s3 remove bucket
aws s3 rb s3://tgsbucket
aws s3 rb s3://tgsbucket –force
# s3 ls commands
aws s3 ls
aws s3 ls s3://tgsbucket
aws s3 ls s3://tgsbucket –recursive
aws s3 ls s3://tgsbucket –recursive –human-readable –summarize
# s3 cp commands
aws s3 cp getdata.php s3://tgsbucket
aws s3 cp /local/dir/data s3://tgsbucket –recursive
aws s3 cp s3://tgsbucket/getdata.php /local/dir/data
aws s3 cp s3://tgsbucket/ /local/dir/data –recursive
aws s3 cp s3://tgsbucket/init.xml s3://backup-bucket
aws s3 cp s3://tgsbucket s3://backup-bucket –recursive
# s3 mv commands
aws s3 mv source.json s3://tgsbucket
aws s3 mv s3://tgsbucket/getdata.php /home/project
aws s3 mv s3://tgsbucket/source.json s3://backup-bucket
aws s3 mv /local/dir/data s3://tgsbucket/data –recursive
aws s3 mv s3://tgsbucket s3://backup-bucket –recursive
# s3 rm commands
aws s3 rm s3://tgsbucket/queries.txt
aws s3 rm s3://tgsbucket –recursive
# s3 sync commands
aws s3 sync backup s3://tgsbucket
aws s3 sync s3://tgsbucket/backup /tmp/backup
aws s3 sync s3://tgsbucket s3://backup-bucket
# s3 bucket website
aws s3 website s3://tgsbucket/ –index-document index.html –error-document error.html
# s3 presign url (default 3600 seconds)
aws s3 presign s3://tgsbucket/dnsrecords.txt
aws s3 presign s3://tgsbucket/dnsrecords.txt –expires-in 60
$ aws s3 ls s3://tgsbucket –recursive
2019-04-07 11:38:19 2777 config/init.xml
2019-04-07 11:38:20 52 config/support.txt
2019-04-07 11:38:20 1758 data/database.txt
2019-04-07 11:38:20 13 getdata.php
2019-04-07 11:38:20 2546 ipallow.php
2019-04-07 11:38:20 9 license.php
2019-04-07 11:38:20 3677 servers.txt
$ aws s3 ls s3://tgsbucket –recursive –human-readable –summarize
2019-04-07 11:38:19 2.7 KiB config/init.xml
2019-04-07 11:38:20 52 Bytes config/support.txt
2019-04-07 11:38:20 1.7 KiB data/database.txt
2019-04-07 11:38:20 13 Bytes getdata.php
2019-04-07 11:38:20 2.5 KiB ipallow.php
2019-04-07 11:38:20 9 Bytes license.php
2019-04-07 11:38:20 3.6 KiB servers.txt
Total Objects: 7
Total Size: 10.6 KiB
$ aws s3 presign s3://tgsbucket/dnsrecords.txt

Amazon Web Services (AWS) Command Line Interface (AWS CLI) – verwenden von High-Level-Befehlen (s3)

Freitag, März 19th, 2021

Amazon Web Services (AWS) Command Line Interface (AWS CLI) – verwenden von High-Level-Befehlen (s3)

Amazon Web Services (AWS) Command Line Interface (AWS CLI) – how to configure and use AWS CLI to manage data with MinIO Server

Donnerstag, März 18th, 2021

Feel free to use this service for testing and development and replace with your own MinIO keys

C:\>aws configure
AWS Access Key ID [None]: < Access Key >
AWS Secret Access Key [None]:  < Secret Access Key >
Default region name [None]: ENTER
Default output format [None]: ENTER

Additionally enable AWS Signature Version ‚4‘ for MinIO server

C:\>aws configure set default.s3.signature_version s3v4

To list your buckets

C:\>aws –endpoint-url s3 ls
2020-12-11 13:06:51 bucket-governance
2021-03-08 14:23:23 bucket-homeoffice
2020-12-12 11:02:20 bucket-versioning
2020-12-11 12:53:16 bucket-worm

To list contents inside bucket

C:\>aws –endpoint-url s3 ls s3://bucket-homeoffice
2021-03-08 14:25:07 43949 S3Browser_01.jpg
2021-03-08 14:25:03 129627 S3Browser_02.jpg

To make a bucket

C:\>aws –endpoint-url s3 mb s3://bucket-homeoffice-02
make_bucket: bucket-homeoffice-02

To add an object to a bucket

C:\>aws –endpoint-url s3 cp c:\tmp\S3Browser_03.jpg s3://bucket-homeoffice-02
upload: tmp\S3Browser_03.jpg to s3://bucket-homeoffice-02/S3Browser_03.jpg

To delete an object from a bucket

C:\>aws –endpoint-url s3 rm s3://bucket-homeoffice-02/S3Browser_03.jpg
delete: s3://bucket-homeoffice-02/S3Browser_03.jpg

To remove a bucket

C:\>aws –endpoint-url s3 rb s3://bucket-homeoffice-02
remove_bucket: bucket-homeoffice-02

To sync (rsync-like) between local storage and s3

C:\>aws –endpoint-url s3 mb s3://bucket-homeoffice-02
make_bucket: bucket-homeoffice-02
C:\>aws –endpoint-url s3 sync c:\tmp s3://bucket-homeoffice-02
upload: tmp\S3Browser_03.jpg to s3://bucket-homeoffice-02/S3Browser_03.jpg
C:\>aws –endpoint-url s3 sync c:\tmp s3://bucket-homeoffice-02
C:\>copy c:\tmp\S3Browser_03.jpg c:\tmp\S3Browser_04.jpg
1 Datei(en) kopiert.

Syncs directories and S3 prefixes recursively copies new and updated files from the source directory to the destination only creates folders in the destination if they contain one or more files

C:\>aws –endpoint-url s3 sync c:\tmp s3://bucket-homeoffice-02
upload: tmp\S3Browser_04.jpg to s3://bucket-homeoffice-02/S3Browser_04.jpg


S3cmd – is a free command line tool and client for uploading retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol

Samstag, März 13th, 2021

Edit the following fields in your s3cmd configuration file ~/.s3cfg

Copy# Setup endpoint
host_base =
host_bucket =
bucket_location = us-east-1
use_https = True
access_key = < Access Key >
secret_key = < Secret Access Key >
# Enable S3 v4 signature APIs
signature_v2 = False

To make a bucket

# 3cmd mb s3://mybucket Bucket ’s3://mybucket/‘ created

To copy an object to bucket

# s3cmd put newfile s3://testbucket upload: ’newfile‘ -> ’s3://testbucket/newfile‘

To copy an object to local system

# s3cmd get s3://testbucket/newfile download: ’s3://testbucket/newfile‘ -> ‚./newfile‘

To sync local file/directory to a bucket

# s3cmd sync newdemo s3://testbucket upload: ’newdemo/newdemofile.txt‘ -> ’s3://testbucket/newdemo/newdemofile.txt‘

To sync bucket or object with local filesystem

# s3cmd sync s3://testbucket otherlocalbucket download: ’s3://testbucket/cat.jpg‘ -> ‚otherlocalbucket/cat.jpg

To list buckets

# s3cmd ls s3:// 2015-12-09 16:12 s3://testbbucket

To list contents inside bucket

# s3cmd ls s3://testbucket/ DIR s3://testbucket/test/
2015-12-09 16:05 138504 s3://testbucket/newfile

To delete an object from bucket

# s3cmd del s3://testbucket/newfile delete: ’s3://testbucket/newfile‘

To delete a bucket

# s3cmd rb s3://mybucket Bucket ’s3://mybucket/‘ removed

Amazon Web Services (AWS) S3 Browser Version 9.5.5 – provides a simple web services interface that can be used to store and retrieve any amount of data at any time from anywhere on the web

Montag, März 8th, 2021

  S3 Browser – improved support for S3-compatible storages (Openstack, Ceph, MinIO)

MinIO – ein Distributed Cluster erstellen

Montag, März 8th, 2021

MinIO – Funktionen im Überblick // SSL / Replikation / Gateway

Sonntag, März 7th, 2021

MinIO – Objekt Storage erklärt & installiert

Sonntag, März 7th, 2021

lakeFS – is an open source platform that delivers resilience and manageability to your existing object-storage based data lake ( compatible with MinIO )

Mittwoch, Januar 6th, 2021

MinIO – introduces continuous availability and active-active bucket replication

Mittwoch, Dezember 30th, 2020

MinIO – setting a bucket for anonymous download

Montag, Dezember 28th, 2020

   MinIO – setting a bucket for anonymous download