Archive for the ‘MinIO S3 Storage’ Category

MinIO Blog – offers high-performance for S3 compatible object storage

Mittwoch, September 15th, 2021

MinIO Blog – offers high-performance for S3 compatible object storage

MinIO – includes multiple mechanisms that ensure data is continuously protected

Mittwoch, September 1st, 2021

   MinIO – Continuous Data Protection with MinIO Versioning and Rewind

MinIO Console – enables efficient cloud scale operations

Montag, August 30th, 2021

MinIO Erasure Code Calculator – will help you determine your raw and usable capacity across a range of erasure coding settings

Dienstag, August 3rd, 2021

Docker Official Image ‚MinIO/KES‘ – Key Encryption Service (KES) a stateless and distributed key service

Samstag, Juli 3rd, 2021

  Docker Official Image ‚MinIO/KES‘ Key Encryption Service – is a stateless and distributed key-management system for high-performance applications we built KES as the bridge between modern applications – running as containers on Kubernetes and centralized KMS solutions therefore  KES has been designed to be simple scalable and secure by default it has just a few knobs to tweak instead of a complex configuration and does not require a deep understanding of secure key-management or cryptography

VMware vSphere ESXi 7.0 Hypervisor(s) – set up a base network topology for a workshop environment with MinIO S3 storage

Montag, Juni 21st, 2021

MinIO – install on Linux hosts running 64-bit Intel/AMD architectures

# wget https://dl.min.io/server/minio/release/linux-amd64/minio
# chmod 775 minio
# mkdir /data
# ./minio server /data

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console an embedded
web-based object browser built into MinIO Server a web browser running on the host machine to http://192.168.27.252:9000 and log in with the
root credentials you can use the Browser to create buckets upload objects and browse the contents of the MinIO server

# wget https://dl.min.io/client/mc/release/linux-amd64/mc
# chmod 775 mc
# ./mc –help

Docker Official Image ‚MinIO‘ – Object Locking

Donnerstag, Mai 27th, 2021

   MinIO – supports a complete object locking framework offering both Legal Hold and Retention with Governance and Compliance modes. Object Locking functionality is a requirement for many regulated industries from financial services to healthcare. Lifecycle management is an increasingly critical element in the data ecosystem. Data is the primary asset in most organizations at this point – more so than the physical assets we normally associate with large enterprises – trains, planes, automobiles or oil rigs. Protecting this data and managing how long to protect it are complex challenges. MinIO now offers the complete suite of lifecycle management capabilities. This includes versioning, object locking and the various derivative components. Every use case is different and our goal is to create software and documentation that allow our community to implement these features with confidence

Governance Mode

When setting a retention period for your objects or buckets Governance mode is used when the goal is to protect objects from being deleted by standard users having said that there are users that will retain the permissions required to modify the retention settings or delete the objects those users will need the s3:BypassGovernanceRetention permission as well as the DeleteObject permission

$ ./mc mb myminio/bucket-worm –with-lock
Bucket created successfully `myminio/bucket-worm`
$ ./mc mb myminio/bucket-governance –with-lock
Bucket created successfully `myminio/bucket-governance`
$ ./mc retention set –default governance 7d myminio/bucket-governance
GOVERNANCE mode is enabled for 7DAYS

Compliance Mode

Compliance mode is more restrictive it cannot be undone within the retention period as a result the Compliance mode should be used when you are certain that you do not want anyone including the root user to be able to delete the objects during its retention period

$ ./mc mb myminio/bucket-compliance –with-lock
Bucket created successfully `myminio/bucket-compliance`
$ ./mc retention set –default compliance 7d myminio/bucket-compliance
COMPLIANCE mode is enabled for 7DAYS

Docker Official Image ‚MinIO‘ – Versioning

Donnerstag, Mai 27th, 2021

   MinIO  – Versioning allows a user to retain multiple variants of an object in the same bucket. Versioning provides a mechanism to preserve, retrieve, and restore every version of every object stored in a bucket. Versioning ensures that objects remain available across a range of failures from applications to human error. Versioning is enabled at the bucket level. When enabled MinIO automatically creates a unique version ID for the object there can be multiple versions of the same object

$ ./mc mb myminio/bucket-versioning
Bucket created successfully `myminio/bucket-versioning`
$ ./mc version enable myminio/bucket-versioning
myminio/bucket-versioning versioning is enabled

$ ./mc ls –versions myminio/bucket-versioning
[2021-05-27 08:20:29 UTC] 726KiB 898d0538-61d6-4c68-a59d-8509d4c8cb02 v3 PUT josef.jpg
[2021-05-27 08:20:06 UTC] 726KiB 2e088685-1262-4b27-8b81-5699df1024ef v2 PUT josef.jpg
[2021-05-27 08:19:54 UTC] 726KiB 36c2b0fa-6858-4f02-947f-1b875d340681 v1 PUT josef.jpg
$ ./mc cp –version-id 36c2b0fa-6858-4f02-947f-1b875d340681 myminio/bucket-versioning/josef.jpg /tmp
…-versioning/josef.jpg: 725.50 KiB / 725.50 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 2.70 MiB/s 0s
$ ./mc du –versions myminio/bucket-versioning
2.1MiB bucket-versioning

$ ./mc rm –version-id 36c2b0fa-6858-4f02-947f-1b875d340681 myminio/bucket-versioning/josef.jpg
Removing `myminio/bucket-versioning/josef.jpg` (versionId=36c2b0fa-6858-4f02-947f-1b875d340681)
$ ./mc du –versions myminio/bucket-versioning
1.4MiB bucket-versioning
$ ./mc ls –versions myminio/bucket-versioning
[2021-05-27 08:20:29 UTC] 726KiB 898d0538-61d6-4c68-a59d-8509d4c8cb02 v2 PUT josef.jpg
[2021-05-27 08:20:06 UTC] 726KiB 2e088685-1262-4b27-8b81-5699df1024ef v1 PUT josef.jpg

s3-benchmark – performance benchmarking methodologies used to test the performance of different cloud storage providers

Montag, Mai 24th, 2021

s3-benchmark – for 10 threads with the default 1MB object size. The benchmark reports for each operation PUT GET and DELETE the results in terms of data speed and operations per second the program writes all results to the log file benchmark.log

root@pve-02-debian-10:~# apt-get update -y
root@pve-02-debian-10:~# apt-get full-upgrade -y

root@pve-02-debian-10:~# apt-get install git

root@pve-02-debian-10:~# git clone https://github.com/wasabi-tech/s3-benchmark.git

root@pve-02-debian-10:~/s3-benchmark# ./s3-benchmark -u http://192.168.1.151:9000 -a <AccessKey> -b benchmark -s <SecretKey> -t 10
Wasabi benchmark program v2.0
Parameters: url=http://192.168.1.151:9000, bucket=benchnark, region=us-east-1, duration=60, threads=10, loops=1, size=1M
Loop 1: PUT time 64.1 secs, objects = 110, speed = 1.7MB/sec, 1.7 operations/sec. Slowdowns = 0
Loop 1: GET time 60.2 secs, objects = 3584, speed = 59.6MB/sec, 59.6 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.3 secs, 85.5 deletes/sec. Slowdowns = 0

root@pve-02-debian-10:~/s3-benchmark#

Docker Official Image ‚MinIO‘ – object browser

Donnerstag, Mai 20th, 2021

Prometheus 2.27.0 – how to monitor MinIO server

Samstag, Mai 15th, 2021

Prometheus – offers a multi-dimensional data model with time series data identified by metric name and key/value pairs the data collection happens via a pull model over HTTP/HTTPS these metrics can be from any MinIO server once per collection e.g.

minio_node_disk_total_bytes  –  Total storage on a disk
minio_node_disk_used_bytes  –  Total storage used on a disk

Docker Official Images – minio/warp S3 benchmarking tool

Samstag, Mai 15th, 2021

https://hub.docker.com/r/minio/warp

S3 Benchmark – from any location using different thread counts and object sizes

Freitag, Mai 14th, 2021
#  curl -OL https://github.com/dvassallo/s3-benchmark/raw/master/build/linux-arm64/s3-benchmark

MinIO Client (mc) – the ‚mirror‘ command synchronizes data between filesystems and object storages similarly to rsync

Donnerstag, Mai 13th, 2021

  MinIO Client (mc) – provides a modern alternative to Linux commands like ls, cat, cp, mirror, diff etc. and it supports filesystems and Amazon S3 compatible cloud storage service (AWS Signature v2 and v4)

MinIO Client (mc) – Quickstart Guide

USAGE:
mc mirror [FLAGS] SOURCE TARGET
FLAGS:
–overwrite overwrite object(s) on target if it differs from source
–fake perform a fake mirror operation
–watch, -w watch and synchronize changes
–remove remove extraneous object(s) on target
–region value specify region when creating new bucket(s) on target (default: „us-east-1“)
–preserve, -a preserve file system attributes and bucket policy rules on target bucket(s)
–exclude value exclude object(s) that match specified object name pattern
–older-than value filter object(s) older than N days (default: 0)
–newer-than value filter object(s) newer than N days (default: 0)
–storage-class value, –sc value specify storage class for new object(s) on target
–encrypt value encrypt/decrypt objects (using server-side encryption with server managed keys)
–encrypt-key value encrypt/decrypt objects (using server-side encryption with customer provided keys)
–help, -h show help
C:\>mc
mc: Configuration written to `C:\Users\Josef\mc\config.json`. Please update your
access credentials.
mc: Successfully created `C:\Users\Josef\mc\share`.
mc: Initialized share uploads `C:\Users\Josef\mc\share\uploads.json` file.
mc: Initialized share downloads `C:\Users\Josef\mc\share\downloads.json` file.
NAME:
mc – MinIO Client for cloud storage and filesystems.
USAGE:
mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS…]
COMMANDS:
alias set, remove and list aliases in configuration file
ls list buckets and objects
mb make a bucket
rb remove a bucket
cp copy objects
mirror synchronize object(s) to a remote site
cat display object contents
head display first ’n‘ lines of an object
pipe stream STDIN to an object
share generate URL for temporary access to an object
find search for objects
sql run sql queries on objects
stat show object metadata
mv move objects
tree list buckets and objects in a tree format
du summarize disk usage recursively
retention set retention for object(s)
legalhold manage legal hold for object(s)
diff list differences in object name, size, and date between two buckets
rm remove objects
version manage bucket versioning
ilm manage bucket lifecycle
encrypt manage bucket encryption config
event manage object notifications
watch listen for object notification events
undo undo PUT/DELETE operations
policy manage anonymous access to buckets and objects
tag manage tags for bucket and object(s)
replicate configure server side bucket replication
admin manage MinIO servers
update update mc to latest release
GLOBAL FLAGS:
–autocompletion install auto-completion for your shell
–config-dir value, -C value path to configuration folder (default: „C:\\User
s\\Josef\\mc“)
–quiet, -q disable progress bar display
–no-color disable color theme
–json enable JSON lines formatted output
–debug enable debug output
–insecure disable SSL certificate verification
–help, -h show help
–version, -v print the version
TIP:
Use ‚mc –autocompletion‘ to enable shell autocompletion
VERSION:
RELEASE.2021-04-22T17-40-00Z

Microsoft Windows Binary Download
https://dl.min.io/client/mc/release/windows-amd64/mc.exe

to disable SSL certificate verification use the option –insecure

C:\>mc config host add myminio http://192.168.1.151:9000 <AccessKey> <SecretKey>
Added `myminio` successfully.
C:\>mc admin update myminio
Server `myminio` updated successfully from 2021-01-16T02:19:44Z to 2021-04-22T15
-44-28Z
C:\>mc mb myminio/photos
Bucket created successfully `myminio/photos`.
C:\>mc cp -r C:\Users\Josef\Photos myminio/photos
…Thumbs.db: 738.00 KiB / 738.00 KiB [======================] 105.47 KiB/s 6s
C:\>mc mirror C:\Users\Josef\Photos myminio/photos
…josef.jpg: 738.00 KiB / 738.00 KiB [======================] 140.04 KiB/s 5s
C:\>mc mirror C:\Users\Josef\Photos myminio/photos
0 B / ? [ =] 0s
C:\>mc mirror C:\Users\Josef\Photos myminio/photos
0 B / ? [ =] 0s
C:\>mc mirror C:\Users\Josef\Photos myminio/photos
0 B / ? [ =] 0s
C:\>

Linux ARM Binary Download

# wget https://dl.minio.io/client/mc/release/linux-arm/mc
# chmod +x ./mc
remove extraneous object(s) on target
# ./mc mirror –remove myminio/photos /mnt/gv0
0 B / ? ┃░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▓┃ 0s
overwrite object(s) on target if it differs from source
# ./mc mirror –overwrite myminio/photos /mnt/gv0
0 B / ? ┃░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▓┃ 0s
watch and synchronize changes
# ./mc mirror –watch myminio/photos /mnt/gv0
0 B / ? ┃░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▓

MinIO – how to setup and run a MinIO Standalone Object Server using a file share to store objects

Mittwoch, Mai 12th, 2021