Archive for the ‘MinIO S3 Storage’ Category

MinIO Client (mc) – troubleshooting with slow MinIO operation

Dienstag, September 27th, 2022
export MC_UPLOAD_MULTIPART_THREADS=1
export GOMAXPROCS=1

DietPi 8.8.1 – einfache Installation von MinIO mit einem Fujitsu Futro S740

Montag, September 26th, 2022

Google Cloud – how to accelerate migration from self-managed object storage to Cloud Storage

Samstag, September 17th, 2022

MinIO – how to use AWS S3 commands

Samstag, September 17th, 2022

MinIO – benchmarks for S3 performance with and without encryption on both hard disk drives and NVMe SSDs

Samstag, September 10th, 2022

MinIO Object Lifecycle Management – allows creating rules for time or date based automatic transition or expiry of objects for backup strategy ‚ProtectIT‘

Mittwoch, September 7th, 2022

 

MinIO does not expire DeleteMarkers even if no other versions of that object exist to expire delete markers when there are no remaining versions for that object specify the –expired-object-delete-marker option when creating the expiration rule

 

root@dpi-iot-jsho-MinIO-01:~# ./mc ilm export myminio/worm-with-versioning-and-retention-camera1
{
„Rules“: [
{
„Expiration“: {
„Days“: 5,
„DeleteMarker“: false
},
„ID“: „cbt8k52mc0l0680hic2g“,
„Status“: „Enabled“
},
{
„Expiration“: {
„DeleteMarker“: true
},
„ID“: „cc11gdimc0l03bsi2bdg“,
„NoncurrentVersionExpiration“: {
„NoncurrentDays“: 2,
„NewerNoncurrentVersions“: 0
},
„Status“: „Enabled“
}
]
}

root@dpi-iot-jsho-MinIO-01:~#

 

… after few days

 

… after few weeks

 

… after few months

MinIO & PwC Canada – announce joint business relationship to build co-sell and co-market cloud-based AI and ML solutions to Canadian enterprises across various industries

Sonntag, September 4th, 2022

MinIO Client (mc) – tree command lists buckets and directories in a tree format

Donnerstag, August 25th, 2022
root@dpi-iot-jsho-MinIO-01:~#
root@dpi-iot-jsho-MinIO-01:~# ./mc tree myminio/worm-with-versioning-and-retention-camera4
myminio/worm-with-versioning-and-retention-camera4
├─ 2022-08-15
├─ 2022-08-16
├─ 2022-08-17
├─ 2022-08-18
├─ 2022-08-19
├─ 2022-08-20
├─ 2022-08-21
├─ 2022-08-22
├─ 2022-08-23
├─ 2022-08-24
└─ 2022-08-25

root@dpi-iot-jsho-MinIO-01:~#

root@dpi-iot-jsho-MinIO-01:~# ./mc tree –files myminio/worm-with-versioning-and-retention-camera4
myminio/worm-with-versioning-and-retention-camera4
├─ 2022-08-15
├─ 2022-08-16
├─ 2022-08-17
├─ 2022-08-18
├─ 2022-08-19
├─ 2022-08-20
│ ├─ 07-23-41.jpg
│ ├─ 07-23-42.jpg
│ ├─ 07-23-43.jpg
│ ├─ 07-23-44.jpg
│ ├─ 07-23-45.jpg
│ ├─ 07-23-46.jpg

    ├─ 10-21-48.jpg
    └─ 10-21-49.jpg

root@dpi-iot-jsho-MinIO-01:~#

MinIO Client (mc) – summarize disk usage recursively

Donnerstag, August 25th, 2022
root@dpi-iot-jsho-MinIO-01:~#
root@dpi-iot-jsho-MinIO-01:~# ./mc du myminio/worm-with-versioning-and-retention-camera4
99MiB 1303 objects worm-with-versioning-and-retention-camera4
root@dpi-iot-jsho-MinIO-01:~# ./mc du –versions myminio/worm-with-versioning-and-retention-camera4
145MiB 1935 versions worm-with-versioning-and-retention-camera4
root@dpi-iot-jsho-MinIO-01:~#

MinIO Client (mc) – perform liveness check

Mittwoch, August 24th, 2022
root@dpi-iot-jsho-MinIO-01:~# ./mc ping myminio/worm-with-versioning-and-retention-camera4
1: http://192.168.1.145:9000 min=20.165ms max=20.165ms average=20.165ms errors=0 roundtrip=20.165ms
2: http://192.168.1.145:9000 min=20.165ms max=24.248ms average=22.207ms errors=0 roundtrip=24.248ms
3: http://192.168.1.145:9000 min=20.165ms max=24.248ms average=22.084ms errors=0 roundtrip=21.84ms
4: http://192.168.1.145:9000 min=7.128ms max=24.248ms average=18.345ms errors=0 roundtrip=7.128ms
5: http://192.168.1.145:9000 min=6.09ms max=24.248ms average=15.894ms errors=0 roundtrip=6.09ms
6: http://192.168.1.145:9000 min=6.09ms max=24.248ms average=14.878ms errors=0 roundtrip=9.795ms
7: http://192.168.1.145:9000 min=6.09ms max=24.248ms average=15.833ms errors=0 roundtrip=21.566ms
8: http://192.168.1.145:9000 min=6.09ms max=24.248ms average=15.265ms errors=0 roundtrip=11.289ms
9: http://192.168.1.145:9000 min=6.09ms max=24.248ms average=14.476ms errors=0 roundtrip=8.161ms
10: http://192.168.1.145:9000 min=6.09ms max=30.967ms average=16.125ms errors=0 roundtrip=30.967ms
11: http://192.168.1.145:9000 min=6.09ms max=30.967ms average=15.217ms errors=0 roundtrip=6.135ms
12: http://192.168.1.145:9000 min=6.09ms max=30.967ms average=14.623ms errors=0 roundtrip=8.091ms

root@dpi-iot-jsho-MinIO-01:~#

MinIO Object Transition (“Tiering”) – supports creating object transition lifecycle management rules where MinIO can automatically move an object to a remote storage tier

Sonntag, August 21st, 2022

MinIO Object Lifecycle Management – allows creating rules for time or date based automatic transition or expiry of objects for backup strategy ‚ProtectIT‘

Sonntag, August 21st, 2022

 

root@dpi-iot-jsho-MinIO-01:~# ./mc ilm export myminio/worm-with-versioning-and-retention-camera1
{
„Rules“: [
{
„Expiration“: {
„Days“: 5,
„DeleteMarker“: false
},
„ID“: „cbt8k52mc0l0680hic2g“,
„Status“: „Enabled“
}
]
}
root@dpi-iot-jsho-MinIO-01:~#
root@dpi-iot-jsho-MinIO-01:~# ./mc ilm ls myminio/worm-with-versioning-and-retention-camera2
ID | Type | Version | Tier | Date/Days | Prefix | Enabled | Tags
———————-|————|————–|——————|————–|—————-|————|————————
cbt8l7amc0l0680hic30 | Expiry | Current | | 5 day(s) | | ✔ |
———————-|————|————–|——————|————–|—————-|————|————————
root@dpi-iot-jsho-MinIO-01:~#

 

root@dpi-iot-jsho-MinIO-01:~# ./mc ilm export myminio/worm-with-versioning-and-retention-camera1
{
„Rules“: [
{
„Expiration“: {
„Days“: 5,
„DeleteMarker“: false
},
„ID“: „cbt8k52mc0l0680hic2g“,
„Status“: „Enabled“
},
{
„ID“: „cc11gdimc0l03bsi2bdg“,
„NoncurrentVersionExpiration“: {
„NoncurrentDays“: 5,
„NewerNoncurrentVersions“: 0
},
„Status“: „Enabled“
}
]
}

root@dpi-iot-jsho-MinIO-01:~#

root@dpi-iot-jsho-MinIO-01:~# ./mc ilm ls myminio/worm-with-versioning-and-retention-camera1
ID | Type | Version | Tier | Date/Days | Prefix | Enabled | Tags
———————-|————|————–|——————|————–|—————-|————|————————
cbt8k52mc0l0680hic2g | Expiry | Current | | 5 day(s) | | ✔ |
———————-|————|————–|——————|————–|—————-|————|————————
cc11gdimc0l03bsi2bdg | Expiry | Non-Current | | 5 day(s) | | ✔ |
———————-|————|————–|——————|————–|—————-|————|————————
root@dpi-iot-jsho-MinIO-01:~#

MinIO Object Lifecycle Management – allows creating rules for time or date based automatic transition or expiry of objects

Montag, August 15th, 2022

DietPi 8.7.1 – einfacher Backup mit einem MinIO S3 Object Storage und einem Raspberry Pi Zero W

Sonntag, August 14th, 2022
root@rpi-iot-jsho-cam-02:~# crontab -l
59 01 * * * /root/MinIOMotionEye-mc-update.sh >> /MotionEye.log/mc-update.log 2>&1
59 02 * * * /root/MinIOMotionEye-mc-rm.sh >> /MotionEye.log/mc-rm.log 2>&1
59 03-23 * * * /root/MinIOMotionEye-mc-mirror.sh >> /MotionEye.log/mc-mirror.log 2>&1

root@rpi-iot-jsho-cam-02:~#
root@rpi-iot-jsho-cam-02:~# vi MinIOMotionEye-mc-mirror.sh
#!/bin/bash
#FileName: MinIOMotionEye-mc-mirror.sh
#set -x
MirrorDay=`date „+%Y-%m-%d“`
sleep 60
echo ———————————————————
date „+Start mc mirror motioneye/Camera1: %Y-%m-%d-%H:%M:%S“
echo
LocalBackupPath=“/var/lib/motioneye/Camera1″
MinioBucket=“myminio/worm-with-versioning-and-retention-camera1″
MCPATH=“/root/mc“
$MCPATH mirror –json –overwrite –remove –preserve $LocalBackupPath/$MirrorDay $MinioBucket/$MirrorDay
echo
date „+End mc mirror motioneye/Camera1: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
sleep 300
echo ———————————————————
date „+Start mc mirror motioneye/Camera2: %Y-%m-%d-%H:%M:%S“
echo
LocalBackupPath=“/var/lib/motioneye/Camera2″
MinioBucket=“myminio/worm-with-versioning-and-retention-camera2″
MCPATH=“/root/mc“
$MCPATH mirror –json –overwrite –remove –preserve $LocalBackupPath/$MirrorDay $MinioBucket/$MirrorDay
echo
date „+End mc mirror motioneye/Camera2: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
sleep 300
echo ———————————————————
date „+Start mc mirror motioneye/Camera3: %Y-%m-%d-%H:%M:%S“
echo
LocalBackupPath=“/var/lib/motioneye/Camera3″
MinioBucket=“myminio/worm-with-versioning-and-retention-camera3″
MCPATH=“/root/mc“
$MCPATH mirror –json –overwrite –remove –preserve $LocalBackupPath/$MirrorDay $MinioBucket/$MirrorDay
echo
date „+End mc mirror motioneye/Camera3: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
sleep 300
echo ———————————————————
date „+Start mc mirror motioneye/Camera4: %Y-%m-%d-%H:%M:%S“
echo
LocalBackupPath=“/var/lib/motioneye/Camera4″
MinioBucket=“myminio/worm-with-versioning-and-retention-camera4″
MCPATH=“/root/mc“
$MCPATH mirror –json –overwrite –remove –preserve $LocalBackupPath/$MirrorDay $MinioBucket/$MirrorDay
echo
date „+End mc mirror motioneye/Camera4: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
root@rpi-iot-jsho-cam-02:~#
root@rpi-iot-jsho-cam-02:~# vi MinIOMotionEye-mc-update.sh
#!/bin/bash
#FileName: MinIOMotionEye-mc-update.sh
echo ———————————————————
date „+Start mc mirror: %Y-%m-%d-%H:%M:%S“
echo
MCPATH=“/root/mc“
$MCPATH update
$MCPATH admin update myminio
echo
date „+End mc mirror: %Y-%m-%d-%H:%M:%S“
echo ———————————————————

 

root@rpi-iot-jsho-cam-02:~# vi MinIOMotionEye-mc-rm.sh
#!/bin/bash
#FileName: MinIOMotionEye-mc-rm.sh
echo ———————————————————
date „+Start mc rm: %Y-%m-%d-%H:%M:%S“
echo
MinioBucket=“myminio/worm-with-versioning-and-retention-camera1″
MCPATH=“/root/mc“
$MCPATH rm –json –force –versions –recursive –older-than 7d $MinioBucket
echo
date „+End mc rm: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
sleep 300
echo ———————————————————
date „+Start mc rm: %Y-%m-%d-%H:%M:%S“
echo
MinioBucket=“myminio/worm-with-versioning-and-retention-camera2″
MCPATH=“/root/mc“
$MCPATH rm –json –force –versions –recursive –older-than 7d $MinioBucket
echo
date „+End mc rm: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
sleep 300
echo ———————————————————
date „+Start mc rm: %Y-%m-%d-%H:%M:%S“
echo
MinioBucket=“myminio/worm-with-versioning-and-retention-camera3″
MCPATH=“/root/mc“
$MCPATH rm –json –force –versions –recursive –older-than 7d $MinioBucket
echo
date „+End mc rm: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
sleep 300
echo ———————————————————
date „+Start mc rm: %Y-%m-%d-%H:%M:%S“
echo
MinioBucket=“myminio/worm-with-versioning-and-retention-camera4″
MCPATH=“/root/mc“
$MCPATH rm –json –force –versions –recursive –older-than 7d $MinioBucket
echo
date „+End mc rm: %Y-%m-%d-%H:%M:%S“
echo ———————————————————
root@rpi-iot-jsho-cam-02:~#

DietPi 8.7.1 – einfache Installation von MinIO auf einem Raspberry Pi Zero W

Dienstag, August 2nd, 2022

root@dpi-iot-jsho-MinIO-01:~# dietpi-launcher
[ .. ] DietPi-Software | Initialising database
[ OK ] DietPi-Software | Initialised database
[ OK ] DietPi-Software | Reading database
[ OK ] DietPi-Software | DietPi-Userdata validation: /mnt/dietpi_userdata
DietPi-Software
─────────────────────────────────────────────────────
Step: Checking for conflicts and missing inputs
DietPi-Software
─────────────────────────────────────────────────────
Step: Checking for prerequisite software
[ OK ] DietPi-Software | Checking IPv4 network connectivity
[ OK ] DietPi-Software | Checking IPv6 network connectivity
[ OK ] DietPi-Software | Checking DNS resolver
[ OK ] DietPi-Software | mkdir -p /mnt/dietpi_userdata/Music /mnt/dietpi_userdata/Pictures /mnt/dietpi_userdata/Video /mnt/dietpi_userdata/downloads /var/www /opt /usr/local/bin
[ OK ] DietPi-Software | chown dietpi:dietpi /mnt/dietpi_userdata/Music /mnt/dietpi_userdata/Pictures /mnt/dietpi_userdata/Video /mnt/dietpi_userdata/downloads
[ OK ] DietPi-Software | chmod 0775 /mnt/dietpi_userdata/Music /mnt/dietpi_userdata/Pictures /mnt/dietpi_userdata/Video /mnt/dietpi_userdata/downloads
[ SUB1 ] DietPi-Services > unmask
[ OK ] DietPi-Services | unmask : cron
[ SUB1 ] DietPi-Services > stop
[ OK ] DietPi-Services | stop : cron
[ INFO ] DietPi-Software | APT update, please wait…
Hit:1 https://archive.raspberrypi.org/debian bullseye InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian bullseye InRelease
Reading package lists…
[ OK ] DietPi-Software | APT update
DietPi-Software
─────────────────────────────────────────────────────
Step: Installing MinIO: S3 compatible distributed object server
[ OK ] DietPi-Software | curl -sSfLo /usr/local/bin/minio https://dl.minio.io/server/minio/release/linux-arm/minio
[ OK ] DietPi-Software | chmod +x /usr/local/bin/minio
[ OK ] DietPi-Software | curl -sSfLo /etc/systemd/system/minio.service https://github.com/minio/minio-service/raw/master/linux-systemd/minio.service
[ OK ] DietPi-Software | Verifying user „minio-user“ to run MinIO
[ OK ] DietPi-Software | useradd -rMU -d /mnt/dietpi_userdata/minio-data -s /usr/sbin/nologin minio-user
[ OK ] DietPi-Software | mkdir /mnt/dietpi_userdata/minio-data
[ OK ] DietPi-Software | chown -R minio-user:minio-user /mnt/dietpi_userdata/minio-data
DietPi-Software
─────────────────────────────────────────────────────
Step: Finalising install
[ OK ] DietPi-Software | systemctl daemon-reload
2022-08-02 16:30:48 [ INFO ] DietPi-RAMlog | Storing /var/log to /var/tmp/dietpi/logs/dietpi-ramlog_store…
2022-08-02 16:30:48 [ OK ] DietPi-RAMlog | Stored /var/log to /var/tmp/dietpi/logs/dietpi-ramlog_store.
[ SUB1 ] DietPi-Services > dietpi_controlled
[ OK ] DietPi-Services | dietpi_controlled : minio
[ OK ] DietPi-Services | dietpi_controlled : cron
DietPi-Software
─────────────────────────────────────────────────────
Step: Install completed
[ OK ] DietPi-Survey | Purging survey data
[ SUB1 ] DietPi-Services > restart
[ OK ] DietPi-Services | restart : minio
[ OK ] DietPi-Services | restart : cron

root@dpi-iot-jsho-MinIO-01:~#
root@dpi-iot-jsho-MinIO-01:~# vi /etc/default/minio
MINIO_ACCESS_KEY=“admin“
MINIO_SECRET_KEY=“<password>“