Archive for the ‘Proxmox’ Category

Amazon Web Services (AWS) – hat seinen Hostingdienst für das bisher genutzte Elasticsearch offiziell in Amazon Opensearch Service umbenannt

Samstag, September 11th, 2021

  Amazon Web Services (AWS) OpenSearch – ist eine verteilte Open Source Such- und Analyse-Suite die für eine breite Palette von Anwendungsfällen wie Echtzeit-Anwendungsüberwachung, Protokollanalyse und Website-Suche verwendet wird. OpenSearch bietet ein hochgradig skalierbares System für den schnellen Zugriff und die Reaktion auf große Datenmengen mit einem integrierten Visualisierungswerkzeug OpenSearch Dashboards das es den Nutzern leicht macht ihre Daten zu untersuchen. Wie Elasticsearch und Apache Solr basiert auch OpenSearch auf der Suchbibliothek Apache Lucene. OpenSearch und OpenSearch Dashboards wurden ursprünglich von Elasticsearch 7.10.2 und Kibana 7.10.2 abgeleitet

Elastic Machine Learning – Outlier Detection

Samstag, August 28th, 2021

Elastic Machine Learning – Tips and Tricks with Categorization

Freitag, August 27th, 2021

Elastic Machine Learning – anomaly detection

Montag, August 23rd, 2021

Elastic Machine Learning – from zero to hero

Sonntag, August 22nd, 2021

Beginner’s Crash Course to Elastic Stack – Troubleshooting Errors Part 6

Sonntag, August 22nd, 2021

Beginner’s Crash Course to Elastic Stack – Mapping Part 5

Sonntag, August 22nd, 2021

Beginner’s Crash Course to Elastic Stack – Aggregations Part 4

Samstag, August 21st, 2021

Elasticsearch Tutorial – Analyzing Logs with Kibana Dashboards

Samstag, August 21st, 2021

Elasticsearch Tutorial For Beginners – start with the installation on Debian 11 „Bullseye“ LinuxContainer (LXC) in Proxmox 7.0-11

Samstag, August 21st, 2021
root@Elasticsearch-10:~#
root@Elasticsearch-10:~# apt-get update -y
root@Elasticsearch-10:~# apt-get full-upgrade -y
root@Elasticsearch-10:~# apt-get install gnupg gnupg2 gnupg2 -y
root@Elasticsearch-10:~# wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add –
OK
root@Elasticsearch-10:~# apt-get install apt-transport-https
root@pve-02-ubuntu-2:~# echo „deb https://artifacts.elastic.co/packages/7.x/apt stable main“ | tee -a /etc/apt/sources.list.d/elastic-7.x.list
deb https://artifacts.elastic.co/packages/7.x/apt stable main
root@pve-02-ubuntu-2:~# apt-get update -y
root@pve-02-ubuntu-2:~# apt-get install elasticsearch
root@pve-02-ubuntu-2:~# systemctl daemon-reload
root@pve-02-ubuntu-2:~# systemctl enable elasticsearch.service
root@pve-02-ubuntu-2:~# systemctl start elasticsearch.service
root@Elasticsearch-10:~# apt-get install curl
root@Elasticsearch-10:~# curl http://localhost:9200
{
„name“ : „Elasticsearch-10“,
„cluster_name“ : „elasticsearch“,
„cluster_uuid“ : „7XCqjwV7SdWbMOCmVW8xVw“,
„version“ : {
„number“ : „7.14.0“,
„build_flavor“ : „default“,
„build_type“ : „deb“,
„build_hash“ : „dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1“,
„build_date“ : „2021-07-29T20:49:32.864135063Z“,
„build_snapshot“ : false,
„lucene_version“ : „8.9.0“,
„minimum_wire_compatibility_version“ : „6.8.0“,
„minimum_index_compatibility_version“ : „6.0.0-beta1“
},
„tagline“ : „You Know, for Search“
}
root@Elasticsearch-10:~#

root@Elasticsearch-10:~# systemctl status elasticsearch.service
* elasticsearch.service – Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-08-18 12:26:01 UTC; 3min 56s ago
Docs: https://www.elastic.co
Main PID: 149 (java)
Tasks: 64 (limit: 17848)
Memory: 8.0G
CPU: 1min 17.404s
CGroup: /system.slice/elasticsearch.service
|-149 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch >
`-498 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Aug 18 12:25:37 Elasticsearch-10 systemd[1]: Starting Elasticsearch…
Aug 18 12:26:01 Elasticsearch-10 systemd[1]: Started Elasticsearch.
root@Elasticsearch-10:~# curl -X GET „localhost:9200/_cat/indices?v&pretty“
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases hBK8M62nQSagipE5KxDD0Q 1 0 42 0 40.1mb 40.1mb
root@Elasticsearch-10:~# curl -XPUT ‚localhost:9200/products/mobiles/1?pretty‘ -H ‚Content-Type: application/json‘ -d‘
{
„name“: „iPhone 7“,
„camera“: „12MP“,
„storage“: „256GB“,
„display“: „4.7inch“,
„battery“: „1,960mAh“,
„reviews“: [„Incredibly happy after having used it for one week“, „Best iPhone so far“, „Very expensive, stick to Android“]
}

{
„_index“ : „products“,
„_type“ : „mobiles“,
„_id“ : „1“,
„_version“ : 1,
„result“ : „created“,
„_shards“ : {
„total“ : 2,
„successful“ : 1,
„failed“ : 0
},
„_seq_no“ : 0,
„_primary_term“ : 1
}
root@Elasticsearch-10:~# curl -X GET „localhost:9200/_cat/indices?v&pretty“
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases hBK8M62nQSagipE5KxDD0Q 1 0 42 0 40.1mb 40.1mb
yellow open products 8AkZHUrdQ1yTv12KjQ5oJA 1 1 1 0 7.2kb 7.2kb
root@Elasticsearch-10:~#
root@Elasticsearch-10:~# curl -XGET „localhost:9200/products/_search?pretty“ -H ‚Content-Type: application/json‘
{
„took“ : 4,
„timed_out“ : false,
„_shards“ : {
„total“ : 1,
„successful“ : 1,
„skipped“ : 0,
„failed“ : 0
},
„hits“ : {
„total“ : {
„value“ : 1,
„relation“ : „eq“
},
„max_score“ : 1.0,
„hits“ : [
{
„_index“ : „products“,
„_type“ : „mobiles“,
„_id“ : „1“,
„_score“ : 1.0,
„_source“ : {
„name“ : „iPhone 7“,
„camera“ : „12MP“,
„storage“ : „256GB“,
„display“ : „4.7inch“,
„battery“ : „1,960mAh“,
„reviews“ : [
„Incredibly happy after having used it for one week“,
„Best iPhone so far“,
„Very expensive, stick to Android“
]
}
}
]
}
}
root@Elasticsearch-10:~# curl -XGET „localhost:9200/products/_search?pretty“ -H ‚Content-Type: application/json‘ -d‘
{
„query“: {
„match_phrase“: {
„camera“: „12MP“
}
}
}‘
{
„took“ : 7,
„timed_out“ : false,
„_shards“ : {
„total“ : 1,
„successful“ : 1,
„skipped“ : 0,
„failed“ : 0
},
„hits“ : {
„total“ : {
„value“ : 1,
„relation“ : „eq“
},
„max_score“ : 0.2876821,
„hits“ : [
{
„_index“ : „products“,
„_type“ : „mobiles“,
„_id“ : „1“,
„_score“ : 0.2876821,
„_source“ : {
„name“ : „iPhone 7“,
„camera“ : „12MP“,
„storage“ : „256GB“,
„display“ : „4.7inch“,
„battery“ : „1,960mAh“,
„reviews“ : [
„Incredibly happy after having used it for one week“,
„Best iPhone so far“,
„Very expensive, stick to Android“
]
}
}
]
}
}

### Loading sample data – the complete works of William Shakespeare suitably parsed into fields shakespeare.json

root@Elasticsearch-10:~# curl -X GET „localhost:9200/_cat/indices?v&pretty“
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases hBK8M62nQSagipE5KxDD0Q 1 0 42 38 40.1mb 40.1mb
yellow open products 8AkZHUrdQ1yTv12KjQ5oJA 1 1 1 0 7.3kb 7.3kb
root@Elasticsearch-10:~# curl -XPUT „localhost:9200/shakespeare/_bulk?pretty“ -H ‚Content-Type: application/json‘ –data-binary @shakespeare_6.0.json
root@Elasticsearch-10:~# curl -X GET „localhost:9200/_cat/indices?v&pretty“
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases hBK8M62nQSagipE5KxDD0Q 1 0 42 38 40.1mb 40.1mb
yellow open shakespeare -YUARRYPSYO170-U1-N6gg 1 1 111396 0 19mb 19mb
yellow open products 8AkZHUrdQ1yTv12KjQ5oJA 1 1 1 0 7.3kb 7.3kb
root@Elasticsearch-10:~# curl -XGET „localhost:9200/shakespeare/_search?pretty“ -H ‚Content-Type: application/json‘ -d‘
{
„query“: {
„match_phrase“: {
„text_entry“: „So shaken as we are“
}
}
}

{
„took“ : 27,
„timed_out“ : false,
„_shards“ : {
„total“ : 1,
„successful“ : 1,
„skipped“ : 0,
„failed“ : 0
},
„hits“ : {
„total“ : {
„value“ : 1,
„relation“ : „eq“
},
„max_score“ : 21.60624,
„hits“ : [
{
„_index“ : „shakespeare“,
„_type“ : „_doc“,
„_id“ : „3“,
„_score“ : 21.60624,
„_source“ : {
„type“ : „line“,
„line_id“ : 4,
„play_name“ : „Henry IV“,
„speech_number“ : 1,
„line_number“ : „1.1.1“,
„speaker“ : „KING HENRY IV“,
„text_entry“ : „So shaken as we are, so wan with care,“
}
}
]
}
}
root@Elasticsearch-10:~#

### Elasticsearch Loader – for batch loading data files (json, parquet, csv, tsv) into Elasticsearch

root@Elasticsearch-10:~# apt-get update -y
root@Elasticsearch-10:~# apt-get install pip
root@Elasticsearch-10:~# pip install elasticsearch-loader

### JSON files with the format of [{„a“: „1“}, {„b“: „2“}]

root@Elasticsearch-10:~# elasticsearch_loader –index licenses –type license json licenses.json
root@Elasticsearch-10:~# curl -X GET „localhost:9200/_cat/indices?v&pretty“
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases hBK8M62nQSagipE5KxDD0Q 1 0 42 42 40.1mb 40.1mb
yellow open licenses EQm3fIu7Qn2JLlqWP01HyA 1 1 9120 0 1.4mb 1.4mb
yellow open shakespeare -YUARRYPSYO170-U1-N6gg 1 1 111396 0 19mb 19mb
yellow open products 8AkZHUrdQ1yTv12KjQ5oJA 1 1 1 0 7.3kb 7.3kb
root@Elasticsearch-10:~# curl -X GET „localhost:9200/licenses/_search?pretty“ -H ‚Content-Type: application/json‘
root@Elasticsearch-10:~# curl -X GET „localhost:9200/licenses/_search?q=CITY:Hamburg“

### Install Kibana with Debian package

root@Elasticsearch-10:~# wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add –
root@Elasticsearch-10:~# apt-get install apt-transport-https
root@Elasticsearch-10:~# echo „deb https://artifacts.elastic.co/packages/7.x/apt stable main“ | tee -a /etc/apt/sources.list.d/elastic-7.x.list
root@Elasticsearch-10:~# apt-get install kibana
root@Elasticsearch-10:~# vi /etc/kibana/kibana.yml
server.port: 5601
server.host: „localhost“
elasticsearch.hosts: [„http://localhost:9200“]
root@Elasticsearch-10:~# systemctl enable kibana
root@Elasticsearch-10:~# systemctl start kibana
root@Elasticsearch-10:~# systemctl status kibana
* kibana.service – Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-08-21 11:30:20 UTC; 9s ago
Docs: https://www.elastic.co
Main PID: 7835 (node)
Tasks: 18 (limit: 17848)
Memory: 242.4M
CPU: 10.772s
CGroup: /system.slice/kibana.service
|-7835 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist –logging.dest=/var/log/kibana/kibana.log –pid.file=/run/kibana/kib>
`-7847 /usr/share/kibana/node/bin/node –preserve-symlinks-main –preserve-symlinks /usr/share/kibana/src/cli/dist –logging.dest=/var/log/kibana/kibana>
Aug 21 11:30:20 Elasticsearch-10 systemd[1]: Started Kibana.
root@Elasticsearch-10:~# apt-get install nginx -y
root@Elasticsearch-10:~# echo „kibana:`openssl passwd -apr1`“ | tee -a /etc/nginx/htpasswd.users
Password:
Verifying – Password:
kibana:$apr1$9r1.Gchv$vpG1jpSrB7yuqxpsbU4.u0
root@Elasticsearch-10:~# vi /etc/nginx/htpasswd.users
kibana:$apr1$9r1.Gchv$vpG1jpSrB7yuqxpsbU4.u0
root@Elasticsearch-10:~# vi /etc/nginx/sites-available/Elasticsearch-10.fritz.box
server {
listen 80;
server_name Elasticsearch-10.fritz.box;
auth_basic „Restricted Access“;
auth_basic_user_file /etc/nginx/htpasswd.users;
location / {
proxy_pass http://locoalhost:5601;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
root@Elasticsearch-10:~# ln -s /etc/nginx/sites-available/Elasticsearch-10.fritz.box /etc/nginx/sites-enabled/Elasticsearch-10.fritz.box
root@Elasticsearch-10:/etc/nginx/sites-available# ls -la
total 12
drwxr-xr-x 2 root root 4096 Aug 21 18:24 .
drwxr-xr-x 8 root root 4096 Aug 21 18:24 ..
-rw-r–r– 1 root root 466 Aug 21 17:54 Elasticsearch-10.fritz.box
root@Elasticsearch-10:/etc/nginx/sites-enabled# ls -la
total 8
drwxr-xr-x 2 root root 4096 Aug 21 18:11 .
drwxr-xr-x 8 root root 4096 Aug 21 18:24 ..
lrwxrwxrwx 1 root root 53 Aug 21 17:26 Elasticsearch-10.fritz.box -> /etc/nginx/sites-available/Elasticsearch-10.fritz.box

Elastic’s Guide – to data visualization in Kibana

Samstag, August 21st, 2021

Beginner’s Crash Course to Elastic Stack – full text queries Part 3

Freitag, August 20th, 2021

Beginner’s Crash Course to Elastic Stack – relevance of a search Part 2

Donnerstag, August 19th, 2021

Beginner’s Crash Course to Elastic Stack – intro to Elasticsearch and Kibana Part 1

Donnerstag, August 19th, 2021

Elastic Stack – how to send multiple types of logs in different indices with Filebeat & Logstash

Freitag, August 6th, 2021