brow3c/bro-app-devopts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sfsdf

Browse Source
This commit is contained in:
Jeonghwa Koo
2021-06-07 20:47:39 +09:00
parent 02175a3c9a
commit a0e6539667
51 changed files with 2257 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
docker/elk/.github/ISSUE_TEMPLATE/config.yml vendored Executable file
View File

@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: Discuss the Elastic Stack
url: https://discuss.elastic.co
about: Please ask questions related to the usage of Elastic products in those forums.
- name: Docker Community Forums
url: https://forums.docker.com
about: Please ask questions related to the usage of Docker products in those forums.
- name: docker-elk Gitter chat room
url: https://gitter.im/deviantony/docker-elk
about: General questions regarding this project can also be asked in the chat.

87
docker/elk/.github/ISSUE_TEMPLATE/issue_report.md vendored Executable file
View File

@@ -0,0 +1,87 @@
---
name: Issue report
about: Report a problem with the docker-elk integration or its documentation.
---
<!--
Thanks for your issue report!
In order for us to be able to reproduce the problem and identify the root cause
quickly, we kindly ask you to include *all* the information requested below in
your issue report. It saves us a lot of effort and allows us to provide you
with a solution with as little delay as possible.
Issues submitted without the requested information will be closed.
Thank you for your understanding.
-->
### Problem description
<!--
Please be as descriptive as possible regarding the encountered issue versus the
expected outcome.
-->
### Extra information
#### Stack configuration
<!--
Please mention all changes performed to the default configuration, including to Dockerfiles.
If possible, provide the output of the `git diff` command.
-->
#### Docker setup
<!--
Please paste the full output of the `docker version` command below.
Example:
Client: Docker Engine - Community
Version: 20.10.2
API version: 1.41
...
-->
```console
$ docker version
[OUTPUT HERE]
```
<!--
Please paste the full output of the `docker-compose version` command below.
Example:
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
...
-->
```console
$ docker-compose version
[OUTPUT HERE]
```
#### Container logs
<!--
Please paste the full output of the `docker-compose logs` command below.
Example:
elasticsearch_1 | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
elasticsearch_1 | {"@timestamp":"2021-01-16T21:53:38.331Z", "log.level": "INFO", "message":"version...
kibana_1 | {"type":"log","@timestamp":"2021-01-16T21:54:10+00:00","tags":["info","plugins-system"],...
...
-->
```console
$ docker-compose logs
[OUTPUT HERE]
```

293
docker/elk/.github/workflows/ci.yml vendored Executable file
View File

@@ -0,0 +1,293 @@
name: CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
test-compose:
name: 'Test suite: Compose'
# List of supported runners:
# https://docs.github.com/en/free-pro-team@latest/actions/reference/specifications-for-github-hosted-runners#supported-runners-and-hardware-resources
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
#####################################################
# #
# Install all dependencies required by test suites. #
# #
#####################################################
- name: Prepare environment
run: |
# Install Linux packages
#
# List of packages pre-installed in the runner:
# https://docs.github.com/en/free-pro-team@latest/actions/reference/specifications-for-github-hosted-runners#supported-software
sudo apt install -y expect
# Pre-build container images
docker-compose build
########################################################
# #
# Ensure §"Initial setup" of the README remains valid. #
# #
########################################################
- name: Set password of every built-in user to 'testpasswd'
run: |
# Change password of 'elastic' user from 'changeme' to 'testpasswd' in config files
sed -i -e 's/\(xpack.monitoring.elasticsearch.username:\) elastic/\1 logstash_system/g' -e 's/\(xpack.monitoring.elasticsearch.password:\) changeme/\1 testpasswd/g' logstash/config/logstash.yml
sed -i 's/\(password =>\) "changeme"/\1 "testpasswd"/g' logstash/pipeline/logstash.conf
sed -i -e 's/\(elasticsearch.username:\) elastic/\1 kibana_system/g' -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' kibana/config/kibana.yml
sed -i -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' -e 's/\(secret_management.encryption_keys:\)/\1 [test-encrypt]/g' extensions/enterprise-search/config/enterprise-search.yml
sed -i 's/\(password:\) changeme/\1 testpasswd/g' extensions/apm-server/config/apm-server.yml
sed -i 's/\(password:\) changeme/\1 testpasswd/g' extensions/metricbeat/config/metricbeat.yml
# Run Elasticsearch and wait for its availability
docker-compose up -d elasticsearch
source .github/workflows/scripts/lib/testing.sh
poll_ready "$(container_id elasticsearch)" "http://$(service_ip elasticsearch):9200/" -u 'elastic:changeme'
# Set passwords
.github/workflows/scripts/elasticsearch-setup-passwords.exp
##########################################################
# #
# Test core components: Elasticsearch, Logstash, Kibana. #
# #
##########################################################
- name: Run the stack
run: docker-compose up -d
- name: Execute core test suite
run: .github/workflows/scripts/run-tests-core.sh
- name: 'debug: Display state and logs (core)'
# https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#jobsjob_idif
# https://docs.github.com/en/free-pro-team@latest/actions/reference/context-and-expression-syntax-for-github-actions#job-status-check-functions
if: always()
run: |
docker-compose ps
docker-compose logs elasticsearch
docker-compose logs logstash
docker-compose logs kibana
##############################
# #
# Test supported extensions. #
# #
##############################
#
# Logspout
#
- name: Execute Logspout test suite
run: |
# Set mandatory Logstash settings
sed -i '$ a input { udp { port => 5000 codec => json } }' logstash/pipeline/logstash.conf
# Run Logspout and execute tests.
# Logstash will be restarted as a result of building the Logspout
# image, so changes above will automatically take effect.
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml up -d logspout
.github/workflows/scripts/run-tests-logspout.sh
# Revert changes to Logstash configuration
sed -i '/input { udp { port => 5000 codec => json } }/d' logstash/pipeline/logstash.conf
- name: 'debug: Display state and logs (Logspout)'
if: always()
run: |
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml ps
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml logs logspout
# next steps don't need Logspout
docker-compose -f docker-compose.yml -f extensions/logspout/logspout-compose.yml stop logspout
# next steps don't need Logstash
docker-compose stop logstash
#
# Enterprise Search
#
- name: Execute Enterprise Search test suite
run: |
# Set mandatory Elasticsearch settings
sed -i '$ a xpack.security.authc.api_key.enabled: true' elasticsearch/config/elasticsearch.yml
# Restart Elasticsearch for changes to take effect
docker-compose restart elasticsearch
# Run Enterprise Search and execute tests
docker-compose -f docker-compose.yml -f extensions/enterprise-search/enterprise-search-compose.yml up -d enterprise-search
.github/workflows/scripts/run-tests-enterprise-search.sh
# Revert changes to Elasticsearch configuration
sed -i '/xpack.security.authc.api_key.enabled: true/d' elasticsearch/config/elasticsearch.yml
docker-compose restart elasticsearch
- name: 'debug: Display state and logs (Enterprise Search)'
if: always()
run: |
docker-compose -f docker-compose.yml -f extensions/enterprise-search/enterprise-search-compose.yml ps
docker-compose -f docker-compose.yml -f extensions/enterprise-search/enterprise-search-compose.yml logs enterprise-search
# next steps don't need Enterprise Search
docker-compose -f docker-compose.yml -f extensions/enterprise-search/enterprise-search-compose.yml stop enterprise-search
#
# APM Server
#
- name: Execute APM Server test suite
run: |
docker-compose -f docker-compose.yml -f extensions/apm-server/apm-server-compose.yml up -d apm-server
.github/workflows/scripts/run-tests-apm-server.sh
- name: 'debug: Display state and logs (APM Server)'
if: always()
run: |
docker-compose -f docker-compose.yml -f extensions/apm-server/apm-server-compose.yml ps
docker-compose -f docker-compose.yml -f extensions/apm-server/apm-server-compose.yml logs apm-server
# next steps don't need APM Server
docker-compose -f docker-compose.yml -f extensions/apm-server/apm-server-compose.yml stop apm-server
#
# Metricbeat
#
- name: Execute Metricbeat test suite
run: |
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml up -d metricbeat
.github/workflows/scripts/run-tests-metricbeat.sh
- name: 'debug: Display state and logs (Metricbeat)'
if: always()
run: |
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml ps
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml logs metricbeat
# next steps don't need Metricbeat
docker-compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml stop metricbeat
##############
# #
# Tear down. #
# #
##############
- name: Terminate all components
if: always()
run: >-
docker-compose
-f docker-compose.yml
-f extensions/logspout/logspout-compose.yml
-f extensions/enterprise-search/enterprise-search-compose.yml
-f extensions/apm-server/apm-server-compose.yml
-f extensions/metricbeat/metricbeat-compose.yml
down -v
test-swarm:
name: 'Test suite: Swarm'
runs-on: ubuntu-latest
env:
MODE: swarm
steps:
- uses: actions/checkout@v2
#####################################################
# #
# Install all dependencies required by test suites. #
# #
#####################################################
- name: Prepare environment
run: |
# Install Linux packages
sudo apt install -y expect
# Enable Swarm mode
docker swarm init
########################################################
# #
# Ensure §"Initial setup" of the README remains valid. #
# #
########################################################
- name: Set password of every built-in user to 'testpasswd'
run: |
# Change password of 'elastic' user from 'changeme' to 'testpasswd' in config files
sed -i -e 's/\(xpack.monitoring.elasticsearch.username:\) elastic/\1 logstash_system/g' -e 's/\(xpack.monitoring.elasticsearch.password:\) changeme/\1 testpasswd/g' logstash/config/logstash.yml
sed -i 's/\(password =>\) "changeme"/\1 "testpasswd"/g' logstash/pipeline/logstash.conf
sed -i -e 's/\(elasticsearch.username:\) elastic/\1 kibana_system/g' -e 's/\(elasticsearch.password:\) changeme/\1 testpasswd/g' kibana/config/kibana.yml
# Run Elasticsearch and wait for its availability
docker stack deploy -c ./docker-stack.yml elk
docker service scale elk_logstash=0 elk_kibana=0
source .github/workflows/scripts/lib/testing.sh
poll_ready "$(container_id elasticsearch)" "http://$(service_ip elasticsearch):9200/" -u 'elastic:changeme'
# Set passwords
.github/workflows/scripts/elasticsearch-setup-passwords.exp swarm
##########################################################
# #
# Test core components: Elasticsearch, Logstash, Kibana. #
# #
##########################################################
- name: Run the stack
run: docker service scale elk_logstash=1 elk_kibana=1
- name: Execute core test suite
run: .github/workflows/scripts/run-tests-core.sh swarm
- name: 'debug: Display state and logs (core)'
if: always()
run: |
docker stack services elk
docker service logs elk_elasticsearch
docker service logs elk_kibana
docker service logs elk_logstash
##############
# #
# Tear down. #
# #
##############
- name: Terminate all components
if: always()
run: docker stack rm elk

29
docker/elk/.github/workflows/docs.yml vendored Executable file
View File

@@ -0,0 +1,29 @@
name: Documentation
on:
schedule:
- cron: '0 0 * * 0' # At 00:00 every Sunday
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
markdown-check:
name: Check Markdown
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Check links
uses: gaurav-nelson/github-action-markdown-link-check@v1
with:
config-file: .github/workflows/mlc_config.json
- name: Lint
uses: avto-dev/markdown-lint@v1
with:
args: '**/*.md'
config: .github/workflows/lint/markdown.yaml

152
docker/elk/.github/workflows/lint/markdown.yaml vendored Executable file
View File

@@ -0,0 +1,152 @@
default: false # includes/excludes all rules by default
# Heading levels should only increment by one level at a time <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md001>
MD001: true
# Heading style <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md003>
MD003: true
# Unordered list style <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md004>
MD004: true
# Inconsistent indentation for list items at the same level <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md005>
MD005: true
# Consider starting bulleted lists at the beginning of the line <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md006>
MD006: true
# Unordered list indentation <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md007>
MD007: true
# Trailing spaces <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md009>
MD009: true
# Hard tabs <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md010>
MD010: true
# Reversed link syntax <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md011>
MD011: true
# Multiple consecutive blank lines <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md012>
MD012: true
# Line length <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md013>
MD013:
line_length: 120
code_blocks: false
# Dollar signs used before commands without showing output <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md014>
MD014: false
# No space after hash on atx style heading <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md018>
MD018: true
# Multiple spaces after hash on atx style heading <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md019>
MD019: true
# No space inside hashes on closed atx style heading <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md020>
MD020: true
# Multiple spaces inside hashes on closed atx style heading <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md021>
MD021: true
# Headings should be surrounded by blank lines <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md022>
MD022: true
# Headings must start at the beginning of the line <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md023>
MD023: true
# Multiple headings with the same content <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md024>
MD024:
allow_different_nesting: true
# Multiple top level headings in the same document <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md025>
MD025: true
# Trailing punctuation in heading <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md026>
MD026: true
# Multiple spaces after blockquote symbol <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md027>
MD027: true
# Blank line inside blockquote <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md028>
MD028: false
# Ordered list item prefix <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md029>
MD029:
style: 'one'
# Spaces after list markers <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md030>
MD030: true
# Fenced code blocks should be surrounded by blank lines <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md031>
MD031: true
# Lists should be surrounded by blank lines <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md032>
MD032: true
# Inline HTML <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md033>
MD033: true
# Bare URL used <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md034>
MD034: true
# Horizontal rule style <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md035>
MD035:
style: '---'
# Emphasis used instead of a heading <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md036>
MD036: true
# Spaces inside emphasis markers <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md037>
MD037: true
# Spaces inside code span elements <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md038>
MD038: true
# Spaces inside link text <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md039>
MD039: true
# Fenced code blocks should have a language specified <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md040>
MD040: true
# First line in file should be a top level heading <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md041>
MD041: true
# No empty links <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md042>
MD042: true
# Required heading structure <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md043>
MD043: false
# Proper names should have the correct capitalization <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md044>
MD044:
names:
- docker-elk
- Elasticsearch
- Logstash
- Kibana
- Docker
- Compose
- macOS
code_blocks: false
# Images should have alternate text (alt text) <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md045>
MD045: true
# Code block style <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md046>
MD046:
style: fenced
# Files should end with a single newline character <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md047>
MD047: true
# Code fence style <https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#md048>
MD048:
style: 'backtick'
# Custom rules:
CHANGELOG-RULE-001: true
CHANGELOG-RULE-002: true
CHANGELOG-RULE-003: true
CHANGELOG-RULE-004: true

5
docker/elk/.github/workflows/mlc_config.json vendored Executable file
View File

@@ -0,0 +1,5 @@
{
"ignorePatterns": [
{ "pattern": "^http:\/\/localhost:" }
]
}

28
docker/elk/.github/workflows/scripts/elasticsearch-setup-passwords.exp vendored Executable file
View File

@@ -0,0 +1,28 @@
#!/usr/bin/expect -f
# List of expected users with dummy password
set user "(elastic|apm_system|kibana_system|logstash_system|beats_system|remote_monitoring_user)"
set password "testpasswd"
# Find elasticsearch container id
set MODE [lindex $argv 0]
if { [string match "swarm" $MODE] } {
set cid [exec docker ps -q -f label=com.docker.swarm.service.name=elk_elasticsearch]
} else {
set cid [exec docker ps -q -f label=com.docker.compose.service=elasticsearch]
}
set cmd "docker exec -it $cid bin/elasticsearch-setup-passwords interactive -s -b -u http://localhost:9200"
spawn {*}$cmd
expect {
-re "(E|Ree)nter password for \\\[$user\\\]: " {
send "$password\r"
exp_continue
}
eof
}
lassign [wait] pid spawnid os_error_flag value
exit $value

114
docker/elk/.github/workflows/scripts/lib/testing.sh vendored Executable file
View File

@@ -0,0 +1,114 @@
#!/usr/bin/env bash
# Log a message.
function log {
echo -e "\n[+] $1\n"
}
# Log an error.
function err {
echo -e "\n[x] $1\n" >&2
}
# Return the ID of the container running the given service.
function container_id {
local svc=$1
local label
if [[ "${MODE:-}" == "swarm" ]]; then
label="com.docker.swarm.service.name=elk_${svc}"
else
label="com.docker.compose.service=${svc}"
fi
local cid
# retry for max 60s (30*2s)
for _ in $(seq 1 30); do
cid="$(docker container ls -aq -f label="$label")"
if [ -n "$cid" ]; then
break
fi
echo -n '.' >&2
sleep 2
done
if [ -z "${cid:-}" ]; then
err "Timed out waiting for creation of container with label ${label}"
return 1
fi
echo "$cid"
}
# Return the IP address at which a service can be reached.
# In Compose mode, returns the container's IP.
# In Swarm mode, returns the IP of the node to ensure traffic enters the routing mesh (ingress).
function service_ip {
local svc=$1
local ip
if [[ "${MODE:-}" == "swarm" ]]; then
#ingress_net="$(docker network inspect ingress --format '{{ .Id }}')"
#ip="$(docker service inspect elk_"$svc" --format "{{ range .Endpoint.VirtualIPs }}{{ if eq .NetworkID \"${ingress_net}\" }}{{ .Addr }}{{ end }}{{ end }}" | cut -d/ -f1)"
node="$(docker node ls --format '{{ .ID }}')"
ip="$(docker node inspect "$node" --format '{{ .Status.Addr }}')"
if [ -z "${ip:-}" ]; then
err "Node ${node} has no IP address"
return 1
fi
echo "$ip"
return
fi
local cid
cid="$(container_id "$svc")"
ip="$(docker container inspect "$cid" --format '{{ (index .NetworkSettings.Networks "docker-elk_elk").IPAddress }}')"
if [ -z "${ip:-}" ]; then
err "Container ${cid} has no IP address"
return 1
fi
echo "$ip"
}
# Poll the given service at the given port:/path until it responds with HTTP code 200.
function poll_ready {
local cid=$1
local url=$2
local -a args=( '-s' '-D-' '-m3' '-w' '%{http_code}' "$url" )
if [ "$#" -ge 3 ]; then
args+=( ${@:3} )
fi
echo "curl arguments: ${args[*]}"
local -i result=1
local output
# retry for max 180s (36*5s)
for _ in $(seq 1 36); do
if [[ $(docker container inspect "$cid" --format '{{ .State.Status}}') == 'exited' ]]; then
err "Container exited ($(docker container inspect "$cid" --format '{{ .Name }}'))"
return 1
fi
output="$(curl "${args[@]}" || true)"
if [ "${output: -3}" -eq 200 ]; then
result=0
break
fi
echo -n 'x' >&2
sleep 5
done
echo -e "\n${output::-3}"
return $result
}

14
docker/elk/.github/workflows/scripts/run-tests-apm-server.sh vendored Executable file
View File

@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -eu
set -o pipefail
source "$(dirname ${BASH_SOURCE[0]})/lib/testing.sh"
cid="$(container_id apm-server)"
ip="$(service_ip apm-server)"
log 'Waiting for readiness of APM Server'
poll_ready "$cid" "http://${ip}:8200/"

59
docker/elk/.github/workflows/scripts/run-tests-core.sh vendored Executable file
View File

@@ -0,0 +1,59 @@
#!/usr/bin/env bash
set -eu
set -o pipefail
source "$(dirname ${BASH_SOURCE[0]})/lib/testing.sh"
cid_es="$(container_id elasticsearch)"
cid_ls="$(container_id logstash)"
cid_kb="$(container_id kibana)"
ip_es="$(service_ip elasticsearch)"
ip_ls="$(service_ip logstash)"
ip_kb="$(service_ip kibana)"
log 'Waiting for readiness of Elasticsearch'
poll_ready "$cid_es" "http://${ip_es}:9200/" -u 'elastic:testpasswd'
log 'Waiting for readiness of Logstash'
poll_ready "$cid_ls" "http://${ip_ls}:9600/_node/pipelines/main?pretty"
log 'Waiting for readiness of Kibana'
poll_ready "$cid_kb" "http://${ip_kb}:5601/api/status" -u 'kibana_system:testpasswd'
log 'Creating Logstash index pattern in Kibana'
source .env
curl -X POST -D- "http://${ip_kb}:5601/api/saved_objects/index-pattern" \
-s -w '\n' \
-H 'Content-Type: application/json' \
-H "kbn-version: ${ELK_VERSION}" \
-u elastic:testpasswd \
-d '{"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}'
log 'Searching index pattern via Kibana API'
response="$(curl "http://${ip_kb}:5601/api/saved_objects/_find?type=index-pattern" -s -u elastic:testpasswd)"
echo "$response"
count="$(jq -rn --argjson data "${response}" '$data.total')"
if [[ $count -ne 1 ]]; then
echo "Expected 1 index pattern, got ${count}"
exit 1
fi
log 'Sending message to Logstash TCP input'
echo 'dockerelk' | nc -q0 "$ip_ls" 5000
sleep 1
curl -X POST "http://${ip_es}:9200/_refresh" -u elastic:testpasswd \
-s -w '\n'
log 'Searching message in Elasticsearch'
response="$(curl "http://${ip_es}:9200/_count?q=message:dockerelk&pretty" -s -u elastic:testpasswd)"
echo "$response"
count="$(jq -rn --argjson data "${response}" '$data.count')"
if [[ $count -ne 1 ]]; then
echo "Expected 1 document, got ${count}"
exit 1
fi

30
docker/elk/.github/workflows/scripts/run-tests-enterprise-search.sh vendored Executable file
View File

@@ -0,0 +1,30 @@
#!/usr/bin/env bash
set -eu
set -o pipefail
source "$(dirname ${BASH_SOURCE[0]})/lib/testing.sh"
cid_es="$(container_id elasticsearch)"
cid_en="$(container_id enterprise-search)"
ip_es="$(service_ip elasticsearch)"
ip_en="$(service_ip enterprise-search)"
log 'Waiting for readiness of Elasticsearch'
poll_ready "$cid_es" "http://${ip_es}:9200/" -u 'elastic:testpasswd'
log 'Waiting for readiness of Enterprise Search'
poll_ready "$cid_en" "http://${ip_en}:3002/api/ent/v1/internal/health" -u 'elastic:testpasswd'
log 'Ensuring that App Search API keys were created in Elasticsearch'
response="$(curl "http://${ip_es}:9200/.ent-search-actastic-app_search_api_tokens_v3/_count?pretty" -s -u elastic:testpasswd)"
echo "$response"
declare -i count
count="$(jq -rn --argjson data "${response}" '$data.count')"
if (( count != 2)); then
echo "Expected search and private keys, got ${count} result(s)"
exit 1
fi

52
docker/elk/.github/workflows/scripts/run-tests-logspout.sh vendored Executable file
View File

@@ -0,0 +1,52 @@
#!/usr/bin/env bash
set -eu
set -o pipefail
source "$(dirname ${BASH_SOURCE[0]})/lib/testing.sh"
cid_es="$(container_id elasticsearch)"
cid_ls="$(container_id logspout)"
ip_es="$(service_ip elasticsearch)"
ip_ls="$(service_ip logspout)"
log 'Waiting for readiness of Elasticsearch'
poll_ready "$cid_es" "http://${ip_es}:9200/" -u 'elastic:testpasswd'
log 'Waiting for readiness of Logspout'
poll_ready "$cid_ls" "http://${ip_ls}/health"
# When Logspout starts, it prints the following log line:
# 2021/01/07 16:14:52 # logspout v3.2.13-custom by gliderlabs
#
# which we expect to find by querying:
# docker.image:"docker-elk_logspout" AND message:"logspout gliderlabs"~3
#
log 'Searching a log entry forwarded by Logspout'
declare response
declare -i count
# retry for max 60s (30*2s)
for _ in $(seq 1 30); do
response="$(curl "http://${ip_es}:9200/_count?q=docker.image:%22docker-elk_logspout%22%20AND%20message:%22logspout%20gliderlabs%22~3&pretty" -s -u elastic:testpasswd)"
count="$(jq -rn --argjson data "${response}" '$data.count')"
if [[ $count -gt 0 ]]; then
break
fi
echo -n 'x' >&2
sleep 2
done
echo -e '\n' >&2
echo "$response"
# Logspout may restart if Logstash isn't ready yet, so we tolerate multiple
# results
if [[ $count -lt 1 ]]; then
echo "Expected at least 1 document, got ${count}"
exit 1
fi

56
docker/elk/.github/workflows/scripts/run-tests-metricbeat.sh vendored Executable file
View File

@@ -0,0 +1,56 @@
#!/usr/bin/env bash
set -eu
set -o pipefail
source "$(dirname ${BASH_SOURCE[0]})/lib/testing.sh"
cid_es="$(container_id elasticsearch)"
cid_mb="$(container_id metricbeat)"
ip_es="$(service_ip elasticsearch)"
ip_mb="$(service_ip metricbeat)"
log 'Waiting for readiness of Elasticsearch'
poll_ready "$cid_es" "http://${ip_es}:9200/" -u 'elastic:testpasswd'
log 'Waiting for readiness of Metricbeat'
poll_ready "$cid_mb" "http://${ip_mb}:5066/?pretty"
# We expect to find one monitoring entry for the 'elasticsearch' Compose
# service using the following query:
#
# agent.type:"metricbeat"
# AND event.module:"docker"
# AND event.dataset:"docker.container"
# AND container.name:"docker-elk_elasticsearch_1"
#
log 'Searching a document generated by Metricbeat'
declare response
declare -i count
# retry for max 60s (30*2s)
for _ in $(seq 1 30); do
response="$(curl "http://${ip_es}:9200/metricbeat-*/_search?q=agent.type:%22metricbeat%22%20AND%20event.module:%22docker%22%20AND%20event.dataset:%22docker.container%22%20AND%20container.name:%22docker-elk_elasticsearch_1%22&pretty" -s -u elastic:testpasswd)"
set +u # prevent "unbound variable" if assigned value is not an integer
count="$(jq -rn --argjson data "${response}" '$data.hits.total.value')"
set -u
if (( count > 0 )); then
break
fi
echo -n 'x' >&2
sleep 2
done
echo -e '\n' >&2
echo "$response"
if (( count != 1 )); then
echo "Expected 1 document, got ${count}"
exit 1
fi

45
docker/elk/.github/workflows/update.yml vendored Executable file
View File

@@ -0,0 +1,45 @@
name: Update Elastic release
on:
schedule:
- cron: '0 0 * * 0' # At 00:00 every Sunday
jobs:
check-and-update:
name: Check and update Elastic release
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Get latest release version
uses: octokit/request-action@v2.x
id: get_latest_release
with:
route: GET /repos/:repository/releases/latest
repository: elastic/elasticsearch
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Update stack version
run: |
source .env
cur_ver="$ELK_VERSION"
new_ver=${{ fromJson(steps.get_latest_release.outputs.data).tag_name }}
# Escape dot characters so sed interprets them as literal dots
cur_ver="$(echo $cur_ver | sed 's/\./\\./g')"
# Trim leading 'v' in semantic version
new_ver="${new_ver:1}"
for f in .env docker-stack.yml README.md; do
sed -i "s/${cur_ver}/${new_ver}/g" "$f"
done
- name: Send pull request to update to new version
uses: peter-evans/create-pull-request@v3
with:
commit-message: Update to ${{ fromJson(steps.get_latest_release.outputs.data).tag_name }}
title: Update to ${{ fromJson(steps.get_latest_release.outputs.data).tag_name }}
delete-branch: true