Compare commits

..

45 Commits

Author SHA1 Message Date
Michel Hollands
232777d71a Remove unused ruler files
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-08 10:47:52 +01:00
Michel Hollands
d9a4d4a964 Merge pull request #106 from grafana/split_up_grafana_template
Split up Grafana yaml
2024-05-08 10:39:29 +01:00
Michel Hollands
57adbf43e2 Split up Grafana yaml
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-08 10:39:02 +01:00
Michel Hollands
add43ae974 Merge pull request #105 from grafana/remove_redundant_variables
Remove unused variables
2024-05-08 10:25:37 +01:00
Michel Hollands
52ec526718 Remove unused variables
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-08 10:24:48 +01:00
Michel Hollands
8a5ed559a2 Merge pull request #104 from grafana/fix_dependency_check
Fix name and indentation of workflow
2024-05-08 09:49:07 +01:00
Michel Hollands
188cd7e56f Fix name and indentation of workflow
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-08 09:46:42 +01:00
Michel Hollands
9e4dbcd44a Merge pull request #100 from grafana/combine_ci
Combine dependency updates
2024-05-08 09:40:07 +01:00
Michel Hollands
28daa27fca Merge pull request #99 from grafana/chore/update-minio
[dependency] Update the Grafana version
2024-05-08 09:38:26 +01:00
Michel Hollands
2de595baf4 Merge branch 'main' into chore/update-minio 2024-05-08 09:37:45 +01:00
Michel Hollands
95257b66d3 Merge pull request #103 from grafana/chore/update-tempo-distributed
[dependency] Update the Tempo Distributed subchart
2024-05-08 09:36:02 +01:00
Michel Hollands
e9b0e57ef0 Merge pull request #95 from grafana/update_grafana
Add CI action to update Grafana version
2024-05-08 09:35:29 +01:00
Michel Hollands
03609ebb35 Merge pull request #102 from grafana/fix_alloy_config_for_traces
Fix the alloy config
2024-05-08 09:34:53 +01:00
MichelHollands
7e38d19814 Update Tempo Distributed 2024-05-08 07:03:26 +00:00
Michel Hollands
32272298d7 Fix the alloy config
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 16:35:00 +01:00
Michel Hollands
3879207e05 Merge pull request #101 from grafana/fix_minio_secret_name
Fix secret name
2024-05-07 14:40:52 +01:00
Michel Hollands
cd42da2197 Fix secret name
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 14:39:20 +01:00
Michel Hollands
56cab04af8 Merge pull request #92 from grafana/use_secret_for_minio
Use a secret for the Minio access
2024-05-07 12:37:07 +01:00
Michel Hollands
c6d0444dfa Combine dependency updates
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 11:26:32 +01:00
Michel Hollands
b99140d3f4 Merge pull request #97 from grafana/chore/update-tempo-distributed
[dependency] Update the Tempo Distributed subchart
2024-05-07 11:00:53 +01:00
MichelHollands
749e271455 Update Tempo Distributed 2024-05-07 09:22:20 +00:00
MichelHollands
d938dbbfe5 Update Grafana version 2024-05-07 09:22:19 +00:00
Michel Hollands
e9125d1a9c Add corrected key
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 10:21:42 +01:00
Michel Hollands
076685ef06 Revert key
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 10:18:55 +01:00
Michel Hollands
b0451d626e Use $. in yaml key
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 10:16:10 +01:00
Michel Hollands
90e949e89a Change version param
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 10:14:50 +01:00
Michel Hollands
06e176e720 Trim the v prefix from the released version
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 10:11:17 +01:00
Michel Hollands
d4c886ba9d Use token from env
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 10:00:55 +01:00
Michel Hollands
643e73f5f1 add token
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 09:54:50 +01:00
Michel Hollands
7e65f3d9c9 Fix sourceid
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 09:46:31 +01:00
Michel Hollands
de91b4dac7 Merge pull request #96 from grafana/chore/update-tempo-distributed
[dependency] Update the Tempo Distributed subchart
2024-05-07 09:22:43 +01:00
MichelHollands
9f6e52d7a1 Update Tempo Distributed 2024-05-07 08:22:14 +00:00
Michel Hollands
26e0ad0b85 Add CI action to update Grafana version
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-07 09:20:51 +01:00
Michel Hollands
025bb5b0c3 Merge pull request #94 from grafana/chore/update-loki
[dependency] Update the Loki subchart
2024-05-07 08:51:44 +01:00
MichelHollands
0b31eae425 Update loki 2024-05-07 07:02:51 +00:00
Michel Hollands
ab42a96949 Update installation instructions
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-06 16:29:33 +01:00
Michel Hollands
386ff25fca Use the secret in the ruler for the dashboards
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-06 16:18:44 +01:00
Michel Hollands
c6889131a7 Use structuredConfig correctly
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-06 16:12:48 +01:00
Michel Hollands
2739bae0c0 Use correct variables
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-03 15:40:36 +01:00
Michel Hollands
cea8076b75 Start using a secret
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-03 15:38:07 +01:00
Michel Hollands
29b831ca00 Merge pull request #91 from grafana/set_step_time_to_1_minute_in_dashboards
Set the interval to 1m to match the scrape interval
2024-05-03 09:58:40 +01:00
Michel Hollands
09cf8f812c Merge pull request #90 from grafana/chore/update-tempo-distributed
[dependency] Update the Tempo Distributed subchart
2024-05-03 09:58:25 +01:00
Michel Hollands
f8436a8e44 Set the interval to 1m to match the scrape interval
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
2024-05-03 09:28:26 +01:00
MichelHollands
2b26abedbb Update Tempo Distributed 2024-05-03 07:02:32 +00:00
Michel Hollands
017c041007 Merge pull request #89 from grafana/add_extra_metrics
Add way to gather extra metrics and logs
2024-05-02 17:15:18 +01:00
30 changed files with 398 additions and 799 deletions

View File

@@ -0,0 +1,30 @@
name: Bump grafana version specified in the values.yaml
sources:
latestGrafanaRelease:
name: Get latest grafana release on Github
kind: githubrelease
spec:
owner: grafana
repository: grafana
token: '{{ requiredEnv "UPDATECLI_GITHUB_TOKEN" }}'
versionfilter:
kind: latest
transformers:
- trimprefix: "v"
conditions:
grafanaImagePublished:
name: Ensure the latest Grafana is published on DockerHub
kind: dockerimage
source-id: latestGrafanaRelease
spec:
image: "grafana/grafana"
targets:
grafana:
name: Update Grafana version in values.yaml
kind: helmchart
spec:
file: values.yaml
key: $.grafana.version
name: charts/meta-monitoring
versionincrement: none
sourceid: latestGrafanaRelease

View File

@@ -16,8 +16,8 @@ env:
UPDATECLI_GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
jobs:
updateLoki:
name: Update the Loki subchart
updateVersions:
name: Update the subcharts
runs-on: "ubuntu-latest"
steps:
- name: Checkout
@@ -26,7 +26,7 @@ jobs:
- name: Install Updatecli
uses: updatecli/updatecli-action@v2
- name: Run Updatecli
- name: Run Updatecli for Loki
id: update-loki
run: |
updatecli apply --config ${UPDATECLI_CONFIG_DIR}/loki.yaml
@@ -34,31 +34,7 @@ jobs:
echo "changed=true" >> "${GITHUB_OUTPUT}"
fi
- name: Create pull request
if: steps.update-loki.outputs.changed == 'true'
uses: peter-evans/create-pull-request@v5
with:
title: "[dependency] Update the Loki subchart"
body: "Updates the Loki subchart"
base: main
author: "${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>"
committer: "GitHub <noreply@github.com>"
commit-message: Update loki
labels: dependencies
branch: chore/update-loki
delete-branch: true
updateGrafanaAlloy:
name: Update the Grafana Alloy subchart
runs-on: "ubuntu-latest"
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Updatecli
uses: updatecli/updatecli-action@v2
- name: Run Updatecli
- name: Run Updatecli for Alloy
id: update-grafana-alloy
run: |
updatecli apply --config ${UPDATECLI_CONFIG_DIR}/alloy.yaml
@@ -66,31 +42,7 @@ jobs:
echo "changed=true" >> "${GITHUB_OUTPUT}"
fi
- name: Create pull request
if: steps.update-grafana-alloy.outputs.changed == 'true'
uses: peter-evans/create-pull-request@v5
with:
title: "[dependency] Update the Grafana Alloy subchart"
body: "Updates the Grafana Alloy subchart"
base: main
author: "${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>"
committer: "GitHub <noreply@github.com>"
commit-message: Update Grafana Alloy
labels: dependencies
branch: chore/update-grafana-alloy
delete-branch: true
updateMimirDistributed:
name: Update the Mimir Distributed subchart
runs-on: "ubuntu-latest"
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Updatecli
uses: updatecli/updatecli-action@v2
- name: Run Updatecli
- name: Run Updatecli for Mimir
id: update-mimir-distributed
run: |
updatecli apply --config ${UPDATECLI_CONFIG_DIR}/mimir-distributed.yaml
@@ -98,31 +50,7 @@ jobs:
echo "changed=true" >> "${GITHUB_OUTPUT}"
fi
- name: Create pull request
if: steps.update-mimir-distributed.outputs.changed == 'true'
uses: peter-evans/create-pull-request@v5
with:
title: "[dependency] Update the Mimir Distributed subchart"
body: "Updates the Mimir Distributed subchart"
base: main
author: "${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>"
committer: "GitHub <noreply@github.com>"
commit-message: Update Mimir Distributed
labels: dependencies
branch: chore/update-mimir-distributed
delete-branch: true
updateTempoDistributed:
name: Update the Tempo Distributed subchart
runs-on: "ubuntu-latest"
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Updatecli
uses: updatecli/updatecli-action@v2
- name: Run Updatecli
- name: Run Updatecli for Tempo
id: update-tempo-distributed
run: |
updatecli apply --config ${UPDATECLI_CONFIG_DIR}/tempo-distributed.yaml
@@ -130,31 +58,7 @@ jobs:
echo "changed=true" >> "${GITHUB_OUTPUT}"
fi
- name: Create pull request
if: steps.update-tempo-distributed.outputs.changed == 'true'
uses: peter-evans/create-pull-request@v5
with:
title: "[dependency] Update the Tempo Distributed subchart"
body: "Updates the tempo Distributed subchart"
base: main
author: "${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>"
committer: "GitHub <noreply@github.com>"
commit-message: Update Tempo Distributed
labels: dependencies
branch: chore/update-tempo-distributed
delete-branch: true
updateMinio:
name: Update the Minio subchart
runs-on: "ubuntu-latest"
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Updatecli
uses: updatecli/updatecli-action@v2
- name: Run Updatecli
- name: Run Updatecli for Minio
id: update-minio
run: |
updatecli apply --config ${UPDATECLI_CONFIG_DIR}/minio.yaml
@@ -163,15 +67,47 @@ jobs:
fi
- name: Create pull request
if: steps.update-minio.outputs.changed == 'true'
if: steps.update-loki.outputs.changed == 'true' || steps.update-grafana-alloy.outputs.changed == 'true' || steps.update-mimir-distributed.outputs.changed == 'true' || steps.update-tempo-distributed.outputs.changed == 'true' || steps.update-minio.outputs.changed == 'true'
uses: peter-evans/create-pull-request@v5
with:
title: "[dependency] Update the Minio subchart"
body: "Updates the Minio subchart"
title: "[dependency] Update the subcharts"
body: "Updates the subcharts"
base: main
author: "${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>"
committer: "GitHub <noreply@github.com>"
commit-message: Update minio
commit-message: Update dependencies
labels: dependencies
branch: chore/update-dependencies
delete-branch: true
updateGrafana:
name: Update the Grafana version
runs-on: "ubuntu-latest"
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install Updatecli
uses: updatecli/updatecli-action@v2
- name: Run Updatecli
id: update-grafana
run: |
updatecli apply --config ${UPDATECLI_CONFIG_DIR}/grafana.yaml
if ! git diff --exit-code > /dev/null; then
echo "changed=true" >> "${GITHUB_OUTPUT}"
fi
- name: Create pull request
if: steps.update-grafana.outputs.changed == 'true'
uses: peter-evans/create-pull-request@v5
with:
title: "[dependency] Update the Grafana version"
body: "Updates the Grafana version"
base: main
author: "${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>"
committer: "GitHub <noreply@github.com>"
commit-message: Update Grafana version
labels: dependencies
branch: chore/update-minio
delete-branch: true

View File

@@ -1,7 +1,7 @@
dependencies:
- name: loki
repository: https://grafana.github.io/helm-charts
version: 6.4.2
version: 6.5.0
- name: alloy
repository: https://grafana.github.io/helm-charts
version: 0.1.1
@@ -10,9 +10,9 @@ dependencies:
version: 5.3.0
- name: tempo-distributed
repository: https://grafana.github.io/helm-charts
version: 1.9.4
version: 1.9.9
- name: minio
repository: https://charts.min.io
version: 5.2.0
digest: sha256:62e737c64e180822d737a94514acee68d3c58c2cfb3e741be2f707a5d3588004
generated: "2024-05-02T07:02:31.487968368Z"
digest: sha256:5328702b5f6b0487aba8f7bc77d6abfcd5e094569e9205cd725971e3e31255dd
generated: "2024-05-08T07:03:21.797461955Z"

View File

@@ -22,7 +22,7 @@ appVersion: "0.0.1"
dependencies:
- name: loki
repository: https://grafana.github.io/helm-charts
version: 6.4.2
version: 6.5.0
condition: local.logs.enabled
- name: alloy
repository: https://grafana.github.io/helm-charts
@@ -33,7 +33,7 @@ dependencies:
condition: local.metrics.enabled
- name: tempo-distributed
repository: https://grafana.github.io/helm-charts
version: 1.9.4
version: 1.9.9
condition: local.traces.enabled
- name: minio
repository: https://charts.min.io

Binary file not shown.

View File

@@ -51,6 +51,7 @@
"overrides": [ ]
},
"id": 1,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -98,6 +99,7 @@
"overrides": [ ]
},
"id": 2,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -157,6 +159,7 @@
"overrides": [ ]
},
"id": 3,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -235,6 +238,7 @@
"overrides": [ ]
},
"id": 4,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -325,6 +329,7 @@
"overrides": [ ]
},
"id": 5,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -403,6 +408,7 @@
"overrides": [ ]
},
"id": 6,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -462,6 +468,7 @@
"overrides": [ ]
},
"id": 7,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -658,6 +665,7 @@
},
"fill": 10,
"id": 8,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -719,6 +727,7 @@
"overrides": [ ]
},
"id": 9,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -766,6 +775,7 @@
"overrides": [ ]
},
"id": 10,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -837,6 +847,7 @@
"hideZeroBuckets": false,
"highlightCards": true,
"id": 11,
"interval": "1m",
"legend": {
"show": true
},
@@ -899,6 +910,7 @@
"hideZeroBuckets": false,
"highlightCards": true,
"id": 12,
"interval": "1m",
"legend": {
"show": true
},
@@ -968,6 +980,7 @@
"overrides": [ ]
},
"id": 13,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1039,6 +1052,7 @@
"overrides": [ ]
},
"id": 14,
"interval": "1m",
"links": [ ],
"options": {
"legend": {

View File

@@ -35,6 +35,7 @@
"fill": 1,
"format": "none",
"id": 1,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -110,6 +111,7 @@
"fill": 1,
"format": "dtdurations",
"id": 2,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -213,6 +215,7 @@
"overrides": [ ]
},
"id": 3,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -260,6 +263,7 @@
"overrides": [ ]
},
"id": 4,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -307,6 +311,7 @@
"overrides": [ ]
},
"id": 5,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -366,6 +371,7 @@
"overrides": [ ]
},
"id": 6,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -413,6 +419,7 @@
"overrides": [ ]
},
"id": 7,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -460,6 +467,7 @@
"overrides": [ ]
},
"id": 8,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -519,6 +527,7 @@
"overrides": [ ]
},
"id": 9,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -566,6 +575,7 @@
"overrides": [ ]
},
"id": 10,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -603,6 +613,7 @@
{
"datasource": "$loki_datasource",
"id": 11,
"interval": "1m",
"span": 6,
"targets": [
{
@@ -616,6 +627,7 @@
{
"datasource": "$loki_datasource",
"id": 12,
"interval": "1m",
"span": 6,
"targets": [
{

View File

@@ -38,6 +38,7 @@
},
"hiddenSeries": false,
"id": 35,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -129,6 +130,7 @@
},
"hiddenSeries": false,
"id": 41,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -216,6 +218,7 @@
},
"hiddenSeries": false,
"id": 36,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -307,6 +310,7 @@
},
"hiddenSeries": false,
"id": 40,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -398,6 +402,7 @@
},
"hiddenSeries": false,
"id": 38,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -489,6 +494,7 @@
},
"hiddenSeries": false,
"id": 39,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -575,6 +581,7 @@
},
"hiddenSeries": false,
"id": 37,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -667,6 +674,7 @@
},
"hiddenSeries": false,
"id": 42,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -759,6 +767,7 @@
},
"hiddenSeries": false,
"id": 31,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -857,6 +866,7 @@
"y": 6
},
"id": 29,
"interval": "1m",
"maxDataPoints": "",
"options": {
"showLabels": false,

View File

@@ -32,6 +32,7 @@
"y": 0
},
"id": 17,
"interval": "1m",
"panels": [ ],
"targets": [ ],
"title": "Main",
@@ -61,6 +62,7 @@
},
"hiddenSeries": false,
"id": 6,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -157,6 +159,7 @@
},
"hiddenSeries": false,
"id": 7,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -251,7 +254,7 @@
},
"hiddenSeries": false,
"id": 2,
"interval": "",
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -347,6 +350,7 @@
},
"hiddenSeries": false,
"id": 4,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -443,6 +447,7 @@
},
"hiddenSeries": false,
"id": 24,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -540,6 +545,7 @@
},
"hiddenSeries": false,
"id": 9,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -647,6 +653,7 @@
},
"hiddenSeries": false,
"id": 12,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -753,6 +760,7 @@
},
"hiddenSeries": false,
"id": 71,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -852,6 +860,7 @@
},
"hiddenSeries": false,
"id": 13,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -960,6 +969,7 @@
},
"hiddenSeries": false,
"id": 72,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -1059,6 +1069,7 @@
},
"hiddenSeries": false,
"id": 10,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -1168,6 +1179,7 @@
},
"hiddenSeries": false,
"id": 14,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -1274,6 +1286,7 @@
},
"hiddenSeries": false,
"id": 73,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -1374,6 +1387,7 @@
},
"hiddenSeries": false,
"id": 15,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -1480,6 +1494,7 @@
},
"hiddenSeries": false,
"id": 74,
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -1583,6 +1598,7 @@
},
"hiddenSeries": false,
"id": 112,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -1667,6 +1683,7 @@
"y": 27
},
"id": 113,
"interval": "1m",
"pageSize": null,
"panels": [ ],
"showHeader": true,
@@ -1780,6 +1797,7 @@
},
"hiddenSeries": false,
"id": 26,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -1874,6 +1892,7 @@
},
"hiddenSeries": false,
"id": 27,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -1966,6 +1985,7 @@
},
"hiddenSeries": false,
"id": 31,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2051,6 +2071,7 @@
"y": 32
},
"id": 29,
"interval": "1m",
"options": {
"showLabels": false,
"showTime": false,
@@ -2085,6 +2106,7 @@
},
"hiddenSeries": false,
"id": 33,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2180,6 +2202,7 @@
},
"hiddenSeries": false,
"id": 32,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2274,6 +2297,7 @@
},
"hiddenSeries": false,
"id": 34,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2368,6 +2392,7 @@
},
"hiddenSeries": false,
"id": 35,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2473,6 +2498,7 @@
},
"hiddenSeries": false,
"id": 36,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2567,6 +2593,7 @@
},
"hiddenSeries": false,
"id": 37,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2659,6 +2686,7 @@
},
"hiddenSeries": false,
"id": 38,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2744,6 +2772,7 @@
"y": 32
},
"id": 39,
"interval": "1m",
"options": {
"showLabels": false,
"showTime": false,
@@ -2778,6 +2807,7 @@
},
"hiddenSeries": false,
"id": 67,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2884,6 +2914,7 @@
},
"hiddenSeries": false,
"id": 106,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -2975,6 +3006,7 @@
},
"hiddenSeries": false,
"id": 108,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -3082,6 +3114,7 @@
},
"hiddenSeries": false,
"id": 102,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -3190,6 +3223,7 @@
"hideZeroBuckets": false,
"highlightCards": true,
"id": 100,
"interval": "1m",
"legend": {
"show": true
},
@@ -3247,6 +3281,7 @@
},
"hiddenSeries": false,
"id": 96,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -3345,6 +3380,7 @@
"hideZeroBuckets": true,
"highlightCards": true,
"id": 98,
"interval": "1m",
"legend": {
"show": true
},
@@ -3418,6 +3454,7 @@
},
"hiddenSeries": false,
"id": 68,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -3512,6 +3549,7 @@
},
"hiddenSeries": false,
"id": 69,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -3604,6 +3642,7 @@
},
"hiddenSeries": false,
"id": 65,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -3723,6 +3762,7 @@
},
"hiddenSeries": false,
"id": 70,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -3834,7 +3874,7 @@
},
"hiddenSeries": false,
"id": 53,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -3938,7 +3978,7 @@
},
"hiddenSeries": false,
"id": 54,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4051,7 +4091,7 @@
},
"hiddenSeries": false,
"id": 55,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4160,7 +4200,7 @@
},
"hiddenSeries": false,
"id": 58,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4273,7 +4313,7 @@
},
"hiddenSeries": false,
"id": 61,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4377,7 +4417,7 @@
},
"hiddenSeries": false,
"id": 62,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4489,7 +4529,7 @@
"y": 10
},
"id": 79,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4592,7 +4632,7 @@
"y": 18
},
"id": 80,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4704,7 +4744,7 @@
"y": 10
},
"id": 79,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4807,7 +4847,7 @@
"y": 18
},
"id": 80,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -4919,7 +4959,7 @@
"y": 10
},
"id": 115,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,
@@ -5022,7 +5062,7 @@
"y": 18
},
"id": 116,
"interval": "",
"interval": "1m",
"legend": {
"alignAsTable": true,
"avg": false,

View File

@@ -90,6 +90,7 @@
]
},
"id": 1,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -191,6 +192,7 @@
]
},
"id": 2,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -253,6 +255,7 @@
"overrides": [ ]
},
"id": 3,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -354,6 +357,7 @@
]
},
"id": 4,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -455,6 +459,7 @@
]
},
"id": 5,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -517,6 +522,7 @@
"overrides": [ ]
},
"id": 6,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -619,6 +625,7 @@
},
"gridPos": { },
"id": 7,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -720,6 +727,7 @@
},
"gridPos": { },
"id": 8,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -782,6 +790,7 @@
},
"gridPos": { },
"id": 9,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -832,6 +841,7 @@
},
"gridPos": { },
"id": 10,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -879,6 +889,7 @@
},
"gridPos": { },
"id": 11,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -926,6 +937,7 @@
},
"gridPos": { },
"id": 12,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1025,6 +1037,7 @@
},
"gridPos": { },
"id": 13,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1126,6 +1139,7 @@
},
"gridPos": { },
"id": 14,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1188,6 +1202,7 @@
},
"gridPos": { },
"id": 15,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1238,6 +1253,7 @@
},
"gridPos": { },
"id": 16,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1285,6 +1301,7 @@
},
"gridPos": { },
"id": 17,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1332,6 +1349,7 @@
},
"gridPos": { },
"id": 18,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1431,6 +1449,7 @@
},
"gridPos": { },
"id": 19,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1532,6 +1551,7 @@
},
"gridPos": { },
"id": 20,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1594,6 +1614,7 @@
},
"gridPos": { },
"id": 21,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1644,6 +1665,7 @@
},
"gridPos": { },
"id": 22,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1691,6 +1713,7 @@
},
"gridPos": { },
"id": 23,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1738,6 +1761,7 @@
},
"gridPos": { },
"id": 24,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1836,6 +1860,7 @@
]
},
"id": 25,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1937,6 +1962,7 @@
]
},
"id": 26,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1999,6 +2025,7 @@
"overrides": [ ]
},
"id": 27,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -2062,6 +2089,7 @@
},
"gridPos": { },
"id": 28,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -2148,6 +2176,7 @@
},
"gridPos": { },
"id": 29,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -2249,6 +2278,7 @@
},
"gridPos": { },
"id": 30,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -2311,6 +2341,7 @@
},
"gridPos": { },
"id": 31,
"interval": "1m",
"links": [ ],
"options": {
"legend": {

View File

@@ -200,6 +200,7 @@
},
"fill": 10,
"id": 1,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -249,6 +250,7 @@
"overrides": [ ]
},
"id": 2,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -333,6 +335,7 @@
"overrides": [ ]
},
"id": 3,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -545,6 +548,7 @@
},
"fill": 10,
"id": 4,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -594,6 +598,7 @@
"overrides": [ ]
},
"id": 5,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -678,6 +683,7 @@
"overrides": [ ]
},
"id": 6,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -890,6 +896,7 @@
},
"fill": 10,
"id": 7,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -939,6 +946,7 @@
"overrides": [ ]
},
"id": 8,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1023,6 +1031,7 @@
"overrides": [ ]
},
"id": 9,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1235,6 +1244,7 @@
},
"fill": 10,
"id": 10,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -1284,6 +1294,7 @@
"overrides": [ ]
},
"id": 11,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1368,6 +1379,7 @@
"overrides": [ ]
},
"id": 12,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1580,6 +1592,7 @@
},
"fill": 10,
"id": 13,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -1629,6 +1642,7 @@
"overrides": [ ]
},
"id": 14,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1713,6 +1727,7 @@
"overrides": [ ]
},
"id": 15,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1925,6 +1940,7 @@
},
"fill": 10,
"id": 16,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -1974,6 +1990,7 @@
"overrides": [ ]
},
"id": 17,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -2058,6 +2075,7 @@
"overrides": [ ]
},
"id": 18,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -2270,6 +2288,7 @@
},
"fill": 10,
"id": 19,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -2319,6 +2338,7 @@
"overrides": [ ]
},
"id": 20,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -2397,6 +2417,7 @@
"overrides": [ ]
},
"id": 21,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -2609,6 +2630,7 @@
},
"fill": 10,
"id": 22,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -2658,6 +2680,7 @@
"overrides": [ ]
},
"id": 23,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -2736,6 +2759,7 @@
"overrides": [ ]
},
"id": 24,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {

View File

@@ -90,6 +90,7 @@
]
},
"id": 1,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -191,6 +192,7 @@
]
},
"id": 2,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -253,6 +255,7 @@
"overrides": [ ]
},
"id": 3,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -317,6 +320,7 @@
},
"fill": 1,
"id": 4,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -423,6 +427,7 @@
"overrides": [ ]
},
"id": 5,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -482,6 +487,7 @@
"overrides": [ ]
},
"id": 6,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -529,6 +535,7 @@
"overrides": [ ]
},
"id": 7,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -590,6 +597,7 @@
},
"fill": 1,
"id": 8,
"interval": "1m",
"legend": {
"avg": false,
"current": false,
@@ -696,6 +704,7 @@
"overrides": [ ]
},
"id": 9,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -743,6 +752,7 @@
"overrides": [ ]
},
"id": 10,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -802,6 +812,7 @@
"overrides": [ ]
},
"id": 11,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -849,6 +860,7 @@
"overrides": [ ]
},
"id": 12,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -896,6 +908,7 @@
"overrides": [ ]
},
"id": 13,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -956,6 +969,7 @@
},
"format": "short",
"id": 14,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1004,6 +1018,7 @@
"overrides": [ ]
},
"id": 15,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1095,6 +1110,7 @@
},
"format": "short",
"id": 16,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1143,6 +1159,7 @@
"overrides": [ ]
},
"id": 17,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1233,6 +1250,7 @@
"overrides": [ ]
},
"id": 18,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1280,6 +1298,7 @@
"overrides": [ ]
},
"id": 19,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -1327,6 +1346,7 @@
"overrides": [ ]
},
"id": 20,
"interval": "1m",
"links": [ ],
"options": {
"legend": {

View File

@@ -90,6 +90,7 @@
]
},
"id": 1,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -191,6 +192,7 @@
]
},
"id": 2,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -253,6 +255,7 @@
"overrides": [ ]
},
"id": 3,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -316,6 +319,7 @@
},
"gridPos": { },
"id": 4,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -405,6 +409,7 @@
},
"gridPos": { },
"id": 5,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -506,6 +511,7 @@
},
"gridPos": { },
"id": 6,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -568,6 +574,7 @@
},
"gridPos": { },
"id": 7,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -618,6 +625,7 @@
},
"gridPos": { },
"id": 8,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -665,6 +673,7 @@
},
"gridPos": { },
"id": 9,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -712,6 +721,7 @@
},
"gridPos": { },
"id": 10,
"interval": "1m",
"links": [ ],
"options": {
"legend": {

View File

@@ -200,6 +200,7 @@
},
"fill": 10,
"id": 1,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -249,6 +250,7 @@
"overrides": [ ]
},
"id": 2,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -345,6 +347,7 @@
"overrides": [ ]
},
"id": 3,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -392,6 +395,7 @@
"overrides": [ ]
},
"id": 4,
"interval": "1m",
"links": [ ],
"options": {
"legend": {
@@ -619,6 +623,7 @@
},
"fill": 10,
"id": 5,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -668,6 +673,7 @@
"overrides": [ ]
},
"id": 6,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -913,6 +919,7 @@
},
"fill": 10,
"id": 7,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -962,6 +969,7 @@
"overrides": [ ]
},
"id": 8,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1207,6 +1215,7 @@
},
"fill": 10,
"id": 9,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -1256,6 +1265,7 @@
"overrides": [ ]
},
"id": 10,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {
@@ -1495,6 +1505,7 @@
},
"fill": 10,
"id": 11,
"interval": "1m",
"linewidth": 0,
"links": [ ],
"options": {
@@ -1544,6 +1555,7 @@
"overrides": [ ]
},
"id": 12,
"interval": "1m",
"links": [ ],
"nullPointMode": "null as zero",
"options": {

View File

@@ -1,555 +0,0 @@
groups:
- name: "mimir_api_1"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_request_duration_seconds_sum[5m])) by (cluster, job) / sum(rate(cortex_request_duration_seconds_count[5m]))
by (cluster, job)"
record: "cluster_job:cortex_request_duration_seconds:avg"
- expr: "sum(rate(cortex_request_duration_seconds_bucket[5m])) by (le, cluster, job)"
record: "cluster_job:cortex_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_request_duration_seconds_sum[5m])) by (cluster, job)"
record: "cluster_job:cortex_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_request_duration_seconds_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_request_duration_seconds_count:sum_rate"
- name: "mimir_api_2"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_request_duration_seconds_bucket[5m]))
by (le, cluster, job, route))"
record: "cluster_job_route:cortex_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_request_duration_seconds_bucket[5m]))
by (le, cluster, job, route))"
record: "cluster_job_route:cortex_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_request_duration_seconds_sum[5m])) by (cluster, job, route)
/ sum(rate(cortex_request_duration_seconds_count[5m])) by (cluster, job, route)"
record: "cluster_job_route:cortex_request_duration_seconds:avg"
- expr: "sum(rate(cortex_request_duration_seconds_bucket[5m])) by (le, cluster, job,
route)"
record: "cluster_job_route:cortex_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_request_duration_seconds_sum[5m])) by (cluster, job, route)"
record: "cluster_job_route:cortex_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_request_duration_seconds_count[5m])) by (cluster, job, route)"
record: "cluster_job_route:cortex_request_duration_seconds_count:sum_rate"
- name: "mimir_api_3"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_request_duration_seconds_bucket[5m]))
by (le, cluster, namespace, job, route))"
record: "cluster_namespace_job_route:cortex_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_request_duration_seconds_bucket[5m]))
by (le, cluster, namespace, job, route))"
record: "cluster_namespace_job_route:cortex_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_request_duration_seconds_sum[5m])) by (cluster, namespace,
job, route) / sum(rate(cortex_request_duration_seconds_count[5m])) by (cluster,
namespace, job, route)"
record: "cluster_namespace_job_route:cortex_request_duration_seconds:avg"
- expr: "sum(rate(cortex_request_duration_seconds_bucket[5m])) by (le, cluster, namespace,
job, route)"
record: "cluster_namespace_job_route:cortex_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_request_duration_seconds_sum[5m])) by (cluster, namespace,
job, route)"
record: "cluster_namespace_job_route:cortex_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_request_duration_seconds_count[5m])) by (cluster, namespace,
job, route)"
record: "cluster_namespace_job_route:cortex_request_duration_seconds_count:sum_rate"
- name: "mimir_querier_api"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_querier_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_querier_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_querier_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_querier_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_querier_request_duration_seconds_sum[5m])) by (cluster,
job) / sum(rate(cortex_querier_request_duration_seconds_count[5m])) by (cluster,
job)"
record: "cluster_job:cortex_querier_request_duration_seconds:avg"
- expr: "sum(rate(cortex_querier_request_duration_seconds_bucket[5m])) by (le, cluster,
job)"
record: "cluster_job:cortex_querier_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_querier_request_duration_seconds_sum[5m])) by (cluster,
job)"
record: "cluster_job:cortex_querier_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_querier_request_duration_seconds_count[5m])) by (cluster,
job)"
record: "cluster_job:cortex_querier_request_duration_seconds_count:sum_rate"
- expr: "histogram_quantile(0.99, sum(rate(cortex_querier_request_duration_seconds_bucket[5m]))
by (le, cluster, job, route))"
record: "cluster_job_route:cortex_querier_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_querier_request_duration_seconds_bucket[5m]))
by (le, cluster, job, route))"
record: "cluster_job_route:cortex_querier_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_querier_request_duration_seconds_sum[5m])) by (cluster,
job, route) / sum(rate(cortex_querier_request_duration_seconds_count[5m])) by
(cluster, job, route)"
record: "cluster_job_route:cortex_querier_request_duration_seconds:avg"
- expr: "sum(rate(cortex_querier_request_duration_seconds_bucket[5m])) by (le, cluster,
job, route)"
record: "cluster_job_route:cortex_querier_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_querier_request_duration_seconds_sum[5m])) by (cluster,
job, route)"
record: "cluster_job_route:cortex_querier_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_querier_request_duration_seconds_count[5m])) by (cluster,
job, route)"
record: "cluster_job_route:cortex_querier_request_duration_seconds_count:sum_rate"
- expr: "histogram_quantile(0.99, sum(rate(cortex_querier_request_duration_seconds_bucket[5m]))
by (le, cluster, namespace, job, route))"
record: "cluster_namespace_job_route:cortex_querier_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_querier_request_duration_seconds_bucket[5m]))
by (le, cluster, namespace, job, route))"
record: "cluster_namespace_job_route:cortex_querier_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_querier_request_duration_seconds_sum[5m])) by (cluster,
namespace, job, route) / sum(rate(cortex_querier_request_duration_seconds_count[5m]))
by (cluster, namespace, job, route)"
record: "cluster_namespace_job_route:cortex_querier_request_duration_seconds:avg"
- expr: "sum(rate(cortex_querier_request_duration_seconds_bucket[5m])) by (le, cluster,
namespace, job, route)"
record: "cluster_namespace_job_route:cortex_querier_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_querier_request_duration_seconds_sum[5m])) by (cluster,
namespace, job, route)"
record: "cluster_namespace_job_route:cortex_querier_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_querier_request_duration_seconds_count[5m])) by (cluster,
namespace, job, route)"
record: "cluster_namespace_job_route:cortex_querier_request_duration_seconds_count:sum_rate"
- name: "mimir_cache"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_memcache_request_duration_seconds_bucket[5m]))
by (le, cluster, job, method))"
record: "cluster_job_method:cortex_memcache_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_memcache_request_duration_seconds_bucket[5m]))
by (le, cluster, job, method))"
record: "cluster_job_method:cortex_memcache_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_memcache_request_duration_seconds_sum[5m])) by (cluster,
job, method) / sum(rate(cortex_memcache_request_duration_seconds_count[5m]))
by (cluster, job, method)"
record: "cluster_job_method:cortex_memcache_request_duration_seconds:avg"
- expr: "sum(rate(cortex_memcache_request_duration_seconds_bucket[5m])) by (le, cluster,
job, method)"
record: "cluster_job_method:cortex_memcache_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_memcache_request_duration_seconds_sum[5m])) by (cluster,
job, method)"
record: "cluster_job_method:cortex_memcache_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_memcache_request_duration_seconds_count[5m])) by (cluster,
job, method)"
record: "cluster_job_method:cortex_memcache_request_duration_seconds_count:sum_rate"
- expr: "histogram_quantile(0.99, sum(rate(cortex_cache_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_cache_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_cache_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_cache_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_cache_request_duration_seconds_sum[5m])) by (cluster, job)
/ sum(rate(cortex_cache_request_duration_seconds_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_cache_request_duration_seconds:avg"
- expr: "sum(rate(cortex_cache_request_duration_seconds_bucket[5m])) by (le, cluster,
job)"
record: "cluster_job:cortex_cache_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_cache_request_duration_seconds_sum[5m])) by (cluster, job)"
record: "cluster_job:cortex_cache_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_cache_request_duration_seconds_count[5m])) by (cluster,
job)"
record: "cluster_job:cortex_cache_request_duration_seconds_count:sum_rate"
- expr: "histogram_quantile(0.99, sum(rate(cortex_cache_request_duration_seconds_bucket[5m]))
by (le, cluster, job, method))"
record: "cluster_job_method:cortex_cache_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_cache_request_duration_seconds_bucket[5m]))
by (le, cluster, job, method))"
record: "cluster_job_method:cortex_cache_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_cache_request_duration_seconds_sum[5m])) by (cluster, job,
method) / sum(rate(cortex_cache_request_duration_seconds_count[5m])) by (cluster,
job, method)"
record: "cluster_job_method:cortex_cache_request_duration_seconds:avg"
- expr: "sum(rate(cortex_cache_request_duration_seconds_bucket[5m])) by (le, cluster,
job, method)"
record: "cluster_job_method:cortex_cache_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_cache_request_duration_seconds_sum[5m])) by (cluster, job,
method)"
record: "cluster_job_method:cortex_cache_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_cache_request_duration_seconds_count[5m])) by (cluster,
job, method)"
record: "cluster_job_method:cortex_cache_request_duration_seconds_count:sum_rate"
- name: "mimir_storage"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_kv_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_kv_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_kv_request_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_kv_request_duration_seconds:50quantile"
- expr: "sum(rate(cortex_kv_request_duration_seconds_sum[5m])) by (cluster, job)
/ sum(rate(cortex_kv_request_duration_seconds_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_kv_request_duration_seconds:avg"
- expr: "sum(rate(cortex_kv_request_duration_seconds_bucket[5m])) by (le, cluster,
job)"
record: "cluster_job:cortex_kv_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_kv_request_duration_seconds_sum[5m])) by (cluster, job)"
record: "cluster_job:cortex_kv_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_kv_request_duration_seconds_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_kv_request_duration_seconds_count:sum_rate"
- name: "mimir_queries"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_query_frontend_retries_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_query_frontend_retries:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_query_frontend_retries_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_query_frontend_retries:50quantile"
- expr: "sum(rate(cortex_query_frontend_retries_sum[5m])) by (cluster, job) / sum(rate(cortex_query_frontend_retries_count[5m]))
by (cluster, job)"
record: "cluster_job:cortex_query_frontend_retries:avg"
- expr: "sum(rate(cortex_query_frontend_retries_bucket[5m])) by (le, cluster, job)"
record: "cluster_job:cortex_query_frontend_retries_bucket:sum_rate"
- expr: "sum(rate(cortex_query_frontend_retries_sum[5m])) by (cluster, job)"
record: "cluster_job:cortex_query_frontend_retries_sum:sum_rate"
- expr: "sum(rate(cortex_query_frontend_retries_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_query_frontend_retries_count:sum_rate"
- expr: "histogram_quantile(0.99, sum(rate(cortex_query_frontend_queue_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_query_frontend_queue_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_query_frontend_queue_duration_seconds_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_query_frontend_queue_duration_seconds:50quantile"
- expr: "sum(rate(cortex_query_frontend_queue_duration_seconds_sum[5m])) by (cluster,
job) / sum(rate(cortex_query_frontend_queue_duration_seconds_count[5m])) by
(cluster, job)"
record: "cluster_job:cortex_query_frontend_queue_duration_seconds:avg"
- expr: "sum(rate(cortex_query_frontend_queue_duration_seconds_bucket[5m])) by (le,
cluster, job)"
record: "cluster_job:cortex_query_frontend_queue_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(cortex_query_frontend_queue_duration_seconds_sum[5m])) by (cluster,
job)"
record: "cluster_job:cortex_query_frontend_queue_duration_seconds_sum:sum_rate"
- expr: "sum(rate(cortex_query_frontend_queue_duration_seconds_count[5m])) by (cluster,
job)"
record: "cluster_job:cortex_query_frontend_queue_duration_seconds_count:sum_rate"
- name: "mimir_ingester_queries"
rules:
- expr: "histogram_quantile(0.99, sum(rate(cortex_ingester_queried_series_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_ingester_queried_series:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_ingester_queried_series_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_ingester_queried_series:50quantile"
- expr: "sum(rate(cortex_ingester_queried_series_sum[5m])) by (cluster, job) / sum(rate(cortex_ingester_queried_series_count[5m]))
by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_series:avg"
- expr: "sum(rate(cortex_ingester_queried_series_bucket[5m])) by (le, cluster, job)"
record: "cluster_job:cortex_ingester_queried_series_bucket:sum_rate"
- expr: "sum(rate(cortex_ingester_queried_series_sum[5m])) by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_series_sum:sum_rate"
- expr: "sum(rate(cortex_ingester_queried_series_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_series_count:sum_rate"
- expr: "histogram_quantile(0.99, sum(rate(cortex_ingester_queried_samples_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_ingester_queried_samples:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_ingester_queried_samples_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_ingester_queried_samples:50quantile"
- expr: "sum(rate(cortex_ingester_queried_samples_sum[5m])) by (cluster, job) / sum(rate(cortex_ingester_queried_samples_count[5m]))
by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_samples:avg"
- expr: "sum(rate(cortex_ingester_queried_samples_bucket[5m])) by (le, cluster, job)"
record: "cluster_job:cortex_ingester_queried_samples_bucket:sum_rate"
- expr: "sum(rate(cortex_ingester_queried_samples_sum[5m])) by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_samples_sum:sum_rate"
- expr: "sum(rate(cortex_ingester_queried_samples_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_samples_count:sum_rate"
- expr: "histogram_quantile(0.99, sum(rate(cortex_ingester_queried_exemplars_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_ingester_queried_exemplars:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(cortex_ingester_queried_exemplars_bucket[5m]))
by (le, cluster, job))"
record: "cluster_job:cortex_ingester_queried_exemplars:50quantile"
- expr: "sum(rate(cortex_ingester_queried_exemplars_sum[5m])) by (cluster, job) /
sum(rate(cortex_ingester_queried_exemplars_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_exemplars:avg"
- expr: "sum(rate(cortex_ingester_queried_exemplars_bucket[5m])) by (le, cluster,
job)"
record: "cluster_job:cortex_ingester_queried_exemplars_bucket:sum_rate"
- expr: "sum(rate(cortex_ingester_queried_exemplars_sum[5m])) by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_exemplars_sum:sum_rate"
- expr: "sum(rate(cortex_ingester_queried_exemplars_count[5m])) by (cluster, job)"
record: "cluster_job:cortex_ingester_queried_exemplars_count:sum_rate"
- name: "mimir_received_samples"
rules:
- expr: "sum by (cluster, namespace, job) (rate(cortex_distributor_received_samples_total[5m]))"
record: "cluster_namespace_job:cortex_distributor_received_samples:rate5m"
- name: "mimir_exemplars_in"
rules:
- expr: "sum by (cluster, namespace, job) (rate(cortex_distributor_exemplars_in_total[5m]))"
record: "cluster_namespace_job:cortex_distributor_exemplars_in:rate5m"
- name: "mimir_received_exemplars"
rules:
- expr: "sum by (cluster, namespace, job) (rate(cortex_distributor_received_exemplars_total[5m]))"
record: "cluster_namespace_job:cortex_distributor_received_exemplars:rate5m"
- name: "mimir_exemplars_ingested"
rules:
- expr: "sum by (cluster, namespace, job) (rate(cortex_ingester_ingested_exemplars_total[5m]))"
record: "cluster_namespace_job:cortex_ingester_ingested_exemplars:rate5m"
- name: "mimir_exemplars_appended"
rules:
- expr: "sum by (cluster, namespace, job) (rate(cortex_ingester_tsdb_exemplar_exemplars_appended_total[5m]))"
record: "cluster_namespace_job:cortex_ingester_tsdb_exemplar_exemplars_appended:rate5m"
- name: "mimir_scaling_rules"
rules:
- expr: |
# Convenience rule to get the number of replicas for both a deployment and a statefulset.
# Multi-zone deployments are grouped together removing the "zone-X" suffix.
sum by (cluster, namespace, deployment) (
label_replace(
kube_deployment_spec_replicas,
# The question mark in "(.*?)" is used to make it non-greedy, otherwise it
# always matches everything and the (optional) zone is not removed.
"deployment", "$1", "deployment", "(.*?)(?:-zone-[a-z])?"
)
)
or
sum by (cluster, namespace, deployment) (
label_replace(kube_statefulset_replicas, "deployment", "$1", "statefulset", "(.*?)(?:-zone-[a-z])?")
)
record: "cluster_namespace_deployment:actual_replicas:count"
- expr: |
ceil(
quantile_over_time(0.99,
sum by (cluster, namespace) (
cluster_namespace_job:cortex_distributor_received_samples:rate5m
)[24h:]
)
/ 240000
)
labels:
deployment: "distributor"
reason: "sample_rate"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
ceil(
sum by (cluster, namespace) (cortex_limits_overrides{limit_name="ingestion_rate"})
* 0.59999999999999998 / 240000
)
labels:
deployment: "distributor"
reason: "sample_rate_limits"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
ceil(
quantile_over_time(0.99,
sum by (cluster, namespace) (
cluster_namespace_job:cortex_distributor_received_samples:rate5m
)[24h:]
)
* 3 / 80000
)
labels:
deployment: "ingester"
reason: "sample_rate"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
ceil(
quantile_over_time(0.99,
sum by(cluster, namespace) (
cortex_ingester_memory_series
)[24h:]
)
/ 1500000
)
labels:
deployment: "ingester"
reason: "active_series"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
ceil(
sum by (cluster, namespace) (cortex_limits_overrides{limit_name="max_global_series_per_user"})
* 3 * 0.59999999999999998 / 1500000
)
labels:
deployment: "ingester"
reason: "active_series_limits"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
ceil(
sum by (cluster, namespace) (cortex_limits_overrides{limit_name="ingestion_rate"})
* 0.59999999999999998 / 80000
)
labels:
deployment: "ingester"
reason: "sample_rate_limits"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
ceil(
(sum by (cluster, namespace) (
cortex_ingester_tsdb_storage_blocks_bytes{job=~".+/ingester.*"}
) / 4)
/
avg by (cluster, namespace) (
memcached_limit_bytes{job=~".+/memcached"}
)
)
labels:
deployment: "memcached"
reason: "active_series"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
sum by (cluster, namespace, deployment) (
label_replace(
label_replace(
sum by (cluster, namespace, pod)(rate(container_cpu_usage_seconds_total[5m])),
"deployment", "$1", "pod", "(.*)-(?:([0-9]+)|([a-z0-9]+)-([a-z0-9]+))"
),
# The question mark in "(.*?)" is used to make it non-greedy, otherwise it
# always matches everything and the (optional) zone is not removed.
"deployment", "$1", "deployment", "(.*?)(?:-zone-[a-z])?"
)
)
record: "cluster_namespace_deployment:container_cpu_usage_seconds_total:sum_rate"
- expr: |
# Convenience rule to get the CPU request for both a deployment and a statefulset.
# Multi-zone deployments are grouped together removing the "zone-X" suffix.
# This recording rule is made compatible with the breaking changes introduced in kube-state-metrics v2
# that remove resource metrics, ref:
# - https://github.com/kubernetes/kube-state-metrics/blob/master/CHANGELOG.md#v200-alpha--2020-09-16
# - https://github.com/kubernetes/kube-state-metrics/pull/1004
#
# This is the old expression, compatible with kube-state-metrics < v2.0.0,
# where kube_pod_container_resource_requests_cpu_cores was removed:
(
sum by (cluster, namespace, deployment) (
label_replace(
label_replace(
kube_pod_container_resource_requests_cpu_cores,
"deployment", "$1", "pod", "(.*)-(?:([0-9]+)|([a-z0-9]+)-([a-z0-9]+))"
),
# The question mark in "(.*?)" is used to make it non-greedy, otherwise it
# always matches everything and the (optional) zone is not removed.
"deployment", "$1", "deployment", "(.*?)(?:-zone-[a-z])?"
)
)
)
or
# This expression is compatible with kube-state-metrics >= v1.4.0,
# where kube_pod_container_resource_requests was introduced.
(
sum by (cluster, namespace, deployment) (
label_replace(
label_replace(
kube_pod_container_resource_requests{resource="cpu"},
"deployment", "$1", "pod", "(.*)-(?:([0-9]+)|([a-z0-9]+)-([a-z0-9]+))"
),
# The question mark in "(.*?)" is used to make it non-greedy, otherwise it
# always matches everything and the (optional) zone is not removed.
"deployment", "$1", "deployment", "(.*?)(?:-zone-[a-z])?"
)
)
)
record: "cluster_namespace_deployment:kube_pod_container_resource_requests_cpu_cores:sum"
- expr: |
# Jobs should be sized to their CPU usage.
# We do this by comparing 99th percentile usage over the last 24hrs to
# their current provisioned #replicas and resource requests.
ceil(
cluster_namespace_deployment:actual_replicas:count
*
quantile_over_time(0.99, cluster_namespace_deployment:container_cpu_usage_seconds_total:sum_rate[24h])
/
cluster_namespace_deployment:kube_pod_container_resource_requests_cpu_cores:sum
)
labels:
reason: "cpu_usage"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- expr: |
# Convenience rule to get the Memory utilization for both a deployment and a statefulset.
# Multi-zone deployments are grouped together removing the "zone-X" suffix.
sum by (cluster, namespace, deployment) (
label_replace(
label_replace(
container_memory_usage_bytes{image!=""},
"deployment", "$1", "pod", "(.*)-(?:([0-9]+)|([a-z0-9]+)-([a-z0-9]+))"
),
# The question mark in "(.*?)" is used to make it non-greedy, otherwise it
# always matches everything and the (optional) zone is not removed.
"deployment", "$1", "deployment", "(.*?)(?:-zone-[a-z])?"
)
)
record: "cluster_namespace_deployment:container_memory_usage_bytes:sum"
- expr: |
# Convenience rule to get the Memory request for both a deployment and a statefulset.
# Multi-zone deployments are grouped together removing the "zone-X" suffix.
# This recording rule is made compatible with the breaking changes introduced in kube-state-metrics v2
# that remove resource metrics, ref:
# - https://github.com/kubernetes/kube-state-metrics/blob/master/CHANGELOG.md#v200-alpha--2020-09-16
# - https://github.com/kubernetes/kube-state-metrics/pull/1004
#
# This is the old expression, compatible with kube-state-metrics < v2.0.0,
# where kube_pod_container_resource_requests_memory_bytes was removed:
(
sum by (cluster, namespace, deployment) (
label_replace(
label_replace(
kube_pod_container_resource_requests_memory_bytes,
"deployment", "$1", "pod", "(.*)-(?:([0-9]+)|([a-z0-9]+)-([a-z0-9]+))"
),
# The question mark in "(.*?)" is used to make it non-greedy, otherwise it
# always matches everything and the (optional) zone is not removed.
"deployment", "$1", "deployment", "(.*?)(?:-zone-[a-z])?"
)
)
)
or
# This expression is compatible with kube-state-metrics >= v1.4.0,
# where kube_pod_container_resource_requests was introduced.
(
sum by (cluster, namespace, deployment) (
label_replace(
label_replace(
kube_pod_container_resource_requests{resource="memory"},
"deployment", "$1", "pod", "(.*)-(?:([0-9]+)|([a-z0-9]+)-([a-z0-9]+))"
),
# The question mark in "(.*?)" is used to make it non-greedy, otherwise it
# always matches everything and the (optional) zone is not removed.
"deployment", "$1", "deployment", "(.*?)(?:-zone-[a-z])?"
)
)
)
record: "cluster_namespace_deployment:kube_pod_container_resource_requests_memory_bytes:sum"
- expr: |
# Jobs should be sized to their Memory usage.
# We do this by comparing 99th percentile usage over the last 24hrs to
# their current provisioned #replicas and resource requests.
ceil(
cluster_namespace_deployment:actual_replicas:count
*
quantile_over_time(0.99, cluster_namespace_deployment:container_memory_usage_bytes:sum[24h])
/
cluster_namespace_deployment:kube_pod_container_resource_requests_memory_bytes:sum
)
labels:
reason: "memory_usage"
record: "cluster_namespace_deployment_reason:required_replicas:count"
- name: "mimir_alertmanager_rules"
rules:
- expr: "sum by (cluster, job, pod) (cortex_alertmanager_alerts)"
record: "cluster_job_pod:cortex_alertmanager_alerts:sum"
- expr: "sum by (cluster, job, pod) (cortex_alertmanager_silences)"
record: "cluster_job_pod:cortex_alertmanager_silences:sum"
- expr: "sum by (cluster, job) (rate(cortex_alertmanager_alerts_received_total[5m]))"
record: "cluster_job:cortex_alertmanager_alerts_received_total:rate5m"
- expr: "sum by (cluster, job) (rate(cortex_alertmanager_alerts_invalid_total[5m]))"
record: "cluster_job:cortex_alertmanager_alerts_invalid_total:rate5m"
- expr: "sum by (cluster, job, integration) (rate(cortex_alertmanager_notifications_total[5m]))"
record: "cluster_job_integration:cortex_alertmanager_notifications_total:rate5m"
- expr: "sum by (cluster, job, integration) (rate(cortex_alertmanager_notifications_failed_total[5m]))"
record: "cluster_job_integration:cortex_alertmanager_notifications_failed_total:rate5m"
- expr: "sum by (cluster, job) (rate(cortex_alertmanager_state_replication_total[5m]))"
record: "cluster_job:cortex_alertmanager_state_replication_total:rate5m"
- expr: "sum by (cluster, job) (rate(cortex_alertmanager_state_replication_failed_total[5m]))"
record: "cluster_job:cortex_alertmanager_state_replication_failed_total:rate5m"
- expr: "sum by (cluster, job) (rate(cortex_alertmanager_partial_state_merges_total[5m]))"
record: "cluster_job:cortex_alertmanager_partial_state_merges_total:rate5m"
- expr: "sum by (cluster, job) (rate(cortex_alertmanager_partial_state_merges_failed_total[5m]))"
record: "cluster_job:cortex_alertmanager_partial_state_merges_failed_total:rate5m"
- name: "mimir_ingester_rules"
rules:
- expr: "sum by(cluster, namespace, pod) (rate(cortex_ingester_ingested_samples_total[5m]))"
record: "cluster_namespace_pod:cortex_ingester_ingested_samples_total:rate1m"

View File

@@ -1,15 +0,0 @@
groups:
- name: "tempo_rules"
rules:
- expr: "histogram_quantile(0.99, sum(rate(tempo_request_duration_seconds_bucket[5m])) by (le, cluster, namespace, job, route))"
record: "cluster_namespace_job_route:tempo_request_duration_seconds:99quantile"
- expr: "histogram_quantile(0.50, sum(rate(tempo_request_duration_seconds_bucket[5m])) by (le, cluster, namespace, job, route))"
record: "cluster_namespace_job_route:tempo_request_duration_seconds:50quantile"
- expr: "sum(rate(tempo_request_duration_seconds_sum[5m])) by (cluster, namespace, job, route) / sum(rate(tempo_request_duration_seconds_count[5m])) by (cluster, namespace, job, route)"
record: "cluster_namespace_job_route:tempo_request_duration_seconds:avg"
- expr: "sum(rate(tempo_request_duration_seconds_bucket[5m])) by (le, cluster, namespace, job, route)"
record: "cluster_namespace_job_route:tempo_request_duration_seconds_bucket:sum_rate"
- expr: "sum(rate(tempo_request_duration_seconds_sum[5m])) by (cluster, namespace, job, route)"
record: "cluster_namespace_job_route:tempo_request_duration_seconds_sum:sum_rate"
- expr: "sum(rate(tempo_request_duration_seconds_count[5m])) by (cluster, namespace, job, route)"
record: "cluster_namespace_job_route:tempo_request_duration_seconds_count:sum_rate"

View File

@@ -48,7 +48,7 @@
{{- define "agent.tempo_write_targets" -}}
{{- $list := list }}
{{- if .Values.local.traces.enabled }}
{{- $list = append $list ("otelcol.exporter.otlp.local.input") }}
{{- $list = append $list ("otelcol.exporter.otlphttp.local.input") }}
{{- end }}
{{- if .Values.cloud.traces.enabled }}
{{- $list = append $list ("otelcol.exporter.otlphttp.cloud.input") }}

View File

@@ -294,9 +294,7 @@ data:
// We don't technically need this, but it shows how to change listen address and incoming port.
// In this case, the Agent is listening on all available bindable addresses on port 4317 (which is the
// default OTLP gRPC port) for the OTLP protocol.
grpc {
endpoint = "0.0.0.0:4317"
}
grpc {}
// We define where to send the output of all ingested traces. In this case, to the OpenTelemetry batch processor
// named 'default'.
@@ -345,6 +343,14 @@ data:
}
{{- end }}
{{- if .Values.local.traces.enabled }}
otelcol.exporter.otlphttp "local" {
client {
endpoint = "http://{{- .Release.Name -}}-tempo-distributor.svc:4318"
}
}
{{- end }}
{{- if .Values.cloud.logs.enabled }}
loki.write "cloud" {
endpoint {

View File

@@ -1,4 +1,4 @@
{{- if and .Values.local.grafana.enabled (or .Values.dashboards.logs.enabled .Values.dashboards.metrics.enabled .Values.dashboards.traces.enabled) }}
{{- if and .Values.local.grafana.enabled .Values.dashboards.logs.enabled }}
---
apiVersion: v1
kind: ConfigMap

View File

@@ -1,4 +1,4 @@
{{- if and .Values.local.grafana.enabled (or .Values.dashboards.logs.enabled .Values.dashboards.metrics.enabled .Values.dashboards.traces.enabled) }}
{{- if and .Values.local.grafana.enabled .Values.dashboards.logs.enabled }}
---
apiVersion: v1
kind: ConfigMap

View File

@@ -1,16 +1,4 @@
{{- if .Values.local.grafana.enabled }}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -32,7 +20,7 @@ spec:
- 0
containers:
- name: grafana
image: grafana/grafana:10.0.0
image: grafana/grafana:{{- .Values.grafana.version }}
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
@@ -65,7 +53,7 @@ spec:
name: grafana-pv
- mountPath: /etc/grafana/provisioning/datasources
name: datasources-provisioning
{{- if or (or .Values.dashboards.logs.enabled .Values.dashboards.metrics.enabled) .Values.dashboards.traces.enabled }}
{{- if .Values.dashboards.logs.enabled }}
- mountPath: /etc/grafana/provisioning/dashboards
name: dashboards-provisioning
{{- end }}
@@ -98,19 +86,4 @@ spec:
- name: agent-dashboards-1
configMap:
name: agent-dashboards-1
---
apiVersion: v1
kind: Service
metadata:
name: grafana
spec:
ports:
- port: 3000
protocol: TCP
targetPort: http-grafana
selector:
app: grafana
sessionAffinity: None
type: ClusterIP # Make this configurable
{{- end }}

View File

@@ -0,0 +1,12 @@
{{- if .Values.local.grafana.enabled }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
{{- end }}

View File

@@ -0,0 +1,15 @@
{{- if .Values.local.grafana.enabled }}
apiVersion: v1
kind: Service
metadata:
name: grafana
spec:
ports:
- port: 3000
protocol: TCP
targetPort: http-grafana
selector:
app: grafana
sessionAffinity: None
type: ClusterIP # Make this configurable
{{- end }}

View File

@@ -1,5 +1,5 @@
{{- if .Values.local.grafana.enabled }}
{{- if and .Values.local.grafana.enabled (or .Values.dashboards.logs.enabled .Values.dashboards.metrics.enabled .Values.dashboards.traces.enabled) }}
{{- if and .Values.local.grafana.enabled .Values.dashboards.logs.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -49,6 +49,9 @@ spec:
- containerPort: 7946
name: memberlist
protocol: TCP
envFrom:
- secretRef:
name: minio
readinessProbe:
failureThreshold: 3
httpGet:

View File

@@ -1,5 +1,5 @@
{{- if .Values.local.metrics.enabled }}
{{- if and .Values.local.grafana.enabled (or .Values.dashboards.logs.enabled .Values.dashboards.metrics.enabled .Values.dashboards.traces.enabled) }}
{{- if and .Values.local.grafana.enabled .Values.dashboards.logs.enabled }}
---
apiVersion: v1
kind: ConfigMap
@@ -10,11 +10,5 @@ data:
{{- if .Values.dashboards.logs.enabled }}
{{ ($.Files.Glob "src/rules/loki-rules.yaml").AsConfig | indent 2 }}
{{- end }}
{{- if .Values.dashboards.metrics.enabled }}
{{ ($.Files.Glob "src/rules/mimir-rules.yaml").AsConfig | indent 2 }}
{{- end }}
{{- if .Values.dashboards.traces.enabled }}
{{ ($.Files.Glob "src/rules/tempo-rules.yaml").AsConfig | indent 2 }}
{{- end }}
{{- end }}
{{- end }}

View File

@@ -3,7 +3,6 @@ namespacesToMonitor:
- loki
# The name of the cluster where this will be installed
clusterLabelValue: "meta-monitoring"
# Set to true to write logs, metrics or traces to Grafana Cloud
# The secrets have to be created first
cloud:
@@ -16,7 +15,6 @@ cloud:
traces:
enabled: true
secret: "traces"
# Set to true for a local version of logs, metrics or traces
local:
grafana:
@@ -28,9 +26,9 @@ local:
traces:
enabled: false
minio:
enabled: false # This should be set to true if any of the previous is enabled
enabled: false # This should be set to true if any of the previous is enabled
grafana:
version: 10.4.2
# Gateway ingress configuration
ingress:
# -- Specifies whether an ingress for the gateway should be created
@@ -38,9 +36,9 @@ grafana:
# -- Ingress Class Name. MAY be required for Kubernetes versions >= 1.18
ingressClassName: ""
# -- Annotations for the gateway ingress
annotations: { }
annotations: {}
# -- Labels for the gateway ingress
labels: { }
labels: {}
# -- Hosts configuration for the gateway ingress, passed through the `tpl` function to allow templating
hosts:
- host: monitoring.example.com
@@ -53,18 +51,14 @@ grafana:
# - secretName: grafana-tls
# hosts:
# - monitoring.example.com
logs:
# Adding regexes here will add a stage.replace block for logs. For more information see
# https://grafana.com/docs/agent/latest/flow/reference/components/loki.process/#stagereplace-block
piiRegexes:
# This example replaces the word after password with *****
# - expression: "password (\\\\S+)"
# source: "" # Empty uses the log message
# replace: "*****""
# The lines matching these will be kept in Loki
piiRegexes: null # This example replaces the word after password with *****
# - expression: "password (\\\\S+)"
# source: "" # Empty uses the log message
# replace: "*****""
# The lines matching these will be kept in Loki
retain:
# This shows the queries
- caller=metrics.go
@@ -78,7 +72,6 @@ logs:
# - caller=push.go
# Additional log lines to retain
extraLogs: []
metrics:
# The list of metrics to retain for logging dashboards
retain:
@@ -179,21 +172,10 @@ metrics:
- promtail_custom_bad_words_total
# Additional metrics to retain
extraMetrics: []
# Set enabled = true to add the default logs/metrics/traces dashboards to the local Grafana
# Set enabled = true to add the default logs dashboards to the local Grafana
dashboards:
logs:
enabled: true
metrics:
enabled: true
traces:
enabled: true
global:
minio:
rootUser: "rootuser"
rootPassword: "rootpassword"
kubeStateMetrics:
# Scrape https://github.com/kubernetes/kube-state-metrics by default
enabled: true
@@ -201,10 +183,8 @@ kubeStateMetrics:
# https://artifacthub.io/packages/helm/prometheus-community/kube-state-metrics/
# is used. Change this if kube-state-metrics is installed somewhere else.
endpoint: kube-state-metrics.kube-state-metrics.svc.cluster.local:8080
# The following are configuration for the dependencies.
# These should usually not be changed.
loki:
loki:
auth_enabled: false
@@ -229,9 +209,9 @@ loki:
common:
storage:
s3:
access_key_id: "{{ .Values.global.minio.rootUser }}"
access_key_id: "${rootUser}"
endpoint: "{{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000"
secret_access_key: "{{ .Values.global.minio.rootPassword }}"
secret_access_key: "${rootPassword}"
compactor:
retention_enabled: true
delete_request_store: s3
@@ -254,9 +234,24 @@ loki:
installOperator: false
lokiCanary:
enabled: false
test:
enabled: false
write:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
read:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
backend:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
alloy:
alloy:
clustering:
@@ -288,37 +283,36 @@ alloy:
maxReplicas: 30
targetMemoryUtilizationPercentage: 90
targetCPUUtilizationPercentage: 90
mimir-distributed:
minio:
enabled: false
global:
extraEnvFrom:
- secretRef:
name: "minio"
mimir:
structuredConfig:
alertmanager_storage:
s3:
bucket_name: mimir-ruler
access_key_id: "{{ .Values.global.minio.rootUser }}"
endpoint: "{{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000"
secret_access_key: "{{ .Values.global.minio.rootPassword }}"
insecure: true
blocks_storage:
backend: s3
s3:
bucket_name: mimir-tsdb
access_key_id: "{{ .Values.global.minio.rootUser }}"
endpoint: "{{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000"
secret_access_key: "{{ .Values.global.minio.rootPassword }}"
insecure: true
ruler_storage:
s3:
bucket_name: mimir-ruler
access_key_id: "{{ .Values.global.minio.rootUser }}"
endpoint: "{{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000"
secret_access_key: "{{ .Values.global.minio.rootPassword }}"
insecure: true
common:
storage:
backend: s3
s3:
bucket_name: mimir-ruler
access_key_id: "${rootUser}"
endpoint: "{{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000"
secret_access_key: "${rootPassword}"
insecure: true
limits:
compactor_blocks_retention_period: 30d
tempo-distributed:
tempo:
structuredConfig:
@@ -328,22 +322,47 @@ tempo-distributed:
s3:
bucket: tempo
endpoint: "{{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000"
access_key: "{{ .Values.global.minio.rootUser }}"
secret_key: "{{ .Values.global.minio.rootPassword }}"
access_key: "${rootUser}"
secret_key: "${rootPassword}"
insecure: true
compactor:
compaction:
block_retention: 30d
distributor:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
ingester:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
compactor:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
querier:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
queryFrontend:
extraArgs:
- "-config.expand-env=true"
extraEnvFrom:
- secretRef:
name: "minio"
traces:
otlp:
http:
enabled: true
grpc:
enabled: true
minio:
rootUser: rootuser
rootPassword: rootpassword
existingSecret: "minio"
buckets:
- name: loki-chunks
policy: none

View File

@@ -25,17 +25,17 @@
```
kubectl create secret generic logs -n meta \
--from-literal=username=<logs username> \
--from-literal=password=<token>
--from-literal=password=<token> \
--from-literal=endpoint='https://logs-prod-us-central1.grafana.net/loki/api/v1/push'
kubectl create secret generic metrics -n meta \
--from-literal=username=<metrics username> \
--from-literal=password=<token>
--from-literal=password=<token> \
--from-literal=endpoint='https://prometheus-us-central1.grafana.net/api/prom/push'
kubectl create secret generic traces -n meta \
--from-literal=username=<OTLP instance ID> \
--from-literal=password=<token>
--from-literal=password=<token> \
--from-literal=endpoint='https://otlp-gateway-prod-us-east-0.grafana.net/otlp'
```
@@ -67,6 +67,14 @@
kubectl create namespace meta
```
1. Create a secret named `minio` with the user and password for the local Minio:
```
kubectl create secret generic minio -n meta \
--from-literal=rootPassword=<password> \
--from-literal=rootUser=<user>
```
1. Create a values.yaml file based on the [default one](../charts/meta-monitoring/values.yaml). An example minimal values.yaml looks like this:
```