Public/grafana-meta-monitoring-chart
2
0
Fork 0
forked from RemoteSync/grafana-meta-monitoring-chart
Code Pull Requests Activity

Compare commits

merge into: Public:add_minio
Branches Tags
Public:main Public:modified Public:chore/update-dependencies Public:chore/update-minio Public:slim-bean-patch-1 Public:update-metrics-port Public:add_loki_team_to_prs Public:test_branch Public:update_version Public:update_main_page Public:cleanup_ci Public:add_ci_install Public:add_ksm_docs Public:fix_cluster_name Public:fix_dashboards Public:fix_loki_write_endpoint Public:fix_traces_forwarding Public:change_default_sampling_type Public:update_ingress_documentation Public:create_new_version Public:add_more_metrics Public:fix_rules Public:remove_rules_files_mimir_tempo Public:split_up_grafana_template Public:remove_redundant_variables Public:fix_dependency_check Public:chore/update-tempo-distributed Public:fix_alloy_config_for_traces Public:fix_minio_secret_name Public:combine_ci Public:update_grafana Public:chore/update-loki Public:use_secret_for_minio Public:set_step_time_to_1_minute_in_dashboards Public:add_extra_metrics Public:more_dashboard_cleanup Public:remove_non_loki_dashboards Public:open_extra_ports_for_otlp Public:get_logs_from_all_namespaces Public:fix_correct_version Public:fix_logic_for_secrets Public:use_updated_dashboards2 Public:filter_cadvisor_kubelet_metrics_on_namespace Public:add_creation_of_dashboard Public:more_docs_fixes Public:add_more_docs Public:update_installation_instructions Public:fix_release_flow Public:add_ci_to_release_helm_chart Public:remove_reviewers Public:chore/update-grafana-alloy Public:chore/update-mimir-distributed Public:fix_action Public:add_helm_chart_updater Public:add_agent_metrics Public:filter_out_errors_local_loki Public:filter_log_lines Public:filter_only_important_log_lines Public:remove_helm_test_step Public:add_create_kind_cluster_to_ci Public:add_lint_to_ci Public:add_ci Public:add_skip_cdrs_to_installation_step Public:add_retention Public:fix_tempo_dashboards Public:fix_mimir_dashboards Public:add_agent_dashboards Public:add_ruler Public:add_tempo_dashboards.yaml Public:add_scraping_of_cadvisor Public:add_mimir_dashboards Public:use_loki_ssd_dashboards Public:add_kube_state_metrics_scraping Public:add_regexes_to_filter_log_lines Public:add_validation Public:enable_dbs_individually Public:add_minio
Public:v1.3.0 Public:v1.2.0 Public:v1.1.0 Public:v1.0.0 Public:v0.0.3 Public:v0.0.2 Public:v0.0.1
...
pull from: Public:use_loki_ssd_dashboards
Branches Tags
Public:modified Public:chore/update-dependencies Public:main Public:chore/update-minio Public:slim-bean-patch-1 Public:update-metrics-port Public:add_loki_team_to_prs Public:test_branch Public:update_version Public:update_main_page Public:cleanup_ci Public:add_ci_install Public:add_ksm_docs Public:fix_cluster_name Public:fix_dashboards Public:fix_loki_write_endpoint Public:fix_traces_forwarding Public:change_default_sampling_type Public:update_ingress_documentation Public:create_new_version Public:add_more_metrics Public:fix_rules Public:remove_rules_files_mimir_tempo Public:split_up_grafana_template Public:remove_redundant_variables Public:fix_dependency_check Public:chore/update-tempo-distributed Public:fix_alloy_config_for_traces Public:fix_minio_secret_name Public:combine_ci Public:update_grafana Public:chore/update-loki Public:use_secret_for_minio Public:set_step_time_to_1_minute_in_dashboards Public:add_extra_metrics Public:more_dashboard_cleanup Public:remove_non_loki_dashboards Public:open_extra_ports_for_otlp Public:get_logs_from_all_namespaces Public:fix_correct_version Public:fix_logic_for_secrets Public:use_updated_dashboards2 Public:filter_cadvisor_kubelet_metrics_on_namespace Public:add_creation_of_dashboard Public:more_docs_fixes Public:add_more_docs Public:update_installation_instructions Public:fix_release_flow Public:add_ci_to_release_helm_chart Public:remove_reviewers Public:chore/update-grafana-alloy Public:chore/update-mimir-distributed Public:fix_action Public:add_helm_chart_updater Public:add_agent_metrics Public:filter_out_errors_local_loki Public:filter_log_lines Public:filter_only_important_log_lines Public:remove_helm_test_step Public:add_create_kind_cluster_to_ci Public:add_lint_to_ci Public:add_ci Public:add_skip_cdrs_to_installation_step Public:add_retention Public:fix_tempo_dashboards Public:fix_mimir_dashboards Public:add_agent_dashboards Public:add_ruler Public:add_tempo_dashboards.yaml Public:add_scraping_of_cadvisor Public:add_mimir_dashboards Public:use_loki_ssd_dashboards Public:add_kube_state_metrics_scraping Public:add_regexes_to_filter_log_lines Public:add_validation Public:enable_dbs_individually Public:add_minio
Public:v1.3.0 Public:v1.2.0 Public:v1.1.0 Public:v1.0.0 Public:v0.0.3 Public:v0.0.2 Public:v0.0.1

13 Commits
add_minio ... use_loki_s

Author SHA1 Message Date
Michel Hollands
f4d5bcc018 Use Loki SSD dashboards 
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
 2023-07-27 13:48:42 +01:00
Michel Hollands
d999ef0110 Merge pull request #4 from grafana/add_regexes_to_filter_log_lines 
Add regexes to filter log lines
 2023-07-27 08:18:15 +01:00
Michel Hollands
9dd6584bee Add full object with stage replace block 
Signed-off-by: Michel Hollands <michel.hollands@gmail.com>
 2023-07-24 13:42:55 +01:00
Michel Hollands
04cf591478 cleanup and fix 
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
 2023-07-24 13:41:34 +01:00
Michel Hollands
9f54397e83 Add documentation for the regex 
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
 2023-07-24 13:41:34 +01:00
Michel Hollands
fa2b01708c Add regexes to filter out PII 
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
 2023-07-24 13:41:34 +01:00
Michel Hollands
a1cd5d36b0 Merge pull request #3 from grafana/add_validation 
Add validation.yaml
 2023-07-07 16:48:20 +01:00
Michel Hollands
a14b07239b Add check for empty list of namespaces to monitor 
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
 2023-07-06 14:11:08 +01:00
Michel Hollands
116586e383 Add check to if anything is enabled 
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
 2023-07-06 14:08:15 +01:00
Michel Hollands
67b13388a7 Add validation.yaml 
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
 2023-07-06 11:58:00 +01:00
Michel Hollands
39a187457c Merge pull request #2 from grafana/enable_dbs_individually 
Enable DBs separately
 2023-06-30 12:45:08 +01:00
Michel Hollands
ed0ab9852a Enable DBs separately 
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
 2023-06-30 12:36:46 +01:00
Michel Hollands
ad187b1ba2 Merge pull request #1 from grafana/add_minio 
Add Minio and use in Mimir
 2023-06-30 08:24:33 +01:00
19 changed files with 769 additions and 230 deletions
Expand all files Collapse all files

8
charts/meta-monitoring/Chart.yaml
View File

@@ -27,19 +27,19 @@ dependencies:
- name: loki - name: loki
repository: https://grafana.github.io/helm-charts repository: https://grafana.github.io/helm-charts
version: "5.8.0" version: "5.8.0"
condition: local.enabled condition: local.logs.enabled
- name: grafana-agent - name: grafana-agent
repository: https://grafana.github.io/helm-charts repository: https://grafana.github.io/helm-charts
version: "0.15.0" version: "0.15.0"
- name: mimir-distributed - name: mimir-distributed
repository: https://grafana.github.io/helm-charts repository: https://grafana.github.io/helm-charts
version: "4.4.1" version: "4.4.1"
condition: local.enabled condition: local.metrics.enabled
- name: tempo-distributed - name: tempo-distributed
repository: https://grafana.github.io/helm-charts repository: https://grafana.github.io/helm-charts
version: "1.4.7" version: "1.4.7"
condition: local.enabled condition: local.traces.enabled
- name: minio - name: minio
repository: https://charts.min.io repository: https://charts.min.io
version: "5.0.11" version: "5.0.11"
condition: local.enabled condition: local.minio.enabled

388
charts/meta-monitoring/src/dashboards/loki-deletion.json
View File

@@ -217,15 +217,15 @@
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ ], "seriesOverrides": [ ],
"spaceLength": 10, "spaceLength": 10,
"span": 6, "span": 4,
"stack": false, "stack": false,
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(increase(loki_compactor_delete_requests_received_total{cluster=~\"$cluster\", namespace=~\"$namespace\"}[1d]))", "expr": "(loki_compactor_delete_requests_received_total{cluster=~\"$cluster\", namespace=~\"$namespace\"} or on() vector(0)) - on () (loki_compactor_delete_requests_processed_total{cluster=~\"$cluster\", namespace=~\"$namespace\"} or on () vector(0))",
"format": "time_series", "format": "time_series",
"intervalFactor": 2, "intervalFactor": 2,
"legendFormat": "received", "legendFormat": "in progress",
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
} }
@@ -233,7 +233,7 @@
"thresholds": [ ], "thresholds": [ ],
"timeFrom": null, "timeFrom": null,
"timeShift": null, "timeShift": null,
"title": "Delete Requests Received / Day", "title": "# of Delete Requests (received - processed) ",
"tooltip": { "tooltip": {
"shared": true, "shared": true,
"sort": 2, "sort": 2,
@@ -293,7 +293,83 @@
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ ], "seriesOverrides": [ ],
"spaceLength": 10, "spaceLength": 10,
"span": 6, "span": 4,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(increase(loki_compactor_delete_requests_received_total{cluster=~\"$cluster\", namespace=~\"$namespace\"}[1d]))",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "received",
"legendLink": null,
"step": 10
}
],
"thresholds": [ ],
"timeFrom": null,
"timeShift": null,
"title": "Delete Requests Received / Day",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": [ ]
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": 0,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
]
},
{
"aliasColors": { },
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"id": 5,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [ ],
"nullPointMode": "null as zero",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [ ],
"spaceLength": 10,
"span": 4,
"stack": false, "stack": false,
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
@@ -361,7 +437,7 @@
"dashes": false, "dashes": false,
"datasource": "$datasource", "datasource": "$datasource",
"fill": 1, "fill": 1,
"id": 5, "id": 6,
"legend": { "legend": {
"avg": false, "avg": false,
"current": false, "current": false,
@@ -381,7 +457,247 @@
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ ], "seriesOverrides": [ ],
"spaceLength": 10, "spaceLength": 10,
"span": 12, "span": 4,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"compactor\"}",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "{{pod}}",
"legendLink": null,
"step": 10
}
],
"thresholds": [ ],
"timeFrom": null,
"timeShift": null,
"title": "Compactor CPU usage",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": [ ]
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": 0,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
]
},
{
"aliasColors": { },
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"id": 7,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [ ],
"nullPointMode": "null as zero",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [ ],
"spaceLength": 10,
"span": 4,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_memstats_heap_inuse_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"compactor\"} / 1024 / 1024 ",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": " {{pod}} ",
"legendLink": null,
"step": 10
}
],
"thresholds": [ ],
"timeFrom": null,
"timeShift": null,
"title": "Compactor memory usage (MiB)",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": [ ]
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": 0,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
]
},
{
"aliasColors": { },
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"id": 8,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [ ],
"nullPointMode": "null as zero",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [ ],
"spaceLength": 10,
"span": 4,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "loki_boltdb_shipper_compact_tables_operation_duration_seconds{cluster=~\"$cluster\", namespace=~\"$namespace\"}",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "{{pod}}",
"legendLink": null,
"step": 10
}
],
"thresholds": [ ],
"timeFrom": null,
"timeShift": null,
"title": "Compaction run duration (seconds)",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": [ ]
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": 0,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
]
}
],
"repeat": null,
"repeatIteration": null,
"repeatRowId": null,
"showTitle": true,
"title": "Compactor",
"titleSize": "h6"
},
{
"collapse": false,
"height": "250px",
"panels": [
{
"aliasColors": { },
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"id": 9,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [ ],
"nullPointMode": "null as zero",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [ ],
"spaceLength": 10,
"span": 6,
"stack": false, "stack": false,
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
@@ -429,19 +745,7 @@
"show": false "show": false
} }
] ]
} },
],
"repeat": null,
"repeatIteration": null,
"repeatRowId": null,
"showTitle": true,
"title": "Failures",
"titleSize": "h6"
},
{
"collapse": false,
"height": "250px",
"panels": [
{ {
"aliasColors": { }, "aliasColors": { },
"bars": false, "bars": false,
@@ -449,7 +753,7 @@
"dashes": false, "dashes": false,
"datasource": "$datasource", "datasource": "$datasource",
"fill": 1, "fill": 1,
"id": 6, "id": 10,
"legend": { "legend": {
"avg": false, "avg": false,
"current": false, "current": false,
@@ -469,7 +773,7 @@
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ ], "seriesOverrides": [ ],
"spaceLength": 10, "spaceLength": 10,
"span": 12, "span": 6,
"stack": false, "stack": false,
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
@@ -523,7 +827,45 @@
"repeatIteration": null, "repeatIteration": null,
"repeatRowId": null, "repeatRowId": null,
"showTitle": true, "showTitle": true,
"title": "Deleted lines", "title": "Deletion metrics",
"titleSize": "h6"
},
{
"collapse": false,
"height": "250px",
"panels": [
{
"datasource": "$loki_datasource",
"id": 11,
"span": 6,
"targets": [
{
"expr": "{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"compactor\"} |~ \"Started processing delete request|delete request for user marked as processed\" | logfmt | line_format \"{{.ts}} user={{.user}} delete_request_id={{.delete_request_id}} msg={{.msg}}\" ",
"refId": "A"
}
],
"title": "In progress/finished",
"type": "logs"
},
{
"datasource": "$loki_datasource",
"id": 12,
"span": 6,
"targets": [
{
"expr": "{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"compactor\"} |~ \"delete request for user added\" | logfmt | line_format \"{{.ts}} user={{.user}} query='{{.query}}'\"",
"refId": "A"
}
],
"title": "Requests",
"type": "logs"
}
],
"repeat": null,
"repeatIteration": null,
"repeatRowId": null,
"showTitle": true,
"title": "List of deletion requests",
"titleSize": "h6" "titleSize": "h6"
} }
], ],

3
charts/meta-monitoring/src/dashboards/loki-logs.json
View File

@@ -6,7 +6,6 @@
"gnetId": null, "gnetId": null,
"graphTooltip": 0, "graphTooltip": 0,
"hideControls": false, "hideControls": false,
"id": 8,
"iteration": 1583185057230, "iteration": 1583185057230,
"links": [ "links": [
{ {
@@ -1070,4 +1069,4 @@
"title": "Loki / Logs", "title": "Loki / Logs",
"uid": "logs", "uid": "logs",
"version": 0 "version": 0
} }

186
charts/meta-monitoring/src/dashboards/loki-mixin-recording-rules.json
View File

@@ -1,41 +1,27 @@
{ {
"annotations": { "annotations": {
"list": [ "list": [ ]
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [ ],
"type": "dashboard"
},
"type": "dashboard"
},
{
"datasource": "${datasource}",
"enable": false,
"expr": "sum by (tenant) (changes(loki_ruler_wal_prometheus_tsdb_wal_truncations_total{tenant=~\"${tenant}\"}[$__rate_interval]))",
"iconColor": "red",
"name": "WAL Truncations",
"target": {
"queryType": "Azure Monitor",
"refId": "Anno"
},
"titleFormat": "{{tenant}}"
}
]
}, },
"editable": true, "editable": true,
"fiscalYearStartMonth": 0, "fiscalYearStartMonth": 0,
"gnetId": null, "gnetId": null,
"graphTooltip": 0, "graphTooltip": 0,
"hideControls": false,
"iteration": 1635347545534, "iteration": 1635347545534,
"links": [ ], "links": [
{
"asDropdown": true,
"icon": "external link",
"includeVars": true,
"keepTime": true,
"tags": [
"loki"
],
"targetBlank": false,
"title": "Loki Dashboards",
"type": "dashboards"
}
],
"liveNow": false, "liveNow": false,
"panels": [ "panels": [
{ {
@@ -599,59 +585,139 @@
"type": "timeseries" "type": "timeseries"
} }
], ],
"schemaVersion": 31, "refresh": "10s",
"rows": [ ],
"schemaVersion": 14,
"style": "dark", "style": "dark",
"tags": [ ], "tags": [
"loki"
],
"templating": { "templating": {
"list": [ "list": [
{ {
"description": null, "current": {
"error": null, "text": "default",
"value": "default"
},
"hide": 0, "hide": 0,
"includeAll": false, "label": "Data Source",
"label": "Datasource",
"multi": false,
"name": "datasource", "name": "datasource",
"options": [ ], "options": [ ],
"query": "prometheus", "query": "prometheus",
"queryValue": "",
"refresh": 1, "refresh": 1,
"regex": "", "regex": "",
"skipUrlSync": false,
"type": "datasource" "type": "datasource"
}, },
{ {
"allValue": null, "allValue": null,
"datasource": "${datasource}", "current": {
"definition": "label_values(loki_ruler_wal_samples_appended_total, tenant)", "text": "prod",
"description": null, "value": "prod"
"error": null, },
"datasource": "$datasource",
"hide": 0,
"includeAll": false,
"label": "cluster",
"multi": false,
"name": "cluster",
"options": [ ],
"query": "label_values(loki_build_info, cluster)",
"refresh": 1,
"regex": "",
"sort": 2,
"tagValuesQuery": "",
"tags": [ ],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "prod",
"value": "prod"
},
"datasource": "$datasource",
"hide": 0,
"includeAll": false,
"label": "namespace",
"multi": false,
"name": "namespace",
"options": [ ],
"query": "label_values(loki_build_info{cluster=~\"$cluster\"}, namespace)",
"refresh": 1,
"regex": "",
"sort": 2,
"tagValuesQuery": "",
"tags": [ ],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"hide": 0,
"label": null,
"name": "loki_datasource",
"options": [ ],
"query": "loki",
"refresh": 1,
"regex": "",
"type": "datasource"
},
{
"allValue": ".+",
"current": { },
"datasource": "$datasource",
"hide": 0, "hide": 0,
"includeAll": true, "includeAll": true,
"label": "Tenant", "label": null,
"multi": true, "multi": false,
"name": "tenant", "name": "tenant",
"options": [ ], "options": [ ],
"query": { "query": "query_result(sum by (id) (grafanacloud_logs_instance_info) and sum(label_replace(loki_tenant:active_streams{cluster=\"$cluster\",namespace=\"$namespace\"},\"id\",\"$1\",\"tenant\",\"(.*)\")) by(id))",
"query": "label_values(loki_ruler_wal_samples_appended_total, tenant)", "refresh": 0,
"refId": "StandardVariableQuery" "regex": "/\"([^\"]+)\"/",
}, "sort": 1,
"refresh": 2, "tagValuesQuery": "",
"regex": "", "tags": [ ],
"skipUrlSync": false, "tagsQuery": "",
"sort": 0, "type": "query",
"type": "query" "useTags": false
} }
] ]
}, },
"time": { "time": {
"from": "now-6h", "from": "now-1h",
"to": "now" "to": "now"
}, },
"timepicker": { }, "timepicker": {
"timezone": "", "refresh_intervals": [
"title": "Recording Rules", "5s",
"uid": "2xKA_ZK7k", "10s",
"version": 9, "30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "utc",
"title": "Loki / Recording Rules",
"uid": "recording-rules",
"version": 0,
"weekStart": "" "weekStart": ""
} }

57
charts/meta-monitoring/src/dashboards/loki-operational.json
View File

@@ -6,7 +6,6 @@
"gnetId": null, "gnetId": null,
"graphTooltip": 0, "graphTooltip": 0,
"hideControls": false, "hideControls": false,
"id": 68,
"iteration": 1588704280892, "iteration": 1588704280892,
"links": [ "links": [
{ {
@@ -567,17 +566,17 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"api_prom_push|loki_api_v1_push\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.99, sum by (le) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"api_prom_push|loki_api_v1_push\", cluster=~\"$cluster\"})) * 1e3",
"legendFormat": ".99", "legendFormat": ".99",
"refId": "A" "refId": "A"
}, },
{ {
"expr": "histogram_quantile(0.75, sum by (le) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"api_prom_push|loki_api_v1_push\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.75, sum by (le) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"api_prom_push|loki_api_v1_push\", cluster=~\"$cluster\"})) * 1e3",
"legendFormat": ".9", "legendFormat": ".9",
"refId": "B" "refId": "B"
}, },
{ {
"expr": "histogram_quantile(0.5, sum by (le) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"api_prom_push|loki_api_v1_push\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.5, sum by (le) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"api_prom_push|loki_api_v1_push\", cluster=~\"$cluster\"})) * 1e3",
"legendFormat": ".5", "legendFormat": ".5",
"refId": "C" "refId": "C"
} }
@@ -673,17 +672,17 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le) (job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.99, sum by (le) (cluster_job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3",
"legendFormat": ".99", "legendFormat": ".99",
"refId": "A" "refId": "A"
}, },
{ {
"expr": "histogram_quantile(0.9, sum by (le) (job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.9, sum by (le) (cluster_job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3",
"legendFormat": ".9", "legendFormat": ".9",
"refId": "B" "refId": "B"
}, },
{ {
"expr": "histogram_quantile(0.5, sum by (le) (job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.5, sum by (le) (cluster_job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3",
"legendFormat": ".5", "legendFormat": ".5",
"refId": "C" "refId": "C"
} }
@@ -779,7 +778,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\"}[5m])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\"}[5m])) by (route)", "expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\"}[$__rate_interval])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\"}[$__rate_interval])) by (route) > 0",
"interval": "", "interval": "",
"legendFormat": "{{route}}", "legendFormat": "{{route}}",
"refId": "A" "refId": "A"
@@ -877,18 +876,18 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.99, sum by (le) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\", cluster=~\"$cluster\"})) * 1e3",
"legendFormat": ".99", "legendFormat": ".99",
"refId": "A" "refId": "A"
}, },
{ {
"expr": "histogram_quantile(0.9, sum by (le) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.9, sum by (le) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\", cluster=~\"$cluster\"})) * 1e3",
"hide": false, "hide": false,
"legendFormat": ".9", "legendFormat": ".9",
"refId": "B" "refId": "B"
}, },
{ {
"expr": "histogram_quantile(0.5, sum by (le) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.5, sum by (le) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\", cluster=~\"$cluster\"})) * 1e3",
"hide": false, "hide": false,
"legendFormat": ".5", "legendFormat": ".5",
"refId": "C" "refId": "C"
@@ -985,7 +984,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\", route=\"/logproto.Pusher/Push\"}[5m])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\"}[5m])) by (route)", "expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\", route=\"/logproto.Pusher/Push\"}[$__rate_interval])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", route=\"/logproto.Pusher/Push\"}[$__rate_interval])) by (route) > 0",
"interval": "", "interval": "",
"legendFormat": "{{route}}", "legendFormat": "{{route}}",
"refId": "A" "refId": "A"
@@ -1085,17 +1084,17 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"}))", "expr": "histogram_quantile(0.99, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"}))",
"legendFormat": "{{route}}-.99", "legendFormat": "{{route}}-.99",
"refId": "A" "refId": "A"
}, },
{ {
"expr": "histogram_quantile(0.9, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"}))", "expr": "histogram_quantile(0.9, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"}))",
"legendFormat": "{{route}}-.9", "legendFormat": "{{route}}-.9",
"refId": "B" "refId": "B"
}, },
{ {
"expr": "histogram_quantile(0.5, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"}))", "expr": "histogram_quantile(0.5, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"}))",
"legendFormat": "{{route}}-.5", "legendFormat": "{{route}}-.5",
"refId": "C" "refId": "C"
} }
@@ -1191,17 +1190,17 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.99, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"})) * 1e3",
"legendFormat": ".99-{{route}}", "legendFormat": ".99-{{route}}",
"refId": "A" "refId": "A"
}, },
{ {
"expr": "histogram_quantile(0.9, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.9, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"})) * 1e3",
"legendFormat": ".9-{{route}}", "legendFormat": ".9-{{route}}",
"refId": "B" "refId": "B"
}, },
{ {
"expr": "histogram_quantile(0.5, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.5, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"api_prom_query|api_prom_labels|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_label|loki_api_v1_label_name_values\", cluster=\"$cluster\"})) * 1e3",
"legendFormat": ".5-{{route}}", "legendFormat": ".5-{{route}}",
"refId": "C" "refId": "C"
} }
@@ -1297,7 +1296,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\", status_code!~\"5[0-9]{2}\"}[5m])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\"}[5m])) by (route)", "expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\", status_code!~\"5[0-9]{2}\"}[$__rate_interval])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\"}[$__rate_interval])) by (route) > 0",
"interval": "", "interval": "",
"legendFormat": "{{route}}", "legendFormat": "{{route}}",
"refId": "A" "refId": "A"
@@ -1396,17 +1395,17 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\", cluster=\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.99, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\", cluster=\"$cluster\"})) * 1e3",
"legendFormat": ".99-{{route}}", "legendFormat": ".99-{{route}}",
"refId": "A" "refId": "A"
}, },
{ {
"expr": "histogram_quantile(0.9, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\", cluster=\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.9, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\", cluster=\"$cluster\"})) * 1e3",
"legendFormat": ".9-{{route}}", "legendFormat": ".9-{{route}}",
"refId": "B" "refId": "B"
}, },
{ {
"expr": "histogram_quantile(0.5, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\", cluster=\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.5, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\", cluster=\"$cluster\"})) * 1e3",
"legendFormat": ".5-{{route}}", "legendFormat": ".5-{{route}}",
"refId": "C" "refId": "C"
} }
@@ -1502,7 +1501,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\"}[5m])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\"}[5m])) by (route)", "expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\"}[$__rate_interval])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", route=~\"/logproto.Querier/Query|/logproto.Querier/Label|/logproto.Querier/Series|/logproto.Querier/QuerySample|/logproto.Querier/GetChunkIDs\"}[$__rate_interval])) by (route) > 0",
"interval": "", "interval": "",
"legendFormat": "{{route}}", "legendFormat": "{{route}}",
"refId": "A" "refId": "A"
@@ -2049,7 +2048,7 @@
"panels": [ ], "panels": [ ],
"targets": [ "targets": [
{ {
"expr": "{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\"} | logfmt | level=\"error\"", "expr": "{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\"} |= \"level=error\"",
"refId": "A" "refId": "A"
} }
], ],
@@ -2100,7 +2099,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\"}[5m])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\"}[5m])) by (route)", "expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\", status_code!~\"5[0-9]{2}\"}[$__rate_interval])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-write\"}[$__rate_interval])) by (route) > 0",
"interval": "", "interval": "",
"intervalFactor": 1, "intervalFactor": 1,
"legendFormat": "{{route}}", "legendFormat": "{{route}}",
@@ -2190,9 +2189,9 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(rate(loki_distributor_ingester_append_failures_total{cluster=\"$cluster\", namespace=\"$namespace\"}[5m])) by (ingester)", "expr": "sum(rate(loki_distributor_ingester_append_failures_total{cluster=\"$cluster\", namespace=\"$namespace\"}[5m])) by (pod)",
"intervalFactor": 1, "intervalFactor": 1,
"legendFormat": "{{ingester}}", "legendFormat": "{{pod}}",
"refId": "A" "refId": "A"
} }
], ],
@@ -3256,7 +3255,7 @@
"panels": [ ], "panels": [ ],
"targets": [ "targets": [
{ {
"expr": "{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\"} | logfmt | level=\"error\"", "expr": "{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\"} |= \"level=error\"",
"refId": "A" "refId": "A"
} }
], ],
@@ -3307,7 +3306,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\", status_code!~\"5[0-9]{2}\"}[1m])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\"}[1m])) by (route)", "expr": "sum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\", status_code!~\"5[0-9]{2}\"}[$__rate_interval])) by (route)\n/\nsum(rate(loki_request_duration_seconds_count{cluster=\"$cluster\", namespace=\"$namespace\", job=~\"($namespace)/(loki|enterprise-logs)-read\"}[$__rate_interval])) by (route) > 0",
"interval": "", "interval": "",
"intervalFactor": 1, "intervalFactor": 1,
"legendFormat": "{{route}}", "legendFormat": "{{route}}",

135
charts/meta-monitoring/src/dashboards/loki-reads-resources.json
View File

@@ -53,6 +53,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -72,6 +77,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\", resource=\"cpu\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"})", "expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"})",
"format": "time_series", "format": "time_series",
@@ -142,6 +155,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -161,6 +179,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\", resource=\"memory\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} > 0)", "expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} > 0)",
"format": "time_series", "format": "time_series",
@@ -509,83 +535,6 @@
"show": false "show": false
} }
] ]
},
{
"aliasColors": { },
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "$datasource",
"fill": 1,
"gridPos": { },
"id": 7,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [ ],
"nullPointMode": "null as zero",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [ ],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "loki_boltdb_shipper_query_readiness_duration_seconds{cluster=~\"$cluster\", namespace=~\"$namespace\"}",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "duration",
"legendLink": null,
"step": 10
}
],
"thresholds": [ ],
"timeFrom": null,
"timeShift": null,
"title": "Query Readiness Duration",
"tooltip": {
"shared": true,
"sort": 2,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": [ ]
},
"yaxes": [
{
"format": "s",
"label": null,
"logBase": 1,
"max": null,
"min": 0,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
]
} }
], ],
"repeat": null, "repeat": null,
@@ -607,7 +556,7 @@
"dashes": false, "dashes": false,
"datasource": "$datasource", "datasource": "$datasource",
"fill": 1, "fill": 1,
"id": 8, "id": 7,
"legend": { "legend": {
"avg": false, "avg": false,
"current": false, "current": false,
@@ -626,6 +575,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -645,6 +599,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\", resource=\"cpu\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"})", "expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"})",
"format": "time_series", "format": "time_series",
@@ -695,7 +657,7 @@
"dashes": false, "dashes": false,
"datasource": "$datasource", "datasource": "$datasource",
"fill": 1, "fill": 1,
"id": 9, "id": 8,
"legend": { "legend": {
"avg": false, "avg": false,
"current": false, "current": false,
@@ -714,6 +676,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -733,6 +700,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\", resource=\"memory\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} > 0)", "expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} > 0)",
"format": "time_series", "format": "time_series",
@@ -783,7 +758,7 @@
"dashes": false, "dashes": false,
"datasource": "$datasource", "datasource": "$datasource",
"fill": 1, "fill": 1,
"id": 10, "id": 9,
"legend": { "legend": {
"avg": false, "avg": false,
"current": false, "current": false,

6
charts/meta-monitoring/src/dashboards/loki-reads.json
View File

@@ -142,7 +142,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.99, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\"})) * 1e3",
"format": "time_series", "format": "time_series",
"intervalFactor": 2, "intervalFactor": 2,
"legendFormat": "{{ route }} 99th Percentile", "legendFormat": "{{ route }} 99th Percentile",
@@ -150,7 +150,7 @@
"step": 10 "step": 10
}, },
{ {
"expr": "histogram_quantile(0.50, sum by (le,route) (job_route:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.50, sum by (le,route) (cluster_job_route:loki_request_duration_seconds_bucket:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\"})) * 1e3",
"format": "time_series", "format": "time_series",
"intervalFactor": 2, "intervalFactor": 2,
"legendFormat": "{{ route }} 50th Percentile", "legendFormat": "{{ route }} 50th Percentile",
@@ -158,7 +158,7 @@
"step": 10 "step": 10
}, },
{ {
"expr": "1e3 * sum(job_route:loki_request_duration_seconds_sum:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\", cluster=~\"$cluster\"}) by (route) / sum(job_route:loki_request_duration_seconds_count:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\", cluster=~\"$cluster\"}) by (route) ", "expr": "1e3 * sum(cluster_job_route:loki_request_duration_seconds_sum:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\"}) by (route) / sum(cluster_job_route:loki_request_duration_seconds_count:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-read\", route=~\"loki_api_v1_series|api_prom_series|api_prom_query|api_prom_label|api_prom_label_name_values|loki_api_v1_query|loki_api_v1_query_range|loki_api_v1_labels|loki_api_v1_label_name_values\"}) by (route) ",
"format": "time_series", "format": "time_series",
"intervalFactor": 2, "intervalFactor": 2,
"legendFormat": "{{ route }} Average", "legendFormat": "{{ route }} Average",

28
charts/meta-monitoring/src/dashboards/loki-retention.json
View File

@@ -52,6 +52,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -71,6 +76,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\", resource=\"cpu\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"})", "expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"})",
"format": "time_series", "format": "time_series",
@@ -140,6 +153,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -159,6 +177,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\", resource=\"memory\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} > 0)", "expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-read.*\"} > 0)",
"format": "time_series", "format": "time_series",
@@ -1534,4 +1560,4 @@
"title": "Loki / Retention", "title": "Loki / Retention",
"uid": "retention", "uid": "retention",
"version": 0 "version": 0
} }

26
charts/meta-monitoring/src/dashboards/loki-writes-resources.json
View File

@@ -128,6 +128,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -147,6 +152,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\", resource=\"cpu\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"})", "expr": "min(container_spec_cpu_quota{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} / container_spec_cpu_period{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"})",
"format": "time_series", "format": "time_series",
@@ -217,6 +230,11 @@
"points": false, "points": false,
"renderer": "flot", "renderer": "flot",
"seriesOverrides": [ "seriesOverrides": [
{
"alias": "request",
"color": "#FFC000",
"fill": 0
},
{ {
"alias": "limit", "alias": "limit",
"color": "#E02F44", "color": "#E02F44",
@@ -236,6 +254,14 @@
"legendLink": null, "legendLink": null,
"step": 10 "step": 10
}, },
{
"expr": "min(kube_pod_container_resource_requests{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\", resource=\"memory\"} > 0)",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "request",
"legendLink": null,
"step": 10
},
{ {
"expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} > 0)", "expr": "min(container_spec_memory_limit_bytes{cluster=~\"$cluster\", namespace=~\"$namespace\", container=\"loki\", pod=~\"(loki|enterprise-logs)-write.*\"} > 0)",
"format": "time_series", "format": "time_series",

6
charts/meta-monitoring/src/dashboards/loki-writes.json
View File

@@ -142,7 +142,7 @@
"steppedLine": false, "steppedLine": false,
"targets": [ "targets": [
{ {
"expr": "histogram_quantile(0.99, sum by (le) (job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.99, sum by (le) (cluster_job:loki_request_duration_seconds_bucket:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-write\"})) * 1e3",
"format": "time_series", "format": "time_series",
"intervalFactor": 2, "intervalFactor": 2,
"legendFormat": "99th Percentile", "legendFormat": "99th Percentile",
@@ -150,7 +150,7 @@
"step": 10 "step": 10
}, },
{ {
"expr": "histogram_quantile(0.50, sum by (le) (job:loki_request_duration_seconds_bucket:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})) * 1e3", "expr": "histogram_quantile(0.50, sum by (le) (cluster_job:loki_request_duration_seconds_bucket:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-write\"})) * 1e3",
"format": "time_series", "format": "time_series",
"intervalFactor": 2, "intervalFactor": 2,
"legendFormat": "50th Percentile", "legendFormat": "50th Percentile",
@@ -158,7 +158,7 @@
"step": 10 "step": 10
}, },
{ {
"expr": "1e3 * sum(job:loki_request_duration_seconds_sum:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"}) / sum(job:loki_request_duration_seconds_count:sum_rate{job=~\"($namespace)/(loki|enterprise-logs)-write\", cluster=~\"$cluster\"})", "expr": "1e3 * sum(cluster_job:loki_request_duration_seconds_sum:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-write\"}) / sum(cluster_job:loki_request_duration_seconds_count:sum_rate{cluster=~\"$cluster\", job=~\"($namespace)/(loki|enterprise-logs)-write\"})",
"format": "time_series", "format": "time_series",
"intervalFactor": 2, "intervalFactor": 2,
"legendFormat": "Average", "legendFormat": "Average",

20
charts/meta-monitoring/templates/agent/_helpers-agent.tpl
View File

@@ -8,21 +8,29 @@
{{- define "agent.loki_write_targets" -}} {{- define "agent.loki_write_targets" -}}
{{- $list := list }} {{- $list := list }}
{{- if .Values.local.enabled }} {{- if .Values.local.logs.enabled }}
{{- $list = append $list ("loki.write.local.receiver") }} {{- $list = append $list ("loki.write.local.receiver") }}
{{- end }} {{- end }}
{{- if .Values.cloud.enabled }} {{- if .Values.cloud.logs.enabled }}
{{- $list = append $list ("loki.write.cloud.receiver") }} {{- $list = append $list ("loki.write.cloud.receiver") }}
{{- end }} {{- end }}
{{- join ", " $list }} {{- join ", " $list }}
{{- end }} {{- end }}
{{- define "agent.loki_process_targets" -}}
{{- if empty .Values.logs.piiRegexes }}
{{- include "agent.loki_write_targets" . }}
{{- else }}
{{- printf "loki.process.PII.receiver" }}
{{- end }}
{{- end }}
{{- define "agent.prometheus_write_targets" -}} {{- define "agent.prometheus_write_targets" -}}
{{- $list := list }} {{- $list := list }}
{{- if .Values.local.enabled }} {{- if .Values.local.metrics.enabled }}
{{- $list = append $list ("prometheus.remote_write.local.receiver") }} {{- $list = append $list ("prometheus.remote_write.local.receiver") }}
{{- end }} {{- end }}
{{- if .Values.cloud.enabled }} {{- if .Values.cloud.metrics.enabled }}
{{- $list = append $list ("prometheus.remote_write.cloud.receiver") }} {{- $list = append $list ("prometheus.remote_write.cloud.receiver") }}
{{- end }} {{- end }}
{{- join ", " $list }} {{- join ", " $list }}
@@ -30,10 +38,10 @@
{{- define "agent.tempo_write_targets" -}} {{- define "agent.tempo_write_targets" -}}
{{- $list := list }} {{- $list := list }}
{{- if .Values.local.enabled }} {{- if .Values.local.traces.enabled }}
{{- $list = append $list ("otelcol.exporter.otlp.local.input") }} {{- $list = append $list ("otelcol.exporter.otlp.local.input") }}
{{- end }} {{- end }}
{{- if .Values.cloud.enabled }} {{- if .Values.cloud.traces.enabled }}
{{- $list = append $list ("otelcol.exporter.otlp.cloud.input") }} {{- $list = append $list ("otelcol.exporter.otlp.cloud.input") }}
{{- end }} {{- end }}
{{- join ", " $list }} {{- join ", " $list }}

34
charts/meta-monitoring/templates/agent/config.yaml
View File

@@ -37,16 +37,35 @@ data:
} }
} }
{{- if or .Values.local.logs.enabled .Values.cloud.logs.enabled }}
loki.source.kubernetes "pods" { loki.source.kubernetes "pods" {
targets = discovery.relabel.rename_meta_labels.output targets = discovery.relabel.rename_meta_labels.output
forward_to = [ {{ include "agent.loki_write_targets" . }} ] forward_to = [ {{ include "agent.loki_process_targets" . }} ]
} }
{{- if not (empty .Values.logs.piiRegexes) }}
loki.process "PII" {
forward_to = [ {{ include "agent.loki_write_targets" . }} ]
{{- range .Values.logs.piiRegexes }}
stage.replace {
expression = "{{ .expression }}"
source = "{{ .source }}"
replace = "{{ .replace }}"
}
{{- end }}
}
{{- end }}
{{- end }}
{{- if or .Values.local.metrics.enabled .Values.cloud.metrics.enabled }}
prometheus.scrape "pods" { prometheus.scrape "pods" {
targets = discovery.relabel.rename_meta_labels.output targets = discovery.relabel.rename_meta_labels.output
forward_to = [ {{ include "agent.prometheus_write_targets" . }} ] forward_to = [ {{ include "agent.prometheus_write_targets" . }} ]
} }
{{- end }}
{{- if or .Values.local.traces.enabled .Values.cloud.traces.enabled }}
// Shamelessly copied from https://github.com/grafana/intro-to-mlt/blob/main/agent/config.river // Shamelessly copied from https://github.com/grafana/intro-to-mlt/blob/main/agent/config.river
otelcol.receiver.otlp "otlp_receiver" { otelcol.receiver.otlp "otlp_receiver" {
// We don't technically need this, but it shows how to change listen address and incoming port. // We don't technically need this, but it shows how to change listen address and incoming port.
@@ -75,20 +94,25 @@ data:
traces = [ {{ include "agent.tempo_write_targets" . }} ] traces = [ {{ include "agent.tempo_write_targets" . }} ]
} }
} }
{{- end }}
{{- if .Values.local.enabled }} {{- if .Values.local.logs.enabled }}
loki.write "local" { loki.write "local" {
endpoint { endpoint {
url = "http://loki-gateway.{{- .Release.Namespace -}}.svc.cluster.local:80/loki/api/v1/push" url = "http://loki-gateway.{{- .Release.Namespace -}}.svc.cluster.local:80/loki/api/v1/push"
} }
} }
{{- end }}
{{- if .Values.local.metrics.enabled }}
prometheus.remote_write "local" { prometheus.remote_write "local" {
endpoint { endpoint {
url = "http://{{- .Release.Name -}}-mimir-nginx.{{- .Release.Namespace -}}.svc:80/api/v1/push" url = "http://{{- .Release.Name -}}-mimir-nginx.{{- .Release.Namespace -}}.svc:80/api/v1/push"
} }
} }
{{- end }}
{{- if or .Values.local.traces.enabled .Values.cloud.traces.enabled }}
// The OpenTelemetry exporter exports processed trace spans to another target that is listening for OTLP format traces. // The OpenTelemetry exporter exports processed trace spans to another target that is listening for OTLP format traces.
// A unique label, 'local', is added to uniquely identify this exporter. // A unique label, 'local', is added to uniquely identify this exporter.
otelcol.exporter.otlp "local" { otelcol.exporter.otlp "local" {
@@ -107,7 +131,7 @@ data:
} }
{{- end }} {{- end }}
{{- if .Values.cloud.enabled }} {{- if .Values.cloud.logs.enabled }}
loki.write "cloud" { loki.write "cloud" {
endpoint { endpoint {
url = "{{- .Values.cloud.logs.endpoint -}}/loki/api/v1/push" url = "{{- .Values.cloud.logs.endpoint -}}/loki/api/v1/push"
@@ -118,7 +142,9 @@ data:
} }
} }
} }
{{- end }}
{{- if .Values.cloud.metrics.enabled }}
prometheus.remote_write "cloud" { prometheus.remote_write "cloud" {
endpoint { endpoint {
url = "{{- .Values.cloud.metrics.endpoint -}}/api/prom/push" url = "{{- .Values.cloud.metrics.endpoint -}}/api/prom/push"
@@ -129,7 +155,9 @@ data:
} }
} }
} }
{{- end }}
{{- if .Values.cloud.traces.enabled }}
otelcol.exporter.otlp "cloud" { otelcol.exporter.otlp "cloud" {
client { client {
endpoint = "{{- .Values.cloud.traces.endpoint -}}" endpoint = "{{- .Values.cloud.traces.endpoint -}}"

2
charts/meta-monitoring/templates/grafana/configmap-1.yaml
View File

@@ -1,4 +1,4 @@
{{ if .Values.local.enabled }} {{- if .Values.local.logs.enabled }}
--- ---
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap

2
charts/meta-monitoring/templates/grafana/configmap-2.yaml
View File

@@ -1,4 +1,4 @@
{{ if .Values.local.enabled }} {{- if .Values.local.logs.enabled }}
--- ---
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap

2
charts/meta-monitoring/templates/grafana/dashboard.yaml
View File

@@ -1,4 +1,4 @@
{{ if .Values.local.enabled }} {{- if .Values.local.logs.enabled }}
--- ---
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap

8
charts/meta-monitoring/templates/grafana/datasources.yaml
View File

@@ -1,4 +1,4 @@
{{ if .Values.local.enabled }} {{- if or (or .Values.local.logs.enabled .Values.local.metrics.enabled) .Values.local.traces.enabled }}
--- ---
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
@@ -20,6 +20,7 @@ data:
datasources: datasources:
# <string, required> Sets the name you use to refer to # <string, required> Sets the name you use to refer to
# the data source in panels and queries. # the data source in panels and queries.
{{- if .Values.local.logs.enabled }}
- name: Loki - name: Loki
# <string, required> Sets the data source type. # <string, required> Sets the data source type.
type: loki type: loki
@@ -39,6 +40,8 @@ data:
# <bool> Allows users to edit data sources from the # <bool> Allows users to edit data sources from the
# Grafana UI. # Grafana UI.
editable: true editable: true
{{- end }}
{{- if .Values.local.metrics.enabled }}
- name: Mimir - name: Mimir
# <string, required> Sets the data source type. # <string, required> Sets the data source type.
type: prometheus type: prometheus
@@ -58,6 +61,8 @@ data:
# <bool> Allows users to edit data sources from the # <bool> Allows users to edit data sources from the
# Grafana UI. # Grafana UI.
editable: true editable: true
{{- end }}
{{- if .Values.local.traces.enabled }}
- name: Tempo - name: Tempo
# <string, required> Sets the data source type. # <string, required> Sets the data source type.
type: tempo type: tempo
@@ -77,4 +82,5 @@ data:
# <bool> Allows users to edit data sources from the # <bool> Allows users to edit data sources from the
# Grafana UI. # Grafana UI.
editable: true editable: true
{{- end }}
{{- end }} {{- end }}

20
charts/meta-monitoring/templates/grafana/grafana.yaml
View File

@@ -1,4 +1,4 @@
{{ if .Values.local.enabled }} {{- if or (or .Values.local.logs.enabled .Values.local.metrics.enabled) .Values.local.traces.enabled }}
--- ---
apiVersion: v1 apiVersion: v1
kind: PersistentVolumeClaim kind: PersistentVolumeClaim
@@ -63,30 +63,34 @@ spec:
volumeMounts: volumeMounts:
- mountPath: /var/lib/grafana - mountPath: /var/lib/grafana
name: grafana-pv name: grafana-pv
- mountPath: /etc/grafana/provisioning/datasources
name: loki-datasources-provisioning
{{- if .Values.local.logs.enabled }}
- mountPath: /etc/grafana/provisioning/dashboards
name: loki-dashboards-provisioning
- mountPath: /var/lib/grafana/dashboards/loki-1 - mountPath: /var/lib/grafana/dashboards/loki-1
name: loki-dashboards-1 name: loki-dashboards-1
- mountPath: /var/lib/grafana/dashboards/loki-2 - mountPath: /var/lib/grafana/dashboards/loki-2
name: loki-dashboards-2 name: loki-dashboards-2
- mountPath: /etc/grafana/provisioning/dashboards {{- end }}
name: loki-dashboards-provisioning
- mountPath: /etc/grafana/provisioning/datasources
name: loki-datasources-provisioning
volumes: volumes:
- name: grafana-pv - name: grafana-pv
persistentVolumeClaim: persistentVolumeClaim:
claimName: grafana-pvc claimName: grafana-pvc
- name: loki-dashboards-provisioning
configMap:
name: loki-dashboards-provisioning
- name: loki-datasources-provisioning - name: loki-datasources-provisioning
configMap: configMap:
name: loki-datasources-provisioning name: loki-datasources-provisioning
{{- if .Values.local.logs.enabled }}
- name: loki-dashboards-provisioning
configMap:
name: loki-dashboards-provisioning
- name: loki-dashboards-1 - name: loki-dashboards-1
configMap: configMap:
name: loki-dashboards-1 name: loki-dashboards-1
- name: loki-dashboards-2 - name: loki-dashboards-2
configMap: configMap:
name: loki-dashboards-2 name: loki-dashboards-2
{{- end }}
--- ---
apiVersion: v1 apiVersion: v1

39
charts/meta-monitoring/templates/validate.yaml Normal file
View File

@@ -0,0 +1,39 @@
{{- if not (eq .Values.local.minio.enabled (or .Values.local.traces.enabled (or .Values.local.logs.enabled .Values.local.metrics.enabled))) -}}
{{- fail "minio.enabled should be set to true if any of the the local databases is enabled " -}}
{{- end -}}
{{- if eq .Values.cloud.logs.enabled true -}}
{{- if or (empty .Values.cloud.logs.endpoint) (or (empty .Values.cloud.logs.username) (empty .Values.cloud.logs.password)) -}}
{{- fail "if cloud.logs is enabled then the endpoint, username and password have to be filled in" -}}
{{- end -}}
{{- end -}}
{{- if eq .Values.cloud.metrics.enabled true -}}
{{- if or (empty .Values.cloud.metrics.endpoint) (or (empty .Values.cloud.metrics.username) (empty .Values.cloud.metrics.password)) -}}
{{- fail "if cloud.metrics is enabled then the endpoint, username and password have to be filled in" -}}
{{- end -}}
{{- end -}}
{{- if eq .Values.cloud.traces.enabled true -}}
{{- if or (empty .Values.cloud.traces.endpoint) (or (empty .Values.cloud.traces.username) (empty .Values.cloud.traces.password)) -}}
{{- fail "if cloud.traces is enabled then the endpoint, username and password have to be filled in" -}}
{{- end -}}
{{- end -}}
{{- if eq .Values.cloud.traces.enabled false -}}
{{- if eq .Values.cloud.metrics.enabled false -}}
{{- if eq .Values.cloud.logs.enabled false -}}
{{- if eq .Values.local.traces.enabled false -}}
{{- if eq .Values.local.metrics.enabled false -}}
{{- if eq .Values.local.logs.enabled false -}}
{{- fail "nothing has been enabled" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- if empty .Values.namespacesToMonitor -}}
{{- fail "No namespaces have been specified in namespacesToMonitor" -}}
{{- end -}}

29
charts/meta-monitoring/values.yaml
View File

@@ -2,24 +2,45 @@ namespacesToMonitor:
- loki - loki
- mimir - mimir
- tempo - tempo
clusterName: "observability" # TODO check if this can be derived clusterName: "meta-monitoring" # TODO check if this can be derived
local: local:
enabled: true
cloud:
enabled: false
logs: logs:
enabled: false
metrics:
enabled: false
traces:
enabled: false
minio:
enabled: false # This should be set to true if any of the previous is enabled
cloud:
logs:
enabled: true
endpoint: endpoint:
username: username:
password: password:
metrics: metrics:
enabled: true
endpoint: endpoint:
username: username:
password: password:
traces: traces:
enabled: true
endpoint: endpoint:
username: username:
password: password:
# Adding regexes here will add a stage.replace block. For more information see
# https://grafana.com/docs/agent/latest/flow/reference/components/loki.process/#stagereplace-block
logs:
piiRegexes:
# This example replaces the word after password with *****
# - expression: "password (\\\\S+)"
# source: "" # Empty uses the log message
# replace: "*****""
global: global:
minio: minio:
rootUser: "rootuser" rootUser: "rootuser"