forked from RemoteSync/grafana-meta-monitoring-chart

Michel Hollands cd42da2197 Fix secret name

Signed-off-by: Michel Hollands <michel.hollands@gmail.com>

2024-05-07 14:39:20 +01:00

5.8 KiB

Raw Permalink Blame History

Install this chart

Preparation for Cloud mode (preferred)

Use an existing Grafana Cloud account or setup a new one. Then create an access token:
1. In Grafana go to Administration -> Users and Access -> Cloud access policies.
2. Click Create access policy.
3. Fill in the Display name field and check the Write check box for metrics, logs and traces. Then click Create.
4. On the newly created access policy click Add token.
5. Fill in the Token name field and click Create. Make a copy of the token as it will be used later on.
Create the meta namespace
```
kubectl create namespace meta
```

Create secrets with credentials and the endpoint when sending logs, metrics or traces to Grafana Cloud.

kubectl create secret generic logs -n meta \
 --from-literal=username=<logs username> \
 --from-literal=password=<token> \
 --from-literal=endpoint='https://logs-prod-us-central1.grafana.net/loki/api/v1/push'

kubectl create secret generic metrics -n meta \
 --from-literal=username=<metrics username> \
 --from-literal=password=<token> \
 --from-literal=endpoint='https://prometheus-us-central1.grafana.net/api/prom/push'

kubectl create secret generic traces -n meta \
 --from-literal=username=<OTLP instance ID> \
 --from-literal=password=<token> \
 --from-literal=endpoint='https://otlp-gateway-prod-us-east-0.grafana.net/otlp'

The logs, metrics and traces usernames are the User / Username / Instance IDs of the Loki, Prometheus/Mimir and OpenTelemetry instances in Grafana Cloud. From Home in Grafana click on Stacks. Then go to the Details pages of Loki and Prometheus/Mimir. For OpenTelemetry go to the Configure page.

Create a values.yaml file based on the default one. Fill in the names of the secrets created above as needed. An example minimal values.yaml looks like this:

namespacesToMonitor:
- loki

cloud:
  logs:
    enabled: true
    secret: "logs"
  metrics:
    enabled: true
    secret: "metrics"
  traces:
    enabled: true
    secret: "traces"

Preparation for Local mode

Create the meta namespace
```
kubectl create namespace meta
```

Create a secret named minio with the user and password for the local Minio:

kubectl create secret generic minio -n meta \
 --from-literal=rootPassword=<password> \
 --from-literal=rootUser=<user>

Create a values.yaml file based on the default one. An example minimal values.yaml looks like this:

namespacesToMonitor:
- loki

cloud:
  logs:
    enabled: false
  metrics:
    enabled: false
  traces:
    enabled: false

local:
  grafana:
    enabled:true
  logs:
    enabled: true
  metrics:
    enabled: true
  traces:
    enabled: true
  minio:
    enabled: true

Installing the chart

Add the repo

helm repo add grafana https://grafana.github.io/helm-charts

Fetch the latest charts from the grafana repo
```
helm repo update grafana
```

Install this helm chart

helm install -n meta -f values.yaml meta grafana/meta-monitoring

Upgrade

helm upgrade --install -f values.yaml -n meta meta grafana/meta-monitoring

Delete this chart:
```
helm delete -n meta meta
```

Installing the dashboards and rules on Grafana Cloud

Installing the dashboards on Grafana Cloud

Only the files for the application monitored have to be copied. When monitoring Loki import dashboard files starting with 'loki-'.

For each of the dashboard files in charts/meta-monitoring/src/dashboards folder do the following:

Click on 'Dashboards' in Grafana
Click on the 'New` button and select 'Import'
Drop the dashboard file to the 'Upload dashboard JSON file' drop area
Click 'Import'

Installing the rules on Grafana Cloud

Select the rules files in charts/meta-monitoring/src/rules for the application to monitor. When monitoring Loki use loki-rules.yaml.
Install mimirtool as per the instructions
Create an access policy with Read and Write permission for Rules. Also create a token and record the token.
Get your cloud Prometheus endpoint and Instance ID from the Prometheus page in Stacks.
Use them to load the rules using mimirtool as follows:

mimirtool rules load --address=<your_cloud_prometheus_endpoint> --id=<your_instance_id> --key=<your_cloud_access_policy_token> *.yaml

To check the rules you have uploaded run:

mimirtool rules print --address=<your_cloud_prometheus_endpoint> --id=<your_instance_id> --key=<your_cloud_access_policy_token>

Configure Loki to send traces

In the Loki config enable tracing:
```
loki:
  tracing:
    enabled: true
```
Add the following environment variables to your Loki binaries. When using the Loki Helm chart these can be added using the extraEnv setting for the Loki components.
1. JAEGER_ENDPOINT: http address of the mmc-alloy service installed by the meta-monitoring chart, for example "http://mmc-alloy:14268/api/traces"
2. JAEGER_AGENT_TAGS: extra tags you would like to add to the spans, for example 'cluster="abc",namespace="def"'
3. JAEGER_SAMPLER_TYPE: the sampling strategy, for example to sample all use 'const' with a value of 1 for the next environment variable
4. JAEGER_SAMPLER_PARAM: 1
If Loki is installed in a different namespace you can create an ExternalName service in Kubernetes to point to the mmc-alloy service in the meta monitoring namespace

5.8 KiB Raw Permalink Blame History