Building Devata Overview: Git to Grafana, End to End

Prerequisites: kubernetes-manifest, configmap, pod-volume, sidecar-container, prometheus, promql, grafana, grafana-dashboard-provisioning, gitops, reconciliation, application

This walkthrough traces the actual Devata Overview dashboard from a file in Git to the first screen Grafana serves. It explains why each layer exists, how the YAML and JSON fit together, why /tmp/dashboards/Devata/devata-overview.json is visible to two containers, and what an operator should do with the finished dashboard.

The implementation merged in lab PR #49 and is live on devata. It is not a drawing of a possible architecture. Every runtime check in the final section was repeated against the cluster on 2026-07-19.

The problem the dashboard solves

kube-prometheus-stack already gave devata Prometheus, Grafana, exporters, rules, and detailed dashboards. The missing piece was a deliberate first page.

Without an overview, opening Grafana still leaves the operator with a routing problem: which of dozens of dashboards should be opened first? During a change or incident that wastes the moment when fast orientation matters most.

Devata Overview is a triage dashboard. It starts with six current-state numbers, then shows host and workload trends, then exposes alert, network, and DNS detail. Links at the top lead to specialist dashboards. Its job is to answer:

1. Is something visibly wrong?
2. Which subsystem offers the strongest first lead?
3. Where should the investigation continue?

It does not replace Prometheus alerts, logs, kubectl, or detailed dashboards. It makes the first decision faster and consistent.

Read this dependency chain first

Come back here after those notes and the implementation becomes one connected pipeline rather than a pile of settings.

The complete path

flowchart LR
  Git[lab Git repository\nmanifest and Helm values]
  Argo[Argo CD Application\nkps]
  API[Kubernetes API\nConfigMap]
  Sidecar[dashboard sidecar\nWATCH label]
  Volume[(shared emptyDir\n/tmp/dashboards)]
  Provider[Grafana file provider\n30 second scan]
  Home[Devata Overview\nhome dashboard]
  Prom[Prometheus\ntime series]

  Git -->|desired state| Argo
  Argo -->|reconcile| API
  API -->|watch event and data| Sidecar
  Sidecar -->|write JSON| Volume
  Volume -->|read JSON| Provider
  Provider -->|provision UID| Home
  Home -->|PromQL queries| Prom
  Prom -->|samples| Home

There are two data paths in this chart:

• the provisioning path moves dashboard definition from Git to Grafana;
• the query path moves live metric results from Prometheus to the rendered panels.

Confusing those paths causes bad diagnoses. A broken sidecar can hide the dashboard while Prometheus remains healthy. A broken Prometheus target can show “No data” while the dashboard file is provisioned correctly.

Step 1: Git holds the desired state

The implementation is split across three existing lab files:

This split follows ownership. The dashboard is a Kubernetes resource. Chart-specific Grafana settings are Helm values. The Application says how Argo assembles the chart, values repository, secrets, and resources into one release.

The Application points its fourth source at the resources directory:

- repoURL: https://github.com/PragalvaXFREZ/lab.git
  targetRevision: main
  path: kubernetes/infra/observability/kps/resources

Its policy enables both correction and cleanup:

automated:
  selfHeal: true
  prune: true

selfHeal restores a live object that drifts from Git. prune removes an object that has been removed from the desired source. Together they make Git authoritative in both directions.

Step 2: One YAML object carries one JSON document

The outer document is Kubernetes YAML:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-devata-overview
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_folder: /tmp/dashboards/Devata
data:
  devata-overview.json: |-
    {
      "uid": "devata-overview",
      "title": "Devata Overview",
      "editable": false,
      "panels": []
    }

The real panels array contains all 14 panel definitions. The outer API only sees a ConfigMap whose data value is a string. The sidecar later writes that string as JSON. Grafana then parses the inner document according to Grafana’s dashboard schema.

This nesting creates two validation boundaries:

1. Kubernetes YAML must be a valid ConfigMap;
2. the embedded value must be valid Grafana dashboard JSON.

A server-side dry run can pass while the JSON is malformed. That is why the implementation also extracted and parsed the embedded JSON with jq, inspected its UID and panel count, and executed every PromQL expression against the live datasource before merge.

Step 3: Metadata forms the sidecar contract

The label and annotation are not descriptive decoration.

labels:
  grafana_dashboard: "1"
annotations:
  grafana_folder: /tmp/dashboards/Devata

The rendered sidecar has LABEL=grafana_dashboard and FOLDER_ANNOTATION=grafana_folder. It watches the Kubernetes API across allowed namespaces. The label makes this ConfigMap part of its input set. The annotation overrides the default output folder for this object.

The data key supplies the final filename. The path is derived, not magic:

/tmp/dashboards           sidecar FOLDER
+ /Devata                 ConfigMap grafana_folder annotation
+ /devata-overview.json   ConfigMap data key
= /tmp/dashboards/Devata/devata-overview.json

Step 4: The Helm chart builds the Pod plumbing

The values.yaml file does not spell out the whole Deployment. It supplies values to the kube-prometheus-stack and nested Grafana Helm charts. Those templates render the actual containers, volumes, provider ConfigMap, and grafana.ini.

The relevant input is small:

grafana:
  grafana.ini:
    dashboards:
      default_home_dashboard_path: /tmp/dashboards/Devata/devata-overview.json
  sidecar:
    dashboards:
      folderAnnotation: grafana_folder
      provider:
        allowUiUpdates: false
        foldersFromFilesStructure: true

The rendered Pod contains grafana-sc-dashboard and grafana as separate containers. Both mount the same sc-dashboard-volume emptyDir at /tmp/dashboards. The sidecar writes; Grafana reads.

/tmp here does not mean the workstation’s /tmp, the Talos node’s host /tmp, or some universal directory shared by every Pod. It is a path inside each container. The common volume mounted at that path makes the bytes shared inside this Pod.

The directory is intentionally ephemeral. The file is a projection of declared state and can be rebuilt. Grafana’s database and Prometheus’s TSDB use persistent Longhorn volumes because those hold state whose loss has a different cost.

Step 5: Grafana provisions the file and chooses it as home

The chart renders a file provider that scans /tmp/dashboards every 30 seconds and mirrors its directory structure into Grafana folders. The Devata directory therefore becomes a Devata folder in the UI.

The JSON UID devata-overview gives the dashboard a stable identity. The provider’s allowUiUpdates: false and the dashboard’s editable: false close the loop around Git. A click in the UI cannot silently become the only copy of a production dashboard.

Provisioning and home selection are separate actions:

• the provider makes the dashboard exist in Grafana;
• default_home_dashboard_path makes it the default landing page.

Both rely on the same exact file path. A dashboard can therefore appear in search while still failing to become home if only the second setting is wrong.

Step 6: Panels ask Prometheus precise questions

The dashboard uses the provisioned Prometheus datasource UID prometheus. Grafana sends each panel’s PromQL expression to Prometheus. The 14 expressions were checked for successful live results before merge.

The first row is intentionally cheap to read:

The remaining panels supply trend and localization: per-node CPU and memory, pods by phase, restarts by namespace, alert details, Hubble flow and drops, and CoreDNS non-success responses.

Colors and thresholds help the eye, but panel descriptions and query meaning remain authoritative. A green tile proves only the condition encoded by that query.

Step 7: Argo defends the pipeline

The dashboard was merged through lab PR #49. Argo’s kps Application then read main, created the ConfigMap, and kept the full release Synced and Healthy.

The strongest verification was deliberate drift:

kubectl -n monitoring delete configmap grafana-dashboard-devata-overview

Because Git still declared the object and selfHeal was enabled, Argo recreated the exact ConfigMap in four seconds. The Application briefly reported OutOfSync, then returned to Synced and Healthy. Grafana’s home API continued returning Devata Overview with 14 panels.

This test proved more than “the dashboard currently exists.” It proved the declared path can recover a deleted runtime object without a manual apply.

Verify every boundary yourself

Run these checks from a machine with access to devata.

1. GitOps object

kubectl -n argocd get application kps \
  -o jsonpath='{.status.sync.status}{" / "}{.status.health.status}{"\n"}'

Expected: Synced / Healthy.

2. ConfigMap contract

kubectl -n monitoring get configmap grafana-dashboard-devata-overview \
  -o jsonpath='{.metadata.labels.grafana_dashboard}{" / "}{.metadata.annotations.grafana_folder}{"\n"}'

Expected: 1 / /tmp/dashboards/Devata.

3. Shared file in both containers

pod=$(kubectl -n monitoring get pod \
  -l app.kubernetes.io/name=grafana \
  -o jsonpath='{.items[0].metadata.name}')
 
for container in grafana-sc-dashboard grafana; do
  kubectl -n monitoring exec "$pod" -c "$container" -- \
    wc -c /tmp/dashboards/Devata/devata-overview.json
done

Both containers should report the same byte count.

4. Rendered Grafana setting

kubectl -n monitoring exec "$pod" -c grafana -- \
  grep '^default_home_dashboard_path' /etc/grafana/grafana.ini

Expected path: /tmp/dashboards/Devata/devata-overview.json.

5. Grafana’s own view

Using a safely supplied credential and the reachable Grafana address:

curl -sS -u "$GRAFANA_USER:$GRAFANA_PASSWORD" \
  http://<grafana-address>/api/dashboards/uid/devata-overview \
  | jq '{title: .dashboard.title, uid: .dashboard.uid, panels: (.dashboard.panels | length), folder: .meta.folderTitle, provisioned: .meta.provisioned}'

Expected: title Devata Overview, UID devata-overview, 14 panels, folder Devata, and provisioned: true.

These checks move from desired state to runtime file to application interpretation. Stopping at only one layer would leave the rest assumed.

Operating the dashboard after installation

Routine scan

Open Grafana and read top to bottom. Look for nonzero failures, then compare trend panels with the previous hour or day. You are building a baseline: what CPU, memory, restarts, flow rate, and DNS behavior look like when devata is healthy.

After a change

Note the change time, select a range that includes it, and watch for:

• a target that stopped scraping;
• pods stuck outside Running or Succeeded;
• unavailable replicas or a restart burst;
• node resource pressure;
• a new Hubble drop reason;
• DNS response-code changes.

Do not declare success immediately after a green sync. Give controllers and time-window queries enough time to expose delayed effects.

During an incident

Start with the first abnormal signal, open the matching specialist dashboard, then cross-check Kubernetes events, logs, recent Git changes, and alert details. Keep a timeline. The dashboard narrows the search; it does not replace evidence.

When changing the dashboard

Use Explore or a scratch dashboard to develop the query. Confirm its producer, labels, units, empty-result behavior, and time window. Move the accepted expression and panel definition into the manifest. Validate YAML, parse the embedded JSON, run the query against Prometheus, review the diff, and let Argo reconcile the merged change.

That is the practical end-to-end SRE loop:

observe -> form a question -> explore -> codify -> review -> reconcile -> verify -> operate

Troubleshooting from the symptom inward

The architecture is useful partly because every arrow has an inspectable boundary. Git, Argo, the API object, sidecar logs, shared file, provider, Grafana API, and Prometheus query results can each be tested independently.

What this implementation proves

The finished system proves that devata has a version-controlled, reviewable, self-healing Grafana home dashboard whose 14 panels query live Prometheus data. It also proves that deleting the live dashboard ConfigMap is recoverable through GitOps.

It does not prove that every future outage will be detected, that every panel threshold is perfect, or that the dashboard itself provides notification. Those are operating questions answered over time by real incidents, alert-rule work, and revisions grounded in evidence.

That is what you look after now: whether the first page keeps answering the right questions as the cluster changes.