Research for devata’s next goals

This is the evidence layer for roadmap. It asks what devata should do next after GitOps, LAN TLS, replicated storage, off-cluster restore, and the public snapshot are working. It deliberately does not repackage an existing backlog item as a new goal.

The conclusion is a portfolio: repair the operating contract first, then grow the lab through one real stateful service, declarative external infrastructure, a bounded delivery system, and physical resilience. The cluster does not have the memory for every interesting controller to become a permanent resident.

Evidence boundary

The live cluster and lab default branch were inspected read-only on 2026-07-26. The observation found:

• three Ready Talos nodes and 17 Synced/Healthy Argo CD Applications;
• Cilium, Gateway API, cert-manager, Sealed Secrets, Prometheus, Grafana, Loki, Longhorn, and Velero live;
• only two Git-managed workload Applications, hello and snapshot-publisher;
• CPU use between 4 and 11 percent, but memory use at 89 percent on the control plane, 68 percent on the 16-core worker, and 49 percent on the OptiPlex worker;
• multiple long-lived objects outside Argo ownership, including bootcamp, networking, GPU proof, and storage-migration remnants;
• Alertmanager disabled while Prometheus had actionable and noisy alerts firing;
• a stale control-plane Kubernetes endpoint in live Talos configuration and unsupported local client skew;
• scheduler-visible GPU capacity that the device cannot actually serve;
• an available Velero backup location but no current Backup, Restore, or Schedule objects.

The memory numbers are one point-in-time sample, not a sizing study. They are strong enough to block an unmeasured software expansion, but not strong enough to justify replacing the stack. A 7 to 14 day range, requests and limits, restart history, and failure-domain reserve must decide capacity.

Explicit exclusions

The following work is already completed or already recorded elsewhere and is not claimed as new research:

• Argo CD adoption, Sealed Secrets, Gateway API and LAN TLS, Longhorn, Velero capability, the Grafana home dashboard, and the public snapshot;
• CPU llama.cpp serving, Cloudflare Tunnel, SLOs, Gatus, kromgo, Ask-devata, and the showcase program;
• NetworkPolicy, Tetragon, admission and supply-chain study, Chaos Mesh, a custom operator, reconstruction, Talos upgrade practice, and HA control-plane evaluation;
• the generic anchor-workload gap and its existing private research/read-later front-runner;
• Promtail-to-Alloy migration, remote access, the dead-GPU experiment, and documentation salvage.

The existing work remains valid where it is current. The new portfolio fills gaps it did not cover.

How candidates were judged

Each candidate was tested against six questions:

1. Does it produce utility or evidence that would be missed if removed?
2. Does it teach a portable operating problem rather than only a product UI?
3. Is it absent from the existing roadmap, journey, issue backlog, and learning-gap records?
4. Does it fit two worker failure domains and the observed memory envelope?
5. Can success be proven by a transaction, failure, or recovery test?
6. Is the continuing maintenance cost justified after the initial learning session?

Popularity was used for discovery, not selection. A 2025 community survey ranked Jellyfin, Immich, Home Assistant, Vaultwarden, Paperless-ngx, and Nextcloud among its most valued services. The same survey found fun, privacy, independence, data control, and learning to be the main motivations. That explains what people run, but not what belongs on devata. Availability needs, data risk, resource demand, and failure-domain fit decide that here.

Immediate integrity gaps

These are not glamorous, but adding more software before resolving them would deepen contradictions.

Desired-state ownership is incomplete

The repository says hand-applied state is drift, while live long-lived resources remain outside Argo. Some map to archive/ or lab-experiments/; others have no current repository owner. The next expansion must not normalize a cluster where nobody can tell whether an object should be recreated after loss.

The right experiment is an ownership sweep:

• inventory non-Argo resources and classify each as retire, graduate, or explicitly temporary;
• remove stale storage only after resolving its rollback value and exact backing volume;
• add a repeatable report for long-lived resources without Argo tracking metadata;
• prove the surviving cluster can be explained entirely by Git, Talos configuration, and named temporary experiments.

Monitoring is visible but not actionable

Grafana and Prometheus can show failures, but Alertmanager is disabled. An operations loop needs one external receiver, a noise budget, and a controlled alert-delivery proof. Installing another dashboard would not close this gap.

Argo CD, Longhorn, and Velero also lack the monitoring coverage needed to answer whether reconciliation, replication, and backup freshness are healthy. These are the three controllers that now carry the recovery story.

Talos and published truth disagree with reality

The live control-plane configuration still references an old API endpoint, local clients are outside safe version skew, and the public snapshot hardcodes replicated storage and GPU posture. This creates a dangerous pattern: a healthy Kubernetes API can hide an invalid operator recovery path, while a generated public surface can repeat a constant instead of observing the cluster.

The repair is complete only when talosctl health uses the current endpoint, supported client versions are pinned, GPU schedulability matches real capability, and snapshot fields are derived from live resources.

What people run, filtered for devata

Why Miniflux before Paperless

Miniflux is intentionally boring. It requires PostgreSQL, produces real writes, and can be abandoned without putting family photos, passwords, or financial records at risk. It lets the database platform prove failover, upgrade, backup, and point-in-time recovery before the data becomes emotionally expensive.

Paperless is the stronger long-term household workload if there is a real scanning habit. Its exporter and importer create a second recovery path, and OCR exercises burst CPU while the originals and database exercise different durability contracts. It should graduate only after a time-boxed utility trial, not because it is popular.

Candidate goal families

1. Operate PostgreSQL, then give it a boring consumer

Decision: adopt as a primary goal after the integrity gate.

CloudNativePG turns PostgreSQL lifecycle into declarative Kubernetes state: instances, replication, failover, backup, recovery, and upgrades are controller responsibilities that can be observed and tested. Miniflux is a small application that requires PostgreSQL, so the platform immediately has a consumer.

Important design constraints:

• two worker nodes are only two physical failure domains; three database Pods do not create a third one;
• two PostgreSQL copies on two-replica Longhorn volumes create four data copies and extra network writes;
• choose database replication, volume replication, and R2 recovery as one deliberate durability model rather than stacking every default;
• use the current Barman Cloud plugin path and a separate, capped R2 bucket or prefix;
• declare whether the experiment accepts asynchronous committed-row loss or is testing synchronous RPO zero;
• keep credentials in Sealed Secrets and database recovery material outside the database namespace.

Completion proof: load a deterministic dataset, fail the primary, measure recovery time and committed-row loss, then restore to a named time into a new cluster and verify row checksums. Miniflux must remain usable through a planned switchover. A successful Pod restart is not enough.

Stop rule: if steady-state memory plus the loss of one worker violates the capacity reserve, remove the platform rather than weakening the reserve. If Miniflux is unused after a four-week trial, keep the database lab only if its recurring recovery exercises still justify the toil.

2. Make the external trust boundary declarative

Decision: adopt as a parallel, mostly off-cluster goal.

Kubernetes is declarative, but Cloudflare DNS, Pages, Access, and future Tunnel state are still primarily dashboard-managed. OpenTofu can import existing Cloudflare resources without recreating them, then make an empty plan the external equivalent of an empty Argo diff.

The ownership boundary must be explicit:

• OpenTofu owns zones, static records, Access, Pages, and future Tunnel configuration;
• ExternalDNS may later own only records derived from selected Gateway API Routes;
• the two systems never manage the same record;
• state is encrypted, recoverable, and stored outside the cluster it helps expose;
• imports start read-only and no apply is allowed until the plan contains zero replacement actions.

ExternalDNS is worth adding when a third route makes manual DNS lifecycle repetitive. Its Gateway sources can derive names from HTTPRoute hostnames and targets from the accepted Gateway address. Use a dedicated Cloudflare token, a narrow domain filter, TXT ownership, and dry-run first. Do not reuse cert-manager’s credential just because the permissions look similar.

Completion proof: import existing resources with zero replacements, reach an empty tofu plan, add and remove one disposable static record through reviewed plans, restore encrypted state in a clean environment, then demonstrate that no controller overlaps another controller’s DNS ownership.

Stop rule: if the provider cannot represent an existing critical resource without replacement or hidden defaults, leave that resource dashboard-managed and record the exception. Declarative coverage is not worth a surprise cutover.

3. Build one bounded delivery path

Decision: adopt after the integrity gate, in parallel with the database lane.

The lab proves reconciliation after merge but not the application path before merge. A small first-party service can provide a golden path without installing Backstage or Crossplane:

1. GitHub Actions tests and builds by commit SHA.
2. The image is pushed by immutable digest with provenance and an SBOM.
3. An Argo CD ApplicationSet Pull Request generator creates a quota-limited namespace for labelled pull requests.
4. A smoke transaction runs against the exact digest.
5. Closing the pull request deletes the Application and all namespaced resources.

ApplicationSet security is part of the experiment. Pull-request data must not template the Argo project, destination cluster, or cluster-scoped resources. Only administrators create ApplicationSets. A GitHub App or token receives read-only pull-request metadata access.

Progressive delivery is a later rung, not a second controller to install immediately. Once a first-party request-serving workload has a stable error metric, Argo Rollouts can abort a deliberately bad canary from a Prometheus query. Preview environments answer “does this candidate work?”; canaries answer “should this release continue in production?”

Completion proof: opening a labelled pull request creates one isolated preview from the expected digest, the smoke test passes, an out-of-scope manifest is rejected, and closing the pull request removes the namespace and storage. A later bad canary must abort automatically without changing the stable Service.

Stop rule: if the only consumer is a toy that no future workload will reuse, retain the documented pattern and remove the always-running preview machinery.

4. Protect the always-on physical system from power loss

Decision: adopt conditionally when compatible UPS hardware is available.

Devata is now described as always on. Longhorn and application databases make abrupt loss more expensive than when the cluster held only disposable workloads. The next physical investment should therefore be a UPS and a tested shutdown path, not more compute.

Network UPS Tools should run its primary control outside devata. An in-cluster controller cannot be the only component responsible for shutting down the cluster that hosts it. The shutdown order is workers first and the single control plane last, with enough battery reserve for the slowest observed flush and detach path.

Completion proof: a staged low-battery or forced-shutdown signal invokes the documented order, nodes power off within the battery budget, restored mains brings the platform back, Longhorn returns healthy, and both a database checksum and the Promtail-to-LogQL transaction pass.

Stop rule: do not automate power control from an unverified USB device, smart plug, or in-cluster-only script. Until compatible hardware exists, keep this as a purchase and design gate, not a software project.

5. Add traces only when there is a request path worth tracing

Decision: keep as the next observability experiment, gated by capacity and a multi-hop workload.

Metrics and logs already describe components, but they do not show causality across one request. An OpenTelemetry gateway Collector plus monolithic Tempo would close that gap. The official Tempo monolithic mode is intended for small, demo, and test deployments; microservices mode and the full OpenTelemetry demo are too large for this cluster.

The experiment should instrument one actual path, such as HTTP to application to PostgreSQL. Short retention, content filtering, and explicit sampling are mandatory. Document text, feed content, prompts, credentials, and query values must not become span attributes by default.

Completion proof: one synthetic request produces a trace across the real services, Grafana can move from the trace to the associated logs and metrics, and an injected latency or error is located from the trace. The Collector failure mode and dropped-span behavior must be observed.

Stop rule: run this as a bounded deployment first. If the retained value does not justify its measured memory, remove Tempo and keep the instrumentation contract for a future larger cluster.

Candidates not promoted to primary goals

Central identity

Authentik or another OIDC provider becomes valuable when at least three real applications need the same human identity and group policy. Today it would add a database-backed critical dependency for one operator. Revisit after Miniflux or Paperless, Grafana, and one more application exist. Preserve local break-glass accounts before centralizing login.

Policy engine and image enforcement

Security policy is already a recorded learning gap, so it is not presented as a new goal. The concrete first implementation should sign the existing first-party snapshot-publisher image through GitHub OIDC, emit provenance, audit current violations, and enforce only Pragalva-owned images. Denying all unsigned third-party images would create an exception catalogue rather than a trust model.

Retrieval over the vault

A measured retrieval service using PostgreSQL and pgvector could become a useful bridge after the recorded LLM serving goal. It is not the next platform goal because both its database and inference prerequisites are unfinished. When revisited, a fixed question set, Recall at 5, reciprocal rank, latency, citations, and a reproducible re-index are the proof, not a chat UI.

Household DNS filtering

AdGuard Home or Pi-hole is useful only if two resolvers survive independently. The sole household resolver must not run inside devata because cluster recovery itself needs DNS. This belongs on a router, spare host, or a mixed external and in-cluster pair.

Backstage, Crossplane, a service mesh, and an event broker

These solve absent scale:

• Backstage and Crossplane add a platform catalogue and API for internal customers who do not exist yet;
• Cilium, Gateway API, and Hubble already cover the current network problem, so a service mesh has no concrete traffic or identity requirement;
• Kafka, Redpanda, or NATS has no producer-consumer contract to serve.

Install one only when an actual workload supplies the requirement and acceptance test.

Recommended portfolio and sequence

flowchart TD
  gate[Integrity and capacity gate]
  ops[Actionable operations loop]
  ext[Declarative Cloudflare boundary]
  db[CloudNativePG recovery lab]
  app[Miniflux canary]
  delivery[Disposable delivery path]
  trace[Bounded trace experiment]
  power[UPS and power-loss recovery]

  gate --> ops
  gate --> db
  db --> app
  gate --> delivery
  delivery --> trace
  app --> trace
  ext -. parallel, off cluster .-> db
  power -. hardware dependent .-> gate

The concurrency rule is two active implementation lanes at most: one cluster-mutating lane and one off-cluster or learning lane. Multiple goals exist to preserve choice and dependencies, not to run six controllers at once.

Source notes

Community and workload discovery:

• Self-Hosted Community Survey 2025
• Most valuable applications in that survey
• Miniflux requirements
• Paperless-ngx setup
• Paperless-ngx administration and recovery
• Immich requirements

Database and recovery:

• CloudNativePG failover
• CloudNativePG recovery and point-in-time recovery

External configuration and DNS ownership:

• Cloudflare resource import
• OpenTofu state and plan encryption
• ExternalDNS Gateway API sources
• ExternalDNS Cloudflare provider
• ExternalDNS TXT ownership registry

Delivery and policy:

• Argo CD ApplicationSet pull-request generator
• Argo Rollouts analysis and progressive delivery
• Kubernetes production considerations
• Kubernetes cluster security controls
• Kyverno policy reports
• Kyverno image validation

Observability and physical resilience:

• OpenTelemetry gateway Collector pattern
• Tempo monolithic deployment
• Network UPS Tools upsmon
• Network UPS Tools upssched