devata next-goal portfolio

This is roadmap v3. It replaces the 2026-07-02 keystone roadmap because that document still treated GitOps adoption, LAN TLS, and storage durability as future work. Those milestones are complete. The evidence and candidate comparison behind this plan are in next-goals-research.

This roadmap covers genuinely new homelab goals. The existing LLM learning plan, CPU inference issue, Cloudflare Tunnel issue, showcase program, learning gaps in kubernetes-coverage, and recovery gaps in reconstructing-devata retain their own records. They are not relabelled here.

Current decision

Devata should move from “a platform that demonstrates its components” to “a platform that operates one useful stateful service under a complete ownership, delivery, recovery, and external-infrastructure contract.”

That transition has six goals:

1. restore desired-state and operator integrity;
2. make monitoring actionable;
3. declare the external Cloudflare boundary;
4. operate PostgreSQL and a small real consumer;
5. prove a disposable pre-merge delivery path;
6. protect the physical system from power loss when compatible hardware is available.

Distributed tracing is the next observability experiment after a real multi-hop request exists and capacity is proven. It is not another permanent component to install immediately.

Operating rule

Run at most two implementation lanes at once:

• one cluster-mutating lane;
• one off-cluster, documentation, or learning lane.

This is not a ban on parallel goals. It protects the ability to attribute a failure to one change and keeps the small cluster from becoming a software catalogue.

Every goal has a stopping rule. A manifest, a healthy controller, or a dashboard is intermediate evidence. The goal closes only on the named end-to-end proof.

Goal 0: restore the operating contract

Priority: first

Why: live state currently contradicts the repository’s declarative-only contract. Long-lived bootcamp, networking, demo, GPU-proof, and migration objects exist outside Argo. The control-plane Talos configuration uses an old Kubernetes endpoint, local clients are outside safe version skew, and the public snapshot hardcodes storage and GPU posture that disagree with runtime.

Work

1. Inventory every durable object without Argo tracking metadata.
2. Classify each as retire, graduate to Git, or explicitly time-bounded experiment.
3. Resolve stale PVCs and PVs only after proving whether they still carry rollback value.
4. Add the non-secret control-plane machine configuration inputs missing from the repository.
5. Correct the stale API endpoint and pin compatible talosctl and kubectl versions.
6. Reconcile scheduler-visible GPU capacity with the hardware verdict.
7. Derive public storage and accelerator posture from live resources instead of constants.
8. Observe 7 to 14 days of memory range, requests, limits, restarts, and node-failure reserve before adding a heavy controller.

Done when

• every surviving long-lived object has a named owner and source;
• a repeatable ownership report finds no unexplained durable objects;
• talosctl health completes against the current endpoint with a supported client;
• the generated snapshot matches scheduler and storage reality;
• the capacity record states how much memory remains available when either worker is lost.

Stop rule

Do not replace the current platform merely because one point sample is high. If the 7 to 14 day record proves that the existing stack prevents a bounded workload from fitting with worker-loss reserve, resolve lab issue #50 using measured consumers and removal tradeoffs.

Goal 1: build an actionable operations loop

Priority: immediately after Goal 0 starts producing a clean signal

Why: Prometheus can detect failures, but Alertmanager is disabled. Argo CD, Longhorn, and Velero carry the desired-state and recovery story but do not yet have complete actionable monitoring.

Work

1. Choose one external alert receiver that still works when devata is unreachable.
2. Inventory current firing alerts and classify each as defect, missing dependency, expected invariant, or noise.
3. Enable Alertmanager with encrypted credentials and a narrow first route.
4. Add reconciliation, volume-replica, backup-location, and backup-freshness signals for Argo CD, Longhorn, and Velero.
5. Attach a runbook and owner to each page-worthy alert.
6. Inject one safe failure and trace detection, routing, receipt, and recovery.

Done when

A controlled failure produces one useful notification outside the cluster, links to a runnable response, and resolves after the end-to-end recovery transaction passes. Known firing alerts are fixed or deliberately silenced with a recorded reason.

Stop rule

If a receiver cannot distinguish a real action from routine controller noise, reduce the alert surface before adding another integration. Notification volume is not operational maturity.

Goal 2: make the external trust boundary declarative

Priority: parallel off-cluster lane

Why: Kubernetes state is reconciled from Git, while Cloudflare DNS, Pages, Access, and future Tunnel state remain dashboard-managed. The external side of the trust boundary needs its own reviewable source and recovery contract.

Work

1. Inventory current Cloudflare resources, IDs, dependencies, and rollback exports.
2. Choose a repository boundary and encrypted OpenTofu state location outside devata.
3. Reproduce existing configuration and import it without recreation.
4. Add plan validation in CI and require a reviewed manual apply for the first phase.
5. Assign static Cloudflare resources to OpenTofu.
6. When route count justifies a controller, assign selected Gateway API-derived records to ExternalDNS with a separate credential, narrow domain filter, TXT owner ID, and dry-run rehearsal.
7. Prohibit OpenTofu and ExternalDNS from owning the same DNS record.

Done when

The imported configuration produces an empty plan, encrypted state can be restored in a clean environment, and one disposable DNS record completes a reviewed create and delete round trip. The ownership map proves that each external resource has exactly one controller.

Stop rule

Leave a critical resource manual with a recorded exception if importing it would replace or silently normalize live behavior. Coverage percentage is less important than a no-surprise plan.

Goal 3: operate PostgreSQL with a useful canary

Priority: primary new workload after Goal 0

Why: storage capability is proven, but devata does not yet operate application data. CloudNativePG plus Miniflux creates a small real workload and forces decisions about database replication, failover, schema upgrades, WAL archive, point-in-time recovery, credentials, and observability.

Design decisions before install

• Use two worker failure domains honestly. Three database Pods do not create a third physical domain.
• Decide whether the experiment tests asynchronous availability or synchronous RPO zero.
• Avoid blindly combining two PostgreSQL copies with two Longhorn replicas per copy.
• Use the current Barman Cloud plugin path and a separate capped R2 location.
• Keep database recovery material and Sealed Secrets recovery material outside the database namespace.

Work

1. Record the replication, volume, RPO, RTO, and object-storage model.
2. Install CloudNativePG through Argo with measured requests, limits, and worker placement.
3. Load a deterministic dataset and capture committed-row checkpoints.
4. Fail the primary and measure promotion time and row loss.
5. Restore to a named point into a new cluster and verify checksums.
6. Deploy Miniflux as the first consumer and use it for four weeks.
7. If a real scanning habit exists, evaluate Paperless-ngx as the next consumer with originals, OCR output, database, and exporter/importer tested as separate recovery surfaces.

Done when

A database primary failure and a point-in-time restore both have measured results, restored checksums match, and Miniflux remains a useful service through a planned switchover. “The operator recreated the Pod” does not close the goal.

Stop rule

Remove the platform if its steady-state memory violates worker-loss reserve. Retire Miniflux if it is unused after four weeks. Do not promote Paperless without documents that will actually be ingested and retrieved.

Goal 4: prove a bounded delivery path

Priority: parallel with Goal 3 after Goal 0

Why: Argo proves post-merge reconciliation, but the cluster has no reusable pre-merge workload contract. A small golden path provides stronger platform-engineering evidence than installing a portal for one user.

Work

1. Select one small first-party workload whose delivery contract will be reused.
2. Test and build by commit SHA, publish by immutable digest, and attach provenance and an SBOM.
3. Use an administrator-owned ApplicationSet Pull Request generator to create only quota-limited namespaced resources for labelled pull requests.
4. Keep Argo project, destination cluster, and cluster-scoped permissions out of pull-request templating.
5. Run a smoke transaction against the exact digest.
6. Delete the Application, namespace, and namespaced storage when the pull request closes.
7. Add Argo Rollouts only after a stable first-party service has a meaningful Prometheus error metric.

Done when

Opening a labelled pull request creates one isolated preview from the expected digest, an out-of-bound manifest is rejected, the smoke transaction passes, and closing the pull request removes the full preview. The later progressive-delivery rung closes only when a deliberately bad canary aborts automatically while the stable Service remains healthy.

Stop rule

If no second workload can reuse the path, document the experiment and remove its resident machinery. Do not install Backstage or Crossplane until there is an actual internal consumer and repeated provisioning problem.

Goal 5: engineer power-loss recovery

Priority: conditional physical lane

Why: devata is intended to run around the clock, and it now carries replicated volumes. Power loss is a more direct threat to that investment than lack of another controller.

Work

1. Measure the cluster’s wall load and required graceful-shutdown runtime.
2. Select a compatible UPS with enough reserve for the slowest storage detach and flush path.
3. Run Network UPS Tools control outside devata.
4. Define worker-first and single-control-plane-last shutdown order.
5. Alert on battery, communications, and failed shutdown execution.
6. Run a controlled forced-shutdown and mains-return exercise.

Done when

The cluster shuts down within the battery budget, returns after mains recovery, Longhorn is healthy, and both a database checksum and the synthetic Promtail-to-LogQL transaction pass.

Stop rule

Until compatible hardware and an off-cluster controller exist, this remains a purchase and design gate. An in-cluster-only shutdown script is not an acceptable safety system.

Next observability experiment: traces

Metrics and logs exist; request causality does not. After Goal 3 or 4 creates a multi-hop path and the capacity gate passes, run a bounded OpenTelemetry gateway Collector plus monolithic Tempo experiment. Use short retention, explicit sampling, and content filtering. Do not deploy Tempo microservices mode or the full OpenTelemetry demo.

It closes when a synthetic request produces a trace across the real request and database path, Grafana links it to logs and metrics, and an injected latency or error is located from that trace. Remove Tempo if its retained value does not justify measured memory.

Sequence

flowchart TD
  integrity[Goal 0: operating contract]
  ops[Goal 1: actionable operations]
  external[Goal 2: external declarative state]
  database[Goal 3: PostgreSQL and Miniflux]
  delivery[Goal 4: bounded delivery path]
  power[Goal 5: power-loss recovery]
  traces[Bounded trace experiment]

  integrity --> ops
  integrity --> database
  integrity --> delivery
  external -. parallel off-cluster lane .-> database
  database --> traces
  delivery --> traces
  power -. hardware dependent .-> integrity

First ten actions

1. Export the live non-Argo object inventory without deleting anything.
2. Classify each object as retire, graduate, or temporary.
3. Correct the Talos endpoint and local client-version contract.
4. Replace hardcoded public storage and GPU fields with derived state.
5. Start the 7 to 14 day capacity and restart observation.
6. Dispose the current firing alerts and select one external receiver.
7. Inventory Cloudflare resources and write the controller-ownership map.
8. Design the PostgreSQL replication and storage model on paper before installing CloudNativePG.
9. Choose the first-party workload for the delivery-path experiment.
10. Inventory UPS load and compatibility without buying hardware until runtime needs are known.

Review points

• After Goal 0: decide whether issue #50 is actually triggered by measured memory, not impressions.
• After the database failover: decide whether Miniflux earns four weeks of operation.
• After the Miniflux trial: choose Paperless, another real consumer, or database-lab retirement.
• After one preview environment: decide whether the pattern has a second consumer before keeping it live.
• Before traces: prove enough worker-loss reserve for a bounded Tempo deployment.
• Before UPS automation: prove the controller is outside the failure domain it shuts down.