Kona
Back to the site

What's under the hood

If you run a rescue on your own server and are thinking about moving to Kona, this page is for you. It's a plain accounting of the infrastructure the site runs on, the redundancy layers that sit behind it, and the gaps that are still being worked on. No marketing numbers. Written for people who already know what half of these words mean.

Compute

The site runs on a 3-node Kubernetes cluster (k3s) with embedded etcd. All three nodes are control-plane plus etcd members, so losing any one of them leaves the cluster writable and the apps running. They live on three separate physical hosts, each a Proxmox hypervisor running its node as a virtual machine, and one of those hypervisors passes a GPU straight through to its node: the card dedicated to the AI-assisted features. Each node is sized to run the full public and hub workload on its own, so a failover from any one of the others doesn't degrade service beyond brief rescheduling latency.

Postgres runs under the CloudNativePG operator, one independent cluster per rescue tenant. Each cluster is three instances (a primary and two streaming standbys) placed on different nodes, each on its own physical host, on local NVMe, with synchronous replication: an acknowledged write is on at least two nodes before the client sees the commit, so a primary failure loses no committed data rather than merely failing over fast. If the node holding a primary dies or is taken down for maintenance, the operator promotes the standby and the app reconnects through the operator-managed service; no human is in that loop. Reads and writes are split at the application layer: a request's plain reads go to the cluster's read-only endpoint, which stays up through a switchover, while writes and any read that follows a write in the same request stay pinned to the primary. Web pods float by default, and the ingress proxy runs as two replicas on two different nodes, deliberately kept off the smallest node so a scheduler decision can never put both where the capacity isn't.

Local AI

Kona's AI-assisted features (drafting adoption bios from photos and foster notes, writing alt-text photo descriptions, reviewing applicant free-text answers and surfacing red flags, drafting interview questions tailored to a specific applicant and a specific dog, proposing social-media post copy, extracting structured fields from uploaded vet documents, and the in-app chat assistant that can reason against an individual dog's record) all run on local GPUs on the same rack as the rest of the stack. No adopter data, no applicant answers, and no dog photos are sent to a cloud AI API. That's an intentional constraint. Model weights and the inference runtime are both on-site.

Inference is routed through a priority plus queue-aware dispatcher across a small GPU fleet:

Scaling this layer is a promotion path, not a rewrite. When the dedicated 5090 is no longer enough on its own, the next step is to dedicate the RTX PRO 6000 Blackwell on a temporary basis while longer-term capacity decisions get made. The scheduler, the failover, and the health checks that drive all of this are already in place; growing the dedicated tier is a configuration change, not a project.

Kona also runs a self-hosted image-embedding model (CLIP-class) on the same GPUs, turning each approved dog photo into a vector stored in Postgres through the pgvector extension: visual dog matching done entirely on-site, with no third-party vision API in the path. Two features build on it: a vet-initiated care-team flow, where a treating vet can find a specific rescue dog (by photo, among other non-photo signals) and open a consent-gated link to its record and a sanctioned line of communication; and a lost-and-found search that matches an uploaded photo against the current dogs. The matching is always advisory and score-graded. It surfaces likely candidates for a human to confirm, and never asserts that two dogs are the same animal.

Storage

Postgres data lives on local NVMe on the nodes that host each cluster's instances, not on NFS. The commit-path fsync latency matters more than the bytes-per-second number on a rescue workload, and pulling NFS out of that path cut per-commit fsync from roughly 12 ms to roughly 3 ms. Consumer NVMe plus full-disk encryption lands in single-digit milliseconds, and it varies by a few milliseconds from node to node, so there's no deeper well to drill without enterprise drives with power-loss protection. It hasn't been the bottleneck in practice.

Shared state that needs to outlive a single node (media uploads, backup artifacts, the in-cluster object store) lives on a ZFS pool on a separate NAS, served over NFS. The pool is three-way RAIDZ2, so any two drives in a vdev can fail before data is at risk, and the dataset is encrypted at rest. Kubernetes consumes it via static PersistentVolumes plus the nfs-csi driver.

The read path for uploaded images gets two layers of caching stacked on top of that NFS backing. Every cluster node runs the Linux kernel's FS-Cache layer with cachefilesd, mounting the upload share with fsc,nosharecache so that the first read of a given file on a node pays the NFS round-trip and every subsequent read on the same node serves from that node's local NVMe transparently. Web pods spread across nodes via soft anti-affinity so each tenant's upload mount gets its own per-node cache. In front of that, Cloudflare holds /uploads/* at the edge for thirty days under a cache rule that respects the origin's Cache-Control header, which means the common case for public photo traffic never reaches the origin at all. When the edge does miss and the node cache is cold, the NFS pull is what the visitor waits on; every layer behind that is measured and bounded.

Network

The public internet only reaches the cluster through a Cloudflare Tunnel. There are no open inbound ports on the edge router, and the tunnel daemon runs as two pod replicas in the cluster so that losing one node doesn't drop the site. DNS, TLS termination, WAF, bot filtering, and rate-limiting all happen at Cloudflare before a request is ever brokered into the cluster.

Internally, the network is Ubiquiti gear. The switching core used to be a single aggregation switch that every uplink in the rack funneled through, which meant one firmware update was a rack-wide blip for the length of a reboot. That is now two independent aggregation switches with a two-member 10G link aggregation between them. Every hypervisor has two 10G SFP+ uplinks in an active-backup bond, one leg on each core side, so a switch failure, a dead optic, or a pulled cable moves it to its other leg in under a second at full speed instead of dropping it to a degraded path. The cluster nodes ride that bond through their hypervisor's bridge on a single virtual NIC, so the failover happens beneath them. The bonds are active-backup rather than LACP on purpose: aggregating across two switches would require the switches to share a control plane (MLAG or a stack), and a shared control plane turns one firmware update into a reboot of both halves, which is exactly the failure this design exists to remove. Per-host bandwidth was traded for core independence.

The two gateways run as a shadow-failover pair on two independent WAN circuits (a symmetric fiber line as primary and a 5G connection as failover), split across the cores the same way: each gateway's WAN and LAN legs land on a different aggregation switch, and the fiber hand-off feeds both cores independently, so no single switch owns the path to the internet. A gateway that loses power or dies outright is covered automatically: the standby takes the WAN address with it, so the public address survives and nothing external sees a renumbering.

One failure in this design still needs a human, and it is worth naming precisely. The pair's failover trigger is a direct heartbeat cable between the two gateways, not a check on whether either one can still reach the network. That is the vendor's behavior, not a setting: pulling power fails over, pulling an uplink does not. So if the core switch that the active gateway sits on dies, that gateway is alive and isolated, the heartbeat stays up, and the standby correctly stays passive. The LAN keeps running on the surviving core and the cluster stays whole; internet access waits on someone promoting the standby by hand. Losing the other core, the one carrying the passive unit, is a non-event. Automating the isolated-gateway case is possible (cut its power and the heartbeat drops on its own), and it is deliberately not built yet, because the trigger for that would have to be free of false positives and a wrong one takes the site down rather than saving it.

Power

The rack is on a UPS, which gives enough runtime to ride out the shorter outages that make up the great majority of utility interruptions. Longer outages are covered by a backup generator. The next incremental step is a second UPS on a separate circuit with an automatic transfer switch in front of the load, so that a UPS electronics failure is no longer a single event that takes the rack down. That hardware is in the queue; the circuit changes to accept it are the main dependency. The honest gap on this side: the utility-pull drill that proves the generator transfer under load has not been run, and transfer gear is exactly the kind of thing that fails silently for months. It belongs on the same drill list as everything else.

Backups and recovery

Four independent backup layers overlap.

The in-cluster object store that holds the first and third layers (the WAL archive and the Velero snapshots) is itself replicated, server-side, to a second independent object store on a different physical host, so losing the host that runs the primary store doesn't take the backups with it. That replica is monitored for replication failures, not just for being reachable. A replica that is up but failing to receive copies pages, not only one that's outright down.

On the NAS itself, the underlying ZFS pool also takes periodic snapshots of the datasets that hold the dumps, the MinIO bucket, and the upload shares: cheap copy-on-write rollback even if a backup job wrote a bad archive. Every one of these restore paths has been exercised on this cluster, not left as a runbook assumption: a point-in-time recovery from the WAL archive, a standalone pg_restore from a nightly dump into an empty database, a pull of the SQL dumps back from the separate off-cluster restic server, and a full namespace rebuild from Velero with a pg_restore on top. In each case the data comes back and the pods serve traffic.

What's still missing is distance. Every copy above lives on the premises. True off-site replication, a copy that survives the building, is the next addition on this side. After that, a small off-site standby server will come online that can serve the core rescue workflows (applications, medical records, photo uploads) at reduced capacity if the primary site is offline. It won't run the AI pieces, so AI-assisted features fall back to a simpler path there.

Security

Storage is encrypted at rest at the block or dataset level. The NAS pool uses native ZFS dataset encryption, and the two cluster nodes that carry the databases and the application workload sit on LUKS2 full-disk encryption. A drive pulled out of the rack for RMA or disposal is ciphertext without the key, and the key material lives in places that don't leave the premises. The exception, named rather than glossed: the third cluster node is the small quorum-edge one, and its underlying pool is not encrypted. It holds an etcd member, which is why the layer below matters.

Kubernetes Secrets are separately encrypted at rest inside etcd (AES-CBC), which is what covers the sensitive material on that unencrypted node: the secrets are ciphertext in the datastore itself, not merely behind a disk that happens to be locked. The etcd encryption key is held in two places: on each k3s server, and in a sealed copy on the NAS that's independent of the cluster, so a total cluster wipe followed by an etcd-snapshot-restore can still decrypt the recovered secrets.

Runtime security monitoring is handled by Falco on every node, plus a separate Falco instance watching the Kubernetes audit log. Anything at warning priority or above is forwarded through a small webhook translator into the triage layer described below; critical-priority rule matches are escalated as critical, and warnings go to automated triage first rather than straight to a person. Each application namespace has its own NetworkPolicies so that cross-tenant traffic and unexpected egress paths get dropped at the CNI level rather than relying on application-level trust.

Secrets that back the application (encryption keys for PII columns, OAuth client secrets, SMTP credentials) are stored in Kubernetes Secrets, never checked into git, and rotated when leaked. Secret scanning is not a pre-commit courtesy: every release candidate is scanned with gitleaks over both the working tree and the full git history, and a hit fails the build.

Shipping changes

Nothing reaches a tenant by hand. A release is a commit, and the pipeline builds a candidate image and then runs the full test suite inside that image against a real Postgres, with coverage, plus a migration replay that applies and reverses the schema change against a restored database. On top of that it runs container vulnerability scanning, static analysis, dependency auditing, and the secret scan mentioned above. A failure at any of those stops the release; there is no override flag.

The deploy step is verified rather than trusted, and it is verified because it once wasn't. In August a security fix shipped under a tag that already existed. The deploy tool asked the cluster to run that tag, the cluster was already running that tag, so nothing was replaced, and the deploy reported success in a couple of seconds across all five tenants. Every one of them served the old code for six days while the pipeline said green. The fix was structural: every deploy now resolves the tag to an image digest and pins the digest, so an unchanged tag is not a silent no-op; after each rollout the tool re-reads every running replica's image digest and the version each pod reports about itself and compares them to what was asked for; and the deploy tool's exit code is no longer the success signal. Being unable to measure the result counts as a failure, not a pass. Separately, a standing check compares every running pod against the registry every thirty minutes, so a drift that gets past the deploy path surfaces on its own.

Capacity

Capacity is modeled and load-tested rather than guessed at. Against the public browse path, the honest current numbers are roughly 1,500 concurrent browsers comfortably, 3,000 with headroom left, and around 6,000 as the point where request queueing becomes the visible symptom (still at zero errors, just slower). The binding constraint is web-worker queueing, not CPU, not Postgres, and not the ingress; every other resource is idle when that wall is hit. That was established by ramping synthetic load against the real site and measuring where it bent, and it is re-run rather than assumed.

The largest single lever is the Cloudflare edge cache in front of anonymous HTML. Measured through the real edge under load, it absorbed 98.2 percent of anonymous public traffic before it reached the origin, and requests carrying a session cookie bypass it entirely so no signed-in user is ever served another person's page. A regression run executes on a schedule every week and compares against the recorded baseline, so a change that quietly costs 30 percent of throughput shows up as a failed gate rather than as a surprise on a busy day.

Observability

Prometheus and Grafana run in-cluster, scraping the kubelet, the k3s control plane where it exposes metrics, cAdvisor, and the application pods. Alertmanager routes pageable alerts through the same webhook translator mentioned above; noisy built-in rules like Watchdog are explicitly dropped so the channel only fires when something actually wants attention. Grafana's own state (users, datasources, dashboards) sits on a retained PVC that survived the most recent cluster rebuild without manual restoration.

Alerts don't go straight to a page. They first pass through a triage layer that classifies each one, deduplicates it against whatever is already firing, and decides where it should go. Routine, well-understood conditions (a pod that needs a restart, a transient probe blip, a service already recovering on its own) are handled by a small model running locally on the same on-site GPUs as the rest of the AI stack, which can apply a fixed, pre-approved set of corrective actions and then verify the fix actually held. Anything it can't safely resolve, or anything above a severity threshold, escalates instead of being touched.

Escalation is tiered by severity. Most conditions are routed to a human as a notification with enough context attached to act on quickly. A narrow band of harder infrastructure problems can be handed up to a more capable frontier model for deeper diagnosis. That path is deliberately scoped to the infrastructure and the application's own behavior (logs, metrics, cluster state, service health) and explicitly not to any adopter, applicant, or animal-record data; it reasons about the platform, never about the people using it. Every automated action is logged, and the actions the automated tiers are allowed to take are a fixed, reviewed list, not an open-ended license to change things.

This layer was walked up deliberately rather than switched on at full autonomy: it began observe-only (classifying alerts and taking no action), and actions were enabled narrowly only after the routing had proven correct over time. The standing bias is to escalate when uncertain rather than to act, so that a real or early-warning signal is never written off as noise and a symptom is never patched in a way that hides its cause. The intent is to widen what a lone operator would otherwise miss, not to substitute for one.

Proving it

Redundancy you haven't tested is a guess. The failure modes above are exercised deliberately: the failure is injected on the live cluster, under a synthetic load against the public site, and what the site actually does is measured. Each one is a repeatable script, not a one-time stunt, so the same failure can be re-run after any change that might have quietly regressed it.

Six drills are still on the list, named here so the list above reads as what it is rather than as a victory lap. Kill a whole core switch under load, not one leg at a time, so all three hosts fail over at once and the root-bridge election is part of the measurement. Cut the link between the two core switches under load (a partition, not an outage; the symptom worth hunting is synchronous Postgres commits stalling rather than erroring). Cut the fiber for real so traffic actually moves onto 5G, and watch which transport the tunnel negotiates on a carrier network rather than only that it comes back. Rerun the gateway drills under synthetic client load so the reported number is the one a visitor would feel. Force a database primary switchover under load now that reads are split off to the replica, and measure what the read path actually does rather than asserting it barely notices. And pull utility power to prove the generator transfer.

What this setup isn't

It isn't a colocation deployment in a tier-IV data center. It isn't multi-region. It currently isn't even multi-site. All three cluster nodes are VMs, each on a different hypervisor, and all three hypervisors sit in the same rack. The network and power redundancy buys availability against the failure modes that actually happen at this scale (drives, power supplies, UPS batteries, NICs, single-circuit outages), not against catastrophic site loss. The off-site pieces called out above are what turn that corner.

If you're running a rescue on a small VPS somewhere and you're wondering whether this is better than that: probably, for the things most rescues actually need. If you're running on a well-maintained managed platform with a real SRE team, this isn't going to out-SRE them. The honest answer to most "is this as good as X" questions is "no, but it doesn't need to be for what we're doing here."

Linked discreetly from the main page and the biovision write-up, and nowhere else. If you're reading it, someone probably sent it to you on purpose. Questions welcome at the contact address on the main page.