Operations

Operations

This page covers compute service configuration, Helm chart values, the gVisor DaemonSet deployment model, and the PVC 24-hour grace pattern used for workspace cleanup.

For gVisor node-level installation troubleshooting, see gVisor Install.

---

Environment variables

The compute server binary reads all configuration from environment variables. In the Helm chart, non-secret tunables come from the ConfigMap rendered by helm/compute/templates/server-configmap.yaml. Secrets (none in Phase 3) would use Kubernetes Secret objects.

Variable	Default	Description
GRPC_PORT	50051	gRPC listen port.
LOG_LEVEL	info	Log verbosity. One of debug, info, warn, error.
OTEL_EXPORTER_OTLP_ENDPOINT	(empty)	OTLP endpoint for trace export. Omit to disable tracing.
OTEL_SERVICE_NAME	compute	Service name injected into traces and logs.
ENV	dev	Deployment environment. One of dev, staging, prod.
NAMESPACE	paper-board	Kubernetes namespace where sandbox pods are created.
SANDBOX_IMAGE	ghcr.io/paper-board/sandbox-python:v0.1.0	Default container image used when CreateSandboxRequest.image is empty.
PVC_STORAGE_CLASS	standard	Default StorageClass for workspace PVCs. Override per-environment (see below).
PVC_WORKSPACE_SIZE	5Gi	Default workspace PVC capacity.

Per-environment PVC_STORAGE_CLASS

Environment	Recommended value	Notes
kind (local)	standard	kind’s default hostpath provisioner.
GCP	standard-rwo	GCP Regional PD, ReadWriteOnce.
AWS	gp3	EKS gp3 EBS. Deferred to Phase 6 hardening per ADR-0015.

---

Helm values

The chart lives at helm/compute/ in the paper-board/compute repository. Install or upgrade with:

helm upgrade --install compute ./helm/compute \
  -n paper-board \
  -f values-<env>.yaml

The Helm pre-install / pre-upgrade Job hook runs the migrator before the server deployment rolls out (no-op in compute because the service has no Postgres schema, but the hook is present for Helm contract parity with other services).

server

server:
  replicas: 1
  image:
    repository: ghcr.io/paper-board/compute-server
    tag: v0.1.0
    pullPolicy: IfNotPresent
  service:
    type: ClusterIP
    port: 50051
  config:
    grpcPort: 50051
    logLevel: "info"
    otlpEndpoint: ""
    otelServiceName: "compute"
    env: "dev"
  resources:
    requests: { cpu: 100m, memory: 256Mi }
    limits:   { cpu: 500m, memory: 512Mi }

sandbox

sandbox:
  image:
    repository: ghcr.io/paper-board/sandbox-python
    tag: v0.1.0
    pullPolicy: IfNotPresent
  pvc:
    storageClass: "standard"      # override per environment
    workspaceSize: "5Gi"

gc

The GC CronJob (cmd/gc) deletes PVCs whose compute.paper-board.io/delete-after annotation has passed. It runs hourly by default.

gc:
  enabled: true
  image:
    repository: ghcr.io/paper-board/compute-gc
    tag: v0.1.0
    pullPolicy: IfNotPresent
  schedule: "0 * * * *"    # hourly
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { cpu: 100m, memory: 128Mi }

gvisor

gvisor:
  enabled: true
  version: "20240212"          # gVisor release date tag; immutable pin
  installer:
    image:
      repository: curlimages/curl
      tag: "8.10.1"
      pullPolicy: IfNotPresent
  nodeSelector: {}             # empty = all nodes; scope to sandbox node pool in prod
  tolerations: []
  scheduling: {}               # passed verbatim into RuntimeClass.scheduling

Full gVisor values reference (node pool scoping, cloud-specific notes) is in gVisor Install.

networkPolicy

Default-deny ingress with an explicit allowlist:

networkPolicy:
  enabled: true
  egress:
    kubeDns:
      enabled: true
      namespace: kube-system
      port: 53
  ingressFrom: []              # add runtime + agents CIDRs / pod selectors here

---

gVisor DaemonSet deployment

The runsc-installer DaemonSet runs on every node in the cluster (or a scoped node pool when gvisor.nodeSelector is set). It:

1. Downloads and verifies runsc from the gVisor release bucket.
2. Installs runsc at /usr/local/bin/runsc on the host.
3. Appends the runsc runtime handler block to /etc/containerd/config.toml and restarts containerd.

After the DaemonSet pods complete their init containers, the RuntimeClass named gvisor (handler: runsc) is registered cluster-wide. All sandbox pods created by ComputeService.CreateSandbox reference this class via runtimeClassName: gvisor.

Verify installation

# All DaemonSet pods should be Running
kubectl get ds -n paper-board -w

# RuntimeClass must exist
kubectl get runtimeclass gvisor

# Confirm a sandbox pod uses gVisor
kubectl get pod <sandbox-pod> -o jsonpath='{.spec.runtimeClassName}'
# expected: gvisor

Upgrade runsc

1. Update gvisor.version in values.yaml to the new release date tag.
2. Bump helm/compute/Chart.yaml version.
3. helm upgrade compute ./helm/compute — the DaemonSet rolls out; each init container detects the version mismatch and reinstalls runsc.

For zero-downtime upgrades in production (Phase 6), drain sandbox pods from a node before the DaemonSet pod reaches it.

---

PVC 24-hour grace pattern

When DestroySandbox is called, the compute server:

1. Deletes the sandbox pod immediately.
2. Annotates the workspace PVC with:

   compute.paper-board.io/delete-after: <RFC3339 timestamp = now + 24h>

The GC CronJob (cmd/gc) runs hourly and deletes any PVC in the paper-board namespace whose delete-after annotation timestamp is in the past.

This pattern preserves workspace data for 24 hours after session end, which is useful for:

• Post-mortem inspection after an unexpected crash.
• Agent frameworks that restart a session and want to rehydrate workspace state.

In Phase 6 (production hardening), VolumeSnapshot backups before GC are planned.

Adjust the grace window

The grace window is a compile-time constant (core.PVCGraceSeconds = 86400). To change it, update the constant in internal/core/sandbox.go, rebuild, and re-deploy. A configurable grace window via Helm values is planned for Phase 6.

Manual PVC cleanup

To force-delete a workspace PVC before the grace period expires:

kubectl delete pvc workspace-<sandbox-id-prefix> -n paper-board

The short prefix is the first 8 hex characters of the sandbox UUID (e.g. workspace-f47ac10b for sandbox_id = f47ac10b-58cc-...).

---

Local development bring-up

# 1. Start a local kind cluster with gVisor support
kind create cluster --config kind-config.yaml

# 2. Install the chart (kind storage class = "standard")
helm upgrade --install compute ./helm/compute \
  -n paper-board \
  --create-namespace \
  --set sandbox.pvc.storageClass=standard \
  --set gvisor.enabled=true

# 3. Wait for the DaemonSet init containers to finish (~60 s on kind)
kubectl rollout status ds/runsc-installer -n paper-board

# 4. Port-forward the compute server
kubectl port-forward svc/compute 50051:50051 -n paper-board

# 5. Smoke-test with grpcurl
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

---

Runbook: sandbox pod stuck in Pending

kubectl describe pod <sandbox-pod> -n paper-board | grep -A 10 Events

Common causes:

• No nodes match RuntimeClass scheduling. Verify kubectl get runtimeclass gvisor and that at least one node has runsc installed.
• PVC not bound. Check kubectl get pvc -n paper-board; a Pending PVC usually means the StorageClass provisioner is unavailable or PVC_STORAGE_CLASS is set to a class that does not exist in the cluster.
• Insufficient node resources. Sandbox pods request cpu: 1000m, memory: 1Gi by default. Scale the node pool or reduce ResourceLimits in the CreateSandboxRequest.

Runbook: GC CronJob not deleting PVCs

kubectl logs job/<gc-job-name> -n paper-board

Check that:

1. The compute.paper-board.io/delete-after annotation is present on the PVC (kubectl get pvc -n paper-board -o yaml | grep delete-after).
2. The timestamp has passed (UTC).
3. The GC service account has delete permissions on persistentvolumeclaims (see helm/compute/templates/gc-rbac.yaml).