Portfolio architecture · after the Red Hat methodology

SymphonyGrid — IBM Spectrum Symphony on OpenShift Virtualization

A compute grid whose Windows Server workers are KubeVirt VMs, scaled on demand by KEDA from a live Symphony backlog signal — same-cluster today, cross-cluster over a private RHSI service plane and WireGuard data plane.

Version 1.0 · July 2026 Head 414d272 State Phase 3 in progress · Phases 1–2 exit criteria met Grid Symphony 7.3.2 eval · OpenShift 4.20 IPI · AWS us-east-1

1

About this architecture #

This document presents the SymphonyGrid lab in Red Hat's three-level Portfolio Architecture form — a logical view of what the system is, a schematic view of how it deploys across sites, and a detail view that animates one scenario: an elastic burst driven by KEDA, from task backlog to scale-out to drain and scale-in.

In scope

  • IBM Spectrum Symphony 7.3.2 (eval) grid: RHEL management VMs, Windows Server 2019 compute VMs, all on OpenShift Virtualization.
  • Self-hosted OpenShift 4.20 IPI on AWS (sym-aws), a second burst cluster (sym-burst), GitOps via Argo CD.
  • KEDA-driven VM autoscaling from a live Symphony pending-task signal; RHSI/Skupper metric plane; per-VM WireGuard data plane.

Out of scope / status notes

  • IBM Cloud ROKS remains the reference burst target but is parked pending an API key and VNI allowlist; current burst work targets sym-burst on AWS (decision D13).
  • Production-grade scale-in (Host Factory KubeVirt provider) is designed, not built — KEDA scale-in is a hard kill, an accepted demo gap (D4).
  • Not an official Red Hat or IBM publication; see footer.

2

Logical view #

Four tiers, no networking detail — the level at which the solution is argued, not wired.

Workload
Compute-intensive SOA workloads
Symphony SOA sessions (demo: symping). Windows hosts serve SOA only — MapReduce is unsupported on Windows.
Grid services
IBM Spectrum Symphony 7.3.2
EGO resource manager + SOAM scheduling (SD/SSM) on redundant RHEL master VMs.
symrest API
REST view of applications and backlog — the demand signal every autoscaler consumes.
Host Factory planned
Grid-aware provision/drain — the production replacement for KEDA's hard-kill scale-in.
Elastic compute
Base pool live
Windows Server 2019 VMs cloned from a golden image; join the grid hands-off in ~9.5 min.
Same-cluster burst live
KEDA scales a second VM pool 0→2 next to the base pool.
Cross-cluster burst proven · torn down
Identical pool on a second cluster, joined over WireGuard; rebuildable from git.
IBM Cloud ROKS planned
Reference burst site on VPC bare metal with VPC-routable VM IPs (VNI).
Platform & operations
OpenShift Virtualization on OCP 4.20 IPI
Self-hosted on AWS; VMs on a flat layer-2 UDN — no NAT between grid hosts.
GitOps — Argo CD
App-of-apps per cluster; every cluster change lands through git.
Golden image pipeline
Tekton builds a sysprepped WS2019 + Symphony DataSource once; pools clone it.
Interconnect
WireGuard hub for grid traffic; RHSI/Skupper for the metric plane; IPsec/Direct Link planned for IBM Cloud.

3

Schematic view #

Deployment topology across sites. Connector color follows the Red Hat convention of encoding flow type; solid boxes are live, dashed are planned or currently torn down. Use the legend buttons to isolate one plane.

Schematic — SymphonyGrid (network + data) PLANNED · IPsec VPN, static routes (D3) → Direct Link (production) · MTU ~1446 / MSS 1360 AWS · SYM-AWS — PRIMARY SITE VPC 10.0.0.0/16 · us-east-1 · ~$5.2/hr while up OpenShift 4.20 · self-hosted IPI (D9) · 3 masters + 3 workers + m5zn.metal virt node GRID UDN · LAYER 2 · 10.10.0.0/20 · NO NAT sym-master1 RHEL 9 VM · EGO/SOAM SD · SSM · vemkd 10.10.0.31 symrest :8050 sym-master2 standby · 10.10.0.34 symphony-metrics-shim pusher · polls 10s K8s API egress only sym-compute-win WS2019 pool ×2 VirtualMachinePool d1 · dedicated CPU joins in ~9.5 min sym-compute-win- burst KEDA-scaled 0→2 ScaledObject → /scale subresource hosts reconciler CronJob (D12) · 1 min qemu-guest-agent guest-exec → rewrites both masters' /etc/hosts + EGO hosts files NS SYMPHONY-METRICS default pod network ConfigMap symphony-metrics symphony-metrics- server :8080 /api/v1/pending :8081 /metrics RHSI Site sym-aws Connector metrics-server linkAccess: none→NLB KEDA Custom Metrics Autoscaler (D4) metrics-api trigger target 8 · poll 15s UWM Prometheus user-workload monitoring ServiceMonitor 15s observability only wg-hub · EC2 t3.small WireGuard hub · routes, never NATs UDP 51820 · 10.20.0.0/24 AWS · SYM-BURST — BURST SITE VPC 10.1.0.0/16 · torn down 07-12 · in git OpenShift 4.20 IPI · compute VMs only (D13) RHSI Site sym-burst Listener symphony-metrics :8080 · dial-out only KEDA ScaledObject sym-compute- win-burst-remote GRID-BURST UDN · 10.12.0.0/20 sym-compute-win-burst-remote WS2019 VirtualMachinePool 0→2 joins AWS masters over WireGuard firstboot: wg0 up → 7869 IBM CLOUD — ROKS PLANNED · parked: API key + VNI allowlist ROKS 4.20+ on VPC manual OpenShift Virtualization install VPC bare-metal workers bx2d/cx2d/mx2d.metal · ×3 VNI · OVN localnet UDN VPC-routable VM IPs — no overlay Host Factory provider (IBM) same 5-script interface, ROKS target INFRASTRUCTURE MANAGEMENT Argo CD · OpenShift GitOps app-of-apps per cluster (D6) Tekton windows-efi-installer golden WS2019+Symphony DataSource installer-repo nginx+WebDAV · eval binaries + ISO git.arsalan.io/anaeem/grid source of truth · HTTPS+PAT :8050 · 10s K8s API PATCH out of UDN GET /api/v1/pending · 15s /scale · replicas 0→2 RHSI link · mTLS · symphony-metrics /scale 0→2 EGO 7869 TCP+UDP WireGuard UDP 51820 · dial-out only routed, never NATed · masters 10.20.0.1/.2 :8081 scrape Argo sync — sym-aws sym-burst
Grid data plane Metric & scale plane Management & observability

4

Detail view — a KEDA burst, step by step #

One scenario, animated slide by slide: how a Symphony task backlog becomes new Windows VMs — first inside sym-aws (Track A, live-proven 2026-07-11/12), then across clusters to sym-burst (Track B). Step through with the buttons, dots, or ← → keys.

Step 1 / 19 Track A — same cluster
AWS · SYM-AWS — PRIMARY SITE OpenShift 4.20 · grid UDN 10.10.0.0/20 · no NAT GRID UDN · LAYER 2 sym-master1 EGO/SOAM · SD · SSM pending: 0 symrest :8050 base port 7869 wg0 10.20.0.1 sym-master2 candidate · wg0 10.20.0.2 symphony-metrics-shim polls :8050 every 10s pusher — dials out only sym-compute-win base pool ×2 · WS2019 win-a win-b 8 SIs each in the live gate run sym-compute-win- burst replicas: 0 burst-1 burst-2 DataVolume clone of ws2019-symphony hosts reconciler · 1 min guest-exec → EGO hosts files on both masters NS SYMPHONY-METRICS default network ConfigMap symphony-metrics symphony-metrics- server :8080 /api/v1/pending :8081 /metrics 5s cache RHSI Site + Connector routingKey symphony-metrics dial-in via NLB link KEDA metrics-api trigger target 8 · poll 15s ScaledObject → HPA UWM Prometheus observability only symphony_pending_tasks wg-hub · EC2 routes, never NATs UDP 51820 inner 10.20.0.0/24 AWS · SYM-BURST — BURST SITE torn down 2026-07-12 · rebuildable from git RHSI Listener symphony-metrics :8080 materializes the sym-aws service locally KEDA (sym-burst) ScaledObject sym-compute- win-burst-remote target 8 · poll 15s GRID-BURST UDN · 10.12.0.0/20 sym-compute-win-burst- remote replicas: 0 rburst-1 rburst-2 firstboot: wg0 up → EGO_MASTER_LIST → LIM restart → 7869 wg-peer reconciler (runs on sym-aws) reads pubkey via guest-exec, assigns IP, registers hub peer + masters

5

Network detail #

The invariants below are the hard-won constraints of this grid — breaking any of them has a documented cost in days of debugging.

Important — no NAT, same base port

EGO host registration is hostname/IP-anchored: every host must use base port 7869 (7 consecutive ports from it), and there must be no NAT between grid hosts. Both flat layer-2 UDNs (10.10.0.0/20, 10.12.0.0/20) and the WireGuard hub (routes, never masquerades) exist to preserve this.

Important — identical MTU everywhere

Proven outage on this deployment (2026-07-10): the UDN inherited the AWS jumbo pod-network MTU (8901) while Windows guests silently ignored DHCP option 26 and stayed at 1500 — masters sent ~3.8 KB LIM UDP payloads as jumbo frames Windows dropped, so compute hosts registered but sat unavail (matches IBM KB ibm16164145). Fix: pin the masters down to 1500 (Windows already defaults there), since the live UDN's MTU field is immutable.

Note — dynamic hosts

EGO_DYNAMIC_HOST_WAIT_TIME must be set (this grid: 120s) or dynamic hosts silently never join; EGO_DYNAMIC_HOST_TIMEOUT (15M here) must be set or dead burst VMs linger forever as ghost hosts. WAN paths additionally need EGO_ENABLE_CHAN_KEEPALIVE=y with keepalives ≥180s and below the tunnel idle timeout.

Note — UDN workloads are ingress islands

OVN-Kubernetes primary-UDN port security means a UDN VM or pod can never relay traffic for another host, and nothing outside can dial in (no ClusterIP/NodePort path in). Every cross-boundary flow in this architecture therefore dials out: the shim pushes via the K8s API, Skupper links dial out from sym-burst, WireGuard peers dial out to the hub.

Table: Symphony & platform port matrix (as deployed)
Port(s)PurposeNotes
7869 TCP+UDPEGO base port — LIMIdentical on every host; 7 consecutive ports allocated from here
7870vemkd (EGO_KD_PORT)First client connection point
7871 / 7872 / 7873PEM / egosc / repository serviceProcess manager, service controller, package distribution
7874 / 7875SD admin / SD SDKSession director
31000–31019SSM ← clients (SSM_SDK_ADDR)Pinned in sd.xml — OS-random by default, reassigned every restart
31020–31039SSM ↔ SIM (SSM_SIM_ADDR)Pinned; PLATCOMMDRV_TCP_KEEPALIVE_TIME=180
8050–8060symrest (this eval build)Doc default is 8180 — discovered live, not assumed; HTTPS + Basic auth
8080 / 8081symphony-metrics-server/api/v1/pending for KEDA · /metrics for UWM
51820 UDPWireGuardAll peers dial out to wg-hub; inner net 10.20.0.0/24
8443 / 8080Symphony web GUITLS / non-TLS
9443 / 9080Host Factory RESTPhase 5
1527 · 4046 · 9200–9202 / 9300–9302Derby DB · plc · ElasticsearchManagement-host internals

6

Design decisions #

Locked decisions D1–D13; D14–D16 are proposed and awaiting a call.

D1 · Footprint — staged minimal: primary AWS cluster + one metal virt node first; IBM Cloud later.
D2 · Windows Server 2019 — WS2022 needs Fix 601711 the eval entitlement cannot obtain.
D3 · Interconnect (IBM) — IPsec VPN for PoC, Direct Link for production. Planned.
D4 · KEDA burst driver — ScaledObject → VirtualMachinePool /scale; scale-in is not grid-aware (accepted demo gap, Host Factory is the production path).
D5 · Eval entitlement — Advanced Edition evaluation, valid to 2026-12-31; full product incl. Windows + Host Factory.
D6 · GitOps — OpenShift GitOps (Argo CD), app-of-apps per cluster.
D7 · Windows shape — Windows VMs on OpenShift Virtualization, not WMCO container nodes.
D8 · Masters as RHEL VMs — same cluster, layer-2 UDN; revisit EC2 placement if NLB exposure gets awkward.
D9 · Platform pivot — self-hosted OCP 4.20 IPI replaces ROSA HCP (Marketplace billing wall).
D10 · ROSA retry — burst target retry; failed on the same billing wall. Superseded by D13.
D11 · Same-cluster burst net — shared grid L2 across namespaces via CUDN migration; gated on research.
D12 · Hosts reconciler — CronJob syncs clone IP/hostname into masters' EGO hosts files; blocking for hands-off burst.
D13 · sym-burst — second self-hosted IPI cluster as the burst site; masters stay on sym-aws; compute VMs only.
D14 · proposed — keep per-VM WireGuard as the grid data plane; every mesh product breaks EGO's IP-anchored identity or lacks generic UDP.
D15 · proposed — adopt the home-lab virt cluster as an ACM hub (Hive cluster lifecycle, hibernate).
D16 · proposed — registry-driven golden images: push once to a registry, DataImportCron on every cluster.

7

Adapting this architecture #

If your inputs differ, re-derive these levers first (the Red Hat convention of naming what a reader must recompute):

  • Scale trigger & threshold — targetValue 8 pending tasks and cooldown 300s are demo-sized; derive from your task duration × slot count.
  • SSM port ranges — two 20-port pinned ranges sized for a ~50-listener NLB budget; widen for more concurrent sessions.
  • Burst compute shape — d1 instance type with dedicated CPU placement; re-derive from workload profile and metal node capacity.
  • Interconnect — WireGuard hub is the lab answer for AWS↔AWS; IPsec/Direct Link apply for IBM Cloud; anything that NATs or proxies the grid plane is disqualified.
  • Scale-in policy — KEDA hard-kills VMs; adopt the Host Factory provider before running workloads that cannot tolerate lost in-flight tasks.
  • Image lifecycle — one golden DataSource per cluster today; D16's registry + DataImportCron model once multi-cluster.