--- title: "Hosted Experience Builder — Architecture & Operations" canonical: "https://help.cartinuum.com/space/WC/4484005900/Hosted%20Experience%20Builder%20%E2%80%94%20Architecture%20%26%20Operations" format: markdown --- **Service**: `instalink-container-host` (`services/container-host/`) **Purpose**: Spin up short-lived, per-tenant ArcGIS Experience Builder (EXB) containers on AWS ECS Fargate, front them with an ALB, and persist app artefacts on EFS/S3. > ℹ️ **Audience**: Engineering and DevOps teams managing the Hosted Experience Builder infrastructure. ## Overview Each tenant that purchases the Hosted EXB feature gets an on-demand ArcGIS Experience Builder authoring environment. When a tenant clicks **Start session**: 1. The frontend calls `POST /config-update` with `action: START` 2. The API registers/verifies the tenant's ArcGIS OAuth client ID (`SIGNIN_INFO`) then publishes a message to `startContainerQueue` SQS 3. The `start-container` Lambda creates an ECS Fargate task, mounts their EFS access point, registers the task with the ALB, and sets a 2-hour auto-stop timer via EventBridge 4. When the timer fires (or the tenant clicks Stop), `stop-container` tears everything down Sessions are isolated per tenant via unique EFS access points, ALB listener rules, ECS task groups, and EventBridge rules. ## Key AWS Services | Service | Role | | --- | --- | | ECS Fargate | Runs per-tenant ExB container | | ALB | HTTPS path-based routing to tenant containers | | EFS | Persistent storage for draft ExB apps per tenant | | S3 | Storage for deployed/published ExB apps | | CloudFront | CDN for serving published apps | | SQS | Queues for start/stop/extend/deploy operations | | EventBridge | Auto-stop timer (2h sessions) | | ECR | Container image registry (built from geovonic repo) | | RDS | Session state and config storage (configs table) | ## Handler Reference | Handler | Queue / Trigger | What It Does | | --- | --- | --- | | `start-container.js` | `startContainerQueue` (SQS) | Creates EFS access point, registers ECS task definition, runs Fargate task, creates ALB target group + listener rule, creates EventBridge auto-stop timer, writes status: active | | `stop-container.js` | `stopContainerQueue` (SQS) | Deletes ALB rule, target group, stops ECS task, removes EventBridge timer, deletes EFS access point, writes status: inactive | | `extend-container.js` | `extendContainerQueue` (SQS) | Extends EventBridge timer by 2h | | `process-container-app.js` | `processContainerAppQueue` (SQS) | DEPLOY: bakes clientId into app zip, uploads to S3. REMOVE: deletes from S3 | | `published-app-list.js` | `GET /container/published-apps` | Lists ExB apps from EFS directory + DB ExbApp records | | `create-container-demo-app/` | `createDemoAppQueue` (SQS) | Seeds a starter demo app in tenant's EFS directory | ## Container-Host Architecture ### ECS Cluster and Fargate Tasks - **Cluster**: `instalink-container-` (one cluster per environment) - **Task Group**: `exb--` (identifies/stops tenant tasks) - **Task Definition Family**: `exb--` (deregistered on stop) - **Image**: from ECR (`public.ecr.aws/.../instalink/exb:latest--`) - **Port**: 3000 (ExB Node.js server) - **POSIX User**: uid/gid 1000 (node user) - **Lambda timeout**: 600s for start-container ### ALB Path-Based Routing Each tenant session gets a unique ALB listener rule: - **Path pattern**: `/container//exb-*` - **Target group**: `exb--` (IP target, port 3000) - **Priority**: `tenantId` (must be unique across all rules) - **Health check**: path = `/container//exb-/`, codes 200-399 - **Default action**: 404 fixed response ### EFS — Draft App Storage Each tenant gets an isolated EFS access point: | Property | Value | | --- | --- | | Name | `exb--` | | Root directory | `/exb--` | | POSIX uid/gid | 1000:1000 (matches container node user) | | Permissions | 0755 | Access points are created by `start-container` and deleted by `stop-container`. Temporary access points with unique suffixes are used by `process-container-app` and `create-container-demo-app`. The ECS task mounts the tenant's access point at `/home/node/ArcGISExperienceBuilder/server/public`. ### S3 — Published App Storage - **Bucket**: `ContainerAppBucket` (named per stage) - **Access**: Private ACL with public-read bucket policy for `s3:GetObject` - **CORS**: Enabled (all origins, GET/PUT/POST) - **Structure**: `container-app//exb-//index.html` ### Security Groups | Resource | Ingress | Egress | | --- | --- | --- | | ALB SG | 80, 443 from 0.0.0.0/0 | All traffic | | ECS Task SG | All from ALB SG | All traffic | | EFS SG | All from ECS Task SG; NFS (2049) from VPC SG | NFS (2049) to VPC SG | ## ECR Image Build Pipeline (geovonic repo) > ⚠️ **Important**: The ECR image build has moved to the **geovonic** repo. After the seed build for the ExB widget is completed, you must manually trigger the Bitbucket Pipeline to build and push the new image. ### Build Types | Build Type | BUILD_TYPE | Description | Region Handling | | --- | --- | --- | --- | | Extensions | `extensions` | Geovonic widgets + Connect widget | AU only | | Connect | `connect` | Connect widget only | Multi-region (au, us, eu) | ### Custom Pipelines (Bitbucket) | Pipeline | Stage | Regions | | --- | --- | --- | | `build-extensions-image-dev` | dev | AU | | `build-extensions-image-beta` | beta | AU | | `build-extensions-image-prod` | prod | AU | | `build-connect-image-dev` | dev | AU | | `build-connect-image-qa` | qa | AU | | `build-connect-image-beta` | beta | AU, US, EU (parallel) | | `build-connect-image-prod` | prod | AU, US, EU (parallel) | ### Build Process 1. **set-docker-env.sh**: Sets stage, region, ExB versions, AWS region, Connect widget URL 2. **setup_pipeline.sh**: Installs AWS CLI v2, configures credentials 3. **post-deploy.sh**: Resolves ECR URI from CloudFormation, runs build.sh 4. **build.sh → buildimage.sh**: Builds widgets, Docker image, pushes to ECR ### Docker Image Two-stage build at `packages/functions/src/extension-module/Dockerfile`: - **Stage 1 (Builder)**: Downloads ExB 1.19, applies patches (portalurl widget, oauth-callback, PWA disable, server fixes), copies widgets, installs deps, builds widgets - **Stage 2 (Runtime)**: Alpine-based, copies server + built client, installs dumb-init/aws-cli/rsync/jq, port 3000 ### ECR Registries | Environment | Registry | | --- | --- | | Non-prod | `public.ecr.aws/x2d6z0j2/instalink/exb` | | Prod | `public.ecr.aws/z1m2f7j5/instalink/exb` | ### Seed.run Deployment The `container-host` Serverless stack is deployed via Seed.run as part of the main `instalink` monorepo. This provisions the ECS cluster, ALB, EFS, S3 bucket, SQS queues, Lambda functions, IAM roles, security groups, and DNS records. ## Published Apps — Deploy Flow Tenants can deploy ExB apps from their hosted authoring session. Deployed apps are static HTML/JS bundles stored in S3 and served via CloudFront. ### Deploy Process 1. Tenant triggers deploy via `POST /config-update` with `action: DEPLOY` 2. `process-container-app` Lambda creates a temporary EFS access point 3. Runs a short-lived ECS task that calls `zipApp(appId, output.zip, clientId)` — bakes the OAuth clientId into the app 4. Runs `aws s3 sync` to upload to `container-app//exb-//` 5. On redeploy, triggers CloudFront invalidation 6. Cleans up the temporary EFS access point ### Deployed App URL `https:///container-app//exb-//index.html` ## App ID Redirect URL Configuration > ⚠️ For ArcGIS Enterprise users, this is **mandatory** — without it, users get `Invalid redirect_uri` errors when authenticating. The App ID redirect URL in the ArcGIS portal must include the correct domain for Hosted ExB to work with ArcGIS authentication. ### Required Redirect URLs | Stage | Redirect URL | | --- | --- | | Production | `https://app.cartinuum.com` | | Beta | `https://app-beta.cartinuumdevhub.com` | | QA | `https://app-qa.cartinuumdevhub.com` | | Dev | `https://app-dev.cartinuumdevhub.com` | See [Action required: updates to Geovonic](https://help.cartinuum.com/space/WC/4260757505/Action+required:+updates+to+Geovonic#1.-Mandatory-Action-for-ArcGIS-Enterprise-(Critical-for-Login/Authentication)) for full instructions. ### clientId Flow The `clientId` is sourced from ArcGIS at START time by `register-client-id.js` (in `services/common/arcgis-rest/`). It searches the tenant's ArcGIS portal for an item named `[env] Hosted Connect for ArcGIS`, and the `client_id` from that item **always overwrites** whatever is in the DB. The `clientId` is also baked into deployed apps at deploy time. If it changes, apps must be re-deployed to pick up the new value. ## Troubleshooting & Ops Runbook ### Stuck "Starting" Session **Symptom**: The UI shows "Starting..." indefinitely. Clicking Start does nothing. **Root cause**: `start-container` Lambda timed out after writing `status: activating`. The guard prevents re-runs. **Fix**: 1. Identify the tenant (query `configs` table for `HOSTED_EXB` with `status = activating`) 2. Reset DB status to `inactive` 3. Clean up orphaned AWS resources (ECS task, ALB rule, target group, EventBridge rule) 4. Click Start session again ### DuplicateTargetGroupName on Retry **Symptom**: `DuplicateTargetGroupNameException` in Lambda logs. **Fix**: Delete the orphaned target group (and ALB rule if attached), then retry. ### Quick Reference | Symptom | Likely Cause | Fix | | --- | --- | --- | | Session stuck at "Starting" forever | ALB health check timeout + missing save bug | Reset DB, clean orphaned resources, retry | | `Missing organisationId` | No RelayServiceKey with organisationId | Create/backfill RelayServiceKey | | `Missing signin-info` | SIGNIN_INFO config row missing | Run onboarding START flow | | `DuplicateTargetGroupNameException` | Orphaned target group | Delete orphaned TG + ALB rule, retry | | `Invalid redirect_uri` in deployed app | Wrong clientId baked at deploy time | Re-deploy app after confirming SIGNIN_INFO | | clientId keeps reverting | register-client-id.js overwrites from ArcGIS | Update redirect URLs on ArcGIS app item | | Task stuck in PENDING | Image pull failure, quota, or SG issue | Check stoppedReason and container logs | | ALB returns 502/503 | Task deregistered or health checks failing | Check ECS task and container logs | ### Known Bugs | Bug | File | Impact | Workaround | | --- | --- | --- | --- | | Missing `await exbCfg.save()` on health check timeout | `start-container.js:321` | DB stays activating forever | Manually reset DB status | | waitForTargetHealthy window too short (100s) | `start-container.js` | ALB needs ≥150s; Lambda gives up too early | Increase maxRetryCount to 40+ | | createTargetGroup not idempotent | `start-container.js` | Retry fails with DuplicateTargetGroupNameException | Delete orphaned target group | ## Data Model ### configs Table | Column | Type | Notes | | --- | --- | --- | | id | INTEGER PK | auto-increment | | config_type | STRING | HOSTED_EXB, SIGNIN_INFO, etc. | | data | JSON | Config payload | | tenant_id | INTEGER FK | References tenants.id | | secure | BOOLEAN | If true, data is encrypted at rest | | deleted_at | DATE | Soft-delete (paranoid mode) | ### Session Status State Machine ``` [not started] activating --(success)--> active --(2h timer / STOP)--> inactive | +--(any AWS error)--> failed ``` ## URL Patterns | Resource | Pattern | | --- | --- | | Hosted ExB session | `https:////container/exb-/` | | Published apps API | `GET https:///container/published-apps` | | Config update API | `POST https:///config-update` | | Deployed app | `https:///container-app//exb-//index.html` | ## Related Docs - `docs/hosted-exb-architecture.md` — source architecture doc - `services/container-host/serverless.yml` — Serverless stack definition - `services/container-host/handlers/` — Lambda handlers - [Action required: updates to Geovonic](https://help.cartinuum.com/space/WC/4260757505/Action+required:+updates+to+Geovonic) — Redirect URL instructions