Skip to content

Dev MCP server on Easysource-internal — design

Date: 2026-09-09 Owner: jeevan@hirequotient.com Status: approved-in-progress (infra: EIP attached; remaining: box setup + deploy-dev.yml)

Goal

Stand up a dev/staging deployment of hq-mcp-server that can be exercised end-to-end through a real claude.ai connector (Google OAuth → MCP → staging BE → outreach-handler mail), so feature branches (starting with feat/via-claude-mail-source) are verified before they merge to main — which is the existing prod build + manual-deploy path (mcp.hirequotient.com).

Non-goal: changing anything about the prod MCP on uat-vm or the main build.yml / deploy.yml pipelines. The dev path is strictly additive.

Decisions (locked)

# Decision Choice
1 Exposure Dedicated subdomain + nginx TLS reverse proxy
2 Hostname mcp-internal.hirequotient.com
3 Deploy trigger Manual deploy-dev.yml (workflow_dispatch, any ref); the branch image is built on-demand inside that workflow, then SSH-deployed
4 BE target Existing staging BE https://easysource-internal.hirequotient.com/api (same box; already runs the feature)
5 Stable IP Fresh EIP 35.153.20.238 attached to the box; both easysource-internal + mcp-internal DNS point at it
6 Google OAuth Add redirect URI https://mcp-internal.hirequotient.com/oauth/google/callback to the existing OAuth client (done by owner in Google Cloud)

Target environment

  • Box: Easysource-internal, instance i-0aa0dcabda4ba8f5e, AZ us-east-1b, subnet subnet-0f0b3a59bc25ee466, private IP 10.0.24.66, account bcg-tech-access (788209399382).
  • EIP: 35.153.20.238 (eipalloc-07348920d5e3e3cad, assoc eipassoc-0cd04d0c71bbc1c0c). Old auto IP 3.92.147.173 released.
  • Co-located services already on the box: staging BE (blue/green, ports 4000/4001) behind nginx; a :3000 frontend. The outreach mail service runs on a separate box outreach-handler-bd-internal (i-03a258162878141c3) — the async end of the [via MCP] chain.

Runtime layout (on the box)

  • One clone ~/hq-mcp-server (the box's only MCP instance — prod MCP lives on uat-vm, so no -dev suffix or second clone is needed).
  • A dev-flavored .env (git-ignored, placed once during setup) + a certs/db-cert.pem for Mongo X509.
  • Docker container hq-mcp on 127.0.0.1:5011 → container :4100 — the scripts/deploy.sh defaults, so no MCP_CONTAINER / MCP_HOST_PORT overrides are needed. Confirm 5011 is free before pointing nginx at it (BE uses 4000/4001, so it should be).
  • deploy.sh pulls the GHCR image for the checked-out commit, replaces the container, and health-checks /healthz. Same tested script prod uses — dev and prod deploy paths cannot silently drift.

Dev .env deltas from prod

Key Dev value
PORT 4100
NODE_ENV non-production (pretty logs)
PUBLIC_BASE_URL https://mcp-internal.hirequotient.com (base only — NO /mcp; the app appends /mcp for the resource and /oauth/google/callback for OAuth)
ALLOWED_HOSTS mcp-internal.hirequotient.com
BE_BASE_URL https://easysource-internal.hirequotient.com/api
EASYSOURCE_WEB_BASE https://easysource-internal.hirequotient.com
GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET existing client (redirect URI added for the dev host)
MONGO_URI / MONGO_CERT_PATH / REDIS_URL / JWT_SECRET_TOKEN reuse the internal/UAT values

Edge (nginx + DNS + TLS)

  • New nginx server block for mcp-internal.hirequotient.com:
  • proxy_pass http://127.0.0.1:5011; passing /mcp, /healthz, /oauth/google/callback.
  • Streamable-HTTP/SSE friendly: proxy_buffering off;, proxy_read_timeout long (e.g. 3600s), proxy_http_version 1.1, forward Host + X-Forwarded-*.
  • DNS: A mcp-internal.hirequotient.com → 35.153.20.238 (new); A easysource-internal.hirequotient.com → 35.153.20.238 (re-pointed). Both done.
  • TLS: certbot certificate for mcp-internal.hirequotient.com.

CI/CD — deploy-dev.yml (new, manual, additive)

  • Trigger: workflow_dispatch with a ref input (branch / tag / sha). Optional internal GitHub Environment for an approval gate.
  • Job 1 — build: checkout ref → docker build → push ghcr.io/hirequotient/hq-mcp-server:<sha>. On-demand build keeps GHCR from accumulating an image per branch-push and leaves build.yml/main untouched.
  • Job 2 — deploy (needs: build): SSH to the box (secrets DEV_DEPLOY_SSH_HOST / DEV_DEPLOY_SSH_USER / DEV_DEPLOY_SSH_KEY / DEV_DEPLOY_SSH_PORT, or an internal environment) → cd ~/hq-mcp-server → git fetch && git checkout <ref> → ./scripts/deploy.sh → health check.
  • concurrency: deploy-dev (no overlapping dev deploys). Prod deploy.yml (→ uat-vm, main-only) is never touched.
  • Runner → box SSH: no SG change needed — the box's SG (Easysource-VMs, sg-03b58206d572449c9) already allows 22/80/443 from 0.0.0.0/0, the same world-open-SSH + key-auth pattern prod's uat-vm (EasySource_SG) uses. Setup is only: generate a deploy keypair, add its public half to the box user's authorized_keys, and store the private half as DEV_DEPLOY_SSH_KEY.

End-to-end test (acceptance gate)

  1. Register a second claude.ai connector "Easysource (dev)" → https://mcp-internal.hirequotient.com/mcp; complete Google OAuth.
  2. Against a test org, run a tool that emits an internal-team mail (e.g. create project / update vetting) and confirm the [via MCP] tag lands: subject tag on sync sites; DB-stamp → outreach-handler on async (service_name="outreach-handler" in Loki).
  3. Confirm the prod connector (mcp.hirequotient.com) and prod mail are unaffected.
  4. Green → merge the feature branch to main (existing prod path).

Setup order (runbook)

  1. ✅ Allocate + attach EIP 35.153.20.238; re-point both DNS records. (done)
  2. Google Cloud: add the dev redirect URI to the existing OAuth client. (owner)
  3. On the box: install Docker (if absent), git clone → ~/hq-mcp-server, drop the dev .env + certs/db-cert.pem, docker login ghcr.io (read:packages).
  4. nginx server block + certbot cert for mcp-internal.
  5. Add deploy-dev.yml + the DEV_DEPLOY_SSH_* secrets; open SG for the runner.
  6. First dev deploy: run deploy-dev.yml with ref=feat/via-claude-mail-source.
  7. Run the end-to-end test above.

Open items / risks

  • ~~SG for the runner~~ — RESOLVED: dev box SG already allows 22/80/443 from 0.0.0.0/0 (matches prod). Only a deploy keypair + authorized_keys entry is needed, not an SG change.
  • Restart resilience: with the EIP attached the public IP is now stable across stop/start, but the box historically loses pm2/serve state on restart (separate concern, tracked elsewhere). The dev container uses --restart unless-stopped, so it self-recovers.
  • 5011 free-port check before wiring nginx.