Skip to content

Dev MCP Server on Easysource-internal — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking. This is an ops/infra plan: most "tests" are verification commands or observations, not unit tests.

Goal: Stand up a dev deployment of hq-mcp-server on the Easysource-internal box, reachable at https://mcp-internal.hirequotient.com/mcp, deployable on-demand from any branch via a new deploy-dev.yml, and verified end-to-end through a real claude.ai connector before feature branches merge to main.

Architecture: Additive dev path parallel to prod. Prod (uat-vm, main-only, mcp.hirequotient.com) is untouched. Dev = one ~/hq-mcp-server clone on the box with a dev .env, a Docker container on 127.0.0.1:5011 behind an nginx TLS vhost, fed by a manual deploy-dev.yml that builds the chosen branch's image on-demand then SSH-deploys it via the existing scripts/deploy.sh.

Tech Stack: GitHub Actions, Docker, GHCR, nginx, certbot, PM2/Docker on Ubuntu, Node 20, MCP Streamable-HTTP.

Global Constraints

  • Target box: Easysource-internal, instance i-0aa0dcabda4ba8f5e, account bcg-tech-access, EIP 35.153.20.238.
  • Public host: mcp-internal.hirequotient.com (DNS A → EIP, done).
  • Dev container: name hq-mcp, 127.0.0.1:5011 → container :4100 (deploy.sh defaults).
  • Reuse internal secrets from the existing local/box .env; change only PUBLIC_BASE_URL, ALLOWED_HOSTS, ENV/NODE_ENV.
  • Never commit .env or certs. Never touch main, build.yml, deploy.yml, or the prod box.
  • Git identity jeevan-hq <jeevan@hirequotient.com>; no Co-Authored-By/Claude-Session trailers.
  • Work branch: feat/dev-mcp-deploy (already created).

Task 1: Author deploy-dev.yml (on-demand build + SSH deploy)

Files: - Create: .github/workflows/deploy-dev.yml

Interfaces: - Consumes: scripts/deploy.sh (existing) via MCP_CONTAINER/MCP_HOST_PORT defaults; GHCR image ghcr.io/hirequotient/hq-mcp-server:<sha>. - Produces: a manually-dispatchable workflow with a ref input that builds that ref's image and deploys it to the dev box. - Secrets required (Task 3): DEV_DEPLOY_SSH_HOST, DEV_DEPLOY_SSH_USER, DEV_DEPLOY_SSH_KEY, DEV_DEPLOY_SSH_PORT (optional).

  • [ ] Step 1: Write the workflow
name: Deploy (dev)

# MANUAL ONLY. Builds the chosen ref's image on-demand, pushes to GHCR, then
# SSH-deploys it to the Easysource-internal dev box. Never triggered by push.
# Prod's build.yml / deploy.yml (uat-vm, main-only) are untouched.
on:
  workflow_dispatch:
    inputs:
      ref:
        description: 'Branch / tag / SHA to deploy to dev'
        required: true
        default: 'feat/via-claude-mail-source'

concurrency:
  group: deploy-dev
  cancel-in-progress: false

jobs:
  build:
    name: Build & push dev image (GHCR)
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    outputs:
      sha: ${{ steps.resolve.outputs.sha }}
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ inputs.ref }}
      - id: resolve
        run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/hirequotient/hq-mcp-server:${{ steps.resolve.outputs.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy:
    name: Deploy to Easysource-internal
    needs: build
    runs-on: ubuntu-latest
    environment: internal
    steps:
      - name: SSH -> checkout ref + deploy.sh
        uses: appleboy/ssh-action@v1.2.0
        with:
          host: ${{ secrets.DEV_DEPLOY_SSH_HOST }}
          username: ${{ secrets.DEV_DEPLOY_SSH_USER }}
          key: ${{ secrets.DEV_DEPLOY_SSH_KEY }}
          port: ${{ secrets.DEV_DEPLOY_SSH_PORT || 22 }}
          command_timeout: 20m
          script_stop: true
          envs: DEPLOY_REF
          script: |
            set -euo pipefail
            cd ~/hq-mcp-server
            git fetch origin --prune --tags
            git checkout "${{ inputs.ref }}"
            git pull --ff-only || true
            ./scripts/deploy.sh
  • [ ] Step 2: Lint the workflow YAML

Run: cd /Users/admin/hq-mcp-server && python3 -c "import yaml,sys; yaml.safe_load(open('.github/workflows/deploy-dev.yml')); print('YAML OK')" Expected: YAML OK (If actionlint is available: actionlint .github/workflows/deploy-dev.yml → no errors.)

  • [ ] Step 3: Commit
cd /Users/admin/hq-mcp-server
git add .github/workflows/deploy-dev.yml
git -c user.name="jeevan-hq" -c user.email="jeevan@hirequotient.com" \
  commit -m "ci(dev): add manual deploy-dev workflow (build ref image + SSH deploy to Easysource-internal)"

Task 2: Add .env.dev.example documenting the dev deltas

Files: - Create: .env.dev.example

Interfaces: - Produces: a committed, secret-free reference of the exact dev .env deltas, so the on-box .env (Task 4) is reproducible.

  • [ ] Step 1: Write the example (NON-SECRET keys + placeholders only)
# Dev overrides for the Easysource-internal box. Copy the box's existing/local
# working .env, then set these three keys. All other keys (MONGO_URI, REDIS_URL,
# JWT_SECRET_TOKEN, GOOGLE_CLIENT_ID/SECRET, etc.) are REUSED unchanged from the
# internal values — do NOT commit real secrets here.
PORT=4100
ENV=development
# BASE ONLY — no /mcp suffix (app appends /mcp and /oauth/google/callback itself).
PUBLIC_BASE_URL=https://mcp-internal.hirequotient.com
ALLOWED_HOSTS=mcp-internal.hirequotient.com
BE_BASE_URL=https://easysource-internal.hirequotient.com/api
EASYSOURCE_WEB_BASE=https://easysource-internal.hirequotient.com
# Mongo cluster uses MONGODB-X509 → REQUIRED (deploy.sh mounts certs/db-cert.pem
# at /run/secrets/db-cert.pem; cert = box BE's /etc/databaseCert/dev_prod_mongodb.pem).
MONGO_CERT_PATH=/run/secrets/db-cert.pem
  • [ ] Step 2: Verify no real secret values are present

Run: cd /Users/admin/hq-mcp-server && grep -nE "(mongodb\+srv|redis://|eyJ|AIza|GOCSPX)" .env.dev.example && echo "LEAK" || echo "clean" Expected: clean

  • [ ] Step 3: Commit
cd /Users/admin/hq-mcp-server
git add .env.dev.example
git -c user.name="jeevan-hq" -c user.email="jeevan@hirequotient.com" \
  commit -m "docs(dev): add .env.dev.example documenting dev host/env overrides"

Task 3: Deploy SSH key + GitHub secrets (manual, out-of-repo)

Files: none (GitHub repo settings + box ~/.ssh/authorized_keys).

  • [ ] Step 1: Generate a dedicated deploy keypair (local)
ssh-keygen -t ed25519 -f ~/.ssh/hq-mcp-dev-deploy -N "" -C "hq-mcp-dev-deploy"

  • [ ] Step 2: Add the PUBLIC key to the box (as the deploy user, e.g. ubuntu): Append ~/.ssh/hq-mcp-dev-deploy.pub to the box's ~/.ssh/authorized_keys. Verify: ssh -i ~/.ssh/hq-mcp-dev-deploy <user>@35.153.20.238 'echo ok' → ok.

  • [ ] Step 3: Set repo secrets (Settings → Secrets → Actions, or gh):

  • DEV_DEPLOY_SSH_HOST=35.153.20.238 (or mcp-internal.hirequotient.com)
  • DEV_DEPLOY_SSH_USER=<deploy user>
  • DEV_DEPLOY_SSH_KEY=<contents of ~/.ssh/hq-mcp-dev-deploy> (private key)

  • DEV_DEPLOY_SSH_PORT=22 Verify: gh secret list -R HireQuotient/hq-mcp-server shows the four DEV_DEPLOY_*.

  • [ ] Step 4: Create the internal Environment (optional approval gate) in repo Settings → Environments.

Task 4: On-box provisioning (clone + dev .env + cert + GHCR login)

Files: on box ~/hq-mcp-server/.env, ~/hq-mcp-server/certs/db-cert.pem.

  • [ ] Step 1: Confirm prereqs & free port 

    docker --version && command -v git
ss -ltnp | grep -E ':5011\b' && echo "5011 IN USE — pick another" || echo "5011 free"

  • [ ] Step 2: Clone the repo 

    cd ~ && git clone https://github.com/HireQuotient/hq-mcp-server.git && cd hq-mcp-server

  • [ ] Step 3: Create the dev .env — copy the internal values, then apply the three deltas from .env.dev.example. Place certs/db-cert.pem if the box uses Mongo X509 (else keep the plain MONGO_URI and drop MONGO_CERT_PATH). Verify: test -f .env && echo ".env present".

  • [ ] Step 4: GHCR login (read:packages) 

    echo "$GHCR_PAT" | docker login ghcr.io -u <gh-user> --password-stdin
    Verify: Login Succeeded.

Task 5: nginx TLS vhost for mcp-internal.hirequotient.com

Files: on box /etc/nginx/sites-available/mcp-internal.conf (+ symlink to sites-enabled).

  • [ ] Step 1: Write the server block 

    server {
    server_name mcp-internal.hirequotient.com;
    location / {
        proxy_pass http://127.0.0.1:5011;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;            # Streamable-HTTP/SSE
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }
    listen 80;
}

  • [ ] Step 2: Enable + test config 

    sudo ln -s /etc/nginx/sites-available/mcp-internal.conf /etc/nginx/sites-enabled/
sudo nginx -t
    Expected: syntax is ok / test is successful.

  • [ ] Step 3: Issue TLS cert (certbot rewrites the block to 443) 

    sudo certbot --nginx -d mcp-internal.hirequotient.com --non-interactive --agree-tos -m jeevan@hirequotient.com
sudo systemctl reload nginx
    Verify: cert issued, block now listen 443 ssl.

Task 6: First dev deploy + health

  • [ ] Step 1: Add the dev redirect URI in Google Cloud (owner): https://mcp-internal.hirequotient.com/oauth/google/callback on the existing OAuth client.

  • [ ] Step 2: Run the workflow — Actions → "Deploy (dev)" → Run workflow → ref=feat/via-claude-mail-source. Or: gh workflow run deploy-dev.yml -R HireQuotient/hq-mcp-server -f ref=feat/via-claude-mail-source.

  • [ ] Step 3: Verify container + health on the box 

    docker ps --filter name=hq-mcp
curl -fsS http://127.0.0.1:5011/healthz
    Expected: container running; /healthz 200.

  • [ ] Step 4: Verify public endpoint Verify from an allowed network: https://mcp-internal.hirequotient.com/healthz → 200; https://mcp-internal.hirequotient.com/mcp reachable.

Task 7: End-to-end connector test (acceptance gate)

  • [ ] Step 1: In claude.ai, add a second connector "Easysource (dev)" → https://mcp-internal.hirequotient.com/mcp; complete Google OAuth. Verify tools list loads.

  • [ ] Step 2: Against a test org, run a tool that emits an internal-team mail (create project / update vetting). Verify the internal mail subject carries [via MCP] (sync sites), and for async (insights / outreach EMAIL STATS) that project.vettingSource / workflowLog.source = mcp in the DB and the outreach-handler mail is tagged. Loki: service_name="outreach-handler".

  • [ ] Step 3: Confirm the prod connector (mcp.hirequotient.com) and prod mail are unaffected (spot-check a prod mail has no tag change from this).

  • [ ] Step 4: Green → the feature branch is cleared to merge to main (existing prod build + manual deploy.yml path).

Self-Review

  • Spec coverage: exposure (Task 5), hostname (Tasks 4/5/6), deploy trigger + on-demand build (Task 1), BE target (Task 2 .env), EIP/DNS (done pre-plan), OAuth (Task 6 Step 1), end-to-end test (Task 7), runbook order preserved. ✓
  • Placeholder scan: none (.env.dev.example intentionally uses documented placeholders for secrets, verified secret-free in Task 2 Step 2).
  • Consistency: container hq-mcp / port 5011 / host mcp-internal.hirequotient.com / EIP 35.153.20.238 used identically across tasks. ✓

Execution status — 2026-09-09

LIVE: https://mcp-internal.hirequotient.com/mcp (dev MCP on Easysource-internal).

Done: - EIP 35.153.20.238 allocated + attached; easysource-internal + mcp-internal DNS re-pointed. ✓ - Task 1 deploy-dev.yml (ref-injection hardened) + Task 2 .env.dev.example — committed on feat/dev-mcp-deploy. ✓ - Box provisioning (Task 4/5): ayush added to docker group; repo seeded via git bundle on feat/via-claude-mail-source (731ba13); dev .env (reused secrets + host/env overrides + MONGO_CERT_PATH=/run/secrets/db-cert.pem); Mongo X509 cert copied from /etc/databaseCert/dev_prod_mongodb.pem → certs/db-cert.pem; one-off BUILD_LOCAL=1 image built; container hq-mcp on 127.0.0.1:5011 (--restart unless-stopped). ✓ - nginx vhost + certbot TLS (expires 2026-12-08, auto-renew) + 80→443 redirect. ✓ - Verified: /healthz → 200 {"ok":true}; /mcp → 401 (OAuth challenge, proxy OK). ✓

Remaining (user-driven / CI wiring): - Google Cloud: add redirect URI https://mcp-internal.hirequotient.com/oauth/google/callback to the existing OAuth client. - Task 7: add claude.ai connector → the dev URL, OAuth login, run a tool, verify [via MCP]. - Task 3 + CI path: generate deploy keypair → box authorized_keys; set DEV_DEPLOY_* GitHub secrets; give the box a read-only git credential (CI deploy job runs git fetch origin); then deploy-dev.yml replaces the manual bootstrap as the repeatable deploy.

Gotchas hit during bring-up (2026-09-09)

  1. PUBLIC_BASE_URL must be the BASE, no /mcp — the app appends /mcp (resource) and /oauth/* itself. A /mcp suffix doubles it (resource=.../mcp/mcp) and breaks OAuth discovery + the Google callback path. Verify via /.well-known/oauth-authorization-server matching prod's shape.
  2. Mongo cert must be world-readable — the container runs as non-root appuser, but the cert copied from the BE is 0640 ayush:ayush, so the app throws EACCES on /run/secrets/db-cert.pem → Mongo never connects → users.findOne() buffers 10s → the OAuth callback hangs. Fix: chmod 644 certs/db-cert.pem. (Re-apply whenever the cert is re-copied.)