=== verify_all.sh started at 2026-04-03T18:28:32Z ===
0Chain Comprehensive Verification
Domain: test.zus.network  Apps: test.vult.network / test.blimp.software / test.bolt.holdings
0box: https://0box.test.zus.network  |  Time: Fri Apr  3 08:28:32 PM CEST 2026

══════════════════════════════════════════════════
  Chain Health
══════════════════════════════════════════════════
  [PASS] miner-1 running (restarts=0)
  [PASS] miner-2 running (restarts=0)
  [PASS] miner-3 running (restarts=0)
  [PASS] miner-4 running (restarts=0)
  [PASS] sharder-1 running (restarts=0)
  [PASS] sharder-2 running (restarts=0)
  [PASS] Sharder-1 responding (round 3419)
  [PASS] Sharder-2 responding (round 3419)
  [PASS] Chain advancing (round 3419 → 3425)
  [PASS] 4/4 miners responding
  [WARN] Cannot determine Kafka lag (chain=3419, 0box=0)

══════════════════════════════════════════════════
  Backend Service Health
══════════════════════════════════════════════════
  [PASS] 0dns reachable (HTTP 200)
  [FAIL] 0box API — HTTP 000 — http://127.0.0.1:9081/
  [PASS] zauth container running
  [PASS] zvault container running
  [PASS] ZS3 server reachable (HTTP 200)
  [FAIL] Elasticsearch — HTTP 000 — http://127.0.0.1:9200/_cluster/health
  [PASS] Gotenberg render service (port 3010) reachable
  [PASS] 0box block_worker → local chain (https://test.zus.network/dns)
  [PASS] 0box deployment_mode=3 (auth bypass — Vult/Blimp/Bolt work without Firebase)
  [PASS] Crawler container running
  [PASS] Crawler allocation valid (expires Sun May  3 08:28:10 PM CEST 2026)
  [FAIL] 0box DB: 0 active miners — Atlus blank
  [FAIL] 0box DB: 0 active sharders
  [WARN] 0box DB: 0 available blobbers — Blimp allocation may fail
  [FAIL] Kafka pipeline: 0 snapshots in 0box DB — events not flowing
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka
  [PASS] Crawler allocation has 11 blobbers (all 11 regular blobbers)

══════════════════════════════════════════════════
  CORS — Vult / Blimp / Bolt
══════════════════════════════════════════════════
  [PASS] CORS zauth ← test.vult.network: origin=https://test.vult.network credentials=true
  [PASS] CORS zauth ← test.blimp.software: origin=https://test.blimp.software credentials=true
  [PASS] CORS zauth ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true
  [PASS] CORS zauth ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true
  [PASS] CORS sharder01 ← test.vult.network: origin=https://test.vult.network credentials=true
  [PASS] CORS sharder01 ← test.blimp.software: origin=https://test.blimp.software credentials=true
  [PASS] CORS sharder01 ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true
  [PASS] CORS sharder01 ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true
  [PASS] CORS miner01 ← test.vult.network: origin=https://test.vult.network credentials=true
  [PASS] CORS miner01 ← test.blimp.software: origin=https://test.blimp.software credentials=true
  [PASS] CORS miner01 ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true
  [PASS] CORS miner01 ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true
  [PASS] CORS 0box ← test.vult.network: origin=https://test.vult.network credentials=true
  [PASS] CORS 0box ← test.blimp.software: origin=https://test.blimp.software credentials=true
  [PASS] CORS 0box ← test.bolt.holdings: origin=https://test.bolt.holdings credentials=true
  [PASS] CORS 0box ← test.atlus.cloud: origin=https://test.atlus.cloud credentials=true
  [FAIL] WebSocket NOT working on 0box port 9081 (HTTP 000) — Fix: bash scripts/deploy_local.sh services

══════════════════════════════════════════════════
  Web App Serving (Vult / Bolt / Blimp / Explorer)
══════════════════════════════════════════════════
  [PASS] vult running on port 3003 (HTTP 200)
  [PASS] bolt running on port 3002 (HTTP 200)
  [PASS] blimp running on port 3006 (HTTP 200)
  [PASS] explorer running on port 3001 (HTTP 200)
  [PASS] UI sub-pages: all 12/12 pages return 200/301
  [WARN] 0box /v2/blobbers: HTTP 502
  [WARN] 0box /v2/graph-txns-count: HTTP 502
  [WARN] UI button API endpoints: 1/3 OK, 2 failing
  [PASS] vult custom domain https://test.vult.network/ reachable (HTTP 200)
  [PASS] bolt custom domain https://test.bolt.holdings/ reachable (HTTP 200)
  [PASS] blimp custom domain https://test.blimp.software/ reachable (HTTP 200)
  [PASS] explorer custom domain https://test.atlus.cloud/ reachable (HTTP 200)
  [PASS] vult: .env has no stale dev/prod URLs
  [PASS] bolt: .env has no stale dev/prod URLs
  [PASS] blimp: .env has no stale dev/prod URLs
  [PASS] shared: .env has no stale dev/prod URLs
  [PASS] Web app processes OK (4/4 ports serving, 00 Next.js procs)

══════════════════════════════════════════════════
  Vult — Storage App Checks
══════════════════════════════════════════════════
  [INFO] Checking Vult custom domain navigation pages (https://test.vult.network)...
  [PASS] Vult domain pages: all 4/4 reachable at https://test.vult.network
  [INFO] Faucet: pouring tokens to local.json...
  [PASS] Faucet: poured tokens successfully
  [FAIL] 0box /v2/twilio/phone/verify/signup UNREACHABLE (HTTP 502) — 0box down or nginx not routing
  [FAIL] Login endpoint UNREACHABLE (HTTP 502) — 0box down
  [WARN] 0box /v2/freestorage: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></cente (will fall back to regular allocation)
  [PASS] Allocation created (regular fallback): e984e37602ecd8ac...
  [FAIL] Allocation e984e37602ecd8ac... not found on chain (creation likely failed) — skipping upload tests
  [FAIL] Could not create allocation — skipping upload/download/carousel/replace checks
  [INFO] Second wallet flow: create wallet → fund → allocate → upload → switch...
  [PASS] Second wallet created (vw2_3800608)
  [PASS] Second wallet allocation: 404cb73c0a32206c...
  [WARN] Second wallet upload: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: 404cb73c0a32206c77abf5fb7d45333df682d41510e16e8c01bc203ecf22b058, error: record not found
  [WARN] Switch wallet: second wallet files not visible: Error fetching the allocation allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: 404cb73c0a32206c77abf5fb7d45333df682d41510e16e8c01bc203ecf22b058, error: record not found
  [PASS] No stale dev/devtest URLs in service configs

══════════════════════════════════════════════════
  Blimp — File Manager Checks
══════════════════════════════════════════════════
  [PASS] Blimp app serving on port 3006
  [PASS] Blimp auth: 0box deployment_mode=3 (no Firebase required)
  [FAIL] 0box: only 0 regular blobbers — Blimp 'no provider match'
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka
  [PASS] Blimp carousel: Gotenberg container healthy (port 3010)
  [PASS] Blimp carousel: Gotenberg reachable from 0box container — carousel path OK
  [WARN] Blimp carousel: 0box /v2/convert-to-pdf HTTP 0000 (expected 400/401)
  [PASS] Faucet: tokens poured for Blimp allocation
  [PASS] Blimp allocation created: e984e37602ecd8ac...
  [WARN] Blimp upload text: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found
  [WARN] Blimp upload PDF: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found
  [WARN] Blimp upload PNG: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found
  [WARN] Blimp: PDF download failed: Error fetching the allocation allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found
  [WARN] Blimp Settings add blobber: no free blobber to add (all in use)
  [WARN] Blimp Settings repair: Error fetching the allocation. allocation_fetch_error: Error fetching the allocation.internal_error: can't get allocation: error retrieving allocation: e984e37602ecd8ac5201638b45bc63d73bd2416bf8b91d9deb514bd6819b16fc, error: record not found
  [WARN] Blimp upload-only mode: Error updating allocation:couldnt_find_allocation: Couldn't find the allocation required for update
  [WARN] Blimp freeze mode: Error updating allocation:couldnt_find_allocation: Couldn't find the allocation required for update
  [PASS] Blimp: .env has no stale dev/prod URLs

══════════════════════════════════════════════════
  Bolt — Wallet App Checks
══════════════════════════════════════════════════
  [PASS] Bolt app serving on port 3002
  [PASS] Faucet SC: pour transaction works
  [PASS] Bolt wallet balance: Balance: 412.498527 ZCN (0.74 USD)
  [PASS] Bolt staking: 3 miners with non-zero stake (visible in Bolt UI)
  [PASS] Bolt auth: 0box deployment_mode=3
  [PASS] Bolt send: wallet has balance=4124985270000 (send transaction possible)
  [PASS] Bolt: .env has no stale dev/prod URLs

══════════════════════════════════════════════════
  Atlus — Explorer Dashboard Data
══════════════════════════════════════════════════
  [PASS] Miners on chain: 4
  [PASS] Miners with stake: 3/4
  [PASS] Sharders with stake: 2/2
  [PASS] Validators on chain: 12 (staked: 12, active: 12)
  [PASS] Blobbers on chain: 16 (enterprise: 5)
  [WARN] No challenge history — crawler may need time to generate activity
  [INFO] Checking blobber health endpoints...
  [PASS] All 16 blobbers healthy
  [WARN] Provider rewards: no providers have earned rewards yet
  [INFO] Fix: check Kafka pipeline flowing — bash scripts/deploy_local.sh fix-kafka
  [WARN] Latest block has 0 transactions (chain idle)
  [WARN] 0box graph write-price: no data points (resp: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad)
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka  (populates graph data)
  [WARN] 0box graph challenges: no data points (resp: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad)
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka  (populates graph data)
  [WARN] 0box graph token-supply: no data points (resp: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad)
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka  (populates graph data)
  [WARN] 0box graph txns-count: no data points (resp: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad)
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka  (populates graph data)
  [WARN] 0box graph used-storage: no data points (resp: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad)
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka  (populates graph data)
  [WARN] 0box graph total-allocations: no data points (resp: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad)
  [INFO] Fix: bash scripts/deploy_local.sh fix-kafka  (populates graph data)
  [FAIL] Elasticsearch not reachable — Atlus search broken
  [INFO] Fix: check elasticsearch container — docker logs elasticsearch --tail 20
  [PASS] Atlus/Explorer app serving on port 3001
  [INFO] Checking Atlus custom domain navigation pages (https://test.atlus.cloud)...
  [PASS] Atlus domain pages: all 7/7 reachable at https://test.atlus.cloud

══════════════════════════════════════════════════
  Blobber Health
══════════════════════════════════════════════════
  [PASS] 9/9 regular blobbers responding
  [PASS] 3/3 enterprise blobbers responding
  [PASS] All infrastructure blobbers health-checked within last hour
  [PASS] Blobber delegate wallets match local.json

══════════════════════════════════════════════════
  Provider Rewards, Crawler & JWT
══════════════════════════════════════════════════
  [FAIL] POST /v2/jwt/token → 000 (JWT creation failed — check 0box deployment_mode=3)
  [PASS] Crawler allocation 10fef084ea9a574a… is active (not finalized)
  [PASS] Crawler allocation has 11 blobbers (need 9)
  [PASS] All blobbers 01-09 present in crawler allocation
  [PASS] 8 blobber(s) have challenge rewards > 0
  [WARN] 3 blobber(s) with 0 rewards: blobber03 blobber08 blobber09 — ensure they have data (crawler uploads)

══════════════════════════════════════════════════
  VERIFICATION SUMMARY
══════════════════════════════════════════════════
  Domain:   test.zus.network  (prefix: test)
  Vult:     https://test.vult.network/
  Blimp:    https://test.blimp.software/
  Bolt:     https://test.bolt.holdings/
  Atlus:    https://test.atlus.cloud/

  PASS: 86   WARN: 27   FAIL: 13

  Failed checks:
    ✗ 0box API — HTTP 000 — http://127.0.0.1:9081/
    ✗ Elasticsearch — HTTP 000 — http://127.0.0.1:9200/_cluster/health
    ✗ 0box DB: 0 active miners — Atlus blank
    ✗ 0box DB: 0 active sharders
    ✗ Kafka pipeline: 0 snapshots in 0box DB — events not flowing
    ✗ WebSocket NOT working on 0box port 9081 (HTTP 000) — Fix: bash scripts/deploy_local.sh services
    ✗ 0box /v2/twilio/phone/verify/signup UNREACHABLE (HTTP 502) — 0box down or nginx not routing
    ✗ Login endpoint UNREACHABLE (HTTP 502) — 0box down
    ✗ Allocation e984e37602ecd8ac... not found on chain (creation likely failed) — skipping upload tests
    ✗ Could not create allocation — skipping upload/download/carousel/replace checks
    ✗ 0box: only 0 regular blobbers — Blimp 'no provider match'
    ✗ Elasticsearch not reachable — Atlus search broken
    ✗ POST /v2/jwt/token → 000 (JWT creation failed — check 0box deployment_mode=3)

  13 FAILED — see above

  ALL fixes must go into scripts/deploy_local.sh or test code.
  Never make manual one-off changes on the server.

=== verify_all.sh finished at 2026-04-03T18:30:30Z exit=1 ===