MainNet validator boots, participant is healthy, but /api/validator/readyz stays at 503

App Development
, , ,
1

Hi everyone,

I’m trying to bring up a MainNet validator with the Docker Compose setup, and I’ve narrowed the issue down quite a bit, but I’m still blocked on validator readiness.

What is already working:

  • participant comes up healthy

  • participant admin API initializes successfully

  • validator gets the expected participant identity

  • validator acquires ledger connection successfully

  • bootstrap script completes successfully

What is still failing:

  • the validator never becomes ready

  • GET /api/validator/readyz keeps returning HTTP/1.1 503 Service Unavailable

I repeatedly get responses like this from inside the validator container:

===== Thu Apr 23 14:52:20 CEST 2026 =====
HTTP/1.1 503 Service Unavailable

===== Thu Apr 23 14:52:26 CEST 2026 =====
HTTP/1.1 503 Service Unavailable

===== Thu Apr 23 14:52:32 CEST 2026 =====
HTTP/1.1 503 Service Unavailable

In earlier cycles, readyz also timed out and the validator shut down / restarted.

At the same time, on the participant side I still see repeated connectivity failures / timeouts to some sequencer endpoints, especially:

  • sequencer-4.sv-1.global.canton.network.orb1lp.mpch.io:443

  • sequencer-4.sv-1.global.canton.network.lcv.mpch.io:443

  • sequencer-4.sv-1.global.canton.network.mpch.io:443

So at this point it looks like:

  • local Docker / Postgres / participant init is mostly correct

  • the remaining blocker is validator readiness / BFT bootstrap

  • and it may be related to scan / sequencer connectivity on only part of the network

My questions:

  1. Is it expected that validator readiness stays at 503 in this situation?

  2. Can validator bootstrap remain blocked if only some sequencer / scan endpoints are timing out?

  3. Is there an officially supported way to bootstrap with the reachable healthy subset, or do all required endpoints need to be reachable from the validator host?

Any guidance would be appreciated. Thanks.

2

AFIK:

  • validator’sapi/validator/readyzendpoint only returns 200 when the validator has successfully completed its bootstrap process which includes establishing a working BFT connection to the synchronizer. If the participant is receiving connectivity timeouts to sequencer endpoints, the validator app will not complete initialization and will continue returning 503 and eventually restart, which is also the expected watchdog behavior.

  • Core issue seems validator’s default configuration requires access to at least 2/3 of the SVs for both scan and sequencer connections to achieve BFT integrity.

  • you should check if your egress IP has not been whitelisted by that SV yet.

  • in docs there is an bootstrap way. “You may, at your option and own risk, configure connection to a single trusted scan and sequencer as described under validator helm chart configuration, at the cost of losing BFT integrity guarantees.”