All, I could use some real help, I’m getting a strange error message and cannot seem to figure this out.
Subject: Multi-party actAs across participants fails NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT even after fixing party multi-hosting – what are we missing?
Setup
Canton 3.5.5 (open-source dpm distribution, dpm 1.0.21, JDK 17), one
Canton process running three participants via docker-compose:
bank– admin-api :5001, ledger-api :5002, JSON API :7575depositor– admin-api :5011, ledger-api :5012, JSON API :7576beneficiary– admin-api :5021, ledger-api :5022, JSON API :7577
One local synchronizer, escrow-domain, bootstrapped via
bootstrap.synchronizer_local("escrow-domain"). All three participants
connect via connect_local. Four parties: PrimaryIssuer and
EscrowMediator hosted on bank, Depositor on depositor,
Beneficiary on beneficiary.
Directly confirmed all three participants are on the same synchronizer:
DEBUG-CONN [bank] connected synchronizers: Vector(Synchronizer 'escrow-domain')
DEBUG-CONN [depositor] connected synchronizers: Vector(Synchronizer 'escrow-domain')
DEBUG-CONN [beneficiary] connected synchronizers: Vector(Synchronizer 'escrow-domain')
Minimal reproducer
A JSON Ledger API v2 CreateCommand for a template whose signatory
clause is owner :: issuer :: beneficiaries (three parties, one per
participant), submitted to depositor’s JSON API (:7576) with actAs
listing all three:
POST http://localhost:7576/v2/commands/submit-and-wait-for-transaction
{
"commands": {
"commandId": "create-multi-1787926458995020000",
"actAs": [
"Depositor::1220d98fd877599dda57f6322895568120162bdb41443784a3c9c092c609a8a1a0ba",
"PrimaryIssuer::12201a7bb5b49510309a0cdb9b747b1308df42b2b32d1ce471098cea211ab903fa99",
"Beneficiary::1220316661edbf3df988e2be03f8a2039729f2ab0c4dda1c18357097c6a2c0d8c1bb"
],
"userId": "Depositor",
"commands": [
{
"CreateCommand": {
"templateId": "<pkgid>:Test.StablecoinEscrowTest:MockHolding",
"createArguments": {
"owner": "Depositor::1220d98fd877...",
"amount": "100.0000000000",
"issuer": "PrimaryIssuer::12201a7bb5b4...",
"assetId": "PROBE-1787926458995020000",
"beneficiaries": ["Beneficiary::1220316661ed..."]
}
}
}
]
}
}
Response:
HTTP 404
{
"code": "NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT",
"cause": "Not connected to a synchronizer on which this participant can submit for all submitters",
"context": {
"participant": "depositor",
"unknownSubmitters": "List(bank::12201a7bb5b4..., beneficiary::1220316661ed..., depositor::1220d98fd877...)"
},
"errorCategory": 11,
"grpcCodeValue": 5
}
Note unknownSubmitters lists all three actAs parties, including
depositor::... itself – the party that IS locally hosted on the
participant this request was sent to.
Two-party case that works fine, same topology, submitted to bank’s
JSON API (:7575):
{
"commands": {
"actAs": [
"PrimaryIssuer::12201a7bb5b4...",
"Depositor::1220d98fd877..."
],
"userId": "PrimaryIssuer",
"commands": [{ "ExerciseCommand": { "templateId": "<InterfacePkg>:Escrow.Interface:Escrow", "contractId": "...", "choice": "Activate", "choiceArgument": {} } }]
}
}
This one routes and executes fine (progresses to a different, unrelated
error further downstream) every time. Any combination involving
beneficiary fails identically to the above, every time, regardless of
which participant’s JSON API receives the request or the order of parties
in actAs.
What we’ve tried
1. Confirmed synchronizer connectivity is not the issue (see debug
output above – all three participants connected to the same synchronizer).
2. Found and fixed a real bug in our own bootstrap script. The
original topology-authorization step had each of the three participants
try to unilaterally add itself to every party’s PartyToParticipant
mapping:
// ORIGINAL (broken) -- every cross-participant attempt here silently failed
allMappings.foreach { case (partyId, host) =>
Seq(bank, depositor, beneficiary).foreach { target =>
ignore(target.topology.party_to_participant_mappings.propose(
party = partyId,
newParticipants = Seq((host.id, ParticipantPermission.Submission)),
store = TopologyStoreId.Synchronizer(syncId),
mustFullyAuthorize = true
))
}
}
Canton logged (visible once we stopped swallowing the exception):
TOPOLOGY_UNAUTHORIZED_TRANSACTION(5,...): Topology transaction is missing
authorizations by namespaces=Set(<depositor-namespace>, <beneficiary-namespace>)
and keys=Set()
i.e. bank has no authority to make a topology statement about a party
whose namespace key it doesn’t hold. Makes sense in hindsight – we fixed
this with the real two-sided flow: each party’s home participant
proposes a mapping listing all three desired hosts (mustFullyAuthorize = false), then each additional host independently submits the identical
proposal to add its own countersignature:
val newParticipants = Seq(
(bank.id, ParticipantPermission.Submission),
(depositor.id, ParticipantPermission.Submission),
(beneficiary.id, ParticipantPermission.Submission)
)
ignore(host.topology.party_to_participant_mappings.propose(
party = partyId, newParticipants = newParticipants,
store = TopologyStoreId.Synchronizer(syncId), mustFullyAuthorize = false
))
allParticipants.filter(_ != host).foreach { cosigner =>
ignore(cosigner.topology.party_to_participant_mappings.propose(
party = partyId, newParticipants = newParticipants,
store = TopologyStoreId.Synchronizer(syncId), mustFullyAuthorize = false
))
}
Verified this actually worked by querying the resulting mappings directly
on all three participants afterward:
DEBUG [bank] party=Beneficiary::1220316661ed... participants=Vector(
(PAR::bank::12201a7bb5b4...,Submission),
(PAR::depositor::1220d98fd877...,Submission),
(PAR::beneficiary::1220316661ed...,Submission))
DEBUG [depositor] party=Beneficiary::1220316661ed... participants=Vector(...) // identical
DEBUG [beneficiary] party=Beneficiary::1220316661ed... participants=Vector(...) // identical
Every party, queried from every participant, shows all three participants
as hosts with Submission permission – fully merged, no partial/pending
state, no errors logged anywhere in this step.
3. Despite that, the exact same NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT
persists, byte-for-byte identical error, for any combination involving
beneficiary.
4. Also granted ledger-api user rights in case this was a separate
authorization layer:
depositor.ledger_api.users.rights.grant(id = "Depositor", actAs = Set(depId, cbId, benId))
beneficiary.ledger_api.users.rights.grant(id = "Beneficiary", actAs = Set(benId, cbId, depId))
No change in behavior.
5. Tried settle time – up to 30s between the topology fix taking
effect (confirmed via query) and the actual command submission, in case of
async propagation lag between the topology store and whatever the command
service’s routing check reads. No change.
6. Party order in actAs doesn’t matter. depositor-first vs
bank-first vs beneficiary-first: same error whenever beneficiary is
present.
Question
- Is there a further topology/permission layer beyond
PartyToParticipant
hosting + ledger-api useractAsrights that governs whether a
participant can be the submitting participant for a multi-party
actAscommand spanning other participants – something at the
mediator, sequencer, or synchronizer-parameter level we haven’t touched? - Is
NO_SYNCHRONIZER_ON_WHICH_ALL_SUBMITTERS_CAN_SUBMIT’s routing check
based on something other thanPartyToParticipanttopology at all? If
so, what console command lets us inspect/set it? - Why would
bank+depositorsucceed as a submitter combination while
depositor+beneficiaryor all three fail identically, when
PartyToParticipanttopology (per direct query, shown above) looks
completely symmetric across all four parties? Is there something
specific tobeneficiary’s participant/party allocation order (it was
allocated and topology-authorized last in our bootstrap script) that
could leave it in a different state despite the query showing otherwise?
Happy to share the complete devnet_init.canton bootstrap script or full
request/response logs if useful.
Versions: Canton 3.5.5, dpm 1.0.21, JDK 17, docker-compose (not a managed
network), single Canton process hosting all three participants + one local
synchronizer.