Load & performance testing for Canton apps — early results benchmarking a Token Standard registry

[Dfrnw]

Hi all,

Canton gets sold into regulated finance, where throughput and latency end up in
contracts. But when an application team asks *“can we settle N transactions a
second at p99 under X ms, and what breaks first?”*, there isn’t really a way to
find out. Generic tools (k6, JMeter, Gatling) can generate HTTP load, but they
have no idea what contention is, or the difference between a transaction that
was rejected and one that was merely slow.

I’ve been building a load/performance harness for Canton applications -
explicitly app-level, driving load at an app’s DARs through the Ledger API. Not
protocol or synchronizer performance; that’s Digital Asset’s domain and I’m
staying well clear of it.

It works, and I’ve used it on real code. Sharing early results because I’d
rather find out now whether this is useful to anyone.

Benchmarking OpenZeppelin’s Canton token template

Measured through the real CIP-0056 interface choice `TransferFactory_Transfer`:

• ~15 transfers/second, p50 ~400 ms, p99 ~1.2 s
• Stops scaling around 22 transfers/s offered
• Their DvP allocation path runs at the same throughput as a plain transfer,
  despite doing strictly more work per operation

The interesting part isn’t the rate, it’s what limits it. Every failure
under concurrency is `CONTRACT_NOT_FOUND` - a second transfer reaching an input
holding that another had already archived. The bottleneck is input-holding
selection, not the factory contract. Their implementation archives inputs
first, deliberately, for a contention guarantee; this measures what that costs
when things run in parallel.

For a wallet author that’s actionable: your achievable transfer rate is governed
by your UTXO selection strategy, not by the registry you’re talking to.

One more result that surprised me

Reaching a registry’s factory requires explicit disclosure (the factory is
signed by the admin, so a wallet can’t see it). Attaching that created-event
blob to every submission costs about 3% throughput but raises p99 by ~45%
(803 ms → 1172 ms). The blob is ~576 bytes and rides on every transfer. It shows
up in the tail, not the median.

I haven’t seen that number published anywhere.

Caveats, up front

All of this is a single-participant sandbox on one laptop. They’re floor
numbers. What transfers to a real deployment is the *shape* - where the
bottleneck is and how the system behaves past it - not the absolute rate. Real
figures need a proper deployment, which is exactly what I’d want funding for.

Also: CIP-0104 traffic cost is plumbed and returns the standard’s shape, but a
sandbox has no traffic control, so it reports zero. Cost-per-transaction is
still an open question.

What I’m actually asking

1. Does anyone have this problem right now? Specifically: a team with a
   launch or an SLA who needs a capacity answer and doesn’t have one.
2. What would you want measured on your app that isn’t in the list above?
3. Is anyone already doing this internally? I’d rather join in than duplicate.

Happy to run it against your app and share the results - the workloads are JSON
files and the reports are self-contained HTML.

Thanks,

dfrnw

[Dfrnw]

Ran it against OpenZeppelin’s Canton token template - third-party code… not mine. Numbers in the image

[Dfrnw]

Going deeper on input selection, and it turned out to be more predictable than I expected.

A random pick fails when it lands on a holding the wallet has already spent. Over a run the spent fraction of the pool grows from zero to f, so the average failure rate should be about half of that - and it should not depend on how many submissions are in flight.

contention ~= f / 2 where f = holdings spent / pool size

I put that prediction in the workload generator before running anything, so what follows is a test of it rather than a curve fitted afterwards.

Fixing operations and concurrency, varying only pool depth:

Contention halves every time the pool doubles, mean error about two points across an eight-fold range.

The result that changed how I think about this

Holding pool depth constant and raising concurrency from 8 to 32 - four times as many submissions in flight - moved contention from 12.5% to 11.7%.

Essentially nothing. If contention were concurrent submissions colliding with each other, that should have moved sharply. It didn’t, because for the most part they are not colliding with each other. They are reaching for coins the wallet has already spent.

That is a different mental model from the one I started with, and I suspect a different one from the one a lot of wallet code is written against.

The law also covers the earlier numbers

The single-wallet figure I posted before - 48.8% - comes from a much deeper turnover regime: that run spent about 80% of its pool, where the model predicts around 40%. The 12.5% here comes from a run that spent 22%, where it predicts 11%.

Same law, four-fold different pool conditions, both inside a couple of points. Which is the useful part: those two numbers looked contradictory and aren’t. Contention wasn’t varying because of the wallet count or the concurrency, it was varying because of how much of the pool each run burned through.

Extending that: the wallet-count sweep I posted earlier moved two things at once, since spreading the same load across more wallets also reduces how far each wallet’s pool is drawn down. I’m re-running it with each arm’s pool sized to its own consumption, which separates the two cleanly and should leave wallet count as the only variable.

Contention is also avoidable outright

Same registry, same wallet, same load. The only change is that each in-flight submission gets a distinct holding instead of picking at random:

Reservation takes contention to zero at every concurrency tested, commits all 240 of 240 offered transfers against 208 with random selection, and costs nothing in latency. At concurrency 32 it is actually faster (p99 1902ms down to 1763ms), and the throughput gain widens under pressure: about 5% at concurrency 8, 15% at 32.

What this gives a wallet author

You can estimate your own contention before writing a line of test code:

contention ~= (transfers / holdings) / 2

And you can drive it to zero by reserving an input per in-flight submission rather than picking at random. Pool depth and selection strategy are the two levers, and neither of them is the registry you picked.

Usual caveats: single participant, in-memory, one laptop. The model under-predicts at high turnover, which is where genuine concurrent collisions begin adding on top - so it is a good estimate and a floor, not an identity.

Thanks to Bernhard Elsner

And still the same open question… if anyone has a load profile they would consider representative of real wallet traffic - wallet count, holdings per wallet, transfer size distribution, overlap between senders - I would rather measure that than keep inventing my own.

Kind regards

dfrwn

[kevmuko]

Hi, Kevin from K2F Labs. We run a self-custodial wallet provider (walley.cc) and a validator on MainNet, with PQS indexing our participant. I went through about 12 months of our data (1.97M committed transactions visible to us, ~620K submitted by our node) to see how much of this holds up outside a sandbox. Some of it does. Some of it I think is the environment talking. Numbers below.

1. The 15/22 tps plateau. We’ve seen our participant commit 30-31 transactions inside a single second on MainNet, with real sequencing and confirmation in the path. Caveat that this is effective_at clustering across everything visible to us, not a controlled load test, but it does put the 22 tps ceiling on the harness/sandbox side rather than on Canton or the app. I’d flag this louder in the writeup honestly, because floor numbers get quoted without the caveats.

2. Latency. On MainNet the commit path is dominated by the synchronizer round (sequencing plus confirmations), which a single-participant sandbox never touches. So the 400ms/1.2s percentiles aren’t a floor or a ceiling for anything deployed. I wouldn’t tune app code against them.

3. The 576-byte blob adding 45% to p99. I’m not certain if this is accurate. If marshalling 576 extra bytes were the cause you’d see it at p50 and in throughput proportionally, not as a p99-only spike next to a 3% throughput dip. That pattern is what tail noise looks like on a single machine. Where disclosure actually costs you on MainNet is traffic: the median transaction we submit pays ~35.5KB (p99 ~57.5KB), which at the current $60/MB extra-traffic price is about $2.13 per transaction. A 576-byte disclosed reference is ~1.6% of that envelope, about 3.5 cents. A fee line item, not a latency cliff. (Also, TransferFactory payloads on MainNet are ~146 bytes median.) I suspect an interleaved A/B with more samples makes most of that 45% disappear.

4. Contention and f/2. The math seems fine. What I’d question is how often the regime it models (random picks from a deep pool, uncoordinated concurrent spends) actually occurs. From our data:

   • Median active Amulet pool across the 19,380 owners we can see is 2 holdings, p90 is 9. Merge automation keeps pools shallow on purpose.
   • Out of ~78K consecutive same-owner spend pairs in the last 30 days, 0.35% happened as separate transactions within 2 seconds of each other.
   • About 40% of the “simultaneous” spends were actually one transaction consuming several inputs (p50 1 input, p90 3).

   So production wallets mostly don’t hit this. They batch inputs into one transaction and serialize per party. Your own 8 vs 32 concurrency run says the same thing: contention stayed flat around 12%, which means the generator was colliding with its own spent inputs, not with concurrent submissions. Reservation is the right fix, agreed, but it’s also what existing wallet automation already does, so I’d call it confirmation rather than a new constraint. Where selection strategy really does matter is treasuries, venues, anything with high fan-out from one party. Those exist (one registry owner we can see has ~465K active holdings) and for that class your model is a genuinely useful sizing tool.

5. Your open items. Two of them we can answer from our own production data.

   Traffic cost: across ~620K submitted transactions, paid traffic per transaction was p50 35,542 bytes, p90 ~44KB, p99 ~57.5KB, max 331KB. At $60/MB that’s roughly $2.13 per transaction at the median, $2.63 at p90, $3.45 at p99, and the worst single transaction cost us about $20. If anything, this is the number I’d want a load harness to surface: at these prices, envelope size dominates the economics of a registry long before latency does. One wrinkle: traffic-based app rewards go to featured confirming parties, not submitters, so the cost side and the reward side of a registry don’t line up, and a sandbox sees neither.

   Load profile: ~19K holding owners visible to us. Pool depth p50 2 / p90 9 / p99 99. Holding lifetime create to spend: p10 5.4s, p50 81s, p90 ~41min. Inputs per transfer p50 1, p90 3. Daily volume 34K-67K transactions, busiest minutes 100-185, sustained average under 1 tps. Very bursty, very shallow pools, almost everything serialized per party.

6. The harness. Worth syncing with DA’s Canton performance work before building this out further. They’re the right owner for the protocol-adjacent layers, and it directly answers your duplication question.

Happy to share more anonymized aggregates if useful.

[Dfrnw]

Kevin, thank you.

Тhis is a far better answer than I expected to get.

Corrections first, because you’re right about most of it.

The 45% p99 on disclosure is wrong and I’m withdrawing it. Your reasoning was right, and the sample size makes it worse: that run was 60 transfers. At n=60 the p99 is just the single worst observation, so I was comparing one GC pause against another. The medians were identical at about 400ms, which was the actual result, and I should have said that instead. I’ve redrawn the card from that post with the row removed.

I also wrote that I hadn’t seen that number published anywhere. That should have
been a hint.

The rule I’m using now is roughly 10 × 100/(100−p) samples before a percentile carries information, so about 1,000 for a p90 and 10,000 for a p99. A couple of my other numbers don’t survive it either.

Your 1.6%-of-envelope framing is more useful than the thing it replaces. A fee line, not a latency problem.

The 22/s is also wrong, but differently. That came from a ramp using random input selection. On the same laptop, same registry, with each in-flight submission reserving a distinct input, it commits 240 of 240 at 42.1/s, that’s in the selection card further up the thread. So 22/s was never a ceiling of anything, it was the wallet shedding load by colliding with itself, which is the same effect the rest of the thread is about. Calling it “scaling stops” was careless.

Which also means your 30-31/s isn’t the contradiction it looks like. And the generator isn’t the limit either, for what it’s worth: it does 8,400-10,300 ops/s against an in-process mock and has been run to 2M operations. What’s being measured at 15-42/s is the sandbox.

Your point about caveats not travelling is the one I’ll change habits over. “Floor numbers” in a footer doesn’t help when a headline tile gets screenshotted on its own.

Latency percentiles: withdrawn as guidance, for the reason you give. A single participant never touches sequencing or confirmation.

On contention I think we agree on the mechanism, and you’ve corrected me on how often it matters. Your reading of the 8-vs-32 result is what I concluded too:
they’re colliding with their own spent inputs, not with each other. But median pool depth 2, and 0.35% of same-owner spends landing inside 2 seconds, is decisive. “Confirmation rather than a new constraint” is fair for consumer wallets.

Two things in your numbers were more useful than that makes it sound. Inputs per
transfer of p50 1 and p90 3: I’d just finished measuring the multi-input case, and it’s worse than I’d assumed. A transfer gathering k inputs fails if any one of them is stale, so it goes as 1−(1−p)^k rather than k·p. I ran k = 1, 2, 3, 4, 6, 8, with the k=3 and k=6 predictions fixed before the runs. “k inputs behaves like a pool k times smaller” understates it roughly twofold, and that’s at your p90, not somewhere exotic.

And the 465K-holding owner is the case I should have been leading with rather than wallets.

Your profile also fixes something I owe this thread. I said I’d re-run the wallet-count sweep with each arm’s pool sized to its own consumption. I was going to pick those pool sizes myself. With p50 2 and p90 9 there’s no reason to.

Traffic is where I think you’re most right, and where I’ve been looking at the wrong thing entirely. I’d written traffic cost off as blocked, because CIP-0104 reads unmetered on a sandbox. But the prepare endpoint hands back the prepared transaction next to the zeroed estimate, and the size is just sitting there. A minimal transfer, three fields and one Decimal, comes to about 11.7 KB. Which is your point from the other side: the payload is tiny and the envelope isn’t.

Two questions:

Your 35.5KB median against ~146-byte payloads. My guess is the dominant term is
informee count, since the confirmation request carries encrypted views per informee. That would make envelope cost mostly a function of how many parties a design makes stakeholders, and barely a function of what the payload holds. Does that match what you see?

And would it actually be useful to you? Meaning “this design costs about $X per transaction, and here’s which decision is driving it”, before deploying rather than after.

On the aggregates - pool depth, inputs per transfer and the traffic percentiles especially. What I’d do with them is stop inventing load profiles. Bursty, shallow, serialised per party, under 1 tps sustained is a very different test from the one I’ve been running.

Noted on rewards going to featured confirming parties rather than submitters. If cost and reward don’t line up for whoever runs a registry, that seems like it matters beyond performance.

And yes on DA, do you have a contact there? I’d rather ask than duplicate.

Thanks again for taking the time on this.

[Dfrnw]

Following up on Kevin’s point that envelope size dominates registry economics long before latency does. He’s right, and it turned out to be measurable without a metered synchronizer, which I had assumed it wasn’t.

The thing I had wrong

CIP-0104 traffic cost reads zero on a sandbox, because no traffic control is configured there. I had been reporting that as UNMETERED and treating cost as blocked until I could get access to a metered network.

But interactive-submission/prepare` interprets a command without submitting it and returns the prepared transaction alongside the zeroed estimate. Its size is a real measurement on any participant, sandbox included. I had been discarding it.

So envelope size needs no permission and no network access. Only the price is external.

What a transfer costs

Same standard `TransferFactory_Transfer`, same six parties, same single-input transfer. Only the registry differs: 11,733 bytes against 13,955, or $0.67 against $0.80 at the $60/MB extra-traffic price.

My first instinct was to report that as “one registry is 19% more expensive”, and I am glad I did not, because it is wrong.

The two registries are not doing the same thing. std-spike completes the transfer. OpenZeppelin, with no preapproval in the context, creates a transfer instruction and a locked holding for the receiver to accept later. So the comparison was a completed transfer against half of a two-step one.

Running each registry’s direct-completion path instead separates them. On that path OpenZeppelin comes in at 11,446 bytes - 287 bytes below std-spike. Its data model is not more expensive, it is slightly cheaper. The entire 2,222-byte gap, and 287 bytes more, is propose/accept versus completing directly: +2,509 bytes, about $0.14 per transfer.

And that still understates it, because a two-step transfer is not finished. The receiver has to accept, which is another transaction with its own envelope. So $0.14 is the floor on what propose/accept costs, not the total.

Worth noting on method: three runs per configuration, and the byte count was identical every time. Not close, identical. That is unlike anything else I have posted here. A latency percentile is a sample from a noisy distribution and needs thousands of observations before it means anything, as I found out the hard way earlier in this thread. Envelope size is not a sample at all, it is a property of the transaction’s shape. One run gets it and repeats confirm it.

Where the bytes go: stakeholders, linearly

The obvious question is what drives it, and the first candidate was informee count, since a confirmation request carries a view per informee. StdTransferFactory carries `users : [Party] as its observer list, so varying that list changes the number of stakeholders while the payload stays byte-identical. Nothing else moves.

create: 878 + 175.5 x N

transfer: 9,110 + 437.8 x N

Both are linear to within about ten bytes across a sixteen-fold range, on envelopes from 1 to 23 KB. But the slope is not the same: a stakeholder costs 175.5 bytes on a simple create and 437.8 bytes on a standard transfer - 2.5x more.

I nearly posted the create figure on its own, which would have been wrong by 150% for the transaction anyone actually cares about. At $60/MB it works out at $0.010 per stakeholder on a create and $0.025 on a transfer - so a party added to a contract seeing 50,000 transfers a day is roughly $1,250 a day.

The useful version of that is not a number to quote but a shape: stakeholder count is priced linearly, and the rate is a property of the transaction. Which is precisely why it needs measuring per operation rather than estimating.

One cross-check worth mentioning, because it is the only independent one I have: the transfer fit predicts 11,737 bytes at six parties, and a separate run measured 11,733. Four bytes out.

Where I would temper that

Stakeholder count is a real mechanism but it is not automatically the dominant term. On the transfer the fixed part is 9,110 bytes, so at six parties stakeholders are about 22% of the envelope, and only past roughly twenty parties do they dominate it.

It is also not what separated the two registries above - both ran with the same six parties, which is what pointed at the settlement model instead.

One caveat on that attribution: I isolated the direct path using each registry’s self-transfer, because a preapproved cross-party transfer would have needed a TransferPreapproval in the context and I have not built that yet. Self-transfer takes the same completing path, but it is a proxy, not the identical operation.

And this does not settle the question I put to Kevin. Mine is a sandbox measurement of a prepared transaction; what a MainNet participant actually pays for is the sequenced request, and whether stakeholder count dominates *there* is still his data to answer, not mine.

Two further caveats. The prepared transaction is not the sequenced confirmation request, which adds encrypted views on top, so every number here is a lower bound. Kevin’s MainNet median of ~35.5KB is 2.6 to 3.1 times my figures, which is the right direction, but that is consistency rather than confirmation. And $60/MB is a quoted price, not one I measured. The tool takes it as a parameter and records it beside any figure it produces, so a cost never appears without the assumption that produced it.

Why this seems worth doing

A wallet author cannot change protocol overhead. A registry author chooses the data model and the stakeholder set, and that choice is priced on every transaction for the life of the registry, currently with no way to see it before deploying.

So, a question for anyone running or building a registry: would a cost-per-transaction figure at design time change what you build? And is there a decision you have already made where you would want to know what it cost?