PQS Scribe and "Failed to export metrics" warning

App Development
1

When I run the PQS Scribe app, I see the following in the logs, every minute:

[otel.javaagent] [OkHttp http://localhost:4317/...]
WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter -
Failed to export metrics.
Server responded with gRPC status code 2.
Error message: Failed to connect to localhost/127.0.0.1:4317

I’m guessing this is Scribe trying to report metrics to something like Prometheus? Am I on the right track?

How do I either disable that or point it at a different host?

2

How are you running the app? Is that a Docker image (which one?), a direct java invocation (please share all arguments), some other method (please give as many details as you can)?

3

I am running this image:

digitalasset-docker.jfrog.io/participant-query-store:0.1.0

As a Docker Compose service.

  scribe1:
    image: digitalasset-docker.jfrog.io/participant-query-store:0.1.0
    volumes:
      - ./configs:/canton/host/configs:rw
    command: pipeline ledger postgres-document --config /canton/host/configs/scribe1.conf
    depends_on:
      - participant1
      - pqs1
Docker Compose services 
  participant1:
    image: digitalasset/canton-open-source:2.7.6
    volumes:
      - ./configs:/canton/host/configs:rw
      - ./.daml/dist/:/canton/host/dist/:rw
    command: daemon --config "/canton/host/configs/participant1.conf" --bootstrap "/canton/host/configs/participant1.canton" --log-profile container --log-level-root WARN
    expose:
      - 5003
      - 5004
    ports:
      - 5003:5003
      - 5004:5004

  pqs1:
    image: postgres:16
    environment:
      - POSTGRES_PASSWORD=postgres
    ports:
      - 5432:5432

  scribe1:
    image: digitalasset-docker.jfrog.io/participant-query-store:0.1.0
    volumes:
      - ./configs:/canton/host/configs:rw
    command: pipeline ledger postgres-document --config /canton/host/configs/scribe1.conf
    depends_on:
      - participant1
      - pqs1

With this configuration:

source.ledger.host = "participant1"
source.ledger.port = 5003
target.postgres.host = "pqs1"

Resulting in these logs:

2024-01-25 08:26:49 13:26:49.173 I [zio-fiber-116] com.digitalasset.zio.daml.ledgerapi.LedgerApiVersion:32 Ledger API used to connect: 2.4.0 (V1)  application=scribe
2024-01-25 08:26:50 [otel.javaagent 2024-01-25 13:26:50:025 +0000] [OkHttp http://localhost:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/127.0.0.1:4317
2024-01-25 08:26:50 13:26:50.440 I [zio-fiber-74] com.digitalasset.scribe.postgres.document.DocumentPostgres.Service:67 Applying schema  application=scribe
2024-01-25 08:26:50 13:26:50.642 I [zio-fiber-74] com.digitalasset.scribe.postgres.document.DocumentPostgres.Service:70 Schema applied  application=scribe
2024-01-25 08:26:50 13:26:50.713 I [zio-fiber-0] com.digitalasset.scribe.pipeline.pipeline.Impl:33 Starting pipeline on behalf of 'alice::12204b39c69952e9352049abdd2dbbadbc978ceb1c773bbcde99186fe34610821543,bob::1220863961cc050d02f36c1d8fb430c3a22ce5ad362bd43bea6e52016e120e60260c,participant1::12204b39c69952e9352049abdd2dbbadbc978ceb1c773bbcde99186fe34610821543,participant2::1220863961cc050d02f36c1d8fb430c3a22ce5ad362bd43bea6e52016e120e60260c'  application=scribe
2024-01-25 08:26:50 13:26:50.904 I [zio-fiber-0] com.digitalasset.scribe.pipeline.pipeline.Impl:60 Last checkpoint is absent. Seeding from ACS before processing transactions with starting offset '000000000000000007'  application=scribe
2024-01-25 08:26:50 13:26:50.975 I [zio-fiber-837] com.digitalasset.zio.daml.ledgerapi.package:201 Contract filter inclusive of 3 templates and 0 interfaces  application=scribe
2024-01-25 08:26:51 13:26:51.086 I [zio-fiber-0] com.digitalasset.scribe.pipeline.pipeline.Impl:84 Continuing from offset '000000000000000007' and index '0' until offset 'INFINITY'  application=scribe
2024-01-25 08:26:51 13:26:51.132 I [zio-fiber-905] com.digitalasset.zio.daml.ledgerapi.package:201 Contract filter inclusive of 3 templates and 0 interfaces  application=scribe
2024-01-25 08:26:55 [otel.javaagent 2024-01-25 13:26:55:041 +0000] [OkHttp http://localhost:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/127.0.0.1:4317
2024-01-25 08:27:44 [otel.javaagent 2024-01-25 13:27:44:972 +0000] [OkHttp http://localhost:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/127.0.0.1:4317
more log lines 
2024-01-25 08:26:44 Picked up JAVA_TOOL_OPTIONS: -javaagent:/open-telemetry.jar
2024-01-25 08:26:44 [otel.javaagent 2024-01-25 13:26:44:692 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.30.0
2024-01-25 08:26:48 OpenTelemetry Java Agent initialized.
2024-01-25 08:26:48 13:26:48.397 I [zio-fiber-72] com.digitalasset.scribe.appversion.package:14 scribe, version: v0.1.0  application=scribe
2024-01-25 08:26:48 13:26:48.554 I [zio-fiber-76] com.digitalasset.scribe.configuration.package:52 Applied configuration:
2024-01-25 08:26:48 health {
2024-01-25 08:26:48     port="8080"
2024-01-25 08:26:48 }
2024-01-25 08:26:48 logger {
2024-01-25 08:26:48     format=Plain
2024-01-25 08:26:48     level=Info
2024-01-25 08:26:48     mappings {}
2024-01-25 08:26:48     pattern=Plain
2024-01-25 08:26:48 }
2024-01-25 08:26:48 pipeline {
2024-01-25 08:26:48     datasource=TransactionStream
2024-01-25 08:26:48     filter {
2024-01-25 08:26:48         contracts="*"
2024-01-25 08:26:48         metadata="!*"
2024-01-25 08:26:48         parties="*"
2024-01-25 08:26:48     }
2024-01-25 08:26:48     ledger {
2024-01-25 08:26:48         start=Latest
2024-01-25 08:26:48         stop=Never
2024-01-25 08:26:48     }
2024-01-25 08:26:48     oauth {
2024-01-25 08:26:48         accessToken=null
2024-01-25 08:26:48         cafile=null
2024-01-25 08:26:48         clientId=null
2024-01-25 08:26:48         clientSecret=null
2024-01-25 08:26:48         endpoint=null
2024-01-25 08:26:48         parameters {}
2024-01-25 08:26:48     }
2024-01-25 08:26:48 }
2024-01-25 08:26:48 source {
2024-01-25 08:26:48     ledger {
2024-01-25 08:26:48         auth=NoAuth
2024-01-25 08:26:48         host=participant1
2024-01-25 08:26:48         port="5003"
2024-01-25 08:26:48         tls {
2024-01-25 08:26:48             cafile=null
2024-01-25 08:26:48             cert=null
2024-01-25 08:26:48             key=null
2024-01-25 08:26:48         }
2024-01-25 08:26:48     }
2024-01-25 08:26:48 }
2024-01-25 08:26:48 target {
2024-01-25 08:26:48     postgres {
2024-01-25 08:26:48         database=postgres
2024-01-25 08:26:48         host=pqs1
2024-01-25 08:26:48         maxConnections="16"
2024-01-25 08:26:48         password="********"
2024-01-25 08:26:48         port="5432"
2024-01-25 08:26:48         tls {
2024-01-25 08:26:48             cafile=null
2024-01-25 08:26:48             cert=null
2024-01-25 08:26:48             key=null
2024-01-25 08:26:48             mode=Disable
2024-01-25 08:26:48         }
2024-01-25 08:26:48         username=postgres
2024-01-25 08:26:48     }
2024-01-25 08:26:48     schema {
2024-01-25 08:26:48         autoApply="true"
2024-01-25 08:26:48     }
2024-01-25 08:26:48 }
2024-01-25 08:26:48   application=scribe
2024-01-25 08:26:49 13:26:49.173 I [zio-fiber-116] com.digitalasset.zio.daml.ledgerapi.LedgerApiVersion:32 Ledger API used to connect: 2.4.0 (V1)  application=scribe
2024-01-25 08:26:50 [otel.javaagent 2024-01-25 13:26:50:025 +0000] [OkHttp http://localhost:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/127.0.0.1:4317
2024-01-25 08:26:50 13:26:50.440 I [zio-fiber-74] com.digitalasset.scribe.postgres.document.DocumentPostgres.Service:67 Applying schema  application=scribe
2024-01-25 08:26:50 13:26:50.642 I [zio-fiber-74] com.digitalasset.scribe.postgres.document.DocumentPostgres.Service:70 Schema applied  application=scribe
2024-01-25 08:26:50 13:26:50.713 I [zio-fiber-0] com.digitalasset.scribe.pipeline.pipeline.Impl:33 Starting pipeline on behalf of 'alice::12204b39c69952e9352049abdd2dbbadbc978ceb1c773bbcde99186fe34610821543,bob::1220863961cc050d02f36c1d8fb430c3a22ce5ad362bd43bea6e52016e120e60260c,participant1::12204b39c69952e9352049abdd2dbbadbc978ceb1c773bbcde99186fe34610821543,participant2::1220863961cc050d02f36c1d8fb430c3a22ce5ad362bd43bea6e52016e120e60260c'  application=scribe
2024-01-25 08:26:50 13:26:50.904 I [zio-fiber-0] com.digitalasset.scribe.pipeline.pipeline.Impl:60 Last checkpoint is absent. Seeding from ACS before processing transactions with starting offset '000000000000000007'  application=scribe
2024-01-25 08:26:50 13:26:50.975 I [zio-fiber-837] com.digitalasset.zio.daml.ledgerapi.package:201 Contract filter inclusive of 3 templates and 0 interfaces  application=scribe
2024-01-25 08:26:51 13:26:51.086 I [zio-fiber-0] com.digitalasset.scribe.pipeline.pipeline.Impl:84 Continuing from offset '000000000000000007' and index '0' until offset 'INFINITY'  application=scribe
2024-01-25 08:26:51 13:26:51.132 I [zio-fiber-905] com.digitalasset.zio.daml.ledgerapi.package:201 Contract filter inclusive of 3 templates and 0 interfaces  application=scribe
2024-01-25 08:26:55 [otel.javaagent 2024-01-25 13:26:55:041 +0000] [OkHttp http://localhost:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/127.0.0.1:4317
2024-01-25 08:27:44 [otel.javaagent 2024-01-25 13:27:44:972 +0000] [OkHttp http://localhost:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/127.0.0.1:4317
2024-01-25 08:28:44 [otel.javaagent 2024-01-25 13:28:44:948 +0000] [OkHttp http://localhost:4317/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/127.0.0.1:4317
4

Scribe is using the Java version of the OpenTelemetry library. Its defaults are reflected in the warning message.

Setting the environment variable OTEL_JAVAAGENT_ENABLED=false avoids the warning in the logs. In the Docker Compose file, that looks like this:

    environment:
      - OTEL_JAVAAGENT_ENABLED=false

HT: Thanks to Peter Garmaz for pointing me in the right direction!

5

For the record, Digital Asset has updated the Participant Query Store Docker image so that this warning should no longer appear out-of-the box. :raised_hands: