XMiDT

Talaria service discovery of Caduceus

Hello,

I’m running Xmidt on Kubernetes and am struggling to get my Talaria (v 0.5.3) to get the addresses of Caduceus (v 0.2.7) out of Consul.
In the Talaria configuration I have:

      outbound:
        method: "POST"
        eventEndpoints:
          default: ["http://caduceus-1:8080/api/v3/notify","http://caduceus-2:8080/api/v3/notify"]
        enableConsulRoundRobin: true

I set up the addresses I want as the defaults to sanity check that these work, which they do. The problem is, only the addresses I set as default are picked up, the addresses from Consul seem to be ignored.
When I start Talaria, I see:

{"caller":"outbounder.go:107","level":"info","msg":"Using consul round robin on service discover","service":"caduceus","ts":"2020-06-05T21:05:40.440644549Z"}
{"eventMap":{"default":["http://caduceus-1:8080/api/v3/notify","http://caduceus-2:8080/api/v3/notify"]},"level":"info","ts":"2020-06-05T21:05:40.44075283Z"}
{"datacenter":"","instances":2,"level":"info","passingOnly":false,"service":"caduceus","tags":"[]","ts":"2020-06-05T21:05:40.453073558Z"}
{"datacenter":"","eventCount":0,"level":"info","msg":"subscription monitor starting","passingOnly":false,"service":"caduceus","tags":null,"ts":"2020-06-05T21:05:40.469794275Z"}
{"datacenter":"","eventCount":1,"instances":["http://caduceus-1:8080","http://caduceus-2:8080"],"level":"error","msg":"service discovery update","passingOnly":false,"service":"caduceus","tags":null,"ts":"2020-06-05T21:05:40.469918836Z"}

The addresses I have in Consul are the same, http://caduceus-1:8080 and http://caduceus-2:8080.
I’m not sure why the last line is an error?

Thanks!
Tom

Screenshot of my Consul:

It looks like you have it configured correctly. The defaultEndpoints will be overwritten by the consul service discovery. The last log line while not an error is used to show that consul is working correctly. Have you tried setting up a wrp-listener to see if events are going through the system?

Hello,

Thanks a lot for the quick response.
The problem is, when I set this in Talaria:

 outbound:
        method: "POST"
        eventEndpoints:
          default: ["http://dummy:8080/api/v3/notify"]
        enableConsulRoundRobin: true

I see this in the logs:

{"caller":"outbounder.go:107","level":"info","msg":"Using consul round robin on service discover","service":"caduceus","ts":"2020-06-09T17:21:18.74736954Z"}
{"eventMap":{"default":["http://dummy:8080/api/v3/notify"]},"level":"info","ts":"2020-06-09T17:21:18.747510461Z"}
{"datacenter":"","instances":2,"level":"info","passingOnly":false,"service":"caduceus","tags":"[]","ts":"2020-06-09T17:21:18.767749892Z"}
{"datacenter":"","eventCount":0,"level":"info","msg":"subscription monitor starting","passingOnly":false,"service":"caduceus","tags":null,"ts":"2020-06-09T17:21:18.794944845Z"}
{"datacenter":"","eventCount":1,"instances":["http://caduceus-1-lb:8080","http://caduceus-2-lb:8080"],"level":"error","msg":"service discovery update","passingOnly":false,"service":"caduceus","tags":null,"ts":"2020-06-09T17:21:18.795032236Z"}
{"error":"dial tcp: lookup dummy on 172.20.0.10:53: no such host","level":"debug","msg":"All HTTP transaction retries failed","retries":1,"ts":"2020-06-09T17:21:20.406716612Z","url":"http://dummy:8080/api/v3/notify"}
{"caller":"workerPool.go:81","error":"Post \"http://dummy:8080/api/v3/notify\": dial tcp: lookup dummy on 172.20.0.10:53: no such host","level":"error","msg":"HTTP transaction error","ts":"2020-06-09T17:21:20.406801033Z"}

with the last line repeating.
With my original configuration, I get messages at a wrp listener. The addresses caduceus-1:8080 and caduceus-2:8080 work, but it seems they are being picked up by the “default” Talaria configuration and not from Consul, which is a bit clearer in this example.

It seems you can recreate this using the xmidt/xmidt repo, if I clone https://github.com/xmidt-org/xmidt and modify the docFiles/talaria-0.yaml file:

      eventEndpoints:
        default: http://dummy:6000/api/v3/notify
      enableConsulRoundRobin: true

I see similar output:

{"caller":"outbounder.go:107","level":"info","msg":"Using consul round robin on service discover","service":"caduceus","ts":"2020-06-09T17:53:34.368836571Z"}
{"eventMap":{"default":["http://dummy:6000/api/v3/notify"]},"level":"info","ts":"2020-06-09T17:53:34.368924984Z"}
{"datacenter":"","instances":0,"level":"info","passingOnly":true,"service":"caduceus","tags":"[stage=dev flavor=docker]","ts":"2020-06-09T17:53:41.263943818Z"}
{"datacenter":"","eventCount":0,"level":"info","msg":"subscription monitor starting","passingOnly":true,"service":"caduceus","tags":["stage=dev","flavor=docker"],"ts":"2020-06-09T17:53:41.400204938Z"}
{"datacenter":"","eventCount":1,"instances":[],"level":"error","msg":"service discovery update","passingOnly":true,"service":"caduceus","tags":["stage=dev","flavor=docker"],"ts":"2020-06-09T17:53:41.400262003Z"}
{"datacenter":"","eventCount":2,"instances":["http://caduceus:6000"],"level":"error","msg":"service discovery update","passingOnly":true,"service":"caduceus","tags":["stage=dev","flavor=docker"],"ts":"2020-06-09T17:53:58.345065619Z"}
{"error":"dial tcp: lookup dummy on 127.0.0.11:53: server misbehaving","level":"debug","msg":"retrying HTTP transaction","retry":1,"statusCode":0,"ts":"2020-06-09T17:54:07.561572063Z","url":"http://dummy:6000/api/v3/notify"}

Thanks,
Tom

I am looking more into this but I believe this is a bug. Thank you for catching this.

I created an issue to track this. https://github.com/xmidt-org/talaria/issues/136