I have, at various times in my career, tried to convince others that there is an awful, awful lot of stuff you can get done with a few copies of nginx.
The first “service mesh” I ever did was just nginx as a forward proxy on dev boxes, so we could reroute a few endpoints to new code for debugging purposes. And the first time I ever heard of Consul was in the context of automatically updating nginx upstreams for servers coming and going.
There is someone at work trying to finish up a large raft of work, and if I hadn’t had my wires crossed about a certain feature set being in nginx versus nginx Plus, I probably would have stopped the whole thing and suggested we just use nginx for it.
I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary. And I kind of feel like Hashicorp missed a big opportunity there. Their DNS solution, while interesting, doesn’t compose well with other things, like putting a cache between your web server and the services.
I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.
> I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary.
To be fair, half of the API Gateways and edge router projects out there are basically nginx with a custom consul-like service bolted on.
You can do get around the nginx Plus requirement by using a module like ngx_mruby to customize the backend selection. I haven't measured the latency, so it may not be suitable for your 10ms example.
Here's a post I wrote on that ~4 years ago that uses an in-process cache [1]. It'd be fairly easy to add an endpoint to update it and pull data from Consul. I agree with you, it's a missed opportunity - there are alternatives, but being able to rely on a battletested server like nginx makes a difference.
It appears that if the consul client has the right permissions it can restart the nginx service after editing the configuration file. It uses the consul templating engine to generate an nginx config file.
The two main components are nerve (health checking of services + writing a "I'm here and healthy" znode into zookeeper, https://github.com/airbnb/nerve) and synapse (subscribes to subtrees in zookeeper, updates nginx/haproxy/whatever configs with backend changes, and gracefully restarts the proxy, https://github.com/airbnb/synapse).
It's fairly pluggable too if you don't want to use haproxy/nginx.
Then you have a dependency on zookeeper when you already have consul... it seems like consul template + nginx or haproxy is the solution hashicorp went with.
I totally agree, especially about being able to serve content out of a cache instead of files. It would simplify some of my configuration especially for static sites that point to a CDN.
I like what Caddy is doing, exposing their entire configuration through a REST interface.
I 100% agree with you, I've been using Consul for four years now to run 100s of services in 1000s of VMs across datacenters distributed globally and not once I saw the need for anything else...
Maybe I just don't have the scale to find service mesh or kubernetes interesting. Nomad however is something I am willing to give a go for stateless workflows that I would usually provision a VM running a single docker container for.
> I have, at various times in my career, tried to convince others that there is an awful, awful lot of stuff you can get done with a few copies of nginx.
under the load point of view, yes. absolutely. no doubt.
under the speed of action, no way. if your k8s cluster is properly managed, you can let developers do most of the operations work themselves, confined into their namespaces, touching only the kind of resources that you tell them to touch.
I personally would advise against using DNS for service discovery, it wasn't designed for that.
The few milliseconds that you get though, most likely is due to your local machine not having DNS caching configured, this is quite common in Linux. Because of that every connection triggers a request to DNS server. You can install unbound for example to do it. nscd or sssd can also be configured to do some caching.
> I personally would advise against using DNS for service discovery, it wasn't designed for that.
It was designed for that but the SRV record requires protocols and their clients to explicitly support it. You can argue that this an unreasonable design choice but load balancers like HAproxy do support SRV records.
Internet as a whole uses it to provide human friendly names.
I'm saying it is not good idea to use DNS for service discovery, there's a way of using it correctly, but it requires software to do the name resolution with service discovery in mind, and you're guaranteed that majority of your software doesn't work that way.
Why you shouldn't use DNS? It's because when you communicate over TCP/IP you need an address that's really the only thing you actually need.
If you use DNS for discovery you probably will set low TTL for the records, because you want to update them quickly, this means for every connection you make you will be checking DNS server providing extra load on the DNS server and adding latency when connecting.
On failure of a DNS server, even if you set a large TTL, you will see an immediate failure on your nodes the reason is that's how DNS cache works. Different clients made the DNS request at different time so the records will expire at different times. If you did not configure a local DNS cache on your hosts (most people don't) then you won't even cache the response and every connection request will go to a DNS server, so upon a failure everything is immediately down.
Compare this to have a service that edits (let say an HAProxy) configuration and populates it with IP addresses. If the source that provides the information goes down, you simply won't have updates during the time, but the HAProxy will continue forwarding requests to IPs (if you use IPs instead of hostnames, then you also won't be affected by DNS outages).
Now there are exceptions to this, certain software (mainly load balancers such as pgbouncer (I think HAProxy also added some dynamic name resolution)) use DNS with those limitations in mind. They basically query DNS service on the start to get IP and then periodically query it for changes, if there's a change it is being applied, if the DNS service is down they will keep the old values.
Since they don't throw away the IPs when a record expires, you don't have this kind of issues. Having said that, majority of software will use system resolver the way DNS was designed to work and will have these issues, and if you use DNS for service discovery, you, or someone in your company will use it with such service and you'll have the issues described above.
>Compare this to have a service that edits (let say an HAProxy) configuration and populates it with IP addresses.
Just edit the hosts file? If you have access to machines that run your code and can edit configuration, and also don't want the downsides of resolvers (pull-based instead of push-based updates, TTLs), DNS still seems like a better idea than some new stacks, plus you can push hosts files easily via ssh/ansible/basically any configuration management software
EDIT: The only issue I see with DNS as service discovery is that you can't specify ports. But usually software should use standard ports for their uses and that's never been a problem in my experience.
This is interesting! Do you have some material on load testing the DNS servers and seeing their breaking point? I've heard as much from other people but never experienced it in practice even using Consul with 0 TTL everywhere.
Perhaps the network infrastructure team always scaled it correctly behind the scenes but they never once complained about the amount of DNS queries.
DNS is fairly lightweight and if you have one local on premises, it might be less noticeable, especially if latency is not critical (in previous places I worked that was the setup, we still had a local cache on every host and I would encourage doing that, it increases resiliency). If latency is critical, not having a cache adds extra round trip on every connection initiated.
If you have hosts on public cloud and use DNS server that is also shared with others the latency typically might be bigger and on high number of requests you might also start seeing SERVFAIL on large number of requests.
I can't find the forum post anymore, but people who had applications that were opening large number of connections (bad design of the app imo, but still) had huge performance degradation when they moved from c4 to c5 instances. It turned out that this was because of the move from Xen to Nitro (based on KVM).
Side effect of using Xen was that the VM Host was actually caching DNS requests by itself, from which all guests benefited. In the KVM, all DNS requests were going directly to the DNS server.
> I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.
Don't resolve DNS inline rather on every DNS update, resolve it and insert new IP addresses.
Correct me if I'm wrong, but I believe Consul, lacking a mesh of its own, is leveraging the early 1990's era trick of using round robin DNS to split load over available servers.
Caching those values for very long subverts the point of the feature.
By way of correction: Consul does not simply "round robin" DNS requests unless you configure it in a particularly naive manner.
Prepared queries [1] and network tomography (which comes from the Serf underpinnings of the non-server agents) [2] allow for a much wider range of topologies just using DNS without requiring proxies (assuming well behaved client software, which is not a given by any stretch).
Furthermore, Consul _does_ have a mesh as around 2 years ago - [3].
You are correct though that long caches subvert much of the benefit.
Not really - resolve all backend servers to IPs and list all of them as the nginx backends. When a backend server is removed, update nginx backends.
Round-robin balancing using DNS towards a small cluster is silly - you know when any new instance is added to the pool or removed from a pool, so why not push that load balancing onto the load balancer which in your case is nginx?
Whatever is the technology that you use to register the active backends in the DNS, rather than doing name => ip address lookup per request, you can resolve all those names => ip address maps upon the service being brought up/taken down and push the resolved map as a set of backends into nginx config, thus removing the need to query DNS per request.
The first “service mesh” I ever did was just nginx as a forward proxy on dev boxes, so we could reroute a few endpoints to new code for debugging purposes. And the first time I ever heard of Consul was in the context of automatically updating nginx upstreams for servers coming and going.
There is someone at work trying to finish up a large raft of work, and if I hadn’t had my wires crossed about a certain feature set being in nginx versus nginx Plus, I probably would have stopped the whole thing and suggested we just use nginx for it.
I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary. And I kind of feel like Hashicorp missed a big opportunity there. Their DNS solution, while interesting, doesn’t compose well with other things, like putting a cache between your web server and the services.
I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.