28 Days - Peerage Deep Dive

I am going to continue off of yesterdays’ post in order to look at how the Peerage library works. Peerage is a pretty awesome library which makes Node discovery in distributed environments very easy.

Peerage Strategies

Peerage offers several strategies which allow for Node discovery. The most common one for me has been DNS, but the others can be just as valuable depending on environment.

At the simplest, a Peerage discovery service must respond to the poll() function. This is a pretty small interface, which would make creating new discovery strategies very easy. The self strategy only returns the current node in a List, which is essentially a no-op.

We can look a little deeper at something that might do some actual connectivity, the list strategy. As we can see from this code, it’s really not complex. The application is setup with a list of Node symbols, and those will try to connect through this strategy.

Now we can finally get into some real work, with the UDP strategy. UDP can be used to connect in a situation where nodes are on the same network, but are generally unware of what other nodes are on the network. Digging into the UDP strategy, we can see that UDP messages are sent over the network, and include the format "Peer#{node()}":

@doc "Broadcast our node name via UDP every 3-7 seconds"
def handle_info(:broadcast, state = %{conn: {addr, port, sock}}) do
  :ok = :gen_udp.send(sock, addr, port, ["Peer:#{ node() }"])
  Process.send_after(self(), :broadcast, :rand.uniform(4_000) + 3_000)
  {:noreply, state}
end

When a UDP packet comes in, matching the format, that node is logged into state.

@doc "Handle UDP packet. If it's a node name broadcast, adds to `state.seen`."
def handle_info({:udp,sock,_,_, "Peer:" <> name}, state = %{seen: ms}) do
  Logger.debug "  -> Peerage.Via.Udp sees: #{ name }"
  :inet.setopts(sock, active: 1)
  {:noreply, %{state | seen: ms |> MapSet.put(name |> String.to_atom)}}
end

Finally, this state is returned synchronously when requested by the poller we saw previously.

I really like this general approach because we can see that there isn’t real magic going on. Nodes are broadcasting their name over the UDP connectionless protocol, and then letting the polling service take over from there (which will be explained below).

Finally, the DNS strategy, can be used in multi-node situations where IPs are mapped to a DNS record. In kubernetes, for example, the nodes can be mapped into a headless service and then resolved from there. The poll mechanism of the DNS strategy is even simpler than UDP. Simply take the provided DNS name, look it up using :inet_res.lookup('domain.com', :in, :a), and then turn the provided IPs into connectable names.

Connection

My favorite part of peerage, by far, is how simple the server is that actually connects nodes together. The polling code triggers the discover method and reschedules for some interval in the future. If you haven’t seen it before, the Process.send_after line here is very common for creating gen servers which do some regular interval’d work.

def handle_info(:poll, state) do
  discover()
  Process.send_after(self(), :poll, interval() * 1000)
  {:noreply, state}
end

Finally, the real simplicity comes out with node connections:

defp discover do
  poll()
  |> only_fresh_node_names
  |> Enum.map(&([&1, Node.connect(&1)]))
  |> log_results
end

The nodes are polled using the mechanisms we saw previously, then compared against currently connected nodes. Node.connect is then called on the symbol provided by the providers, and attempted to connect to. When a Node connects, erlang handles all of the security and heartbeat, as laid out in the previous post.

This simple and elegant library has been a cornerstone for me connecting nodes together in kubernetes. It is great to see that there is no hacky implementations under the covers. It is simply easy to follow and debug Elixir code, falling back on known erlang libraries.

Thanks for reading the 6th post in my 28 days of Elixir. Keep up through the month of February to see if I can stand subjecting myself to 28 days of straight writing. I anticipate a few more posts around networking, such as cookie gotchas, distillery release networking, and pg2.