28 Days - Elixir Node Networking Basics
One of my favorite points in Elixir, some form of magic perhaps, is how simple distributed networking can be. The language itself seems to make establishing networked nodes and sending messages a pain-free exercise. While breaking down networking techniques is more than a single post, I am going to look into some of the basics of networking in Elixir. Specifically, what happens when we connect nodes together?
Nodes in Elixir
In Elixir, a Node could be defined as a single running instance. There can be
multiple nodes running on a single machine. Let’s use this to take a look at some
basics of node communication, before diving into what is going on.
# In shell 1
iex --name [email protected]
# In shell 2
iex --name [email protected]
# In shell 1
Node.self() # :"[email protected]"
Node.list() # []
Node.connect(:"[email protected]") # true
Node.list() # [:"[email protected]"]
# In shell 2
Node.list() # [:"[email protected]"]
# Leave open
From the above interactive example, it’s possible to see that 2 nodes are initially
started, with a particular name of [email protected] or test2. These nodes start as
disconnected, but can be manually connected. Once connected, both nodes are aware of the
other node’s existence.
Finally, we can test out sending a message in the above processes:
# In shell 1
Node.spawn(:"[email protected]", fn -> IO.inspect(Node.self()) end)
#PID<13084.113.0>
:"[email protected]"
In the above example, a function is set to be executed on :"[email protected]". We can see
that the result of the execution is a remote pid (doesn’t start with 0) as well the output
containing the node’s name.
The Node module contains lots of interesting tidbits that are actually implemented pretty thin on top of erlang modules. This post won’t go into the ins and outs of node communication, although it’s suffice to say that communication is generally done in better ways than spawning functions between the nodes.
Diving into a connection
Node.connect provides
a really simple 1-liner over :net_kernel.connect_node/1. Let’s dig into this function and
what some ramifications of using it are.
Deep inside of the erlang OTP libraries, we start to see some interesting code in connection. Specifically, there is code to handle automatic “magical” connection between nodes, vs more explicit connection dependencies.
Digging even further,
we discover that the net_kernel symbol is actually a process on the system. Running Process.whereis(:net_kernel)
will return the pid of this net_kernel process.
The first time that a Node is connected to, that connection is not present in the ets lookup table. This leads to setup being called and initializing that connection.
By digging into Process.whereis(:net_kernel) |> :sys.get_state(), it’s possible to see that
there is a structure like:
{:listen, #Port<0.609>, #PID<0.49.0>,
{:net_address, {{0, 0, 0, 0}, 56479}, 'Steves-MBP', :tcp, :inet},
:inet_tcp_dist}
This state is documented and helps to let us trace the module that will actually connect our nodes together. Finally, we are able to track down the setup code that is creating the TCP socket between the nodes.
One small caveat of connecting to nodes is that “cookies” have to match up. This is
essentially an atom that is initialized on boot and read from ~/.erlang.cookie. Finding
this was pretty challenging in erlang, but I traced it back to
dist_util
module. The code above (even if you can’t read erlang) is doing some interesting challenge/reply
protocol to ensure that the nodes are allowed to talk to each other.
After the connection
Of course, connecting via TCP here is just the beginning. There is significantly more at play, cookies to authenticate nodes and heartbeats to ensure connectivity between nodes for starters. The distributed erlang guide from learnyousomeerlang.com goes over some of these concepts in detail. I’ll be tackling them in a future post as well.
Addendum
It was asked in the Elixir slack group if it’s possible to customize the distribution mechanism from TCP/IP to something else. It is! It’s fairly involved C code, but there is an example walking through a OS socket level distribution.
In addition, some other drivers are provided out of the box, such as a SSL driver for communicating over SSL.
I don’t really read erlang code, so this post was very interesting today. However, a lot of great things can be learned from digging into the erlang code and module documentation and not relying solely on Elixir docs. For instance, the documentation for node networking brings up great points around security and TLS node communication. Dive into the docs and see what you go from there; it might be useful one day when there’s a problem that you just can’t figure out.
Thanks for reading the 5th post in my 28 days of Elixir. Keep up through the month of February to see if I can stand subjecting myself to 28 days of straight writing. I anticipate a few more posts around networking, such as cookie gotchas, distillery release networking, and pg2.