Understanding Compile Time Dependencies in Elixir - A Bug Hunt
This blog post will cover a fairly trivial but still interesting problem that I encountered at work today. I think it’s worth writing about simply because it’s a bit non-obvious and will probably happen to other people as well. It also is a good time to reinforce our understanding of how the Elixir compiler works and why the order of things matters.
The issue started after upgrading one of our repositories to the latest Elixir version (from 1.6) and upgrading Distillery from 1.5 -> 2.0. The problem manifested itself as our instrumentation disappearing in DataDog. The graphs was there before the deployment, then immediately became empty after.
The immediate thought is that something could have been wrong with 1.7 or the upgrade libraries (and not something we did). However, this seems unlikely given that Elixir 1.7 has been out in the wild for a while now and has had time to get any kinks worked out (and something this major would be unlikely anyways). We power our instrumentation with Instruments as seen in a past post and so we verified that the application was started and configured with all of our probes…everything was visibly okay on this front.
After confirming that instruments should be working, we noticed that DataDog was reporting some stats for our application that looked an awful lot like what we wanted…it was just missing our “.probes” namespace. The problem isn’t that the stats weren’t reporting, but that they were reporting without our probe_prefix.
The Code Problem
Inside of Instruments, there is a line of code which loads the provided probe prefix and places it into a module attribute:
# https://github.com/discordapp/instruments/blob/89a620d181ba4a04ed0ac01c47057c018d645428/lib/probe/definitions.ex#L11
@probe_prefix Application.get_env(:instruments, :probe_prefix)
This line of code happens inside of a defmodule, which means that it is read from at compile time and not at run-time. It would be possible to use Application.get_env/2 at run-time using something like def probe_prefix, but there is nothing wrong with the module attribute approach.
In Distillery 1, all config was provided at compile time and probe_prefix would always be present. In Distillery 2 (with the awesome new config providers), config can optionally be provided at run time; this simplifies and fixes quite a few deployment quirks. Our standard for Distillery 2 config providers has been to move config/prod.exs to rel/config/runtime.exs.
The Fix
In hindsight, we should not have moved the entire contents of config/prod.exs into our config provider. As Paul mentions in his release blog post, “You can still use all of the config files under config/ in your project, but you should use those for compile-time config and default values only”.
The fix is as simple as making sure our runtime.exs file only has dynamic values in it. This means that we provide a config/prod.exs file for static configuration, and a rel/config/runtime.exs file for dynamic configuration.
Compilation vs Run-time
The reason that I liked this small bug is that it reinforces understanding of compilation and how we can very quickly end up blurring the line between run-time and compile-time. As users and authors of code, we have to be cognizant of where are values are coming from and where provided values are going. It often will be perfectly “okay” to do things incorrectly (because there are many situations where it wouldn’t be a problem), but it can be difficult to track down when it is not okay. In particular, this issue looked like probe_prefix was setup correctly, and only become clear through reading of the source code that it was not setup correctly.
Thanks for reading! I’ll be speaking at Lonestar ElixirConf about bringing Elixir to production, looking at both human and tech challenges in doing so.