Introducing Elixir Response Snapshot Testing
I’m excited to introduce an Elixir testing library that I’ve been working on, as well as to explain the general ideas behind it. The library is called Response Snapshot and can be found on hex. The work is based on my rspec-rcv gem which provides a very similar interface and seeks to achieve the same goals.
What is snapshot testing?
I’m borrowing this term from Jest, but I think that the concept is fairly simple and true to the name. Snapshot testing is testing an API response by collecting a valid snapshot of the output and comparing future tests against it. The entire API response can be asserted against with a single line of code, and visually verified as correct by a human.
In addition to being able to quickly test response outputs and ensure their shape doesn’t change over time, the snapshots can be utilized by frontend specs or other client systems as an example of valid output. If the frontend specs always use the most recent snapshots, any changes to the backend which would break the frontend should fail when the frontend specs run.
Where does snapshot testing fall short?
I would hazard a guess that the idea of snapshot testing makes TDD practitioners cringe a bit. In particular, a human has to verify that the snapshot is accurate and then commit that into source control. Humans are prone to errors and so this could be considered worse than standard tests on a response. I do think that there is a place for response assertions in addition to a snapshot, although snapshots significantly lower the barrier to entry on testing responses. Without snapshot testing, the tests for a response can end up being long, redundant, and a chore to update.
One of the big challenges in snapshot testing is handling values that change between
test runs. This is really common with ids, dates, and information generated from
a library like faker. ResponseSnapshot introduces an optional :keys mode which
will not error out if any values are modified. This is most useful when the shape
of the data should be asserted, but not the exact values. Addition and removal of
keys would be an error in this mode. Another solution to avoid using keys mode is
ignored keys. With this, specific path value changes can be ignored. Paths can be
absolute or wildcards.
Getting started in Elixir
To get started in Elixir, follow the installation instructions.
Capturing your first test involves invoking store_and_compare!/2. Let’s see an
example:
test "widgets are listed out", %{conn: conn} do
conn
|> get("/api/widgets")
|> json_response(200)
|> ResponseSnapshot.store_and_compare!(path: "widgets/index.json")
end
In practice, you will want to setup your fixture path base and ignore keys that change between test runs (such as ids and dates):
config :response_snapshot,
path_base: "test/fixtures",
ignored_keys: [
{"id", :any_nesting},
{"created_at", :any_nesting},
]
The output of the test run will be a fixture file located at test/fixtures/widgets/index.json.
This file will include the test file path, the recording time, and the JSON response
of the API.
Next steps
For next steps, I’d like to bring some best practices to the library, such as using Dialyzer. However, the core of the library seems to be operating as expected and includes a lot of lessons learned from rspec-rcv over the past 2 years. Please reach out on Github issues if you see anything you’d like fixed or have any questions!