Observer in Erlang/Elixir release

When I teach OTP I always show observer application. Observer is a graphical user interface capable of displaying supervision trees and providing information about processes. It is really cool to show how processes get restarted and what structure they follow. It is also suitable for using with production systems too!

Erlang example for teaching purposes

You can start observer by typing this inside Erlang shell.

observer app
observer app

Cowboy and ranch are great examples of using OTP principles in action.

WARNING: You need at least Erlang 18 installed to work with cowboy examples from master.

git clone
cd cowboy/examples/rest_hello_world
make run
(rest_hello_world_example@ observer:start().
** exception error: undefined function observer:start/0

You can easily start the example but not observer.

Cowboy uses to build the apps. It automatically pulls dependencies, compiles code and generates a release. Releases are self contained packages that include Erlang. They can be copied to another server and they should run fine even if Erlang is not installed there (as long as the other server has the same CPU architecture). They also strip all unnecessary applications to make the final output lightweight. This includes observer.

Releases are great for deploying to production but for teaching purposes I would like to use observer with the generated release. How can I do that? After running make run for the first time, ebin directory is created. Edit file ebin/ like this:

{applications, [kernel,stdlib,cowboy,runtime_tools,wx,observer]},

Those three applications: runtime_tools, wx and observer are all you need to launch observer gui. Just type make run again. For producation systems you probably don’t want to include entire observer with wx widgets. It would be better to leave out the graphical part and just add bare minimum to the release so we can inspect it from outside. Lets do it this time with an Elixir example.

Elixir example for production purposes

We can apply similar trick. Lets build an Elixir release using exrm. After adding it as dependency we can run:

MIX_ENV=prod mix compile
MIX_ENV=prod mix release
rel/my_app_name/bin/my_app_name console

This should result in something like this:

iex(my_app_name@MacBook-Pro-Tomasz)1 Node.get_cookie
iex(my_app_name@MacBook-Pro-Tomasz)2 :observer.start
** (UndefinedFunctionError) undefined function :observer.start/0 (module :observer is not available)

As before our release stripped all additional stuff. This time instead of adding all three applications needed to start observer, lets only add bare minimum that will enable inspecting running node from outside. Inside mix.exs application section add :runtime_tools:

applications: [:phoenix, :phoenix_html, :cowboy, :logger, :gettext,
               :phoenix_ecto, :postgrex, :runtime_tools]]

Now repeat those three steps:

MIX_ENV=prod mix compile
MIX_ENV=prod mix release
rel/my_app_name/bin/my_app_name console

But this time open another iex console starting completely different Erlang node:

iex --sname watcher --cookie my_app_name
iex(watcher@MacBook-Pro-Tomasz)1 :"my_app_name@MacBook-Pro-Tomasz"
iex(watcher@MacBook-Pro-Tomasz)2 :observer.start

Now in observer choose from menu Nodes and then my_app_name@MacBook-Pro-Tomasz. Adding just one application to our release (the :runtime_tools) enabled us to connect to our release from outside and start inspecting it! Runtime tools is an application that delivers “low footprint tracing/debugging tools suitable for inclusion in a production system”.

If you want to play with it, try going to Applications tab. You can right click on any process and kill it with arbitrary reason. What happens with children PIDs?


Nested preload in Ecto

I am writing this post, because it took me some time to fully understand how to nest preloads in Ecto. Nesting mechanism isn’t actually difficult, but I find using keyword list syntax in the docs really confusing.

Preloads in Ecto is a cool feature that allows programmer to fetch nested data sets from database.

Lets imagine Phoenix application with couple of models.

mix blog
cd blog
mix phoenix.gen.model User users name
mix phoenix.gen.model Post posts title user_id:references:users
mix phoenix.gen.model Draft drafts title user_id:references:users
mix phoenix.gen.model Likes likes post_id:references:posts user_id:references:users
mix phoenix.gen.model Comment comments text user_id:references:users post_id:references:posts
mix ecto.create
mix ecto.migrate

It is a very basic app structure where a user can have many posts and drafts. Posts can have many comments and likes. We need to set the has_many relations manually inside generated models:

schema "users" do
  field :name, :string
  has_many :posts, Blog.Post
  has_many :drafts, Blog.Draft


schema "posts" do
  field :title, :string
  belongs_to :user, Blog.User
  has_many :comments, Blog.Comment
  has_many :likes, Blog.Like


Lets now seed our database with some data. Key this in in priv/repo/seeds.exs

alias Blog.Repo
alias Blog.User
alias Blog.Post
alias Blog.Draft
alias Blog.Like
alias Blog.Comment
import Ecto.Query

user = Repo.insert!(%User{name: "Kowal"})
post1 = Repo.insert!(%Post{title: "First post", user_id:})
post2 = Repo.insert!(%Post{title: "Second post", user_id:})
draft1 = Repo.insert!(%Draft{title: "First draft", user_id:})
draft2 = Repo.insert!(%Draft{title: "Second draft", user_id:})
like1 = Repo.insert!(%Like{user_id:, post_id:})
like2 = Repo.insert!(%Like{user_id:, post_id:})
like3 = Repo.insert!(%Like{user_id:, post_id:})
like4 = Repo.insert!(%Like{user_id:, post_id:})
comment1 = Repo.insert!(%Comment{text: "asdf", user_id:, post_id:})
comment2 = Repo.insert!(%Comment{text: "asdf", user_id:, post_id:})
comment3 = Repo.insert!(%Comment{text: "asdf", user_id:, post_id:})
comment4 = Repo.insert!(%Comment{text: "asdf", user_id:, post_id:})

Then run:

mix run priv/repo/seeds.exs
iex -S mix phoenix.server

We have now one user with two posts and two drafts. Both posts have two likes and two comments. The author doesn’t know the netiquette and likes its own posts. Twice! But lets keep things simple for the example :) You can try following examples in Elixir console, but first lets import and alias couple of things:

alias Blog.Repo
alias Blog.User
import Ecto.Query

We can now query for all users like this:

Repo.all(from u in User)

It returns a list of %User structs where posts and drafts are not loaded:

posts: #Ecto.Association.NotLoaded

If we want user with its posts, we can preload them:

Repo.all(from u in User, preload: :posts)

For simplicity, lets assume that a preload is either:
* an atom :relation_name
* nested list of atoms [:relation_name OR {:relation_name, preload}, ...]

The nested list part may not be fully clear at first. It means that a list item might be either an atom or a tuple {:relation_name, preload} where preload can be an atom or a list itself. The definition is recursive.

So there are three differnt equivalent solutions to preload posts:

Repo.all(from u in User, preload: :posts)
Repo.all(from u in User, preload: [:posts])
Repo.all(from u in USer, preload: [{:posts, []}])

If we want to preload both posts and drafts, the first option is not applicable:

Repo.all(from u in User, preload: [:posts, 
Repo.all(from u in User, preload: [:posts, 
                                   {:drafts, []}])

It means we can preload even very nested structures:

Repo.all(from u in User, preload: [{:posts, 

Repo.all(from u in User, preload: [{:posts, 

Last example loads users with its posts comments and even users who posted the comments!

The downside of this query is that it makes 6 queries to the database. One for each model. The preload queries look like SELECT something FROM models WHERE id IN [list of ids]. After fetching the data Ecto makes sure that fetched entities will land in proper places inside our nested result. If you use joins, Ecto can load nested results in one query, but it uses a little bit different syntax.

Why I was confused by the keyword list syntax? Lets start with simple example.

Repo.all(from u in User, preload: [posts: :likes])

This looks OK, but now if you want to add drafts at the same level as posts what do you do? This solution:

Repo.all(from u in User, preload: [posts: :likes, :drafts])

obviously results in syntax error. Keyword lists only work at the end of argument list, so lets try something else:

Repo.all(from u in User, preload: [:drafts, posts: :likes])

surprisingly this works even though it is equivalent to:

Repo.all(from u in User, preload: [:drafts, [{:posts, :likes}]])

Actually, we can nest the lists as many times as we want to:

Repo.all(from u in User, preload: [[[[[:drafts, [{:posts, :likes}]]]]]])

This is why I wrote “lets assume” what a nested list of atoms is. We can make the structure arbitrary complicated and confusing.

Even though there is an example of keyword list syntax in the docs I don’t really recommend using it. The structure is not keyword list after all, but nested atom list, so I would stick with atoms and tuples.

What do you think about the syntax for preloads? Let me know in comments.