My summary of Elixir Conf EU 2016

Elixir Conf EU is held annually in some European City. Last year it was Kraków, this time it was Berlin and in 2017 it will probably be in Barcelona. All the talks are about Elixir programming language.

I recognized four main trends during the conference: decoupling, scalability, game programming and hardware. Some of the problems are solved borrowing solutions from networking.

Decoupling

Most of the talks were about decoupling. There are entire classes of problems that come from one concept being responsible for two things.

José Valim (the creator of Elixir language) showed us that simple one for one supervision strategy is so different from other supervisors that it should be in its own module. During last Elixir Conf he was talking about making concurrency easier using processing pipelines. This time he showed that those should be split into couple of subproblems like making processes talk to each other and handling overload by protocols that advertise how much data they can process (which is very similar to TCP sliding window).

The talk about Credo (static code analysis tool for Elixir) showed that we can’t treat all kinds of errors in the same way. This allows prioritizing and tailoring linter to project needs. But the coolest thing about Credo is that it is also great learning tool, because of its friendly and comprehensive messages.

Discovering Processes can be a hard problem if you couple it with supervision tree which is a common practice. Saša showed us tools like gproc and pg2 that separate those two concerns.

By following OTP principles inside Erlang VM you can built resilient systems, but what about connecting with outside world? Andrea focused on talking with external services using TCP in Erlang. It is good idea to decouple the connection handling from protocol logic.

Want to break your monolith? It might be good to hear others people experience in this area. It also shows a thing that I am really excited about which is using Elixir with Elm.

Almost all Elixir developers use Ecto on a day to day basis. Michał Muskała told us that Ecto moves from being your model to being a set of tools that make creating data mappings easy which allows decoupling models from database.

The last talk of the conference was about treating Phoenix as a tool. If you can decouple your application logic from the web and storage capabilities of Phoenix it should be easier to maintain and extend the app.

Game Programming

With help of Phoenix Channels it is now easier than ever to create multiplayer games. Channels allow you to push the game state or updates via websockets to all connected clients in couple of lines of code.

Building Multiplayer Games with Phoenix and Phaser showed that JavaScript part of building web and mobile games can also be nice. With the two biggest problems of multiplayer games made much easier nothing stops you from building your own great online game :)

Another gaming talk showed lessons learned from making three online games in Elixir. Definitely check out how it went!

Scalability

Does Erlang scale? It is easy to make clusters made out 50 to 100 nodes, but after that connecting new nodes to a full mesh becomes expensive. Zandra works on solving that problem. Her talk shows how can we prevent connecting with full mesh and still being able to communicate between all nodes. I saw many parallels to networking. Building spanning trees between nodes, discovering them like in the DNS scheme and even routing messages. Hard topics, but very exciting!

During his keynote Chris showed us the solution to presence problem in distributed system using CRDTs. This feature is not only great for chats, but also for service discovery and other transient data in a cluster. Nice work!

Another scalability talk showed The Road to 2 Million Websocket Connections in Phoenix. It is interesting to see how do you solve scalability problems in the BEAM ecosystem and that it might be easier than it sounds if you know your tools well.

Another very interesting scalability talk was about using Phoenix with Riak Core. This new architectural design can help you make your state distributed in turn making your applications even more fault tolerant. You just have to check it out!

Embedded systems

There is an absolutely amazing set of tools for building embedded systems in Elixir called Nerves. You can see how they work and why they work this way here. It is time to dust my raspberry pi and start hacking :D

There was also a talk about making a robot using Nerves. If you dream of something more than just rearranging pixels on the computer screen it is time to start hacking robots!

 

There were also other nice talks like the one about security and lightning talks. Don’t miss the one by @wojtekmach. I am waiting for all the presentations to be published online. Apart from talks it was a fantastic event where I could met many people passionate about their work. It showed me new perspectives on tools I use every day at work. It motivated me to check out both embedded systems and game programming in Elixir. It is also great to see all the architectural and scalability problems that come up. This are nice problems to have for such a young community. I hope to see you all again in Barcelona!

Posted on by tkowal 0

Observer in Erlang/Elixir release

When I teach OTP I always show observer application. Observer is a graphical user interface capable of displaying supervision trees and providing information about processes. It is really cool to show how processes get restarted and what structure they follow. It is also suitable for using with production systems too!

Erlang example for teaching purposes

You can start observer by typing this inside Erlang shell.

observer:start().
observer app
observer app

Cowboy and ranch are great examples of using OTP principles in action.

WARNING: You need at least Erlang 18 installed to work with cowboy examples from master.

git clone https://github.com/ninenines/cowboy.git
cd cowboy/examples/rest_hello_world
make run
(rest_hello_world_example@127.0.0.1)1 observer:start().
** exception error: undefined function observer:start/0

You can easily start the example but not observer.

Cowboy uses erlang.mk to build the apps. It automatically pulls dependencies, compiles code and generates a release. Releases are self contained packages that include Erlang. They can be copied to another server and they should run fine even if Erlang is not installed there (as long as the other server has the same CPU architecture). They also strip all unnecessary applications to make the final output lightweight. This includes observer.

Releases are great for deploying to production but for teaching purposes I would like to use observer with the generated release. How can I do that? After running make run for the first time, ebin directory is created. Edit file ebin/rest_hello_world.app like this:

{applications, [kernel,stdlib,cowboy,runtime_tools,wx,observer]},

Those three applications: runtime_tools, wx and observer are all you need to launch observer gui. Just type make run again. For producation systems you probably don’t want to include entire observer with wx widgets. It would be better to leave out the graphical part and just add bare minimum to the release so we can inspect it from outside. Lets do it this time with an Elixir example.

Elixir example for production purposes

We can apply similar trick. Lets build an Elixir release using exrm. After adding it as dependency we can run:

MIX_ENV=prod mix compile
MIX_ENV=prod mix release
rel/my_app_name/bin/my_app_name console

This should result in something like this:

iex(my_app_name@MacBook-Pro-Tomasz)1 Node.get_cookie
:my_app_name
iex(my_app_name@MacBook-Pro-Tomasz)2 :observer.start
** (UndefinedFunctionError) undefined function :observer.start/0 (module :observer is not available)

As before our release stripped all additional stuff. This time instead of adding all three applications needed to start observer, lets only add bare minimum that will enable inspecting running node from outside. Inside mix.exs application section add :runtime_tools:

applications: [:phoenix, :phoenix_html, :cowboy, :logger, :gettext,
               :phoenix_ecto, :postgrex, :runtime_tools]]

Now repeat those three steps:

MIX_ENV=prod mix compile
MIX_ENV=prod mix release
rel/my_app_name/bin/my_app_name console

But this time open another iex console starting completely different Erlang node:

iex --sname watcher --cookie my_app_name
iex(watcher@MacBook-Pro-Tomasz)1 :net_adm.ping :"my_app_name@MacBook-Pro-Tomasz"
:pong
iex(watcher@MacBook-Pro-Tomasz)2 :observer.start
:ok

Now in observer choose from menu Nodes and then my_app_name@MacBook-Pro-Tomasz. Adding just one application to our release (the :runtime_tools) enabled us to connect to our release from outside and start inspecting it! Runtime tools is an application that delivers “low footprint tracing/debugging tools suitable for inclusion in a production system”.

If you want to play with it, try going to Applications tab. You can right click on any process and kill it with arbitrary reason. What happens with children PIDs?

Posted on by tkowal 1

Nested preload in Ecto

I am writing this post, because it took me some time to fully understand how to nest preloads in Ecto. Nesting mechanism isn’t actually difficult, but I find using keyword list syntax in the docs really confusing.

Preloads in Ecto is a cool feature that allows programmer to fetch nested data sets from database.

Lets imagine Phoenix application with couple of models.

mix phoenix.new blog
cd blog
mix phoenix.gen.model User users name
mix phoenix.gen.model Post posts title user_id:references:users
mix phoenix.gen.model Draft drafts title user_id:references:users
mix phoenix.gen.model Likes likes post_id:references:posts user_id:references:users
mix phoenix.gen.model Comment comments text user_id:references:users post_id:references:posts
mix ecto.create
mix ecto.migrate

It is a very basic app structure where a user can have many posts and drafts. Posts can have many comments and likes. We need to set the has_many relations manually inside generated models:

schema "users" do
  field :name, :string
  has_many :posts, Blog.Post
  has_many :drafts, Blog.Draft

  timestamps
end

schema "posts" do
  field :title, :string
  belongs_to :user, Blog.User
  has_many :comments, Blog.Comment
  has_many :likes, Blog.Like

  timestamps
end

Lets now seed our database with some data. Key this in in priv/repo/seeds.exs

alias Blog.Repo
alias Blog.User
alias Blog.Post
alias Blog.Draft
alias Blog.Like
alias Blog.Comment
import Ecto.Query

user = Repo.insert!(%User{name: "Kowal"})
post1 = Repo.insert!(%Post{title: "First post", user_id: user.id})
post2 = Repo.insert!(%Post{title: "Second post", user_id: user.id})
draft1 = Repo.insert!(%Draft{title: "First draft", user_id: user.id})
draft2 = Repo.insert!(%Draft{title: "Second draft", user_id: user.id})
like1 = Repo.insert!(%Like{user_id: user.id, post_id: post1.id})
like2 = Repo.insert!(%Like{user_id: user.id, post_id: post2.id})
like3 = Repo.insert!(%Like{user_id: user.id, post_id: post1.id})
like4 = Repo.insert!(%Like{user_id: user.id, post_id: post2.id})
comment1 = Repo.insert!(%Comment{text: "asdf", user_id: user.id, post_id: post1.id})
comment2 = Repo.insert!(%Comment{text: "asdf", user_id: user.id, post_id: post2.id})
comment3 = Repo.insert!(%Comment{text: "asdf", user_id: user.id, post_id: post1.id})
comment4 = Repo.insert!(%Comment{text: "asdf", user_id: user.id, post_id: post2.id})

Then run:

mix run priv/repo/seeds.exs
iex -S mix phoenix.server

We have now one user with two posts and two drafts. Both posts have two likes and two comments. The author doesn’t know the netiquette and likes its own posts. Twice! But lets keep things simple for the example :) You can try following examples in Elixir console, but first lets import and alias couple of things:

alias Blog.Repo
alias Blog.User
import Ecto.Query

We can now query for all users like this:

Repo.all(from u in User)

It returns a list of %User structs where posts and drafts are not loaded:

posts: #Ecto.Association.NotLoaded

If we want user with its posts, we can preload them:

Repo.all(from u in User, preload: :posts)

For simplicity, lets assume that a preload is either:
* an atom :relation_name
* nested list of atoms [:relation_name OR {:relation_name, preload}, ...]

The nested list part may not be fully clear at first. It means that a list item might be either an atom or a tuple {:relation_name, preload} where preload can be an atom or a list itself. The definition is recursive.

So there are three differnt equivalent solutions to preload posts:

Repo.all(from u in User, preload: :posts)
Repo.all(from u in User, preload: [:posts])
Repo.all(from u in USer, preload: [{:posts, []}])

If we want to preload both posts and drafts, the first option is not applicable:

Repo.all(from u in User, preload: [:posts, 
                                   :drafts])
Repo.all(from u in User, preload: [:posts, 
                                   {:drafts, []}])

It means we can preload even very nested structures:

Repo.all(from u in User, preload: [{:posts, 
                                    :comments}, 
                                   :drafts])

Repo.all(from u in User, preload: [{:posts, 
                                    [{:comments,
                                      :user},
                                     :likes]},
                                   :drafts])

Last example loads users with its posts comments and even users who posted the comments!

The downside of this query is that it makes 6 queries to the database. One for each model. The preload queries look like SELECT something FROM models WHERE id IN [list of ids]. After fetching the data Ecto makes sure that fetched entities will land in proper places inside our nested result. If you use joins, Ecto can load nested results in one query, but it uses a little bit different syntax.

Why I was confused by the keyword list syntax? Lets start with simple example.

Repo.all(from u in User, preload: [posts: :likes])

This looks OK, but now if you want to add drafts at the same level as posts what do you do? This solution:

Repo.all(from u in User, preload: [posts: :likes, :drafts])

obviously results in syntax error. Keyword lists only work at the end of argument list, so lets try something else:

Repo.all(from u in User, preload: [:drafts, posts: :likes])

surprisingly this works even though it is equivalent to:

Repo.all(from u in User, preload: [:drafts, [{:posts, :likes}]])

Actually, we can nest the lists as many times as we want to:

Repo.all(from u in User, preload: [[[[[:drafts, [{:posts, :likes}]]]]]])

This is why I wrote “lets assume” what a nested list of atoms is. We can make the structure arbitrary complicated and confusing.

Even though there is an example of keyword list syntax in the docs I don’t really recommend using it. The structure is not keyword list after all, but nested atom list, so I would stick with atoms and tuples.

What do you think about the syntax for preloads? Let me know in comments.

Posted on by tkowal 5

What problems does elm solve?

“Yo, check out this new shiny piece of technology!”
“Why?”
“Because it is functional, distributed, cloud enabled, big data, IoT, bla, bla, bla platform”
“But what problem does it solve?”

When I see product descriptions, I often have no idea, what problems do those products solve. Take docker for example: “An open platform for distributed applications for developers and sysadmins”. Now I know why people like docker, but it is only because I used it.

Recently, I’ve been talking with friends about elm and they asked questions like: “Can I use it with Angular or other framework?” or “Is is hard to debug generated JavaScript?” I used elm a little bit and those questions simply do not make sense. People ask them because they tend to think about languages compiled to JavaScript as of syntactic sugar making JS less painful. Elm is so much more than that! Elm makes writing browser apps enjoyable! Lets see what problems does it solve.

Debugging runtime exceptions

Elm forces you to write correct code. Static typing and functions without side effects force the developer to structure the code nicely. If it compiles, it will run without errors. 100% guarantee! The program logic may still be wrong, but hey! We are talking about compilers – not oracles. If someone asks you, how do you debug JavaScript provided by elm, the answer is: you don’t. I was never forced to read generated JavaScript and I don’t have to know, how it works. It just does. Fixing compilation errors is much simpler than dealing with JS errors, because of great error messages.

Templating

One of standard elm libraries is called Html, which is used to generate (surprise, surprise!) HTML. Every element is a function, that takes two arguments: a list of attributes and a list of child elements. Check it out!. This is seriously huge! No more templating languages with their limitations. I have the full power of a real language. I can easily print recursive structures and group elements or entire widgets as functions, that can be parameterized. All things that are hard to do in templating languages, become easy. The notation is also very succinct and readable.

CSS preprocessor

SASS, LESS and other CSS preprocessors give me variables, arithmetic operations and other things, that make CSS more manageable. In Elm I can do the same things out of the box, because style is simply one of attributes. If my templating language gives me the full power of a regular programming language, then I can use it for CSS too! I am not sure, if it is good idea… But it is possible!

Semantic package versioning

Have you ever upgraded a library (only minor release) to see, that it actually changed API? Elm static typing can detect, if the library API changed and enforce incrementing the major version number. Sweet!

Updating DOM

Updating DOM is painful in JavaScript. Multiple functions can update a DOM element. Some of them are triggered by changing some other DOM elements. It can lead to a big mess. John Carmack once said: “Everything that is syntactically legal that the compiler will accept will eventually wind up in your codebase”. It is not enough to be careful. It is natural, that JavaScript/JQuery apps become a mess.

Elm has only one way of updating DOM. It is a function that takes application model and returns Html. No messing with the state – everything is generated from scratch on every model update. OK, not exactly. It works like in ReactJS. It generates a virtual dom, diffs it with the previous virtual dom and applies a minimal set of changes to the actual DOM. It is fast and it is conceptually simple.

Structuring code

Elm doesn’t allow circular dependencies between the modules. Hurray! There is also tutorial about creating reusable code and composition in elm.

Testing

Testing is much easier, when I don’t have to deal with the state. No more mocks! Only functions with arguments and return values.

Signals

Learning elm requires learning couple of unfamiliar concepts too. Functions in elm do not have side effects, so how can I change something on the screen? I can define values changing over time. They are called Signals. The user inputs are signals. Combining user input with the current model produces a new model, so there is a signal of models. Models are translated into views to produce a signal of HTML. In elm every application is just a signal of HTML. It may sound very formal, but it really makes sense.

Signal of user inputs -> user input + previous model = new model -> Signal of models -> model renders to Html -> Signal of Html.

Higher order functions on signals

I use functional programming in my day job, so I am used to transforming lists with higher order functions. The same concept is applied to Signals. It took me some time to grasp, that stream of events can be processed with the same set of functions as normal lists. Beautiful idea!

Addresses and mailboxes

Signals have their mailboxes with addresses. If an element will be used to generate user input, it needs to have an address. It looks a little bit like in Erlang, so I like it very much, but it may seem foreign for newcomer.

Interacting with JS libraries

JavaScript can be hard to write, but there are too many proven libraries to throw it all away. Elm lets developers interact with JS libraries, but they need to show, what types are expected on the elm side of things. It requires some boilerplate, but is easy to do.

Summing up

Elm is really easy to learn, much easier than Haskell for example. Still, it is not exactly effortless. On the other hand, it promises easy to extend, maintainable code. It is not only a language better than JavaScript. It is a full platform for writing browser applications, better than anything I’ve seen so far! And I haven’t even mentioned time traveling debugger yet!

After only couple of days of learning, I felt much more productive writing webapps. So? What are you waiting for?

Posted on by tkowal 6

The principle of immediate feedback

“Getting feedback is good” seems like a commonsense principle everyone agrees with. Code reviews are important, getting a second opinion makes programmer’s life easier. And yet it seems programmers often act as if to purposefully make giving feedback hard (if not impossible). Instead of showing incremental steps of their work while they implement it, they make changes locally on their branches and they push out a swarm of commits as a pull request to review.

There are a couple of problems with that approach. Firstly it makes the reviewer’s job difficult and avoiding that is not just about being kind to others – it’s also a matter of self-interest. It is not enough to just get feedback. In order to make use of it, one needs to get it quickly. If an error is discovered at an early stage of work, it’s easier to correct it.

In this post I would like to show you a couple of trivial and non trivial examples of using that rule. The rule I call “the principle of immediate feedback”.
Immediate feedback helps us learn faster. There are countless records of that, some more surprising than others. An application providing immediate feedback to children proved more effective in teaching kids in overcrowded Malawian classes that qualified teachers in the UK with class sizes of no more than 30.

A question remains – how can I use the principle of immediate feedback in my day to day work?

  1. Spellchecker in emacs
    When I started blogging, I didn’t use any spellchecker. After writing the whole post, people pointed out to me, that I’ve made spelling errors. Instead of discussing actual content, I was focusing on getting this right. The cost of fixing mistake later was relatively high. So I started using a spellchecker after I was done with writing. This was much better, but still took me some time to make everything straight. I also sometimes made mistakes while fixing things, so another spell checker run uncovered new errors. Gosh, this was frustrating! Now I use an interactive spellchecker spacemacs with flycheck-mode and this is wonderfully simple. Even though I get feedback as I type, I still want to perform a full check after I am done. I use M-x flyspell-buffer – it usually doesn’t uncover new errors.

  2. zsh colours
    There are plugins for your command line, that colour your commands, while you are typing. They can make it red, if you made spelling mistake even before you start typing arguments. This may save only seconds of time, but it also makes it easier to stay in flow. It was a little bit annoying to type entire sed expressions just to get error, because I started with sd instead of sed. Now I am using oh-my-zsh with syntax highlighting plugin. If everything is OK, my command is green, otherwise it is red, so I can spot errors even before I hit enter.

  3. compile errors in editor
    If my editor doesn’t show me syntax errors, while I am writing, I lose serious amounts of time on the save – compile – fix cycle. Some people like to turn the colours off, because it forces them to focus on code. That is fine, as long as something flashes, when they use invalid syntax. Without syntax highlighting, you get the same problem as without spell checker. You can introduce errors while fixing errors.

  4. browser auto reload
    If I am working with web project, automating browser reloading can sometimes save huge amounts of time. Some frameworks make it default behaviour now. Recently I am using Phoenix framework which is a web framework written in Elixir programming language Every time I save a file, it gets recompiled, reloaded and browser is refreshed to reflect changes. This isn’t new idea, but it is awesome!

  5. TDD
    TDD is a great example of applying immediate feedback principle. On low level, there is red – green – refactor cycle. Why I have to start with a failing test case? To have immediate feedback, that the test case works. Otherwise it may happen, that the test case always passes. If it was written after the code, I have no way to know that. Then I work until the test is green, so I actually worked on my feedback loop before I started actual work. This is genius! I put effort to make my feedback loop as tight as possible. Fixing mistakes right where I made them is much less time consuming than hunting for them later.

  6. Code Reviews
    As I said before – code reviews are great, but can you make them even better? Can you make the feedback instantaneous? Yes! Pair programming to the rescue! It works like a constant code review. Pair programming produces higher quality code with less bugs. If your manager says, it is too costly to do pair programming, you can reply, that it costs more not to pair program.

  7. CI
    Continues integration is also providing immediate feedback answering the question: “Are my changes working OK with everybody else changes?”. If I push code to CI every half an hour, it simpler to retract the whole commit from repository, than fixing or merging code. Unless you really like merging.

  8. CD
    Continues delivery is like testing applied on your company clients. You can get immediate feedback about your changes and if something doesn’t work properly or clients don’t like it, you quickly roll back. Compare it with monthly releases, where after a month worth of work clients don’t like the changes. Looks like immediate feedback enhances learning not only for individuals, but also for entire companies.

  9. Elon Musk making rockets
    Almost everyone knows, that Elon Musk has built first reusable rocket, but do you know why? Money wasn’t the only reason. Even more important one was to iterate quickly! Do changes and have feedback as fast as possible. Research always requires many failed attempts. Making the feedback loop tight requires making the cost of mistakes as low as possible. “Fail fast!”

  10. Pitching startup ideas.
    Founders who develop they product in secret before big launch usually have much worse results than companies, that start with pitching the idea. They can get feedback before they even start any costly or time consuming tasks. Success lies in details, that you can get right only by asking many people what they think of them. Don’t ever be afraid of someone stealing your idea. Even if someone does steal it, you are way ahead of him, if you had more insight into it. And you do, because it is your idea. This insight comes from others people feedback.

All of those things rely on one principle. Make your feedback loop as tight as possible to iterate quickly. Next time when you’ve made a mistake stop and think: “Was I able to catch the error faster? How?”

I would like to finish with a link to one more resource. There is a great talk by Bret Victor called “Inventing on principle”, where he preaches about the principle of immediate feedback. He has more interesting examples, like “time travelling debuggers”. He is also great speaker, so it is really pleasure to watch the talk. Enjoy!

Posted on by tkowal 5

Failing fast and slow in Erlang and Elixir

I am recently teaching programming with Elixir and Phoenix in Kraków. During classes, I saw, that new Erlang and Elixir programmers have problems with concept of “failing fast”, so I’ll explain it with examples, but first I need to show you…

The Golden Trinity of Erlang

Torben Hoffman in many of his webinars about Erlang programming language mentions “The Golden Trinity of Erlang”
ndc-london-2014-thinking-like-an-erlanger-9-638

The three principles are:

  • Fail fast
  • Share nothing
  • Failure handling

There are many articles explaining how sharing nothing is great for concurrency. Failure handling is usually explained when teaching OTP supervisors. But what does it mean to “fail fast”?

Failing fast principle

“Fail fast” principle isn’t exclusive to Erlang. In agile methodologies, it expands to:

  1. don’t be afraid to try something new;
  2. evaluate it quickly;
  3. if it works – stick with it;
  4. if not – abandon it fast, before it sucks too much money/energy.

This business approach translates almost directly to programming practice in Erlang.

Happy path programming

When I write in Erlang, I usually don’t program for errors. I can treat most errors or edge cases as if they don’t exist. For example:

{ok, Data} = file:read_file(Filename),
do_something_with_data(Data)

There is no code for handling situation, where file does not exist or I don’t have permissions to open it. It makes the code more readable. I only specify business logic instead of myriad of edge cases.

Of course, I can match on some errors. Maybe I want to create a file if it doesn’t exist. But in that case it becomes application logic, so my argument about not programming for edge cases still holds.

case file:read_file(Filename) of
  {error, enoent} -> create_file();
  {ok, Data} -> do_something_with_data(Data)
end,

This style of programming is called “happy path programming”. It doesn’t mean, that I don’t anticipate errors. It just means that I handle them somewhere else (by supervision trees and restarting).

Failing fast case study 1 – reading from a file

This style of programming requires the code to fail quickly, when problem occurs. Consider this code:

{_, Data} = file:read_file(Filename),
do_something_with_data(Data)

Reading the file could actually return {error, Reason} and then I treat the Reason atom as Data. This propagates the error further, where it is harder to debug and can pollute state of other processes. Erlang is dynamically typed language, so do_something_with_data/1 can pass atom many levels down the call stack. The displayed error will say, that it can’t treat atom as text and the bug gets tricky to find. Even functions, that are used purely for their side effects should match on something to check, if they worked, so instead of:

file:write_file(FileName, Bytes)

it is usually better to use:

ok = file:write_file(FileName, Bytes)

Failing fast case study 2 – calling gen_server

It is even more important to fail before sending anything wrong to another process. I once wrote about it in this blog post. Sending messages using module interfaces helps keep the damage made by errors contained. Crashing caller instead of server is “quicker”, so it doesn’t contaminate application state. It allows failure handling strategies much simpler than preparing for all possible edge cases. Most of the time those strategies are based on restarting processes with clean state. Processes and computation are cheap and can be restarted, but data is sacred.

Case study 3 – tight assertions

Lets consider another example. A test suite:

ok = db:insert(Value),
Value = hd(db:get(Query))

It tests database code by inserting single value to empty database and then retrieving it. However, if we assume, that the database was empty before test execution, we can make sure, that id doesn’t return anything else. Second line above is equivalent to:

[Value | _] = db:get(Query)

But I can make the assertion stronger by writing:

[Value] = db:get(Query)

It asserts both value and number of elements in the list. Sweet!

“Fail fast” is another example of applying “immediate feedback principle” in programming. It allows happy path programming, which makes programs more readable, but requires treating each line as an assertion. It is easy to do this with pattern matching.

Failing fast and supervision trees = ♥♥♥

Posted on by tkowal 2

Building docker images for Elixir applications

TL;DR: Use exrm to speed up working with Elixir and Docker. Time of running docker pull dropped from 5m to 16s.

Docker among many other things solves problem of deploys. It makes them easy to perform and repeatable. I can deploy the same docker image many times on different machines. Developers, testers, QAs and Ops can work with almost identical environment. Performing manual tests, automated tests and stress tests in parallel saves a lot of time.

Up to last week during docker pull, docker had to perform number of steps:

During pull docker waits for layers, if next layer depends on it. It means, that it can’t pull Elixir layer before Erlang layer is ready and it can’t pull application before Elixir is ready.
On my development machine (with the base image precached) docker pull took about five minutes to complete. In case, you are interested, we used this docker image, that installed Elixir and Erlang.

As I said before those 5 minutes pulls are performed many times during the day.

5 minutes * number of environments * number of features we want to push to production adds up quickly.

Can we somehowe speed up the process? Yes! Erlang introduces concept of releases. A release is a minimal self contained build. Releases include Erlang runtime, so you don’t have to have Elixir or Erlang installed.

Releases were historically painful to build, so there are tools that do if for you using sane defaults. relx for Erlang and exrm for Elixir.

Now docker pull performs only two steps:

  • Get the base image
  • Pull our application files

And it takes only 16s!

Is there a catch? Yes, there is.

We also liked to perform unit tests in docker image with mix tests, but exrm contains only application code. No tests code, no mix at all.

We use the old image with all dependencies for unit tests. After they finish, we build a release and create new docker image, which is then pulled many times by other teams.

If you are working with Phoenix web framework, there is a great step by step guide for setting up relx with Phoenix.

Posted on by tkowal 1

How to make your console sing?

Problem:

Sometimes I am running command and I have no idea, how long it will take to execute.
There is this project Makefile, which you think only compiles sources,
but actually it fetches dependencies from git repositories and it takes 1 minute to complete.
Or I start copying things and they are havier than I expected.
So I just go to other windows to do some other work and I am simply forgetting about previous task.

Solution:

Add completion sound after every command.

Additional rationale:

“Pragmatic Thinking and Learning: Refactor Your Wetware” [1]
teaches, that to use your full brain potential, you have to use multisensory input.
That is why mechanical keyboard with nice click sound and button reaction are just nicer to type on.
If you spend a lot of time in terminal, you should make it pretty and maybe even add some sounds.

Preface:

Before you yell at me, that I overcomplicated the final solution, I will walk you through the intermediate steps.
If you are in a hurry, just grep for “Solution2”.
I use Mac OSX, iterm2, oh-my-zsh with bira theme and very often work with tmux over ssh.

First thing, how do you play sounds in terminal on Mac?
afplay /System/Library/Sounds/Submarine.aiff
OSX has couple of nice system sounds.
I decided to use Submarine for succesful command completion and Blow for failure.

Solution1a:

My first idea was to use .zshrc
There is a function called precmd(), that is run always just before displaying prompt.
I could use $? variable to decide, which sound shoud I play.

play_success() { afplay /System/Library/Sounds/Submarine.aiff }
play_failure() { afplay /System/Library/Sounds/Blow.aiff }
precmd() { if [ $? -eq 0 ]; then play_success; else play_failure; fi }

Problems:

The sounds were played in the same thread of execution as the terminal.
This meant, that after typing ls, I had to wait entire second for the sound to stop playing,
before I could write anything.
Not cool!
I have to put it in background.

Solution1b:

play_sound() { afplay $1 }
play_sound_in_background() { play_sound $1 & }
play_success() { play_sound_in_background /System/Library/Sounds/Submarine.aiff }
play_failure() { play_sound_in_background /System/Library/Sounds/Blow.aiff }
precmd() { if [ $? -eq 0 ]; then play_success; else play_failure; fi }

Problems:

Doing something in bacground displayed job notifications in my console:

[1] 14099
[1]  + 14099 done       afplay /System/Library/Sounds/Submarine.aiff

Solution1c:

Surround play sound with “()” makes it run in child console, so there is no output.

play_sound() { afplay $1 }
play_sound_in_background() { ( play_sound $1 & ) }
play_success() { play_sound_in_background /System/Library/Sounds/Submarine.aiff }
play_failure() { play_sound_in_background /System/Library/Sounds/Blow.aiff }
precmd() { if [ $? -eq 0 ]; then play_success; else play_failure; fi }

Problems:

It obviously doesn’t work with ssh.
Even if I install my zsh config on a remote machine.
I don’t have the sounds there.
Abandon solution.

Solution2:

Zsh config is local to the machine, but I always use iTerm2,
so maybe it has some helpful features.
Yes, it does!
Triggers [2] can do something every time, some pattern is printed on terminal.
In bira [3] zsh theme, if command fails, than it displays error code and unicode sign: 1 ↵
Triggers use regular expressions, so it is easy to match on a letter, that rearely appears in normal work.
I went to iTerm2 preferrences -> Profiles -> Advanced -> Triggers -> Edit and put there something like this:

0 ↵$          Run Command    afplay /System/Library/Sounds/Submarine.aiff &
[1-9]d* ↵$    Run Command    afplay /System/Library/Sounds/Blow.aiff &

First pattern catches 0 exit code and the other non zero exit codes.
Mind the & at the end of commands. They are silent, but without it, they run in the same thread
and cause the same problem as in the solution1a.

I also had to modify bira to start displaying the 0 code:

local return_code="%(?.%{$fg[green]%}%? ↵%{$reset_color%}.%{$fg[red]%}%? ↵%{$reset_color%})"

Problems:

In zsh, I can scroll through files in directory using tab.
Every time I did that, the line with status code of previous command is reprinted,
which triggers the sound.

Solution2b:

Bira splits the prompt into two lines, so I simply moved the status code to the first line:

PROMPT="╭─${user_host} ${current_dir} ${rvm_ruby} ${git_branch} ${return_code}
╰─%B$%b "
# RPS1="${return_code}"

Problems:

Now I am quite far, with what I want.
I have sounds, that work even over ssh.
The problem is tmux.
When I scroll in tmux, entire pane gets repainted
and iTerm2 plays all the status codes, that are visible!

Solution2c:

Use iTerm2 tmux integration.
I installed tmux2.0 on the server
(from source, it was the easiest method).
Now I am able to connect to it via ssh with:

ssh user@host -t 'tmux -CC attach'

Problems:

Non yet, TBD.

Wrapping up:

Terminals don’t like sounds.
I can set 256 colors for foregrounds and backgrounds in my terminal emulator,
but I can’t change the beep sound easily.
After couple of days, I think, it was worth it.
I really got used to having this additional feedback,
even when I am listening to music.

Please, comment, if you find it useful.
If you don’t, let me know why.

[1] https://pragprog.com/book/ahptl/pragmatic-thinking-and-learning
[2] https://www.iterm2.com/documentation-triggers.html
[3] https://github.com/robbyrussell/oh-my-zsh/blob/master/themes/bira.zsh-theme

Posted on by tkowal 0

The unintuitive latency over throughput problem

Since 15.01.2015, I am teaching a course about Elixir programming language. It is created by José Valim, who is Rails Core Team Member. He knew, that Erlang virtual machine easily solves problems, that are hard to solve in Ruby, mainly concurrency. During my first presentation, I had couple of slides, that showed, what is so great about Erlang VM, that José picked it for implementing the Ruby successor.

I will not go into details about all technical aspects, but there was one, that was particularly hard to understand: latency over throughput. To understand this problem better, you have to know one thing about scheduling. Erlang gives you very lightweight processes, but this feature is not unique. Go has goroutines, Scala has Akka library and other programming languages start to provide libraries to mimic this. But Erlang gives you also preemptive scheduling, which is really unique feature.

I tried to find something about preemptive scheduling in other languages. I’ve found articles about plans to add it in Go and Akka, but as far as I know, it is not quite there yet (correct me in comments, if I am wrong!).

But what is preemptive scheduling? Without going into details: it means, that a long running process can be paused to give other processes CPU time. Like in operating system, but using light processes and with much, much less overhead :)

Why is this detail important? Because this can reduce latency greatly, which makes user happy. :) How exactly does it work? To answer that problem, lets ask another question.

We have single core, no hyper-threading CPU. There is no parallelism involved. We are testing two web server implementations. We fire 1 000 000 requests and wait until web server returns all responses. First server has no preemptive scheduling. It returns last response after 60 seconds. All responses are correct. Next, we do the same to the server with preemptive scheduler. It finishes processing last request after 90 seconds. Again – all responses are correct.

Which webserver has higher throughput? Which one has lower latency? Which one would you choose?

Throughput question is easy: first one has 1 000 000 / 60 requests per second, which is 16 667 rps. Second one has 11 111 rps, which is worse.

Latency question: It might be tempting to say, that first server has lower latency. If processing all requests was faster, than avarage processing time must be lower, right? WRONG! And I will prove it to you, using counter example consisting of only two requests!

latency-throughput

Lets say, there is one CPU intensive request, that will last for 5 time units and one quick one, which will last for 1 time unit. The longer is first in processing pipeline. In webserver without preemptive scheduling, it has to be processes from start to end. After that, we can get to the second one. We also count the time between next requests (one unit). Lets calculate the avarage response time. First response was sent after 5 tu, second one was sent after 7 tu. The average latency is 6 tu.

In preemptive web server, first request is processed only for one time unit and then it gets preempted, so that second one has a chance. It gets processed quickly and then, we come back to the firs one. Here, first request is finished after 8 tu and second one after 3 tu, giving avarage latency of 5.5 tu.

We have worse throughput and better latency at the same time! This is possible, because preemptive scheduling is fair. It minimises waiting time of requests and makes sure, that quick requests are served faster.

In real world, the longer request might be not 5 times, but 1000 times longer. Also context switch time is usually really small fraction of the minimal time spent on processing. This means, that you can get MUCH better results with preemptive scheduling.

Of course, it is not a silver bullet. If your application processes data and you need all data points for next step of computation, you will go with lower throughput. There is no need to optimise for latency in that case. But if you are building website, where you serve independent users and you want to be fair with them – check out the Erlang VM, you will not regret!

Posted on by tkowal 2

Managing documentation with GitHub and Jekyll

In my company, we have an open source project, that is running for some time. It is a chat server written in Erlang https://github.com/esl/MongooseIM. Recently, we stumbled on a problem. All of our documentation was on wiki pages on github. MongooseIM team puts great effort to keep it always up to date. But what if someone needed docs for old version of Mongoose?

The solutions seemed to be easy. Lets generate html version of docs for every new git tag!

Current docs used Markdown, so to have better versioning, we moved the docs to the repository. This way, when we update the code, we can update the docs in the same commit. If someone checks out some old tag, he will have matching doc in doc folder. Cool!

Now, the hard part! We would like to generate html docs from markdown ones. Why? They would be easier to read and we have greater control over presentation. We can also show docs for couple of recent version without the need to checkout the repo or switching to tags.

The static page generator choice was a no-brainer. Jekyll is not only easy to set up, generates static html from markdown, but also has native GitHub pages support. We used categories to indicate releases and changed permalinks to use the scheme /:categories/:title/ After prepending dates to file names, our docs automatically became posts. That was easy!

Then, we realised, that not everything is so cool…

1. Links between files were broken. In GitHub Flavoured Markdown, when you specify link [some title](chapter1.md) in file table-of-contents.md, GitHub searches for the file chapter1.md in the same directory as table-of-contents.md. But html generated by Jekyll has <a href=”chapter1.md”>some title</a>, which means, that instead of going from /MongooseIM/1.5/table-of-contents.md/ to /MongooseIM/1.5/chapter1.md, it will go to /MongooseIM/1.5/table-of-contents.md/chapter1.md. It will just append the href to the existing url, which does not make any sense.

We thought about changing the base_url, but it is one for entire page, so we would have problems with different docs versions, which was one of the main points of doing the whole html generation!

The easiest solution, we could think of, was going through all markdown files and changing (chapter1.md) to (../chapter1.md). We simply used sed for that. This looks more like an ugly hack, than a solution, though.

2. Erlang code quite often has code like this: {{atom, Something}, Options}. This is syntax for tuples nested in each other. Jekyll uses Liquid for its templating language and two curly braces are used to print variables. The problem is, we wanted to use unmodified markdown files. We don’t have any variables! We even don’t have the jekyll header with title date and other stuff. We were able to squeeze all that into _config.yml.

This lead us to another hack: we’ve added {% raw %} and {% endraw %} to every file before the generation step.

3. All the titles got capitalised. Instead of mod_mam, we now have Mod_mam. It is problematic, because Erlang is case sensitive, so those titles mean two different things…

We haven’t found easy solution to this problem yet. Lets call it a day, we will think about this tomorrow.

How do you manage documentation for your open source projects? Do you use some other tool? Maybe you know, how to make jekyll generate links in some clever way? Do you know, how to disable Liquid for all posts in category? Where do you configure Jekyll to stop capitalising post titles? If you know one of those things, please post a comment :)

Posted on by tkowal 0