A first concurrency in Erlang

Tue, 13 Mar 2007 13:09:39 +0000

I've recently started learning the Erlang language using Joe Armstrong's new, in-beta, book. In the past I learned some Lisp but I ended up admiring Lisp more in the abstract than I liked it in practice and so I didn't stick at it. Lately I've had a yen to return to functional programming because so much of what I find elegant about Ruby seems to have it's roots in the FP world and I feel I've so much more to learn yet.

Getting started with a new programming problem for me is about picking a problem this is useful, not too difficult, but exercises a reasonable spectrum of the language. In this case, because Erlang is so strong in it's support for concurrency, I needed a problem that would naturally have an elegant concurrent solution. Since I have a very large collection of PDF's which includes quite a lot of duplicates and, since I don't like any of the de-dupe solutions I've tried, I thought a good first problem would be scanning for and finding duplicate PDF's across all my disks.

I spent a bit of time looking at the library that comes with Erlang and was pleased to discover filelib:fold_files that neatly took care of most of the actual work of iterating the file-system what remained was learning how to build the resulting datastructure in functional style bearing in mind Erlang immutable datastructures and single-assignment variables. My approach to finding duplicates was to calculate an MD5 hash of the file size along with the first & last 16 bytes of the file which looks like this:

get_file_hash( FilePath, Size ) ->
  {ok,[Hunk1,Hunk2]} = myfile:open( FilePath, [read], fun( File ) -> file:pread( File, [{0,16},{Size-16,16}] ) end ),
    hex:hex( binary_to_list( erlang:md5( lists:append( integer_to_list( Size ), lists:append( Hunk1, Hunk2 ) ) ) ) ).

As an aside I really enjoy the way Erlang functions are declared, e.g.

length( [] ) -> 0;
length( [H|T] ) -> 1 + length( T ).

which uses pattern matching on the arguments to find the right clause of the function. It's elegant although I did get caught out by not realising that clauses of a function with the same name but different arity do no belong together, e.g.

find( Path ) -> expr;
find( Path, [] ) -> expr.

are not clauses of the same function (being instead clauses of find/1 and find/2 respectively) which can lead to an, if you're not expecting it, perplexing head mismatch error.

My definition of myfile:open(), above, reflects my failure to close any files in my first cut at the solution leading to some strange emfile errors. I can't remember the last time I didn't pass a block to Ruby's File#open so I think Matz got that dead right, hence:

open( Filename, Modes, Block ) when is_function( Block ) ->
  {ok,File} = file:open( Filename, Modes ),
  RetVal = Block( File ),
  file:close( File ),
  RetVal.

that demonstrates another neat aspect of Erlang function clauses, guard expressions.

I was surprised at how quickly I had a solution to my basic problem, almost Ruby speed. However what the basic solution didn't do was take advantage of Erlangs concurrency primitives spawn & co. Initially I was quite confused about how to parallelize my solution since so much of the work was going on in fold_files. After some pondering I concluded that what I was looking for was futures). I looked through the library that comes with Erlang but couldn't recognize was I was looking for, so I cooked up the following:

-module(futures).
-export([new/1,value/1]).

new( Fun ) ->
  Pid = spawn( fun oneshot/0 ),
  Pid ! { self(), Fun },
  Pid.

value( Pid ) ->
  receive
    { Pid, { ok, Result } } -> Result
  end.

  oneshot() ->
    receive
      { Sender, Fun } ->
        Sender ! { self(), { ok, Fun() } };
      Any ->
        io:format( "Oneshot(~p) received unknown message ~p~n", [self(),Any] )
    end.

Which, it turns out, isn't a shocking travesty ;-) I did get some useful suggestions from the fine folks in #erlang including looking at Joe Armstrong's definition of pmap. Now I was able to make essentially a two line change to have a fully concurrent solution:

scan( Path ) ->
  lists:map(
    fun( Future ) -> futures:value( Future ) end,
      filelib:fold_files(
        Path,
        ".*.pdf",
        true,
        fun( F, Acc ) -> [ futures:new( fun() -> file_info:file_info( F ) end ) | Acc ] end,
        []
    )
  ).

Creating the future spawns an Erlang process that runs the embedded function (that does all the nasty mucking about with files and MD5 hashing). Later the future is referenced for it's value which will block until a value is available (but by which time all the processes have been spawned off anyway). Crucial to this being efficient is Erlangs ability to cope with hundreds of thousands of processes (I was able to run tests creating millions of processes on my MacBook Pro CD2 with 2GB memory). For more about that take a look here.

The way I have defined the future module highlights what is for me, so far, the only thing I don't like about Erlang (and probably all functional languages). I miss the ability to wrap the future into a nicely packaged class. Calling:

futures:value( Pid )

seems less elegant than (hypothetically):

future.value()

Maybe it's a small point but modules do feel very clunky after using Ruby classes for such a long time.

I've still got a few bugs to work through (the odd future call seemingly to returnin a pids instead of the expected result which is a bit weird) but otherwise I have quickly arived at 80% of a fast neat solution to my problem. The last step of which will be to have it create a customisable script for handling the duplicate files.

I'm beginning to like Erlang a lot.

Curiouser and Curiouser! on functional-programming

A first concurrency in Erlang