Easy Python from LFE/Erlang

Recently there has been a great deal of interest in running Python from Erlang/LFE. In particular, these two posts have gotten a lot of attention. This post introduces a new library for LFE and Erlang that builds upon the work demonstrated in those posts, and furthermore, points towards a future of distributed Python … a future that is just a few days away.

The library is py. It has an extensive README which is, in essence, a tutorial. As such this post won't cover all the material provided there – we'll take a look at some samples and show some new features that haven't been added to the README yet :-)

Getting the Code

Be sure to check out the requirements here! Then grab the code and get everything ready:

$ git clone git@github.com:lfex/py.git
$ cd py
$ make
$ . ./python/.venv/bin/activate
$ make repl-no-deps

This will put you into the LFE REPL with everything running and ready to execute some Python.

Using py

Let's start off simple by taking a look at all the versions you have:

> (py-util:get-versions)
(#(erlang "17")
 #(emulator "6.2")
 #(driver-version "3.1")
 #(lfe "0.9.0")
 #(erlport "0.9.8")
 #(py "0.0.1")
 #(python
   ("3.4.2 (v3.4.2:ab2c023a9432, Oct  5 2014, 20:42:22)"
    "[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]")))

Now a Python call:

> (py:func 'datetime.datetime 'now)
#("datetime" #(2014 12 27 17 58 14 810870 undefined))

Module-level constants:

> (py:const 'math 'pi)
3.141592653589793

Instantiating objects:

> (py:init 'collections 'UserDict)
#("UserDict" ())
> (py:int #b("101010") '(#(base 2)))
42
> (py:init 'datetime 'date '(1923 4 2))
#("date" #(1923 4 1))

Calling methods:

> (set now (py:func 'datetime.datetime 'now))
#("datetime" #(2014 12 23 23 14 37 677463 undefined))
> (py:method now 'strftime '(#b("%Y.%m.d %H:%M:%S")))
"2014.12.d 23:14:37"

Object attributes:

> (py:attr now 'year)
2014
> (py:attr now 'microsecond)
677463

Operations on objects:

> (set later (py:func 'datetime.datetime 'now))
#("datetime" #(2014 12 23 23 21 25 714474 undefined))
> (set earlier now)
#("datetime" #(2014 12 23 23 14 37 677463 undefined))
> (set diff (py:sub later earlier))
#("timedelta" 0 408 37011)
> (py:attr diff 'seconds)
408

Python builtins:

> (py:dict)
#("dict" ())
> (py:dict '(#("a" 1) #("b" 2)))
#("dict" (#("b" 2) #("a" 1)))
> (py:any '(true true false false false true))
true
> (py:all '(true true false false false true))
false
> (py:round 0.666666667 5)
0.66667
> (py:range 7 42)
#($erlport.opaque python
  #B(128 2 99 95 95 98 117 105 108 ...))
> (py:len (py:range 7 42))
35
> (py:pylist (py:range 7 42))
(7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
 31 32 33 34 35 36 ...)

Python operators:

> (py:add 37 5)
42
> (py:mul 7 6)
42
> (py:sub -108 -150)
42
> (py:truediv 462 11)
42.0
> (py:floordiv 462 11)
42
> (py:gt 7 6)
true
> (py:le 7 6)
false
> (py:eq 42 42)
true
> (py:and- 60 13)
12
> (py:or- 60 13)
61
> (py:xor 60 13)
49
> (py:inv 60)
-61
> (py:rshift 60 2)
15
> (py:lshift 60 2)
240

Erlang?

Yes, all of this is easily available to Erlang too :-) That's the beauty of LFE – 100% Erlang Core compatible :-)

There is a link in the project's README file that points to example Erlang usage – be sure to check that out!

The py-sup LFE Supervisor

py runs ErlPort Python servers using an Erlang supervision tree. Since py comes with an OTP application, the supervision tree gets started with the application (this happens automatically when you use the make repl and make repl-no-deps targets).

To get a list of running Python server Erlang process IDs:

> (py:get-python-pids)
(<0.36.0>)

The supervisor process ID:

> (py:get-sup-pid)
<0.35.0>

If your Python server process crashes, the supervisor will restart it. You can see this in action with the following:

> (erlang:exit (car (py:get-python-pids)) 'kill)
true
> (py:get-python-pids)
(<0.37.0>)

Notice the incremented process ID for the Python server.

Wait a few seconds and try it again:

> (erlang:exit (car (py:get-python-pids)) 'kill)
true
> (py:get-python-pids)
(<0.38.0>)

The default restart policy for py is to only restart if the Python server hasn't crashed more than 3 times in 1 second. This prevents run-away restarts in the cases of bad code, pathologically configured Python environment, etc.

Try killing the child Python server process rapidly several times in a row:

> (erlang:exit (car (py:get-python-pids)) 'kill)
true
> (erlang:exit (car (py:get-python-pids)) 'kill)
true
> (erlang:exit (car (py:get-python-pids)) 'kill)
true
> (erlang:exit (car (py:get-python-pids)) 'kill)
true
>
=INFO REPORT==== 27-Dec-2014::18:17:21 ===
    application: py
    exited: shutdown
    type: temporary

Three was okay – but not four!

Support for a supervision tree is just the beginning, though …

The Future

Current development is focused on three new features for the library:

1. Running multiple Python servers at the same time
2. Creating sophisticated schedulers that determine which Python node to execute on next
3. Distributed Python: the crowning glory of Erlang/LFE

To contribute, you can check out the existing tickets here. You can also hop on the LFE mail list and ask questions, make suggestions, etc.