Friday, March 26, 2010

Erlang Factory: Martin Logan, Eric Merritt, Richard Carlson: Writing a Technical Book



Where to start


Writing a technical book is at least as hard as you think it is, porbably even more. The main prerequisite is believing you have something important and worth saying. After that the next thing to do is to choose a medium. In this case a publisher. They chose manning.


The first step is writing a proposal. It is 9 pages long. Involves having a competitive analysis to prove book will do well. Also, propose a table of contents.


Gathering co-authors is good, especially to establish credibility. Then begin writing. Focus more on the writing than on the tools you will use to write the book. Make sure to follow publishers guidelines. Getting the ideas on paper is most important, then continue to do revisions and refactoring.


1/3 Review


This is where you meet your readers and get feedback. Make sure to incorporate it and rewrite sections that need to be fixed. At this time you need to start to solidify the overall structure and ideology of the book. You need to have an overall story, so the chapters flow well together.


This is the time where your social life becomes a distant reality.


Productivity


It is important to keep productivity up and avoid obstructions. Treat the book as if it were a job. Schedule milestones and stick to the deadlines. Try to be consistent about writing often. Used dropbox as a way of sharing the book which was a binary blob.


2/3 Review


Still had lots of negative reviews. Main complaints were inconsistent voice and loose terminology. Needed to choose an audience(chose intermediates) and began another major rewrite.


Final Review

Finally had some positive response. Main responses were good raw material, nice story line, well targeted, and easier to follow. But, you are not done, not even anywhere close.


Next Steps


Need to start polishing book. Did cross-editing where they edited each others chapters instead of their own. Even at this point, don't be afraid to rewrite.


Production


This where you start indexing, final edits, technical editing, production editing, and printing. With Manning, they were required to create index on their own manually.


OTP Challenges


Terminology is ambiguous and in some cases non-existant. There are implicit standards and conventions. The documentation was incomplete.


Closing


Hoping to launch book in Summer 2010.

Erlang Factory: Jack Moffit: Erlang Is Our Superpower

Introduction to Erlang


Started as a game, chess park. Play chess online. Were python guys, so wrote it in python. Decided to use XMPP for all the wonderful features it has. Started using jabbberd. Weren't happy with it. Decided to look at ejabberd, which led them to Erlang.


New Product


Wanted to have a place where you could find out what is currently happening on a particular topic. Publishers post data to them. Data is aggregated into a single firehose. Data is transformed and filtered base on type of data. They do spam filtering, language annotation, classification, and keyword filtering. Data is then persisted to disk. Finally, results are streamed to clients.


Building Collecta


Had to address the problem of mass quantities of data as well as high levels of traffic and queries. DB, message pipeline, and other components has to be distributed for scalability purposes.


Ejabberd


Used as base of distributed messaging pipeline. Ejabberd made it easy to filter and modify packets. Wrote custom PubSub module to achieve this.


Webmachine


Forms the API glue. Use it for client-side analytics.


CouchDB


Uses cloudant to store data. Store API keys and user accounts. Fetch documents by key. RabbitMQ handles the final persistence chain.


Riak and Riak Search


Chose over solr and Lucene, since they don't scale well.


Major reasons for using Erlang


-Best of class for most applications
-Rich set of libraries for many tasks

-Hot code swapping

-It actually works


Going Big


MySpace had put Collecta on a page about Haiti disaster. First had traffic that was 3x more than previous maximum. Went to 5x, then 8x, then 12x. The system held until 12x. Looked into problem and had one bug. They never load tested for when everyone on the system was searching for the same thing at the same time. Came down to list of subscribes in mnesia blowing up. Now normal traffic is 40x.

Erlang Factory: Michael Truog The Cloud as an Interface

A Cloud as an Interface - http://cloudi.org/

Cloudi - not for webapps or data cloud, but for processing. For managing processes and making sure they are fault tolerant.

Cloudi is a flexible framework for private cloud computing. Its dynamically loud balanced and scheduled. Distributed execution of C/C++ work.

Cloudi is an alternative to paying for a blackbox commercial cloud.

Erlang coordinates all the work and any crash of the C/C++ code.

Cloudi is meant for long running intensive data processing like text processing and numerical computations.

Riak Search

Riak Search

Presented by John Muellerteile from Basho.

Fulltext and general indexing methods.

Agenda

* Intro
* A search Tale
* Riak search

"I used to make stuff" (but now I'm a database hacker)

Got sick of having to roll own indexes.

Act 1: I love lucene

Life is good, fast and predictable.

Act 2: Cluster Luck

A few shards, good performance.

Act 3: SANFU

Lots of shards. Operational nightmare. Diminishing returns. Costs more to index and more to fail.

Where does it hurt?

Complexity: more shards. Failure cases go from 'oops' to 'epic'.

Riak Search Goals

Decentralized. Distributed: only worry about local data. All nodes do the same thing. Adding nodes adds performance. Flexible, query platform that's efficient and extensible.

Arch: Query Processor -> System Catalog/Index Store

System catalog is an index of indexes.

No unique snowflakes. All pieces in the cloud are the same.

Index:

'Column-esque': use as many as you want
Every column has a primary key and each key points to a document id
Can also store facets.

System Catalog:

Searchable index of node-local columns
Can contain arbitrary metadata.
Built to be small and fast and always in memory.

example:

+index:"products"
+term:"auto"
+category:"books"

result:

products.automobile,
products.auto, products.automata, ...

Indexing Crash Course:

Tokenize->Term->Column->store

Index, docid, field, field/value, facet, facet/value

One column for each term.

Will distribute columns across nodes.

Query Processor:

Parses query, creates a query in an intermediate language, plans and executes.

They have implemented all of lucene search operators.

Protoype uses lucene parser to create 'primitives' the planner and executor can deal with.

QIL, intermediate language:

Used to express data flow. A network of simple components, producers, filters and accumulators.

Executing query. Locate columns, plan (optimize to use co-located data, optimize where to place filter processes) the query and execute that plan (starts accumulator, filters and then producers).

The intermediate language is built for pipelining and extending.

Why Erlang?

Queries execute as process networks
Each process an automaton, shared nothing
Stability in a storm of short lived processes
Sane ways to reason about and amange them

Project Status

Prototype
It will be open source (Apache)
Production version has been in use

Git Basics

Mastering Git Basics

Given by @mojombo the cofounder and cto of github.

Initial Config:

git config --global user.name [NAME]
git config --global user.email [EMAIL]

git config --global color.ui true

Last one gives nice colors for diffs and such.

Creating and Committing

Make a dir for your project:

$ cd patch/to/repos
$ mkdir hello
$ cd hello

git init

$ git init # converts empty directory into a git repo. Creates .git dir.

Behold the index

Create file hello with content "echo hello". The index is stored in .git directory

$ git add hello # adds hello.sh to the index in .git

Commit

$ git commit -m "first commit"

Git takes snapshots of index and makes a commit out of them.

$ git log

Shows history of commits.

Status

$ git status

Shows current status of repo. All modified files.

Diff

$ git diff

Shwows difference between index and working directory.

Example after adding and commiting a a file hello and then changing it:

12:28:43 $ git diff
diff --git a/hello b/hello
index 09990d4..953a442 100644
--- a/hello
+++ b/hello
@@ -1 +1 @@
-echo "hello"
+echo "hello 2"

Interactive add:

12:29:12 $ git add -p
diff --git a/hello b/hello
index 09990d4..953a442 100644
--- a/hello
+++ b/hello
@@ -1 +1 @@
-echo "hello"
+echo "hello 2"
Stage this hunk [y,n,q,a,d,/,e,?]?

Will ask for each hunk.

After saying yes we can see the diff is now also in the index, ready for committing.

Now 'git diff' is empty since the diff is in the index. There is no difference between the index and the working direcotry.

What if we want to review whats in the index, but different from first commit?

12:30:54 $ git diff --staged
diff --git a/hello b/hello
index 09990d4..953a442 100644
--- a/hello
+++ b/hello
@@ -1 +1 @@
-echo "hello"
+echo "hello 2"

Now make a second commit.

12:31:38 $ git commit -m "second commit"
[master 6cb636e] second commit
1 files changed, 1 insertions(+), 1 deletions(-)

Remember:

1) make changes to working dir
2) stage these changes to index
3) commit the current stat

Now we have 2 commits. Everything is consitent now. Now changes from index to working dir and no changes from last commit to index.

Every commit keeps record of the parent.

Branching and Merging

12:32:52 $ git branch
* master

Shows the current branch with star and all other branches. master is the default branch.

Branches are just pointers to commits. Thus the master branch is simply pointing to our second commit. This is why branches are so easy in git and powerful.

Easiest to think of the working directory corresponding to a branch.

As you commit the branch moves with you.

Create a new branch:

12:36:19 $ git branch uppercase

12:38:25 $ git branch
* master
uppercase

Both branches point to the same commit and the working directory points still to the master branch.

Switch to a branch:

12:38:34 $ git checkout uppercase
Switched to branch 'uppercase'

12:39:09 $ git branch
master
* uppercase

Now our working directory corresponds to the uppercase branch.

When switching branches it will try to apply anything staged to that new branch -- may conflict.

Note: 'git stash' makes a temp commit to save info for moving branches and can 'git stash apply' later.

Make a change in the new branch and add.

diff --git a/hello b/hello
index 953a442..1ba14ad 100644
--- a/hello
+++ b/hello
@@ -1 +1 @@
-echo "hello 2"
+echo "HELLO 2"
Stage this hunk [y,n,q,a,d,/,e,?]?

Now commit:

12:43:41 $ git commit -m "commit 2"
[uppercase d5e17c4] commit 2
1 files changed, 1 insertions(+), 1 deletions(-)

List commits a branch points to:

12:43:47 $ git branch -v
master 6cb636e second commit
* uppercase d5e17c4 commit 2

Just shows the first 7 chars of commit hash, since usually enough to tell the difference.

Switch back to master and working dir changes:

12:45:28 $ cat hello
echo "HELLO 2"
[tristan@duff:~/mastering-git-basics/hello]
12:45:30 $ git checkout master
Switched to branch 'master'
[tristan@duff:~/mastering-git-basics/hello]
12:45:35 $ cat hello
echo "hello 2"

Started late... Moving to 'reset'

git reset --hard

Dangerous, use with care. Deletes uncommitted changes. Changes branch to point somewhere else from the current commit. If no branch is now pointing to that commit you can use reflog to find the hash of that commmit and get it back, for 30 days.

git reflog

Shows previous values of where HEAD points

12:47:59 $ git reflog
6cb636e HEAD@{0}: checkout: moving from uppercase to master
d5e17c4 HEAD@{1}: commit: commit 2
6cb636e HEAD@{2}: checkout: moving from master to uppercase
6cb636e HEAD@{3}: commit: second commit
d02f69f HEAD@{4}: commit (initial): first commit

git diff two commits

12:51:26 $ git diff master uppercase
diff --git a/hello b/hello
index 953a442..1ba14ad 100644
--- a/hello
+++ b/hello
@@ -1 +1 @@
-echo "hello 2"
+echo "HELLO 2"

git merge

Merging in git is super easy. Lets merge uppercase and master.

12:53:44 $ git merge uppercase
Updating 6cb636e..d5e17c4
Fast forward
hello | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

Changes working directory. This is a fast forward commit since it was a direct descendant of the master's current commit. Simply moves pointer, no real merge needed.

Now both point to same commit.

12:54:51 $ git branch -v
* master d5e17c4 commit 2
uppercase d5e17c4 commit 2

Now we can delete uppercase branch if we don't need it.

12:55:28 $ git branch -d uppercase
Deleted branch uppercase (was d5e17c4).

All that happens is pointer to that commit named uppercase is gone, but no work gone. -d only lets you delete if commits are contained in current history. -D removes even if there are commits not in the history.

What if we have two diverging histories to merge.

Shortcut to create new branch and check it out:

12:56:19 $ git checkout -b greeting
Switched to a new branch 'greeting'

12:58:14 $ git branch -v
* greeting d5e17c4 commit 2
master d5e17c4 commit 2

Lets make some changes to files hello and a new file goodbye.

12:59:11 $ git add -p
diff --git a/hello b/hello
index 1ba14ad..653f53d 100644
--- a/hello
+++ b/hello
@@ -1 +1 @@
-echo "HELLO 2"
+echo "Yo"
Stage this hunk [y,n,q,a,d,/,e,?]? y

12:59:40 $ git add goodbye

12:59:47 $ git commit -m "new greetings"
[greeting 07527ab] new greetings
2 files changed, 3 insertions(+), 1 deletions(-)
create mode 100644 goodbye

13:02:17 $ git checkout master
Switched to branch 'master'

Create README and add 'some great scripts'

[tristan@duff:~/mastering-git-basics/hello]
13:05:49 $ git add .

Adds everything.

Will be making a git cast out of presentation since it was cut short and had to jump around... Will post later.

Erlang Factory: Tony Garnock-Jones: What is messaging and why should we care?



Why messaging?


Scaling, load balancing

Delayed jobs, task queues

Multicast and broadcast

Trusted store-and-forward

Management monitoring

Decoupling of components


What is messaging?


Messaging involves relaying, filtering, buffering, queueing, and transfer of responsibility.


Where is messaging?


See it in enterrpise, p2p, etc.


Why AMQP?


Wanted to get rid of current closed uninterruptable system. Other exisiting alternatives have limitations. AMQP is a general model for messaging. Having wire-levle protocol allows avoidance of vendor lock-in. It is API-neutral and language independent.


AMQP Model


App publishes to exchanges. Exchanges bind to queues. Queues deliver to other apps. Based on Producer/Exchange/Consumer model.


RabbitMQ


Is an Erlang OTP implementation of AMQP. Supports C, Java, C#, Python, Ruby, PHP, etc.


Beyond core AMQP


Sometimes AMQP is not quite enough. RabbitMQ has plugins to address this.


rabbitmq-status – broker health check web page

jsonrpc-channel – AMQP over JSONRPC

rfc4627 – JSONC and JSONRPC

mochiweb – integrate with mochiweb


AMQP limitations


No message transformation. No custom message routing. No stateful multicast.


Example 1 – Github commit hooks.


Want url-encoded JSON data blob delivered from HTTP POST to own XMPP client.Need to respond to http post and transform payload. Make RabbitHub to replace a custom http proxy gateway. Then decode and prettyify data by passing to transforming operator, then back to another exchange.


Example 2 – Stock Ticker


Have price cache with hash table for last value cache. Created 'x-lvc' exchange type to address these needs.


Writing own plugin


Write erlang OTP app, include .app. Package it up as an .ez(zip) file. Look at umbrella.

Erlang Factory: Yogish Baliga, Mark Zweifel: Deploying Erlang at Yahoo - A Case Study

We are going to do this collaboratively. We have worked together for over a year, we talk a lot together and thats what we are going to do here.

When you try to bring in a new language to a new company you don't want to be tagged as someone who chose something to be clever. So we did something in three stages.

We will give you the back ground as well as what we did. The critical thing here is to choose a problem that so nasty no one wants to touch it. You get the opportunity to kick the tires with the language and form an opinion. We did this with delicious.

This project already failed three or four times. Failed in the sense that it took longer then expected and didn't fit the project scale.

This problem was something that didn't require any long term commitment from the language. Management didn't care what language you chose. This seemed to be the domain for this language. lets try to use it.

What we got out of it.

No one agreed with us. As long as the problem is hard it was fine. The project already failed so it couldn't be worse. Mark had a solid understanding of the language at this time. Mark was my tutor for the language at this point.

So based on those experiences, you get a handle on what you can get from erlang. You have prior experience with your tool change, so you know where those converge. You can kick the pants off your coworkers and deliver something quite nice.

We got a problem that allowed us to prove that Erlang was better for this. We created a team to do open search service. Exposing the yahoo index to front end developers. Here the main point was we didn't want reinvent the wheel. The initial implementation was in php and it didn't scale. We expected a small amount of queries from the developer community. As time went on we weren't able to scale on the php infrastructure.

The put all the burden on us, and we had less time to market. The manager was a startup guy, he said 'I want the results I don't care about the technology' so we went with erlang. Management was always looking at us and asking what technology we where using.

If you get it out and it works its a lot harder to cancel then if you get caught. Its not always good for job security. we dealt with a wide range of emotions. Its a mind bender.

At this point we showed that Erlang scales and it is production quality. The next problem was a bit nasty. More of a distributed problem. We inherited some components from other teams and no one wanted to rewrite them. Our main concern was how to coordinate all these jobs. It was an indexing pipeline problem. One of our coworkers said 'lets do it in java' and leverage the libraries. We didn't want to go that route because, from past experience, we know it doesn't scale.

This was basically a control pane component. A fleet of gen_fsms that deploy across the fleet. We where able to map out work to a group of machines that where doing different phases of the project. Interop was absolutely required. This is where Erlang ports came in very helpful. We also wrote some jinterface and c nodes.

What every is user facing should be an open technology. This was one of the requirements for the second problem we where solving.

So again, one of the side effects is that by working this way you have to start writing binaries that integrate into the build cycle. You begin having a beginning of a workflow for the company. Building up a bit of a ground swell. We have written a bunch of interfaces for existing libraries already. It took a year and a half to port these to java. We did it in three months in Erlang. Hopefully we will be able to do it for many other tools.

The (delicious problem) migration time was 5 weeks to compute differential. Erlang brought it down to 36 hours. Another success was scaling BOSS. When we started we where doing 40 requests per second in php, with the Erlang infrastructure we could do 800 requests per second on one box.

One of the deficiencies we saw in erlang was text processing. When it comes to text processing especially xml processing it is slow. We took process one version of the expat xml processor and we are using that.

Indexing pipeline control, was manual. We where trying to commercialize this and make it a turn key solution to customers. Thats where the real win came from. We where able to make it where you could see this movement through the entire cloud.

Obstacles Encountered.
first of all, the major problem was other team members and explaining async distributed systems to engineers. On paper its really easy but in life its really hard. They just don't get it. We found it very difficult to explain it to them. Now they are slowly getting it.

the second point is if you start taking these legacy systems and wireing up with erlang, making them non-stop, you start finding problems in other non-erlang components. This makes more work but is a good thing.

Legacy systems are not geared towards uptime. Adding concurrency and uptime from erlang finds many faults. This doesn't make you popular with your coworkers. Pushing your coworkers out of their comfort zone can be risky.

A couple of coworkers came up on Erlang. It was hard for them to let go of existing imperative structures and adopt functional styles.

100 nodes doesn't make Erlang a distributed language. the number of nodes is always 'too low' even though java only scales to 1 out of the box. This was a constant problem.

There are also security concerns. With Erlang you have protected yourself from a fair class of problems. However, tapping into nodes sounds bad to a security professional. Its fairly difficult to find the right types of proof cases to allay those concerns. We are solving that by putting iptables restrictions on top of it. This restricts ports and allows one thing to connect to another.

Some surprising things

Erlang is a lot of fun to work with. We have learned a couple of things. People really get annoyed when you say you can do systems programing in Erlang (For various reasons). Single assignment is a great simplifier. It simplifies your thinking and reasoning about programs. It is fast enough in most cases.

We also haven't missed objects that much. Erlang seems to be what Alan Kay refers to as the essence of OOP.

Erlang stack traces are really nice. The fact that the language/runtime so easy to debug is quite nice.

Functional decomposition is one of my favorites. Makes the system tracing much, much easier.

Many system problems already fixed in erlang.

we love Erlang, we like being fan-boys. We also realize that for some things we need to use other languages. I appreciated matzs debugging of the nine nines.

New Approaches

We find the we are spending a lot more time on specification.
We are thinking more and typing less now-adays.
bang and receive are more useful then for and extends for many of the problems we have to solve.

Erlang Factory: Joe Armstrong: Keynote - What are the Important Ideas in Erlang?



Plan


-History

-3 things missing
-1 big mistake

-2 good ideas

-3 great ideas


Erlang crawled along starting from 1986. Picked up in 1996-1998, then died out for a bit, and now is becoming popular once again.


1986-89 were productive years. 1989-96 was at war with the world. 1996-98 peace broke out. 1998-2000 midlife crisis. 2001-2008 continuous slow growth. 2008 upturn


1985-1989


Started with trying to program POTS. Started with a Smalltalk model of POTS, then came up with a new telephony algebra. Wanted to find a better way of programming POTS. Didn't initially want to create a new language, but it became apparent it was necessary.


Using the telephony algebra based on prolog, could describe POTS with 15 rules. Could only describe one process though, not concurrent processes. Decided to add reduction methodology to Prolog using term rewriting. Allowed calculations to be suspended at any time.


1986-12-18 version 1.06 of Erlang is out.


1989


Project came along that wanted to use Erlang, but Erlang wasn't fast enough. Did a study and found out it needs to be 40x time faster. Was called JAM at the time. Settled on requirements for the language at this time: fault tolerance, interaction with hardware, distributed over several computers, reconfiguration without stopping software, continuous operation over years, etc.


Middle years 1990


Had bigger budget and more time to develop Erlang. Added BEAM, HIPE, OTP, bit syntax, type tools


Midlife Crisis


Were banned inside Ericsson in 1998. Made it open source in 1998. Joe and a lot of people quit Ericsson. Started own company, then got bought out.


Back at Ericsson OTP maintains a low profile. Since project was banned, they decided to rename the project.


Currently


It is becoming mainstream now. Many success stories. More demand for books, documentation, teaching. Code now on Github which allows lots of small quick changes.


3 Things Missing from Erlang


-Hashmaps


-Higher order modules(HOMS) and introspection

module_to_list(lists)

[{append,2,F1},{sort,1,F2}..]

function_to_conc(F1).

“append([H|T],L)” → ...


-Receive a function

F = fun({foo,X}) → … end

recieve(fun)


1 Big Mistake


Lost too much prolog.


2 Good Things


Showed Lightweight processes are OK. Java proved GC(garbage collection), Smalltalk proved messaging. Erlang proved process belong to the PL(programming language) not the OS.


OTP Behaviours. They are like higher order functions, but can encapsulate non-functional concepts in a precise way. It allows large teams to work together.


3 Great Things


Bit syntax for pattern matching. Due to Klacke(Claes Vikstrom).


Links. Can link processes together so if one crashes and EXIT message is sent to the linked processes. Encourages “let it crash” programming.


Non-defensive programming. Program only the happy case and let some other process fix the error.

Thursday, March 25, 2010

Erlang Factory: Mats Cronqvist :The nine nines and how to get there

Rumination, a calm and lengthy intent consideration.

Mats will rumnate on 9 9s of uptime.
We will cover debugging the axd switch
and generally on debugging Erlang itself.

Mats is going to pick on Joe Armstrong because he does not care... all these internet sites talk about 9 9s of uptime. It is based on an internal Ericsson slide that you can't find on the internet. This slide had to do with some reports from British Telecom. They reported 9 9s of uptime.

They were down for 100 milliseconds in x number of years. This was really not too believable. This was never made public.

The Reaction to 9 9s


What Mats Remembers

The custmer BT claimed 9 9s
This whole 9 9s thing was not too accurate. But the AXD was very stable.


Why was it so stable

Did they have higher quality programmers? No! superior system architecture? No!

Realistic project management - no waterfall? Yes!
Highly stable infrastructure? Yes!
Properties of the Erlang lanaguage help? Yes!
*Highly productive small team
*no memory management
*selective receive
*debugging made much easier (Mats thinks this is the most important point) The C++ team building the other switch competing if you will with the AXD coudl not even debug their system.

Something rotten on Denmark

Enbedded system running OTP R5
Live in denmark
There was no way to log into the CPU
There was no way to load new code
There was no usable application debugging tool
You could physically connecto to a terminal.

This system was manually rebooted every 90 days or so.

Tracing

Mats wrote a little programm, a one liner, which ran for one minute trace on the struggling node in denmark before and after the reboot. The results were mailed to Mats.

Mats wrote a little comparison profiler. He found that the ets:lookups were 100 times slower than they should have been. Rebooting fixed this. The reason was that the hash function was broken for bignums. The hash function would mis store large numbers. Kind of dumb.

The OTP team fixed the problem in a later version and back ported to R5 and fixed the issue.

What is the point of this story? If you want a system that is reliable it must be debuggable. Debugability is a property of a system.

The 3 kinds of bugs

Crashes randomly (easy to find)
This can be discovered by logging and monitoring, narrowing this down requires knowing the context of the crash.

It over-consumes resources
This can be discovered by real time logging. This is more advanced that discovering crashes. Load testing must also be employed.

It is failing but you don't even notice - you get bad values (these are tough)
To discover these types of bugs you need to create contracts and test heavily.

Polling for debug info

For the first two problems listed above we use some bifs that the vm has to track them

erlang:memory
erlang:system_info
erlang:statistics
erlang:process info
inet:i()

gperf

Another tool used to help wth this with the AXD was called gperf a clone of the old solaris profiling tool. This pulled info from the vm using the BIFs above and plotted them. The testers got used to healthy patterns and then could see unhealthy ones and flag them under testing.

etop

A classic tool. Everyone has seen it. He will not even talk about it. The copied this for debugging and called it dtop. It has a similar interface but displaying erlang processes. This predates the etop program that is in OTP now.

Interrupts

The VM outside of polling offers interrupt mechanisms. You can tell it to let you know about intersting events. Two very useful functions for this are:

erlang:trace/3 (redbug)
erlang:system_monitor/2 (watchdog)

Trace is similar to unix strace. You tell the unix runtime to interrupt system calls and tell you about what is happening. This is what trace does. OTP has a tool called DBG that uses the trace biff. The issue is that it runs a bit hot so it was not safe. It was also complex. It took down their system more than once.

They needed something simpler and safer. than dbg. They created initially the "frankfurter".


Redbug

redbug:start("erlang:now->stack").

This was simploer to use than DBG and it was safe. It would shutdown if it was hurting the host. The command above would tell the VM to show a stack trace everytime someone called erlang:now.

redbug trace patterns

redbug:start("ets:lookup")
redbug:start("ets:lookup(_, foo)")
redbug:start("ets:lookup(_, X) when X == foo")

This was very key, in the AXD ets lookup was called 5k times a second. The type of narrowing of what would be sent up by interrupt was key under this type of volume.

redbug:start("ets:lookup(_, X) when X == foo -> stack")
redbug:start("ets:lookup(_, X) when X == foo -> return")

The first will show the stack, the second will show the return value. The second will show you application level errors, like could not find a value, or badarg returns or so on.


Ok we are back. Redbug is safe, it will turn itself off if it gets too many messages or gets backed up. This is key to avoid the dbg like node killing issues. This is all done by dbg extensions under the hood.

Conclusions

Reliability is easier in Erlang than C++
but it is not automatic - not a silver bullet
to get high reliability you need testing
... and debugging
debuggability is a core strength of Erlang
... especially the call tracing




Erlang Factory: Kenji Rikitake: Erlang Secure RPC and the SSH module

My Erlang Activities
Discovered erlang in 2008, via Joe's book. Patches accepted, TAI leap second, SSH aes123-cbc, backporting freebsd patches, there for quite a long time.

First I am going to talk about the security weakness in erlang in general and then why ssh is needing and finally an ssh protocol review, four how erlang supports ssh, six prototype implementation and support.

Lets talk about the security weakness in erlang. Its actually a very secure platform. No pointer assignment. Variable assignment is once and only once, message passing based system. It minimizes or eliminates sharing is the only way to build a stable system. Erlang's restricted access for io devices makes it more secure. Otp supports a secure communication module, used to make crytpo graphically secure programs.

Distributed Erlang has a weakness. Internode tcp links, this is not encrypted at all. Big problem on a public network. Weak internode authentication, recently a writer complained about this on the list, but its ulf defended. EPMD is totally unprotected, is very hard to enforce security on. Many applications depend on EPMD.

I will not address the sandboxing Erlang nodes/processes. The basic assumption in the execution module is that all processes are equally trustable. Sandboxing in this environment is very difficult. If once you take control of an erlang node inside a distributed network the node will be free to generate attacks on others. I will not address these issues.

There are three kinds of communication links, between nodes, nodes and epmd, epmd and epmd. We must protect all three kinds of communication, if we are going to protect erlang communication. That is very difficult, so the practical work around is putting the transitive network into a perimeter, along with a gateway processer. The gateway processor will handle potentially hostile contact from the internet. There are many implementation things to consider here.

There is another way to protect links, if all nodes are on the internet there may be no perimeter. In this case, the links should be encrypted. ipsec maybe used for this purpose. The current implementations are host based and don't allow per program configuration. This makes it untenable.

Why ssh for Erlang RPC. The reason I chose ssh is that its easier to me as an administrator then ssl. Erlang OTP already has full ssh capability, it already includes an ssh deamon in the language. OTP has a behaviour called an ssh channel. SSH will not break traditional RPC at all, they are completely independent.

related works,

jungerl ssh. Source code is very old, not maintained since 2006 and not working with current implementations fo erlang.

RPC Ideas
BERT-RPC, generic RPC through Erlang
SDIST by Dave 'dizzyd' Smith of basho, multilevel authentication and security models. This helped me a lot in the implementation of this project.

SSH protocol overview

ssh has three different protocols, authentication, connection, and transport. Very well documented in RFCs. SSH built on top of TCP.

SSH communication protocol
handles multiple streams in the same connection, each connection has psuedo tty or tcp tunnels. Tcp is an encrypted tcp tunnel. I will not go deeper in this.

Application over ssh works in three modes. shell, interactive shell; exec, one time execution; subsystem, named service over ssh. The Erlang ssh module supports all of these modes. If you try to make an ssh application it will be not so hard, not easy but not hard.

The other function of the ssh communication protocol which is important for flow control is to have a window of send/receive. This will not cause a flooding of sending messages without acknowledgement. Window size is adjustable per request.

SSH Transport Protocol

The protocol has several functions, server authentication, diffie hellman protocol. Does negotiation of transport protocols. These protocols are based on public key exchange.

SSH Auth Protocol

Three forms of auth, public/private key, password authentication, and host based authentication. Banner message handling.

What Erlang/OTP provides

password and public key user authentication. If you use public key authentication you can't encrypt the public key itself. There is an interactive shell implementation. one time ssh execution, frameworks for ssh subsystems. ssh_channel behaviour of OTP programming.

Prototye Code (already implemented)

remote execution of functions, non-blocking call handling, still synchronous call but the user doesn't have to wait until execution finish and OTP will not be blocked. subsystem has to handle buffering.

Basically, distributed erlang format with attached lengths. Each message is an Erlang tuple that is marshaled using term_to_binary/binary_to_term. Two types of messages {mfa, M, F, A}, {answer, Term}. Had to modify the ssh module a little bit (aes128-cbc). only added one function to crypto. The aes_cbc_ivec, everything else was there, but this function was not. After testing I found a bug in ssh_transport:unpack, that has been accepted. Other algorithms can be added as well.

Implementation status as of 21/mar/2010. Server code complete, multipacket ssh can be handled, basic client code, not-blocking otp code, Code is available on github.

(demo)

Performance is not that great, because this is not well optimized code. Even on limited hardware you can get 500 calls per second. CPU tends to vary a lot, unsure of what the details of this is. However, suspect that the garbage collection contributes.

SSH module needs more fixes and features. When I did this project I had to think about lots of things, like the priority of choice of shared key cryptography. currently this is hardcoded as aes128-cbc, 3des-cbc. More algorithms should be included.

on R13B04 IPv6 client connections fails, this now works in the current release. More comprehensive testing is needed. Regarding these bugs and implementation issues is concerned more testing is needed.

Future plans and thoughts. I just implemented synchronous call handling. async call handling should be implemented. This can and should be done. There are other ideas, for example spawning a remote process and sending a message to the process on the other end. There is discussion on the list about limiting the modules and functions that could be involved. You can not seal one part from the other, so some problem with this. Many subsystems can be run concurrently. You can build your own subsystem monitoring, control, logging whatever.


Erlang Factory: Todd Lipcon: Thrift Avro/Erlang bindings

Thrift and avro bindings with Erlang.

Previously Todd built many websites in the standard languages for doing such things. He worked for Amie Street and Songza which is where he had the chance to work with Erlang.

This talk is really not for those doing hobby websites. This talk is for those doing enterprize level websites. This is for websites with multi functional teams.

Popular Web Languages

PHP
ASP.net
Ruby
Python
Scala? Clojure?

Where PHP Excels

There are acutally good things about PHP. One good thing is there is no concurrency model. There are no threads.

It has powerful templating and string manipulation

Implicit casting, sloppy semantics

Availability of web frameworks, common code, etc...

Availability of designers and programmers

Large existing code bases

Where Erlang Excels

Great concurrency model
Great reliability features
Achieving 5 nines is relatively easy
Dealing with inter-process communication and async processes is natural

Where PHP Sucks

No concurrency possible

All inter-request communication must go through an external party. This is an odd thing, think about a chat application. One user hitting the website, another hitting the website; you need intermediate systems to get the users to communicate.

Each thread ties up a web server process

Async actions are really hard. If you want to send an email without making the user wait for it. You need the third party tool for this as well.

Ever seen a daemon written in PHP? Besides facebook? It is hard to do. Did you still see that daemon 100Million requests later - NOPE!

Where Erlang Sucks

Syntax is weird looking for new users. This makes it very hard for template designers to pick it up in the website building context.

Immutability feels unnatural. This especially for imperative programmers.

String manipulation, unicode support, etc... not very good. Every string a list of numbers is not very good.

Error printouts are not very easy to understand - not even having line numbers.

An observation

The things PHP sucks at Erlang is good and and vica-versa. Lets glue them together!

Enter Thrift

Thrift is the glue that makes multilingual development easy.

Cross language RPC and serialization is what it is for.

A touch of history

Thrift originally developeed by Facebook (mainly a php shop)

Open sourced in Spring 2007.

Now in apache incubator, 0.2.0 releaced in Dec 09.

Reasonably widespread usage. Facebook has about 140 different services in thrift.

Thrift Features

- Serialization
* Primitives and complex datatypes
* cross platform cross language
* multiple protocol implementations
* backwards compatibility built in. This is very userful in multi team situations and complex deployments.
* useful for long-term storage too.

-RPC
* makes remote interlanguage function calls feel like local ones
* seriaalizes the calls, results, exceptions over a transport (eg socket)
* provide service and client abstractions
* comes with well-written client and server skeletons

Why design with services

essentially what this give you is modularity. It forces you to think about your api. If you never think about your API you end up implementing it very slopply. This is very difficult for multi team developement.

This also provides vertical partitioning. You have your C++ code with is really efficient for search. It could be some hardware specialized setup along with it too. You doln't want your webservers running on that sort of hardware. Running on separate hardware connected via thrift is the way to go here.

Long lived data in ram is another reason. You can keep the data in an effcient serview on the network and query it from webservers and son on via thrift.

Thrift vs other options

CORBA - ugly no one likes it
Protocol Buffers - OSS version does not provide RPC stack
Roll your own - bug prone and tedious, though can be marginally more efficient.
HTTP/REST/JSON - deep structures without types are inconvenient. These are sort of rubbish. No types no nothing just agreement loosely does not work well.

The steps to use thrift

first write a .thrift file to define interfaces.
Then run thrift-gen to generate code in various languages.
Then do the actual work to write services around it.
Then rake in the profit.


he is generating code in Erlang and Python and enabling them to communicate.

Case Study

Amie Street is an online music store. You buy mp3s like any other music site. The pricing is dynamic though. The price is a function of the number of previously completed buys.

This is actually really tricky.

Imagine that Alice and Bob both to to Amie street to buy music. Both see price x and put in there credit card data. Well, who gets the price they saw at x?

We needed Erlang to solve this problem. They used tickets just like ticket master. You get x number of minutes to get your prices.

Cards were modeled as processes linked to ticket_releasers which handle cleanup etc... No idea how this would have been solved in PHP!

Joe says that will not work? But Todd answers quickly and Joe is satisfied!


Thrift allows the protocols we are seeing to evolve independently with respect to multiple services due to the backwards compatibility.

The code for this was shipped in 4.5 months. 4k lines of code. Thrift separated difficult distributed system code from the PHP code.

Kind of a black box in a good way - front end people did not care how the backend stuff worked. They interoped nicely through thrift.

The whole thing was very stable and performant.

What is next - Avro

More compact serialization scheme
more oriented towards large dataset storage
Erlang bindings are in progress

Links

thrift http://bit.ly/thrift






Erlang Factory: Chad DePue: Lua Integration with Erlang

Problems we start having with it is when you start running multiple servers (monit). A couple of cool things about monit and god, nice because its turing complete, monitor with distributed monitoring like with m/monit. Problem with it is that its running in ruby.

Erlang, monitoring code is in erlang, configuration in lua. connected nodes share the same configuration.

How does that interact with lua. We use a lot of closures and do a lot of event callbacks.

(example of hooking up a monitor in lua)

Have a global config state that sets up a lua state. Any time it detects a config state it will reload the config. When we reload the config we create a new state. We can grab all our monitors and restart our monitors based on that new config. This example is how we walk through our config state to find our monitor. Its a simple matter of walking a list.

We use a webui built in nitrogen, we actually forked lua.

What don't you like about GPL?

It limits what you can do with the tool. For me its mostly personal preference.

Is the problem with configuration crashing the vm a problem with lua code or the way you are handling errors?

It's more in the way we process the exceptions, its mostly my bad c code. erlua and erl-lua both early stage process and its.

(a couple of examples)

Erlang Factory: Sean Cribbs: Achieving Parsing Sanity with Neotoma

Cucumber

Sean worked on a testing framework for doing acceptance testing called cucumber. Cucumber let you write agile stories that were executable.

Sean had a thought, why not write cucumber in Erlang?

Gherkin uses Treetop

A simple language for expressing tests. Why not use leex and yecc to do the parser generation for handling Gherkin/Treetop. The problem with using leex and yecc was that the grammer exploded. There was too much generation going on for each production, there was an explosion in the grammer, due to the need to express optional tokens explicitly.

PEGs Parsing Expression Grammars

Brian Ford 2002 thesis and related papers. This is a superset of regular expressions. Ordered choice removes ambiguities.

if a then b then c else d

which if does the else belong to? With PEGS you can say explicitly that the else part belongs to the b part.

PEGs are not good at expressing natural language but very good for machine recognizable languages.

PE -> function

e f
sequence(e,f)

This translates to calling sequence of e and f

e /f
choice(e,f)

(e)
e // has no effect but good for grouping

e+
one _or_more(e)

e*
zero_or_more(e)

!e
not(e) // this parsing expression will not match

tag:e
label("tag", e)

.
anything()

if we take all these parsing expressions and turn them into functions it translates into Erlang quite nicely.

Functions are Data

This is the primary way we construct our parser. PEG functions are higher order. Combnators take functions and produce parsing functions.

Recursive Descent

functions call other functions to recognize and produce output.

Predictive recursive descent

functions call other functions and recognize and consume impot.

Stream lookahead to determine which branch to take (firsts and follows)

fail early, retry very little

Predictive Parsers are Expensive to Build

Packrat Parsers (memoizing parsers) work like recursive decent parsers but remember the work they have done. They are fast but use a fair amount of memory. Memoization is also a higher order parsing function which translates to a functional language like Erlang well.

Memoization has an advantage over recursive decent parsers because they can recall the parsed values created when pursuing failed branches.

Neotoma

Deines a metagrammer
generates packrat parsers
transformation code inline
memozation uses ets
parses and generates itself

Build a CSV Parser with Neotoma

rows <- row (crlf row)* - / '';
row <- field (field_sep field)* / '';
field <- quited_field / (f!field_set ! crlf .)*;
quited_field <- '"' string:

can't quite keep up with his grammer. Looks complex but I am sure it is nice to the trained eye.

The Erlang code that the above grammar translates to is rather straightforward I must say.

to create a parser you take peg files and run them through neotoma to create erl, then compile the erl file.

erl -pa ../neotoma/ebin
> neotoma:file("csv.peg")
> c(csv.erl)

Functional programming plus PEG + packrat = win.

Clear relationship between grammar and generated code.
Powerful abstractions and higher order functions make this much easier.



Erlang Factory: Patrick Nyblom: Erlang SMP Support - Behind the Scenes

Erlang Multicore Support

If you went to EUC this year, this is a good time to take a nap. This talk will be very similar to that one.

The erlang multicore support has been in the product for a couple of releases. we started five or six years ago with the single cpu virtual machine. Its still there, if you start on a single core machine you get the single threaded virtual machine. Now we have multicore support for the scheduler.

When we started out we had one scheduler and four priorities. You usually use just one, which is good. We had one queue and picked from the highest priority. We had one queue and depending events (reductions, io, etc) you where descheduled. All the IO drivers and things worked in the same way as drivers you write yourself. These drivers had exclusive access to the scheduler.

Look at this we had a 'bucket' of execution time and a bunch of 'balls' of execution time. Now we have multiple buckets and the balls of execution time are distributed through them. (interesting slide graphic). We did this in a number of conversion steps. We did that with

multiple schedulers - 1 instance per physical core,
then parrallelized io, then parallelized memory allocation.
finally we had multiple run queues.

Starting with multiple schedulers we developed some tools. some synthetic benchmarks (big bang - opposite of the ring case). That was good for this step because we only delt with processes and io. To ensure no deadlocks in the system we imposed a strict locking order. If you follow that order you can't have a deadlock. We imposed that locking order with a lock checker. If you compile a debug version of the vm you will notice that it says 'checking locks' this is the lock checker.

The techniques we had at hand was our own threading library. Not all versions of pthreads was good enough. No possibility to use bitwise operations on atomic variables, test and set operations, etc. No these are in gcc, but at that time they weren't. A guy at upsalla university implemented a thread library for all the processors he loved - which proved to be a lot of processors. Reading that code is really pleasant ;)

We also used a technique called lock tables, more or less hashing the locks. If you need mutual exclusion and you have a million processes you will get a lot of locks and take a lot of memory. There is very little congestion on this locks. so there is an opportunity to share the lock. This allowed a reduced number of locks. Having these lock tables and custom lock processes and a lot of icky code actually reduces the memory you need. Normal mutual exclusion lock takes several words in a pthread implementation. There will be a lot of memory consumption just for the locks. So have a custom implementation of locks and these locking tables.

Even if our own thread library isn't available, you can fall back on pthread implementations. Thats when we send code to upsalla and get new code.

This worked with the big bang test. At that time multicore with more then two cores where not common and more then four didn't exist. You have to make tradeoffs between memory and speed. We also found that lock order enforcement was essential. (Debugging deadlocks is not what you want to do).

Now we had a toy application (that took one year). You could run parallel processes, do graphs, etc. However, you couldn't run a real system because you couldn't do IO.

We ran more bench marks doing IO. We could measure see that the bottleneck was IO. so we had to do something about that. So we started to do more fine grained locking when it came to io. Most of the work was around the driver implementation. We had drivers that we hadn't written, so we had to invent a differentiated locking for drivers that we didn't write ourselves. So we had two different levels of locking.

So we had to reimplement the leading of code, and we had to reimplement on windows.

When you do io, when you do this command thing you are in the middle of executing a process. so you have three locks. That gets complicated, you need to release a lock here. We realized, why not schedule the io operations. So when you get into a complex situation that you can't solve without 100 lines of interprocess locking you schedule the task for execution.

So the result was that we got parallel applications for real. Our first customers started to benifit from them. Doing things at the right time helps an application like this. We also got better performance from it.

Now we got our hands on 'vtune' an application from intel. It has a tool called a thraed profiler. On windows it worked. We ran that on windows and saw horrible things happening. What we deduced was that it was memory allocation that was the problem. We had our own memory application, we never used malloc. We have several instances of memory allocation with different semantics depending on what type of memory it was. You can see that in the code, every allocation has a little tag that tells you what types.

So we already had a bunch of memory allocators. In the big bang we immediately got congestion on the locks. We realized that you cant' have a central source. We created one memory allocator per scheduler per type of memory. There is still some more work to do here. You may not need one for each type of memory.

So what happened was that we greatly improved performance on cpu intense applications with many cores. There is still locking, but we don't get collisions on those locks.

Then we went on to looking at the run queues. We implemented some more tools, like the lock counters. This is custom for our own application and understands the semantics of that application. Its not a lot of code, it exists in the virtual machine.

We got our hands on some tilera and nehalem processors, which at that time was special. Tilera is a really cool processor and a really cold processor. It has many cores running very cool, it doesn't get hot. So we started, where trying to distribute the jobs for the queues over different schedulers. That proved to be harder then you might think. You have to balance long running and short running jobs but you can't balance to often, so it makes it hard. We came up with some fancy algorithms for this and had much better performance for this.

What is the distribution marshaler

its simply a something for marshaling over the network and it had some locks.

So distributed systems also benefit from this step. Nehelam performance got really great, core two wasn't as good. What we realized, the big insight, is that no global lock will ever fail to create a bottleneck. If you have four core processes you may be able to get away with a lock. If you have more then that you will not be able to get away with it.

What we did with the multiple run queues. What we have now is (for example) a three core system (which is very common ;) you have one run queue per scheduler and you have this migration logic which I might try to explain. The logic is, when you have executed a number of reductions globally in the vm you start load balancing. You look at the maximum run queue length of each scheduler during this period and divide it by the number of schedulers and that creates a limit. If a scheduler has more jobs then the limit migration queues are set up, if it has less then immigration queues are set up. The jobs balance across these queue slowly during execution.

You slowly take jobs from each other. You can run out of jobs, so you have work stealing. We also do, when you have too little work, when you look at a processes from a load balancing state you can also set them up to sleep. We want to have a number of cores working at maximum. If we have to many cores and too little jobs we can set them up to sleep. If they don't need to work they shouldn't. We also look at the average number of reductions consumed by the scheduler. An indication that the core isn't exclusive to the vm is that it doesn't use many reductions. We can then send it less work.

Thats a little complicated but it works really good. looking at the tilepro64 (graph) red line is with multiple run queues, blue line is multiple run queues. Why the drop at 32 cores? thats the limit after that they start sharing, that is bad.

We compare also to clovertown xeon and gainstown xeon. 'gainstown' is the nehelam cpu. We see the speed up with multiple run queues is almost linear up to 8 cores.

Interprocess communication benchmarks are also linear on the nehelam. Nehelam is an amazing processor (I don't work at intel)>

Insights,

No global lock goes unpunished
You have to distribute data as well as execution over threads
(Did I mention that global locks are no good?)
A lot of locks cost a lot of memory, its a tradeoff hard to do perfectly
You can see the difference between the core2 and nehelam architecture. (Big difference).
Massive multicore CPUs are different, we may need multiple approaches.
Use tools, it might even be worth buying a tool from now and then. You can also implement things in your own code, might be better but also costs time.
Work incrementally, some increments took a year.

Tools
We used vtune, our own lock checker
oprofile is open source and does the same thing as vtune. Both are hard to use.
We used our lock counter which you can use also.
Percept is also a good tool. Use to look at why some things didn't parrellelize like we wanted to.
We used customer code and our own benchmarks
We also used valgrind

What now we are looking at now is non-uniform memory access. We want to have schedulers use the right memory and so on. Delaying and deallocation and scheduled deallocation of memory. One scheduler allocates a message and another consumes it. Then you don't need locks. Even if we don't remove locks delayed deallocation has proven to be effective. You usually want to deallocate the memory when you are done and not at some later point. Performance increases. So we need to develop our libraries, like mnesia and other big applications like that need to be optimized and profiled.

We need to use less memory.

How can you access the lock counter

Conditional compilation (make lock counter, or configure for it). it should be documented.

Can you say how these results might change for multi-socket systems

The difference between multisocket systems and multicore systems is caches. Multisocket is non-shared cache, other is shared cache. Better to keep memory on one process. That is an issue for R14+.

Are the schedulers preemptive

Yes in a way, its not a timeslice in terms of actual time. Its the number of reductions. When the number of reductions is consumed its scheduled out. It works fairly well except for built in functions.

Reading the release nots R13B03 - B04 I don't see much to affect disk io? But we saw better performance.

Its a complete surprise to me. We haven't done anything in that area as far as I know.

Erlang Factory: Cliff Moon: Fast Enough

I had a bunch of slow erlang that I ported to C and now I have two problems. So we are going to talk about how to be fast enough.

Performance Tuning, Profiling all the things you need to do before this point. Through put, Online transaction processing, 99.9% latency. These are the things that typically matter. How do we go about profiling.

Identify bottle necks.
Optimize algorithms
Port critical sections to C
Repeat until satisfied.

Erlang VM

In R13B on scheduler per core. Each scheduler has its on run queue. Async thread pool and driver managed threads. These things are what we have to work with.

Three methods of porting

NIFs
Linked in Drivers
C Nodes

We are going to be talking about the first two. Both are dynamic libraries that are loaded into the vm. they can crash the vm.

Lets talk about nifs since they are the latest greatest shiny. Have interesting run time characteristics. All of this only holds for R13B04 thats what I did this work for. The internals change enough for that to matter.

Chracteristics of NIF
Low overhead
must be reentrant
no setup or teardown
explicitly tied to module versions.

(Nif developement cycle)
goes from unloaded to loaded, gets called when on module loaded. reload on reload, upgrade during code upgrade and unload when no more modules in the system that actually talk to this.

This is all tied to the shared library itself. Any sort of things that need to happen during these lifecycle functions. Nifs give you an opaque box for your environment, priv_data stash for state between nif calls. load_info is a way to take terms from the erlang side and put them into these callbacks for load and unload.

(nif example)
How many know C (most of room). How many know Erlang VM C code (slightly less). The vm C code is very nice code in comparison to other vms like ruby/python. Nif code is very clean, clear code. nif is transition to calling a function pointer.

Case Studies. FNV Hash.

Went through a couple of iterations to make it faster before I reimplemented it in c. First two implementations where slow. finally moved it to a nif.

Nif is simple, manpages are great. I don't want to turn this into a set of manpages on slides.

(c side nif example)

(erlang side nif example)

new on_load directive. Indicates a function to call when a module is loaded into the vm.

Benchmarking NIFs

simple example, built with rebar. just compiled hash example. Lets see what we can do here. The bench marking just creates a really big binary and runs through the three implementations of nif.

three runs, nifs where an order of magnitude faster then the fastest Erlang implementation.

times are in wall times. Its just a demo more then a bench mark.

Linked In Drivers (nothing to do with social networks)

Runtime Characteristics.

Going to go into some of the common uses. Don't want it to devolve into man pages.

(linked in driver example)

The way you interact with linked in drivers (after open port) there are several different ways. The fully synchronous call is via the 'control', control is expected to return a value. It can interface with async facilities, but ment to be synchronous.

Thing to remember with linked in drivers is that unless you are managing async calls everything is synchronous. Async thread pool is mainly done via the driver_async work.

You can do things like thread affinity, labels, etc. Async calls must be thread safe, you must do the threading yourself.

(slide, async thread pool handling)

Async is a pretty good thread to reward ratio. If you need async behaviour in a linked in driver please use the async thread pool.

If you need fine grained concurrency control you need to use user managed threads. There is a set of callbacks that will handle thread queues and queue data. It gives you a lot more control over the concurrency level.

(side, user managed thread pool)

The third way to handle things is the Event based async model.
driver_select/ready_input/ready_output/ErlDrvEvent. These get called when your file descriptor is ready for reading or writing.

Choosing an Async Model

You have a wide array of stuff to choose from.
Erlang managed threads are easiest.
thread pool needs to be configured
stack size needs to be configured - Small amount available by default. You may need more - be aware.
User Managed threads can be customized
Higher complexity, but give you the customization you might need.

When linked in drivers are tieing up scheduler threads. can get thread starvation. Ends up with an unresponsive erlang process of doom.

Libgd, case study
Done a lot of work on systems that handle and manipulate images. Gd is a library that resizes and crops images. These are long running operations. Async drivers come into play.

(shows async linked in driver support).
(running the erlang demo)
everything worked, image was resized.

Close things up.
NIF for simple CPU bound operations
Linked in Drivers for IO and fine grained concurrency

Erlang Factory: Kresten Thorup: Erjang

This is my project to learn Erlang. There are a lot of questions in the community about wether this is possible at all. This ignited me, so I had to go prove that it is possible.

I have a software company of around a 100 people and we are mostly doing java. We are the goto guys for java in denmark. We where a very early adapter of java. We have been very successful with that. Recently I have been getting the sense that it is starting to java is starting fall apart, in addressing this complexity. There seem to be some cracks that are starting to show up. We do a lot of coordination around integrating webserves. Utilizing multicore stuff that is starting to be more and more of an issue in Java.

This coordination stuff tends to be like the error handling stuff in C++. Coordination stuff tends to look like this. We have been doing systems level programming in Java. I have been doing a lot of threads and locks and its really really complicated. there are a lot of books on the topic of concurrency in Java. They are survival guides. They are not about makeing your code more elegant. There is a constant push for more and more complexity. I think a lot of this complexity comes from coordination.

We are being pushed to do more and more. (graph) over time there is a demand for more and more complexity. There are a lot of jumps. The jumps are via performance enablers. The jumps come also from new ways to understand and manage complexity. A good example of this is OO. However, there is a big thing thats misising here. That is al of this coordination modeling.

I am not sure that Erlang is the next java of application programmers, but it certainly has a bunch of its parts right.

the whole goal of this project was for me to learn erlang. To know where to push this in my own company. Its not easy to push this into a company of 120 people. I also wanted to use this to talk about actor modeling and understand how to structure concurrent programmers. I also learn this as a vehicle to learn really nice people.

One of the things we do is organize conferences. This is an opportunity for me to meet potential speakers for my conferences. We do conferences at various places around the world.

Enough context, on to the action. Before I do that, I have been giving talks on this many times. I have given this talk mostly to audiences that do not know Erlang. This is different to give to an audience that knows Erlang. This time I will go strait to Erjang.

Erjang is an execution engine for executing beam files, written in java. It does compilation from beam code to jvm byte code. The principle difference between Erlang and Erjang is that there is noly one memory model. that is the shared heap model. Everything is written in java. It sits right in the middle of the stack right where erts would sit.

When its complete it should be able to run any Erlang program. Lets show what it can do right now. Mind you at EUC I was just contemplating this project trying to convince people that it was doable.

(demo)
Its run just as a jar file. It must know where the OTP root directory is located. For a long time this was my milestone (running the erl shell). The measurement was the number of erlang modules that would successfull load from beam into jvm code. Just getting to this point, there was a lot of stuff running.

erlang:processes().

Many services up and running already at this point. Getting to the point where basic commands worked in the shell was a major milestone.

I will talk just a little bit more about where things changed. The big difference is the shared heap. Thats the only fundamental language feature that I don't think I can do. That is have a per process heap as beam/erlang typically has it. The JVM has been making a lot of effort to make the heap and memory management in a shared heap efficient.

It also doesn't do anything natively. That just isn't in the java way of doing things. Thats nice in that you can't load any code in the jvm that will make it fail. So there might be some nice security features on the jvm.

I hav some samples I will show you here for performance. Of course, the real interesting behaviour is when you get real applications up and running. There also might be some advantages in IO. In Erjang its not being copied into the heap.

Lots of good reasons for going with java. Its everywhere, it has a lot of libraries and integrations, lots of engineering and performance stuff, and the jvm is pretty fast. I have tried to push and leverage all of this existing engineering on the jvm. Thats a science in itself. Getting around the assumptions of the JVM is hard.

People ask me, why would you use this? I don't care, I am just learning Erlang. But for what is there its a pretty good piece of work. Its not just a toy, I am doing a real thing.

You could imagine running Erlang in a java friendly environment. Many of our customers are Java only, even IBM websphere only. So you can deploy anything as long as it runs in Websphere. This makes a beam engine running in Java very interesting. You can also run it places where Erlang currently does not run.

Its also nice to make java products available in the erlang environment. Basically make some java component be a proper player in a Erlang cluster.

It could also be that some places Erjang is just better. Depending on tooling, Performance characteristics.

How does a jvm make programming run fast.

Polymorphic dispatch really dominates performance. That is a level of indirection that is costly. The jvm does a good job of removing this indirection. The special thing is that it does all of this at run time. Modern JVMS start in interpreted mode and gathers information about the things running. Then it jits the critical paths.

Really making java programs fast is preserving type information, so you don't loose type information across calls. There are a lot of challenges.

How to have a lot of processes.
How to do tail recursion
How to implement bignums
How to do pattern matching
How to be typesafe
You might have to insert type information (smalltalk may be a better platform)

Realtime Behaviour - every once in awhile you will get global garbadge collections, so realtime behaviour isn't possible. In some cases this might be faster.

The core functionality of Erjang is the JIT compiler.
Beam Reader -> Type Analysis -> JVM Codegen -> *.jar

I cache these jar files by giving them a checksum name so I don't have to compile them all the time. If I delete my code cache and try to run it again

(demo) see lots of things being compiled -> beam files to jvm byte code.
ls() ->



Does code reloading on the fly. Erjang makes use of the jvm code reloading. A small problem in that there is no code unloading, but with a bit of effort can make sure the code gets unloaded.

In OTP what happens is that when a module is undefined it will call an error handler, that will propagate the error. I am working directly in that core infrastructure.

An Erlang module becomes a class, a function becomes a static method + an EFun object, values become an specific kind of object instance.
Each of these things have an overhead.

Recusive function call, along with pattern matching, and tail calls.

What does this look like in java land. Look what the compiler generates.


It was a challenge reverse engineering what all those beam extensions ment. Its not documented. Translating x and y registers into a flat register space was very difficult.

So the java code actually looks very similar.


One of the core concepts of improving type information is virtual method calls in the value objects. This is pretty strait forward java code that might end up running pretty fast.

If we can make the code look really nice there is a good chance that the jvm can do a good job.

There is a whole heirarchy of different java classes that map onto erlang types.


Some of these classes end up getting generated during the compilation process.

Cons cells are an area where I am in active experimentation to figure out how to optimize these cons using specific subclasses of the Cons class.

Numbers have heirarchy, its one of the main areas of performance problems. Its an area where I could leverage more sophisticated compiler technology.


Performance

I start out much slower then Erlang, but for larger values of fibonacci, I get faster.

The big issues is light weight processes. What I use is a tool called Kilim. You don't need to do anythnig in the common case. I can generate the code as you saw before and pass it through the kilim byte code processor. Part of Kilim is a CPS transformer that I pulled out of kilim.

The essence is that it generates a lot of code. Does a simple code transformation, every function turns into a big switch (checking various states). There are a lot of instructions but they are all in the cache.

If you try to run 'The Ring' example. Actually I am doing pretty well. 10000 process ring, What you are testing is how fast you can context switch. With all that code, tuned erjang, out of the box erlang, in the beginning erjang is slower, but as the jit kicks in the erjang version becomes a bit faster. We are doing context switching on the order of 1-1.5 microseconds. At the same speed that beam can do it. I was quite happy with this. Erlang helps me because stacks are not that deep (due to tail recursion). If you add a little payload, I can be arbitrarily faster. Thats the prime advantage.


Going Forward

lots of things that doen't work yet.

Surprising how stable the OTP framework is in the presence of a bad vm.