The Erlang User Conference is the oldest and most well-attended Erlang conference in the world. Held annually in Stockholm in mid-November, it pulls together an eclectic mix of software developers interested in building extremely reliable, large-scale, distributed computing systems with Erlang. Like last year, the conference was held at the Astoria, a renovated movie theater on Nygobratan in Central Stockholm, and was organized by Erlang Solutions, a UK based company also responsible for the Erlang Factory conferences.
Below is a talk-by-talk recap of the conference:
Please email me with any corrections.
Bjarne Dacker began the conference by reviewing the growth of Erlang measured by conference attendees. Main points were:
May need to find a new venue, as they reached capacity this year with ~350 people.
There is now a stronger Erlang community than ever, with EUC, Erlang Factory San Francisco, Erlang Factory London, Erlang Lite Krakow, Erlang Lite Los Angeles, ACM Sigplan Erlang Workshops, and Tokyo Erlang.
Taken together, the numbers for Erlang conference attendees show exponential growth.
Jia Wang, Tobbe Tornkvist, David Evans, Jordi Chacon, and Tobias Lindahl traded off the microphone to describe three different tools created at Klarna in Erlang to facilitate their rapid growth. Klarna uses the Kanban development methodology, the agility is necessary as Klarna is quickly expanding their online payment solution from Scandivia to other parts of Europe. Challenges include a fast-changing code base that must remain extremely stable, and the pain of understanding business rules and internationalizing for many different countries.
Klarna is the largest Erlang-based company, with ~300 employees and ~50 attendees at EUC.
The first project covered how Klarna modified Dialyzer to produce dependency graphs of modules between projects. The problem was evident upon reviewing the graphs, as many dependencies went in two directions between the projects.
The second project covered POlish, a web GUI built to manage the workflow between developers and translators. Developers make code changes that include translations placeholders. They then launch the tool on their workstations, then email a URL to the translators. The translators log in with OpenID and perform the translation, which changes files on the local Git repo and then alerts the developer. Translation of the 25k line PO files used to take the equivalent of a single full time developer, so the tool represents a large savings.
The third project covered using Fitnesse / Slim / SlimErl to perform acceptance testing. Fitnesse is an existing tool that provides a wiki interface, where a developer or business user can provide a prose-description of the code as well as a truth table of values. At the push of a button, the test software then grabs the truth table, constructs test code on the fly, performs the test, and updates the wiki with results.
Klarna plans to release all of the code to the open source community in the future.
Links:
Melinda Toth presented RefactorErl (http://plc.inf.elte.hu/erlang/dl/). As you would expect from the name, RefactorErl allows you to perform the usual basic list of code refactoring functions (ie: moving, renaming, and changing signatures).
But RefactorErl is much, much more than that; it is actually a tool to help you gain insight into your Erlang code. RefactorErl provides a full-featured XPath style query language with selectors, properties, and filters to allow you to (for a contrived example) list all non tail-recursive functions that are called by a specific function consisting of less than X lines of code. The OCD side of me wants to start using this immediately to enforce code standards such as maximum levels of case statements, line length, function length, etc. RefactorErl can be called from the Erlang shell, integrates with Emacs and also offers a web-based GUI.
This is definitely on my post conference list of things to investigate further.
Links:
Graham Crowe gave a storied overview of his quality-focused career through Ericsson, describing what has lead him to believe in using Erlang/Quickcheck to test both Erlang and non-Erlang code. Graham is now involved in the testing effort for some extremely complicated and very profitable hardware for Ericsson.
The talk was slightly confusing for someone not versed in Ericsson acronyms, but given that Ericsson develops Erlang, ran the conference until recently, sponsors the conference, and counts a significant portion of the audience as either current or previous employees, the lack of background is excusable.
The most important and significant part of this talk was not the content, but that it happened at all. I’ve been to a half-dozen Erlang conferences in the past two years, yet this is the first time I’ve seen Ericsson really stand up and be counted as a production user of Erlang in public, and this is on a high profile project.
Links:
David Almroth gave a great overview of the challenges that game companies face trying to scale. With games such as World of Warcraft or Eve Online, it is a challenge to get more than ~50 players in the same place at the same time, otherwise the load overwhelms a server. This is partially due to the complexity of the games, and partially due to the game engines (such as Unity) that are coded along a single-threaded model.
Generally, the solution is a combination of the following:
PikkoServer creates technology that attempts to solve this problem. Their software has the ideas of users and floating masts. The users are assigned to different masts based on in-game geography, so users switch between the different masts as they move around the universe. As users congregate, the masts automatically shift to where the action is. Game engines need to write serialization code and some other bridge code in order to facilitate handoff of users between masts, and receiving events from users assigned to different masts. PikkoServer currently integrates with Unity, but other engines are on the way.
Excitingly, David announced that PikkoServer is aiming for a world record 1000 users in the same area at the same time. Current record is currently 250.
Links:
Joseph Norton presented Hibari, a NoSQL project developed by Gemini Mobile. Hibari breaks from the current set of popular NoSQL projects by using a Chain Replication architecture, allowing the system to support fast reads, failure tolerance, and strong consistency at the expense of slower writes.
Hibari is used in at least three different production systems developed by Gemini, the largest of which, a WebMail solution for a Japanese mobile carrier and Internet provider, launched in April
Gemini open-sourced Hibari in July 2010. Future plans include more benchmarks and Hadoop Integration, and Gemini is actively cultivating the NoSQL community in Japan. Overall, the project looks like a welcome addition to the growing list of SQL alternatives on the market.
Links:
Marcus Kern from MIG described a nail-biting scenario of building an SMS/MMS gateway project under a high-pressure time schedule (e.g.: hit the deadline or we fail.) MIG is the fastest growing privately owned technology business in the UK, and integrates mobile and digital communications and provides things like billing, messaging, advertising, gateways, etc. (Disclaimer, MIG is a Basho customer.)
Marcus described the development and evolution of “Project Buzzard”, an SMS/MMS gateway. MIG partnered with Erlang Solutions, and implemented a number of agile practices including pair coding, TDD, daily standups, regular retrospectives, stories/points/velocity, etc. For tools, they used screen sharing, Campfire, Skype, GitHub, and Pivotal tracker.
Impressively, given the scope of the project, the complexity, and the number of 3rd party integrations, the team was able to launch successfully and use Erlang’s hot code loading to apply patches and avoid downtime post launch.
Links:
Simon Thompson and Clara Benac Earle gave quick updates on a number of ProTest projects. Simon spoke about Wrangler, a code refactoring and clone detection tool, describing some of the challenges in detecting and removing duplicate code. (Hint: It’s hard.) He then spoke about updates to QuickCheck that will allow QuickCheck to infer state machines from eunit tests, which also sounds difficult, but quite exciting. Simon stressed that one of the important parts of inferring the tests is that you can see what you are actually testing, rather than what you think you are testing.
Clara then took the stage and described the upcoming integration between QuickCheck and McErlang. McErlang is a model checker that works by replacing core parts of the Erlang runtime, and the executing a depth first run through all the branches of a state space. This allows you to find timing bugs and concurrency bugs that may not arise under normal conditions, or even under a random scheduler like PULSE.
Links:
Samuel Rivas presented a fun allegory of how property-based testing leads directly to huge profits. It served as a great introduction to the power of property-based testing. Property-based testing, in contrast to unit testing, helps you describe what the software should do in more general terms. This allows you to create tests that test more edge cases, have more longevity, and are more clear than standard unit tests. Software can generalize from the properties in a way that it can’t based on single unit tests.
Links:
Thomas Arts gave a highly energized presentation touching on some of the potentials of QuickCheck in the automotive industry. The basic backstory is:
Thomas also described a slick library in the next version of Quickcheck that will assist in testing C code. The software (eq_c) analyzes C header files and auto generates interface code in a single line, allowing you to make native C calls extremely easily. Sadly, it’s only available with a paid Quickcheck license.
Links:
Raghav Karol and Torben Hoffman from Motorola described using Erlang and Quickcheck to test an application that connects two TETRA systems (TETRA is a mobile telecommunications standard.) The big challenge is that the system had to go into production with little to no chance to test with the other systems in advance.
The team found that Erlang allowed them to create a solution with lower development cost then expected, and Quickcheck allowed them to deliver with a high level of confidence. Post deployment, the team relied on Quickcheck to vet code before applying hot patches.
Links:
During the break, Bjarne Däcker awarded Kresten Krab Thorup the Erlang User of the Year award for his work on Erjang. To summarize the Bjarne, “Many people have started building Erlang ports to Java, but few people have followed it down to the gritty details. Krestin did.”
Krestin had no idea he was to receive the award. He gave a great “acceptance speech”, thanking his new friends and humbly stating that he had started the project after EUC 2009 as a way to learn about Erlang and distributed computing. (Wow.)
Links:
Tino Breddin, in one of the most polished polished talks of the day, presented what Erlang Solutions is doing in order to do their part of being the first line of support for Erlang. The basic idea is for Erlang Solutions to provide first level support for Erlang applications and users, reduce load on the Ericsson team. Critical issues and bug fixes still get routed through to the Erlang/OTP team.
Some highlights:
Links:
Jacoby Thwaites gave a mind-bending presentation spawned by the basic premise that servers are easy to test because they simply respond to requests, clients are hard to test because they maintain state, therefore if you had an architecture where you removed all idea of “clients” and just kept “servers”, it would be a big win. He explored this concept as part of his Google 20% time, and the result is an architecture called MadCloud.
The result feels like an inversion of control system, where the exact operations called depend on which variables are present in state as well as what variables are present. Taking the idea a step farther, since the state is functional and independent, it can be scattered across different datacenters.
Links:
I spoke about creating distributed applications with no single point of failure using Riak Core. Riak Core is basically a generalized version of Amazon’s Dynamo Architecture developed in Erlang. At its heart is the idea of splitting the dataset across virtual nodes based on consistent hashing, assigning virtual nodes to physical nodes, and allowing the physical nodes to sort out amongst themselves who is responsible for which virtual nodes based on a gossip protocol.
Links:
Rickard Green described improvements to the RWLocks machinist, replete with many graphs showing how and when the new mechanism improved Erlang VM performance. I stepped out during this talk, so don’t have a complete summary.
Links:
Kenneth Lundin described highlights for the upcoming R14B01 release, including, as well as a possible road map for the new R15 release.
Possible new features include:
Finally, he discussed erlang.org traffic. Ranked by country, United States is first, Sweden second, and then, interestingly, China and Russia are third and fourth.
Links:
Finally, Bjarne Däcker took the stage again, thanked all of the organizers, the speakers, the attendees, and the venue.
Content © 2006-2021 Rusty Klophaus