Planet Fp-Syd

Perdition 1.19-rc3 Released

2010-07-29T08:16:17+00:00

I have released version 1.19-rc3 of Perdition. This release fixes several problems. Perhaps most significantly, Thunderbird 3.1 can now re-connect to perdition SSL/TLS.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

kexec-tools 2.0.2 Released

2010-07-29T05:01:27+00:00

I have released version 2.0.2 of kexec-tools. the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen.

There are changes since 2.0.2-rc1.

The code is available as a tarball here and in git here. More information is available in the announcement email.

kexec-tools recently moved, so the URLs above are new.

Moved

2010-07-29T03:23:52+00:00

My home page and blog have moved. The new locations are:

Home: horms.org/
Feed: horms.org/pleb_blossom/rss.xml

R.I.P. Nedit

2010-07-27T12:18:00+00:00

For serious programmers, the text editor they user is an intensely personal thing. Try suggesting to an Emacs user that they should switch to Vim or vice-versa. Most would shudder at the thought.

My choice of editor for the last 15 years has been Nedit, the Nirvana Editor. Nedit has been an outstanding editor; feature full yet easy to use. When I first started using it, Nedit was a closed source binary-only download but sometime in the late 1990s, it was released under the GNU GPL.

Unfortunately Nedit has been suffering from bit rot and neglect for a number of years. The main problem is that it uses the Motif widget toolkit. For open source, there are basically two options for Motif; Lesstif, an LGPL reimplementation of Motif which has been basically unmaintained for a number of years, or OpenMotif released under a license which is in no way OSI approved. On top of that, Nedit still doesn't support UTF-8, mainly because Lesstif doesn't support it.

I have, in the past, tried to fix bugs in Nedit, but the bugs are not really in Nedit itself, but in an interaction between Nedit whichever Motif library it is linked against and the underlying X libraries. Depending on whether Nedit is linked against Lesstif and OpenMotif, Nedit will display different sets of bugs. I have tried fixing bugs in Nedit linked against Lesstif, but got absolutely nowhere. Lesstif is one of the few code bases I have ever worked on that I was completely unable to make progress on.

With Nedit getting flakier with each passing year I finally decided to switch to a new editor. I had already discounted Emacs and Vim; switching from Nedit to either of those two archaic beasts was going to be way too painful. Of all the FOSS editors available, Gedit seemed to be the closest in features to Nedit.

Unfortunately, Gedit does not compare well with Nedit feature wise. To me it seems to try to be simultaneously as simple as possible and to have as many features as possible and the features don't seem to fit together all that well from a usability point of view. On top of that, it lacks the following:

Regex search and regex search/replace. Apparently there is a regex search/replace plugin, but that uses a different hot key combination that literal search/repalce. Nedit on the other hand uses the same dialog box for literal and regex search/replaces; with a toggle button to switch between literal and regex searches.
Search and replace within the selected area only.
Highlighting of matching braces and brackets.
Language specific editing modes and auto indentation.
A macro language allowing further customisation.
A simple, quick way to go to a particular line number (for Gedit, Control-L is supposed to work, but doesn't).

On top of that Gedit could also do with some improved key bindings and some improvements to its syntax highlighting patterns. The Ocaml syntax highlighting is particularly poor.

I'm now going to try to use Gedit, by customising its setup and and using the plugin system to see if I can regain the features that made Nedit such a pleasure to use.

FP-Syd #26.

2010-07-25T12:11:00+00:00

On Thursday July 15th, we held the 26th meeting of the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had 18 people show up to hear our two presenters.

First up we had your correspondent (thats me) with a presentation titled An LLVM Backend for DDC. This presentation covered the problems with the current C backend, gave a description of LLVM, the options for using LLVM from Haskell, why the LLVM code from GHC was chosen and how if fits into the DDC compile pipeline. Finally I demoed the very wonderful LLVM CGI script which allows you to enter a small C program and view the LLVM output.

Our second presenter for the evening was Eric Willigers who presented on the subject of the ATS programming langauge. ATS is interesting because it offers functional programming with an advanced type system with things like dependent types and linear types but has excellent performance as shown on the Computer Language Benchmarks Game. Eric was able to demonstrate dependent types on a couple of list operations which certainly showed some of the promise of dependent types. ATS certainly does seem interesting but also seems to lack quite a bit of polish.

A big thanks to Eric for presenting and Google for providing the meeting venue and refreshments.

Goanna Command Style

2010-07-20T03:40:57+00:00

Most users will use Goanna integrated into their development environment, either Visual Studio or Eclipse. However, we also have a command line version called Goanna Central. Since I am mostly working on the analysis engine this is the version I use most often. And part of this entails to find open source projects and run Goanna over it. So, if you have an open source project, we might be watching you :)

Most open source projects provide configure scripts to generate makefiles. If that is the case using Goanna is a matter of configuring it with Goanna. There are two executables, goannacc and goannac++, that behave like gcc and g++. Configuring then just means to execute:

goanna@KITTYHAWK:~$ ./configure CC=goannacc CXX=goannac++

After this you can make you project as you are used to, with the difference that you will get feedback from Goanna.

Sometimes open source project do not provide a configure script. Last week I got my hands on an open source model checker - it is always some guilty pleasure to model check a model checker - and this project only included a makefile. Once all the necessary libraries were installed - the once provided were incompatible with my machine - such that the project could be build with g++, all that remained was to edit the makefile. It is always exciting to edit a file that says right at the top: Automatically-generated file. Do not edit!. To use Goanna required to find all occurrences of, in this case, g++ and replace them with goannac++. And then to make the project.

The output looks like this:

Building file: ../src/Ned.cpp Invoking: GCC C++ Compiler goannac++ -DDEBUG -I../include -O0 -g3 -Wall -c -fmessage-length=0 -Wextra -MMD -MP -MF”src/Ned.d” Goanna - analyzing file ../src/Ned.cpp Number of functions: 3 ../src/Ned.cpp:28: warning: Goanna[COP-assign-op] Missing assignment operator for class `Ned’ which uses dynamic memory allocation ../src/Ned.cpp:28: warning: Goanna[COP-copy-ctor] Missing copy constructor for class `Ned’ which uses dynamic memory allocation Total runtime : 6.65 seconds

That was about all it took. BTW: Kittyhawk is the name of my machine, and it is aptly named.

Flying

2010-07-20T00:55:18+00:00

Due to tyre troubles with my other two bikes I was "forced" to ride my Cannondale road bike to work this morning. The first time I've ridden it since well before Hikari was born last November. Wow, that bike is light and voom it flies. (And I put the brakes back together correctly after shipping it from Sydney!)

In semi-related news, its really warm in Tokyo at the moment.

First release of the CUDA backend for Accelerate

2010-07-18T14:31:07+00:00

During the first Australian Haskell Hackathon, AusHac2010, we finally managed to complete the integration of the CUDA backend, written by Sean Lee and Trevor McDonell, into the array EDSL Accelerate. It still doesn't cover the complete functionality of the current version of the Accelerate EDSL, but already allows for interesting GPU computations. The package source (version 0.7.1.0) can be obtained from Hackage as usual. There is now also a bug tracker.

Permalink | Leave a comment »

LLVM Backend : Milestone #1.

2010-07-18T12:18:00+00:00

About 3 weeks ago I started work on the LLVM backend for DDC and I have now reached the first milestone.

Over the weekend I attended AusHac2010 and during Friday and Saturday I managed to get DDC modified so I could compile a Main module via the existing C backend and another module via the LLVM backend to produce an executable that ran, but gave an incorrect answer.

Today, I managed to get a very simple function actually working correctly. The function is trivial:


  identInt :: Int -> Int
  identInt a = a

and the generated LLVM code looks like this:


  define external ccc %struct.Obj* @Test_identInt(%struct.Obj* %_va)  
  {
  entry:
      ; _ENTER (1)
      %local.slotPtr = load %struct.Obj*** @_ddcSlotPtr
      %enter.1 = getelementptr inbounds %struct.Obj** %local.slotPtr, i64 1
      store %struct.Obj** %enter.1, %struct.Obj*** @_ddcSlotPtr
      %enter.2 = load %struct.Obj*** @_ddcSlotMax
      %enter.3 = icmp ult %struct.Obj** %enter.1, %enter.2
      br i1 %enter.3, label %enter.good, label %enter.panic
  enter.panic:
      call ccc void ()* @_panicOutOfSlots(  ) noreturn
      br label %enter.good
  enter.good:
      ; ----- Slot initialization -----
      %init.target.0 = getelementptr  %struct.Obj** %local.slotPtr, i64 0
      store %struct.Obj* null, %struct.Obj** %init.target.0
      ; ---------------------------------------------------------------
      %u.2 = getelementptr inbounds %struct.Obj** %local.slotPtr, i64 0
      store %struct.Obj* %_va, %struct.Obj** %u.2
      ; 
      br label %_Test_identInt_start
  _Test_identInt_start:
      ; alt default
      br label %_dEF1_a0
  _dEF1_a0:
      ; 
      br label %_dEF0_match_end
  _dEF0_match_end:
      %u.3 = getelementptr inbounds %struct.Obj** %local.slotPtr, i64 0
      %_vxSS0 = load %struct.Obj** %u.3
      ; ---------------------------------------------------------------
      ; _LEAVE
      store %struct.Obj** %local.slotPtr, %struct.Obj*** @_ddcSlotPtr
      ; ---------------------------------------------------------------
      ret %struct.Obj* %_vxSS0
  }

That looks like a lot of code but there are a couple of points to remember:

This includes code for DDC's garbage collector.
DDC itself is still missing a huge number of optimisations that can added after the compiler actually works.

I have found David Terei's LLVM AST code that I pulled from the GHC sources very easy to use. Choosing this code was definitely not a mistake and I have been corresponding with David, which has resulted in a few updates to this code, including a commit with my name on it.

LLVM is also conceptually very, very sound and easy to work with. For instance, variables in LLVM code are allowed to contain the dot character, so that its easy to avoid name clashes between C function/variable names and names generated during the generation of LLVM code, by making generated names contain a dot.

Finally, I love the fact that LLVM is a typed assembly language. There would have been dozens of times over the weekend that I generated LLVM code that the LLVM compiler rejected because it would't type check. Just like when programming with Haskell, once the code type checked, it actually worked correctly.

Anyway, this is a good first step. Lots more work to be done.

Singleton: A general-purpose dependently-typed assembly language

2010-07-17T12:31:05+00:00

In the paper Singleton: A general-purpose dependently-typed assembly language, we introduce Singleton, a typed assembly language that embeds a variant of the calculus of inductive constructions (CiC) as an assertion language in its type system. Moreover, Singleton procedures can abstract over propositions and proof terms. Singleton's name comes form its support for generalised singleton types that allow the values from arbitrary inductive types to be associated with the contents of registers and memory locations. Together these features allow strong statements to be made about the functional behaviour of Singleton programs.

Singleton's development was motivated by the desire for a typed assembly language that can establish high-level properties that are not implied by type safety alone, a requirement that we encountered in the context of runtime verification. For more details on that context, see our previous papers Secure Untrusted Binaries - Provably! and On the automated synthesis of proof-carrying temporal reference monitors.

Simon Winwood, on whose PhD work the paper is based, established basic properties of Singleton's type system, namely type safety and a type erasure property, using the Coq proof assistant: proof scripts.

Permalink | Leave a comment »

kexec-tools 2.0.2-rc2 Released

2010-07-13T03:01:14+00:00

I have released version 2.0.2-rc2 of kexec-tools. the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen. The only change since 2.0.2-rc1 is to include a number of files that were missing from the tarball which caused build failures on x86_64, ppc, ppc64 and arm.

The code is available as a tarball here and in git here. More information is available in the announcement email.

Perdition 1.19-rc2 Released

2010-07-10T01:13:03+00:00

I have released version 1.19-rc2 of Perdition. There is a single change since 1.19-rc1 to fix a crash-bug on invocation of perdition in managesieve mode. Merge madness on my part. Thanks to Filipe Carvalho for reporting it. The fix is here.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

Haskell 2010 is complete!

2010-07-07T10:36:00+00:00

The final version of the Haskell 2010 edition has been released: http://www.haskell.org/pipermail/haskell/2010-July/022189.html This completes my stewardship of the Haskell foreign function interface specification. It is now part of the language standard proper. The next language feature I personally like to see being integrated are type families, but that still requires much work.

Permalink | Leave a comment »

Goanna Studio 2.0

2010-07-07T03:48:47+00:00

It is out! We just released a major upgrade to Goanna Studio version 2.0. There has been a lot of work going into the new version and some of the new key features include:

Full (whole program) interprocedural analysis to track effects across functions and files
Incremental analysis to minimize time for reanalyzing files/projects
Around 100 classes of checks, up almost 70% compared to the previous release
Much improved precision and elimination of some existing false positives
Improved Path Simulator to display error traces
New project reporting mechanism and export facilities

For existing customers:

We are also happy to announce that all existing customers have the possibility to upgrade to 2.0 free of charge!
If you were a trial user in the past and need a trial extension visit: http://redlizards.com/trial-extension

Overall, the new version is another leap forward and enables to detect more and deeper critical issues early in the development cycle.

Perdition 1.19-rc1 Released

2010-07-06T02:47:19+00:00

I have released version 1.19-rc1 of Perdition, a mail retrieval proxy. This release includes numerous bug fixes and support for the manage sieve protocol.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

Perdition 1.19-rc1 Released

2010-07-06T02:47:19+00:00

I have released version 1.19-rc1 of Perdition, a mail retrieval proxy. This release includes numerous bug fixes and support for the manage sieve protocol.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

kexec-tools 2.0.2-rc1 Released

2010-07-05T07:15:14+00:00

I have released version 2.0.2-rc1 of kexec-tools, the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen. The code is available as a tarball here and in git here. More information is available in the announcement email.

LLVM Backend for DDC.

2010-06-28T20:51:00+00:00

With the blessing of Ben Lippmeier I have started work on an new backend for his DDC compiler. Currently, DDC has a backend that generates C code which then gets run through GNU GCC to generate executables. Once it is working, the new backend will eventually replace the C one.

The new DDC backend will target the very excellent LLVM, the Low Level Virtual Machine. Unlike C, LLVM is specifically designed as a general retargetable compiler backend. It became the obvious choice for DDC when the GHC Haskell compiler added an LLVM backend which almost immediately showed great promise. Its implementation was of relatively low complexity in comparison to the existing backends and it also provided pretty impressive performance. This GHC backend was implemented by David Terei as part of an undergraduate thesis in the Programming Languages and Systems group an UNSW.

Since DDC is written in Haskell, there are two obvious ways to implement an LLVM backend:

Using the haskell LLVM bindings available on hackage.
Using David Terei's code that is part of the GHC compiler.

At first glance, the former might well be the more obvious choice, but the LLVM bindings have a couple of drawbacks from the point of view of using them in DDC. In the end, the main factor in choosing which to use was Ben's interest in boostrapping the compiler (compiling the compiler with itself) as soon as possible.

The existing LLVM bindings use a number of advanced Haskell features, that is, features beyond that of the Haskell 98 standard. If we used the LLVM bindings in DDC, that would mean the DDC would have to support all the features needed by the binding before DDC could be bootstrapped. Similarly, the LLVM bindings use GHC's Foreign Function Interface (FFI) to call out the the LLVM library. DDC currently does have some FFI support, but this was another mark against the bindings.

By way of contrast, David Terei's LLVM backend for GHC is pretty much standard Haskell code and since it generates text files containing LLVM's Intermediate Representation (IR), a high-level, typed assembly language, there is no FFI problem. The only downside of David's code is that the current version in the GHC Darcs tree uses a couple of modules that are private to GHC itself. Fortunately, it looks like these problems can be worked around with relatively little effort.

Having decided to use David's code, I started hacking on a little test project. The aim of the test project to set up an LLVM Abstract Syntax Tree (AST) in Haskell for a simple module. The AST is then pretty printed as a textual LLVM IR file and assembled using LLVM's llc compiler to generate native assembler. Finally the assembler code is compiled with a C module containing a main function which calls into the LLVM generated code.

After managing to get a basic handle on LLVM's IR code, the test project worked; calling from C into LLVM generated code and getting the expected result. The next step is to prepare David's code for use in DDC while making it easy to track David's upstream changes.

FP-Syd #25.

2010-06-28T08:08:00+00:00

On Thursday June 17th, we held the 25th meeting of the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we about 20 people show up to hear our two presenters.

First up we had Eric Torreborre on the subject of Specs a Scala library designed as an alternative to JUnit for testing Scala and Java projects. Specs seems to remove a lot of the boilerplate normall associated with unit testing.

Our second presenter for the evening was Shane Stephens who presented on the subject of Arrows, a generalisation of Monads. Shane's presentation was originally scheduled to be a 5-7 minute lightning talk but due to a lack of other lightning presenters, Shane managed to pad it out to a full talk in the hour before he presented. According to Shane, his presentation was based quite closely on the original paper "Generalising Monads to Arrows" by John Hughes.

A big thanks to Eric and Shane for presenting and Google for providing the meeting venue and refreshments.

A few words on code style

2010-06-25T02:29:35+00:00

I wish that I was writing about a new code for style but instead I'm writing about poor coding style. I'll keep it brief.

If you have a very large function that is doing lots of things, its a sign that some refactoring might be in order. If not for yourself, to aid others who have to work on the code.
If the things are interlaced for no apparently good reason, you've created a maintenance monster.

New paper: An LLVM Backend For GHC

2010-06-15T05:31:53+00:00

In An LLVM Backend For GHC, we describe the design and implementation of an LLVM backend for the Glasgow Haskell Compiler (GHC). We outline the conceptual obstacles that we had to overcome and give a detailed performance analysis of the new backend compared to the existing C backend and native code generator.

Permalink | Leave a comment »

One week with iPad

2010-06-07T01:16:23+00:00

When I decided to by an iPad, I had one killer application in mind: reading. I was hoping the device would also be useful otherwise, but cutting down on printing papers and the like was my main aim. As a researcher, I inevitably read a lot: research papers, books, reports, manuals, thesis drafts, the drafts of my own work-in-progress papers, and so on. I have got reading material on my desk at home, in my office, in my bag...it's a mess. The prospect of consolidating it all into one device, which would hopefully also cover most of my note taking and sketching out of ideas, was enough to sell me on the iPad.

So, how did it work out? So far, very well — but slightly more than a week of use makes it hard to draw any final conclusions. The printed area of a paper is about the same as the size of the iPad screen, which combined with the high quality of the display makes for a good reading experience. Instead of printing a paper or other document I like to read, I now sync it to my iPad —usually using Dropbox— and read it with GoodReader for iPad. If I expect that I may want to refer to a paper at a later time, I'm adding it to my mobile library kept in Papers. If I need to annotate a document —for example, for a review— I currently use iAnnotate PDF. I like Dropbox, GoodReader, and Papers, but I'm only using iAnnotate as there doesn't seem to be an alternative at the moment — the user interface is pretty rough.

In addition, I found over the last week that iPad is also the perfect Twitter device (I'm currently using Twitterific for iPad), and great for reading blogs and online articles, which I often file away for later reading on my iPad using Instapaper.

Permalink | Leave a comment »

FP-Syd #24.

2010-05-30T00:08:00+00:00

On Thursday May 20th, we held the 24th meeting of the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we a bit over 20 people show up to hear our two presenters.

First up we had Raphael Speyer presenting his impressions on leaning the Ocaml programming language and applying to writing a version of the game "Four in a Row" also known as "Connect Four". The game is a two player game and Raphael implemented the game rules and some basic artificial intelligence so that a single player can play against the AI. Rapahel seemed to have a positive experience with Ocaml and the Batteries Included libraries. The performance was quite as good as he would have liked, but discussion seemed to suggest that switching from lists to arrays would probably improve things. Raphael has made his code available as a github project.

Our second presenter for the evening was Eric Willigers who presented on the subject of the Scala language's Actor library, a library implementation of Erlang's message passing primitives. Eric gave us a number of demos that showed how Actors behave.

A big thanks to Raphael and Eric for presenting and Google for providing the meeting venue and the snacks.

More SMH Bashing

2010-05-17T12:43:29+00:00

I read an interesting article in the SMH a few days ago about a technique for preparing children to learn to read by increasing their ‘phonemic awareness’.

The first thing I think when I read about any innovation is “does it really work?” — the evidence was addressed by a single paragraph:

After 13 weeks of the program, 5 per cent of children improved in language tests administered by the foundation workers and the teachers. The number of three-year-olds who passed the tests after the program doubled.

Which tells us very little. Is that 5 per cent figure net of the children who did worse? Or is it gross? Why is the three year old improvement stated differently? Did the journalist just feel that it read better to avoid a similar turn of phrase, or did the educator phrase it that way for some reason? Obviously it makes a difference whether the 3 year old pass rate increased from 1% to 2% or 50% to 100%. Most importantly — as 13 weeks is a long time in a child’s life, especially a three year old — you would expect improvements no matter what. A trial with a control group, and blind administration of the tests would make the figures much more convincing.

No doubt there is better evidence available, as “other preschools in the area are clamouring for the program”, so it’s a shame that the journalist couldn’t have presented that evidence in their article.

FP-Syd #23.

2010-04-28T11:25:00+00:00

On Thursday April 15th, we held the 23rd meeting of the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had 28 people show up to hear our two presenters.

First up we had Ben Lippmeier demonstrating Gloss, a library for 2D vector graphics, animations and simulations. Gloss grew out of a library that Ben and others were using the teach Haskell to first year computer science students at ANU. It is now also being used at UNSW. The library was designed to allow students to get animations running without having to really tangle with monads. Programs using the library can achieve very impressive results with very few lines of code.

Our second presenter for the evening was Barry Jay demonstrating his Bondi programming language. Bondi is an Ocaml-like language in that it is strictly statically typed, is mainly functional but allows imperative constructs as well, is impure with respect to I/O and has an object system.

Bondi came about as a proof of concept language aiming to show that the the ideas behind Pattern Calculus actually have practical application. Bondi shows very clearly the value of being able to pattern match on arbitrary compounds objects without writing code to deal with the specifics of the compound. A simple example Barry came up with was defining a completely new data type Complex and having Bondi's pattern matching immediately able to figure out how to print it using the toString function as follows (the "~~" below is the Bondi REPL command prompt):


  ~~ datatype Complex = Cmplx of Float and Float ;;
  Cmplx: Float -> Float -> Complex

  ~~ toString (Cmplx 1.1 2.2) ;;
  it: String
  it = "Cmplx 1.1 2.2"

Towards the end, the questions and resulting discussion turned back to the Pattern Calculus its relationship to Lambda Calculus. This discussion continued at the pub where I think we decided that the Pattern Calculus is a super-set of the Lambda Calculus. For more on this topic I would recommend the slides to Barry's presentation at SAPLING '09 and the full paper version titled "A Combinatory Account of Internal Structure".

A big thanks to Ben and Barry for presenting and Google for providing the meeting venue and the snacks.

Goanna 1.4 release

2010-04-21T06:10:58+00:00

We are happy to announce a new release of our Goanna static analysis solution. After a previous release for Visual Studio 2010 we are now bringing new technology with new benefits to all our products:

Some of the advances include:

up to 300% speed improvement through improved core analysis engine
even less false positives due to improved check accuracy
updated output format to easier identify critical issues
32-/64-bit cross-compilation support in Visual Studio
support of Visual Studio property sheets
initial support of new C++0x standard (”auto” and other features)

If you like to test the new features and require a 30 day Trial Extension for your current Goanna version please complete this trial extension request form.

Regular, shape-polymorphic, parallel arrays in Haskell

2010-04-06T05:34:00+00:00

The paper Regular, shape-polymorphic, parallel arrays in Haskell describes a new Haskell library, named Repa, for multi-dimensional, regular arrays based on our work on Data Parallel Haskell and a delayed array representation avoiding unnecessary data structures. As the following table shows, the sequential performance already gets close to the performance of C programs. More importantly, the Haskell code transparently makes use of multicore hardware — something that is much more difficult in C.

The Haskell code is purely functional, using collective array operations. Here is matrix-matrix multiplication:

mmMult :: (Num e, Elt e)

=> Array DIM2 e -> Array DIM2 e -> Array DIM2 e

mmMult arr brr = sum (zipWith (*) arrRepl brrRepl)

where

trr = force (transpose2D brr)

arrRepl = replicate (Z :.All :.colsB :.All) arr

brrRepl = replicate (Z :.All :.All :.rowsA) trr

(Z:. colsA:. rowsA) = extent arr

(Z:. colsB:. rowsB) = extent brr

In addition to good performance, the library facilitates reuse through the use of shape polymorphism.

Permalink | Leave a comment »

Robert and Lynne's Wedding

2010-03-27T09:15:03+00:00

Today Chiz, Hikari and I were lucky enough to join Robert and Lynne, and their friends and family at Eden Gardens in North Ryde to celebrate their wedding. It was a very happy and relaxed wedding, very fitting for Robert and Lynne. Congratulations to you both.

Photos here

GHC 6.12.1 in Debian Testing.

2010-03-14T07:58:00+00:00

Joachim Breitner recently announced on the Debian Haskell mailing list that version 6.12.1 of the Glasgow/Glorious Haskell compiler was about to transition from Debian unstable to Debian testing. That has now happened. This means there is a very good chance it will be part of the next stable release of Debian.

A big thanks is due to Kari Pahula, the Debian maintainer for GHC who managed to get this version of GHC working on a bunch of CPU architectures not officially supported by the upstream GHC maintainers. Deserving of equal attention are Joachim Breitner and Marco Túlio Gontijo e Silva who did a large amount of real quality work to improve the way Haskell libraries are packaged in Debian.

The big change recently was drastic improvements in the way library dependencies are tracked across packages which will make it much easier to write tools to automatically check for broken dependency chains. Packaging Haskell libraries for Debian is now a relatively trivial and fool proof exercise. Packaging a library which is on Hackage can take as little as 5 minutes.

With the current version of GHC in Debian and a large and growing collection of Haskell libraries, writing Haskell code on Debian using nothing but Debian packages is now a pleasure. Ubuntu and other Debian and Ubuntu derived distributions of course also benefit from this work.

FP-Syd #22.

2010-03-13T00:57:00+00:00

On Thursday February 25th we held the first meeting for 2010 of the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had 17 people show up to hear our two presenters.

First up we had your correspondent (Erik de Castro Lopo), giving a presentation titled "Hacking DDC" on my bug fixing work on Ben Lippmeier's DDC compiler. I explained a little about what DDC and Disciple were; a Haskell like language with some interesting extensions to the type system. I then suggested that anyone curious as to why these extensions were interesting should read the first chapter of Ben's PhD thesis "Type Inference and Optimisation for an Impure World". I then went on how using Darcs for the revision control made it easy to use one branch per bug or feature I'm working on, specifically, it allowed me to work on one until I got stuck and then move on to another without the debugging of the first interfering with the second.

Our second presenter for the evening was Tim Docker who gave us an explanation of a Domain Specific Language (DSL) for handling dates in financial systems. Code written in his DSL looked a lot like Ocaml, but the implementation was in C++.

A big thanks to Tim for presenting and Google for providing the meeting venue and the snacks.

Intel Embedded Graphics Driver Fail.

2010-03-11T09:24:00+00:00

In my day job I do Linux embedded work and as people in the embedded world know, Linux is a pretty commonly used embedded OS. Today I was evaluating a new board and found it had an Intel graphics chip that was not properly detected by Ubuntu 9.10. The ever trusty lspci said this:


  00:02.0 VGA compatible controller: Intel Corporation System Controller Hub
        	(SCH Poulsbo) Graphics Controller (rev 07)

We all know that Intel employs a bunch of well known Xorg developers, so this shouldn't be a problem, right?

Unfortunately, it is a problem. Intel's offering for this chipset is the Intel® Embedded Graphics Drivers web page where they offer a 124 megabyte download (registration required). After registration you get to choose which driver pack you want and which OS you are downloading it for. Ubuntu was not on the list and neither was Debian. I chose Fedora 10 (released in 2008) as that was the most recent one.

Now, you can image my surprise when the driver download for Fedora Linux contained just four files:


  Archive:  /tmp/IEGD_10_3_GOLD.zip
      testing: UsersGuide_10_3_1525.pdf   OK
      testing: IEGD_10_3_GOLD_1525.exe    OK
      testing: IEGD_SU_10_3_GOLD.pdf      OK
      testing: RELNOTES_10_3_1525.txt     OK
  No errors detected in compressed data of /tmp/IEGD_10_3_GOLD.zip.

Yep, thats right, the driver download for Fedora Linux contains two PDF files, a text file and an executable installer for Windows.

Being the curious (and paranoid) type I decided to explore this further, by running the installer under WINE in a chroot. After the installer you get left with several metric craploads of Java Jar files, and another windows executable iegd-ced.exe that supposedly configures this nightmare. I ran it (again, under WINE in a chroot) but it didn't seem to do anything sensible or worthwhile so I looked around amongst the other installed files and found IEGD_10_3_Linux.tgz.

Inside that tarball there are a bunch of Xorg library binaries (for several different versions of Xorg), a large chunk of source code that gets compiled into the Linux kernel and even better yet, a couple of Microsoft Visual Studio project files. WTF?

Unbe-fscking-lievable. Needless to say, I will avoid any hardware which uses this chipset and any other hardware that requires binary only kernel blobs packaged this badly. Doing so makes my life easier.

The people at Intel who thought this was a good idea must have their own personal mother-lode of stupid.

Goanna 1.2 released

2010-03-10T10:50:09+00:00

Goanna version 1.2 has been released. Download it now.

The major change is More Checks, in fact 40% more than were previously available in v1.1. Over the next few months we will continue to add new checks with each release. You can expect to see up to 100 additional high quality checks within the coming 6 months, which as usual will be free for all existing customers. Additionally, should you require a 30 day Trial Extension for your version 1.2 update please complete this trial extension request form.

We are also very pleased to announce the Beta release of Goanna for Command Line. This new command line version enables more flexibility and freedom for those wishing to integrate our powerful C/C++ static analyzer into their own development process. The Beta is currently available for Linux users and a version for Windows users is scheduled to be available in May. Linux users can now access a fully gcc-compatible solution integrated with over 60 classes of flow-sensitive quality checks to detect critical bugs as early as possible in the development cycle.

Inter-procedural analysis is also well under way, so stay tuned for a public Beta release soon!

Visual Studio 2010

2010-03-05T01:42:10+00:00

We’re proud to have been selected for simultaneous shipment of our Goanna static analysis extension with Microsoft Visual Studio 2010. Here is a short introductory video demonstrating our Visual Studio 2010 integration, and we’re on schedule for April release:

We have some further news regarding recent developments (more high quality checks being one) and we’ll be posting more information next week.

Conroy : Is He a Duck?

2010-02-19T21:26:00+00:00

The Australian Federal Minister for Communications, Stephen Conroy, may not actually be corrupt; I certainly have no evidence that he is, but a number of recent incidents sure look like corruption to me. For instance:

Conroy's recommendation of a former ALP Staffer for a highly paid senior position (government relations) with the National Broadband Network an organisation operating in the area of Conroy's ministry.
Conroy playing golf with billionaire media mogul James Packer on the same day the government announced a $250 million licence fee rebate for free-to-air television stations (one of which is owned by Packer).
Conroy skiing with media mogul Kerry Stokes in Colorado a month before the $250 million licence fee rebate was announced. James Packer and Kerry Stokes are the owners of 2 of the 3 main non-government free-to-air television stations.

The real irony is that under Conroy's proposed scheme to filter the internet in Australia comments like this blog entry may end up being censored. The problem with Conroy's filter is not that it filters porn, but rather that the list of what is being filtered is secret and hence could easily include web sites which contain comments which the government or the Minister for Communications want silenced.

More Fun Tracing Parallel Haskell Programs

2010-02-19T07:08:17+00:00

I recently wrote about profiling sparse-matrix vector multiplication implemented with Data Parallel Haskell. Let's take this a step further and look at a program parallelised with package parallel, instead of Data Parallel Haskell. Specifically, we look at the dense matrix-matrix multiplication benchmark of the parallel section of the nofib benchmark suite. This benchmark uses a linewise version of the torus-based Gentleman algorithm implemented with vanilla Haskell lists. As we see in the following profile, the algorithm parallelises very well on two cores. It also has a low allocation rate; hence, there is little garbage collection.

In the above profile and that of the sparse-matrix vector multiplication that I showed in my previous post, the width of the time slices of the mutator threads and of garbage collection are determined by the wall clock time of the run-thread/stop-thread and gc-start/gc-end events, respectively. These events are generated by the Haskell thread scheduler and do not —they cannot— take the OS scheduling of the Haskell Execution Contexts (HECs) running Haskell threads into account. In other words, a Haskell thread may be scheduled on a HEC for a long time, but if the HEC itself (i.e., the operating thread on which it executes) is descheduled by the operating system, then the Haskell thread still won't make any progress. To visualise this effect, the custom instruments used here also support graphing the virtual time that expired in a time slice. The following profile is a slightly magnified version of the initial portion of the previous profile, where the height of a mutator time slice indicates its virtual extent. The screenshot also shows the configuration pane for one mutator track. The virtual extent of time slices in the garbage collection pane can also be displayed in this alternative form, although that is not shown here.

The custom instruments for GHC provide addition information that is by default not displayed in the track pane, including the wall clock percentage during which the runtime performs garbage collection and the percentage of utilisation of the mutators threads. However, all information is accessible via Instrument's tabular detail view. In particular, the library function GHC.Exts.traceEvent can be used to inject user-defined messages into the event stream. In the DPH benchmarking framework, we use this for example to delimit the start and end of the benchmark payload, as opposed to benchmark set up, data generation, and tear down code. The following snapshot shows part of the detail view for the sparse-matrix vector multiplication benchmark, where the event indicating the start of benchmarking is highlighted.

By clicking on the user message events in the detail view, we can easily mark the relevant section of the profile — it is the section highlighted in blue. Below, we see that the sequential part of the profile is only part of the set up, but not of the benchmarked code.

If you like to try for yourself, download the trace template GHC HEC 2-core and use it in Instruments.

A current caveat: Instruments supports zooming the track views horizontally (on the time axis) with a slider. Unfortunately, in version 2.0.1 of Instruments, the maximum resolution is not sufficient to see the exact relative timing of mutator and garbage collection time slices if garbage collection is very frequent. Moreover, when using the Block Graph style, blocks that are very closely spaced may be coloured very lightly or appear transparent — if you are suspicious of a section that seems to show no activity in a profile, use the Peak Graph style to double check. Instruments collects all the data to provide a better resolution; it's just a matter of supporting a wider range in the zoom slider. (If anybody knows a work around for that, please post it in the comments.)

Permalink | Leave a comment »

Tracing Sparse Matrix-Vector Multiplication with DTrace

2010-02-14T10:10:46+00:00

The following is an example trace of a Data Parallel Haskell program multiplying a sparse matrix (10k x 10k elements with 10% non-zero elements) with a dense vector. The program runs on both cores of an Intel Core 2 Duo processor and uses GHC's new DTrace support on Mac OS X to gather the trace data, which I visualised in Instruments. Of the three tracks, the topmost shows garbage collection activity in blue. The other two show the activity of the two HECs (Haskell Execution Contexts) running the application code on two cores in green. The program starts by loading the matrix from disk and generating a random vector — this is where only HEC #1 does any work. Then, during the actual matrix multiplication, both cores are utilised almost evenly.

There is little garbage collection during the parallel matrix multiplication as GHC successfully unboxes the numeric code resulting in little heap allocation.

Permalink | Leave a comment »

Communication

2010-02-02T04:06:07+00:00

Is it ok to use chat to communicate with my wife?
If she is sitting at the next desk?
If there is a sleeping baby in between?

Is any of this relevant if she laughs?

Australia Day Fireworks

2010-01-26T11:47:52+00:00

This evening Chiz, Hikari and I went down to Darling Harbour to watch the Australia Day Fireworks. Although there were rather a lot of people down there we managed to find a good vantage point. Close enough to be impressed by the show. Much to my surprise I managed to take some reasonable snaps of the show. Photos here.

Golden Section - Jumping the Gun (1993).

2010-01-25T11:04:00+00:00

Almost two decades ago, when I was at university I played bass guitar in a couple of rock bands; Golden Section and Fishtank. Both bands playing an all original set although Golden Section did cover a song by Fishtank and Fishtank had a song without lyrics but found by accident that the lyrics to the Beatles' "Lucy in the Sky with Diamonds" worked, despite our melody being different.

Recently one of my bands mates in Golden Section posted a video of Golden Section playing live at the Palais Hotel in Newcastle (Australia) in 1993. The first real glimpse of me playing bass is at 1:10.

Playing in Golden Section was just so much fun! Working up to the 1991 Newcastle University Band Competition (which we won) we were rehearsing twice a week and as a result reached a state where the band synced like clockwork. We could play light and reasonably heavy (Fishtank was heavier) and we were even a bit funky on certain songs.

I absolutely loved playing in this band. Good times, good memories.

Profiling Garbage Collection in Haskell with DTrace & Instruments

2010-01-24T12:54:20+00:00

I recently added DTrace probes to the GHC runtime system. In the following, I will explore the use of these probes for profiling the garbage collection behaviour of Haskell programs. Mac OS X comes with a graphical tool, called Instruments, that supports visualising data gathered with DTrace. Let's start by having a look at visualising the DTrace events gc-start and gc-end for the standard nfib function (compiled with GHC 6.13 and -O).

The two most important parts of the Instruments window above are the track pane, which graphs numerical values computed by the DTrace probes, and the detail pane, which displays all probe data in a tabular view. Here, the track pane contains graphs for

time slices of the mutator,
time slices spent on garbage collection,
the running sum of the mutator time slices,
the running sum of garbage collection time slices, and
the running percentage of the runtime spent on garbage collection.

The height of each spike in the graphs is proportional to the length of the time slice (or the running sum), measuring virtual time (i.e., process time determined using DTrace's vtimestamp). It is important to note that the x-axis of the Instruments track pane uses wall clock time instead. (That is crucial when using Instruments to correlate the behaviour of two or more processes or when inspecting overall system behaviour.) This explains the gap about one third into the above run. During that time, the process executing our Haskell program simply was not scheduled by the operating system.

The detail pane in the above screenshot displays the numeric values corresponding to the DTrace events represented as spikes in the track pane. In addition, it also displays the event names.

Overall, our nfib implementation spends very little time on garbage collection, only about 1%. This becomes very clear in the following alternative few of the track pane, where we overlay the time slices for the mutator and garbage collection. Black spikes are mutator time slices, whereas red ones are garbage collection slices.

Let's have a look at a look at a more interesting program, namely Andrew Tolmach and Thomas Nordin's constraint solver from the GC portion of the nofib benchmark suite. We compare executing the program with two different heap-size settings, namely with a 10MB versus a 100MB heap (the program was again compiled with GHC 6.13 and -O, and we used an input value of 10). With Instruments, we can directly compare the 10MB and the 100MB run.

Remember that the x-axis of the track pane is wall clock time — i.e., the constraint solver runs approximately twice as fast with a 100MB than with a 10MB heap. The longer 10MB run shows evenly spaced major garbage collections after an initial startup phase (the red peaks in the second graph from the top). As GHC by default uses a generational garbage collector, we see many small red events in between the major collections — these correspond to the much less time consuming minor collections. Except in the start up phase, the mutator graph has only very short peaks, and overall garbage collection accounts for more than half of the overall runtime.

The 100MB run presents a dramatically different picture. Here, we only have three time consuming garbage collection events during the whole program execution. Other than that, there are only (barely visible in the screenshot) minor collections that briefly interrupt the mutator. Overall, the program only spends 14% of its runtime on garbage collection. The graph of the 100MB run clearly illustrates that we graph events (as peaks) and the gaps between events are event free. The height of a peak is the virtual duration of the corresponding mutator or garbage-collection time-slice and it will only correspond to the gap between events if the process was continuously running. In addition to graphing peaks, Instruments supports block diagrams, too — here, the same two program runs using a block diagram.

The height of a block still corresponds to a virtual time slice and its width is wall clock time. In the same manner as we overlayed the peak diagrams of the mutator and garbage-collection time-slices in the nfib example, we can do so with block diagrams.

For simple diagrams, this may seem to be the most easily understood graph. However, its interpretation does require some care. The human mind likes to assign meaning to the area of the blocks in a block diagram, but that leads to the wrong intuition here. In the above figure, the 10MB run appears to be entirely dominated by the red garbage collection blocks. Although garbage collection accounts for a hefty 65% of the execution time, the diagram suggests that the mutator makes nearly no progress, which is an incorrect conclusion. To avoid such misinterpretation, peak diagrams are the default.

You may like to argue that Instruments should arrange for block diagrams to use the area of blocks in such a manner that it is consistent with an intuitive interpretation. However, this runs contrary to using wall clock time for the x-axsis and the event value for the y-axsis (that is the height of a peak or block), which in turn is important to understand overall system behaviour and to correlate multiple processes. Let's look at a simple example, where wall clock time matters. The following is a profile of GHC itself when compiling a small Haskell program — in fact, it is compiling the previous constraint-solver benchmark.

We notice two points: firstly, GHC spends quite a bit of time on garbage collection —45% of its runtime, to be precise— and secondly, there is a big gap in the end, where nothing appears to happen. Indeed, we can verify that the process executing GHC is not active by adding another instrument that profiles process activity (the Instruments application comes with a whole library of pre-defined instruments).

GHC invokes the system assembler and linker to produce an executable. This obviously accounts for a large fraction of the overall execution time.

If you like to try this for yourself, you need to build the current GHC HEAD (6.13) and get this trace template for Instruments: GHC GC single. This trace template is designed for single-threaded programs — i.e., programs linked against the vanilla GHC runtime without the -threaded link-time option. DTrace support for GHC is currently only implemented for Mac OS X. However, it shouldn't be hard to port to another DTrace implementation (contact me if you need advise). I don't know whether graphical DTrace frontends are available on other platforms. To profile GHC's garbage collection and multi-threading behaviour, you can also use ThreadScope (which I found difficult to build on Mac OS).

Permalink | Leave a comment »

In Search of the Linux Laptop.

2010-01-07T11:14:00+00:00

I'm shopping for a laptop to run Linux on and I'm finding it a really frustrating process. I would like a high end, small form factor machine like the Dell Studio XPS 13. I've seen one of these machines in the real life and held it in my hands. It seems to be excellent quality and have the right features at a reasonable price. Unfortunately, on the Dell Australia web site, there is no option to purchase this machine with Linux pre-installed.

My current laptop is a Dell Latitude X1, and my wife has a Dell too. At work I have a Dell Precision workstation. All of these machine run Linux 100% of the time and for every single one of these machines I had to pay for a Microsoft Windows license that was never used. As most people know, Microsoft uses a small portion of its revenue to fund attacks on Linux and Free/Open Source Software like the SCO debacle and the ramming of OOXML through the ISO standards process.

Considering Microsoft's malignant presence in the computer industry and the fact that I don't use their products makes me reluctant to buy a machine with Windows pre-installed. I want to buy a laptop where I get my operating system of choice and just as importantly, I know Microsoft doesn't get any part of the money I pay.

Looking around for laptops with Linux pre-installed in Australia I found VG Computing who have a range of laptops which can be shipped with Linux. Unfortunately, their order page says that they get the machines with Windows which they remove to install Linux. I do realise that as a small vendor, there is not much that VG can do about this, but as far as I am concerned, thats a fail.

Another company thats been around for ages is Pioneer Computers but their machines always seemed a bit old, a bit under powered and a bit over priced. Looking at that site just recently I found their DreamBook Light M73 which was close to what I was looking for. Ordering machines from Pioneer with Windows XP costs $89 more than the same machine with Ubuntu. Purchased with Linux, this is genuinely a Microsoft free machine.

Taking a trip out to Pioneer in Alexandria I was able to see one up close and I must say I was disappointed. Compared to the Dell XPS 13, the DreamBook felt flimsy and poorly constructed. On top of that the DreamBook had a smaller keyboard (much like the Compaq M300 I had years ago) than than the Dell and SiS graphics whereas I was really hoping for Nvidia. Another machine crossed off the list.

In the US, there are a number vendors that sell laptops with Linux. In late November 2009 I contacted two of them, System 76 and ZaReason Inc. System 76 wasn't willing to ship to Australia for warranty reasons. ZaReason will ship to Australia, but to take advantage of any warranty work, I'd have to ship the machine to and from the US which would be rather inconvenient.

Despite the concerns over warranty, by early December 2009 I had come to the decision to purchase a ZaReason Alto 3550 with a bunch of extras like faster CPU, more RAM and a bigger disk. Unfortunately, while trying to purchase it online I ran afoul of their payment system and by the time we'd figured that out they had run out of stock.

I contacted them and asked about the ETA for new stock. They said, Dec 19th, which came and went with no new stock on the website. I contacted them again and was told another week. A week passed with no new stock so I contacted them again when I was informed that their distributor doesn't have any more of the Alto 3550 available. I would go for the Alto 2550 but that has an Intel graphics chip whereas I was hoping for an Nvidia like in the Alto 3550.

So now I have to decide, do I go with the ZaReason Alto 2550 with the Intel chip and then worry about warranty issues or do I buy the Dell locally, where I know that the support and service will be excellent and then try to get a refund for the Windows license I don't need or want.

Debugging Memory Leaks in a GTK+ House of Cards.

2010-01-03T00:38:00+00:00

Recently I've been hacking on Conrad Parker's sound editing, audio mangling and DJing tool Sweep. As part of my bug fixing and clean up work I ran Sweep under the Linux world's favourite memory debugging tool Valgrind. Even after running valgrind with the officially sanctioned environment variables and gtk suppressions file, the resulting 500k gzipped output file was a little shocking.

Now I'm pretty sure a number of those leaks are in Sweep, but a significant number of them seem to be in GTK+ and Glib. Since trying to differentiate the leaks in Sweep from the leaks in GTK+ was proving to be a very difficult and frustrating task I decided to look at the behaviour of a simple GTK+ program under Valgrind. The program I chose was the helloworld example from the GTK+ tutorial.

Compiling that on Ubuntu 9.10 and running it under valgrind using the following commands:


  export G_SLICE=always-malloc
  export G_DEBUG=gc-friendly
  valgrind --tool=memcheck --leak-check=full --leak-resolution=high \
    --num-callers=50 --show-reachable=yes --suppressions=gtk.suppression \
    helloworld > helloworld-vg.txt 2>&1

resulted in a memcheck summary of:


  ==22566== LEAK SUMMARY:
  ==22566==    definitely lost: 1,449 bytes in 8 blocks
  ==22566==    indirectly lost: 3,716 bytes in 189 blocks
  ==22566==      possibly lost: 4,428 bytes in 107 blocks
  ==22566==    still reachable: 380,505 bytes in 7,898 blocks
  ==22566==         suppressed: 35,873 bytes in 182 blocks

The full memcheck report is available here.

The simplest GTK+ hello world program is 100 lines of code and results in a leak report of over 8000 leaked blocks even when using the recommended valgrind suppressions file and GTK+ debugging environment variables. If someone modifies that code and adds another leak, trying to find that leak needle in the GTK+ leak haystack is going to be a needlessly difficult task.

Researching this some more I find that GTK+ is known to do a large number of allocations that are done once per program run and are never released. Furthermore the GTK+ developers seem to think this is ok and from the point of view of a user running a GTK+ program this is true. However for developers coding against GTK+ and hoping to use Valgrind to find leaks in their own code, this is a royal PITA. Leaks in the developer's code can easily be swamped and hidden by GTK+ leaks. My guess is that most people don't even bother checking unless their program's memory footprint grows over time for no good reason.

Obviously, I'm not the first to realise how hard it is too debug memory leaks in a program when the library it links against throws up so many warnings. In fact, back in 2001 a bug was raised in the GTK+ bug tracker requesting the addition of a call to be used only during debugging that would release all memory so that client programs are easier to debug. That bug has remained open and without action for over 8 years.

As far as I am concerned, this is completely unacceptable. ~~If this was my code, I would be too ashamed to put my name on it.~~ Edit: Being able to valgrind GTK+ client code is worth the effort and cost of changing the otherwise perfectly reasonable behaviour of not accounting for lifetime-of-the-app data structures (thanks Andrew).

Note: Anyone who wishes to comment on this can do so on reddit.

Using DTrace to track scheduler events of GHC's runtime

2009-12-13T06:11:27+00:00

I finally got around to committing my instrumentation of the GHC runtime system with DTrace probes (commit message). The current set of probes tracks all the events of the eventlog framework used by ThreadScope. A detailed description of the probes is on the new Using DTrace with GHC page of the GHC developer wiki.

Permalink | Leave a comment »

FP-Syd #21.

2009-12-05T05:19:00+00:00

Two weeks ago, on November 19th we held the 21st meeting of FP-Syd, the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had 22 people show up to hear our two presenters.

First up we had repeat offender Mark Wotton give us a presentation titled "Hubris : A Trojan Horse for Haskell" on his project called Hubris, a bridge between Ruby (ie Rails) and Haskell. The specific use case Mark has in mind for this bridge is call fast compiled Haskell code from Ruby web code. The code this is is available at:


  git clone git://github.com/mwotton/HaskellHubris.git

but requires the 6.12 release candidate of the GHC Haskell compiler.

Our second presenter for the evening was Tony Sloane who gave us an introduction to "Embedding a Rewriting DSL in Scala". Tony's DSL of interest is the Stratego Language which he is embedding in Scala using the Kiama library. Tony's particular interest seems to be in program transformation; desugaring of high level languages, evaluation by reduction rules, optimisation and source to target translation.

A big thanks to both our speakers and to Dan, Mark and Google for providing the meeting venue and the light refreshments.

FP-Syd will be having brief hiatus over the Xmas/New Year period and our next meeting will be in February.

Goanna 1.1 release

2009-11-26T03:52:44+00:00

Goanna for Visual Studio 1.1 has been released. Download it now. Changes include:

Fixed a constructor initialization false positive.

Fixed several unused variable false positives related to complex types in C++.

Include paths can now end in a backslash.

Accelerator keys: Alt+F1 (run Goanna on the Solution) and Alt+F2 (Run Goanna on the active project).

Several new checks, including:

Comparison never holds

Comparison always holds

Switch case is unreachable

Expanded the interval analysis.

Checks are now organized by category in the settings dialog.

Underlining (”Squiggles”) of warning-relevant code in the Visual Studio text editor.

Statistics page for monitoring Goanna’s progress.

Analysis of assert() statements for variable bounds.

Improved traces.

Much more internal work has been done, laying the groundwork for inter-procedural analysis and user-defined checks. Visual Studio 2010 support is well underway.

Haskell 2010

2009-11-25T01:30:02+00:00

Four years ago the Haskell' Committee was assembled, charged with the task to produce a new revision of the Haskell language standard (aka Haskell 98). There was a lot of enthusiasm and many wild ideas in the beginning, but it turned out that it was hard to agree on the scope of changes and that it was very hard to find a person willing to shoulder the rather large editorial burden that a major revision of Haskell entails.

The Committee finally decided that our only realistic chance of making progress was to break the mammoth task into smaller chunks. More concretely, we decided to evolve the language in small increments with a release every year, while rotating the committee on an annual basis to avoid burn out.[1] Whether this fine-grained evolutionary approach to language design will be successful in the long run remains to be seen. Nevertheless, the new approach helped us to make progress and to agree on a revision for this year: behold Haskell 2010!

[1] Details about The Haskell Prime Process

Permalink | Leave a comment »

DDC : Man or Boy?

2009-11-17T11:03:00+00:00

Computer scientist Donald Knuth came up with something he called the Man or Boy Test as a way of evaluating implementations of the ALGOL60 language (standardized in 1963) to distinguish compilers that correctly implemented "recursion and non-local references" from those that did not. Knuth said:

"I have written the following simple routine, which may separate the 'man-compilers' from the 'boy-compilers'."

My first attempt at solving this problem in Disciple resulted in me raising bug #148 in the DDC bug tracker with the following code:


  -- Compiler needs a little help inferring the types.
  a :: Int -> a -> a -> a -> a -> a -> Int
  a k x1 x2 x3 x4 x5
   = do    b () = do { k := k - 1 ; a k b x1 x2 x3 x4 }
           if k = 0 then x4 () + x5 () else b ()

  fn n = \() -> n

  main ()  -- Function 'a' should return -67
   = do    out = a 10 (fn 1) (fn -1) (fn -1) (fn 1) (fn 0)
           if out /= -67
               then println $ "Output was " % show out % ". Should have been -67."
               else println "Passed!"

Fiddling around with the problem a bit, I suddenly realised that the Disciple language has call-by-reference semantics by default (by way of contrast, the C programming language has default call-by-value semantics with optional call-by-reference semantics using pointers).

While chatting with Ben on IRC he suggested using a copy to create a local copy of the function parameter that gets mutated so that mutation doesn't change the value outside call frame.

Here are two correct solutions to the Man or Boy problem:


  a0 :: Int -> a -> a -> a -> a -> a -> Int
  a0 k x1 x2 x3 x4 x5
   = do   b () = do { k := k - 1 ; a0 (copy k) b x1 x2 x3 x4 }
          if k = 0 then x4 () + x5 () else b ()


  a1 :: Int -> a -> a -> a -> a -> a -> Int
  a1 k x1 x2 x3 x4 x5
   = do   m = copy k
          b () = do { m := m - 1 ; a1 m b x1 x2 x3 x4 }
          if k = 0 then x4 () + x5 () else b ()

  fn n = \() -> n

  main ()
   = do   out0 = a0 10 (fn 1) (fn -1) (fn -1) (fn 1) (fn 0)
          out1 = a1 10 (fn 1) (fn -1) (fn -1) (fn 1) (fn 0)

          println "All outputs below should be equal to -67."
          println $ "Output 0 : " % show out0
          println $ "Output 1 : " % show out1

Both of these Disciple solutions are significantly less complex than the equivalent Haskell solution.

While I have no problem with function parameters being passed by reference, I don't think its a good idea to have those parameters being mutable by default (ie with the values also changing in the calling function).

I need to play with this some more.

Hacking DDC.

2009-11-15T10:28:00+00:00

Over the last couple of months I've been doing a bit of hacking on an experimental compiler called DDC. This has been some of the most interesting, gratifying and challenging hacking I have done in years. Having this much fun should probably be illegal!!

I was introduced to DDC at the April 2008 meeting of FP-Syd when Ben Lippmeier, its author, gave a presentation titled "The Disciplined Disciple Compiler". The two main reasons this compiler is interesting are:

Its written in Haskell an advanced purely functional programming language.
The language it compiles (Disciple) has some interesting solutions to the problems of side effects and mutability.

The Disciple language is very Haskell-like but has some extra features in the type system which allows the compiler to track mutability and side effects in the type system. The important differences between the Disciple language and the Haskell language are listed on the DDC web page as:

Strict Evaluation Order is the default, laziness is introduced explicitly.
Type directed Field Projections complement type classing.
All data objects support Destructive Update.
The Effect System tracks what computational effects are being used in a program, without the need for state monads.
The Class System ensures that effects and destructive update play nicely with laziness.
Closure Typing is used to track data sharing, and to preserve soundness in the presence of Polymorphic Update.

Obviously a compiler that is doing all this really clever stuff has to be pretty complicated, but it still only weighs in at about 50k lines of code.

The main challenge in working on this is that i am not a very experienced Haskell programmer. There are also large chunks of the compiler doing some very complicated stuff that I don't even have a hope of understanding without reading and understanding Ben's PhD thesis.

Despite that, Ben was willing to give me commit access to the Darcs repo and I have been able to significantly reduce the number of bugs in the DDC bugtracker. Since I was already pretty familiar with the concepts of lexing and parsing as well as being familiar with Parsec (probably the most widely used parsing tool in the Haskell community) I started off fixing some simple lexer and parser bugs like:

I then managed to hack in support for Int64 and Float64 (#106) followed by some significant re-factoring of the Parsec parser which reduced the usage of the Parsec.try construct allowing Parsec to produce much better error messages.

Once I'd done all that, I ran into a very busy time at work and didn't mess with DDC for a couple of months. When I finally got back to looking at DDC, I realised that nearly all of the remaining bugs were much deeper than the bugs I had tackled so far. Tackling these deeper bugs required a new strategy as follows:

Scan the bug list for reports that either had test cases already or give enough information for me to proceed.
Create a new darcs branch for each bug. This allowed me to work on multiple different bugs at once so that if I got stuck on any one specific bug, I could just leave it and move on to another.
Create a reproducible test case if one didn't exist already.
Create a shell script in the root directory of each branch which did make and then ran the specific test case for this specific bug.
Use Haskell's Debug.Trace module in conjunction with Haskell's very wonderful Show Type Class to add debug statements to the code.
Use the Wolf Fencing debugging technique to narrow down the problem to specific area of the code.

Once the problem had been narrowed down to a piece of code, all that remained was to develop a fix. In many cases this resulted in me asking Ben how he'd like it fixed, either in email or on IRC. I also often came up with an ugly fix at first which was refined and cleaned up before being applied and pushed upstream.

With the above methodology I was able to fix a number of deeper and more complex bugs like the following:

I'm now getting a pretty good idea of how the compiler is put together and I'm stretching my hacking into feature enhancements.

My enthusiasm for DDC was recently validated by functional programming guru Oleg Kiselyov's comment on the haskell-cafe mailing list:

"One may view ML and Haskell as occupying two ends of the extreme. ML assumes any computation to be effectful and every function to have side effects. Therefore, an ML compiler cannot do optimizations like reordering (even apply commutative laws where exists), unless it can examine the source code and prove that computations to reorder are effect-free. .....

Haskell, on the other hand, assumes every expression pure. Lots algebraic properties become available and can be exploited, by compilers and people. ....

Hopefully a system like DDC will find the middle ground."

Anyway, back to hacking ....

Home

2009-11-15T07:04:56+00:00

Hikari and Chiz came home from hospital today. It was nice and uneventful and everyone seems relaxed now.

To celebrate I've expanded Hikari's photo page, its here.

Hikari Horman

2009-11-12T06:45:03+00:00

Born at 1:47 pm on the 11th November 2009 at the Royal Hospital for Women in Randwick. 2.72kg healthy boy.

Photos here

Status Update of the Glasgow Haskell Compiler, October 2009

2009-11-04T04:58:00+00:00

If you are interested in Haskell generally or the Glasgow Haskell Compiler (GHC) in particular, you may want to have a look at the latest issue of the biannual GHC status updates. It includes a summary of our recent progress in the Data Parallel Haskell project.

Permalink | Leave a comment »

First benchmark of regular, multidimensional arrays in DPH

2009-11-04T03:47:55+00:00

Download now or preview on posterous

delayed-regular-arrays-Sep09.pdf (417 KB)

Permalink | Leave a comment »

Finally found the ghci bug on Snow Leopard

2009-10-30T04:35:05+00:00

After much digging through otool output and tracing Haskell code with dtrace, log messages, and gdb, I finally found the cause of the SIGBUS errors with ghci on Snow Leopard. At the end, only a tiny change was required[1]: the homegrown dynamic linker of ghci had simply ignored a relocation type that ld on Snow Leopard chose to use.

What this really shows again is that re-implementing parts of basic infrastructure in an ad hoc manner always comes back to bite you. Such infrastructure is usually full of low-level, platform-dependent details, which you are bound to get wrong once in a while and then it can be rather involved to find the problem. In this case, the relocation of the memory access of a bit of C code in GMP, reading a global variable, went wrong. As a result, the wrong memory location was accessed, which happened to contain a NULL pointer. Upon dereferencing the NULL, the program crashed.

All of this was further obscured by the fact that the homegrown linker approach of ghci implies that we have two copies of the basic libraries in memory and in use at the same time. One copy has been statically linked with the ghc executable and the second copy is dynamically loaded by ghci. The relocation in the statically linked copy was perfectly fine, but the relocation in the dynamically loaded one was broken.

[1] http://www.haskell.org/pipermail/cvs-ghc/2009-October/050863.html

Permalink | Leave a comment »

Data-oriented programming and the vectorisation transformation

2009-10-27T00:55:02+00:00

I just realised that the data layout and the resulting code organisation in data-oriented programming, as proposed for games design, in many aspects resembles the data layout and code organisation favoured by the vectorisation transformation that is at the core of Data Parallel Haskell. This should not be surprising as both have similar goals, namely to maximise data throughput, to minimise stalls by utilising the memory hierarchy, and to maximise parallelism.

However, the parallels are interesting given the very different origins of both approaches. The big difference is of course that data-oriented programming is a design methodology for programmers, whereas the vectorisation transformation is a program transformation automatically applied by a compiler. Nevertheless, we may regard the vectorisation transformation as a program transformation that turns purely functional code into code that is structured in a data-oriented manner, where the layout of bulk data shapes the organisation of the code.

This raises the question of how useful nested data parallel programming and the vectorisation transformation may be to games programming.

Permalink | Leave a comment »

FP-Syd #20.

2009-10-25T06:49:00+00:00

On October 22th we held the 20th meeting of FP-Syd, the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had about 28 people show up to hear our two presenters.

First up we had Roman Leshchinskiy and his presentation "Loop Fusion in Haskell", work that is part of GHC's Data Parallel Haskell library. Loop fusion depends on the ability to convert operations on arrays into operations on streams. Then, when applying multiple stream operations, adjacent conversions to and from streams can be dropped, allowing further inlining. The real beauty of this approach is that stream operations and data parallelism can be written as a library outside of the GHC compiler and then depend on the compiler to do most of the heavy lifting. Romain then moved on to explain that this work was just as applicable to general parallel computation on multicore systems as it was to clusters and GPUs.

Our second presenter for the evening was Barry Jay who gave us an introduction to "Pattern Calculus". Barry's work on Pattern Calculus was inspired by the fact that while lambda calculus is able to adequately explain computation, it does not explain operations on data well. In particular, lambda calculus does not distinguish between variables and data constructors while in the pattern calculus constructors are treated as a separate class; matchable symbols. The ideas behind Pattern Calculus are explained more fully in Barry's book:

A big thanks to both our speakers and to Shane Stephens and Google for providing the meeting venue and the light refreshments.

Fun with higher-order functions in C and Objective-C

2009-10-20T12:48:27+00:00

The latest release of Mac OS X (Snow Leopard) came with an upgrade to the C and Objective-C languages, adding lambda abstractions — which they call blocks. Matt Gallagher has a nice blog post that describes some of the trickier aspects of blocks: http://cocoawithlove.com/2009/10/ugly-side-of-blocks-explicit.html

Interesting is the function 'newDoubleToIntComparison' (in Section "Declaring a block that returns a block"), which is a curried[1] version of 'compareDoubleToInt'. Together with the explicit memory management for returned blocks and the need for involved type annotations and casts, it becomes clear why functional languages like Haskell make curried functions the default as well as support garbage collection and type inference out of the box.

[1] http://en.wikipedia.org/wiki/Currying

Permalink | Leave a comment »

It’s always the cover up which gets you.

2009-10-20T11:31:55+00:00

I suppose Watergate is the archetypal example of the consequences of the cover up being worse than the original offence, but you see it all the time.

The example I have in mind has less geopolitical significance, but will probably be more interesting for most readers.

A company named PopCap Games publish a nice little game called ‘Plants vs. Zombies’. I played the demo for the allotted 60 minutes when it was originally released, and then forgot about it when my trial expired.

About a week ago they released a free Flash version, which uses a subset of the plants and zombies from the full game, and playing that got me interested in buying the full version again.

The game sells for USD20 or AUD30, which at current exchange rates is a bit rich — the Australian dollar is buying 92 US cents at the moment. Just yesterday I bought another game, Machinarium — which also sells for USD20 — for AUD22.32.

So I sent a tweet to PopCap enquiring about their pricing:

@popcap_games why is PvZ AUD30, when USD20 is only AUD22?

To their credit, they replied quickly, but I didn’t find their answer convincing:

@tgdavies USD price displayed is pre-tax which is added in the cart. International prices include sales tax.

To get 22 to 30 you need more than 30% tax — Australia’s GST is only 10%, and I think paying that is my responsibility as the person ‘importing’ the game.

Of course, the real reason is that as recently as March the Aussie dollar was only buying 65 US cents — which gives you about AUD30 for USD20. PopCap simply haven’t adjusted their prices.

I wish they could have said that, or just “I don’t know”, instead of invoking some mythical tax!

PopCap aren’t the only company whose Australian prices are out of whack — the cheapest 15″ MacBook Pro is AUD600 more expensive than a simple exchange rate + tax calculation would suggest.

The Stupidity of Fat Elf (was OSX Universal Binaries).

2009-10-20T09:53:00+00:00

Early in 2008 I drafted a blog post about something on Mac OS X called Universal binaries. Now the same stupid idea is available for Linux, and planned for other ELF systems. Its called Fat Elf.

The original post follows.

As the main author and maintainer of libsndfile I've recently been involved in a number of email exchanges along the lines of this:


  OS X User : Is it possible to build a Universal Binary of libsndfile?
  Me        : No, please see the FAQ.
  OS X User : Autoconf sucks!
  Me        : Even if you use Xcode, the problem of testing remains.
  OS X User : Other projects can build Universal Binaries, why can't yours?

to which I should probably respond "Have you stopped beating your wife/ girlfriend yet?".

For those lucky enough to be blissfully unaware of the issue, Universal Binaries are Apple's solution to the marketing problem that arose from their decision to ditch the PowerPC processors it had been using in its machines in favour of CPUs from Intel. Obviously, this change of CPU would have an impact on users and Apple chose a two pronged approach to ease the migration of its users from one CPU to another; Rosetta a PowerPC emulator which allows PowerPC code to run on Intel based Macs, and the idea of the Universal Binary (UB), a way of packaging binary code for PowerPC and Intel into a single monolithic file so that the OSX operating system can run which ever is appropriate for the CPU it is being run on.

Building Universal Binaries on Mac OS X using Apple's own development toolset Xcode is apparently quite trivial. Under the hood, Xcode uses a version of the GNU GCC compiler which Apple hacked to allow it to take in a single input source code file, pass it through both an Intel and a PowerPC back-end and then generate a single binary containing both Intel and PowerPC executable code.

Apple has obviously gone to a lot of trouble to make the process easy. So easy, that the average OSX developer expects it to just work, without realising exactly what it is they are doing and even more importantly, without realising that the process itself contains one very large flaw (which I'll get to shortly).

Building Universal Binaries of FLOSS (Free/Libre/Open Source ...) software is a whole different matter. A large majority of FLOSS software uses things like scons or autoconf and its related tools to detect characteristics of the target platform so that the source code can be compiled correctly for the target.

The problem is, that people who know that the Xcode version of GCC can generate a universal binary from one pass over a source code file expect to take a project like libsndfile, set the CFLAGS environment variable, run configure and build a working universal binary. Anyone who knows autoconf can guess that this is simply not going to work. The main output of the configure script generated by autoconf is a header file containing definitions to handle the detected features of the CPU and operating system. The problem is that the configure script is only run once and the characteristics detected on that run are then used to compile both the Intel and the PowerPC version of the executable code.

The result of the above is that people developing FLOSS like myself get emails from OSX users complaining that their UB version of libsndfile doesn't work. Since I first heard of Apple's Universal Binaries back in early 2006 I have personally spent over 50-100 hours trying to come up with a solution to an issue that only affects an operating system that I personally am not very interested in. Thats a bunch of hours that I could have spent in much more enjoyable pursuits.

Now, consider how many other FLOSS project would have been hit by similar problems. So lets say 10 hours per project across a thousand projects. Thats one hell of a lot of developer time that could have been spent doing more useful things. Why didn't Apple foresee this? Why didn't Apple spend some of its resources trying to mitigate the effect of UB on FLOSS projects. Why didn't Apple devote say 500 man hours of its developer's time to help ease the transition of FLOSS projects to UB; 500 man hours of work on autoconf and automake would likely have gone a long way.

The difficulty of generating UB for code using the autotools is not even the biggest problem with UB. What I see as the biggest problem is the huge green elephant standing in the corner that no one wants to talk about; testing (maybe that should have a blink tag).

For nearly a decade, testing, especially reliable, repeatable, automated testing has been considered an important part of the process of creating reliable software. Now Apple is encouraging the building of Universal Binaries, but I for one am very curious as to how these are being tested. I can think of the following possibilities:

Ad-hoc and manual testing. Obviously this is a poor substitute for automated testing.
Automated testing relying on copying test binaries to another machine and running the tests for the other architecture there. This requires easy access to the other machine and considerable effort to make the tests work.
Automated testing relying on Rosetta. Like the previous option this requires considerable effort to make the tests work.

Given that all these options are difficult it is highly likely that testing falls by the wayside. In short Apple is effectively encouraging the release of untested binaries to users. When these untested binaries go wrong, who do users blame? The authors of that software even when those FLOSS authors are not Mac users and have little or no interest in Apple and its operating system.

Furthermore, Apple is about to add another spanner to the works; quad binaries. Quad binaries are like Universal Binaries that contain code for both 32 bit and 64 bit versions of PowerPC and Intel. Quad binaries will be even more difficult to test, more difficult to get right and still more pain for FLOSS developers.

FP-Syd #19.

2009-10-19T08:46:00+00:00

On September 17th we held the 19th meeting of FP-Syd, the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had about 18 people show up to hear our two presenters.

First up we had Raphael Speyer and his presentation "Rhino, ES5 & GSoC". Raphael explained how he was accepted into the Google Summer of Code project to work on bring the Rhino, a Javascript engine that runs on the Java Virtual machine, into line with the upcoming EcmaScript 5 standard.

For the second presentation we had Shane Stephens one of the Google people working on Google Wave giving us an overview of Google's Operational Transform algorithm which is at the core Google Wave. He then walked us through a reimplementation of the algorithm in Haskell.

A big thanks to Shane Stephens and Google for providing the meeting venue and the light refreshments.

Multicore Haskell Now!

2009-10-18T06:16:44+00:00

Don Stewart summarised the state of play of parallel programming in Haskell at "ACM Reflections | Projections 2009". He covers strategies, Concurrent Haskell, STM, and Data Parallel Haskell: http://donsbot.wordpress.com/2009/10/17/multicore-haskell-now-acm-reflections-projections-2009/

Permalink | Leave a comment »