Using the ZeptoProg II on OS X

I’ve decided that I should be using xmega AVR microcontrollers rather than the older series — the timer architecture is much nicer for one thing. The only downside is that they are 3.3V only — and have a different programming interface.

I bought the ZeptoProg II and a simple header board from Justin Mattair at mattairtech.com.

The programmer isn’t explicitly supported on OS X, but in fact it works fine once you set it up correctly.

I use avrdude for programming AVRs, and when I first ran it with the ZeptoProg II I got:

Blonder:avrtest tomd$ avrdude -c avrisp2 -p x128a4 -P usb:000200012345 -U flash:r:flash.hex:h -v -v -v -v

avrdude: Version 5.11.1, compiled on Feb 16 2012 at 22:11:20
    Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
    Copyright (c) 2007-2009 Joerg Wunsch

    System wide configuration file is "/usr/local/CrossPack-AVR-20120217/etc/avrdude.conf"
    User configuration file is "/Users/tomd/.avrduderc"

    Using Port                    : usb:000200012345
    Using Programmer              : avrisp2
avrdude: usbdev_open(): Found AVRISP mkII, serno: 000200012345
avrdude: usbdev_open(): using read endpoint 0x82
avrdude: Sent: . [01] 
avrdude: usbdev_recv_frame(): usb_bulk_read(): usb_bulk_read: An error occured during read (see messages above)
avrdude: stk500v2_recv_mk2: error in USB receive
avrdude: usbdev_send(): wrote -104 out of 1 bytes, err = usb_bulk_write: An error occured during write (see messages above)
avrdude: stk500_send_mk2(): failed to send command to serial port

I contacted Justin, who explained that this meant that the ZeptoProg was in AVR Studio mode, not avrdude mode. There’s a Java application which allows you to change this (it also allows you to use the ZeptoProg as a logic analyzer and signal generator).

There was one gotcha — the jar containing this application bundles the librxtxSerial.jnilib library which it needs to talk to the USB port, but only a 32 bit version. Java on my OS X version is 64 bit only, so I got messages like:

Blonder:xmega tomd$ java -jar ~/Downloads/ZeptoProg_II.jar 
java.lang.UnsatisfiedLinkError: /private/var/folders/tv/58qc8nq53x50q8f7r31z_6280000gn/T/natives3881310/librxtxSerial.jnilib: dlopen(/private/var/folders/tv/58qc8nq53x50q8f7r31z_6280000gn/T/natives3881310/librxtxSerial.jnilib, 1): no suitable image found.  Did find:
/private/var/folders/tv/58qc8nq53x50q8f7r31z_6280000gn/T/natives3881310/librxtxSerial.jnilib: no matching architecture in universal wrapper thrown while loading gnu.io.RXTXCommDriver
Exception in thread "main" java.lang.UnsatisfiedLinkError: /private/var/folders/tv/58qc8nq53x50q8f7r31z_6280000gn/T/natives3881310/librxtxSerial.jnilib: dlopen(/private/var/folders/tv/58qc8nq53x50q8f7r31z_6280000gn/T/natives3881310/librxtxSerial.jnilib, 1): no suitable image found.  Did find:
/private/var/folders/tv/58qc8nq53x50q8f7r31z_6280000gn/T/natives3881310/librxtxSerial.jnilib: no matching architecture in universal wrapper
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1939)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1868)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
at java.lang.Runtime.loadLibrary0(Runtime.java:845)
at java.lang.System.loadLibrary(System.java:1084)
at gnu.io.CommPortIdentifier.<clinit>(CommPortIdentifier.java:83)
at GUITerminal.getAvailableSerialPortNames(GUITerminal.java:234)
at GUITerminal.<init>(GUITerminal.java:134)
at ZeptoProgGUI.<init>(ZeptoProgGUI.java:56)
at ZeptoProg_II.main(ZeptoProg_II.java:25)

The strange /private/var/folders/tv/58qc8nq53x50q8f7r31z_6280000gn/T... path in the message above is where the jnilib from the jar is unpacked.

To fix this I needed to:

  • download this file http://code.google.com/p/create-lab-commons/source/browse/trunk/java/lib/rxtx/librxtxSerial64.jnilib (this contains 64 bit and 32 bit architectures)
  • rename it to librxtxSerial.jnilib (possible not necessary)
  • replace the file in the jar: jar uf ZeptoProg_II.jar librxtxSerial.jnilib
  • then it runs fine with java -jar ZeptoProg_II.jar
  • Put the board into multitool mode, then a file /dev/tty.usbmodem513281 will appear (not sure if the number is always the same) and you can refresh the COM port list, select it and connect. When the board is not in tool mode that device is not present in /dev.

Once set up like that the ZeptoProg works with avrdude.

More SMH Bashing

I read an interesting article in the SMH a few days ago about a technique for preparing children to learn to read by increasing their ‘phonemic awareness’.

The first thing I think when I read about any innovation is “does it really work?” — the evidence was addressed by a single paragraph:

After 13 weeks of the program, 5 per cent of children improved in language tests administered by the foundation workers and the teachers. The number of three-year-olds who passed the tests after the program doubled.

Which tells us very little. Is that 5 per cent figure net of the children who did worse? Or is it gross? Why is the three year old improvement stated differently? Did the journalist just feel that it read better to avoid a similar turn of phrase, or did the educator phrase it that way for some reason? Obviously it makes a difference whether the 3 year old pass rate increased from 1% to 2% or 50% to 100%. Most importantly — as 13 weeks is a long time in a child’s life, especially a three year old — you would expect improvements no matter what. A trial with a control group, and blind administration of the tests would make the figures much more convincing.

No doubt there is better evidence available, as “other preschools in the area are clamouring for the program”, so it’s a shame that the journalist couldn’t have presented that evidence in their article.

It’s always the cover up which gets you.

I suppose Watergate is the archetypal example of the consequences of the cover up being worse than the original offence, but you see it all the time.

The example I have in mind has less geopolitical significance, but will probably be more interesting for most readers.

A company named PopCap Games publish a nice little game called ‘Plants vs. Zombies’. I played the demo for the allotted 60 minutes when it was originally released, and then forgot about it when my trial expired.

About a week ago they released a free Flash version, which uses a subset of the plants and zombies from the full game, and playing that got me interested in buying the full version again.

The game sells for USD20 or AUD30, which at current exchange rates is a bit rich — the Australian dollar is buying 92 US cents at the moment. Just yesterday I bought another game, Machinarium — which also sells for USD20 — for AUD22.32.

So I sent a tweet to PopCap enquiring about their pricing:

@popcap_games why is PvZ AUD30, when USD20 is only AUD22?

To their credit, they replied quickly, but I didn’t find their answer convincing:

@tgdavies USD price displayed is pre-tax which is added in the cart. International prices include sales tax.

To get 22 to 30 you need more than 30% tax — Australia’s GST is only 10%, and I think paying that is my responsibility as the person ‘importing’ the game.

Of course, the real reason is that as recently as March the Aussie dollar was only buying 65 US cents — which gives you about AUD30 for USD20. PopCap simply haven’t adjusted their prices.

I wish they could have said that, or just “I don’t know”, instead of invoking some mythical tax!

PopCap aren’t the only company whose Australian prices are out of whack — the cheapest 15″ MacBook Pro is AUD600 more expensive than a simple exchange rate + tax calculation would suggest.

The SMH gets more numbers wrong

From the lifestyle section:

USING a solarium increases a person’s risk of developing melanoma by 15 per cent and the risk rises to 75 per cent for those who used them when they were younger than 30, a major review has found.

I suspect that the melanoma rate among under 30s sunbed users isn’t 75%, and what the Herald meant to say was “the risk rises by 75 per cent” or “the increase in the risk rises to 75 per cent”.

The mistake may be obvious, but why not take the trouble to say what you mean, instead of something completely different?

Functional Programming on Android

I’ve integrated CAL with a variety of Java based frameworks/environments: Tapestry, GWT and Google AppEngine for instance.

Recently someone asked on the CAL Google Group whether CAL programs could run on Android mobile devices. Android doesn’t use a standard JVM, it uses Dalvik, and it provides a subset of the normal JRE library.

I have put together a trivial proof of concept to show that CAL does indeed work on Android — although it doesn’t cover a great deal of CAL’s runtime behaviour, so there might be problems with larger programs which I haven’t detected.

My sample application just accumulates keypresses and displays the resulting string.

The Android Activity

This class delegates calls to onCreate and onKeyUp to CAL functions.

The state of the application (a String in this case) is initialised to the value returned by the CAL onCreate function, and is then passed to and returned from the onKeyUp function.

public class CalActivity extends Activity {
    private ExecutionContext ec;
    private TextView tv;
    private String state = null;

    public CalActivity() {
        System.setProperty("org.openquark.cal.machine.lecc.non_interruptible", "true");
        ec = StandaloneRuntime.makeExecutionContext(CalActivity.class);
    }

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        tv = new TextView(this);
        try {
            state = Test.onCreate(CAL_Opaque.make(this), CAL_Opaque.make(savedInstanceState), ec);
            tv.setText(state.toString());
        } catch (CALExecutorException e) {
            tv.setText(e.getMessage());
        }
        setContentView(tv);
    }

    @Override
    public boolean onKeyUp(int keyCode, KeyEvent event) {
        char c = (char)event.getUnicodeChar();
        // ignore events which aren't characters
        if (c != 0) {
            try {
                state = Test.onKeyUp(CAL_Opaque.make(this), c, state, ec);
            } catch (CALExecutorException e) {
                tv.setText(e.getMessage());
            }
            tv.setText(state.toString());
            return true;
        }
        return super.onKeyUp(keyCode, event);
    }
}

The CAL Module

module Org.Kablambda.Android.Test;

... imports omitted ...

onCreate :: Activity -> Bundle -> String;
public onCreate activity bundle = "Hello from CAL!\n";

onKeyUp :: Activity -> Char -> String -> String;
public onKeyUp activity ch s = s ++ (fromChar ch);

Activity and Bundle are declared as foreign types, but don’t have any functions defined on them at present.

Modifications to the build process

I added some new targets to the standard build.xml file created by the android SDK.

<property name="quark.dir" location="${user.home}/dev/tools/cal-1.7.1" />
<property name="moduleName" value="Org.Kablambda.Android.Test"/>
<property name="className" value="org.kablambda.android.Test"/>

These properties set the location of our CAL installation, the name of the CAL module we want to call from Java, and the name of the class we want that module’s functions to be exposed in.

The copy-quark-jars target copies the jar files needed by CAL at runtime. They are copied into a temporary directory.

<target name="copy-quark-jars">
    <mkdir dir="tmp"/>
    <copy
            todir="tmp"
            flatten="true">
          <fileset dir="${quark.dir}/Quark/bin/java/release">
            <include name="**/calLibraries.jar" />
            <include name="**/calRuntime.jar" />
            <include name="**/calUtilities.jar" />
          </fileset>
          <fileset dir="${quark.dir}/Quark/lib/Resources/External/java/">
            <include name="**/icu4j.jar" />
            <include name="**/log4j.jar" />
          </fileset>
        </copy>
  </target>

The build-quark-standalone target compiles the CAL workspace workspace.cws into a jar file. This workspace must contain the module specified above and its dependencies. This target assumes that ${quarkdir}/Quark is on the PATH.

  <target name="build-quark-standalone" depends="copy-quark-jars">
    <exec executable="quarkc.sh">
        <arg value="workspace.cws"/>
        <arg value="-lib"/>
        <arg value="${moduleName}"/>
        <arg value="public"/>
        <arg value="${className}"/>
        <arg value="tmp/calmodule.jar"/>
        <arg value="-src"/>
        <arg value="calmodule-src.zip"/>
        <env key="QUARK_CP" value="src:bin:${sdk-location}/platforms/android-1.5/android.jar"/>
    </exec>
  </target>

Because several of the CAL jars contain duplicate files, which Android doesn’t tolerate, we merge all our jars into a single file, ignoring duplicates.

<target name="merge-jars" depends="build-quark-standalone">
    <zip destfile="libs/cal.jar" duplicate="preserve">
        <zipgroupfileset dir="tmp" includes="*.jar"/>           
    </zip>
</target>

To deploy your application to the Android Emulator you need to run ant merge-jars reinstall.

Note that during the process of converting the cal.jar file to Dalvik bytecode you’ll see a large number of error messages like this:

[apply] ...while processing com/ibm/icu/util/TimeZoneData.class
[apply] warning: Ignoring InnerClasses attribute for an anonymous inner class that doesn't come with an associated EnclosingMethod attribute. (This class was probably produced by a broken compiler.)

This is something to do with the rather old version of IBM’s International Components for Unicode library — it would be interesting to recompile CAL with a more recent version — or just recompiling that version of icu4j with the Sun compiler.

Further Work

The Android API needs to be exposed to CAL, and a Monad written to sequence Android API operations which have side effects. The CalActivity implementation should store the state as an opaque CalValue, so that implementations can choose what type to use for their state, and of course it needs to delegate all the possible events to CAl functions. There’s plenty of scope for more scaffolding on the CAL side.

As I don’t have an Android phone I doubt that I’ll do any further work on this — unless Dalvik gets an iPhone port :-)

Innumeracy in the SMH

From an article in today’s SMH:

The State Transit Authority’s own figures show the number of people travelling without a ticket or with the incorrect ticket has shot up by more than 40 per cent in the past year.

Is this true? Unfortunately it’s contradicted by the very next paragraph:

From March 1 to May 31, 2008, officers from the STA’s Revenue Protection Unit inspected 10,151 buses, fined 2628 passengers and cautioned 1590 others. In the same period this year, they inspected 14,134 buses, fining 3780 passengers and cautioning 1857 others.

So the number of people fined has shot up, but the number of busses inspected has shot up by about the same amount, leaving the people fined per bus inspected figure unchanged at about 1 in 4.

If you add people fined to people cautioned, then the figure per bus has declined very slightly, so perhaps the number of people travelling without a ticket has fallen!

Reconsidering my Combinators

Since writing my set of function combinators for matching HTTP requests I’ve realised that there are better approaches.

While the combinators read nicely, once you start extracting attributes from the request things become messy, due to the hAnd combinator composing results as nested pairs.

Here’s an example:

skitchPutHandler =
    let
        storeImage p =
            let (p2, path) = p;
                (maybeContentType,imageData) = p2;
                ...
            in ...
    in
        matchMethod "PUT" `hThen` matchUrl "/skitch" `hThen`
            (getHeader "Content-Type") `hAnd` requireRawData `hAnd` requirePathInfo `hApply` storeImage;

Unpacking the values extracted from the request is cumbersome, and because many are Strings, error prone.

A better approach is to simply use the Maybe monad with the same extraction functions:

simpleSkitchPutHandler req =
    let
        maybeContentType = requireHeader "Content-Type" req;
        ...
    in
        matchMethod "PUT" req `anonymousBind` matchUrl "/skitch" req `anonymousBind`
            requireRawData req `bind` (\rawData ->
            requirePathInfo req `bind` (\path ->
            return $ storeImage path rawData (actualContentType path)));

Optional values are extracted in let expressions, and required values and tests are composed with the monadic bind operators in the main expression. You can make the function a bit more concise and ‘point free’ by writing a monad which composes the Maybe and Reader monads to supply the HTTP request parameter to functions implicitly:

readerMonadSkitchPutHandler =
    let
        ...
    in
        runHttpRequestM (
        method "PUT" `anonymousBind` matchUrlM "/skitch" `anonymousBind`
            rawData `bind` (\rawData ->
            pathInfo `bind` (\path ->
            (optional (header "Content-Type")) `bind` (\maybeContentType ->
            return $ storeImage path rawData (actualContentType path maybeContentType)))));

The function optional has the type HttpRequestM a -> HttpRequestM (Maybe a), that is, it allows a function to succeed by wrapping its result in a ‘Just’.

I’m not convinced that the improvement in readability gained by removing the request parameter is worth the increase in complexity.

So I’ll remove the combinators from my Http module and will rewrite Hangman using just the Maybe monad. I’ll keep the more complicated Monad on the back-burner in case I discover a compelling advantage.

I’d be interested to know what someone with a deeper understanding of functional programming would say.

CAL Hangman on GAE Part 2 — The Datastore

Introduction

My implementation of Hangman has only very simple data storage requirements, so my CAL module Datastore doesn’t cover the entire capabilities of Bigtable.

All the application needs is to be able to save a Game instance, put the key of that instance into a cookie in the user’s browser, and then retrieve and update that instance as the user makes each guess, or delete it if the user chooses to restart the game.

Important unexplored areas are:

  • The creation of entities having another entity as a parent. This is important in Bigtable, because a transaction can only operate on entities in the same ‘entity group’ — that is, entities which share a common parent.

  • The persistence of references to other entities, and special treatment of these references when performing operations on the entity containing the reference — that is, there is no support for modelling relationships between entities.

  • No error handling support — any exception will result in a 500 response. In particular, ConcurrentModificationException should be treated differently, as this indicates that a transaction failed due to another transaction modifying the same records. A failure of this type should either be retried, or reported to the user in an application specific manner.

  • The module is limited to storing types with a single constructor, although there is no difficulty in extending this to support multiple constructor ADTs.

It is implemented using the low level api, not JDO.

Datastore Module

The Datastore module provides a Monad instance for using the AppEngine data store, and a Storeable type class. The source is here.

The Storeable Type Class

Algebraic data types which need to be persisted must be instances of Storeable, which provides metadata needed to persist instances of the type. The metadata includes:

  • The ‘kind’ of the record — analogous to the name of the table to store instances of the type in, not to kind in the type theory sense.

  • The data store property names to use for each of the constructor arguments.

When using CAL in statically compiled mode the names of types and the names of constructor parameters are not easily available at runtime. The constructor arguments could simply be stored in properties named arg0...n, but this would make querying and debugging more difficult.

data public Store a = public Store kind :: String fields :: [String];

public class (Outputable a, Inputable a) => Storeable a where
    store :: a -> Store a;
    ;

Note that the type parameter a is not used in the declaration of the constructor arguments — it simply labels the store to ensure that the store used with, for example fromEntity below, is consistent with the type of Storeable we are expecting fromEntity to return. For this reason functions which construct a Store must be explicitly typed, like gameStore below, as otherwise no more specific type than Store a can be inferred.

So the instance for a game of hangman is:

data public Game = 
    private Game word :: String guesses :: String deriving Show, Inputable, Outputable;

gameStore :: Store Game;
public gameStore = Store "Org.Kablambda.AppEngine.Test.Game" ["word","guesses"];

instance Storeable Game where
    store = gameStoreA;
    ;

private gameStoreA a = gameStore;

The public gameStore function is used when we wish to query.

Converting between CAL values and AppEngine Entities

Two conversion functions are used, one in each direction. In both cases the bulk of the work is carried out in Java functions.

From a CAL value to an Entity

toEntity :: Storeable a => a -> Entity;
public toEntity !r =
    jObjectToEntity (store r).Store.kind (outputList $ (store r).Store.fields) (output r);

where:

foreign unsafe import jvm "static method org...jObjectToEntity" 
    public jObjectToEntity :: String -> JList -> JObject -> Entity;

The parameters passed to jObjectToEntity are the kind of entity to create, the list of names to use for the fields of the CAL value, and the CAL value, converted to a JObject by the output function. Storeable instances should all derive Outputable, so output r uses the default implementation of output to convert the algebraic data type instance to an instance of the CAL Java support class AlgebraicValue. The fields are transferred to the AppEngine entity like this:

Entity entity = new Entity(kind);
AlgebraicValue av = ...;
entity.setProperty(DC_NAME, av.getDataConstructorName());
entity.setProperty(DC_ORDINAL, av.getDataConstructorOrdinal());
for (String fieldName : fieldNames) {
    entity.setProperty(fieldName, av.getNthArgument(i++));
}

From an Entity to a CAL value

fromEntity :: Storeable a => Store a -> Entity -> a;
public fromEntity !s !e =
    input $ jEntityToJObject (outputList s.Store.fields) e;

where:

foreign unsafe import jvm "static method org...entityToJObject" 
   public jEntityToJObject :: JList -> Entity -> JObject;

jEntityToJObject reverses the process, creating an AlgebraicValue which is converted to a CAL value by the input function.

The Data Store Monad

Because data store operations operate via side effects we need to create a Monad instance to manage them. This is required or two reasons:

  • We need to ensure that our data store operations happen in a definite sequence — our puts must happen before we commit the transaction, for example. Lazy evaluation won’t guarantee this without special attention.

  • Some operation don’t produce results, or produce results which will be ignored, so lazy evaluation won’t cause these operations to happen at all.

We define the following functions for use with our Monad (the type of which is DSM):

put :: Storeable a => a -> DSM Key;

The put function creates a new record, returning the Key it was assigned by the data store.

update :: Storeable a => Entity -> a -> DSM Key;

The update function replaces a previously retrieved entity with a new value, returning the Key (which will always be the same as the old key). Note that the original Entity is used only for the value of its key — all the replacement fields come from the new Storeable value.

delete :: Entity -> DSM ();

The delete function deletes a previously retrieved entity.

find :: (Storeable a) => Store a -> Key -> DSM (Maybe (Entity,a));

The find function looks for a record using a Key and returns it or Nothing if it does not exist. The return value consists of a pair of values: the raw Entity and its value when converted to a Storeable instance. This allows update to be used later. An alternative design would have been to require Storeable instances to be able to store a key, but I think the current design better separates the concerns of data storage from the domain objects.

query :: (Storeable a, Outputable b) => Store a -> [(String,FilterOperator,b)] -> [(String,SortDirection)] -> DSM [(Entity,a)];

The query function returns a sorted list of (Entity,Storeable) pairs, based on a list of filter criteria applied to attributes of the records.

commit :: DSM ();

The runDSM function described below starts and commits the transaction within which the monad is being run, but if you need to operate on entities from more than one entity group you must commit the original transaction and start a new one using the commit function.

These functions are combined with the normal monad operators of bind and anonymousBind, and then the resulting function is ‘run’ with runDSM:

runDSM :: DSM a -> a;

Key Conversion

In the context of a web application we may wish to store keys in cookies or generated URLs. The following functions are provided for extracting Keys from Entity instances, and converting Keys to and from strings.

public keyToString :: Key -> String; 
public stringToKey :: String -> Key;
public getKey :: Entity -> Key;

Example

Suppose we want to retrieve a Game, add a letter to the set of guesses and store it again:

    updateGame :: Key -> Char -> Key;
    updateGame key letter =
        runDSM (find gameStore key `bind` (\p ->
            let (entity, game) = fromJust p;
            in update entity (addGuess letter game)
        ));

where addGuess returns a new game with the given letter added to its set of guesses. This function will terminate with an error if the Game is not found, that is, if find returns Nothing.

CAL Hangman on Google AppEngine, Part 1

This post describes a simple implementation of the game ‘Hangman’, written in CAL for Google Appengine.

Introduction

Hangman is a simple word guessing game — the computer chooses a word, telling you only how many letters it has. You that it contains a particular letter. Correct guesses fill in those letters in the word, until you have guessed all the letters in the word, or you make too many incorrect guesses and lose.

You can try my Hangman for AppEngine implementation to understand the game. The source is available at launchpad.

Hangman is a very simple game — the state is just the word being guessed, and the letters guessed so far. The point of implementing it in CAL on AppEngine is to develop CAL modules for the AppEngine API classes.

Design

It would be easy to implement hangman as a mostly client-side browser based application, but that wouldn’t have exercised much of the AppEngine API.

To make the task interesting, I decided that the game state would be persisted on the server, and a user’s current game would be identified by a cookie in their browser. So you can make some guesses, close your browser, and come back to the same game later.

Http Module

The Http module provides function for creating a server which handles requests and creates responses. The source is here.

The Server Function

All requests go through a single servlet, which is configured with init-params to reflectively call a static method generated by the CAL stand-alone jar builder.

The function called by the servlet must have the type HttpServletRequest -> JHttpServletResponse. These CAL types are simply imports of the Java HttpServletRequest and HttpServletResponse classes.

data foreign unsafe import jvm "javax.servlet.http.HttpServletRequest"
    public HttpServletRequest deriving Inputable, Outputable;

data foreign unsafe import jvm "javax.servlet.http.HttpServletResponse"
    private JHttpServletResponse deriving Inputable, Outputable;

JHttpServletResponse is private to the module because it is mutable and so shouldn’t be exposed to clients of the module. Instead Http provides a data type HttpServletResponse:

data public HttpServletResponse =
    private HttpServletResponse cookies :: [Cookie] headers :: [Header] body :: HttpServletResponseBody;

data public HttpServletResponseBody =
    private Ok contentType :: String content :: (Array Byte) |
    private Error code :: Int message :: (Maybe String) |
    private Redirect url :: String;

This allows the client to assemble the state of a response, which is then applied to the response in a single operation by the Http module. A monad to sequence operations on the Java response object would have been another alternative, but that wouldn’t enforce correct construction of the response to the same degree — e.g. the client might not set a content type. Note that the actual constructors are private so that the structure of the type is not exposed to clients of this module.

The current data type doesn’t allow the response to be streamed.

As clients can’t see or use JHttpServletResponse they need to use the server function to create a web server:

server :: [HttpServletRequest -> Maybe HttpServletResponse] -> HttpServletRequest -> JHttpServletResponse -> ();

This function takes a list of handlers, each of which is tried in sequence until one returns a response. So the server function for the hangman web application is:

public service =
    server [currentGameHandler, newGameHandler, guessHandler, restartHandler];

Writing Handlers

The Http module provides functions to test and retrieve properties of the request, and combinators to build handlers from these individual functions. The combinators are inspired by Mark Tullsen’s First Class Patterns.

Each combinator combines two functions, each of which extracts a property of the request, and returns a function of the type HttpServletRequest -> Maybe a.

It isn’t clear to me that this is the very best approach — experimenting with a Maybe/Reader monad stack would be interesting.

Combinators

hOr :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe a);

hOr requires that at least one of two extraction functions succeeds, i.e. returns Just x. The final result is the result of the first extraction function to succeed (if the first extraction function succeeds, the second is not evaluated).

hThen :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe b) -> (HttpServletRequest -> Maybe b);

hThen requires that both the extraction functions succeed, but it discards the result of the first extraction function.

hAnd :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe b) -> (HttpServletRequest -> Maybe (a,b));

hAnd requires that both extraction functions succeed, and returns their results as a pair.

hNot :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe ());

hNot reverses the sense of an extractor, succeeding if it fails and vice-versa.

The following combinators have specialised functions, rather than acting to combine two extraction functions.

hCompose :: (HttpServletRequest -> Maybe a) -> (a -> b) -> (HttpServletRequest -> Maybe b);

hCompose transforms the result of an extraction function.

hApply :: (HttpServletRequest -> Maybe a) -> (a -> HttpServletResponse) -> (HttpServletRequest -> Maybe HttpServletResponse);

hApply combines a sequence of extraction functions with a function that produces a response.

hAttempt :: (HttpServletRequest -> Maybe a) -> (a -> Maybe HttpServletResponse) -> (HttpServletRequest -> Maybe HttpServletResponse);

hAttempt is like hApply, but the response generation function returns Maybe HttpServletResponse, allowing further conditional processing outside the context of combined extraction functions.

Request Attribute Extractors

These functions look at attributes of the response and return their values (which may amount to just signalling their presence or absence).

getParameter :: String -> HttpServletRequest -> Maybe (Maybe String);

getParameter extracts the value of a named parameter from the request. It always succeeds, returning Just Nothing or Just $ Just paramValue.

requireParameter :: String -> HttpServletRequest -> Maybe String;

requireParameter extracts the value of a named parameter, but fails (i.e. returns Nothing) if it is not set.

There are similar get/require Header/Cookie functions.

matchUrl :: String -> HttpServletRequest -> Maybe ();

matchUrl takes a string containing a regular expression, and succeeds if the regular expression matches the path info of the request. It would be useful to have another function which returns the values of capturing groups of the regular expression.

Creating the Response

Utility functions are provided for each common response type.

ok :: HasByteRep a => String -> a -> HttpServletResponse;
public ok contentType content = ...

textHtml :: HasByteRep a => a -> HttpServletResponse;
public textHtml = ok "text/html";

redirect :: String -> HttpServletResponse;
public redirect url = ...

err :: Int -> Maybe String -> HttpServletResponse;
public err code msg = ...

These functions all create requests with default headers. Headers and Cookies are added by applying a function to the request.

addHeader :: Header -> HttpServletResponse -> HttpServletResponse;
addCookie :: Cookie -> HttpServletResponse -> HttpServletResponse;  

Example

Rather than an example from the hangman game, here’s a simple example which uses the functions discussed directly.

helloHandler =
    let
        page p = 
            let (firstName, maybeLastName) = p;
            in ok "text/plain" (
                case maybeLastName of
                    Just lastName -> "Greetings, " ++ firstName ++ " " ++ lastName;
                    Nothing -> "Hi there " ++ firstName;
                );
    in
        matchUrl "/hello" `hThen`
        (requireParameter "firstName") `hAnd`
        (getParameter "lastName") `hApply` page;

This handler will return a page for the url /hangman/hello if the firstName parameter is present. To keep the example as simple as possible it uses the text/plain content type to avoid the verbiage of an HTML page.

To be continued…

In the next instalment I’ll describe the CAL module which supports the AppEngine datastore.

Functional Programming on Google AppEngine

I’ve always hoped that Java would be the next Google App Engine language. Google have provided a solid platform with a ‘real’ JVM and few restrictions.

I deployed the sample application created with the Eclipse plugins a few minutes after my account was upgraded to support Java, and then I went on to deploy an application written in CAL.

There are two ways of integrating CAL and Java. The first is to dynamically load and compile a workspace at run time, and then reflectively call functions in the workspace’s modules. I’ve used this approach in the past, for example for CAL/GWT integration.

There are two reasons not to use this approach for an AppEngine application. Firstly, when CAL loads a workspace it uses a SecurityManager, which GAE doesn’t allow. It’s just used to get the class of the caller of a function so the code could probably be replaced by looking at the stack trace of an un-thrown Exception, but that would involve a custom CAL build. I suspect that this approach would also want to create some local files, which also wouldn’t work on AppEngine.

The other reason not to use this approach is that AppEngine runs your application on (potentially) many different JVM instances, and may stop your application when it is idle and restart it when it receives a request. Nothing I’ve read so far indicates how often this happens, but start up overhead which would be perfectly acceptable in a normal application server is likely to be a bad idea in AppEngine.

CAL modules can also be compiled statically to a ’stand-alone’ jar file.

Each public function in the module becomes a static function on the resulting class, e.g.:

square :: Int -> Int;
public square n = n * n;

becomes:

public static int square(int n, ExecutionContext context)

If we hadn’t explicitly typed square, it would have the inferred type Num a => a -> a and the function would have the signature

public static CalValue square(CalValue n, ExecutionContext context);

Where a CalValue can hold any Java types which CAL knows how to convert to CAL types via the input function, and ExecutionContext really just caches Constant Applicative Forms.

So the components of a minimal CAL application for AppEngine are:

  1. A module which imports HttpServletRequest and HttpServletResponse and provides suitable functions to get their properties and to set attributes of the response.
  2. A function with the type HttpServletRequest -> HttpServletResponse -> ().
  3. A servlet which calls that function.
  4. Extensions to the example build.xml provided by Google which package up the CAL modules into a stand-alone jar, and include the runtime support jars CAL needs.

Additionally all the other AppEngine Services (Data Store, Cache, Mail etc.) need to be made available to CAL (in Monads of course :-) ), but that can be done piecemeal as you need them.

I’ll just give one example from the Http module, as it illustrates the issue of dealing with null being returned by Java methods.

First I import the HttpServletRequest.getParameter method:

foreign unsafe import jvm "method getParameter"
    jGetParameter :: HttpServletRequest -> String -> String;

But it isn’t made public. The ‘real’ getParameter function is defined like this:

public getParameter req s =
    let v = jGetParameter req s;
    in if isNullString v then Nothing else Just v;

Allowing the Maybe type to handle the possibility that a parameter may not be set.

The build.xml changes are:

<target name="copy-quark-jars">
    <copy
            todir="war/WEB-INF/lib"
            flatten="true">
          <fileset dir="${quark.dir}/Quark/bin/java/release">
            <include name="**/calLibraries.jar" />
            <include name="**/calPlatform.jar" /> <!-- maybe don't need? -->
            <include name="**/calRuntime.jar" />
            <include name="**/calUtilities.jar" />
          </fileset>
          <fileset dir="${quark.dir}/Quark/lib/Resources/External/java/">
            <include name="**/icu4j.jar" />
            <include name="**/log4j.jar" />
          </fileset>
        </copy>
</target>

<target name="build-quark-standalone" depends="copyjars">
    <exec executable="quarkc.sh">
        <arg value="workspace.cws"/>
        <arg value="-lib"/>
        <arg value="${moduleName}"/>
        <arg value="public"/>
        <arg value="${className}"/>
        <arg value="war/WEB-INF/lib/calmodule.jar"/>
        <env key="QUARK_CP" value="src:war/WEB-INF/lib/appengine-api-1.0-sdk-1.2.0.jar:${sdk.dir}/lib/shared/geronimo-servlet_2.5_spec-1.2.jar"/>
    </exec>
</target>

Now to do something useful with it!