While the combinators read nicely, once you start extracting attributes from the request things become messy, due to the hAnd combinator composing results as nested pairs.

Here’s an example:

skitchPutHandler =
    let
        storeImage p =
            let (p2, path) = p;
                (maybeContentType,imageData) = p2;
                ...
            in ...
    in
        matchMethod "PUT" `hThen` matchUrl "/skitch" `hThen`
            (getHeader "Content-Type") `hAnd` requireRawData `hAnd` requirePathInfo `hApply` storeImage;

Unpacking the values extracted from the request is cumbersome, and because many are Strings, error prone.

A better approach is to simply use the Maybe monad with the same extraction functions:

simpleSkitchPutHandler req =
    let
        maybeContentType = requireHeader "Content-Type" req;
        ...
    in
        matchMethod "PUT" req `anonymousBind` matchUrl "/skitch" req `anonymousBind`
            requireRawData req `bind` (\rawData ->
            requirePathInfo req `bind` (\path ->
            return $ storeImage path rawData (actualContentType path)));

Optional values are extracted in let expressions, and required values and tests are composed with the monadic bind operators in the main expression. You can make the function a bit more concise and ‘point free’ by writing a monad which composes the Maybe and Reader monads to supply the HTTP request parameter to functions implicitly:

readerMonadSkitchPutHandler =
    let
        ...
    in
        runHttpRequestM (
        method "PUT" `anonymousBind` matchUrlM "/skitch" `anonymousBind`
            rawData `bind` (\rawData ->
            pathInfo `bind` (\path ->
            (optional (header "Content-Type")) `bind` (\maybeContentType ->
            return $ storeImage path rawData (actualContentType path maybeContentType)))));

The function optional has the type HttpRequestM a -> HttpRequestM (Maybe a), that is, it allows a function to succeed by wrapping its result in a ‘Just’.

I’m not convinced that the improvement in readability gained by removing the request parameter is worth the increase in complexity.

So I’ll remove the combinators from my Http module and will rewrite Hangman using just the Maybe monad. I’ll keep the more complicated Monad on the back-burner in case I discover a compelling advantage.

I’d be interested to know what someone with a deeper understanding of functional programming would say.

My implementation of Hangman has only very simple data storage requirements, so my CAL module Datastore doesn’t cover the entire capabilities of Bigtable.

All the application needs is to be able to save a Game instance, put the key of that instance into a cookie in the user’s browser, and then retrieve and update that instance as the user makes each guess, or delete it if the user chooses to restart the game.

Important unexplored areas are:

• The creation of entities having another entity as a parent. This is important in Bigtable, because a transaction can only operate on entities in the same ‘entity group’ — that is, entities which share a common parent.

• The persistence of references to other entities, and special treatment of these references when performing operations on the entity containing the reference — that is, there is no support for modelling relationships between entities.

• No error handling support — any exception will result in a 500 response. In particular, ConcurrentModificationException should be treated differently, as this indicates that a transaction failed due to another transaction modifying the same records. A failure of this type should either be retried, or reported to the user in an application specific manner.

• The module is limited to storing types with a single constructor, although there is no difficulty in extending this to support multiple constructor ADTs.

It is implemented using the low level api, not JDO.

Datastore Module

The Datastore module provides a Monad instance for using the AppEngine data store, and a Storeable type class. The source is here.

The Storeable Type Class

Algebraic data types which need to be persisted must be instances of Storeable, which provides metadata needed to persist instances of the type. The metadata includes:

• The ‘kind’ of the record — analogous to the name of the table to store instances of the type in, not to kind in the type theory sense.

• The data store property names to use for each of the constructor arguments.

When using CAL in statically compiled mode the names of types and the names of constructor parameters are not easily available at runtime. The constructor arguments could simply be stored in properties named arg0...n, but this would make querying and debugging more difficult.

data public Store a = public Store kind :: String fields :: [String];

public class (Outputable a, Inputable a) => Storeable a where
    store :: a -> Store a;
    ;

Note that the type parameter a is not used in the declaration of the constructor arguments — it simply labels the store to ensure that the store used with, for example fromEntity below, is consistent with the type of Storeable we are expecting fromEntity to return. For this reason functions which construct a Store must be explicitly typed, like gameStore below, as otherwise no more specific type than Store a can be inferred.

So the instance for a game of hangman is:

data public Game = 
    private Game word :: String guesses :: String deriving Show, Inputable, Outputable;

gameStore :: Store Game;
public gameStore = Store "Org.Kablambda.AppEngine.Test.Game" ["word","guesses"];

instance Storeable Game where
    store = gameStoreA;
    ;

private gameStoreA a = gameStore;

The public gameStore function is used when we wish to query.

Converting between CAL values and AppEngine Entities

Two conversion functions are used, one in each direction. In both cases the bulk of the work is carried out in Java functions.

From a CAL value to an Entity

toEntity :: Storeable a => a -> Entity;
public toEntity !r =
    jObjectToEntity (store r).Store.kind (outputList $ (store r).Store.fields) (output r);

where:

foreign unsafe import jvm "static method org...jObjectToEntity" 
    public jObjectToEntity :: String -> JList -> JObject -> Entity;

The parameters passed to jObjectToEntity are the kind of entity to create, the list of names to use for the fields of the CAL value, and the CAL value, converted to a JObject by the output function. Storeable instances should all derive Outputable, so output r uses the default implementation of output to convert the algebraic data type instance to an instance of the CAL Java support class AlgebraicValue. The fields are transferred to the AppEngine entity like this:

Entity entity = new Entity(kind);
AlgebraicValue av = ...;
entity.setProperty(DC_NAME, av.getDataConstructorName());
entity.setProperty(DC_ORDINAL, av.getDataConstructorOrdinal());
for (String fieldName : fieldNames) {
    entity.setProperty(fieldName, av.getNthArgument(i++));
}

From an Entity to a CAL value

fromEntity :: Storeable a => Store a -> Entity -> a;
public fromEntity !s !e =
    input $ jEntityToJObject (outputList s.Store.fields) e;

where:

foreign unsafe import jvm "static method org...entityToJObject" 
   public jEntityToJObject :: JList -> Entity -> JObject;

jEntityToJObject reverses the process, creating an AlgebraicValue which is converted to a CAL value by the input function.

The Data Store Monad

Because data store operations operate via side effects we need to create a Monad instance to manage them. This is required or two reasons:

• We need to ensure that our data store operations happen in a definite sequence — our puts must happen before we commit the transaction, for example. Lazy evaluation won’t guarantee this without special attention.

• Some operation don’t produce results, or produce results which will be ignored, so lazy evaluation won’t cause these operations to happen at all.

We define the following functions for use with our Monad (the type of which is DSM):

put :: Storeable a => a -> DSM Key;

The put function creates a new record, returning the Key it was assigned by the data store.

update :: Storeable a => Entity -> a -> DSM Key;

The update function replaces a previously retrieved entity with a new value, returning the Key (which will always be the same as the old key). Note that the original Entity is used only for the value of its key — all the replacement fields come from the new Storeable value.

delete :: Entity -> DSM ();

The delete function deletes a previously retrieved entity.

find :: (Storeable a) => Store a -> Key -> DSM (Maybe (Entity,a));

The find function looks for a record using a Key and returns it or Nothing if it does not exist. The return value consists of a pair of values: the raw Entity and its value when converted to a Storeable instance. This allows update to be used later. An alternative design would have been to require Storeable instances to be able to store a key, but I think the current design better separates the concerns of data storage from the domain objects.

query :: (Storeable a, Outputable b) => Store a -> [(String,FilterOperator,b)] -> [(String,SortDirection)] -> DSM [(Entity,a)];

The query function returns a sorted list of (Entity,Storeable) pairs, based on a list of filter criteria applied to attributes of the records.

commit :: DSM ();

The runDSM function described below starts and commits the transaction within which the monad is being run, but if you need to operate on entities from more than one entity group you must commit the original transaction and start a new one using the commit function.

These functions are combined with the normal monad operators of bind and anonymousBind, and then the resulting function is ‘run’ with runDSM:

runDSM :: DSM a -> a;

Key Conversion

In the context of a web application we may wish to store keys in cookies or generated URLs. The following functions are provided for extracting Keys from Entity instances, and converting Keys to and from strings.

public keyToString :: Key -> String; 
public stringToKey :: String -> Key;
public getKey :: Entity -> Key;

Example

Suppose we want to retrieve a Game, add a letter to the set of guesses and store it again:

    updateGame :: Key -> Char -> Key;
    updateGame key letter =
        runDSM (find gameStore key `bind` (\p ->
            let (entity, game) = fromJust p;
            in update entity (addGuess letter game)
        ));

where addGuess returns a new game with the given letter added to its set of guesses. This function will terminate with an error if the Game is not found, that is, if find returns Nothing.

Introduction

Hangman is a simple word guessing game — the computer chooses a word, telling you only how many letters it has. You that it contains a particular letter. Correct guesses fill in those letters in the word, until you have guessed all the letters in the word, or you make too many incorrect guesses and lose.

You can try my Hangman for AppEngine implementation to understand the game. The source is available at launchpad.

Hangman is a very simple game — the state is just the word being guessed, and the letters guessed so far. The point of implementing it in CAL on AppEngine is to develop CAL modules for the AppEngine API classes.

Design

It would be easy to implement hangman as a mostly client-side browser based application, but that wouldn’t have exercised much of the AppEngine API.

To make the task interesting, I decided that the game state would be persisted on the server, and a user’s current game would be identified by a cookie in their browser. So you can make some guesses, close your browser, and come back to the same game later.

Http Module

The Http module provides function for creating a server which handles requests and creates responses. The source is here.

The Server Function

All requests go through a single servlet, which is configured with init-params to reflectively call a static method generated by the CAL stand-alone jar builder.

The function called by the servlet must have the type HttpServletRequest -> JHttpServletResponse. These CAL types are simply imports of the Java HttpServletRequest and HttpServletResponse classes.

data foreign unsafe import jvm "javax.servlet.http.HttpServletRequest"
    public HttpServletRequest deriving Inputable, Outputable;

data foreign unsafe import jvm "javax.servlet.http.HttpServletResponse"
    private JHttpServletResponse deriving Inputable, Outputable;

JHttpServletResponse is private to the module because it is mutable and so shouldn’t be exposed to clients of the module. Instead Http provides a data type HttpServletResponse:

data public HttpServletResponse =
    private HttpServletResponse cookies :: [Cookie] headers :: [Header] body :: HttpServletResponseBody;

data public HttpServletResponseBody =
    private Ok contentType :: String content :: (Array Byte) |
    private Error code :: Int message :: (Maybe String) |
    private Redirect url :: String;

This allows the client to assemble the state of a response, which is then applied to the response in a single operation by the Http module. A monad to sequence operations on the Java response object would have been another alternative, but that wouldn’t enforce correct construction of the response to the same degree — e.g. the client might not set a content type. Note that the actual constructors are private so that the structure of the type is not exposed to clients of this module.

The current data type doesn’t allow the response to be streamed.

As clients can’t see or use JHttpServletResponse they need to use the server function to create a web server:

server :: [HttpServletRequest -> Maybe HttpServletResponse] -> HttpServletRequest -> JHttpServletResponse -> ();

This function takes a list of handlers, each of which is tried in sequence until one returns a response. So the server function for the hangman web application is:

public service =
    server [currentGameHandler, newGameHandler, guessHandler, restartHandler];

Writing Handlers

The Http module provides functions to test and retrieve properties of the request, and combinators to build handlers from these individual functions. The combinators are inspired by Mark Tullsen’s First Class Patterns.

Each combinator combines two functions, each of which extracts a property of the request, and returns a function of the type HttpServletRequest -> Maybe a.

It isn’t clear to me that this is the very best approach — experimenting with a Maybe/Reader monad stack would be interesting.

Combinators

hOr :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe a);

hOr requires that at least one of two extraction functions succeeds, i.e. returns Just x. The final result is the result of the first extraction function to succeed (if the first extraction function succeeds, the second is not evaluated).

hThen :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe b) -> (HttpServletRequest -> Maybe b);

hThen requires that both the extraction functions succeed, but it discards the result of the first extraction function.

hAnd :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe b) -> (HttpServletRequest -> Maybe (a,b));

hAnd requires that both extraction functions succeed, and returns their results as a pair.

hNot :: (HttpServletRequest -> Maybe a) -> (HttpServletRequest -> Maybe ());

hNot reverses the sense of an extractor, succeeding if it fails and vice-versa.

The following combinators have specialised functions, rather than acting to combine two extraction functions.

hCompose :: (HttpServletRequest -> Maybe a) -> (a -> b) -> (HttpServletRequest -> Maybe b);

hCompose transforms the result of an extraction function.

hApply :: (HttpServletRequest -> Maybe a) -> (a -> HttpServletResponse) -> (HttpServletRequest -> Maybe HttpServletResponse);

hApply combines a sequence of extraction functions with a function that produces a response.

hAttempt :: (HttpServletRequest -> Maybe a) -> (a -> Maybe HttpServletResponse) -> (HttpServletRequest -> Maybe HttpServletResponse);

hAttempt is like hApply, but the response generation function returns Maybe HttpServletResponse, allowing further conditional processing outside the context of combined extraction functions.

Request Attribute Extractors

These functions look at attributes of the response and return their values (which may amount to just signalling their presence or absence).

getParameter :: String -> HttpServletRequest -> Maybe (Maybe String);

getParameter extracts the value of a named parameter from the request. It always succeeds, returning Just Nothing or Just $ Just paramValue.

requireParameter :: String -> HttpServletRequest -> Maybe String;

requireParameter extracts the value of a named parameter, but fails (i.e. returns Nothing) if it is not set.

There are similar get/require Header/Cookie functions.

matchUrl :: String -> HttpServletRequest -> Maybe ();

matchUrl takes a string containing a regular expression, and succeeds if the regular expression matches the path info of the request. It would be useful to have another function which returns the values of capturing groups of the regular expression.

Creating the Response

Utility functions are provided for each common response type.

ok :: HasByteRep a => String -> a -> HttpServletResponse;
public ok contentType content = ...

textHtml :: HasByteRep a => a -> HttpServletResponse;
public textHtml = ok "text/html";

redirect :: String -> HttpServletResponse;
public redirect url = ...

err :: Int -> Maybe String -> HttpServletResponse;
public err code msg = ...

These functions all create requests with default headers. Headers and Cookies are added by applying a function to the request.

addHeader :: Header -> HttpServletResponse -> HttpServletResponse;
addCookie :: Cookie -> HttpServletResponse -> HttpServletResponse;  

Example

Rather than an example from the hangman game, here’s a simple example which uses the functions discussed directly.

helloHandler =
    let
        page p = 
            let (firstName, maybeLastName) = p;
            in ok "text/plain" (
                case maybeLastName of
                    Just lastName -> "Greetings, " ++ firstName ++ " " ++ lastName;
                    Nothing -> "Hi there " ++ firstName;
                );
    in
        matchUrl "/hello" `hThen`
        (requireParameter "firstName") `hAnd`
        (getParameter "lastName") `hApply` page;

This handler will return a page for the url /hangman/hello if the firstName parameter is present. To keep the example as simple as possible it uses the text/plain content type to avoid the verbiage of an HTML page.

To be continued…

In the next instalment I’ll describe the CAL module which supports the AppEngine datastore.

I deployed the sample application created with the Eclipse plugins a few minutes after my account was upgraded to support Java, and then I went on to deploy an application written in CAL.

There are two ways of integrating CAL and Java. The first is to dynamically load and compile a workspace at run time, and then reflectively call functions in the workspace’s modules. I’ve used this approach in the past, for example for CAL/GWT integration.

There are two reasons not to use this approach for an AppEngine application. Firstly, when CAL loads a workspace it uses a SecurityManager, which GAE doesn’t allow. It’s just used to get the class of the caller of a function so the code could probably be replaced by looking at the stack trace of an un-thrown Exception, but that would involve a custom CAL build. I suspect that this approach would also want to create some local files, which also wouldn’t work on AppEngine.

The other reason not to use this approach is that AppEngine runs your application on (potentially) many different JVM instances, and may stop your application when it is idle and restart it when it receives a request. Nothing I’ve read so far indicates how often this happens, but start up overhead which would be perfectly acceptable in a normal application server is likely to be a bad idea in AppEngine.

CAL modules can also be compiled statically to a ’stand-alone’ jar file.

Each public function in the module becomes a static function on the resulting class, e.g.:

square :: Int -> Int;
public square n = n * n;

becomes:

public static int square(int n, ExecutionContext context)

If we hadn’t explicitly typed square, it would have the inferred type Num a => a -> a and the function would have the signature

public static CalValue square(CalValue n, ExecutionContext context);

Where a CalValue can hold any Java types which CAL knows how to convert to CAL types via the input function, and ExecutionContext really just caches Constant Applicative Forms.

So the components of a minimal CAL application for AppEngine are:

1. A module which imports HttpServletRequest and HttpServletResponse and provides suitable functions to get their properties and to set attributes of the response.
2. A function with the type HttpServletRequest -> HttpServletResponse -> ().
3. A servlet which calls that function.
4. Extensions to the example build.xml provided by Google which package up the CAL modules into a stand-alone jar, and include the runtime support jars CAL needs.

Additionally all the other AppEngine Services (Data Store, Cache, Mail etc.) need to be made available to CAL (in Monads of course ), but that can be done piecemeal as you need them.

I’ll just give one example from the Http module, as it illustrates the issue of dealing with null being returned by Java methods.

First I import the HttpServletRequest.getParameter method:

foreign unsafe import jvm "method getParameter"
    jGetParameter :: HttpServletRequest -> String -> String;

But it isn’t made public. The ‘real’ getParameter function is defined like this:

public getParameter req s =
    let v = jGetParameter req s;
    in if isNullString v then Nothing else Just v;

Allowing the Maybe type to handle the possibility that a parameter may not be set.

The build.xml changes are:

<target name="copy-quark-jars">
    <copy
            todir="war/WEB-INF/lib"
            flatten="true">
          <fileset dir="${quark.dir}/Quark/bin/java/release">
            <include name="**/calLibraries.jar" />
            <include name="**/calPlatform.jar" /> <!-- maybe don't need? -->
            <include name="**/calRuntime.jar" />
            <include name="**/calUtilities.jar" />
          </fileset>
          <fileset dir="${quark.dir}/Quark/lib/Resources/External/java/">
            <include name="**/icu4j.jar" />
            <include name="**/log4j.jar" />
          </fileset>
        </copy>
</target>

<target name="build-quark-standalone" depends="copyjars">
    <exec executable="quarkc.sh">
        <arg value="workspace.cws"/>
        <arg value="-lib"/>
        <arg value="${moduleName}"/>
        <arg value="public"/>
        <arg value="${className}"/>
        <arg value="war/WEB-INF/lib/calmodule.jar"/>
        <env key="QUARK_CP" value="src:war/WEB-INF/lib/appengine-api-1.0-sdk-1.2.0.jar:${sdk.dir}/lib/shared/geronimo-servlet_2.5_spec-1.2.jar"/>
    </exec>
</target>

Now to do something useful with it!

I began by working through Google’s Guestbook example, and then replaced its use of Google’s Users API with single sign on via Crowd.

Crowd Single Sign On

A Crowd client application authenticates via SOAP calls to a Crowd server. The sequence looks like this:

Duck Punching httplib for SOAP

I used the soaplib Python SOAP library to talk to Crowd. This library uses httplib to talk to the SOAP server, which is a problem as AppEngine only allows applications to use Google’s HTTP request mechanism.

I adapted soaplib to use Google’s fetch function by ‘Duck Punching’, also known as ‘Monkey Patching’. This globally overrides a library class with a class of your own:

import httplib

class AppEngineHTTPConnection(object):
... delegate the functions called by soaplib to Google's fetch ...

class Crowd(object):
def __init__(self, path, applicationName, applicationPassword):
    # do some monkeypatching
    httplib.HTTPConnection = AppEngineHTTPConnection

Simple, and safe in this case, as we want any client of httplib to use our replacement.

By default, soaplib puts an empty namespace on strings and arrays, so I cut, pasted and renamed these classes, changing them to explicitly set the correct namespace. A Pythonista could probably duck-punch their way out of that with less duplication

My modified Guestbook application adds login and logout URLs which authenticate with Crowd and create the appropriate cookie.

Caveats and Conclusions

AppEngine doesn’t support SSL, so your username and password are transmitted unencrypted to the application. To avoid this you could write a small authentication application to be hosted with your Crowd server under SSL.

Your Crowd server and your Google AppEngine application need to be on the same domain, in order to share the Crowd SSO cookie. This should be possible by assigning your domain to a Google Apps account, but I haven’t managed to add my Google Apps domain to my Google AppEngine application yet.

Python

I strongly dislike weakly typed languages such as Python. IDEs can’t sensibly provide code completion, and I found that I made many errors which were only caught at runtime. Most of these were in code which could easily have been type checked at compile time.

The sooner type inference is added to Python or AppEngine supports JVM languages, the happier I’ll be. I expect the latter to happen first.

Code

The code for this AppEngine app can be found in my crowd-appengine Git repository. As usual, get a snapshot if you don’t have a git client installed.