CAL Hangman on GAE Part 2 — The Datastore


My implementation of Hangman has only very simple data storage requirements, so my CAL module Datastore doesn’t cover the entire capabilities of Bigtable.

All the application needs is to be able to save a Game instance, put the key of that instance into a cookie in the user’s browser, and then retrieve and update that instance as the user makes each guess, or delete it if the user chooses to restart the game.

Important unexplored areas are:

  • The creation of entities having another entity as a parent. This is important in Bigtable, because a transaction can only operate on entities in the same ‘entity group’ — that is, entities which share a common parent.

  • The persistence of references to other entities, and special treatment of these references when performing operations on the entity containing the reference — that is, there is no support for modelling relationships between entities.

  • No error handling support — any exception will result in a 500 response. In particular, ConcurrentModificationException should be treated differently, as this indicates that a transaction failed due to another transaction modifying the same records. A failure of this type should either be retried, or reported to the user in an application specific manner.

  • The module is limited to storing types with a single constructor, although there is no difficulty in extending this to support multiple constructor ADTs.

It is implemented using the low level api, not JDO.

Datastore Module

The Datastore module provides a Monad instance for using the AppEngine data store, and a Storeable type class. The source is here.

The Storeable Type Class

Algebraic data types which need to be persisted must be instances of Storeable, which provides metadata needed to persist instances of the type. The metadata includes:

  • The ‘kind’ of the record — analogous to the name of the table to store instances of the type in, not to kind in the type theory sense.

  • The data store property names to use for each of the constructor arguments.

When using CAL in statically compiled mode the names of types and the names of constructor parameters are not easily available at runtime. The constructor arguments could simply be stored in properties named arg0...n, but this would make querying and debugging more difficult.

data public Store a = public Store kind :: String fields :: [String];

public class (Outputable a, Inputable a) => Storeable a where
    store :: a -> Store a;

Note that the type parameter a is not used in the declaration of the constructor arguments — it simply labels the store to ensure that the store used with, for example fromEntity below, is consistent with the type of Storeable we are expecting fromEntity to return. For this reason functions which construct a Store must be explicitly typed, like gameStore below, as otherwise no more specific type than Store a can be inferred.

So the instance for a game of hangman is:

data public Game = 
    private Game word :: String guesses :: String deriving Show, Inputable, Outputable;

gameStore :: Store Game;
public gameStore = Store "Org.Kablambda.AppEngine.Test.Game" ["word","guesses"];

instance Storeable Game where
    store = gameStoreA;

private gameStoreA a = gameStore;

The public gameStore function is used when we wish to query.

Converting between CAL values and AppEngine Entities

Two conversion functions are used, one in each direction. In both cases the bulk of the work is carried out in Java functions.

From a CAL value to an Entity

toEntity :: Storeable a => a -> Entity;
public toEntity !r =
    jObjectToEntity (store r).Store.kind (outputList $ (store r).Store.fields) (output r);


foreign unsafe import jvm "static method org...jObjectToEntity" 
    public jObjectToEntity :: String -> JList -> JObject -> Entity;

The parameters passed to jObjectToEntity are the kind of entity to create, the list of names to use for the fields of the CAL value, and the CAL value, converted to a JObject by the output function. Storeable instances should all derive Outputable, so output r uses the default implementation of output to convert the algebraic data type instance to an instance of the CAL Java support class AlgebraicValue. The fields are transferred to the AppEngine entity like this:

Entity entity = new Entity(kind);
AlgebraicValue av = ...;
entity.setProperty(DC_NAME, av.getDataConstructorName());
entity.setProperty(DC_ORDINAL, av.getDataConstructorOrdinal());
for (String fieldName : fieldNames) {
    entity.setProperty(fieldName, av.getNthArgument(i++));

From an Entity to a CAL value

fromEntity :: Storeable a => Store a -> Entity -> a;
public fromEntity !s !e =
    input $ jEntityToJObject (outputList s.Store.fields) e;


foreign unsafe import jvm "static method org...entityToJObject" 
   public jEntityToJObject :: JList -> Entity -> JObject;

jEntityToJObject reverses the process, creating an AlgebraicValue which is converted to a CAL value by the input function.

The Data Store Monad

Because data store operations operate via side effects we need to create a Monad instance to manage them. This is required or two reasons:

  • We need to ensure that our data store operations happen in a definite sequence — our puts must happen before we commit the transaction, for example. Lazy evaluation won’t guarantee this without special attention.

  • Some operation don’t produce results, or produce results which will be ignored, so lazy evaluation won’t cause these operations to happen at all.

We define the following functions for use with our Monad (the type of which is DSM):

put :: Storeable a => a -> DSM Key;

The put function creates a new record, returning the Key it was assigned by the data store.

update :: Storeable a => Entity -> a -> DSM Key;

The update function replaces a previously retrieved entity with a new value, returning the Key (which will always be the same as the old key). Note that the original Entity is used only for the value of its key — all the replacement fields come from the new Storeable value.

delete :: Entity -> DSM ();

The delete function deletes a previously retrieved entity.

find :: (Storeable a) => Store a -> Key -> DSM (Maybe (Entity,a));

The find function looks for a record using a Key and returns it or Nothing if it does not exist. The return value consists of a pair of values: the raw Entity and its value when converted to a Storeable instance. This allows update to be used later. An alternative design would have been to require Storeable instances to be able to store a key, but I think the current design better separates the concerns of data storage from the domain objects.

query :: (Storeable a, Outputable b) => Store a -> [(String,FilterOperator,b)] -> [(String,SortDirection)] -> DSM [(Entity,a)];

The query function returns a sorted list of (Entity,Storeable) pairs, based on a list of filter criteria applied to attributes of the records.

commit :: DSM ();

The runDSM function described below starts and commits the transaction within which the monad is being run, but if you need to operate on entities from more than one entity group you must commit the original transaction and start a new one using the commit function.

These functions are combined with the normal monad operators of bind and anonymousBind, and then the resulting function is ‘run’ with runDSM:

runDSM :: DSM a -> a;

Key Conversion

In the context of a web application we may wish to store keys in cookies or generated URLs. The following functions are provided for extracting Keys from Entity instances, and converting Keys to and from strings.

public keyToString :: Key -> String; 
public stringToKey :: String -> Key;
public getKey :: Entity -> Key;


Suppose we want to retrieve a Game, add a letter to the set of guesses and store it again:

    updateGame :: Key -> Char -> Key;
    updateGame key letter =
        runDSM (find gameStore key `bind` (\p ->
            let (entity, game) = fromJust p;
            in update entity (addGuess letter game)

where addGuess returns a new game with the given letter added to its set of guesses. This function will terminate with an error if the Game is not found, that is, if find returns Nothing.

Post a Comment

Your email is never shared. Required fields are marked *