Writing a Haskell API Server, Part 2

Making the Domain Work for You

Back in part 1 we spent a fair amount of time working to make sure that as much of the low-level data logic of our Haskell api server was kept in the database as possible. Things like auto-incrementing ids, created_at/updated_at dates, nulls, referential integrity, and uniqueness are best dealt with at the lowest possible layer. We’ve found that unless these concerns are dealt with there, they’ll need to be dealt with over and over again on all higher layers.

In part two, we’ll move on into the Haskell realm where we can build up true domain logic rather than simple data integrity logic. As often happens in software, we begin to see a layered structure emerge. Just like we encapsulated a lot of the low-level data integrity in the data layer, we likewise started to build up domain logic encapsulated in types. We found that designing types carefully at the outset yields lots of benefits down the road. If we can write types in such a way as to make certain bugs lead to compiler errors, then that’s awesome; it’s a huge win. That translates to a whole bunch of tests that we don’t even have to write (because if we did write such a test, it wouldn’t even compile). Yaron Minsky coined the term “make illegal states unrepresentable” for this tactic. Scott Wlaschin expounds wonderfully on this topic in “Domain modelling with the F# type system.” Ken Fox has also written a great post on how this sort of type-first programming meshes really well with TDD. All of these resources are excellent if you are interested in what we’re up to here.

We found a number of techniques to be really helpful for modeling our domain. A common thing that Jon and I would say to one another is that we were “front-loading the headache,” meaning that we had to put in some effort upfront — which turned out to be well worth it as time went on. This definitely felt odd for me at first, but I got used to it. These techniques fell into a number of patterns which I’ll now illustrate with some examples.

Newtype all the things!

The first surprising thing that we found is that almost nothing in an application is actually a string. Sure, many things are commonly represented as strings, but they’re not really strings. Most of the time when we use a string for some data, there’s actually more structure embedded in there. Email addresses are a perfect example. We can write them out and store them in strings, but they really have a lot of structure:

name@example.com
 ^     ^
 |     |
 |     ` server part
 ` local part

In fact, if something is significant to the business domain, then probably isn’t a vanilla string (or int, or float, etc.). A good test for this is if you have to inspect inside a string, then that’s a strong hint that it isn’t just a string. As a developer, it’s our job to find out what those constraints are that prevent something from really being a string.

To that end, in our app we wrapped most data in newtype wrappers. This tells the compiler to treat the wrapped data as if it were completely novel even if at runtime the data is stored precisely as a string would be.

One entity in the API server is a Contact, this type represents a user’s contact information. Here’s what it looks like:

data ContactFields = ContactFields
  { con_name     :: ContactName
  , con_email    :: ContactEmail
  , con_address  :: Maybe ContactAddress
  , con_url      :: Maybe ContactURL
  , con_nickname :: Maybe Nickname
  , con_twitter  :: Maybe TwitterHandle
  , con_linkedIn :: Maybe LinkedInHandle
  } deriving (Eq, Show, Typeable)

You may notice that this datatype is for ContactFields rather than just Contact — that’s very significant and I’ll explain it in the Separate lifecycle phases with types section. For now, just note that each field is named something like ContactName and etc. Each of these is a newtype, defined like this:

newtype ContactName = ContactName ST.Text
  deriving (Byteable, Eq, Show,
            FromJSON, ToJSON,
            Convertible SqlValue, Parsable)

Here you can see that a ContactName is actually just Text, but it is wrapped in such a way that the compiler will catch any errors in using it. A ContactName could’t be used where we were expecting a BusinessName (and so on). There is surely some (syntactic) overhead to adding all these newtypes, but they save us from writing a lot of tests and they document the API well. We found the tradeoff to be worth it.

Border security

The next piece that we attacked was the issues around the “border.” This is any place where we either get or receive data from outside the application. This turns out to be a fair number of places even in a simple API server! Examples are:

• when we serialize or de-serialize data to the database
• when we convert to or from JSON
• when we read parameters from a web request

Any time that we do one of these actions, we must parse the data and/or deal with any errors in communication. If we read some data from a POST, we must make sure that it actually conforms to our idea of ContactFields.

Haskell has some really nice libraries for doing this. In the case of JSON we can use Aeson to attempt to parse JSON into a datatype (this is as shown before, just pay attention to the FromJSON and ToJSON in the deriving clause):

newtype ContactName = ContactName ST.Text
  deriving (Byteable, Eq, Show,
            FromJSON, ToJSON,
            Convertible SqlValue, Parsable)

Aeson can almost figure out how to automatically convert to and from our datatype, but we have to provide some help since we’re using newtypes pervasively:

instance FromJSON EmailAddress where
  parseJSON (String text) = maybe mzero pure . emailAddress $ SE.encodeUtf8 text
  parseJSON _ = mzero

instance ToJSON EmailAddress where
  toJSON = String . SE.decodeUtf8 . toByteString

In a few places, we have to provide these instances to show Aeson how to convert from an EmailAddress into a String (and vice versa). This is where you can really see that, by using a newtype, Haskell really has no idea what this type is.

Once we’ve defined the proper conversions, reading ContactFields from a POST looks like this:

instance FromJSON ContactFields where
  parseJSON (Object v) =
    ContactFields <$> v .: "name"
                  <*> v .: "email"
                  <*> v .:? "address"
                  <*> v .:? "url"
                  <*> v .:? "nickname"
                  <*> v .:? "twitter"
                  <*> v .:? "linkedin"
  parseJSON _ = mzero

We extract each individual field from the incoming JSON and then build a ContactFields with the result. The .:? operator is for fields that are optional. If these are null, we’ll be left with Nothings in the corresponding field. This lines up exactly with our datatype where, for example, an address (or ContactAddress) is optional. These are modeled in the type as Maybes.

So that is how we convert from a JSON value, but how do we convert ContactFields to a JSON value? We can’t. We simply do not provide a ToJSON instance for ContactFields. This prevents us from ever inadvertently sending ContactFields to the API client. This begins to blend with the next section, “Separate lifecycle phases with types.” When writing the application, we determined that there’s no valid business reason to send the client what is essentially an incomplete and unsaved Contact.

How do we create and save a Contact? That`s what the next section is about.

Separate lifecycle phases with types

We determined early on that there’s a big difference from an entity that has been saved in the database and one that hasn’t. Predictably, we enforce this distinction in the type system. You’ve already seen the datatype for ContactFields:

data ContactFields = ContactFields
  { con_name :: ContactName
  , …Omitting this part…
  } deriving (Eq, Show, Typeable)

As you saw in part one of this series, we only create auto-incrementing IDs in the data layer. Note that a Contact is just a ContactID paired with the ContactFields that you’ve already seen:

data Contact = Contact
  { con_id :: ContactID
  , con_fields :: ContactFields
  } deriving (Eq, Show, Typeable)

That is, we’re distinguishing between unsaved and saved Contacts. This gives us a convenient lifecycle handle on contacts:

1. ContactFields for a new Contact are POSTed to a controller action

2. The application validates that this is a reasonable and allowed thing to do

3. The application stores the ContactFields in the database

4. The database returns (via the RETURNING sql statement) a fully-formed Contact

5. The application deserializes the new Contact. Since Contact is an instance of ToJSON, the application can return it to the client as a response to the CREATE API action.

In almost every one of those steps, constraints on the validity of our data (enforced through types) limited the reasonable steps in our controller action. We end up with a very small and obvious controller action. Here’s an example of editing a Contact for a given User.

edit :: User -> CardMeActionM s ()
edit user = do
  cid <- getContactIdFor user
  contact <- Contact cid <$> fromParams
  saved <- reqQuery $ Contact.update contact
  json saved

Reading from the </span>do onward the events are nicely sequential.

1. We get the ContactID for the given user, since this is a protected route we get the User via an authenticate helper (this involves checking headers & etc.).
2. We use that ContactID and the supplied params, via fromParams to build a new Contact. fromParams is typed in such a way as to gather the POST parameters necessary to create ContactFields — this is a polymorphic return type in action again.
3. Next we run a query reqQuery that updates an existing Contact with the one that we just created. The new Contact is returned as saved
4. Finally, we send the contact as a JSON response. Remember, we can only do this with a fully-formed Contact, the program wouldn’t have compiled if we had attempted to send back the ContactFields that we got from fromParams.

Error handling

Though it looks like we don’t do any error checking here, each step can possibly fail. Let’s take the line, contact fromParams. Here we’re creating a new Contact from the submitted parameters. This requires that we’ve described a way to go from params to a ContactFields datatype:

instance FromParams ContactFields where
  fromParams = ContactFields <$> reqParam "name"
                             <*> reqParam "email"
                             <*> optParam "address"
                             <*> optParam "url"
                             <*> optParam "nickname"
                             <*> optParam "twitter"
                             <*> optParam "linkedin"

This sort of code should be starting to look familiar by now. We’ve specified how to convert from individual params into ContactFields data. The reqParam and optParam determine if the param is required or not:

optParam :: (Parsable a) => T.Text -> CardMeActionM s (Maybe a)
optParam key = do
  val  params
  case val of
    Just unparsed -> case parseParam unparsed of
      Left _  -> raise $ MalformedParam key
      Right x -> return $ Just x
    _ -> return Nothing

reqParam :: (Parsable a) => T.Text -> CardMeActionM s a
reqParam key = do
  val  return x
    _      -> raise $ MissingParam key

There’s a lot going on in these examples, but let me sketch it out for you a little. Required parameters (reqParam) are enforced by reusing the code for optional param, but then raise-ing if the param wasn’t provided. So that means all the heavy lifting is in optParam. Let’s look at what `optParam` does.

First we look up the param among all the params, params, by the provided key. If the param is missing, we immediately return a Nothing to the controller action via a return Nothing, the param wasn’t provided and that’s okay because it was optional. If the val is present, we now have an unparsed, possibly-invalid param. We then run parseParam on the unparsed param. Haskell is polymorphic in the return type of functions, so whatever type (the a in the type signature) we’re expecting will be what the parameter is expected to parse as. If that parse fails, we have a present-but-malformed parameter and we raise that to the controller action. Otherwise, we’ve got a present-and-valid parameter so we return it.

I mentioned raise a second ago. Despite the name, this is actually just a normal function, albeit one that can abort the Controller action with more information. The amazing thing is that this is all accomplished without any special handling by Haskell. At any stage within our `CardMeActionM` monad, we can interrupt the flow with an early return (an exception). These errors are propagated back to the outer function where they are handled. In our routes we’ve set up a fall-back route that looks for errors:

fallback :: CardMeException -> CardMeActionM s ()
fallback err = case err of
  MissingAuthToken   -> status status401 >> text "authorization header required"
  UnauthorizedUser   -> status status401 >> text "unauthorized"
  NoQueryResults     -> status status404 >> text "file not found"
  NoContactForUser   -> status status404 >> text "user has no associated contact"
  (MissingParam p)   -> status status422 >> text (LT.concat ["missing parameter: ", p])
  (MalformedParam p) -> status status422 >> text (LT.concat ["malformed parameter: ", p])
  ServerError t      -> status status500 >> text t

-- using it in the routes:

routes :: CardMeServerM s ()
routes = defaultHandler fallback >> do

    post "/capture"        $ User.authenticate Contact.capture
    -- rest of the controller actions follow...

These functional exceptions allow us to separate out error-handling code while still keeping local control flow (as opposed to the non-local control flow of regular exceptions).

Summary

We talked a lot in this post about doing some type-first designing to encode domain logic in the very types that we use to represent those concepts. We also did a lot of testing, which you haven’t seen in this post. Tests revolved around questions like can we round-trip a Contact through the database? These revealed the bulk of our bugs. Combined with a few issues while deploying (e.g. deserializing port numbers to Haskell) we almost got the magical worked on the first try effect.

My big takeaway from this portion of the work was just how untrustworthy the outside world is. All the bugs that we ran into happened at the borders of the application. Any time we read data in, that was also where bugs crept in. Parse early and mercilessly but then rest easy after that. That being said, debugging was fairly effortless. When we noticed undesired behavior, we found the relevant border crossing that the infraction must be located at and fixed it without much fuss.

Thanks

• Bendyworks for professional development time & the room to try new stuff
• Jon for putting in all the time that actually makes a project like this work. He’s always been the one that fixes the half-baked stuff that I write. Also, it has been immensely rewarding to pair-program on Haskell professionally.
• And last, but certainly not least, Joe Nelson for his utterly indefatigable work on everything Haskell, all of which I depend on constantly: haskell-vim-now, heroku-buildpack-ghc, postgrest and more. Also I want to thank Joe for hours of deep conversations during BayHac2014  about what’s possible using databases, Haskell, and so much more. My understanding about how to design applications like this wouldn’t have been possible without his insights. His urging (even as a voice in the back of my head) has caused me to finish more projects than anything else. Thanks Joe!

Posts in this series