This post aims to document a practical design implementation we came up with when designing some APIs in Scala. Of course the concept is not Scala specific and applies to other functional languages.
If you don’t want to go into the full introduction, this post talks about how Applicatives are too restrictive and breaking them into two independent components can be interesting for Contravariant and Invariant Functors. Jump to implementation attached.
We are taking JSON serialization/deserialization as a motivating example.
Writing JSON we have its dual:
Implementing Writes provides a way to write JSON for a type
We have also a type that represents both, being able to read and write a JSON for an
Reads happens to be a monad, which is interesting but we are not going to talk about this much more. What we are interested in is having a simple way of combining JSON serializers, meaning if we have
it could be very interesting to have a
This is actually quite easy to achieve with Applicative Functors. To simplify, Applicative Functors are Functors (having the map or fmap function implemented) together with the Applicative part which adds the power we need to achieve the combination we want to do. Explaining Applicative Functors is out of scope and there is a lot of material on the web explaining these.
This composition simply means that if we have two readers of two different types, they could read from the same JSON and provide two values which I can pack into another type (User here).
Now that we solved our problem of combining the Reads with Applicative Functors, wouldn’t it be even more interesting if we could combine similarly the Writes? Having:
then we could have a
Almost. If we have two writers (capable of writing two different JsValue from two different types, Int and String here) then we end up with two
Let’s do a new type that only writes
Now it makes sense to have one writer out of our two writes above ( writeAge, writeName) by simply merging the resulting objects. So logically it makes sense, except that we can’t use our Applicative based builder since an OWrites is not a Functor at all!
Actually our OWrite is, naturally, the dual of a Functor, a CoFunctor (or Contravariant). Put simply, this means that to change the
So or status here, we know how to transform the type of OWrites, we know how to merge two OWrites but all of that doesn’t help us to use the builder which based on an Applicative Functor.
What if we split our problem into two components. Let’s introduce a new type that we call FunctionalCanBuild:
Meaning for a given container M, we can some how compose two instances into one with the tuple of the type parameters. So for Reads it will be:
and for the OWrites it will make:
But that gets us only half way through what we want to achieve. What we want is actually a Reads and OWrites of User and not tuple of Int, String.
So it seems that separating the problem into two sub problems is working for us. But what about format? Format is not a Functor and a Contra, it is actually Invariant. This means that it need functions in both directions,
Now we can implement FunctionalCanBuild[OFormat] :
and once we have the
This led us to an API that is capable of building compositions for Functors (
Bottom Line: By breaking
ps: FunctionalCanBuild[M[_]] may look like a monoid but it is not exactly one.
ps4: This work is integrated into Play’s included JSON library
ps5: A potentially more readable version as a gist
Realtime Web Applications
Realtime Web Applications are applications making use of Websockets, Server Sent Events, Comet or other protocols offering/simulating an open socket between the browser and the server for continuous communication. Basically, these applications offer to users delivery of information as it is published without having the user periodically pinging the service.
There are quite a few web frameworks that target the development of this type of applications. Mostly, however, the solution is by providing an API that allows developers to push/receive messages from/to an open channel, something like:
Though this kind of API offers an opportunity to get started doing Realtime Web, it doesn’t offer a programming model for dealing with the challenges encountered when programming with streams of data, including creating, adapting, manipulating, filtering and merging streams of data and all of the synchronization involved.
A Programming Model
Since development of Realtime Web Apps is mostly built around manipulation of streams of data, it is crucial to have a programming model which identifies clearly what a stream of data is and defines composable components to deal with it.
It is obvious that the above channel api falls short when manipulating a rich set of streams of data, but even the classic InputStream/OutputStreams interfaces are not sufficient. Forgetting about their inefficient blocking runtime properties, they don’t carry enough signals/information to allow building rich stream manipulation api.
Play2 uses Iteratees together with Futures for dealing with streams of data, providing a very rich model for programming rich Realtime Web Applications.
A sample Realtime Web App
The goal of this text isn’t to provide a detailed description of what Iteratees and pals are. Nevertheless I will go into a fast introduction and then move into an example illustrating few aspects of how powerful this approach is.
Our sample application features two streams, one is a stream of financial operations:
Here we are generating random values at random distances (of maximum 500ms). In the real world this stream could be coming from a datastore or an open socket with another server.
The other stream we have is a stream of system messages, messages that talk about the status of the system:
This stream can be coming from another server or datastore. With these two streams at hand, we can prooduce one single stream that contains messages of both by interleaving them:
Actually those not comfortable using symbolic operators can use
Now the model part of our application looks like:
Our sample Realtime Web App features:
Our application will publish this stream of
You can only see
You can filter the stream by range of interest in the amount of the operation. By providing an upper and lower bounds you get only corresponding operations, for this we create another
Our App will be pushing JSON messages to the browser, that’s why we need one more
For convenience, let’s produce one single adapter out of the three we got, for that we can use the
We’re almost done. Now all what we need is to respond to the browser with an
Here we pass our stream through the
Our application now looks like:
feed = new EventSource(‘/feed?role=@role&lower=’ + min + ‘&higher=’ + max)
Snapshots of our Realtime Web App:
Realtime Web involves dealing with different streams of data from different sources. It is hard to do any non-trivial application without having a programming model that contains an appropriate representation of a stream and necessary API for creating, adapting, filtering and consuming streams of data. Play2 uses
Note: Full working application source was created by me @sadache and @guillaumebort for one of our talks