Dispatch is a library for conducting HTTP interaction. You tell it what to do in pure and uninhibited Scala, and Dispatch carries out your orders using its trusty HttpClient backend.
The simplest way to try Dispatch is to clone the Twine example application from github.
git clone https://github.com/n8han/dispatch-twine.git
If you don’t have git available, you can download and extract the project’s current zip archive.
Twine is built with sbt 0.10. If you haven’t yet setup sbt, now
is a good time to do that. Once you have sbt on your executable search
path, you can enter its interactive console for the Twine project.
cd dispatch-twine
sbt
In sbt’s console, call the console task.
console
On the first run this task will download Dispatch dependencies and
compile the Twine app. After that, you should see a message welcoming
you to the console and a friendly scala> prompt.
First, import the main Dispatch classes and objects into scope.
import dispatch._
Then, we’ll need an HTTP executor to carry out our requests.
val h = new Http
Requests are described with dispatch.Request objects, and one way to
construct them is with a URL.
val req = url("http://www.scala-lang.org/")
This works because you’ve imported an object called url from the
dispatch package, and the object is itself a function that creates
request objects.
Now, we have an executor and a request. Dispatch needs to know how to handle the request.
val handler = req >>> System.out
With this complete request-response handler, Dispatch can execute the request. Assuming you have a network connection, that is.
h(handler)
And that was the source of the Scala home page. Of course, we don’t usually assign all these component parts to values unless we need to reuse them. Typically, the handler above would be written in one line.
h(url("http://www.scala-lang.org/") >>> System.out)
If you can’t wait to see more Dispatch verbs like >>>, here’s a
colorful cheat sheet.
Dispatch supports arbitrarily many types of HTTP executors—you could make your own, and you wouldn’t be the first to do so. Executors carry out your request handlers and callbacks.
Because executors are defined separately from HTTP interaction, you can model HTTP interaction without demanding a particular implementation. For example, the same interface to a web API can be used with an NIO executor and a Google App Engine executor.
Dispatch includes several executors, defined in various modules and
packages. These executors are all called “Http”, so application code
typically imports only the base dispatch package and refers to
executors by their subpackage name, as seen below.
To follow along with the examples below, use a console of the Twine example app described in the preceding page.
The traditional and most common Dispatch executor is located in the
base dispatch package. It performs its operations in the current
program thread, allowing it to return whatever value the handler
builds. If you request a string response body with as_str, the
executor’s return type is String.
import dispatch._
val h = new Http
h(url("http://www.scala-lang.org/") as_str)
You may notice in the Scala console that the last expression above
reports a return type of HttpPackage. This wrapping type is required
to generalize the executor model, but in this executor it is defined
as the wrapped type itself. It can be used any place where the wrapped
type is expected.
val s: String = h(url("http://www.scala-lang.org/") as_str)
An important caveat of the executor defined above is that it is not thread-safe. You should not share references to it across threads. Instead, Dispatch applications commonly define a function in some convenient object that returns a newly constructed executor. This also allows you replace it with a customized executor, later.
def http = new Http
Another option is to use a thread-safe executor, which maintains a shared connection pool and may (or may not) provide better performance. Dispatch provides a trait for this behavior:
val http = new Http with thread.Safety
The Http singleton object is an executor constructed with this
trait, so you can use it from any thread.
Http(url("http://www.scala-lang.org/") as_str)
Another executor that does not depart from the traditional model is the background thread executor.
import dispatch._
val h = new thread.Http
val f = h(url("http://www.scala-lang.org/") as_str)
This executor application evaluates to a future for the string value to be constructed by the handler. A future in Dispatch is a function.
type Future[T] = Function0[T] {
def isSet: Boolean
}
When applied, the future waits until its underlying value is
available, and isSet reports this availability. These are useful
when handling a batch of requests, for example.
val urls: Traversable[String] = < some urls I need to get >
val futures = urls.map { u => h(url(u) as_str) }
< do something else that's slow... >
val (ready, working) = futures.partition { _.isSet }
consume(ready.map { _() }) // mapped to Traversable[String]
This executor uses Java NIO and does not block a current or background thread while carrying out requests. You can have n active requests without tying up n threads or waiting for available threads from a pool.
import dispatch._
val h = new nio.Http
val f = h(url("http://www.scala-lang.org/") as_str)
Like the background thread executor, the NIO executor returns a Dispatch future; you can operate on collection as in the previous example. One caveat of this executor is that it must be shutdown.
h.shutdown()
If you’re using this executor in a console and forget to shut it down, the console may not close when asked.
This executor operates on the current thread, like dispatch.Http,
but it is specially modified to work with Google App Engine which
does not permit the direct use of sockets.
import dispatch._
val h = new gae.Http
val s = h(url("http://www.scala-lang.org/") as_str)
Note that this executor is not on Twine’s classpath and should only be used with App Engine.
The examples above all use apply which will execute the handler if
the resulting HTTP statusCode is in the range 200 to 204. This range
allows the standard handlers to fail-fast in situations where the response
wouldn’t be handled. If you need finer control please review the docs
for the when and x methods on trait HttpExecutor.
Dispatch is divided into a number of modules so that client applications need only depend on the parts of Dispatch they use. Some of the modules depend on other modules, and dependencies managers will add these transitive dependencies automatically.
Each module is cross-built against several versions of Scala and published to the scala-tools repository with the organization-id “net.databinder”. The modules have the Scala version they are built against appended. For Scala 2.9.1, the full artifact names are as follows:
Because each web API and third party Scala library has its own release timeline, some modules are not part of the main Dispatch build and distribution. Instead, they are external modules hosted on github and tagged for release as needed.
Since these modules are build from source, binary compatibility with Scala and Dispatch versions is not an issue. More importantly, they can be trivially initted, forked, and used by anyone.
If a module is lacking an API method that you need right now, you can fork it, fix it, tag it, and push it without waiting on anyone else. At that point it’s “released” under your name. You should also fork this documentation to add your integration module to the list:
When using sbt with Scala binary dependencies, it’s best to have the
Scala version automatically appended so it will always match
your project’s. In a sbt 0.11.x build.sbt:
libraryDependencies ++= Seq(
"net.databinder" %% "dispatch-http" % "0.8.7"
)
To use source dependencies with sbt, create a project build such
as project/build.scala. This is an example using dispatch-lift-json:
import sbt._
object MyApp extends Build
{
lazy val root =
Project("root", file(".")) dependsOn(dispatchLiftJson)
lazy val dispatchLiftJson =
uri("git://github.com/dispatch/dispatch-lift-json#0.1.0")
}
The Twine example application uses both source and binary Dispatch dependencies in its sbt project.
With Maven, you can depend on Dispatch’s binary modules by specifying the full artifact id with Scala version:
<dependency>
<groupId>net.databinder</groupId>
<artifactId>dispatch-http_2.9.1</artifactId>
<version>0.8.7</version>
</dependency>
To use source dependencies with Maven, your best bet is to check out the project as a submodule.
Dispatch (including this documentation) lives on GitHub, and many of its features and bug fixes originate in forks of the project there. Dispatch’s documentation is generated via Pamflet. We also have a mailing list.
This section explores basic request definition and response handling with Dispatch. You’ll need a Scala console with Dispatch on the classpath to follow along.
Hosts are defined with the :/ verb. It should remind you (a
little) of the characters that come before hosts in URLs.
import dispatch._
val sl = :/("www.scala-lang.org")
This value sl is a request to the root path of the Scala home
domain. What if we want some sub-path of it? There’s an obvious verb
for that: /
val learnScala = sl / "node" / "1305"
Now we have a reference to a page (the “Learning Scala” page
actually), and sl is unchanged.
Request definitions such as these are immutable; by appending a path to one you are creating a new request object, similar to string concatenation in Java. If we don’t need to use the host for different requests, we would probably have defined this all at once:
val learnScala2 = :/("www.scala-lang.org") / "node" / "1305"
But—can’t you define this with its actual URL? Of course!
val learnScala3 = url("http://www.scala-lang.org/node/1305")
The url verb is also imported from the dispatch package and it
produces a request object like the others. It is most useful when
dealing with URLs stored externally, or discovered at runtime. For
fixed requests in your application code, building up from host and
path components is usually preferred as it lends itself to reuse.
Request definitions, by themselves, are not enough information for Dispatch to do its job. The response that the server returns may have no body at all—or it could contain too much data to hold in memory.
Before Dispatch makes a request, then, you must tell it how to handle the server’s response. For this we have handlers. Handlers are created from request definitions using handler verbs. Taking up the request defined on the last page, we could simply ignore the response.
val ignore = learnScala >|
This would be very unusual for a GET request in particular, which shouldn’t have any side effects. (It’s also not speaking well of our commitment to learning Scala.)
To use a handler, you pass it to a Dispatch executor.
Http(ignore)
Okay, but let’s say you actually want to do something with the response. Assuming it’s text of a reasonable size, you could retrieve it as a string:
Http(learnScala >- { str =>
str.length
})
Like most handlers this one takes a function from the transformed
response body to any type. Here the function is of type String =>
Int. It merely returns the string length of the response.
How about trying something more interesting with the body, like extracting the page’s title?
import XhtmlParsing._
Http(learnScala </> { nodes =>
(nodes \\ "title").text
})
The </> handler processes the response as XHTML and passes a
scala.xml.NodeSeq to the supplied function. The \\ projection
function finds the <title> node, and text gives its contents.
Note: The parser imported here is very brittle. See the next section for alternatives that support real-world HTML.
While you can easily weild Dispatch’s stream and reader handlers to bind to any parser you like, several binding modules are available to make it even simpler.
Lift-JSON is developed and supported by the Lift framework. These are some basic instructions for using it with Dispatch.
Using Lift-JSON is very similar to using Dispatch’s internal JSON representation. As always, start by importing the main Dispatch methods.
import dispatch._
But instead of importing the internal JSON methods, import Dispatch’s interface to lift-json:
import dispatch.liftjson.Js._
We will now be able to use the ># operator in our handlers. For
example, if we wanted to search for scala podcasts, we might use
something like
import net.liftweb.json.JsonAST._
val http = new Http()
val u = url("http://gpodder.net/search.json") <<? Map("q" -> "scala")
http(u ># { json =>
(json \ "title" children) flatMap( _ match {
case JField("title", JString(d)) => Some(d)
case JString(d) => Some(d)
case _ => None
})
})
This script starts by importing lift’s JSON values (for pattern matching) and then creates the http executor and the gpodder url. We then execute with a handler which accepts a lift JValue and returns a list of the podcast titles. Simple, no?
TagSoup is a SAX-compliant parser, that parses HTML instead of well formed XML. TagSoup is very resilient, allowing you to load HTML as found in the wild, to be processed as a scala NodeSeq.
To process the response with TagSoup you first have to make the handler verbs available. This can be done in several ways.
import dispatch.tagsoup.TagSoupHttp._
Now we can use the operators </>, tagsouped and as_tagsouped in
our handlers. If we want to find the title of a HTML page it can look
something like this.
import dispatch.tagsoup.TagSoupHttp._
val title = Http(:/("example.org") </> { ns =>
(ns \ "title").text
})
TagSoup let’s you work with the HTML as a scala.xml.NodeSeq and as a
convenience you can use as_tagsouped to retrieve it.
val ns = Http(:/("example.com") as_tagsouped)
JSoup is a library for working with real-world HTML. It parses HTML and provides a convenient API for extracting and manipulating data. JSoup is similar to TagSoup as they both lets you work with real-world HTML, but JSoup provide more functionality for working with the extracted result.
To use JSoup you have to make it handler verbs available. This is easiest done by importing them from JSoupHttp.
import dispatch.jsoup.JSoupHttp._
Now we can use the operators </>, jsouped, as_jsouped and as_jsoupedNodeSeq
in our handlers. As a start we can extract the title from a HTML page.
import dispatch.jsoup.JSoupHttp._
val title = Http(:/("example.org") </> { doc =>
doc.title
})
JSoup parse the HTML to a DOM like structure org.jsoup.nodes.Document. There
is a rich set of find and search methods returning org.jsoup.nodes.Elements.
Elements implements java List so they are very easy to use with scala
collections, just make the usual import scala.collection.JavaConversions._.
So, to extract all links from a page and put them in a list as absolute paths looks like this:
import dispatch.jsoup.JSoupHttp._
import scala.collection.JavaConversions._
val request = :/("example.org") / "test.html"
val list = Http(request </> { doc =>
doc.select("a[href]").map(_.attr("abs:href"))
})
JSoup is a great api for processing HTML, and scala makes it even better. To learn more of it’s capabilities take a look in the JSoup Cookbook.
Once you’ve mastered the basic request and response verbs, you can compose almost any kind of HTTP interaction.
Dispatch has so many handy response handlers—but what if you want to use two with the same request? You can’t do that with a stream of the response body, since this can only be consumed once, but for headers it’s essential and straightforward.
The recommended header-handling verb >:+ provides a header Map and
a request object to chain a second handler for the body.
http(:/("dispatch.databinder.net") >:+ { (headers, req) =>
headers("content-type").filter {
_.contains("text/html")
}.headOption.map { _ =>
req </> { nodes => (nodes \\ "h1").text }
}.getOrElse {
req >> { _ => "unknown content type" }
}
})
To facilitate case-insensitive handling, the keys in the header
Mapare uniformly lowercase.
The function passed to >:+ should produce a second handler for the
body of the response. In this case, it produces different response
handlers depending on the headers present.
Many people are writing Android applications in Scala and want to do their HTTP communications with Dispatch. Unfortunately, this is not as easy as it should be.
The Android platform ships with the Apache HttpClient library. It is packaged in the same JAR as the “Java” standard library. From an application perspective it is just there, always.
When packaging for Android, you should be sure not to include any HttpClient jars; the classloader will always find the stock client’s classes first. This allows apps that require HTTP communication—nearly all apps—to save space.
The downside of shipping libraries in the standard classpath is that the environment becomes rigidly coupled to them. If you want to take advantage of a new feature in a library, too bad. You’d have to wait until all of your users upgrade their operating system to the new version—if it’s ever made available for their devices.
But wait, it gets worse! Updates to the HttpClient libraries in Android stalled. There was a procedural issue with binary compatibility; see this thread for the gory details. The short of it is, we can rule out just waiting for those parties to fix the situation.
Versions of Dispatch through 0.7.x can work with Android’s random old version of HttpClient just fine. Version 0.8.x however depends on newer major versions of the HttpComponents libraries and certain interfaces that are not compatible with the code shipped with Android.
It is possible to include the newer HttpComponents artifacts in the application package and to trick the classloader into finding them. How? You can obfuscate the class and package names with ProGuard, eliminating the namespace collisions.
But really: isn’t programming hard enough without some bytecode analyzer changing the names of all your symbols prior to runtime? For most of us, yes. That’s why it’s recommended here that you use Dispatch 0.7.x with Android. That works without any tricks and it’s still a lot nicer than having to code to the client directly.
This unfortunate problem underscores the need for Dispatch to be independent of any single backend. The primary advancement of 0.8 was to support HttpComponet’s async NIO client in addition to its standard blocking client, but that (like the rest of 0.8) is unusable on Android without some serious hackery.
Dispatch 0.9 will therefore need to support entirely different backends such as Netty or one of the higher level async clients built on Netty. And main reason to choose a backend like that for Andorid apps will be, ironically enough, its absence in the platform distribution.
This clever and attractive table lists Dispatch’s request and response verbs all in one place.
Generated documentation is available for the latest
version of Dispatch. Since most Dispatch interaction is defined by
verbs that are added implicitly to requests, the interesting
classes are RequestVerbs, CallbackVerbs, and so on.
These are some projects and companies that have used Dispatch. Are you using Dispatch? Fork this page on github.
Dispatch forms the foundation of an in-house SOAP client at Novus together with scalaxb. Thanks to these tools our standard response to “Can you do SOAP?” is “Bring it on!” instead of “Ew, what’s this on my shoe?!”
In a past life, maxaf wrote an internal baseball stats REST API client that was used to feed data into some very gruelling PDF generation code. Dispatch was used to cleanly produce HTTP requests & parse responses without hurting engineer morale.
conscript uses Dispatch to fetch program descriptors from github.
giter8 uses Dispatch to fetch templates from github.
kiln2rally uses Dispatch along with lift-json to hit Rally’s awesome web API.
The posterous-sbt plugin uses Dispatch to publish project release notes.
Unfiltered uses Dispatch in its system tests.