The Power of Programmatic Builds
Using a build tool that provides the full power of a real programming language makes many tasks much easier than build tools that use a declarative markup language. To cut to the chase, I’m referring to sbt vs Maven (or Ant). Just wanted to write a quick post that shows how we have customized our sbt builds in two important ways: using different versions of dependencies based on which version of Scala we’re using, and publishing to different snapshot and release repositories depending on the project’s current version.
In the land of Scala, it’s common to publish library jars compiled with multiple versions of Scala (long story). So sbt supports cross-building. In this project’s build.properties file, we specify to compile it with both Scala 2.9.1 and 2.8.1:
build.scala.versions=2.9.1 2.8.1
That’s great and all, but one of the dependencies we want to use, Specs, only goes up to version 1.6.8 with Scala 2.8.1, then switches to version 1.6.9 for Scala 2.9.1. Might seem like a problem, but because we define our build in Scala, we just throw in an if-statement:
class Project(info: ProjectInfo) extends DefaultProject(info) {
//specs version depends on Scala version
val specsVersion = if ("2.9.1" == buildScalaVersion) "1.6.9" else "1.6.8"
val specs = "org.scala-tools.testing" %% "specs" % specsVersion % "test"
}
Run +update on that and you’ll get Specs 1.6.8 for Scala 2.8.1 and Specs 1.6.9 for Scala 2.9.1.
It’s standard practice to use separate Maven repositories for snapshot versions and release versions. Case in point: scala-tools snapshot & release repos. When your project is at version 0.9-SNAPSHOT you should publish its jars to the snapshot repo. Then when version 0.9 is released, publish to the release repo.
Again, because we’re defining the build in Scala, we just do some programming to set this up exactly how we want it:
class Project(info: ProjectInfo) extends DefaultProject(info) {
//publish destination depends on build version
override def managedStyle = ManagedStyle.Maven
def suffix = if (version.toString endsWith "-SNAPSHOT") "snapshots/" else "releases/"
val publishTo = "Scala Tools Nexus" at "http://nexus.scala-tools.org/content/repositories/" + suffix
Credentials(Path.userHome / ".ivy2" / ".scala_tools_credentials", log)
}
Pretty simple examples, but the point is that we’re not locked in to the XML elements & attributes that some Maven plugin author exposed to us. With sbt we can do basically whatever we need to in the build file to customize our build to meet project requirements.
Database Actors in Lift
We’ve been slowly migrating our Lift web app to more of an event-driven architecture. This approach offloads non-essential processing out of the HTTP request/response cycle into actors, and makes things a lot more flexible on the back-end. However, as we’ve discovered during this process, there are several things to be aware of by doing database processing in actors. In this post we’ll examine those problems and present our current solution, which is also demonstrated in an example Lift app and was recently discussed on the Lift mailing list.
Background
The app is very simple: users say things and these quotes show up on the home page. Other users can then “like” the quotes. The most-liked quotes are also shown on the home page.
Let’s take a look at the Like link in more detail. When you click it, a function in a snippet runs on the server-side.
This function toggles the like status between a Quote and a User. So after you like a quote, the link changes to Unlike so you can take back your like.
def likeLink(q: Quote): NodeSeq = a(likeLinkText(q), "id" -> likeLinkId(q)) {
for (u <- User.currentUser)
u toggle q
Replace(likeLinkId(q), likeLink(q))
}
This toggling makes its way to the QuoteLike meta mapper, to either the like or unlike method. In both cases, after the action is processed we send a message to the QuotePopularity actor.
object QuoteLike extends QuoteLike with LongKeyedMetaMapper[QuoteLike] with Logger {
...
def like(u: User, q: Quote) =
if (!exists_?(u, q)) {
val ql = QuoteLike.create.user(u).quote(q).saveMe
debug("User " + u.id.is + " liked Quote " + q.id.is)
QuotePopularity !<> q
Full(ql)
} else
Empty
def unlike(u: User, q: Quote) =
for (ql <- find(u, q)) {
ql.delete_!
debug("User " + u.id.is + " unliked Quote " + q.id.is)
QuotePopularity !<> q
}
def toggle(u: User, q: Quote) = if (exists_?(u, q)) unlike(u, q) else like(u, q)
}
This actor updates that quote’s popularity. The Popular Quotes section on the home page renders the top 10 quotes by descending popularity.
object QuotePopularity extends DbActor with Logger {
protected def messageHandler = {
case q: Quote =>
val p = q.likeCount
q.popularity(p).save
debug("Quote " + q.id.is + " popularity = " + p)
}
}
While we could easily have done this update in the QuoteLike like and unlike methods, we chose to offload this processing to an actor, since it is not essential to the AJAX response after the user clicks the Like/Unlike link. It’s a simple calculation in this example app, but imagine a more complex app where a lot of number-crunching must take place to determine trending items (*cough*Pongr*ahem*). We don’t want the AJAX response delayed, so we let an actor update the popularity sometime later. And the popularity of quotes is then updated & cached for the next home page load.
Problem
While this is a very common use case for actors (asynchronous “later” processing), what’s not immediately obvious is the dreaded database transaction. It’s standard in Lift to wrap every HTTP request inside a transaction. This is configured in Boot:
S.addAround(DB.buildLoanWrapper)
So at the end of our AJAX HTTP req/resp cycle due to the user clicking the Like/Unlike link, the new (or deleted) QuoteLike object is committed to the database, and can be read by other parts of our app. So far, so good.
However, by default, Lift actors are not wrapped in database transactions. So as soon as you send that message to the QuotePopularity actor, it may start updating the quote’s popularity. We have no guarantees as to when that actor will execute; it may be immediately in which case it won’t see the new/deleted QuoteLike, or it may happen to be after the QuoteLike is committed.
Another problem occurs if this actor makes changes to the database itself. Since it’s executing outside of a transaction, these changes are committed immediately, leaving some partially updated entities open to discovery by other parts of the app. Definitely not something we want to happen.
Solution
Our approach to solving this problem is the following simple DbActor trait:
trait DbActor extends LiftActor {
override protected def aroundLoans = List(DB.buildLoanWrapper)
def !<>(msg: Any): Unit = DB.performPostCommit { this ! msg }
}
We now follow these two best practices for actors that use the database:
- Extend DbActor instead of LiftActor
- Only send messages to the actor using the !<> method
So #1 ensures that this actor’s messageHandler method is executed inside a transaction. LiftActor has this awesome aroundLoans method, where we can simply wrap the actor in a DB.buildLoanWrapper (just like HTTP requests). Database changes made by the actor will now all be committed when the actor is finished. Our QuotePopularity actor above extends DbActor. Actors executing inside database transactions: check.
The !<> method in #2 ensures that this actor will only execute after the current database transaction commits. Again, Lift comes to the rescue with the DB.performPostCommit method, which does exactly what we want. So the QuotePopularity actor above will only execute after the AJAX HTTP request that creates/deletes a QuoteLike has been committed to the database. This way, we know for sure that the QuotePopularity actor will see the QuoteLike changes. Actors executing after the current transaction commits: check.
Conclusion
So just remember: if your Lift actor does anything with the database, follow the two best practices above. Wrap actor execution in a transaction, and send messages to the actor so that the actor executes after the current transaction commits.
It’s worth pointing out that we’re hard-coding the notification of the QuotePopularity actor above in the like and unlike methods. This is OK, but a better solution would be a generic pub-sub event system, where those methods would publish “quote liked/unliked” events, and QuotePopularity would just subscribe to those events. Similarly, QuotePopularity could publish a “quote popularity updated” event when it’s done, and something else like a comet actor could receive that event and update the Popular Quotes section of the home page. But that’s a topic for another blog post…
Lift + Ostrich
We’ve been doing some profiling on http://pongr.com recently and have started using Ostrich. Our site uses Lift and I thought I’d put together a brief tutorial showing how to use Ostrich in a Lift-based web app. Our approach was heavily inspired by the usage of Ostrich in ESME, is described below and is available on Github.
In this example we’ll use Lift 2.1 and Ostrich 2.2.10. Update: As Steve commented, there is a newer release of Ostrich (2.3.3 as of this post) but it requires Scala 2.8.1, so to use it you would also need to use Lift 2.2. You’d also need to add the Twitter maven repo to the sbt project.
Setup
First off, we need to add the JBoss maven repo and the Ostrich dependency to the sbt project:
val jbossRepo = "jboss" at "http://repository.jboss.org/nexus/content/groups/public/" override def libraryDependencies = Set( "net.liftweb" %% "lift-webkit" % liftVersion % "compile->default", "net.liftweb" %% "lift-mapper" % liftVersion % "compile->default", "org.mortbay.jetty" % "jetty" % "6.1.22" % "test->default", "junit" % "junit" % "4.5" % "test->default", "org.scala-tools.testing" %% "specs" % "1.6.5" % "test->default", "com.h2database" % "h2" % "1.2.138", "com.twitter" % "ostrich_2.8.0" % "2.2.10" ) ++ super.libraryDependencies
Ostrich will provide stats via HTTP, so we need to define the port it will use in default.props:
#Enable and set port for Ostrich web access (JSON output) admin_http_port=9990
Ostrich needs to be started when the web app boots up:
// Ostrich setup
val runtime = new RuntimeEnvironment(getClass)
val config = new Config
config("admin_http_port") = Props.getInt("admin_http_port") openOr 9990
ServiceTracker.startAdmin(config, runtime)
As far as configuration goes, that’s it! Ostrich is now ready to collect data and provide stats.
Counters
The first type of data we’ll collect is the count of something. Could be number of users registered, number of new blog posts, etc. For this example we’ll count each time the HelloWorld.howdy snippet is rendered:
def howdy(in: NodeSeq): NodeSeq = {
Stats.incr("howdy-renders")
Helpers.bind("b", in, "time" -> date.map(d => Text(d.toString)))
}
Gauges
A gauge is just some value of something at a particular point in time, perhaps a measurement from a sensor or some calculated value. In our example we’ll use WTFs/min:
import scala.math._
Stats.makeGauge("wtfs-per-min") { rint(random * 10) }
We set up this particular gauge in Boot.scala and just generate random data. You just name a gauge and provide a function that always returns the current value.
Timings
Ostrich can also record how long it takes to execute certain blocks of code. Simply wrap the code you want to time in a call to Stats.time. For extra convenience, it will return whatever your code block returns.
In this example, we suspect our important number calculation is slowing down page loads, so we’ll time it:
def importantNumber(in: NodeSeq): NodeSeq = {
val n = Stats.time("important-number-calculation") {
import scala.math._
Thread.sleep(round(2000*random + 1000))
7
}
<span>{n}</span>
}
Stats
Now that we’re collecting all three types of data, let’s pull some stats from Ostrich. After loading our home page at http://localhost:8080 several times, we can get the latest stats from http://localhost:9990/stats.
{"counters":{"howdy-renders":7},"timings":{"important-number-calculation":{"count":7,"standard_deviation":621,"p75":3405,"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"maximum":2878,"p9999":3405,"p90":3405,"p25":2015,"p99":3405,"average":2179,"minimum":1447,"p50":2619,"p999":3405}},"jvm":{"nonheap_committed":46530560,"heap_max":1908932608,"thread_peak_count":54,"heap_committed":120782848,"uptime":436119,"nonheap_max":224395264,"thread_daemon_count":12,"num_cpus":8,"thread_count":54,"nonheap_used":45256464,"start_time":1292277570344,"heap_used":9656904},"gauges":{"wtfs-per-min":5.0}}
By default, Ostrich returns stats in JSON format which is great for parsing for display in another app or viewing with JSONView. However, for this blog post perhaps the plaintext version at http://localhost:9990/stats.txt is more readable:
counters: howdy-renders: 7 gauges: wtfs-per-min: 2.0 jvm: heap_committed: 120782848 heap_max: 1908932608 heap_used: 8911680 nonheap_committed: 46530560 nonheap_max: 224395264 nonheap_used: 45252880 num_cpus: 8 start_time: 1292277570344 thread_count: 54 thread_daemon_count: 12 thread_peak_count: 54 uptime: 430468 timings: important-number-calculation: (average=2179, count=7, maximum=2878, minimum=1447, p25=2015, p50=2619, p75=3405, p90=3405, p99=3405, p999=3405, p9999=3405, standard_deviation=621)
We can see that the howdy snippet was rendered 7 times and we’re currently seeing 2 WTFs/min. We also have collected 7 timings for the important number calculation and we see various stats like min/max/avg, etc. Now we know precisely how much time is being spent calculating important numbers, and we can choose to optimize if needed.
While not as detailed or comprehensive as a profiling tool like VisualVM, Ostrich is a great, simple tool for collecting performance data in specific parts of your Scala app. And integration with Lift really could not be easier.
Jersey + Guice + Scala
At Pongr, our RESTful web services are built using Jersey, Guice and Scala (among many other technologies). Here’s a quick post that shows how to set up an example project that uses all three. By the end we’ll be able to declare Guice bindings in our own custom module and have them injected into Jersey resource classes.
We’ll manage everything with Maven, so first create your pom.xml. I know it looks like a lot, but only has three Jersey dependencies (and one on the Servlet 2.5 API to make things compile). The rest of this junk is mainly configuring plugins for Scala – don’t worry about it. The upshot is that you can a) run this all using mvn jetty:run, and b) can generate an Eclipse project using mvn eclipse:eclipse.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.pongr</groupId>
<artifactId>jerseyguicescala</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>jerseyguicescala</name>
<properties>
<jersey-version>1.1.1-ea</jersey-version>
</properties>
<dependencies>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
<version>${jersey-version}</version>
</dependency>
<dependency>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-guice</artifactId>
<version>${jersey-version}</version>
</dependency>
<dependency>
<groupId>com.sun.jersey.contribs</groupId>
<artifactId>jersey-scala</artifactId>
<version>${jersey-version}</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<!-- Run with "mvn jetty:run" -->
<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>maven-jetty-plugin</artifactId>
<version>6.1.19</version>
<configuration>
<contextPath>/</contextPath>
</configuration>
</plugin>
<!-- Begin scala plugins, inspired by: http://scala-tools.org/mvnsites/maven-scala-plugin/usage_java.html -->
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-eclipse-plugin</artifactId>
<configuration>
<sourceIncludes>
<sourceInclude>**/*.scala</sourceInclude>
</sourceIncludes>
<buildcommands>
<buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
</buildcommands>
<additionalProjectnatures>
<!-- This nature gets put after org.eclipse.jdt.core.javanature in .project so Eclipse has a J badge on the project instead of an S -->
<projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
</additionalProjectnatures>
<classpathContainers>
<classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
<classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
</classpathContainers>
</configuration>
</plugin>
<!-- http://groups.google.com/group/liftweb/browse_thread/thread/3dac7002f9e59546/3918bba2f7a92cd3?pli=1 -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-test-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<!-- End scala plugins -->
</plugins>
</build>
</project>
Next up, create the src/main/webapp/WEB-INF/web.xml file. This just registers our special GuiceConfig class that, uh, configures Guice.
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_5.xsd" metadata-complete="false" version="2.5">
<listener>
<listener-class>com.pongr.GuiceConfig</listener-class>
</listener>
<filter>
<filter-name>guiceFilter</filter-name>
<filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>guiceFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
</web-app>
We’ll create an example Guice module in the src/main/java/com/pongr/ExampleModule.java file. It just binds a message String that we’ll inject into our resource classes later.
package com.pongr;
import com.google.inject.*;
import com.google.inject.name.*;
public class ExampleModule extends AbstractModule
{
@Override
protected void configure()
{
bindConstant().annotatedWith(Names.named("message")).to("Hello, World!");
}
}
Next up is src/main/java/com/pongr/GuiceConfig.java where we connect Jersey to Guice. This is where we create the Guice Injector using our ExampleModule and configure any Jersey properties, like the package(s) that contain our resource classes.
package com.pongr;
import java.util.*;
import com.google.inject.*;
import com.google.inject.servlet.*;
import com.sun.jersey.api.core.*;
import com.sun.jersey.guice.spi.container.servlet.*;
public class GuiceConfig extends GuiceServletContextListener
{
@Override
protected Injector getInjector()
{
final Map<String, String> params = new HashMap<String, String>();
params.put(PackagesResourceConfig.PROPERTY_PACKAGES, "com.pongr");
return Guice.createInjector(new ExampleModule(), new ServletModule()
{
@Override
protected void configureServlets()
{
serve("/*").with(GuiceContainer.class, params);
}
});
}
}
Now for the fun! This src/main/java/com/pongr/JavaResource.java file gets the message String injected and displays it.
package com.pongr;
import javax.ws.rs.*;
import javax.ws.rs.core.*;
import com.google.inject.*;
import com.google.inject.name.*;
@Path("java")
public class JavaResource
{
private String message;
@Inject
public JavaResource(@Named("message") String message)
{
this.message = message;
}
@GET
@Produces(MediaType.TEXT_PLAIN)
public String get()
{
return "From Java: " + message;
}
}
And for good measure here’s src/main/scala/com/pongr/ScalaResource.scala. Same as JavaResource, except in Scala. The Guice @Inject and @Named annotations can be a bit tricky for constructor injection, so here’s how it’s done.
package com.pongr
import javax.ws.rs._
import javax.ws.rs.core._
import com.google.inject._
import com.google.inject.name._
@Path("scala")
class ScalaResource @Inject() (@Named("message") message: String) {
@GET
@Produces(Array(MediaType.TEXT_PLAIN))
def get(): String = "From Scala: " + message
}
That’s it! Now just run mvn jetty:run on your command line and hit http://localhost:8080/java and http://localhost:8080/scala in your browser to see these resources in action. As always, you can also visit http://localhost:8080/application.wadl to get an overview of the available resources. Enjoy!
Dynamically Generating Zip Files in Jersey
We often need to pull a large number of rows from a database table, split those rows up into n groups, and write each group out to a separate text file. These text files are then processed by another application. Each text file starts with the number of rows in the file on the first line, and then contains each row on its own line after that. It became a pain to generate these files by hand, so I added a new resource to our Jersey-based web service that would generate all of the files and wrap them all up into a single .zip file. The processing app also used to be run by hand, but it’s now totally automated, so it obtains the .zip file, unzips it, and processes each individual file.
Here is the resource we ended up with, except for this blog I’m just generating a list of 100 random strings instead of pulling rows from a real database:
import java.io.*;
import java.util.*;
import java.util.zip.*;
import javax.ws.rs.*;
@Path("/files")
public class FilesResource
{
@GET
@Produces("application/x-zip-compressed")
public InputStream getZipFile(@QueryParam("per_file") @DefaultValue("25") final int perFile)
throws IOException
{
//we write to the PipedOutputStream
//that data is then available in the PipedInputStream which we return
final PipedOutputStream sink = new PipedOutputStream();
PipedInputStream source = new PipedInputStream(sink);
//apparently we need to write to the PipedOutputStream in a separate thread
Runnable runnable = new Runnable()
{
public void run()
{
//PrintStream => BufferedOutputStream => ZipOutputStream => PipedOutputStream
ZipOutputStream zip = new ZipOutputStream(sink);
PrintStream writer = new PrintStream(new BufferedOutputStream(zip));
try
{
//break the strings up into multiple files
List<String> strings = getStrings();
int stringCount = strings.size();
int fileCount = (int) Math.ceil((double) stringCount / (double) perFile);
for (int file = 0; file < fileCount; file++)
{
zip.putNextEntry(new ZipEntry("file" + (file + 1) + ".txt"));
int first = file * perFile;
int last = Math.min((file + 1) * perFile, stringCount);
int imagesInFile = last - first;
writer.println(imagesInFile);
for (int i = first; i < last; i++)
writer.println(strings.get(i));
writer.flush();
zip.closeEntry();
}
//also include a single file with all strings
writer.println(stringCount);
zip.putNextEntry(new ZipEntry("file.txt"));
for (int i = 0; i < stringCount; i++)
writer.println(strings.get(i));
writer.flush();
zip.closeEntry();
}
catch (IOException e)
{
}
writer.flush();
writer.close();
}
};
Thread writerThread = new Thread(runnable, "FileGenerator");
writerThread.start();
return source;
}
private List<String> getStrings()
{
List<String> strings = new ArrayList<String>();
for (int i = 0; i < 100; i++)
strings.add(String.valueOf(Math.random()));
return strings;
}
}
The getZipFile() method will be called in response to a GET /files request. I also registered the .zip URI extension so it will also respond to GET /files.zip. By default it will split up the strings into groups of 25, but the per_file query parameter can be specified in the request to change that, like GET /files.zip?per_file=50 to split them into groups of 50.
The easiest way to create a .zip file in Java is using ZipOutputStream. Once you have that created you call its putNextEntry() method to start a new file within the .zip file. We also wrapped the ZipOutputStream in a PrintStream, and write the contents of the text files by calling println() on the PrintStream (there’s also a BufferedOutputStream in there for good measure).
Jersey is smart enough to read the data from an InputStream and use it as the HTTP response body, so ultimately we need to return an InputStream. By hooking the ZipOutputStream up to a PipedOutputStream, and then connecting the PipedOutputStream to a PipedInputStream, Jersey can read this all of this zip file data from the PipedInputStream. It’s best to write-to and read-from these piped streams in different threads so we start a new writing thread and return the PipedInputStream right away.
This ends up working perfectly: the resource at http://site.com/files.zip feels like a static zip file, but is really generated dynamically on every request. It also can now be accessed overy HTTP from any machine and processed automatically. If it was really expensive to generate we could cache it and re-generate only when one of the rows in the database changed, but for our purposes it’s a cheap .zip file to generate.
iPhone + Gmail = You Need GPush
GPush is an awesome new iPhone app by Tiverias Apps that notifies you when you receive an email to your Gmail account. Its first few days were a bit sketchy but it seems to be working great now. I have Gmail open in Firefox all day and GPush pretty much always notifies me of a new email before I see the little (1) on the Gmail tab.
Entrepreneurs and product managers everywhere should take note of this product. First off, the problem they tackled meets the three important criteria:
- Pervasive: everyone with an iPhone and a Gmail account has to wait for the iPhone to poll Gmail for new emails, every 15 mins, 30 mins, or 1 hour depending on settings (while their friends with BlackBerries get emails instantly)
- Urgent: that market has had the problem since the iPhone was originally released
- Willing to pay: FWIW I’m part of this market and was more than happy to pay $0.99 for a solution to this problem
Their solution is just about perfect. It’s incredibly simple: you install the app, give it your Gmail username & password, and that’s it. And as long as they keep the service working properly it works great: you’re notified instantly when you get a new email to your Gmail address.
I know there have been a lot of unhappy GPush users so far, but it’s only been out for about a week. I’m willing to cut them some slack at this point, and I’ve seen an incredible improvement over the last few days. If GPush turns out to be a big fail long-term I’ll be happy to criticize it, but at this point I think Tiverias has done an awesome job!
Example of Implicit Conversions, Options, Extractors and Tuples
While writing a new web service resource today I had the chance to use some pretty awesome parts of Scala, and wanted to share some of them. We integrate with Twitter and have a POJO like this TwitterUser class:
package com.pongr.twitter;
public class TwitterUser
{
private String name;
public TwitterUser()
{
this(null);
}
public TwitterUser(String name)
{
setName(name);
}
public String getName()
{
return name;
}
public void setName(String name)
{
this.name = name;
}
}
It’s been stripped down to only the relevant property, in this case the user’s full name that they specified on their Twitter account. This new resource sends back various information about a user to our mobile apps, including their Twitter info. Since I wrote this new resource in Scala I decided to “scalafy” (scalify?) the TwitterUser class a bit.
First off, one of our users may not have linked to their Twitter account, so the TwitterUser object might be null. And even if they have, the name we get back from the Twitter API will be null if they haven’t entered a name, or may just be a duplicate of their screen name (like @zcox). We need to split up the name into valid first and last names, so this seemed like a good place to be using an Option[String] instead of just a String, which is a best practice from the Beginning Scala book. So the TwitterInfo class ended up looking like this:
package com.pongr.twitter
class TwitterInfo(user: Option[TwitterUser]) {
val NameRegex = """(.*?)\s+(.*)""".r
val name = user.flatMap(_.getName match {
case NameRegex(first, last) => Some((first, last))
case _ => None
})
val firstName = name.map(_._1)
val lastName = name.map(_._2)
}
object TwitterInfo {
implicit def twitterUser2TwitterInfo(user: TwitterUser) = new TwitterInfo(if (user == null) None else Some(user))
}
Another thing to note is the use of a regular expression as an extractor to split up the Twitter name into separate first and last names. I picked up this technique from the Programming in Scala book and I have to say that it makes the code incredibly intuitive. I also made the name field a tuple to reuse this regular expression extraction: the firstName and lastName fields are then initialized to the first and second parts of it, respectively.
Implicit conversions continue to amaze me so I created the twitterUser2TwitterInfo method that converts a TwitterUser object into a TwitterInfo. Putting all of this together, we can use this TwitterInfo class to safely obtain the user’s Twitter name:
package com.pongr.twitter
import TwitterInfo._
object App {
def main(args : Array[String]) : Unit = {
print(new TwitterUser)
print(new TwitterUser("asdf"))
print(new TwitterUser("John Smith"))
}
def print(user: TwitterUser) = println(user.lastName.getOrElse("N/A") + ", " + user.firstName.getOrElse("N/A"))
}
The implicit conversion lets us (appear to) call TwitterInfo methods right on the TwitterUser object, and since the firstName and lastName fields are each an Option[String] we can gracefully handle the case where we don’t have the Twitter name. Running the above test app prints out the following:
N/A, N/A N/A, N/A Smith, John
I know this is a pretty simple example, but I thought it was a practical demonstration of some of the techniques that make Scala such a fun language to program with.
Simple JDBC Queries in Scala
The Beginning Scala book has a great example of using partially applied functions to automatically close JDBC connections. Today I needed to use some complex SQL outside of our ORM and extended this code sample to make it incredibly simple & safe.
The using and bmap methods are from the book; the query and queryEach methods are my creations:
object Control {
def using[Closeable <: {def close(): Unit}, B](closeable: Closeable)(getB: Closeable => B): B =
try {
getB(closeable)
} finally {
closeable.close()
}
import scala.collection.mutable.ListBuffer
def bmap[T](test: => Boolean)(block: => T): List[T] = {
val ret = new ListBuffer[T]
while(test) ret += block
ret.toList
}
import java.sql._
/** Executes the SQL and processes the result set using the specified function. */
def query[B](connection: Connection, sql: String)(process: ResultSet => B): B =
using (connection) { connection =>
using (connection.createStatement) { statement =>
using (statement.executeQuery(sql)) { results =>
process(results)
}
}
}
/** Executes the SQL and uses the process function to convert each row into a T. */
def queryEach[T](connection: Connection, sql: String)(process: ResultSet => T): List[T] =
query(connection, sql) { results =>
bmap(results.next) {
process(results)
}
}
}
The using method just ensures that something with a close() method gets closed after it’s used, while bmap collects the results of some function into a list. I found the using method resulted in a fair amount of boilerplate code, so I factored that out into the query method. You just give it a Connection object and an SQL string and it will handle creating the Statement and ResultSet, passing the ResultSet to your custom processing function, and then closing everything safely for you.
The queryEach method also eliminates some common boilerplate. It builds on the query method by using each row in the ResultSet to create some domain object and collecting all of those objects into a list. As the example usage below shows, you just give queryEach your connection and SQL, as well as some simple code to process a single row in the ResultSet:
import com.whatever.Control._
import java.sql._
val conn: Connection = ...
val people = queryEach(conn, "SELECT * FROM person") {rs =>
new Person(rs.getString("name"), rs.getInt("age"), rs.getBoolean("valid"))
}
I think this results in some very compact and intuitive code. And reusing all of the boilerplate in the Control methods ensures that you don’t forget to close a Statement or ResultSet.
URI Extensions in Jersey
Jersey provides an excellent system for easily supporting multiple representations of resources. You use the @Produces annotation to define which formats a resource method supports:
@GET
@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
public Thing getThing()
{
Thing thing = ...;
return thing;
}
In this case clients can obtain the representation of the thing resource in either XML or JSON format (assuming you’ve registered corresponding MessageBodyWriters for the Thing class). Just by including the @Produces annotation, Jersey handles sending the response back in the proper format and will even send a 415 Unsupported Media Type response automatically if the client requests the resource in an unsupported format.
By default Jersey uses the Accept request header to determine which response format to use. Setting request headers in client apps can be cumbersome though, so a more convenient approach is to use URI extensions. So if you want the http://site.com/thing resource as XML you GET http://site.com/thing.xml, or GET http://site.com/thing.json if you want its JSON representation.
To tell Jersey to use URI extensions, you create a cusom PackagesResourceConfig implementation and set up the mappings between extensions and media types:
import java.util.*;
import javax.ws.rs.core.*;
import com.sun.jersey.api.core.*;
/** Registers URI extensions for some common media types. This lets clients specify the desired
* response format right in the URI like http://site.com/whatever.xml instead of
* http://site.com/whatever with an Accept:application/xml header. */
public class UriExtensionsConfig extends PackagesResourceConfig
{
private Map<String, MediaType> mediaTypeMap;
public UriExtensionsConfig()
{
super();
}
public UriExtensionsConfig(Map<String, Object> props)
{
super(props);
}
public UriExtensionsConfig(String[] paths)
{
super(paths);
}
@Override
public Map<String, MediaType> getMediaTypeMappings()
{
if (mediaTypeMap == null)
{
mediaTypeMap = new HashMap<String, MediaType>();
mediaTypeMap.put("json", MediaType.APPLICATION_JSON_TYPE);
mediaTypeMap.put("xml", MediaType.APPLICATION_XML_TYPE);
mediaTypeMap.put("txt", MediaType.TEXT_PLAIN_TYPE);
mediaTypeMap.put("html", MediaType.TEXT_HTML_TYPE);
mediaTypeMap.put("xhtml", MediaType.APPLICATION_XHTML_XML_TYPE);
MediaType jpeg = new MediaType("image", "jpeg");
mediaTypeMap.put("jpg", jpeg);
mediaTypeMap.put("jpeg", jpeg);
mediaTypeMap.put("zip", new MediaType("application", "x-zip-compressed"));
}
return mediaTypeMap;
}
}
You register the UriExtensionsConfig in web.xml like this:
<servlet>
...
<init-param>
<param-name>com.sun.jersey.config.property.resourceConfigClass</param-name>
<param-value>com.pongr.rest.config.UriExtensionsConfig</param-value>
</init-param>
</servlet>
This provides yet another way to make your web services more user-friendly. Clients can still use the Accept header but can also use standard URI extensions to specify the desired response format.
Same Snippet, Different Designs
You’re building a web site and the designers send you XHTML templates with essentially the same form on two different pages, but each page uses a slightly different layout. Being the good coder you are, you want to reuse the same submit-handling code on the back-end to prevent code duplication. This is easy to handle using Lift snippets.
The form lets a user add a new restaurant to our restaurant review site. So we create a snippet like this (it just prints the submitted data for now):
import scala.xml._
import net.liftweb.util.Helpers._
import net.liftweb.http._
class CreateRestaurant {
def render(xhtml: NodeSeq): NodeSeq = {
var name = ""
var address = ""
var phone = ""
def createRestaurant() = println("Created " + name + ", " + address + ", " + phone)
bind("form", xhtml, "name" -> SHtml.text(name, name = _),
"address" -> SHtml.text(address, address = _),
"phone" -> SHtml.text(phone, phone = _),
"submit" -> SHtml.submit("Add", createRestaurant))
}
}
Next we can create two different forms in our XHTML templates and bind them to the same CreateRestaurant snippet. We are free to modify the design inside the snippet, as long as we bind the same form elements in each one.
Here’s the first form template:
<lift:CreateRestaurant form="POST">
<h3>Add a new restaurant</h3>
<table>
<tr><td>Name</td><td><form:name><input type="text"/></form:name></td></tr>
<tr><td>Address</td><td><form:address><input type="text"/></form:address></td></tr>
<tr><td>Phone</td><td><form:phone><input type="text"/></form:phone></td></tr>
<tr><td></td><td><form:submit><input type="submit"/></form:submit></td></tr>
</table>
</lift:CreateRestaurant>
And here’s the second form template:
<lift:CreateRestaurant form="POST">
<table>
<tr>
<th>Name</th>
<th>Address</th>
<th>Phone</th>
<th></th>
</tr>
<tr>
<td><form:name><input type="text"/></form:name></td>
<td><form:address><input type="text"/></form:address></td>
<td><form:phone><input type="text"/></form:phone></td>
<td><form:submit><input type="submit"/></form:submit></td>
</tr>
</table>
</lift:CreateRestaurant>
Same form, just different layouts: the first has the fields arranged horizontally while the second lays them out vertically. When Lift renders them they both look different but behave exactly the same way on the back-end, because they both reuse the same snippet code.
No more need to duplicate back-end form submit handling code!
