Character Networks with Go and d3

I've had the opportunity to create a practical application in Go. I'm working on a research project in grad school that involves constructing character networks from literature text. After In order to get a grasp on what we might be able to "say" about networks in the corpus.

After pre-processing the text with Stanford's Named Entity Recognizer, I then searched the text for people, and began constructing the relationships. Details on the methods will be upcoming. First I wrote the processing code in R, and it took around 10 seconds to run. After rewriting in Go, the processing takes less than one second. I don't think it should be a surprise that it performs better. The important part is that this allows me to create a tool that will update parameters of the algorithm and update the network for display in real time.

Right now the display is being done in using D3's force layout. I think it should work out, but there are some kinks that I need to resolve first.

More to come.

WAID server

Bare Bones Martini App

Its pretty easy to add handlers to routes. And because the Handlers are simply functions of a specific type, it is easy to separate these into different files. Martini also has some helpers that make it trivial to deal with basic stuff like json request, and parameters. Below are a couple of route examples.

I'm using gorp to handle my database connections

// index route
m.Get("/entries", func(r render.Render) {
 r.JSON(200, entry.All(dbMap))
})

// create route
m.Post("/entries", binding.Json(entry.Entry{}), func(params martini.Params, e entry.Entry, r render.Render) {
 err := dbMap.Insert(&e)
 if err != nil {
  r.JSON(404, "Unable to update entry.")
  return
 }
 r.JSON(200, e)
})

Adding routes for other operations is pretty easy once the first couple are working.

Creating arbitrary HTTP request types.

Go's http library has some built in methods for sending GET and POST requests, but not for DELETE, PUT, or PATCH. I added a simple helper function that creates and sends a request of arbitrary type, then stuffs the response data into a passed interface

func jsonRequest(reqType string, path string, v interface{}) {
 var err error
 client := http.Client{}

 //prepare json data to send to the server
 jsonData, err := json.Marshal(v)
 doPanic(err)
 body := bytes.NewBuffer(jsonData)

 // create req uest
 req, err := http.NewRequest(reqType, fmt.Sprintf("%s%s", SERVER_URL, path), body)
 doPanic(err)
 resp, err := client.Do(req)
 doPanic(err)

 // error status code
 if resp.StatusCode >= 400 {
  panic(fmt.Sprintf("Request %d: %s:", resp.StatusCode, resp.Status))
 }

 // set returned "1data to interface, this will crash if the types aren't good
 entryData, err := ioutil.ReadAll(resp.Body)
 doPanic(err)

 // construct json data as requested struct
 json.Unmarshal(entryData, &v)
}

Go's json marshal and unmarshall made this very easy. A flaw I notice right now, is that the method assume that you'll want the submitted object to be modified by the server, this doesn't apply to DELETE requests (they should send back an empty response). But really this is still an exercise in learning Go for me.

Handing JSON payload with Martini

Initially I was trying to use Martini's form handler for the requests, but it was easier to write the client code for requests with a json payload. Luckily Martini also can handle JSON just as easily.

The net affect is that the client is much simpler, and separating the data handling and the interface has made adding new features easier.

Future Work

The database configurations is pretty funny. It's just a sqlit3 file in the user's home directory. I'd like to make that slightly more sophisticated. I'd also like to add some config files so that the user can, you know, configure that application without recompile. After those goals have been met. A deployment scheme is next.