He's gotten much larger since then, but here's a picture of Tucker and Olivia .

I'm not sure exactly how this ended up on the internet, but the address is correct and that looks my messy scribble handwriting. I have no idea where the rose came from.

http://www.gdao.org/items/show/811999

Wow. The California DMV website is like a bad joke. "Hey, since we can't make you wait in line at the DMV, we'll build a really slow, sucky, unreliable web site and you can wait trying to use that!"

So OmniPlan is a pretty nice project planning tool. Only problem is that its charts are pretty ugly and rather difficult to customize. Timeline3D from BeeDocs looks pretty nice, only its UI for entering what they call rows and events is pretty awful. Ok, I'll just take my data from OmniPlan and import it into Timeline3D. Wait, you can't do that. What? Hmm... Looks like Timeline3D allows you to import from OmniFocus. OmniPlan allows you to export to OmniOutliner and OmniGraffle. What about OmniFocus? Or, better yet, how about OmniPlan support in Timeline3D? Think about it guys...

So in the last installment, we saw a few problems with ABCL, maven and libraries to be supplied by maven. I've tracked a few of these things down, learned a few things, and released a trivial new library.

Maven 3.0.3 vs 3.0.4

It turns out I had maven 3.0.3 installed. I'm not sure where this came from. XCode perhaps? In any event, the ABCL maven stuff requires version 3.0.3 or later, so I was OK there, but it depends on some features that are only found in 3.0.4 (some HttpWagon or something or other).

Removing the 3.0.3 maven and installing homebrew's maven 3.0.4 fixes this problem. If there's an easy way to make the ABCL maven-embedder stuff work with 3.0.3 or 3.0.4, that would be nice.

Other Remote Repositories

I'm still relying on the freehep 2d graphics libraries and these aren't in maven central, but rather in the freehep maven repo. How can we tell the ABCL maven stuff to search this repository? There may be a way, but if so I haven't found it yet.

Using Sharpsign-quote

It turns out one can do (#"foo" ...) instead of (java:jcall "foo" ...), so I've switched over my code to this style.

ABCL-CDK

It's a farily trivial package at this point, but I've released abcl-cdk which provides some examples of calling the CDK from ABCL.

in which I attempt to write some Common Lisp code to be run in a Common Lisp environment that runs inside a virtual machine designed to support a C-like language that incorporated a few lispy features, so that I can use a library written in said C-like language with my Common Lisp code, or something like that.

Ok, it's time to see if I can get the CDK and ABCL playing nicely together.

CDK

The CDK (Chemistry Development Kit) is java library for dealing with various type of chemistry data, elements, atoms, bonds, molecules, etc... and various computed or measured properties thereof. I should point out that the CDK isn't really just one library, but rather a family of various related libraries. We'll come back to building an appropirate version of CDK in a moment, but, for now, let's move on.

ABCL

ABCL is an implementation of the Common Lisp programming language that runs on the JVM. Besides running (in theory) on any platform that supports the JVM, ABCL provides for relatively smooth interoperability with other code (such as Java libraries) that run on the JVM.

Building CDK

First, we need the CDK. Some of the main things I want to do with the CDK are to instantiate a molecule from a SMILES string, get a 2D representation of the molecule, and compute various properties (molecular weight, charge, etc...) of the molecule. The only problem with that is the main CDK doesn't actually support 2D rendering. Before we get into how to get a CDK that does 2D rendering, I should take this opportunity to gripe about the various versions of the CDK for a moment.

Sourceforge's CDKs

One of the things that bothers me about sourceforge-hosted projects is that there are often too many "home pages" for a project. For the CDK we have two:

http://sourceforge.net/projects/cdk/

and

http://cdk.sf.net, which in turn redirects to:

http://sourceforge.net/apps/mediawiki/cdk/index.php?title=Main_Page

Ok, so there's a bunch of info on the sf.net page and the CDK Development with Git page is kind enough to point us over to github:

 $ git clone git://github.com/cdk/cdk.git 

which, of course implies that there is something of a home/project page over at https://github.com/cdk/cdk. And, sure enough, there is.

From there we can see that github's cdk/cdk project is actually a fork of Egon Willighagen's cdk git repository.

JChemPaint

Of course none of these (at least on first glance) contain the 2D rendering code we want. It turns out that's not part of the core CDK, but rather part of the JChemPaint code. The JChemPaint project is another effort, closely related to CDK, that has applets/applications for interactive 2D molecule editing, 2D structure rendering code, etc... So, on the JChemPaint page we see links to various downloads where we have CDK, JChemPaint, CDK-JChemPaint, etc...

Wait, what? CDK-JChemPaint? Hang on a second! We'll come back to that in a moment. First we see that the CDK code is moving ahead rapidly but that the JChemPaint is from September 2011 and the JChemPaint (development) code is from November 2010! Hmm...

So, near as I can tell, JChemPaint was a separate, but related-to-CDK project and at some point somebody cribbed some of the reusable bits from JChemPaint and put them into CDK-JChemPaint.

But then it seems like maintaining a separate CDK-JChemPaint seemed a bit silly and egonw (?) has been maintaining a branch of the CDK with some of the JChemPaint (or is it CDK-JChemPaint?) functionality incorporated: https://github.com/egonw/cdk/tree/13-unsorted-patches. This is what I orginally used for the 2D rendering code. It turns out that there is a newer, better (?) version of the CDK with the appropriate JChemPaint bits added, the 381-14x-renderextra branch.

Back to building CDK...

First we get the code

git clone git://github.com/cdk/cdk.git 

Then we need to pull from egonw's branch (I suppose we could have just cloned this first):

git remote add egonw git://github.com/egonw/cdk.git  
git pull egonw 

And now let's checkout the branch we want:

git checkout 381-14x-renderextra 

Ok, now we've got the code. We build it with ant:

ant 

Assuming we have java properly setup, things should build fine. Now we have a brazillion jar files in cdk/dist/jar. Wait, that's not what we want. We want a single CDK jar that we can (presumaly) point our CLASSPATH to, or at least do whatever the ABCL equivalent is. Turns out there's a "dist-large" target in the CDK build.xml file so we can build that with:

ant dist-large 

Ok, now we have dist/jar/cdk-1.4.8.git.jar.

Installing CDK

So what are we supposed to do with that? Well, it appears that some folks in the Java world use this thing called maven for both remote and local package fetching/deployment/whatever-you-call-it-in-the-java-world.

So, assuming we have maven around, we can install a CDK which we can later, hopefully, use with ABCL with the following:

export CDK_VERSION=1.4.8-SNAPSHOT  
export CDK_BUILD_VERSION=1.4.8.git  
mvn install:install-file -DgroupId=org.openscience.cdk -DartifactId=cdk \  
    -Dversion=${CDK_VERSION} -Dpackaging=jar \  
    -Dfile=dist/jar/cdk-${CDK_BUILD_VERSION}.jar 

Notice that we need two distinct version identifiers as maven wants nice clean version numbers (and doesn't really like the 1.4.8.git version) and most maven-ized projects seem to use the SNAPSHOT suffix for in-progress releases. On the other hand, the CDK build.props file sets the version to 1.4.8.git. We use the two identifiers here so that cdk-1.4.8.git.jar gits installed as org.openscience.cdk/cdk version 1.4.8-SNAPSHOT.

Fetching Java Dependency Libraries

At this point I should point out that I'm not exactly a big maven fan. It's no quicklisp. But there must be a reason why folks in the Java world use it. Let's see what it takes to download some more dependencies (presumably other jar files we're going to use later). So, we fire off some queries on our favorite search engine for, say, "maven fetch", and we see things like http://stackoverflow.com/questions/1895492/how-can-i-download-a-specific-maven-artifact-in-one-command-line and http://stackoverflow.com/questions/4568633/use-maven-just-to-fetch-some-library-jars. Oh, man. I just want to download some jars and now I'm being told to use Ivy (whatever that is) or some crazy maven plugin where all I need to do is edit my ~/.m2/settings.xml and a ~/.m2/plugin-registry.xml file? No thanks!

(Note: I think there's some built-in functionality in ABCL to handle this next task -- but I couldn't get it to work!)

Fortunately, the clojure folks, who occasionally drink a little too much Java toolchain (tooling?) Kool-aid for my taste, but at least have enough taste to want a lisp-ish language, have gotten here first and the standard tool for these kinds of jobs seems to be Phil Hagelberg's leiningen. I'm going to assume for the moment that you actually have leningen lying around, or that you're smart enough to figure out some other way to get these dependencies installed if not.

So, to trick leiningen into doing some dirty work for us, we make a project.clj that looks as follows:

(defproject abcl-cdk-hacking "0.0.0"  
  :description "Fake project for fetching abcl-cdk-hacking dependencies"  
  :dependencies [[org.freehep/freehep-graphics2d "2.1.1"]  
                 [org.freehep/freehep-graphicsio-pdf "2.1.1"]  
                 [org.freehep/freehep-graphicsio-svg "2.1.1"]]  
  :repositories {"freehep" "http://java.freehep.org/maven2"}) 

Once we have this we can do:

lein deps 

which will install the dependencies for us somewhere in ~/.m2 (let's forget about system-wide installs for the moment).

Using Java Dependency Libraries

Ok, we should be ready to figure out how to make ABCL talk to CDK now. First we just have to figure out how to make ABCL talk to CDK. Wait, wasn't that what I just said? Yes, but, how do we do it? Fortunately, the ABCL guys anticpated this problem and added what they call abcl-asdf. By doing a (require 'abcl-asdf) (oh wait, and a (require 'abcl-contrib) before that, I think), we can tell our ASDF system how to tell ABCL to tell the JVM where to find the jars we need put on the CLASSPATH, or something like that.

(eval-when (:compile-toplevel :load-toplevel :execute)  
  (cl:require 'abcl-contrib)  
  (cl:require 'abcl-asdf))  
 
(asdf:defsystem :abcl-cdk-hacking  
  :name "abcl-cdk-hacking"  
  :author "Cyrus Harmon"  
  :serial t  
  :default-component-class asdf:cl-source-file  
  :components  
  ((:mvn "org.freehep/freehep-graphics2d" :version "2.1.1")  
   (:mvn "org.freehep/freehep-graphicsio-pdf" :version "2.1.1")  
   (:mvn "org.freehep/freehep-graphicsio-svg" :version "2.1.1")  
   (:mvn "org.openscience/cdk" :version "1.4.8-SNAPSHOT")  
   (:file "abcl-cdk-hacking"))) 

We can add :mvn components to our ASDF system and the abcl-asdf machinery will add the maven artifact (?) or jar file or whatever to the CLASSPATH, or at least somehow make it so the classes are available to the JVM.

Well, that's the theory anyway. In practice this doesn't work with a stock ABCL because of the following bug: http://trac.common-lisp.net/armedbear/ticket/204. Once this is fixed (via the patch attached to the bug report), and ABCL rebuilt, a simple:

(asdf:load-system 'abcl-cdk-hacking) 

will load the dependencies into the JVM and we should be off and running, finally.

Calling Static Java Methods

Ok, now we need to do some Java interop stuff with CDK. First thing we want to do is call a static Java method.

We're going to need an instance of the org.openscience.cdk.DefaultChemObjectBuilder class. We can get this via the static getInstance method as follows:

(defparameter *dcob*  
  (java:jcall  
   (java:jmethod (java:jclass "org.openscience.cdk.DefaultChemObjectBuilder")  
                 "getInstance")  
   nil)) 

So, we have the java:jclass function to lookup a class, the java:jmethod function to lookup a method and the java:jcall function to invoke the method. So far so good.

Creating Java Objects

Now we're going to need to create a Java object. Turns out we can do that with the java:jnew function:

(defparameter *smiles-parser*  
  (java:jnew "org.openscience.cdk.smiles.SmilesParser" *dcob*)) 

This gives us a new instance of the org.openscience.cdk.smiles.SmilesParser class.

Calling Methods on Java Objects

Finally, we can call a java method with java:jcall, as we do with the parseSmiles method here:

(defparameter *caffeine*  
  (java:jcall "parseSmiles" *smiles-parser* "CN1C=NC2=C1C(=O)N(C(=O)N2C)C")) 

Java Class Identifiers

Well, the clojure folks have figured out that some people, at least, hate typing long java class names all over the place, and the ABCL java interop stuff seems to require lots of typing of long java names. In a perhaps misguided attempt to relieve this burden and provide something more like clojure's syntax, I present the jimport macro:

(defmacro jimport (java-package class &optional package)  
  `(defparameter ,(apply #'intern class  
                         (when package (list package)))  
     (concatenate 'string (symbol-name (quote ,java-package))  
                  "."  
                  (symbol-name (quote ,class))))) 

This macro allows one to do:

(jimport |org.openscience.cdk| |DefaultChemObjectBuilder|) 

which then defines the value of the |DefaultChemObjectBuilder| symbol (in the current packaage, at least if not specified in the jimport call) to be "org.openscience.cdk.DefaultChemObjectBuilder", so now we can do:

(java:jcall  
 (java:jmethod  
  (java:jclass |DefaultChemObjectBuilder|) "getInstance")  
 nil) 

Not a huge win, but it does allow the compiler to ensure that we're seeing identified symbols, rather than just potentially random strings for Java classes.

Java List<Foo>'s

One of the CDK classes, org.openscience.cdk.renderer.AtomContainerRenderer, has a constructor that expects a List<IGenerator<IAtomContainer>> as one of its arguments. How do we invoke the constructor with one of those? Well, it turns out we can't just use a lisp list as the argument. We have to make a java List of some sort. It turns out there's some infrastructure provided by ABCL to help with this, although nothing I can find that does exactly what I need. The extensible-sequence stuff allows us to make a lisp sequence that is actually some sort of instance of the java.util.List interface. I use a java.util.Vector and provide a helper function called jlist as follows:

(defun jlist (&rest initial-contents)  
  (sequence:make-sequence-like  
   (java:jnew |Vector|) (length initial-contents)  
   :initial-contents initial-contents)) 

So, now we've got a way to create java lists that we can pass on to the constructor.

Getting a java stream from a lisp stream

One final bit of consternation, we'd like to be able to create streams using the sane lisp syntax like:

(with-open-file (out-stream pathname :direction :output  
                                       :if-exists :supersede  
                                       :element-type :default)  
  ...) 

but then use the corresponding streams where we need java streams. In particular the freehep SVG and PDF libraries want java streams for files. It turns out there's a function to get the java output stream associated with a lisp stream, getWrappedOutputStream. We use that to get the java.io.Stream or whatever and we're good to go.

Now we can define our mol-to-svg function as follows:

(defun mol-to-svg (mol pathname)  
  (with-open-file (out-stream pathname :direction :output  
                                       :if-exists :supersede  
                                       :element-type :default)  
    (let*  
        ((r (java:jnew |AtomContainerRenderer|  
                       (jlist  
                        (java:jnew |BasicAtomGenerator|)  
                        (java:jnew |BasicBondGenerator|)  
                        (java:jnew |BasicSceneGenerator|))  
                       (java:jnew |AWTFontManager|)))  
         (vg (java:jnew |SVGGraphics2D|  
                        (java:jcall "getWrappedOutputStream" out-stream)  
                        (java:jnew |Dimension| 320 320)))  
         (adv (java:jnew |AWTDrawVisitor| vg)))  
      (java:jcall "startExport" vg)  
      (java:jcall "generateCoordinates"  
                  (java:jnew |StructureDiagramGenerator| mol))  
      (java:jcall "setup" r mol (java:jnew |Rectangle| 0 0 100 100))  
      (java:jcall "paint" r mol adv  
                  (java:jnew (java:jconstructor |Rectangle2D$Double| 4)  
                             10 10 300 300)  
                  java:+true+)  
      (java:jcall "endExport" vg)))) 

Finally, we can render our molecule of choice, caffeine to an SVG file thusly:

(mol-to-svg *caffeine* "/tmp/caffeine.svg") 

And we see:

Voila!

Next time hopefully we can explore integrating chemicl and CDK directly with ABCL, but I think that requires fixing an ABCL bug that prevents it from successfully compiling plexxipus-xpath.

In anticipation of adding icalendar/xCal support to cl-vcard, I've renamed the project to soiree. It can be now found at https://github.com/slyrus/soiree.

Ever since taking Stuart Russell's Knowledge Representation and Reasoning class (Holy smokes, that was my first real introduction to Common Lisp and that was eleven years ago! How time flies...), I've always felt like I don't really have my data unless I have it in some machine readable form -- and not just in an unstructured text document or in some proprietary GUI application, but rather in a place where I can reasonably query, update, search, etc... the data.

So, one kind of data that I haven't really had is my address book. For a while I kept things locally in Apple's Address Book application, but that really only worked for a single machine, at least at first. At some point I discovered the awesomeness of DAViCal, which is a CALDAV/CARDDAV server for serving up calendar and contact/address book information. Great! OK, so, I jumped through the requisite hoops to get things working between DAViCal and Address Book (and iCal) and all was good -- except for the fact that I still didn't have a nice convenient API for searching/querying/retrieving/updating the data in DAViCal.

So, that motivated me to write some code for working with my data from Address Book and from DAViCal, which leads me to this blog post. It turns out that there is an IETF standard for exchanging contact information, the VCARD format. Apparently I can export and import data from Address Book in this form, and this is how the CardDAV data is stored in DAViCal, I think. Great. Now all I need to do is to parse the VCARD data and I'm good to go. Fortunately, the spec lays out the file format nicely and this shouldn't be too terribly hard to parse.

• Data Model?

But, that leads to the next question of, having parsed the data, what I'm I going to do with the data? Or, put another way, how am I going to represent/model the data contained in the VCARD file? Just slurping the VCARD bits into a buffer of characters doesn't help me find, for instance, the email address of a particular person. One approach would be to define a data model with CLOS classes and generic functions that operate on those classes to allow for reading/writing/querying the data. Another approach would be to use a more generic data structure like lists or, probably better yet, nested hash-tables that would allow traversal of a graph of data objects via key/value relationships. The VCARD format itself could be thought of as a (somewhat unwieldy) data model.

• The VCARD Specification

The VCARD 3.0 specification formally describes what VCARD data should look like. The problem is that transferring back and forth either between CLOS objects or hash-tables and VCARD data sounds like a cumbersome and error-prone process, especially as either the objects or hash-tables, on the one hand, or the VCARD format itself evolves over time. Surely there's a better approach.

The other problem with the VCARD specification is that the specification isn't machine readable. Yes, there are bits of ABNF grammars in there, but the spec, as a whole, isn't easily parsed and queried. If one was going to rely on a specification to define the data model, it sure would be nice if the specification itself could be read, interrogated, and used by our programs.

• xCard: VCARD XML Representation

Enter the xCard specification. The xCard specification does two things, it provides a model for reading and writing vCard data (that is the data that can be represented in VCARDs, not the actual VCARD syntax) and, more importantly, it provides a machine-readable representation of the specification in the form of a RELAX NG schema.

• RELAX NG

RELAX NG itself is an XML schema for describing XML schemas. It provides for a human-friendly but machine-readable format for specifying RELAX NG schemas, the RELAX NG Compact Syntax.

• Back to xCard

So, we find the compact syntax for the xCard schema at the end of the standard. For whatever reason, I can't locate a canonical version of the schema outside of the spec, but it's simple enough to cut and paste the bits out of the spec and place it in an RNG file. Great. Now we've got data in VCARD format, a nice machine-readable (and human-friendly) description of the data model in the RELAX NG compact syntax. How does this solve the problem of representing/querying/modifying the data that I mentioned above? Well, one approach would be automagically generate CLOS classes and interfaces that match the RELAX NG schema. This seemed like an interesting approach, but an awful lot of work. Since we've got a schema that defines the vCard semantics, as represented by an XML document, perhaps we can just use an in-memory representation of the XML data itself as our "data model" for reading/writing/querying/etc... the address book data. This is the approach I've taken with cl-vcard, and we'll come back to it momentarily.

• Parsing VCARDs parser-combinators

For the moment, before we get into what are we transforming the data to, we need to consider what we're transforming the data from and how to do so. A simple, hand-coded recursive descent parser would probably be the most straightforward way to go, but I'm exceedingly lazy and wanted someone else to do the bulk of the heavy lifting of parsing for me. Enter Jakub Higersberger's awesome parser-combinator library, inspired by Haskell's parsec monadic parser-combinator library.

While it may be overkill, parser-combinator provides for a nice, clean API for writing parsers. The core of the parsing routine is shown below:

(defun content-line? (&optional name)  
  ;; [group "."] name *(";" param) ":" value CRLF  
  (named-seq?  
   (<- group (opt? (hook?  
                    #'first  
                    (seq-list? (group?) "."))))  
   (<- name (if name name (name?)))  
   (<- params (many? (named-seq?  
                      ";"  
                      (<- param (param?))  
                      param)))  
   ":"  
   (<- value (value?))  
   (<- long-lines (many? (long-line-extension?)))  
   (seq-list? #\Return #\Newline)  
   (when name  
     (list group name params (apply #'concatenate 'string value long-lines))))) 

• fset

Parser combinators are designed to work with functional data structures, as it they may backtrack, and modifying data structures as one goes with parser combinators can lead to problems. Therefore, I use fset, a functional collections library, for building the parsed representation of the data as I go. I could probably get away without doing this, but, it seems like a good idea to use a functional parser as such, rather than abusing the idea that we're not taking advantage of backtracking such that we can modify our data structures along the way as we do the parse.

• Putting it all together

Here's an example of using cl-vcard to parse a simple vCard:

(asdf:load-system 'cl-vcard)  
 
(cl:defpackage #:cl-vcard-example  
  (:use #:cl #:cl-vcard))  
 
(cl:in-package #:cl-vcard-example)  
 
(defparameter *baba-oriley-vcard*  
  "BEGIN:VCARD  
VERSION:3.0  
N:O'Riley;Baba;;;  
FN:Baba O'Riley  
ORG:Polydor Records  
TITLE:Field Worker  
PHOTO;VALUE=URL;TYPE=GIF:http://www.example.com/dir_photos/my_photo.gif  
ADR;type=WORK;type=pref:;;Trafalgar Square;London;England;;UK  
TEL;TYPE=WORK,VOICE:(415) 555-1212  
TEL;TYPE=HOME,VOICE:(415) 555-1213  
EMAIL;TYPE=PREF,INTERNET:thewho@example.com  
END:VCARD  
")  
 
(defparameter *baba* (parse-vcard *baba-oriley-vcard*)) 

(note that the vcard string needs CRLF line endings!)

Doing this at the REPL, we see:

CL-VCARD-EXAMPLE> (defparameter *baba* (parse-vcard *baba-oriley-vcard*))  
*BABA*  
CL-VCARD-EXAMPLE> *baba*  
#.(CXML-STP-IMPL::DOCUMENT  
   :CHILDREN '(#.(CXML-STP:ELEMENT  
                  #| :PARENT of type DOCUMENT |#  
                  :CHILDREN '(#.(CXML-STP:ELEMENT  
                                 #| :PARENT of type ELEMENT |#  
                                 :CHILDREN '(#.(CXML-STP:ELEMENT  
                                                #| :PARENT of type ELEMENT |#  
                                                :CHILDREN '(#.(CXML-STP:ELEMENT  
                                                               #| :PARENT of type ELEMENT |#  
                                                               :CHILDREN '(#.(CXML-STP:TEXT  
                                                                              #| :PARENT of type ELEMENT |#  
                                                                              :DATA "O'Riley"))  
                                                               :LOCAL-NAME "surname"  
                                                               :NAMESPACE-URI "urn:ietf:params:xml:ns:vcard-4.0")  
 
... 

So we have a big, hairy document tree that (hopefully) has the vcard data we want in it in a form that will, eventually, prove to be easy for us to work with. But, before we get into actually doing anything with the data, let's take a digression into XML data representation and validation.

• STP, CXML, RNG, CXML-RNG

Notice that we're using stp:serialize. CXML-STP is a document-based interface to XML data, written by David Lichteblau, and somewhat like the DOM, only with a much nicer interface (in my subject opinion, of course), inspired by the XOM. A comparison between the DOM and STP interfaces can be found here.

In the guts of the parser we use functions like stp:make-element and stp:append-child to construct the document tree. These, and the rest of the STP API, sit on top of Gilbert Baumann's Closure XML (or CXML) library.

• What's the damn phone number?

OK, so far this has all been a lot of work and we haven't even gotten to access any of our data. We're almost there. We just need one more XML library (of course...). We could work with the STP document directly:

CL-VCARD-EXAMPLE> (stp:string-value  
 (stp:find-child  
  "text"  
  (stp:find-child  
   "tel"  
   (stp:find-child  
    "vcard"  
    (stp:find-child  
     "vcards" *baba*  
     :key #'stp:local-name :test 'equal)  
    :key #'stp:local-name :test 'equal)  
   :key #'stp:local-name :test 'equal)  
  :key #'stp:local-name :test 'equal))  
"(415) 555-1212" 

But this grows tedious. Fortunately, there's a better way.

• Plexippus-XPath

Plexippus XPath is another excellent library in the CXML family, written by Ivan Shvedunov. Plexippus provides an implementation of the XPath spec that works with CXML documents. Using plexippus instead of walking the tree by hand as above, we can do:

(xpath:with-namespaces ((nil cl-vcard:*vcard-namespace*))  
  (xpath:evaluate "string(/vcards/vcard/tel/*/text())" *baba*)) 

Ignoring the with-namespaces macro invocation for a moment, we see a single line of code that gets us the information we want. Win! This is a very simple example, but the XPath language allows us to write much more interesting queries. There are two housekeeping matters we need to take care of first.

First, let's make a macro to handle the namespace stuff. Plexippus is (rightfully) rather picky about making sure that XML element and attribute names are properly qualified. We can set the default namespace as above, but we'll do this in a macro in case we want to change this later:

(defmacro with-vcard-namespace (&body body)  
  `(xpath:with-namespaces ((nil cl-vcard::*vcard-namespace*))  
     ,@body)) 

Second, we'll make a little function for converting XPath query results to text. We'll probably use something else in an real application using this stuff, but it's nice for playing around with the API and for interactive development.

(defun join-xpath-result (result)  
  (if (xpath:node-set-p result)  
      (format nil "~{~a~^~&~}"  
              (xpath:map-node-set->list #'xpath:string-value result))  
      (xpath:string-value result))) 

Now, back to XPath. If, for instance, I wanted to see the formatted names of every one in my family vcard database who has a photo in the database, I could do:

(join-xpath-result  
 (with-vcard-namespace  
   (xpath:evaluate "/vcards/vcard[count(photo)>0]/fn/text" *family*))) 

There's lots more one can do, but that should give you a flavor of how XPath can be used to effectively walk the document tree.

• Validation

As mentioned earlier, the xCard specification gives us a nice Relax NG (RNG) schema for data about individuals and other entities (as the spec somewhat vacuously says). The nice thing about this is that we can use the schema to validate our in memory representation of the xCard data -- even if there's never an xCard file per se:

(stp:serialize *baba* (cxml-rng:make-validator *vcard-rng-schema*)) 

Once we've got that the document in place we can validate it against the Relax NG schema. The VCARD -> xCard transformation may not be complete (which it isn't yet), but at least we know that the (so far tested) output is valid XML, that complies with the Relax NG schema.

• Really putting it all together

Here's another simple example of getting some data out of the xCard document:

CL-VCARD-EXAMPLE> (xpath:with-namespaces ((nil cl-vcard::*vcard-namespace*))  
  (format nil "~A is a ~A who works at ~A and can be reached via e-mail at ~A"  
          (xpath:evaluate "string(/vcards/vcard/fn/*/text())" *baba*)  
          (xpath:evaluate "string(/vcards/vcard/title/*/text())" *baba*)  
          (xpath:evaluate "string(/vcards/vcard/org/*/text())" *baba*)  
          (xpath:evaluate "string(/vcards/vcard/email/*/text())" *baba*)))  
 
"Baba O'Riley is a Field Worker who works at Polydor Records and can be reached via e-mail at thewho@example.com" 

Of course we can write more interesting queries, make a proper front end to the data, write it back out, talk to an address book server, etc... but those exercises are left for the reader.

Wait a minute, did I say talk to a server? DRAKMA would be perfect for that, but there's one problem. In the next blog post, I'll go into how one can talk to a CardDAV server and what one needs to change in DRAKMA to make this work. Next time...

In the mean time, cl-vcard can be found on github.

• References

** Libraries that made this all possible

1. CXML
2. CXML-STP
3. CXML-RNG
4. cl-parser-combinators
5. fset
6. DRAKMA

** Standards

1. VCARD
2. ABNF
3. xCard
4. Relax NG
5. RELAX NG Compact Syntax
6. XPATH

Well, the previous attempts at the pixel setf-expander got most of the way there, but there are a couple of important changes since the last blog post, that I figured I should document for posterity's sake, lest someone run across the old post and attempt to base some future setf-expander off of the almost-but-not-quite-fully-working version contained therein.

First of all, Utz Uwe-Haus provided a number of fixes to get the fast path setf-expander working on Allegro. The first step was to get %get-image-dimensions working via a cltl2-signature-compatible version of variable-information. The second step was to look for types of the form (integer 0 255) instead of (unsigned-byte 8), which is how Allegro apparently reports (unsigned-byte 8)'s. Finally, it turns out that Allegro is finicky about needing things at compile-time in slightly different ways than SBCL is and it needs +max-image-channels+ define at compile-time, which sounds like the right thing to do in any case.

So with those changes in place, we have:

;;; support functions/constants for the pixel setf-expander need to  
;;; exist at compile time  
(eval-when (:compile-toplevel :load-toplevel :execute)  
  (defun %get-array-dimensions-from-type-decl (type-decl)  
    "Extract the array dimension specifier from type declaration TYPE-DECL."  
    #+(or sbcl ccl)  
    (and type-decl  
         ;; here we expect e.g. (TYPE SIMPLE-ARRAY (UNSIGNED-BYTE 8) (* * 3))  
         (listp type-decl)  
         (= (length type-decl) 4)  
         (fourth type-decl))  
    #+allegro  
    (and type-decl  
         ;; here we expect e.g. (TYPE (SIMPLE-ARRAY (INTEGER 0 255) (* * 3)))  
         (listp type-decl)  
         (= (length type-decl) 2)  
         (= (length (second type-decl)) 3)  
         (third (second type-decl))))  
 
  (defun %get-image-dimensions (image-var env)  
    #+(or sbcl ccl allegro)  
    (when (symbolp image-var)  
      (multiple-value-bind (binding-type localp declarations)  
          (opticl-cltl2:variable-information image-var env)  
        (declare (ignore binding-type localp))  
        (let ((type-decl (find 'type declarations :key #'car)))  
          (%get-array-dimensions-from-type-decl type-decl)))))  
 
  (defconstant +max-image-channels+ 4)) 

Ok, enough for the Allegro fixes. Now into the pixel setf-expander itself. There were a couple problems here. First, we weren't expanding image-var itself. This meant things would break if we tried to do:

(defmacro foo ()  
  `(make-8-bit-gray-image 4 4 :initial-element 32))  
 
(let ((moose))  
  (setf (pixel (setf moose (foo)) 0 0) 4)  
  moose) 

It turns out that we need to expand image-var itself with get-setf-expansion and deal with the 5 return values as appropriate. I think, that I can ignore the storing form, since I'm not actually, changing the value referred to by image-var and that I can just use the accessing form in the (setf (aref ...)) calls in the expander. If any language lawyers have any input here, it would be appreciated. Also, it's important to keep in mind that we need to return the temporary variables and their value forms from the get-setf-expansion. Ugh... This is all kind of a mess, but the end product is pretty neat! A non-consing idiomatic way to set pixel values, assuming we've declared the type of the image, but at least we can do so using the languages own (declare ...) mechanism rather than resorting to some sort of (with-fast-pixels ...) macro around all of the pixel/setf pixel calls.

Here's the final product:

(define-setf-expander pixel (image-var y x &environment env)  
  (multiple-value-bind (dummies vals newval setter getter)  
      (get-setf-expansion image-var env)  
    (declare (ignore newval setter))  
    (let ((image-dimensions (%get-image-dimensions getter env)))  
      (if image-dimensions  
          (let ((arity (or (and (= (length image-dimensions) 3)  
                                (third image-dimensions))  
                           1))  
                (temp-y (gensym))  
                (temp-x (gensym)))  
            (if (= arity 1)  
                (let ((store (gensym)))  
                  (values `(,@dummies ,temp-y ,temp-x)  
                          `(,@vals ,y ,x)  
                          `(,store)  
                          `(setf (aref ,getter ,temp-y ,temp-x) ,store)  
                          `(aref ,getter ,temp-y ,temp-x)))  
                (let ((stores (map-into (make-list arity) #'gensym)))  
                  (values `(,@dummies ,temp-y ,temp-x)  
                          `(,@vals ,y ,x)  
                          stores  
                          `(progn (setf ,@(loop for i from 0  
                                             for store in stores  
                                             collect `(aref ,getter ,temp-y ,temp-x ,i)  
                                             collect store))  
                                  (values ,@stores))  
                          `(values ,@(loop for i from 0 below (length stores)  
                                        collect `(aref ,getter ,temp-y ,temp-x ,i)))))))  
          (let ((syms (map-into (make-list +max-image-channels+) #'gensym)))  
            (let ((temp-y (gensym))  
                  (temp-x (gensym)))  
              (values `(,@dummies ,temp-y ,temp-x)  
                      `(,@vals ,y ,x)  
                      syms  
                      `(ecase (array-rank ,getter)  
                         (3 (let ((d (array-dimension ,getter 2)))  
                              (case d  
                                (1  
                                 (values  
                                  (setf (aref ,getter ,temp-y ,temp-x 0) ,(elt syms 0))))  
                                (2  
                                 (values  
                                  (setf (aref ,getter ,temp-y ,temp-x 0) ,(elt syms 0))  
                                  (setf (aref ,getter ,temp-y ,temp-x 1) ,(elt syms 1))))  
                                (3  
                                 (values  
                                  (setf (aref ,getter ,temp-y ,temp-x 0) ,(elt syms 0))  
                                  (setf (aref ,getter ,temp-y ,temp-x 1) ,(elt syms 1))  
                                  (setf (aref ,getter ,temp-y ,temp-x 2) ,(elt syms 2))))  
                                (4  
                                 (values  
                                  (setf (aref ,getter ,temp-y ,temp-x 0) ,(elt syms 0))  
                                  (setf (aref ,getter ,temp-y ,temp-x 1) ,(elt syms 1))  
                                  (setf (aref ,getter ,temp-y ,temp-x 2) ,(elt syms 2))  
                                  (setf (aref ,getter ,temp-y ,temp-x 3) ,(elt syms 3))))  
                                (t (loop for i below d  
                                      collect (setf (aref ,getter ,temp-y ,temp-x i) (elt (list ,@syms) i)))))))  
                         (2 (setf (aref ,getter ,temp-y ,temp-x) ,(elt syms 0))))  
                      `(ecase (array-rank ,getter)  
                         (3  
                          (let ((d (array-dimension ,getter 2)))  
                            (case d  
                              (1  
                               (values  
                                (aref ,getter ,temp-y ,temp-x 0)))  
                              (2  
                               (values  
                                (aref ,getter ,temp-y ,temp-x 0)  
                                (aref ,getter ,temp-y ,temp-x 1)))  
                              (3  
                               (values  
                                (aref ,getter ,temp-y ,temp-x 0)  
                                (aref ,getter ,temp-y ,temp-x 1)  
                                (aref ,getter ,temp-y ,temp-x 2)))  
                              (4  
                               (values  
                                (aref ,getter ,temp-y ,temp-x 0)  
                                (aref ,getter ,temp-y ,temp-x 1)  
                                (aref ,getter ,temp-y ,temp-x 2)  
                                (aref ,getter ,temp-y ,temp-x 3)))  
                              (t (values-list  
                                  (loop for i below d  
                                     collect (aref ,getter ,temp-y ,temp-x i)))))))  
                         (2 (aref ,getter ,temp-y ,temp-x))))))))))  
 
(defmacro pixel (image-var y x &environment env)  
  (let ((image-dimensions (%get-image-dimensions image-var env)))  
    (if image-dimensions  
        (progn  
          (ecase (length image-dimensions)  
            (2 `(aref ,image-var ,y ,x))  
            (3 `(values ,@(loop for i below (third image-dimensions)  
                             collect `(aref ,image-var ,y ,x ,i))))))  
        `(ecase (array-rank ,image-var)  
           (2 (aref ,image-var ,y ,x))  
           (3 (ecase (array-dimension ,image-var 2)  
                (1 (values  
                    (aref ,image-var ,y ,x 0)))  
                (2 (values  
                    (aref ,image-var ,y ,x 0)  
                    (aref ,image-var ,y ,x 1)))  
                (3 (values  
                    (aref ,image-var ,y ,x 0)  
                    (aref ,image-var ,y ,x 1)  
                    (aref ,image-var ,y ,x 2)))  
                (4 (values  
                    (aref ,image-var ,y ,x 0)  
                    (aref ,image-var ,y ,x 1)  
                    (aref ,image-var ,y ,x 2)  
                    (aref ,image-var ,y ,x 3))))))))) 

Finally, if you've gotten this far and you want to see opticl in action, check out spectacle a CLIM application for viewing images that uses opticl for the image loading, representation, etc... On SBCL, and presumably Allegro, it has nice responsive scrolling/zooming/rotating/etc..., but if the pixel stuff conses (as it seems to do on CCL), it can be a bit sluggish.

I'd like to officially announce the availability (and perhaps more importantly, the quicklisp-installability) of opticl, a new image processing library for Common Lisp with a BSD-style license.

Opticl can be found on the opticl github page. To install opticl from quicklisp, do:

(ql:quickload 'opticl) 

from a suitable lisp with quicklisp installed. Opticl has been mostly developed on SBCL, but should work on any Common Lisp, and has seen some limited testing on CCL and ABCL. Patches to more fully support other lisps would be most welcome, should they be needed.

Opticl picks up many of the ideas and concepts from my earlier ch-image image processing library and Matthieu Villenueve's IMAGO library, but offers some advantages over both packages, such as the direct use of common lisp arrays for images and the efficient access to both getting and setting pixel values using mulitple-values, a setf-exapnder and, where available, CLtL2-style variable information to provide hints to the compiler to generate efficient code using standard lisp type declaration expressions.

Some of the core features of opticl are:

• representation of various types of 2-d images in common lisp arrays and routines for making the appropriate arrays
• routines for efficiently performing affine transformations of images providing for operations such as resizing, scaling, rotating and skewing images
• support for discrete convolution with arbitrary kernels, with built-in kernels for blurring and sharpening images
• support for morphological operations with arbitrary kernels, with built-in kernels for dilating and eroding images
• simple drawing primitives
• performing gamma computations on images
• I/O routines to read and write from various file formats; currently supported filetypes are JPEG, PNG, TIFF, PBM, PGM, PPM and GIF.
• routines for converting between various image types
• k-means clustering of pixels in images

More details about opticl can be found in the README, and in the opticl-test and opticl-examples packages. Note that these packages have been broken out into their own repositories in order to keep the size of a core opticl installation to a minimum. Currently opticl checks in around 3,500 lines of lisp code and the code compiles to approximately 900k of fasl files on SBCL/x86-64.