I made a lot of bad decisions early in my lisp coding days. Lately I've been trying to undo some of the most egregious mistakes. One of the big mistakes was that I had way too many dependencies between my various libraries. Everything I wrote depended on ch-util and ch-asdf. There was no need for this.

I've released new versions of clem and ch-image that don't depend on ch-util and ch-asdf. There may still be some references in the doc building stuff, but the core libraries and their associated tests no longer depend on these.

To mark the milestone of finally bringing hunchentoot-auth, hunchentoot-vhost, hunchenoot-cgi and nuclblog into the present such that they work properly with the hunchentoot-1.0 release, I've rolled up the following packages:

• hunchentoot-auth_0.2.tar.gz
• hunchentoot-vhost_0.2.tar.gz
• hunchentoot-cgi_0.2.tar.gz
• nuclblog_0.5.tar.gz

Of course these are probably best gotten from the git repo's, but for those of you who like released versions, I figured I'd roll new ones since it had been quite some time (almost two years in some cases!).

Well, after months of instability with my hunchentoot-based webserver, I finally, once again, got around to trying to figure out the source of the instability was. I had come to blame SBCL's sb-ext:run-program functionality as I was able to fairly reliably crash the server using apachebench. I was also seeing sporadic crashes somewhat randomly after the server being up for a week or so. So, this was a pretty strong hit that it had something to do with sb-ext:run-program. Folks who were much more knowledgeable than I about the SBCL internals, including Francois-Rene Rideau and Gabor Melis, looked at cleaning up possible sources of race conditions and generally robustifying sb-ext:run-program but none of the fixes seemed to make the situation better. Compounding my difficulties was the fact that I was running the server on FreeBSD, which doesn't see quite the level of SBCL testing/hacking that, say, linux does, so I thought it possible that there may be a bug either in the way SBCL handles signals on FreeBSD or in FreeBSD itself. Finally, I got around to replicating, roughly, my setup on another computer. In this case a MacOS box which, when subjected to the same stressful conditions, gave me a helpful error message that said something about being unable to open a pipe or perhaps that there were too many open pipes. This got me thinking "wait a minute, I'm just calling the program via sb-ext:run-program and getting a stream to read data back from the program; who's closing the stream and getting rid of the process?" Then it dawned on me that perhaps nobody was and perhaps these processes were sticking around, consuming scarce resources, like pipes, and, eventually, causing the server to crash. Sure enough, waiting for the process to finish and then closing the process cleared up my problem.

I should point out that SBCL's sb-ext:run-program has an argument that seems relevant here, which is the :wait arugment. One can specify :wait t which will wait until the process has finished. This seemed to work in some cases, but fail in others. Eventually, it occurred to me that it was failing in the cases where the output was larger than in the cases where it was succeeding. I think what was going on was that the external program was writing data to the stream which would fill up some buffer, which then blocked waiting for data to be read, which wasn't going to happen until after the process returned. There could be something else, going on here, but it seems to me that :wait t, while somewhat in spirit what I want, isn't going to do it from my. In this case, I'm just launching a process and expecting to get some data back from it, this isn't, say, a window manager that's going to live on for the life of the SBCL process, or beyond. But, :wait t didn't seem to do what I need either, so I was back to :wait nil. Now that I figured out I needed to close the process I came up with:

(defmacro with-input-from-program ((stream program program-args environment)
                                   &body body)
  "Creates an new process of the specified by PROGRAM using
PROGRAM-ARGS as a list of the arguments to the program. Binds the
stream variable to an input stream from which the output of the
process can be read and executes body as an implicit progn."
  #+sbcl
  (let ((process (gensym)))
    `(let ((,process (sb-ext::run-program ,program
                                          ,program-args
                                          :output :stream
                                          :environment ,environment
                                          :wait nil)))
       (when ,process
         (unwind-protect
              (let ((,stream (sb-ext:process-output ,process)))
                ,@body)
           (sb-ext:process-wait ,process)
           (sb-ext:process-close ,process)))))
  #-sbcl
  `(error "Not implemented yet!"))

(with-input-from-program (in path nil env)
  (loop for line = (chunga:read-line* in)
     until (equal line "")
     do (destructuring-bind
              (key val)
            (ppcre:split ": " line)
          (setf (hunchentoot:header-out key) val)))
  (let ((out (flexi-streams:make-flexi-stream
              (tbnl:send-headers)
              :external-format tbnl::+latin-1+)))                   
    (copy-stream in out 'character)))

Some of you may have noticed by now that this site is quite often down. I just wanted to give a brief explanation for this. This site certainly could be more robust, but, at least for the moment, I'm running a development instance of hunchentoot on one of the generally less-well supported SBCL platforms, FreeBSD, and am using things like my hunchentoot-cgi interface and nuclblog which certainly haven't been well tested, and probably aren't used by anyone else either. All this adds up to a lot of moving parts that occasionally fail in less-than-graceful ways.

In fact, just today the site crashed because SBCL ended up in LDB. I try to avoid this and try to reset the thing soon after this happens, but occasionally a good chunk of time passes. Until recently, I was seeing very frequent failures in which SBCL would run out of processes and the server would die. It turns out this was caused by hunchentoot-cgi and the way it was calling sb-ext:run-process. By calling sb-ext:run-process with :wait nil, the problem seemed to go away and all was right with the world for a couple of weeks. I'm not sure what's up with the latest failure and will keep on eye things to hopefully minimized the downtime if the server ends up back in LDB again.

Why am I using hunchentoot-cgi, you ask? Well, that seemed like the easiest way to use gitweb on my current setup. Hopefully it won't cause to many other problems. Thanks for your patience to this of you who actually might want to access the site.

[some years pass...] So, thanks to some gentle prodding and patches from Daniel Herring, I've finally gotten around to making clem and ch-image run on lisps besides SBCL. In particular, they both work on ccl and clisp, with ecl support hopefully not too far behind.

A combination of bogus things on my part, the occasional bug in the lisp implementation itself, and idiosyncrasies of the spec meant that various things didn't work right. The most-common offender was that I was assuming that things like double-float and fixnum defined classes, and that one could use these as method specializers. Apparently, the lisp spec says otherwise.

Also, there were a few bugs in ccl, especially in the 32-bit x86 version, that the ccl team fixed straightaway that allow trunk builds of ccl to load and run clem and ch-image. Thanks to R. Matthew Emerson for the prompt fixes and to Gary Byers for the helpful discussion about whether or not (setf (find-class 'foo) (find-class 'bar)) should create a new type for foo.

In any event, clem can be found at http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=clem.git and ch-image at http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=ch-image.git.

I should point out that these changes haven't found there way into official releases, per se, so if you want to use this stuff on ccl, clisp, ecl, etc... make sure you get the latest source from the git repo.

The TIFF image file format has been around a long time, and lisp even longer. Yet, I couldn't find any common lisp libraries for reading TIFF images. Perhaps there's one out there I missed, but the only one I could find was my previous attempt, tiff-ffi, which consists of some wrapper functions around FFI calls to libtiff. I wanted a native common lisp TIFF library that wouldn't require the libtiff library so, at Robert Strandh's urging, I wrote retrospectiff.

The retrospectiff git repository can be found here.

Currently, there is no support for writing TIFF files, and only a fraction of the TIFF image formats are supported, but RGB and ARGB images, both uncompressed and with LZW compression, can be read. Grayscale support should come next and, hopefully, support for writing TIFF files before too long.

Enjoy.

Ok, after a couple of years... I've finally gotten around to incorporating (some of) my changes to the cl-jpeg library into the upstream sources. There's a new version (1.023) up on the common-lisp.net cl-jpeg site.

To go along with this, the latest git sources of ch-image now use cl-jpeg.asd instead of my hacked up jpeg.asd. My jpeg project will now disappear and should be replaced with the upstream sources.

I know, I know... Why on earth would you want to run CGIs from hunchentoot? Well, while it may seem counterintuitive, I've found a need for this as I want to run the gitweb cgi interface behind hunchentoot and don't feel like setting up apache as the front-end to an otherwise happy hunchentoot site (this one), so I rigged up a little CGI interface for hunchentoot.

To use, check out the hunchentoot-cgi::create-cgi-dispatcher-and-handler function.

Ok, so I've finally gotten around to moving (at least some of) my projects over to git. The good news, besides having the code in a modern version control system, is that the repos are now publicly accessible. The list of projects can be found at:

http://git.cyrusharmon.org/cgi-bin/gitweb.cgi

SBCL recently removed the sb-ext:find-executable-in-search-path or whatever it was called function, so ch-util and ch-asdf were broken. The new ch-util and ch-asdf should fix that.

Ok, following closely on the heels of the last release, here are new releases of nuclblog, hunchentoot-vhost and hunchentoot-auth.

New features include:

• nuclblog now uses esc instead of str for textareas for editing blog entries so that, e.g., > doesn't get converted into > (thanks to Timothy Ritchey for the patch).
• the hunchentoot-vhost dispatch mechanism was simplified to remove the &optional vhost argument from dispatch functions, which means that standard hunchentoot dispatch functions can be used, allowing for the removal of create-virtual-host-folder-dispatcher-and-handler. Added the *virtual-host* special variable to replace the &optional vhost arg.
• Added some preliminary test code to hunchentoot-auth and exported some more symbols

The new releases can be found here:

• nuclblog 0.4.14
• hunchentoot-vhost 0.1.7
• hunchentoot-auth 0.0.11

Ok, a number of bugs and design flaws have been fixed. One can now be logged into multiple blogs on the same server with different user names and can log in and out of one without effecting the status of the other blogs. Also, some internal API cleanup for the blog handler functions. Oh, also the realm stuff in hunchentoot-auth was simplified and nuclblog now does a better job of keeping track of the information regarding which ports to use.

• nuclblog 0.4.12
• hunchentoot-vhost 0.1.4
• hunchentoot-auth 0.0.9

After I released 0.4.5, it was pointed out to me that there is a new cl-who that uses *downcase-tokens-p* instead of *downcase-tags-p*. nuclblog 0.4.6 has been changed to address this and, as a consequence, requires cl-who 0.11.0 or later.

Also, as a bonus, there's a new release of ch-util (0.3.3) that cleans up the build system a bit, removing some hacks I thought were neat when I was first learning ASDF and restoring buildability on (at least some) non-SBCL lisps. Thanks to Michele Pasin for the bug reports.

last time it was hunchentoot-vhost (of which there is a new release as well), this time it's hunchentoot-auth. Along the way, there's a new nuclblog release that supports the latest and greatest virtual hosting and user authentication facilities in hunchentoot-vhost and hunchentoot-auth, respectively.

• nuclblog
• hunchentoot-auth
• hunchentoot-vhost

I've updated hunchentoot-vhost to include Edi's copyright, as a couple of the functions are blatant cut-and-paste-and-edit versions of hunchentoot functions. But, more importantly, I've also updated nuclblog to work with hunchentoot-vhost so that now you can 1) have multiple blogs per host, but also have multiple hosts going, each potentially with multiple blogs.

• nuclblog
• hunchentoot-vhost

Of course, the down-side to this is that it introduces yet another dependancy to nuclblog.

I've made a rather trivial extension to hunchentoot that allows for so-called virtual hosting such that one can host web sites for multiple domains in a single server. hunchentoot-vhost project page and hunchentoot-vhost_0.0.2.tar.gz have more details.

No examples or documentation other than docstrings yet.

I've made an asdf-packaged version of the Latin Modern fonts for use with cl-pdf. One step closer to being able to wean myself of latex.

Now, if somebody would just make a nice beamer equivalent for cl-pdf, I'd be all set. Well, minus all the equations and stuff.

cl-pdf-lm 0.0.2

nuclblog 0.3.1 has been released.

Thanks to Timothy Ritchey for pointing out that I had broken the default startup on port 4242 (for http) and 4243 (for https). Now this works out of this box. Also, thanks to Brian Mastenbrook for pointing out a couple places where I needed some locks.

I've finally gotten around to packaging up a release of smarkup with cl-typesetting support. Unfortunately, there is no documentation and there are no examples for this yet. Hopefully those will appear in a future release release.

Enjoy.

smarkup

nuclblog now optionally supports https for authorized pages. Now the question is what to do about comments. Or, perhaps more to the point, how to allow comments with a minimal amount of blogspam.

nuclblog_0.2 can be found here. nuclblog is a rewrite of Brian Mastenbrook's cl-blog package for use with Hunchentoot. There's a demo blog included and it's fairly easy to get up and running:

(asdf:oos 'asdf:load-op :nuclblog-demo)
(nuclblog-demo:start-services)
(nuclblog::add-user nuclblog-demo::*blog* "demouser" "demopass")

Enjoy!

Oh, I should point out that an addition to ch-asdf, nuclblog also requires a number of other packages to run, including hunchentoot, cl-who, puri, cl-store and probably a bunch of other packages I'm forgetting at the moment.

yay. nuclblog now supports adding new blog entries and editing old ones. Hunchentoot is a joy to work with. I think all of the features of cl-blog that I used are in nuclblog now. There's no trackback, which I had disabled because of all the spam anyway. Also, it would be really nice to have some sort of comments facility, but that will have to wait for another day.

If anyone's interested in checking out nuclblog, let me know and I'll post a release of the code.

Ok, I've finally gotten around to, at least, starting to port cl-blog to hunchentoot. Right now it's pretty bare bones, but it has all my old blog posts and should see more features in the coming days. Let me know if things don't work as expected.

Moving on with the new releases... CLEM 0.4 has a whole slew of changes. The biggest change is largely under-the-covers and allows for n-dimensional matrices. Most stuff still expects 2-d matrices, but doing things like adding two 4-d matrices should work (multiplying them not so much, of course).

There are also a number of miscellaneous API cleanup issues. In particular, the mat-scale!/mat-add!/etc... variants have gone away, being replaced by (mat-add m n :in-place t). I'm not 100% the :in-place keyword is the way to go, but I got tired of maintaining an the ! version of every matrix operation, so I'm giving this a go. Feedback, as always, appreciated.

• http://cyrusharmon.org/static/releases/clem_0.4.tar.gz
• http://cyrusharmon.org/static/releases/ch-image_0.3.tar.gz

Following closely on the heels of the tiff-ffi 0.2 release is freetype-ffi 0.2 which also doesn't require gcc-xml-ffi just to load the sb-alien definitions.

Visit the project page or pick up a copy here: http://cyrusharmon.org/static/releases/freetype-ffi_0.2.tar.gz

[well, that was fast... fixed a bug in the tiff-ffi-gen depnedencies. now we're at version 0.2.2]

Some folks have complained about needing to install gcc-xml in order to get the libtiff FFI bindings working. Well, I've finally gotten around to separating out the generation of the xml file and sb-alien file from the C header files from the loading of a pre-existing sb-alien file. Now you can just download tiff-ffi and asdf:oos 'asdf:load-op it directly, without needing gcc-xml-ffi. If you want to rerun the gcc-xml-ffi auto-generation stuff, you can just do:

For details see the tiff-ffi project page or download it from http://cyrusharmon.org/static/releases/tiff-ffi_0.2.2.tar.gz.

As an added bonus, tiff-ffi no longer requires UFFI.

Enjoy

In the spirit of trying new things, I've decided to try using vox to blog TED. It's my first time at TED and it's an interesting event so far. While there is at least one other computational biologist here, I get the feeling I'm the only lisp hacker in the audience, BICBW.

If you want to read more, check out: http://cyrus.vox.com/.

After some more help from Juho Snellman in tracking down some nasty bugs, including one in the debugging code down in print.c, I was able to get sbcl/x86-64/darwin up and running without the sb-ext:*evaluator-mode* hacks. This experimental version has been checked into the SBCL tree as version 1.0.3.16 and most of the tests pass. There are still test failures in float.pure.lisp, debug.impure.lisp, foreign-stack-alignment.impure.lisp and run-program.impure.lisp.

Test reports on x86-64/darwin (and other platforms to make sure I didn't break anything) are most welcome.

At Berkeley, when you file your dissertation, you get a lollipop that says "PhinisheD" on it.

Here's Olivia's impression of what I would look like when I filed:

If you're interested in reading about an atlas of spatial patterns of gene expression in Drosophila melanogaster, the dissertation has been filed and can be found here.

The appendix describes some of the Common Lisp packages used in creating the atlas. Hopefully a paper describing these in more detail will appear soon.

Hmm... the search engines don't seem to be aware that dissertation == thesis. Perhaps this will help.

Well, it's been far too long since I've updated this page to reflect the status of threads and what not on SBCL/MacOS. In the meantime, the lutex stuff has landed on the trunk and the threads stuff has been cleaned up a bit, but is still somewhat unstable and doesn't pass the threads tests without catching an illegal instruction error. To address this, I, along with some major help from Alastair Bridgewater, have added an experimental feature for SBCL to use mach exception handling instead of (just) BSD-style signals.

The good news, irrespective of threads, is that this fixes the long-standing "CrashReporter" problem that many have complained about and it makes it so that one can use GDB with SBCL. Previously, GDB choked on SBCL's strategy of using mprotect for protecting memory in non-exceptional cases by preventing GDB from stepping across the EXC_BAD_ACCESS (the mach exception equivalent of a SIGBUS or SIGSEGV) and mach using mach exception handlers gets around this. Anyway, this has been checked in to the SBCL trunk but, at least for the moment, should be considered experimental and must be enabled by building with the :mach-exception-handler feature. Oh, and there's no PPC port for this yet.

As for threads, they are still not quite there, but certainly seem better and I have been using them for development work for some time. Hopefully the added debuggability will help in tracking down the remaining issues.

Well, it's getting closer. The lutex branch no longer kernel panics, thanks to mutex locks around the i386_set_ldt calls, and the garbage collection-caused memory corruption seems to be fixed. So it builds, builds itself, and all tests pass, usually. Unfortunately, it's the "usually" that is the problem. About 10% of the time, the threads test hangs with a thread waiting for a mutex lock that it's never going to get. I'm not sure if the problem is a subtle race condition in the code or if there are problems with MacOS' pthreads mutex/condition variable implementation, but it happens often enough that there is definitely some sort of problem somewhere. Hopefully this will get merged onto the head before too long, after the 0.9.13 release. Oh, and slam.sh still doesn't work.

A while back I asked something to the effect of "how do you frob the EIP in a mach exception handler?" Well, the answer is that you get the MACHINE_THREAD_STATE (i386_THREAD_STATE in this case) via thread_get_state, adjust the eip in the thread_state_t, and then call thread_set_state to get the changes to take effect. I wasn't calling thread_set_state before and assumed that this would behave like a Unix signal handler where you can just adjust the machine context and everything happens automatically. In the case of mach exception handlers, you need to explicitly set the state.

SBCL makes extensive use of POSIX signals for such tasks as garbage collection, error handling, and ensuring that atomic operations are executed atomically. MacOS X supports POSIX-style signals, but there are some problems with Apple's BSD-style, as they call it, signalling mechanism. The main problem, viz SBCL, is that MacOS X's signal implementation and GDB interact in such a way that renders GDB basically useless for debugging SBCL. SBCL's strategy of protecting memory pages with mprotect and then using a signal (usually SIGBUS or SIGSEGV, and SIGBUS in the case of MacOS X) handler to either adjust the memory protection mode and take apporpriate action or to signal an error causes MacOS to issue a mach exception (EXC_BAD_ACCESS) which is then caught be the kernel and causes a SIGBUS to be sent to the offending process. Unfortunately, GDB can't be used to continue past the offending mach exception, so the process just continues to send the exception and never issues the signal to the listening process or moves the program counter past the offending instruction.

In addition to the SIGBUS debugging problems, the signalling mechanism of MacOS X on Intel poses other problems in that MacOS' delivery of SIGTRAP signals is unreliable. It generally works, but only about 95% of the time. This is unacceptable for SBCL's use as a mechanism for signalling when operations that are supposed to be atomic have been interrupted and that approriate action needs to be taken. We have worked around the SIGTRAP problems by using the UDA2 instruction to cause a SIGILL signal to be delivered to the SBCL process. This works reliably, but causes the MacOS X on Intel code to differ from other Intel-based platforms.

Finally, Apple's crash reporter doesn't realize that we might be using memory protections and the associated SIGBUS messages for non-crashing, expected behavior and generates a crash log message or, depending on the Crash Reporter preferences, a dialog message to appear on the screen.

These issues are enough to motivate me to consider using Mach exceptions instead of or, more likely, in addition to POSIX/BSD-style signals.

If only it were that simple. Gory details to follow...

So, now that SBCL works on MacOS X/Intel, and given that MacOS is proving to be rather recalictrant when it comes to running a threaded version of SBCL, I thought I would share my list of the top 10 reasons why the life of an SBCL developer is made needlesly difficult by the current state of MacOS X.

1. No OS Source. The source to Darwin used to be available. Apparently that is no longer the case, at least for the Intel version. This is rather unfortunate. When developing for Linux one can dive down into the source to see if, for instance, user provided thread stacks are available to be freed after a pthread_join or not. (Note: in this case, the source to the pthread libraries is in fact available, but, given the design of MacOS' Mach kernel, this is mostly glue around calls down the to the Mach thread layer, the sources to which are not available).
2. GDB can't step across an EXC_BAD_ACCESS/SIGSEGV. SBCL makes rather extensive use of signals (or, in Mach parlence, exceptions) in the "this is a somewhat unusual, but also expected event and should be appropriately dealt with"-sense, not the "this is an error, most likely caused by programmer error or system failure, maybe you should print an error message before you quit"-sense. One critical example of this is the use of memory protection to trigger a SIGSEGV (or SIGBUS depending on the archictecture) to inform the system when bits of memory are being written to. This is a normal event and it triggers a Mach Exception (EXC_BAD_ACCESS) that GDB cannot step across, attempting to do so just causes the event to be refired. Setting gdb to "handle pass noprint" this type of exception just causes GDB to hang there. This makes GDB basically unusable, except for certain cases where you can attach to a running core, which will then proceed to work until an EXC_BAD_ACCESS is triggered again.
3. INT3 traps are not reliably delivered. Another example of where a signalling mechanism is used to handle slightly unusual, but totally expected, cases is the use of the INT3 trapping mechanism. SBCL uses this for error handling and, especially, as part of the mechanism for achieving fast atomic operations without having to go to the kernel for a lock. INT3 traps basically work on MacOS, except when they don't, which is about 2-5% of the time. Basically the trap signal is just lost and is not reliable delivered to the signal handler. This causes all sorts of problems for a system that expects to get these traps and was the source of major headaches in the MacOS X/Intel porting effort.
4. sem_init is not implemented. sem_open is implemented, but this takes a pathname and is a much more expensive call than creating an anonymous semaphore. The ironic thing is that the underlying Carbon and Mach APIs do support anonymous semaphores and they machinery to associate file system path names to Mach semaphores (one presumes) for use with sem_open. It would seem trivial to support sem_init.
5. The mach semaphores are a private API. Here's a useful API for doing semaphore stuff, but for some reason it's a private API. One can use Carbon semaphores, and can link in the Carbon framework, but this seems rather unneccessary.
6. Problems freeing a thread's stack(?). If I provide a stack for a thread to use with pthread_attr_setstack or pthread_attr_setstackaddr, I see major problems with the threads test suite if I free the stack. If I don't free the stack, the test suite is happier, until the kernel panics. See below.
7. Kernel panics. I have seen quite a number of kernel panics caused by (one presumes) use or misuse of the pthreads and semaphore APIs. I could understand it if this were a KEXT or even a root process, but these are all user processes. No user process should be able to so easily cause kernel panics.
8. No futexes. MacOS has a whole bunch of different locking APIs, none of which seem as nice as futexes. It would be great if the kernel (or a KEXT?) could provide futex support for MacOS. I'm assuming that it's not just SBCL and that other language environments, databases, and other highly-concurrent applications will take advantage of futexes and their efficient locking properties on Linux. It will be a shame if those applications are relegated to only using pthread condition variables and mutexes on MacOS X.
9. No POSIX RT signals. The POSIX RT signalling stuff provides for much more reasonable behavior of the delivery of signals than the original POSIX stuff. We have had to jump through hoops to get the threaded version of SBCL as far as it is without RT signals. It would be great if future versions of MacOS X supported RT signals (I would be happy to trade Spotlight for RT signals, although I have a hard time seeing a guy in jeans and a black turtleneck running around a stage talking about how great POSIX RT signals are :) ).
10. LDTs are not reused even after being freed. This is a rather obscure bug, and certainly one can manange their own set of LDTs, but the LDT API provides a way to return these to the OS, however they are not recycled and one runs out of LDTs after 0x2000 LDTs, or so, have been set up.

Anyway, perhaps this degree of systems inadequacy is present on all Operating Systems and perhaps there is a certain amount of pain in making things work properly on new OS/architecture environments, but when one compares the UNIX guts of OS X to the guts of, say, Linux or Solaris, OS X appears lacking and makes life difficult for the UNIX application programmer. I'm sure there are many benefits to Apple's microkernel architecture, but there's still a long way to go before it catches up to the other modern UNIXes, at least inasmuch as one treats MacOS as a UNIX, which is what is needed for the low-level of a sophisticated, cross-platform language environment such as Lisp in general and SBCL in particular. If these areas were addressed, it would make it easy for SBCL developers (those who use SBCL, not just those who develop and maintain SBCL) to both develop cross-platform lisp tools and to use MacOS X's sophisticated features, such as Cocoa, QuickTime, Aqua, Bonjour, etc... to develop first-class MacOS applications using SBCL.

On the positive side, this MBP is blazingly fast for SBCL development. It compiles all of SBCL in a hair longer than it takes my desktop x86-64 box on linux, and in half the time of a 2x2GHz G5 desktop.

Ok, I finally got around to hacking up a page listing my lisp projects. It can be found here. Enjoy!

Ok, an unofficial SBCL 0.9.10.37 binary for x86/darwin is available. Hopefully this will be the last binary released before the official 0.9.11 release. The last release had two problems. Some tests were failing due to the way they were being run and SLIME couldn't connect to SBCL due to some changes to cl:listen. Those issues have both been fixed in this release. This should work well enough to tide folks over until the 0.9.11 release, both for running this directly and for building new versions.

At long last, a binary release of SBCL 0.9.10.35 for x86/darwin is available.

Hopefully there will be an official 0.9.11 binary for darwin/x86 on the SBCL home page at the beginning of next month. Until then, this should allow one to build from current sources on x86/darwin without needing to do a cross-compile. Enjoy!

Ok, I think I've got a fix for the stability problems I was seeing with x86/darwin SBCL. At first everything looked great, then I noticed some sporadic failures. Doing things like running SBCL while the system load was high seemed to exacerbate the problem and increase the likelihood of failing with a SIGSEGV. After much debugging, telepathic and otherwise, and thanks to the help of Juho Snellman, Alastair Bridgewater and the rest of the #lisp crew, it became apparent that the problem was that the SIGTRAP handler wasn't reliably being called. I made some test cases that showed this to be the case, independent of SBCL, and that also demonstrated that the problem exists with Mach exception handlers as well. So, now what?

Well, thankfully Andrew Pinski and Alastair Bridgewater both suggested using x86 instructions that would generate a SIGILL instead and using that instead of SIGTRAP. Sure enough, that seems to ensure that the signal handler is reliably called. This means that SBCL on x86/darwin finally seems to work, for real this time. Knock on wood... I'll commit the changes and roll a binary for public consumption sometime in the next day or so.

Thanks to everyone who helped me debug and workaround this problem. It would be great if Apple would consider making sure that SIGTRAP is reliably called when an 0xCC instruction is encountered on x86. I've got testcases if you want them.

Also, if anyone knows how to get at and modify the EIP inside of a mach exception handler, let me know. I suppose digging through the GDB sources should provide some insight.

Experimental support for x86/darwin has been added to the SBCL source tree. No need to use my patch anymore, just grab the latest from source. Of course one has to cross-build at this point. I'll put up a binary release shortly that will enable folks to grab the release and the source and build it themselves without resorting to a cross-build.

Ok, this patch seems to work pretty well. Feedback and more testing welcome.

There's one remaing issue which is that we should consider using a sigaltstack so that signal handlers get a stack that is properly (16 bytes per the ABI) aligned. Currently, we don't do so and this might leave the door open for bad things to happen.

Ok, everything seems to be working now. I'll make a patch later today or tomorrow and hopefully this will hit the tree sometime in the next week. Thanks to Alistair Bridgewater, Juho Snellman, Reaper, Christophe Rhodes, Andreas Fuchs, et al., for remedial x86 assembly instruction, telepathic debugging help and moral support.

I've been working on trying to get SBCL running on Darwin/X86. Unfortunately, it doesn't work yet. But if anyone else wants to play along, here's a patch of the latest stuff.

Whoops. First pass at this didn't include the new files. New diff posted.

Currently, it builds and starts up, but dies in !COLD-INIT.

Comments and suggestions greatly appreciated.

Well, it took quite some work, but the generational garbage collector now works on PPC for both MacOS and Linux. The latest and greatest patch can be found here. The patch is to 0.9.9.29, but it should work on any of the very recent CVS versions.

I thought this was basically done a few days ago, but there was a really nasty and hard-to-find bug in the PPC assembly language routines where we were loading a 32-bit constant into a register with LIS and ADDI. The problem is that the ADDI instruction was sign-extending it's argument if the high-bit of the low-word was set. The solution is to use ORI which does not sign-extend. The problem manifested itself in a hosed LRA register, which, thankfully, Christophe Rhodes was able to hunt down.

This was not an easy task, but it was a great way to learn a lot about the internals of SBCL in a hurry. Thanks to everyone who listened to my griping and who chipped in with ideas and debugging help, especially Raymond Toy, Christophe Rhodes, Juho Snellman. And thanks to KingNato and Raymond Toy getting the ball rolling with the initial SBCL work and the CMUCL port, respectively.

Coming to a CVS repository near you soon. Hopefully.

Now, on to locking and threads...

So it turns out ASDF is very useful for packaging all kinds of documents in a form that can be easily distributed. With the help of some code that walks the ASDF system definition, I can automatically make tarball releases with all of the source code and other files, configuration files, shell scripts, documentation, images, etc... and have these be available to the user who downloads this distribution without worrying about where the package gets installed, relative paths, environment variables, etc...

This has been great, but I find myself writing a lot of code that looks like this:

  (let ((images-component
         (asdf:find-component
          (asdf:find-component
           (asdf:find-system "ch-imageio-test") "test")
          "images")))
    (let ((inputfile
           (asdf:component-pathname
            (asdf:find-component
             images-component "sunset-lzw")))
          (imagedir (asdf:component-pathname images-component)))
           ...

Well, it's a lot easier to do something like this:

(ch-asdf::asdf-lookup "asdf:/ch-imageio-test/test/images/sunset-lzw")

This is a URI of scheme asdf with host nil and the path (:absolute "ch-imageio-test" "test" "images" "sunset-lzw"). Notice that this isn't a path in the filesystem, but rather a path in asdf space. The first element of the path (after the :abosulte) corresponds to the asdf system and the other elements correspond to ASDF components so we can find them with the following code:

(defun asdf-lookup (path)
  (cond ((and path (listp path))
         (reduce #'asdf:find-component (cdr path)
                 :initial-value (asdf:find-system (car path))))
        ((stringp path)
         (let ((uri (puri:parse-uri path)))
           (when uri
             (let ((scheme (puri:uri-scheme uri)))
               (when (and (or (null scheme)
                              (eql scheme :asdf))
                          (puri:uri-parsed-path uri))
                 (asdf-lookup (cdr (puri:uri-parsed-path uri))))))))))

Again, note that the elements of the path are not file names, but rather then names of ASDF components from the following asdf file:

(defsystem :ch-imageio-test
  :version "0.1.2-20050724"
  :depends-on (ch-util ch-imageio tiff-ffi)
  :components
  ((:module
    :test
    :components
    ((:ch-imageio-test-cl-source-file "defpackage")
     (:ch-imageio-test-cl-source-file "test-ch-imageio" :depends-on ("defpackage"))
     (:module
      :images
      :components
      ((:tiff-file "euc-tiff" :pathname #p"euc.tiff")
       (:tiff-file "eucgray-tiff" :pathname #p"eucgray.tiff")
       (:jpeg-file "euc-jpeg" :pathname #p"euc.jpeg")
       (:jpeg-file "eucgray-jpeg" :pathname #p"eucgray.jpeg")
       (:tiff-file "sunset-lzw" :pathname #p"sunset-lzw.tiff")
       (:jpeg-file "sanfran" :pathname #p"sanfran.jpeg")
       ))))))

It's a trivial little hack, but it saves me 6 lines of code all over the place.

Raymond Toy has recently added a generational garbage collector to CMUCL/ppc. Unfortunately, CLEM breaks it. Here's a release of the latest CLEM which breaks the gengc. I really need to get anonymous SVN working one of these days.

CLEM

Work continues on CLEM (Common-Lisp Egregious Matrix), my matrix math package for common-lisp. CLEM now has a bunch of macros which generate type-specific methods for various matrix operations so that I can do fast, non-consing matrix math operations like addition, multiplication, etc... CLEM uses the MOP to define a standard-matrix-class which serves as the meta-class for matrix classes. This allows one to define typed matrix classes as subclasses of matrix, with class attributes stored in the instance of the metaclass. So now we have matrices for {u,s}b-{8,16,32}, fixnum, integer, single-float, double-float, float, real, complex, number, and even t, although this last one is probably suspect.

In addition to the matrix types and the standard matrix operations (add, subtr, scalar multiply, matrix multiply, hadamard product, abs, log, etc...), aggregate operations (min, max, variance, sum), CLEM supports discrete convolution, affine transformation and a couple different types of interpolation, morphological operations (dilate, and erode), and thresholding. Most of these are relatively non-consing, although there are probably a few cases that need to be re-written in the new macro scheme.

Overall, it seems to work, but it is rather slow to compile and results in large fasl files. this shouldn't be too big of a problem, as I'd rather have a fast matrix package that took a while to compile than a slow package that compiles quickly.

Speeds are decent. I approach naive C algorithms and get to within a factor of 10 for highly-optimized matrix multiplication hand-coded in assembly. More benchmarking and further attention to things like the size of the blocks for the blocked matrix multiply would probably be a good thing.

ch-image

Using CLEM, I've developed a trivial image representation/manipulation package. I should probably follow the lisp tradition of calling this trivial-image except that this isn't really trivial, as it requires CLEM. I've thought about trying to abstract away the core matrix stuff from CLEM into a package that works on arrays, and then trivial-image could use that (trivial-matrix?), but I'm getting sidetracked. ch-image supports images of a few types including ub8, rgb8 and argb8. Adding additional types should be straightforward and I hope to work on this soon.

ch-imageio

ch-image is nice, but rather useless if you can't get images into and out of it. This is where ch-imageio comes in. ch-imageio does the conversion between either files in various formats, or from the results of other file reading packages that load the image files into their own format. Currently, ch-imageio supports the following: 1) reading and writing JPEG via the cl-jpeg library, 2) reading and writing TIFF files via libtiff using interfaces defined by gcc-xml-ffi, and 3) writing PNGs using Zach Beane's SALZA library.

gcc-xml-ffi

gcc-xml-ffi generates FFI definitions from C (and C++ code, sort of) using gcc-xml-ffi. Currently, it spits out sb-alien definitions, although the original incarnation did UFFI. In theory, other backends, like CMUCL's alien interface and CFFI should be fairly trivial, but I haven't gotten around to it yet. UFFI isn't really adequate here as it doesn't support callbacks.

ch-asdf

To facilitate generating xml files from gcc-xml and for generating common-lisp FFI declaration files from gcc-xml-ffi, I've made some extensions to asdf. I've also aped the SBCL asdf extensions for dealing with unix-dsos. This is a straightforward thing that everybody seems to do, but the less-obvious part, for me, was how to package them up in such a way that these extensions can be used by other asdf systems. ch-asdf was my attempt at solving this and allows me to add :unix-dso components without having to redefine what a unix-dso is and what is load and compile methods are in every asd file. Also, I can declare a component as a gcc-xml-c-source-file and the right things happen. Dependencies are sort of tracked, but there are a couple places where things break down. In any event, it makes writing asdf systems for packages that use gcc-xml-ffi and unix-dsos much easier.

tiff-ffi

tiff-ffi uses gcc-xml-ffi (and ch-asdf) to wrap the libtiff library. There are also some rather trivial glue functions that help facilitate things a bit. ch-imageio uses this to read and write TIFF files.

carbon-ffi

carbon-ffi allows one, in theory, to develop native Mac OS applications using SBCL. This sort of works and I have some screenshots of simple apps. I've even built "package"-style apps that allow for double-click launching. The negatives are that this is Carbon only (no Cocoa) and that in order to make this work one needs to use some undocumented Apple API functions. This proved to be a huge pain as getting this to work properly exposed some issues with SBCLs stack alignment where we were not properly aligning the stack on a 16-byte boundary, which was making AltiVec rather unhappy.

quicktime-ffi

Similar to the carbon-ffi, this is an FFI wrapper for quicktime. This also suffered from the stack alignment issue, but these are now fixed. Now one can use the QuickTime API from SBCL directly to make and read movies, etc... I havent't tried the GUI stuff (QT movie playing functionality, e.g., but it should work).

congeal

I have implemented a version of the congealing algorithm in common-lisp. This uses clem and the various image stuff to learn a set of transforms that bring a stack of images into registration and can learn the shape of the item represented in the stack of images.

clsr

I have begun to connect SBCL up with R so that I can evaluate R expressions from common-lisp using the R C API and can get the results back to lisp. Next steps are to have a representation of lisp data objects in R and vice versa so that I can, for instance, call lisp functions from R and to connect up the R plotting stuff so that I can make nice plots from SBCL. This is an area where a clean common-lisp API that wraps the R graphing APIs might be a nice thing.

ch-photo

ch-photo is a library based around FFI definitions to the dcraw package to read NEF and DNG files (come to think of it, this should be rolled into ch-imageio, but that hasn't happened yet). In addition, ch-photo contains some scripts for importing RAW files and for organzing them into file system heirarchies based on date of import and file type.

fftw-ffi

fftw-ffi is a wrapper for the fftw (Fastest Fourier Transform in the West) library. In addition to the FFI stuff, there are routines to translate data between ch-image and fftw compatible representations so that one can use fftw to do ffts of images in ch-image.

SBCL stack alignment issues

In order to get carbon-ffi and quicktime-ffi working properly, I had to fix some bugs with stack alignment on ppc. Thanks to Gary Byers for pointing out the bug after I was at wits end with bizarre results coming back from quicktime due to misaligned stack data being munged by altivec (without complaints). These have no been fixed and everything seems to be OK here.

callback fixes

The SBCL developers have been working on adding callbacks to SBCL. The initial ppc port had a number of bugs that caused problems with non-32-bit arguments, long longs, mixing arguments of different sizes, etc... Raymond Toy fixed a bunch of these problems for CMUCL and I ported these over to SBCL. After the initial round of fixes, there were some more problems with how the arguments got pulled off of the stack in the lisp trampoline, but these have been fixed as well. These patches have not yet hit the tree, but hopefully this will happen after the 0.9.8 release.

SBCL sb-alien field alignment issues

Finally, in order to make the carbon-ffi and quicktime-ffi packages work properly, I had to deal with the fact that some of the MacOS toolbox data structures use a bizarre alignment scheme that is a holdover from the m68k days. The bad news is that a lot of these are core structures for things like graphics and IO. In order to support these weird alignments, Apple's hacked up version of GCC has some pragma directives that take care of the alignment issue. Fortunately for me, someone else must have been having similar problems as the CVS versions of gcc-xml started dumping out the offset of structure members. Now that the quasi-compiler was giving me this information, I had to hack up SBCL's alien interface to allow me to use it to specify non-standard alignment of struct elements. This works and now we can properly use these funky MacOS structures.

Ok, that's the overview. One of the main missing pieces is documentation. I need to go back in and document all of this stuff soon. Perhaps that will be my first New Year's Resolution: I well document as much or more code than I write this year.

Happy New Year,

Cyrus

For both of the other people using callbacks in SBCL on PPC, I've fixed some more callback bugs. The latest patches sent to sbcl-devel fix the following:

• Arguments of mixed sizes. Before, we were calculating the offsets of the arguments on the stack wrong. Bascially, we were moving up the stack by the size of the next argument, not the size of the current argument so that if you passed in a double followed by a float, or vice versa, you would get the wrong results.
• Callbacks with lots of arguments. Before when we ran out of registers, we would throw an error. This is a mistake. What we needed to do was to copy the registers onto the stack so that the alien-callback-lisp-lambda-wrapper or whatever it's called could get the registers off of the stack. Once we run out of registers, we don't need to do anything as the value is on the stack.

Previous fixes contained in the latest patch set include:

• Ported Raymond Toy's CMUCL fixes for non-32 bit callback args (doubles, long longs, chars, shorts (although these didn't work in combination before the latest patch)).
• Ported Raymond Toy's CMUCL fixes for calling vararg functions. This isn't a callback fix, but rather an ffi-fix, but it's in roughly the same area. The problem was that we need to put the arguments on the stack as well as in the registers when calling vararg functions.

Speaking of rtoym, there's still an outstanding issue in SBCL that he fixed in CMUCL. One of the ffi param registers is being used as reg_FDEFN. This should be moved down one, I think. I'll look into this after the callback changes hit the tree.

Thanks to rtoy for the cmucl fixes. I think the new fixes are going to affect CMUCL as well, so hopefully he'll find time to add those in, assuming he's not too busy with the gengc port! :-) Go rtoy, go! Thanks to luis and the cffi crew for motivating test cases and to lemonodor for complaining about the too many args thing.

Once again, here are some more new releases. ch-util and clem today. clem, in particular, has some MOP fixes that allow for inheritiance of meta-class options from superclasses that makes it easier to define matrix subclasses.

Ok, here are some more new releases. This time, it's gcc-xml-ffi, tiff-ffi, ch-util and ch-asdf.

Release notes to follow, eventually.

Here are some new releases of my matrix and utility packages.

Thanks to Gary Byers for pointing out that he had seen similar problems with funky lines in Mac OS windows when the stack was misaligned. For some reason SBCL's stack is misaligned when we try to call a C routine. This patch ensures 16-byte alignment and makes Mac OS windows (and QuickTime movies) look much nicer:

Ok, after a lengthy hiatus, I've been trying to get SBCL and Carbon to play nicely together again. Unfortunately, I seem to be at a roadblock. Here's what I've got:

As you can see, the window has some funky lines on it and the menu bar colors are hosed. This doesn't happen when I run essentially the same code from C. So something horrible must be happening somewhere like Carbon's initialization routines or something. I've used the undocumented hacks to get foreground operation and I've tried faking a bundle a la OpenMCL, but I can't seem to get things to look any better this. Suggestions greatly appreciated.

Here are some new releases of my matrix and image packages. Notable new features include (fastish) affine transformations and discrete convolution, various code cleanups, a bug fix in gcc-xml-ffi where I wasn't allocating space for the NULL at the end of C strings, etc... Enjoy.

(Note that the iamgeio download is rather large as it now contains some sample images.)

(Furthermore, note that I just bumped the version # for clem to fix an asdf problem and some code that was inadvertently commented out.)

Ok, it's time for me to bundle up some of this stuff and ship it out, in case anyone is interested. What is it, you ask? Why, some nascent libraries for image processing, matrix math, image i/o, interfacing with C libraries via gcc-xml, etc...

Some of this was released yeseterday, but there are brand new versions of all of this stuff. Comments/suggestions appreciated. Oh yeah, all of these pretty much require SBCL at the moment. I'd like to resurrect OpenMCL support and support CMUCL in the future.

For those of you have run into the 128M dynamic space limit in SBCL on PPC, the following patch provides for a 768M heap. It's rather trivial, and I imagine most folks who have run into this limit can come up with something like this or better, but nevertheless, here are the parameters I've been using successfully for the last few weeks.

I'd like to see something like this rolled into the main SBCL repository at some point, but until then, enjoy, or at least let me know if you have a better approach. On a related note, porting the gencgc to ppc (or migrating rtoy(?)'s CMUCL version of it to SBCL) would be a great project, but unfortunately I'm not sure I have the skills and I definitely don't have the time at the moment.

Index: sbcl/src/compiler/ppc/parms.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/compiler/ppc/parms.lisp,v
retrieving revision 1.19
diff -u -r1.19 parms.lisp
--- sbcl/src/compiler/ppc/parms.lisp	11 Mar 2005 17:10:10 -0000	1.19
+++ sbcl/src/compiler/ppc/parms.lisp	18 May 2005 17:49:34 -0000
@@ -93,19 +93,19 @@
 ;;; FIXME: this is a gross violation of OAOO, done purely to support
 ;;; the #define of DYNAMIC_SPACE_SIZE in validate.c -- CSR, 2002-02-25
 ;;; (these numbers should match dynamic-0-*)
-(def!constant dynamic-space-start   #x40000000)
-(def!constant dynamic-space-end     #x47fff000)
+(def!constant dynamic-space-start   #x10000000)
+(def!constant dynamic-space-end     #x3ffff000)
 
 ;;; nothing _seems_ to be using these addresses 
-(def!constant dynamic-0-space-start #x40000000)
-(def!constant dynamic-0-space-end   #x47fff000)
-(def!constant dynamic-1-space-start #x48000000)
-(def!constant dynamic-1-space-end   #x4ffff000)
+(def!constant dynamic-0-space-start #x10000000)
+(def!constant dynamic-0-space-end   #x3ffff000)
+(def!constant dynamic-1-space-start #x40000000)
+(def!constant dynamic-1-space-end   #x6ffff000)
 
 #!+darwin
 (progn
-  (def!constant linkage-table-space-start #x50000000)
-  (def!constant linkage-table-space-end   #x51000000)
+  (def!constant linkage-table-space-start #x0a000000)
+  (def!constant linkage-table-space-end   #x0b000000)
   (def!constant linkage-table-entry-size 16))
 
 ;;;; Other miscellaneous constants.

After spending a couple of days trying to get things back to normal after upgrading to Tiger (trying to sort out compiler versions, fink packages, R and bioconductor, etc...), it occurred to me that the upgrade process for SBCL and OpenMCL has been a breeze. I haven't upgraded to the latest bleeding-edge OpenMCL as I've been using SBCL for most of my work these days, but I understand it runs and OpenMCL 0.14.3 seems to still be running OK. SBCL builds and runs great with Tiger. Thanks to Gary Byers, Brian Mastenbrook and the rest of the OpenMCL and SBCL development teams who made sure that the Tiger upgrade would be a snap, at least from the (or should I say my) open-source lisp point of view.

Now, back to figuring out which combination of the fortran and C compilers (and which libraries) will actually compile R and successfully install Bioconductor...

Oh, and thanks to the SLIME and carbon emacs guys too!

LTU has a piece on a special issue of JStatSoft on the demise of LispStat. I haven't read through all of the articles yet, but it does seem clear that lisp-stat isn't much of a player these days and that R is the lingua franca of the statistics world. This isssue is timely for me as I have been working in R on and off for the past few months and while I have come to appreciate some of the features of R, I definitely miss many, many nice features of common lisp when using R. S and R are shining examples of Greenspun's Tenth Law in practice. But at least the R guys were smart enough to build R in C by implementing a lisp-like intermediate language. Nevertheless, there are a ton of things in common lisp that are either just starting to make their way into R or are way off on the horizon that the R community will need to grapple with eventually. Then there's the whole issue of completeness, standardization, maturity of the language, etc... But, in any event, it does seem to be the case that R has taken over the, academic at least, statisticl computing landscape.

But, I think it may be too early to write lisp off for statistics off just yet. I'm just delving into the land of lisp-stat, but it doesn't seem to me that lisp-stat is the end-all, be-all of statistics in common lisp. Just as the R and S guys no doubt learned from lisp-stat, I'm sure there are lessons from R that could be applied to a lisp-based statistical package. Witness matlisp as an example of a lisp-based package inspired by matlab. Speaking of matrices and linear algebra, and granted I need to catch up with the past 15 years of progress in lisp-stat, but it seems that lisp-stat wasn't designed with what I view as modern statistic in mind. When I think of modern statistics I think of big matrices and methods to operate on them efficiently. The main focus of lisp-stat seems to be on dynamic graphing tools and matrices and linear algebra get 5 or 6 pages in the orginal book. I'm sure this may have changed, but in R, as in matlab, the focus seems to be on efficient matrix math from the get go. As for the dynamic graphing stuff, yes, I can see this being nice, but I see this as more of an "add-on" thing than a core feature. A core feature (or a damn important library) of any statistics package, on the other hand, needs to be high-quality print graphics. It's great to be able to spin and zoom and all that, but ultimately scientists want to present their data for publication. R has 1) a good interface to BLAS/LAPACK (with decent notation), 2) great graphing facilities (even if support for png, jpeg, etc... are a bit janky and require either X or ghostscript), 3) a ton of in-depth (if not complete) library packages for doing a little bit of everything.

I think one of the major things holding lisp-stat back was the lack of good freely available common lisp implementations. The lisp-stat book claims that (at least in 1990) common lisps were expensive and that the free (subset of) common lisp available didn't have a compiler. Obviously things have changed a great deal since then. I haven't been around long enough to know the history well enough to do all the previous implementations justice, but I can say that first CMUCL and later SBCL have done an amazing job of bringing a fantastic common lisp implementation to the masses. All that aside, I still think that lisp-based systems have a deployment problem. If you look at successful lisp based systems, I'd argue that most of them include their own (perhaps half-assed or limited) lisp implementations, emacs and autocad being cases in point. One of the advantages of emacs and R is that they both just build and run out of the box. There isn't this whole build the language, get the libraries, build them, get the application, build it, get the app. libraries, build them, etc... rigamorole. Granted R libraries aren't necessarily the cleanest thing in the world, but the whole R CMD INSTALL thing is easier, for some reason, for new users to grasp than the asdf install process.

Nevertheless, I'm still hopeful that the emergence of high quality, freely available lisp systems, and the success of at-least-partially-lisp-inspired (at least in the guts) stats packages such as R suggest that there's still hope for doing linear algebra and statistics in lisp. I'm looking forward to trying to do some of what I currently do in R in common lisp in the future.

So in my continued adventures in ffi-land, I decided it would be an interesting exercise to make a MOP metaclass that provided for transparent access to FFI structs. I took a first stab at this a while back using OpenMCL and it kinda worked. As part of my recent efforts to switch over to SBCL, I decided to pick this up again and see if I could make this work with UFFI. Sure enough, it works! And I've cleaned up the code a bit to create a package called meta-ffi which provides the MOP infrastructure and another package called vimage which uses the GCC-XML stuff and my GCC-XML-UFFI declarations to automatically parse the vImage (a MacOS framework for doing HW accelerated image manipulation) headers, generate an XML file of the appropriate declarations, generate a CL file with UFFI declarations and finally define a CL class which provides for transparent access to underlying structs. It's an awful lot of machinery, but it should be reusable in other contexts and might help write CL code that uses foreign libraries. Clearly this can be done without all of this machinery, but the point was to try to automate as much as possible. Now I can do something like this:

(defclass vimage-affine-transform ()
  ((a :accessor a :initarg :a :foreign-field '|a|)
   (b :accessor b :initarg :b :foreign-field '|b|)
   (c :accessor c :initarg :c :foreign-field '|c|)
   (d :accessor d :initarg :d :foreign-field '|d|)
   (tx :accessor tx :initarg :tx :foreign-field '|tx|)
   (ty :accessor ty :initarg :ty :foreign-field '|ty|))
  (:metaclass uffi-foreign-struct-class)
  (:foreign-type |vImage_AffineTransform|))

 (make-instance 'vimage::vimage-affine-transform
                 :a 1.0 :b 0.0 :c 0.0 :d 1.0 :tx 0.0 :ty 0.0)

The MOP stuff under the covers takes care of the foreign struct allocation and making sure that slot-value and setf (slot-value... do the right thing.

I think this is kinda cool, but I can't decide if it's really useful or just Brucio-cool. Time to do some more hacking to see if this really helps. I think it could prove useful for the development of Carbon apps in SBCL.

I'm sure there are more pressing matters in getting SBCL to 0.9, 1.0 and beyond, but I'd like to suggest that the SBCL_HOME/INSTALL_ROOT thing is too fragile and should be fixed. To require that INSTALL_ROOT be set at build and install and runtime seems overkill. I would prefer to see SBCL_HOME/INSTALL_ROOT be set at build and install time, leaving an sbcl executable that doesn't require the setting of SBCL_HOME. I think this will help make SBCL useable for new users. I understand that 1) this is a relatively trivial bug, 2) there is a number of simple (including some currently documented) workarounds and, 3) there are more important things to work on, but, nevertheless, I think it would help adoption of SBCL is this were to be fixed.

Ok, the DSL outage has been fixed. Now I can get back to work. I always think I'll be more productive without a net connection, but it's never the case.

In any event, the gcc-xml work continues. One problem is 64-bit support. C long long should be useable as args and return values, but UFFI doesn't support this. I'm not quite sure what to do here. Perhaps using SB-ALIEN and other native FFI interfaces might be appropriate for 64-bit stuff. It doesn't really matter if everything goes through UFFI or not, as long as it works, and it's probably easier to stick the 64-bit support in here directly rather than trying to convince KMR to support 64-bit types in UFFI, which seems like a losing battle. Might need some special C glue support too, which probably has no reason to be in UFFI.

A second problem is dealing with the huge number of declarations generated. Clearly processing the XML file and generating the lisp decls every time is not a good idea (nor is generating the .xml from the .h/.c every time) and compiling the lisp forms is probably not a good idea either. Loading a compiled fasl helps, but still yields a 30-60 second delay. Saving a lisp core might work, but I understand there are some issues WRT lisp cores on foreign libraries and I haven't worked through these yet. As a step in this direction, I tried save-lisp-and-die (SBCL's dump custom lisp core command) and things failed with some a 65536 is not an UNSIGNED-BYTE 16 error. Changing the element type of the compact-info-entries-index (or something like that) to be 32 bits fixed the problem, but I have no idea if this is correct much less wise, although it gets me one step closer to trying it out. More in a bit...

Work is continuing on generating UFFI-based lisp bindings to foreign libraries via gcc-xml. Now I can successfully parse all the GCC-XML declarations from Carbon.h. This (and some apparently undocumented (at least not officially) MacOS hackery) enabled me to do this:

So I've been working on automatically generating UFFI declarations from GCC-XML and it's almost there. The idea is that one feeds a C file (.c or .h) to GCC-XML which outputs an XML file containing XML elements for the declarations in the file, including function defintions, typedefs, structs, enums, etc... I feed this file into some code which parses the XML file and spits out a file of UFFI declarations, which is then eval'ed to produce the appropriate UFFI types and what not. This should make it easier to port code that uses OpenMCL's interface database to SBCL. Don't get me wrong, the interface database is very cool, but it is OpenMCL specific and it would be very nice to be able to develop Carbon applications, e.g., on SBCL. And there's nothing OS X specific about this; it should work on any platform with GCC-XML and a modern Lisp implementation supported by UFFI.

There are some potential issues:

1. Names: How should lisp names for functions, etc... be constructed? Should we attempt to map things like SuperCoolOpenGLBuffer_Fast to super-cool-open-gl-buffer-fast? Or should we just intern the C name yielding a function call like this (|EnableOpenGLMonkeyButter| ...) or whatever? I'm leaning towards the mixed case symbol approach which means that the symbols have to be enclosed in vertical bars, but I'd arguet that this is a feature as it makes more obvious which symbols refer to external declarations.
2. Packages: To which package should the C declarations go?
3. Callbacks: For full functionality of many C APIs we're going to need callbacks. My understanding is that SBCL still doesn't support them. Hopefully this will get fixed in the near future.

Anyway, hopefully I will have a release of this up in the next week or so. Or send me email if you're interested in seeing this sooner rather than later.

(setf (ref l :foo :ban :barry) 17)

(defun hash-table-to-plist (h &aux l)
  (if (hash-table-p h)
      (progn (maphash
              #'(lambda (key val)
                  (setf l (cons (hash-table-to-plist val)
                                (cons key l)))) h)
             (nreverse l))
      h))

(defun hash-ref (h &rest keys)                                                                                  
  (reduce #'(lambda (h k) (gethash k h)) keys :initial-value h))                                                

(defun %put-hash-ref (new-value h key &rest more-keys)
  ;; not quite Perl-style autovivification, but we do create
 ;; appropriate list structure for intermediate keys that
can't be found
  (unless (hash-table-p h) (setf h (make-hash-table :test 'equal)))
  (let* ((sub (gethash key h))
         (val (if more-keys
                  (apply #'%put-hash-ref new-value sub more-keys)
                  new-value)))
    (progn (setf (gethash key h) val) h)))

(define-setf-expander hash-ref (place &rest props
                                  &environment env)                                                             
  ;; %put-ref may cons new structure or mutate its argument.
  ;; all this magic is just so that we can
  ;; (let ((l nil)) (setf (ref l :foo :bar) t))
  (multiple-value-bind (temps values stores set get)
      (get-setf-expansion place env)
    (let ((newval (gensym))
          (ptemps (loop for i in props collect (gensym))))
      (values `(,@temps ,@ptemps )
              `(,@values ,@props )
              `(,newval)
              `(let ((,(car stores) (%put-hash-ref ,newval ,get ,@ptemps)))
                 ,set
                 ,newval)
              `(hash-ref ,get ,@ptemps)))))

CL-USER> (let ((l nil)) (setf (hash-ref l :foo :bar) 47) (hash-table-to-plist l))
(:FOO (:BAR 47))                                                                     

[I was going to send this to sbcl-devel, but I've already pestered the folks on the list with a couple trivial matters this week and I'm sure the folks have better things to do than listen to me whine about perceived (by me) weaknesses in SBCL without a patch to address them. So, here goes...]

I've been working on adding prepared query support and support for blobs to the CLSQL postgresql backend.

• clsql-cyrus-20050207.tgz

It works pretty well (i.e. it passes the few simple tests I wrote and works in the limited situations in which I tested it), but is currently not used at all by the oodml stuff, which would be nice, as it offers the potential to greatly speed up repeated accesses to the database

I'd like to get this stuff cleaned up and integrated into the CLSQL source tree, but it hasn't happened yet. Hopefully soon.

SBCL's sb-bsd-sockets and sb-posix packages directly export a number of symbols, instead of using the :export cluase in defpackage. This seems to cause a problem on subsequenct recompilation of the defpackage.lisp file as the symbols are exported by the package but not in the :exports clause. This patch fixes things these two packages.

Christope Rhodes kindly pointed out that the consquences of calling defpackage on existing packages that have been modified are undefined. This suggests that my proposed patch is probably the wrong way to handle this. Xophe suggests the following:

  (eval-when (:compile-toplevel :load-toplevel :execute)
    (unless (find-package "FOO")
      (make-package "FOO" :use ("BAR") ...)))

Sounds good to me, but this hasn't made it into the tree yet, AFAICT.