The aforementioned articles about computational biology in Science make some reasonable points. But I find it ironic that I'm reading article in PDF form where there are no links to the references. Here we are discussing the future of science and the scientific publishing community still doesn't have links to references contained in the paper in the PDF! The HTML version of the paper is a little better, as some of the references have links to Medline (and CrossRef and Web of Science, whatever those are) but, in general, the references aren't hyperlinks, as one would hope in any modern publication. I agree that mathematical training is important for the future of the field, but it's also important that we bring scientific publishing into the '90s.

In this article, John Timmer makes some interesting commentaries on the recent Science articles by Pevzner and Shamir and Robeva and Laubenbacher. Worth a read.

To mark the milestone of finally bringing hunchentoot-auth, hunchentoot-vhost, hunchenoot-cgi and nuclblog into the present such that they work properly with the hunchentoot-1.0 release, I've rolled up the following packages:

• hunchentoot-auth_0.2.tar.gz
• hunchentoot-vhost_0.2.tar.gz
• hunchentoot-cgi_0.2.tar.gz
• nuclblog_0.5.tar.gz

Of course these are probably best gotten from the git repo's, but for those of you who like released versions, I figured I'd roll new ones since it had been quite some time (almost two years in some cases!).

Well, after months of instability with my hunchentoot-based webserver, I finally, once again, got around to trying to figure out the source of the instability was. I had come to blame SBCL's sb-ext:run-program functionality as I was able to fairly reliably crash the server using apachebench. I was also seeing sporadic crashes somewhat randomly after the server being up for a week or so. So, this was a pretty strong hit that it had something to do with sb-ext:run-program. Folks who were much more knowledgeable than I about the SBCL internals, including Francois-Rene Rideau and Gabor Melis, looked at cleaning up possible sources of race conditions and generally robustifying sb-ext:run-program but none of the fixes seemed to make the situation better. Compounding my difficulties was the fact that I was running the server on FreeBSD, which doesn't see quite the level of SBCL testing/hacking that, say, linux does, so I thought it possible that there may be a bug either in the way SBCL handles signals on FreeBSD or in FreeBSD itself. Finally, I got around to replicating, roughly, my setup on another computer. In this case a MacOS box which, when subjected to the same stressful conditions, gave me a helpful error message that said something about being unable to open a pipe or perhaps that there were too many open pipes. This got me thinking "wait a minute, I'm just calling the program via sb-ext:run-program and getting a stream to read data back from the program; who's closing the stream and getting rid of the process?" Then it dawned on me that perhaps nobody was and perhaps these processes were sticking around, consuming scarce resources, like pipes, and, eventually, causing the server to crash. Sure enough, waiting for the process to finish and then closing the process cleared up my problem.

I should point out that SBCL's sb-ext:run-program has an argument that seems relevant here, which is the :wait arugment. One can specify :wait t which will wait until the process has finished. This seemed to work in some cases, but fail in others. Eventually, it occurred to me that it was failing in the cases where the output was larger than in the cases where it was succeeding. I think what was going on was that the external program was writing data to the stream which would fill up some buffer, which then blocked waiting for data to be read, which wasn't going to happen until after the process returned. There could be something else, going on here, but it seems to me that :wait t, while somewhat in spirit what I want, isn't going to do it from my. In this case, I'm just launching a process and expecting to get some data back from it, this isn't, say, a window manager that's going to live on for the life of the SBCL process, or beyond. But, :wait t didn't seem to do what I need either, so I was back to :wait nil. Now that I figured out I needed to close the process I came up with:

(defmacro with-input-from-program ((stream program program-args environment)
                                   &body body)
  "Creates an new process of the specified by PROGRAM using
PROGRAM-ARGS as a list of the arguments to the program. Binds the
stream variable to an input stream from which the output of the
process can be read and executes body as an implicit progn."
  #+sbcl
  (let ((process (gensym)))
    `(let ((,process (sb-ext::run-program ,program
                                          ,program-args
                                          :output :stream
                                          :environment ,environment
                                          :wait nil)))
       (when ,process
         (unwind-protect
              (let ((,stream (sb-ext:process-output ,process)))
                ,@body)
           (sb-ext:process-wait ,process)
           (sb-ext:process-close ,process)))))
  #-sbcl
  `(error "Not implemented yet!"))

(with-input-from-program (in path nil env)
  (loop for line = (chunga:read-line* in)
     until (equal line "")
     do (destructuring-bind
              (key val)
            (ppcre:split ": " line)
          (setf (hunchentoot:header-out key) val)))
  (let ((out (flexi-streams:make-flexi-stream
              (tbnl:send-headers)
              :external-format tbnl::+latin-1+)))                   
    (copy-stream in out 'character)))

Some of you may have noticed by now that this site is quite often down. I just wanted to give a brief explanation for this. This site certainly could be more robust, but, at least for the moment, I'm running a development instance of hunchentoot on one of the generally less-well supported SBCL platforms, FreeBSD, and am using things like my hunchentoot-cgi interface and nuclblog which certainly haven't been well tested, and probably aren't used by anyone else either. All this adds up to a lot of moving parts that occasionally fail in less-than-graceful ways.

In fact, just today the site crashed because SBCL ended up in LDB. I try to avoid this and try to reset the thing soon after this happens, but occasionally a good chunk of time passes. Until recently, I was seeing very frequent failures in which SBCL would run out of processes and the server would die. It turns out this was caused by hunchentoot-cgi and the way it was calling sb-ext:run-process. By calling sb-ext:run-process with :wait nil, the problem seemed to go away and all was right with the world for a couple of weeks. I'm not sure what's up with the latest failure and will keep on eye things to hopefully minimized the downtime if the server ends up back in LDB again.

Why am I using hunchentoot-cgi, you ask? Well, that seemed like the easiest way to use gitweb on my current setup. Hopefully it won't cause to many other problems. Thanks for your patience to this of you who actually might want to access the site.

[some years pass...] So, thanks to some gentle prodding and patches from Daniel Herring, I've finally gotten around to making clem and ch-image run on lisps besides SBCL. In particular, they both work on ccl and clisp, with ecl support hopefully not too far behind.

A combination of bogus things on my part, the occasional bug in the lisp implementation itself, and idiosyncrasies of the spec meant that various things didn't work right. The most-common offender was that I was assuming that things like double-float and fixnum defined classes, and that one could use these as method specializers. Apparently, the lisp spec says otherwise.

Also, there were a few bugs in ccl, especially in the 32-bit x86 version, that the ccl team fixed straightaway that allow trunk builds of ccl to load and run clem and ch-image. Thanks to R. Matthew Emerson for the prompt fixes and to Gary Byers for the helpful discussion about whether or not (setf (find-class 'foo) (find-class 'bar)) should create a new type for foo.

In any event, clem can be found at http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=clem.git and ch-image at http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=ch-image.git.

I should point out that these changes haven't found there way into official releases, per se, so if you want to use this stuff on ccl, clisp, ecl, etc... make sure you get the latest source from the git repo.

So now that I have more than a passing interest in chemistry (and therefore cheminformatics/chemi-informatics/chemoinformatics/whatever-you-call-it), I'd like to see what the state of the art is for representing chemical information and if there are any decent libraries for working with these representations. At first glance, I'm in luck. There's CML, the Chemical Markup Language, there's the Blue Obelisk set of projects for open source/open data/open standards in chemsitry, and there's the CDK, the Chemistry Development Kit. This all sounds promising. Let's dive in.

Working in reverse order, let's start with CDK. Google CDK and it shows up as the first hit -- even before the Cyclin Dependent Kinases. This is good. Now let's follow the link. Uh oh. It's a link to sourceforge. That's alright, we'll click through and hope for the best. Ah, not only is it on sf.net, but it's a wiki site: http://apps.sourceforge.net/mediawiki/cdk/index.php?title=Main_Page.

Ugh. Alright, I'll try to overcome my biases and keep plugging away. It's not that all wikis are bad, but rather that, IMO, they are a poor substitute for a properly designed web site for a project. They certainly have a place, but the idea that all web content gets wiki-ized can lead to some rather difficult-to-follow web pages, again, IMO. The wikipedia example is a good counterexample to my claim, but, most other wiki sites don't have the complete, polished feel of wikipedia. In any event, let's keep plugging ahead with CDK.

Ah, here we go. Two publications in the peer-reviewed literature. This should help give us an overview of what CDK has and where it is going. One is in the Journal of Chemcal Information and Modeling (although it seems that when the article was published it was called the Journal of Chemcal Information and Computer Sciences. Alright, sounds promising. Click through the DOI link, which takes us to a page of American Chemical Society, the world's largest scientific society. Surely, this being a paper about on open-source toolkit and ACS being a society for the betterment of society, this is going to be an open-access journal, right? Or at least an open-access publication in a mixed-access journal, right? Click on the link to get the PDF... Get PDF -- WRONG! $30 for 48 hours of access. Damn. Ok, well, let's get the other paper, there were two on the CDK wiki. The next one is in something called Current Pharmaceutical Design. Uh oh. This doesn't sound promising. And, sure enough: "The full text electronic article is available for purchase. You will be able to download the full text electronic article after payment. $55.10 plus tax." This isn't getting us anywhere. At least CDK is open source. Let's go get the source. Well, first let's browse the documentation anyway.

Click over to the documentation look on sourceforge: http://apps.sourceforge.net/mediawiki/cdk/index.php?title=Documentation. How is the documentation different than what I was looking at before, or what's the difference between the documentation and the main page? Who knows. In any case, this looks promising: "A great source of CDK documentation or introductory reading is the CDK News, the quarterly newsletter of the CDK team." Click through. Ok, there's a picture of the (presumably) most recent issue, which is a link to the table of contents and a note about getting the PDF: "The full issue can be downloaded as PDF from http://sf.net/projects/cdk/". Hmm... Ok, click through that... And we're back on sf.net. Oh wait, that's not a link. Just some text. Cut the URL and paste into the nav bar in the browser... Hmm. Now we're at another sf.net page. So far we've got the "main" page, the "documentation" page and now the, presumably, "project" page. But now that we're there, we see that there is no mention of CDK News on this page. Damn. Alright, let's start clicking and see what we find. Ok, under the "Download -- Browse All Packages" link we get to a page that has CDK News on it. Maybe now we're getting somewhere. Click through that and we have a nice list of the various "Releases" of CDK News (this seems like an abuse of the Release mechanism, if you ask me -- these aren't different versions of the same thing, rather distinct issues, all of which shoud live on, but, OK, I think I see what they did there). Let's start at the beginning. Click on 1/1. Hmm... That just expands the HTML a bit to show another link for cdknews1.1.pdf. Ok, click that. Ah yes, no I remember why I hate sf.net. Do I get the PDF in my browser? NO! I get a window with a whole bunch of orange reminding me the name of the web site I know hate so much, some links to "share" the project (whatever the hell that means!), for related stuff and for forums (yet another set of pages to not get the info I want?) some google ads and some guy who looks like he hasn't slept in a week carrying stacks of cash, presumably, in an ad for nortel. Oh, and I link telling me who's providing this oh-so-handy mirror. Oh, and I almost forgot, some nice sponsor links! As opposed to the ads, I suppose. Where's my damn PDF? Who knows. Ah, this is helpful. Please use this "direct link": http://voxel.dl.sourceforge.net/sourceforge/cdk/cdknews1.1.pdf. Why the hell didn't they just give me that link the first place??? Ok, finally got CDK News 1.1. I suppose a (direct) link off of the CDK home page (or at least one of them) would have been to helpful for irritating newbies like myself. Ok, time to go read the first CDK News, since I can't get the peer reviewed articles about the project. More later.

The TIFF image file format has been around a long time, and lisp even longer. Yet, I couldn't find any common lisp libraries for reading TIFF images. Perhaps there's one out there I missed, but the only one I could find was my previous attempt, tiff-ffi, which consists of some wrapper functions around FFI calls to libtiff. I wanted a native common lisp TIFF library that wouldn't require the libtiff library so, at Robert Strandh's urging, I wrote retrospectiff.

The retrospectiff git repository can be found here.

Currently, there is no support for writing TIFF files, and only a fraction of the TIFF image formats are supported, but RGB and ARGB images, both uncompressed and with LZW compression, can be read. Grayscale support should come next and, hopefully, support for writing TIFF files before too long.

Enjoy.

So, attempting to get on MUNI this morning, I had a fun experience trying to actually purchase a ticket. To recreate the problem, try buying a ticket to get on MUNI at the Embarcadero station with only a US $10 bill. Turns out, you need to pay for your ticket with coins only.

Ok, so how do you get coins? Well, there's a machine that takes $1 bills and gives you a $1 coin. Has to be the only change machine that doesn't in fact give you "change", at lost not as one might expect, but rather changes your bill into a coin. Oh, and the machine takes $5 bills as well. Great. But I've got a $10. Now what?

Well, let's ask the oh-so-helpful person on the other side of the bulletproof glass talking at me through the loudspeaker. Why they have bulletproof glass when they only accept coins is a mystery, but that seems to be the case. In any event, she tells me I have to go over to the BART change machines which only accept $10 and $20 bills and use those machines which, much to my surprise, actually give out $5 bills in exchange for the $10 and $20 bills. Again, a new twist on the change machine. I've never seen a change machine that takes bills and gives you more bills, just as I had never seen a change machine give you a dollar for a dollar, but, OK.

Great, now I've got a $5 bill. Now what? Ok, go back to the $1/$5 change machine and get 5 $1 coins. Perfect, but, with the BART transfer, I only need $1.25. What do I do now? Well, ask the oh-so-helpful lady, of course. And, of course, she sends me back to the BART change machines. Well, OK, not the BART change machines, but the BART ticket machines, where you put in your dollar coin and then hit cancel and, lo and behold, the machine gives you 4 $.25 pieces for your $1 coin. Amazing.

All of this instead of just being able to buy a card that would automagically charge my credit card when I got on the damn train. Well, I guess such a thing might exist, but you certainly can't buy it at the train station, at least not at Embarcadero. All you have is a BART ticket station with signs all over the place saying NO MUNI TICKETS. So, it's not like a bunch of other people haven't tried to show up there and buy MUNI tickets, but, instead of, oh, you know, figuring out how to sell MUNI tickets, they put up a sign telling you to f**k off and leave you to go talk to the scary people behind the glass in order to figure out how to use three change machines to get the exact (coin) change required for the train.

Look, I'm all for SUPERTRAINS, but perhaps we should start by firing whomever is in charge for the ticket collection at Embarcadero MUNI and then we can work on a decent rail system for the bay area.

From tonight's Kirin takeout: "You will be showered with good luck before your next birthday." Well, let's hope so anyway.

So, now that the presidential race is heating up, what, exactly does it mean to "play the race card"? The McCain camp is accusing Obama of doing this, but I don't know what this even means and everyone talks about it like it's obvious. What are they trying to say here?

This one is pretty funny:

Ok, after a couple of years... I've finally gotten around to incorporating (some of) my changes to the cl-jpeg library into the upstream sources. There's a new version (1.023) up on the common-lisp.net cl-jpeg site.

To go along with this, the latest git sources of ch-image now use cl-jpeg.asd instead of my hacked up jpeg.asd. My jpeg project will now disappear and should be replaced with the upstream sources.

I know, I know... Why on earth would you want to run CGIs from hunchentoot? Well, while it may seem counterintuitive, I've found a need for this as I want to run the gitweb cgi interface behind hunchentoot and don't feel like setting up apache as the front-end to an otherwise happy hunchentoot site (this one), so I rigged up a little CGI interface for hunchentoot.

To use, check out the hunchentoot-cgi::create-cgi-dispatcher-and-handler function.

Ok, so I've finally gotten around to moving (at least some of) my projects over to git. The good news, besides having the code in a modern version control system, is that the repos are now publicly accessible. The list of projects can be found at:

http://git.cyrusharmon.org/cgi-bin/gitweb.cgi

SBCL recently removed the sb-ext:find-executable-in-search-path or whatever it was called function, so ch-util and ch-asdf were broken. The new ch-util and ch-asdf should fix that.

Ok, following closely on the heels of the last release, here are new releases of nuclblog, hunchentoot-vhost and hunchentoot-auth.

New features include:

• nuclblog now uses esc instead of str for textareas for editing blog entries so that, e.g., > doesn't get converted into > (thanks to Timothy Ritchey for the patch).
• the hunchentoot-vhost dispatch mechanism was simplified to remove the &optional vhost argument from dispatch functions, which means that standard hunchentoot dispatch functions can be used, allowing for the removal of create-virtual-host-folder-dispatcher-and-handler. Added the *virtual-host* special variable to replace the &optional vhost arg.
• Added some preliminary test code to hunchentoot-auth and exported some more symbols

The new releases can be found here:

• nuclblog 0.4.14
• hunchentoot-vhost 0.1.7
• hunchentoot-auth 0.0.11

Ok, a number of bugs and design flaws have been fixed. One can now be logged into multiple blogs on the same server with different user names and can log in and out of one without effecting the status of the other blogs. Also, some internal API cleanup for the blog handler functions. Oh, also the realm stuff in hunchentoot-auth was simplified and nuclblog now does a better job of keeping track of the information regarding which ports to use.

• nuclblog 0.4.12
• hunchentoot-vhost 0.1.4
• hunchentoot-auth 0.0.9

I just hope Nate Longshore's ankle isn't hurt too badly or it will be an interesting next couple weeks.

Nice play by Ezeff to seal the win.

After I released 0.4.5, it was pointed out to me that there is a new cl-who that uses *downcase-tokens-p* instead of *downcase-tags-p*. nuclblog 0.4.6 has been changed to address this and, as a consequence, requires cl-who 0.11.0 or later.

Also, as a bonus, there's a new release of ch-util (0.3.3) that cleans up the build system a bit, removing some hacks I thought were neat when I was first learning ASDF and restoring buildability on (at least some) non-SBCL lisps. Thanks to Michele Pasin for the bug reports.

last time it was hunchentoot-vhost (of which there is a new release as well), this time it's hunchentoot-auth. Along the way, there's a new nuclblog release that supports the latest and greatest virtual hosting and user authentication facilities in hunchentoot-vhost and hunchentoot-auth, respectively.

• nuclblog
• hunchentoot-auth
• hunchentoot-vhost

I've updated hunchentoot-vhost to include Edi's copyright, as a couple of the functions are blatant cut-and-paste-and-edit versions of hunchentoot functions. But, more importantly, I've also updated nuclblog to work with hunchentoot-vhost so that now you can 1) have multiple blogs per host, but also have multiple hosts going, each potentially with multiple blogs.

• nuclblog
• hunchentoot-vhost

Of course, the down-side to this is that it introduces yet another dependancy to nuclblog.

I've made a rather trivial extension to hunchentoot that allows for so-called virtual hosting such that one can host web sites for multiple domains in a single server. hunchentoot-vhost project page and hunchentoot-vhost_0.0.2.tar.gz have more details.

No examples or documentation other than docstrings yet.

apparently this is the preferred mode of transportation in dot-com-land:

I've made an asdf-packaged version of the Latin Modern fonts for use with cl-pdf. One step closer to being able to wean myself of latex.

Now, if somebody would just make a nice beamer equivalent for cl-pdf, I'd be all set. Well, minus all the equations and stuff.

cl-pdf-lm 0.0.2

nuclblog 0.3.1 has been released.

Thanks to Timothy Ritchey for pointing out that I had broken the default startup on port 4242 (for http) and 4243 (for https). Now this works out of this box. Also, thanks to Brian Mastenbrook for pointing out a couple places where I needed some locks.

I've finally gotten around to packaging up a release of smarkup with cl-typesetting support. Unfortunately, there is no documentation and there are no examples for this yet. Hopefully those will appear in a future release release.

Enjoy.

smarkup

nuclblog now optionally supports https for authorized pages. Now the question is what to do about comments. Or, perhaps more to the point, how to allow comments with a minimal amount of blogspam.

Joe DeRisi is on the case of the missing bees.

The slides from my recent RECOMB2007 talk Comparative Analysis of Spatial Patterns of Gene Expression in Drosophila melanogaster Imaginal Discs can be found here.

nuclblog_0.2 can be found here. nuclblog is a rewrite of Brian Mastenbrook's cl-blog package for use with Hunchentoot. There's a demo blog included and it's fairly easy to get up and running:

(asdf:oos 'asdf:load-op :nuclblog-demo)
(nuclblog-demo:start-services)
(nuclblog::add-user nuclblog-demo::*blog* "demouser" "demopass")

Enjoy!

Oh, I should point out that an addition to ch-asdf, nuclblog also requires a number of other packages to run, including hunchentoot, cl-who, puri, cl-store and probably a bunch of other packages I'm forgetting at the moment.

yay. nuclblog now supports adding new blog entries and editing old ones. Hunchentoot is a joy to work with. I think all of the features of cl-blog that I used are in nuclblog now. There's no trackback, which I had disabled because of all the spam anyway. Also, it would be really nice to have some sort of comments facility, but that will have to wait for another day.

If anyone's interested in checking out nuclblog, let me know and I'll post a release of the code.

Ok, I've finally gotten around to, at least, starting to port cl-blog to hunchentoot. Right now it's pretty bare bones, but it has all my old blog posts and should see more features in the coming days. Let me know if things don't work as expected.

Moving on with the new releases... CLEM 0.4 has a whole slew of changes. The biggest change is largely under-the-covers and allows for n-dimensional matrices. Most stuff still expects 2-d matrices, but doing things like adding two 4-d matrices should work (multiplying them not so much, of course).

There are also a number of miscellaneous API cleanup issues. In particular, the mat-scale!/mat-add!/etc... variants have gone away, being replaced by (mat-add m n :in-place t). I'm not 100% the :in-place keyword is the way to go, but I got tired of maintaining an the ! version of every matrix operation, so I'm giving this a go. Feedback, as always, appreciated.

• http://cyrusharmon.org/static/releases/clem_0.4.tar.gz
• http://cyrusharmon.org/static/releases/ch-image_0.3.tar.gz

Following closely on the heels of the tiff-ffi 0.2 release is freetype-ffi 0.2 which also doesn't require gcc-xml-ffi just to load the sb-alien definitions.

Visit the project page or pick up a copy here: http://cyrusharmon.org/static/releases/freetype-ffi_0.2.tar.gz

[well, that was fast... fixed a bug in the tiff-ffi-gen depnedencies. now we're at version 0.2.2]

Some folks have complained about needing to install gcc-xml in order to get the libtiff FFI bindings working. Well, I've finally gotten around to separating out the generation of the xml file and sb-alien file from the C header files from the loading of a pre-existing sb-alien file. Now you can just download tiff-ffi and asdf:oos 'asdf:load-op it directly, without needing gcc-xml-ffi. If you want to rerun the gcc-xml-ffi auto-generation stuff, you can just do:

For details see the tiff-ffi project page or download it from http://cyrusharmon.org/static/releases/tiff-ffi_0.2.2.tar.gz.

As an added bonus, tiff-ffi no longer requires UFFI.

Enjoy

In the spirit of trying new things, I've decided to try using vox to blog TED. It's my first time at TED and it's an interesting event so far. While there is at least one other computational biologist here, I get the feeling I'm the only lisp hacker in the audience, BICBW.

If you want to read more, check out: http://cyrus.vox.com/.

After some more help from Juho Snellman in tracking down some nasty bugs, including one in the debugging code down in print.c, I was able to get sbcl/x86-64/darwin up and running without the sb-ext:*evaluator-mode* hacks. This experimental version has been checked into the SBCL tree as version 1.0.3.16 and most of the tests pass. There are still test failures in float.pure.lisp, debug.impure.lisp, foreign-stack-alignment.impure.lisp and run-program.impure.lisp.

Test reports on x86-64/darwin (and other platforms to make sure I didn't break anything) are most welcome.

At Berkeley, when you file your dissertation, you get a lollipop that says "PhinisheD" on it.

Here's Olivia's impression of what I would look like when I filed:

If you're interested in reading about an atlas of spatial patterns of gene expression in Drosophila melanogaster, the dissertation has been filed and can be found here.

The appendix describes some of the Common Lisp packages used in creating the atlas. Hopefully a paper describing these in more detail will appear soon.

Hmm... the search engines don't seem to be aware that dissertation == thesis. Perhaps this will help.

Well, it's been far too long since I've updated this page to reflect the status of threads and what not on SBCL/MacOS. In the meantime, the lutex stuff has landed on the trunk and the threads stuff has been cleaned up a bit, but is still somewhat unstable and doesn't pass the threads tests without catching an illegal instruction error. To address this, I, along with some major help from Alastair Bridgewater, have added an experimental feature for SBCL to use mach exception handling instead of (just) BSD-style signals.

The good news, irrespective of threads, is that this fixes the long-standing "CrashReporter" problem that many have complained about and it makes it so that one can use GDB with SBCL. Previously, GDB choked on SBCL's strategy of using mprotect for protecting memory in non-exceptional cases by preventing GDB from stepping across the EXC_BAD_ACCESS (the mach exception equivalent of a SIGBUS or SIGSEGV) and mach using mach exception handlers gets around this. Anyway, this has been checked in to the SBCL trunk but, at least for the moment, should be considered experimental and must be enabled by building with the :mach-exception-handler feature. Oh, and there's no PPC port for this yet.

As for threads, they are still not quite there, but certainly seem better and I have been using them for development work for some time. Hopefully the added debuggability will help in tracking down the remaining issues.

Well, it's getting closer. The lutex branch no longer kernel panics, thanks to mutex locks around the i386_set_ldt calls, and the garbage collection-caused memory corruption seems to be fixed. So it builds, builds itself, and all tests pass, usually. Unfortunately, it's the "usually" that is the problem. About 10% of the time, the threads test hangs with a thread waiting for a mutex lock that it's never going to get. I'm not sure if the problem is a subtle race condition in the code or if there are problems with MacOS' pthreads mutex/condition variable implementation, but it happens often enough that there is definitely some sort of problem somewhere. Hopefully this will get merged onto the head before too long, after the 0.9.13 release. Oh, and slam.sh still doesn't work.

A while back I asked something to the effect of "how do you frob the EIP in a mach exception handler?" Well, the answer is that you get the MACHINE_THREAD_STATE (i386_THREAD_STATE in this case) via thread_get_state, adjust the eip in the thread_state_t, and then call thread_set_state to get the changes to take effect. I wasn't calling thread_set_state before and assumed that this would behave like a Unix signal handler where you can just adjust the machine context and everything happens automatically. In the case of mach exception handlers, you need to explicitly set the state.

SBCL makes extensive use of POSIX signals for such tasks as garbage collection, error handling, and ensuring that atomic operations are executed atomically. MacOS X supports POSIX-style signals, but there are some problems with Apple's BSD-style, as they call it, signalling mechanism. The main problem, viz SBCL, is that MacOS X's signal implementation and GDB interact in such a way that renders GDB basically useless for debugging SBCL. SBCL's strategy of protecting memory pages with mprotect and then using a signal (usually SIGBUS or SIGSEGV, and SIGBUS in the case of MacOS X) handler to either adjust the memory protection mode and take apporpriate action or to signal an error causes MacOS to issue a mach exception (EXC_BAD_ACCESS) which is then caught be the kernel and causes a SIGBUS to be sent to the offending process. Unfortunately, GDB can't be used to continue past the offending mach exception, so the process just continues to send the exception and never issues the signal to the listening process or moves the program counter past the offending instruction.

In addition to the SIGBUS debugging problems, the signalling mechanism of MacOS X on Intel poses other problems in that MacOS' delivery of SIGTRAP signals is unreliable. It generally works, but only about 95% of the time. This is unacceptable for SBCL's use as a mechanism for signalling when operations that are supposed to be atomic have been interrupted and that approriate action needs to be taken. We have worked around the SIGTRAP problems by using the UDA2 instruction to cause a SIGILL signal to be delivered to the SBCL process. This works reliably, but causes the MacOS X on Intel code to differ from other Intel-based platforms.

Finally, Apple's crash reporter doesn't realize that we might be using memory protections and the associated SIGBUS messages for non-crashing, expected behavior and generates a crash log message or, depending on the Crash Reporter preferences, a dialog message to appear on the screen.

These issues are enough to motivate me to consider using Mach exceptions instead of or, more likely, in addition to POSIX/BSD-style signals.

If only it were that simple. Gory details to follow...

So, now that SBCL works on MacOS X/Intel, and given that MacOS is proving to be rather recalictrant when it comes to running a threaded version of SBCL, I thought I would share my list of the top 10 reasons why the life of an SBCL developer is made needlesly difficult by the current state of MacOS X.

1. No OS Source. The source to Darwin used to be available. Apparently that is no longer the case, at least for the Intel version. This is rather unfortunate. When developing for Linux one can dive down into the source to see if, for instance, user provided thread stacks are available to be freed after a pthread_join or not. (Note: in this case, the source to the pthread libraries is in fact available, but, given the design of MacOS' Mach kernel, this is mostly glue around calls down the to the Mach thread layer, the sources to which are not available).
2. GDB can't step across an EXC_BAD_ACCESS/SIGSEGV. SBCL makes rather extensive use of signals (or, in Mach parlence, exceptions) in the "this is a somewhat unusual, but also expected event and should be appropriately dealt with"-sense, not the "this is an error, most likely caused by programmer error or system failure, maybe you should print an error message before you quit"-sense. One critical example of this is the use of memory protection to trigger a SIGSEGV (or SIGBUS depending on the archictecture) to inform the system when bits of memory are being written to. This is a normal event and it triggers a Mach Exception (EXC_BAD_ACCESS) that GDB cannot step across, attempting to do so just causes the event to be refired. Setting gdb to "handle pass noprint" this type of exception just causes GDB to hang there. This makes GDB basically unusable, except for certain cases where you can attach to a running core, which will then proceed to work until an EXC_BAD_ACCESS is triggered again.
3. INT3 traps are not reliably delivered. Another example of where a signalling mechanism is used to handle slightly unusual, but totally expected, cases is the use of the INT3 trapping mechanism. SBCL uses this for error handling and, especially, as part of the mechanism for achieving fast atomic operations without having to go to the kernel for a lock. INT3 traps basically work on MacOS, except when they don't, which is about 2-5% of the time. Basically the trap signal is just lost and is not reliable delivered to the signal handler. This causes all sorts of problems for a system that expects to get these traps and was the source of major headaches in the MacOS X/Intel porting effort.
4. sem_init is not implemented. sem_open is implemented, but this takes a pathname and is a much more expensive call than creating an anonymous semaphore. The ironic thing is that the underlying Carbon and Mach APIs do support anonymous semaphores and they machinery to associate file system path names to Mach semaphores (one presumes) for use with sem_open. It would seem trivial to support sem_init.
5. The mach semaphores are a private API. Here's a useful API for doing semaphore stuff, but for some reason it's a private API. One can use Carbon semaphores, and can link in the Carbon framework, but this seems rather unneccessary.
6. Problems freeing a thread's stack(?). If I provide a stack for a thread to use with pthread_attr_setstack or pthread_attr_setstackaddr, I see major problems with the threads test suite if I free the stack. If I don't free the stack, the test suite is happier, until the kernel panics. See below.
7. Kernel panics. I have seen quite a number of kernel panics caused by (one presumes) use or misuse of the pthreads and semaphore APIs. I could understand it if this were a KEXT or even a root process, but these are all user processes. No user process should be able to so easily cause kernel panics.
8. No futexes. MacOS has a whole bunch of different locking APIs, none of which seem as nice as futexes. It would be great if the kernel (or a KEXT?) could provide futex support for MacOS. I'm assuming that it's not just SBCL and that other language environments, databases, and other highly-concurrent applications will take advantage of futexes and their efficient locking properties on Linux. It will be a shame if those applications are relegated to only using pthread condition variables and mutexes on MacOS X.
9. No POSIX RT signals. The POSIX RT signalling stuff provides for much more reasonable behavior of the delivery of signals than the original POSIX stuff. We have had to jump through hoops to get the threaded version of SBCL as far as it is without RT signals. It would be great if future versions of MacOS X supported RT signals (I would be happy to trade Spotlight for RT signals, although I have a hard time seeing a guy in jeans and a black turtleneck running around a stage talking about how great POSIX RT signals are :) ).
10. LDTs are not reused even after being freed. This is a rather obscure bug, and certainly one can manange their own set of LDTs, but the LDT API provides a way to return these to the OS, however they are not recycled and one runs out of LDTs after 0x2000 LDTs, or so, have been set up.

Anyway, perhaps this degree of systems inadequacy is present on all Operating Systems and perhaps there is a certain amount of pain in making things work properly on new OS/architecture environments, but when one compares the UNIX guts of OS X to the guts of, say, Linux or Solaris, OS X appears lacking and makes life difficult for the UNIX application programmer. I'm sure there are many benefits to Apple's microkernel architecture, but there's still a long way to go before it catches up to the other modern UNIXes, at least inasmuch as one treats MacOS as a UNIX, which is what is needed for the low-level of a sophisticated, cross-platform language environment such as Lisp in general and SBCL in particular. If these areas were addressed, it would make it easy for SBCL developers (those who use SBCL, not just those who develop and maintain SBCL) to both develop cross-platform lisp tools and to use MacOS X's sophisticated features, such as Cocoa, QuickTime, Aqua, Bonjour, etc... to develop first-class MacOS applications using SBCL.

On the positive side, this MBP is blazingly fast for SBCL development. It compiles all of SBCL in a hair longer than it takes my desktop x86-64 box on linux, and in half the time of a 2x2GHz G5 desktop.

Ok, I finally got around to hacking up a page listing my lisp projects. It can be found here. Enjoy!

Ok, an unofficial SBCL 0.9.10.37 binary for x86/darwin is available. Hopefully this will be the last binary released before the official 0.9.11 release. The last release had two problems. Some tests were failing due to the way they were being run and SLIME couldn't connect to SBCL due to some changes to cl:listen. Those issues have both been fixed in this release. This should work well enough to tide folks over until the 0.9.11 release, both for running this directly and for building new versions.

At long last, a binary release of SBCL 0.9.10.35 for x86/darwin is available.

Hopefully there will be an official 0.9.11 binary for darwin/x86 on the SBCL home page at the beginning of next month. Until then, this should allow one to build from current sources on x86/darwin without needing to do a cross-compile. Enjoy!

Ok, I think I've got a fix for the stability problems I was seeing with x86/darwin SBCL. At first everything looked great, then I noticed some sporadic failures. Doing things like running SBCL while the system load was high seemed to exacerbate the problem and increase the likelihood of failing with a SIGSEGV. After much debugging, telepathic and otherwise, and thanks to the help of Juho Snellman, Alastair Bridgewater and the rest of the #lisp crew, it became apparent that the problem was that the SIGTRAP handler wasn't reliably being called. I made some test cases that showed this to be the case, independent of SBCL, and that also demonstrated that the problem exists with Mach exception handlers as well. So, now what?

Well, thankfully Andrew Pinski and Alastair Bridgewater both suggested using x86 instructions that would generate a SIGILL instead and using that instead of SIGTRAP. Sure enough, that seems to ensure that the signal handler is reliably called. This means that SBCL on x86/darwin finally seems to work, for real this time. Knock on wood... I'll commit the changes and roll a binary for public consumption sometime in the next day or so.

Thanks to everyone who helped me debug and workaround this problem. It would be great if Apple would consider making sure that SIGTRAP is reliably called when an 0xCC instruction is encountered on x86. I've got testcases if you want them.

Also, if anyone knows how to get at and modify the EIP inside of a mach exception handler, let me know. I suppose digging through the GDB sources should provide some insight.

Ok, so on the MBP, after migration I was getting this dialog box over and over asking me to connect to my old machine when starting mozilla, firefox, camino, etc... or when opening up certain pages. It was driving me insance.

Turns out it was a bogus alias in "/Library/Internet Plug-ins" for the Real Player plug-in. Deleting these aliases gets everything back to normal. I doubt they would have worked anyway, being PPC plug-ins, even if the alias wasn't bogus. I'm not about to go find out though.

At leaset things are back to normal.

Now, to figure out why the fortran compiler can't build matlisp...

Experimental support for x86/darwin has been added to the SBCL source tree. No need to use my patch anymore, just grab the latest from source. Of course one has to cross-build at this point. I'll put up a binary release shortly that will enable folks to grab the release and the source and build it themselves without resorting to a cross-build.

Ok, this patch seems to work pretty well. Feedback and more testing welcome.

There's one remaing issue which is that we should consider using a sigaltstack so that signal handlers get a stack that is properly (16 bytes per the ABI) aligned. Currently, we don't do so and this might leave the door open for bad things to happen.

Ok, everything seems to be working now. I'll make a patch later today or tomorrow and hopefully this will hit the tree sometime in the next week. Thanks to Alistair Bridgewater, Juho Snellman, Reaper, Christophe Rhodes, Andreas Fuchs, et al., for remedial x86 assembly instruction, telepathic debugging help and moral support.

I've been working on trying to get SBCL running on Darwin/X86. Unfortunately, it doesn't work yet. But if anyone else wants to play along, here's a patch of the latest stuff.

Whoops. First pass at this didn't include the new files. New diff posted.

Currently, it builds and starts up, but dies in !COLD-INIT.

Comments and suggestions greatly appreciated.

Well, it took quite some work, but the generational garbage collector now works on PPC for both MacOS and Linux. The latest and greatest patch can be found here. The patch is to 0.9.9.29, but it should work on any of the very recent CVS versions.

I thought this was basically done a few days ago, but there was a really nasty and hard-to-find bug in the PPC assembly language routines where we were loading a 32-bit constant into a register with LIS and ADDI. The problem is that the ADDI instruction was sign-extending it's argument if the high-bit of the low-word was set. The solution is to use ORI which does not sign-extend. The problem manifested itself in a hosed LRA register, which, thankfully, Christophe Rhodes was able to hunt down.

This was not an easy task, but it was a great way to learn a lot about the internals of SBCL in a hurry. Thanks to everyone who listened to my griping and who chipped in with ideas and debugging help, especially Raymond Toy, Christophe Rhodes, Juho Snellman. And thanks to KingNato and Raymond Toy getting the ball rolling with the initial SBCL work and the CMUCL port, respectively.

Coming to a CVS repository near you soon. Hopefully.

Now, on to locking and threads...

So it turns out ASDF is very useful for packaging all kinds of documents in a form that can be easily distributed. With the help of some code that walks the ASDF system definition, I can automatically make tarball releases with all of the source code and other files, configuration files, shell scripts, documentation, images, etc... and have these be available to the user who downloads this distribution without worrying about where the package gets installed, relative paths, environment variables, etc...

This has been great, but I find myself writing a lot of code that looks like this:

  (let ((images-component
         (asdf:find-component
          (asdf:find-component
           (asdf:find-system "ch-imageio-test") "test")
          "images")))
    (let ((inputfile
           (asdf:component-pathname
            (asdf:find-component
             images-component "sunset-lzw")))
          (imagedir (asdf:component-pathname images-component)))
           ...

Well, it's a lot easier to do something like this:

(ch-asdf::asdf-lookup "asdf:/ch-imageio-test/test/images/sunset-lzw")

This is a URI of scheme asdf with host nil and the path (:absolute "ch-imageio-test" "test" "images" "sunset-lzw"). Notice that this isn't a path in the filesystem, but rather a path in asdf space. The first element of the path (after the :abosulte) corresponds to the asdf system and the other elements correspond to ASDF components so we can find them with the following code:

(defun asdf-lookup (path)
  (cond ((and path (listp path))
         (reduce #'asdf:find-component (cdr path)
                 :initial-value (asdf:find-system (car path))))
        ((stringp path)
         (let ((uri (puri:parse-uri path)))
           (when uri
             (let ((scheme (puri:uri-scheme uri)))
               (when (and (or (null scheme)
                              (eql scheme :asdf))
                          (puri:uri-parsed-path uri))
                 (asdf-lookup (cdr (puri:uri-parsed-path uri))))))))))

Again, note that the elements of the path are not file names, but rather then names of ASDF components from the following asdf file:

(defsystem :ch-imageio-test
  :version "0.1.2-20050724"
  :depends-on (ch-util ch-imageio tiff-ffi)
  :components
  ((:module
    :test
    :components
    ((:ch-imageio-test-cl-source-file "defpackage")
     (:ch-imageio-test-cl-source-file "test-ch-imageio" :depends-on ("defpackage"))
     (:module
      :images
      :components
      ((:tiff-file "euc-tiff" :pathname #p"euc.tiff")
       (:tiff-file "eucgray-tiff" :pathname #p"eucgray.tiff")
       (:jpeg-file "euc-jpeg" :pathname #p"euc.jpeg")
       (:jpeg-file "eucgray-jpeg" :pathname #p"eucgray.jpeg")
       (:tiff-file "sunset-lzw" :pathname #p"sunset-lzw.tiff")
       (:jpeg-file "sanfran" :pathname #p"sanfran.jpeg")
       ))))))

It's a trivial little hack, but it saves me 6 lines of code all over the place.

Raymond Toy has recently added a generational garbage collector to CMUCL/ppc. Unfortunately, CLEM breaks it. Here's a release of the latest CLEM which breaks the gengc. I really need to get anonymous SVN working one of these days.

CLEM

Work continues on CLEM (Common-Lisp Egregious Matrix), my matrix math package for common-lisp. CLEM now has a bunch of macros which generate type-specific methods for various matrix operations so that I can do fast, non-consing matrix math operations like addition, multiplication, etc... CLEM uses the MOP to define a standard-matrix-class which serves as the meta-class for matrix classes. This allows one to define typed matrix classes as subclasses of matrix, with class attributes stored in the instance of the metaclass. So now we have matrices for {u,s}b-{8,16,32}, fixnum, integer, single-float, double-float, float, real, complex, number, and even t, although this last one is probably suspect.

In addition to the matrix types and the standard matrix operations (add, subtr, scalar multiply, matrix multiply, hadamard product, abs, log, etc...), aggregate operations (min, max, variance, sum), CLEM supports discrete convolution, affine transformation and a couple different types of interpolation, morphological operations (dilate, and erode), and thresholding. Most of these are relatively non-consing, although there are probably a few cases that need to be re-written in the new macro scheme.

Overall, it seems to work, but it is rather slow to compile and results in large fasl files. this shouldn't be too big of a problem, as I'd rather have a fast matrix package that took a while to compile than a slow package that compiles quickly.

Speeds are decent. I approach naive C algorithms and get to within a factor of 10 for highly-optimized matrix multiplication hand-coded in assembly. More benchmarking and further attention to things like the size of the blocks for the blocked matrix multiply would probably be a good thing.

ch-image

Using CLEM, I've developed a trivial image representation/manipulation package. I should probably follow the lisp tradition of calling this trivial-image except that this isn't really trivial, as it requires CLEM. I've thought about trying to abstract away the core matrix stuff from CLEM into a package that works on arrays, and then trivial-image could use that (trivial-matrix?), but I'm getting sidetracked. ch-image supports images of a few types including ub8, rgb8 and argb8. Adding additional types should be straightforward and I hope to work on this soon.

ch-imageio

ch-image is nice, but rather useless if you can't get images into and out of it. This is where ch-imageio comes in. ch-imageio does the conversion between either files in various formats, or from the results of other file reading packages that load the image files into their own format. Currently, ch-imageio supports the following: 1) reading and writing JPEG via the cl-jpeg library, 2) reading and writing TIFF files via libtiff using interfaces defined by gcc-xml-ffi, and 3) writing PNGs using Zach Beane's SALZA library.

gcc-xml-ffi

gcc-xml-ffi generates FFI definitions from C (and C++ code, sort of) using gcc-xml-ffi. Currently, it spits out sb-alien definitions, although the original incarnation did UFFI. In theory, other backends, like CMUCL's alien interface and CFFI should be fairly trivial, but I haven't gotten around to it yet. UFFI isn't really adequate here as it doesn't support callbacks.

ch-asdf

To facilitate generating xml files from gcc-xml and for generating common-lisp FFI declaration files from gcc-xml-ffi, I've made some extensions to asdf. I've also aped the SBCL asdf extensions for dealing with unix-dsos. This is a straightforward thing that everybody seems to do, but the less-obvious part, for me, was how to package them up in such a way that these extensions can be used by other asdf systems. ch-asdf was my attempt at solving this and allows me to add :unix-dso components without having to redefine what a unix-dso is and what is load and compile methods are in every asd file. Also, I can declare a component as a gcc-xml-c-source-file and the right things happen. Dependencies are sort of tracked, but there are a couple places where things break down. In any event, it makes writing asdf systems for packages that use gcc-xml-ffi and unix-dsos much easier.

tiff-ffi

tiff-ffi uses gcc-xml-ffi (and ch-asdf) to wrap the libtiff library. There are also some rather trivial glue functions that help facilitate things a bit. ch-imageio uses this to read and write TIFF files.

carbon-ffi

carbon-ffi allows one, in theory, to develop native Mac OS applications using SBCL. This sort of works and I have some screenshots of simple apps. I've even built "package"-style apps that allow for double-click launching. The negatives are that this is Carbon only (no Cocoa) and that in order to make this work one needs to use some undocumented Apple API functions. This proved to be a huge pain as getting this to work properly exposed some issues with SBCLs stack alignment where we were not properly aligning the stack on a 16-byte boundary, which was making AltiVec rather unhappy.

quicktime-ffi

Similar to the carbon-ffi, this is an FFI wrapper for quicktime. This also suffered from the stack alignment issue, but these are now fixed. Now one can use the QuickTime API from SBCL directly to make and read movies, etc... I havent't tried the GUI stuff (QT movie playing functionality, e.g., but it should work).

congeal

I have implemented a version of the congealing algorithm in common-lisp. This uses clem and the various image stuff to learn a set of transforms that bring a stack of images into registration and can learn the shape of the item represented in the stack of images.

clsr

I have begun to connect SBCL up with R so that I can evaluate R expressions from common-lisp using the R C API and can get the results back to lisp. Next steps are to have a representation of lisp data objects in R and vice versa so that I can, for instance, call lisp functions from R and to connect up the R plotting stuff so that I can make nice plots from SBCL. This is an area where a clean common-lisp API that wraps the R graphing APIs might be a nice thing.

ch-photo

ch-photo is a library based around FFI definitions to the dcraw package to read NEF and DNG files (come to think of it, this should be rolled into ch-imageio, but that hasn't happened yet). In addition, ch-photo contains some scripts for importing RAW files and for organzing them into file system heirarchies based on date of import and file type.

fftw-ffi

fftw-ffi is a wrapper for the fftw (Fastest Fourier Transform in the West) library. In addition to the FFI stuff, there are routines to translate data between ch-image and fftw compatible representations so that one can use fftw to do ffts of images in ch-image.

SBCL stack alignment issues

In order to get carbon-ffi and quicktime-ffi working properly, I had to fix some bugs with stack alignment on ppc. Thanks to Gary Byers for pointing out the bug after I was at wits end with bizarre results coming back from quicktime due to misaligned stack data being munged by altivec (without complaints). These have no been fixed and everything seems to be OK here.

callback fixes

The SBCL developers have been working on adding callbacks to SBCL. The initial ppc port had a number of bugs that caused problems with non-32-bit arguments, long longs, mixing arguments of different sizes, etc... Raymond Toy fixed a bunch of these problems for CMUCL and I ported these over to SBCL. After the initial round of fixes, there were some more problems with how the arguments got pulled off of the stack in the lisp trampoline, but these have been fixed as well. These patches have not yet hit the tree, but hopefully this will happen after the 0.9.8 release.

SBCL sb-alien field alignment issues

Finally, in order to make the carbon-ffi and quicktime-ffi packages work properly, I had to deal with the fact that some of the MacOS toolbox data structures use a bizarre alignment scheme that is a holdover from the m68k days. The bad news is that a lot of these are core structures for things like graphics and IO. In order to support these weird alignments, Apple's hacked up version of GCC has some pragma directives that take care of the alignment issue. Fortunately for me, someone else must have been having similar problems as the CVS versions of gcc-xml started dumping out the offset of structure members. Now that the quasi-compiler was giving me this information, I had to hack up SBCL's alien interface to allow me to use it to specify non-standard alignment of struct elements. This works and now we can properly use these funky MacOS structures.

Ok, that's the overview. One of the main missing pieces is documentation. I need to go back in and document all of this stuff soon. Perhaps that will be my first New Year's Resolution: I well document as much or more code than I write this year.

Happy New Year,

Cyrus

For both of the other people using callbacks in SBCL on PPC, I've fixed some more callback bugs. The latest patches sent to sbcl-devel fix the following:

• Arguments of mixed sizes. Before, we were calculating the offsets of the arguments on the stack wrong. Bascially, we were moving up the stack by the size of the next argument, not the size of the current argument so that if you passed in a double followed by a float, or vice versa, you would get the wrong results.
• Callbacks with lots of arguments. Before when we ran out of registers, we would throw an error. This is a mistake. What we needed to do was to copy the registers onto the stack so that the alien-callback-lisp-lambda-wrapper or whatever it's called could get the registers off of the stack. Once we run out of registers, we don't need to do anything as the value is on the stack.

Previous fixes contained in the latest patch set include:

• Ported Raymond Toy's CMUCL fixes for non-32 bit callback args (doubles, long longs, chars, shorts (although these didn't work in combination before the latest patch)).
• Ported Raymond Toy's CMUCL fixes for calling vararg functions. This isn't a callback fix, but rather an ffi-fix, but it's in roughly the same area. The problem was that we need to put the arguments on the stack as well as in the registers when calling vararg functions.

Speaking of rtoym, there's still an outstanding issue in SBCL that he fixed in CMUCL. One of the ffi param registers is being used as reg_FDEFN. This should be moved down one, I think. I'll look into this after the callback changes hit the tree.

Thanks to rtoy for the cmucl fixes. I think the new fixes are going to affect CMUCL as well, so hopefully he'll find time to add those in, assuming he's not too busy with the gengc port! :-) Go rtoy, go! Thanks to luis and the cffi crew for motivating test cases and to lemonodor for complaining about the too many args thing.

Well, I've been playing with Aperture for a couple of days now. My overall impression is that I like it much more than I thought I would. I have to admit to buying it before I was quite sure 1) what it did, 2) who it was marketed to, and 3) why I needed it. I think I understand who it was designed for and what it does now. It's a digital photography productivity tool. It's basically iPhoto for professionals. I have developed my own workflow system for importing RAW images from digital cameras (or CF cards), for managing these files and for editing and printing pictures. Aperture essentially wants to do all of these things. And it seems to do a pretty good job with most of these things.

The best aspects of Aperture are that it addresses a few problems that I was having before but that I mangaged to convince myself weren't that big of a deal. Once you have solved these problems, you really don't want to go back. I imagine every digital photographer has there own unique system, and mine was certainly quirky, but I had a comfort level with my scripts that did the importing, bridge to do the initial screening and RAW conversion, and photoshop to do the editing, adjusting, retouching and printing. Aperture rethinks this whole approach by providing a single application that does all of these tasks. I was reluctant to hand-off control, and I still have not given aperture the task of importing my images as I don't quite trust it. On the other hand, Aperture's approach has two major advantages: seemless viewing, adjusting and printing of pictures and the ability to print many pictures simultaneously. These aspects alone are enough to make me want to remove Bridge and Photoshop from my dock.

The on-the-fly adjustments provide for a much more streamlined approach to editing lots of pictures, but, moreover, I no longer have to maintain different sets of imported and edited pictures. There all in the same place and when I make adjustments to an image, those adjustments are dynamically applied to the original image, so I don't have to keep lots of copies around.

The ability to print lots of pictures in the same batch may sound like a trivial feature, and was supported in iViewMedia Pro, but I couldn't figure out how to do this from Bridge and Photoshop. One of the strengths of Aperture is that Apple has put so many features in one place that you can spend less time context-switching (metaphorically, not unix-process-illy, speaking).

On the downside, Aperture wants to manage a big-old database of all of your files, so things have to be imported into projects within Aperture, all of which live in your Aperture Library. This is very much like the iTunes approach to asset mangagement and I find it to be rather annoying. I would rather manage my files how I want and have Aperture deal accordingly, rather than have to import everything into Aperture's monolithic database. I'm worried about what happens if that were to get corrupted. (And I have had a couple of crashes when importing large batches of RAW files, although I have not had any data corruption issues.) Also, this means that the Library can only be as large as your volume. This isn't a huge deal as one can always stripe disks and make bigger partitions, but if you're hoping to use space on multiple disks, version 1.0 doesn't allow this. It would be great if you could dynamically add spaces to the library. The whole library thing is a bit annoying, but so far the benefits far outweigh the cost of having to import things into Aperture's Library.

It still feels a bit raw as it as version 1.0. The HTML galleries look very nice, but there is no way to customize the CSS, only a few preset options. This is fine for a consumer application, but not for a professional application. Hopefully this will get fixed. I've had some bugs printing large batches where after 8 or 10 prints they all start coming out solid black. Obviously not good. I had a few crashes importing NEF files. The interface is a bit hard to get used to with all of the little buttons and what not, but it grows on you. The concept of project/folder/album/gallery is a bit counterintuitive and strikes me as a case of arbitrary distinctions being made in the name of usability that just go to confuse things. What's the difference between an album and a folder? Well, an album can have pictures in it, but not folders, and a folder can have albums in it but not pictures. It all seems a bit contrived and a simpler heirarchy would seem to make more sense to me.

All in all, it's a joy to use and I feel like I'm much more productive with Aperture than without. I hope it continues to evolve. And I want a 30-inch monitor, or two, to use with it.

Strengths

• Dynamic adjustments means that things like levels, white balance, sharpness, etc... can be adjusted at any time.
• Great for printing out large numbers of different pictures
• Nice-looking web galleries
• Don't need to keep lots of different versions of files round - Aperture does this behind the scenes, although I think it is doing much of this dynamically so it doesn't need to store as much data as a PSD file for each version

Weaknesses

• HTML galleries aren't customizable
• Confusing Folder/Project/Album/Gallery heirarchy
• Printing Bug(s)
• Importing Bug(s)
• Have to store all of your images in Aperture's library

Ok, Thiemo Seufer has committed a change to gc-common.c that gets rid of an extra instruction cache flush on non GENCGC platforms (that is, CHENEYGC). Turns out this is a no-op on x86 and x86-64 due to their cache-coherency, but might be required for a generational collector on other platforms should one ever get ported to, say, PPC or MIPS.

As for the additional sync call, this may be necessary on multi-processor systems as the instructions seen in the icache by one processor may be seen as data in another and the data-cache syncing is necessary. So this should stay as is.

Here are some cl-bench results from recent SBCL builds on PPC with a couple minor caching changes. The first is the vanilla SBCL, the second is with an os_icache_flush removed from the cheneygc and the second is with this change and a sync removed from ppc_flush_cache_line. Here's the patch:

 
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/runtime/gc-common.c,v
retrieving revision 1.33
diff -u -r1.33 gc-common.c
--- src/runtime/gc-common.c     5 Dec 2005 18:01:29 -0000       1.33
+++ src/runtime/gc-common.c     10 Dec 2005 21:47:35 -0000
@@ -314,8 +314,10 @@
         fheaderl = fheaderp->next;
         prev_pointer = &nfheaderp->next;
     }
+#ifdef LISP_FEATURE_GENCGC
     os_flush_icache((os_vm_address_t) (((long *)new_code) + nheader_words),
                     ncode_words * sizeof(long));
+#endif
 #ifdef LISP_FEATURE_GENCGC
     gencgc_apply_code_fixups(code, new_code);
 #endif
Index: src/runtime/ppc-assem.S
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/runtime/ppc-assem.S,v
retrieving revision 1.7
diff -u -r1.7 ppc-assem.S
--- src/runtime/ppc-assem.S     23 Oct 2005 19:29:00 -0000      1.7
+++ src/runtime/ppc-assem.S     10 Dec 2005 21:47:35 -0000
@@ -574,7 +574,6 @@
        dcbf 0,REG(3)
        sync
        icbi 0,REG(3)
-       sync
        isync
        blr
        SET_SIZE(ppc_flush_cache_line)

 

Once again, here are some more new releases. ch-util and clem today. clem, in particular, has some MOP fixes that allow for inheritiance of meta-class options from superclasses that makes it easier to define matrix subclasses.

Ok, here are some more new releases. This time, it's gcc-xml-ffi, tiff-ffi, ch-util and ch-asdf.

Release notes to follow, eventually.

Here are some new releases of my matrix and utility packages.

Thanks to Gary Byers for pointing out that he had seen similar problems with funky lines in Mac OS windows when the stack was misaligned. For some reason SBCL's stack is misaligned when we try to call a C routine. This patch ensures 16-byte alignment and makes Mac OS windows (and QuickTime movies) look much nicer:

Ok, after a lengthy hiatus, I've been trying to get SBCL and Carbon to play nicely together again. Unfortunately, I seem to be at a roadblock. Here's what I've got:

As you can see, the window has some funky lines on it and the menu bar colors are hosed. This doesn't happen when I run essentially the same code from C. So something horrible must be happening somewhere like Carbon's initialization routines or something. I've used the undocumented hacks to get foreground operation and I've tried faking a bundle a la OpenMCL, but I can't seem to get things to look any better this. Suggestions greatly appreciated.

(I've posted this to the cclan list, but I thought I'd put a copy of this here for posterity's sake.) So I'm trying to package up some C code to use as a test for gcc-xml-ffi. I have some C code that I'd like to use to generate a shared library, which will then be loaded by SBCL at runtime. To do this I can 1) hand-code the Makefile rules with the approrpriate platform-specific compiler/linker incantations to get the shared library, 2) use autotools, or 3) try to rig up some sort of asdf-based system to do this. I've done 1 for darwin and got 2 to work, but I wasn't super happy with distributing this giant mess of m4 files just to compile some shared libraries, so I decided to explore 3. Actually, as I was considering this, I decided that I also wanted to use asdf to invoke gccxml to generate the xml for gcc-xml-ffi.

So far so good. SBCL has examples of how to do this in sb-posix (and somewhere else, IIRC this is a violation of OAOO, but let's put that aside for the moment.

But I've run into a couple issues:

Clearly, asdf is an extensible system and through the right extensions I can make this work, but if anyone has any suggestions or comments on how to address these issues, I'd appreciate it.

Here are some new releases of my matrix and image packages. Notable new features include (fastish) affine transformations and discrete convolution, various code cleanups, a bug fix in gcc-xml-ffi where I wasn't allocating space for the NULL at the end of C strings, etc... Enjoy.

(Note that the iamgeio download is rather large as it now contains some sample images.)

(Furthermore, note that I just bumped the version # for clem to fix an asdf problem and some code that was inadvertently commented out.)

Ok, it's time for me to bundle up some of this stuff and ship it out, in case anyone is interested. What is it, you ask? Why, some nascent libraries for image processing, matrix math, image i/o, interfacing with C libraries via gcc-xml, etc...

Some of this was released yeseterday, but there are brand new versions of all of this stuff. Comments/suggestions appreciated. Oh yeah, all of these pretty much require SBCL at the moment. I'd like to resurrect OpenMCL support and support CMUCL in the future.

Ok, now new functionality in here, other than the package mechanism, which is in fact in the chutil package. In any event, here are new versions of chutil and meta-ffi. Enjoy.

• chutil
• meta-ffi

Not that would want to, but if you want to pacakge up a distribution, typing make dist should do the right thing.

whiskey tango foxtrot indeed... thanks for the feedback Xach!

It seems the good people of #lisp aren't a big fan of i) autotools, ii) m4, and iii) needlessly large distributions. Addressing these three complaints in one fell swoop, I present new versions of chutil and meta-ffi.

• chutil
• meta-ffi

Ok, I'm finally getting my lazy act together and trying to package some of the software I've been working on for release. The first two packages are rather trivial, chutil and meta-ffi. They don't do much. chutil is some relatively trivial utilities and meta-ffi is a MOP metaclass for doing FFI to foreign structs. meta-ffi allows for making a :metaclass uffi-foreign-struct-class and then define slots with :foreign-field class. It's pretty trivial but a nice way for me to learn how to make a metaclass. There's no examples or documentation, but I hope to fix that in the near future.

• chutil
• meta-ffi

For those of you have run into the 128M dynamic space limit in SBCL on PPC, the following patch provides for a 768M heap. It's rather trivial, and I imagine most folks who have run into this limit can come up with something like this or better, but nevertheless, here are the parameters I've been using successfully for the last few weeks.

I'd like to see something like this rolled into the main SBCL repository at some point, but until then, enjoy, or at least let me know if you have a better approach. On a related note, porting the gencgc to ppc (or migrating rtoy(?)'s CMUCL version of it to SBCL) would be a great project, but unfortunately I'm not sure I have the skills and I definitely don't have the time at the moment.

Index: sbcl/src/compiler/ppc/parms.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/compiler/ppc/parms.lisp,v
retrieving revision 1.19
diff -u -r1.19 parms.lisp
--- sbcl/src/compiler/ppc/parms.lisp	11 Mar 2005 17:10:10 -0000	1.19
+++ sbcl/src/compiler/ppc/parms.lisp	18 May 2005 17:49:34 -0000
@@ -93,19 +93,19 @@
 ;;; FIXME: this is a gross violation of OAOO, done purely to support
 ;;; the #define of DYNAMIC_SPACE_SIZE in validate.c -- CSR, 2002-02-25
 ;;; (these numbers should match dynamic-0-*)
-(def!constant dynamic-space-start   #x40000000)
-(def!constant dynamic-space-end     #x47fff000)
+(def!constant dynamic-space-start   #x10000000)
+(def!constant dynamic-space-end     #x3ffff000)
 
 ;;; nothing _seems_ to be using these addresses 
-(def!constant dynamic-0-space-start #x40000000)
-(def!constant dynamic-0-space-end   #x47fff000)
-(def!constant dynamic-1-space-start #x48000000)
-(def!constant dynamic-1-space-end   #x4ffff000)
+(def!constant dynamic-0-space-start #x10000000)
+(def!constant dynamic-0-space-end   #x3ffff000)
+(def!constant dynamic-1-space-start #x40000000)
+(def!constant dynamic-1-space-end   #x6ffff000)
 
 #!+darwin
 (progn
-  (def!constant linkage-table-space-start #x50000000)
-  (def!constant linkage-table-space-end   #x51000000)
+  (def!constant linkage-table-space-start #x0a000000)
+  (def!constant linkage-table-space-end   #x0b000000)
   (def!constant linkage-table-entry-size 16))
 
 ;;;; Other miscellaneous constants.

My knowledge of unix memory management is fairly limited, but i was unpleasantly surprised to discover that the following:

(defparameter a1 (make-array '(1024 1024 128) :element-type '(unsigned-byte 8)))
(gc)

will cause SBCL to rather unceremoniously exit. Apparpently, there are some compiler settings that determine the runtime memory layout and the gc sections only get 128MB each. I need to do some homework to figure out how to address this. I'd like to be able to use SBCL for image processing but, images from the d2x are 75MB each and I'd like to have more than one in memory at a time (or at least more than one representation of an image - especially a floating point representation and 16-bit per channel per pixel representation.

I really like the headline Times Panel Proposes Steps to Build Credibility. I like that the didn't use words like enhance, improve, etc... and just flat out admit that the Times needs to build credibility. We'll see what this all means. In a limited defense of the New York Times, it must be somewhat of a thankless job, as everybody on all sides of all ideological debates is going to at some point take issue with what's in the paper. Nevertheless, the Times behavior in the past decade has been only marginal better than bona fide propoganda outlets such as Fox News (or Talon News for that matter).

One thing the article says is that they will expand the coverage of religion. This is good, if they mean covering religion as the cover the crime beat, like who is peddling what sort of theocratic, dominionist crap to whom, or perhaps even as literature or theater, which it is, of course, and of course including a strong element of multi-level marketing and brainwashing, but I digress. If by coverage, they mean lifestyle pieces showing how wonderful the works of the righteous are, I'll pass think you.

Curtailing the use of anonymous sources is a good thing. I think one needs to be careful with anonymous sources and can't dismiss them entirely. While not black or white, a major, initial, test for a justification of anonymity should be do my claims support the official position of the organization that I claim to represent? If so, why the anonymity? You can imagine pentagon staffers claiming "hey, all this WMD baloney, it's crap!" might want to remain anonymous for fear of losing their jobs. On the other end, "senior pentagon officials" claiming on background that we know where the weapons are and therefore we need to go to war? That's just a way of skirting accountability and passing of rumor and innuendo as justification for war. That doesn't meet the test in my mind.

Anyway, cautious kudos to the Times. Clearly, modern technology (the internet, blogs, 24/7 news coverage, satellite telephones, etc... greatly enhance the ability to transmit, dessiminate and, more over, integrate, analyze and synthesize information from a variety of sources. It's high time the Times took a step back and asked how they can be doing a better job to bring the world the news, at least the news from a at-least-statedly-objective, American-but-at-least-in-theory-not-rah-rah-blindly-patriotic perspective.

I'm eager to see how this all unfolds. BTW, did they give out Judy Miller's email address yet? I'm sure there are some folks in the blogosphere who would like to ask her some questions. It's still amazing how much more airtime Jayson Blair got than Judy Miller's amazingly successful propoganda project.

After spending a couple of days trying to get things back to normal after upgrading to Tiger (trying to sort out compiler versions, fink packages, R and bioconductor, etc...), it occurred to me that the upgrade process for SBCL and OpenMCL has been a breeze. I haven't upgraded to the latest bleeding-edge OpenMCL as I've been using SBCL for most of my work these days, but I understand it runs and OpenMCL 0.14.3 seems to still be running OK. SBCL builds and runs great with Tiger. Thanks to Gary Byers, Brian Mastenbrook and the rest of the OpenMCL and SBCL development teams who made sure that the Tiger upgrade would be a snap, at least from the (or should I say my) open-source lisp point of view.

Now, back to figuring out which combination of the fortran and C compilers (and which libraries) will actually compile R and successfully install Bioconductor...

Oh, and thanks to the SLIME and carbon emacs guys too!

LTU has a piece on a special issue of JStatSoft on the demise of LispStat. I haven't read through all of the articles yet, but it does seem clear that lisp-stat isn't much of a player these days and that R is the lingua franca of the statistics world. This isssue is timely for me as I have been working in R on and off for the past few months and while I have come to appreciate some of the features of R, I definitely miss many, many nice features of common lisp when using R. S and R are shining examples of Greenspun's Tenth Law in practice. But at least the R guys were smart enough to build R in C by implementing a lisp-like intermediate language. Nevertheless, there are a ton of things in common lisp that are either just starting to make their way into R or are way off on the horizon that the R community will need to grapple with eventually. Then there's the whole issue of completeness, standardization, maturity of the language, etc... But, in any event, it does seem to be the case that R has taken over the, academic at least, statisticl computing landscape.

But, I think it may be too early to write lisp off for statistics off just yet. I'm just delving into the land of lisp-stat, but it doesn't seem to me that lisp-stat is the end-all, be-all of statistics in common lisp. Just as the R and S guys no doubt learned from lisp-stat, I'm sure there are lessons from R that could be applied to a lisp-based statistical package. Witness matlisp as an example of a lisp-based package inspired by matlab. Speaking of matrices and linear algebra, and granted I need to catch up with the past 15 years of progress in lisp-stat, but it seems that lisp-stat wasn't designed with what I view as modern statistic in mind. When I think of modern statistics I think of big matrices and methods to operate on them efficiently. The main focus of lisp-stat seems to be on dynamic graphing tools and matrices and linear algebra get 5 or 6 pages in the orginal book. I'm sure this may have changed, but in R, as in matlab, the focus seems to be on efficient matrix math from the get go. As for the dynamic graphing stuff, yes, I can see this being nice, but I see this as more of an "add-on" thing than a core feature. A core feature (or a damn important library) of any statistics package, on the other hand, needs to be high-quality print graphics. It's great to be able to spin and zoom and all that, but ultimately scientists want to present their data for publication. R has 1) a good interface to BLAS/LAPACK (with decent notation), 2) great graphing facilities (even if support for png, jpeg, etc... are a bit janky and require either X or ghostscript), 3) a ton of in-depth (if not complete) library packages for doing a little bit of everything.

I think one of the major things holding lisp-stat back was the lack of good freely available common lisp implementations. The lisp-stat book claims that (at least in 1990) common lisps were expensive and that the free (subset of) common lisp available didn't have a compiler. Obviously things have changed a great deal since then. I haven't been around long enough to know the history well enough to do all the previous implementations justice, but I can say that first CMUCL and later SBCL have done an amazing job of bringing a fantastic common lisp implementation to the masses. All that aside, I still think that lisp-based systems have a deployment problem. If you look at successful lisp based systems, I'd argue that most of them include their own (perhaps half-assed or limited) lisp implementations, emacs and autocad being cases in point. One of the advantages of emacs and R is that they both just build and run out of the box. There isn't this whole build the language, get the libraries, build them, get the application, build it, get the app. libraries, build them, etc... rigamorole. Granted R libraries aren't necessarily the cleanest thing in the world, but the whole R CMD INSTALL thing is easier, for some reason, for new users to grasp than the asdf install process.

Nevertheless, I'm still hopeful that the emergence of high quality, freely available lisp systems, and the success of at-least-partially-lisp-inspired (at least in the guts) stats packages such as R suggest that there's still hope for doing linear algebra and statistics in lisp. I'm looking forward to trying to do some of what I currently do in R in common lisp in the future.

Ok, so it looks like the kewl kids on #lisp don't want Xach to syndicate my blog on planet.sbcl.org as that's reserved for output from CVS commit logs, chandler/chavatar's sbcl bulid-tester (very cool, BTW), and comments from ranking SBCL developers. I still don't want all of my drivel going to planet.lisp.org, which means it's relegated to the SBCL category of my blog, which means nobody will ever see it, but that's fine with me, at least for the moment.

Ok, on to my next SBCL problem. Profiling fails on PPC. I sent a bug to sbcl-devel, but so far the interest in learning more about the problem or fixing it seems rather low. I guess nobody else is trying to profile SBCL code on PPC at the moment. Changing *timer-overhead-iterations* from 500000 to 100000 fixes sb-profile:report, at least in the trivial case. This is helpful, but I'm not quite sure what the 500000 was for in the first place, or, more to the point, why bumping this down to 100000 fixes the problem.

I still don't understand how or why we get a large negative number instead of an unsigned-byte. Clearly more investigative work is required here.

Now that Xach has set up planet.sbcl.org, I'll move my gripes, helfpful (yeah, right...) comments, opinions, etc... about SBCL to the new SBCL category of my blog and maybe Xach will be kind enough ot pick up the feed.

As a first entry, let me continue to try to convince the SBCL gurus on #lisp that changing the size of compact-info-entries-index is a good thing.

As I have mentioned before, I've been trying to generate FFI definitions for the MacOS Carbon APIs and have run into a problem attempting to run purify, which gets called automatically by save-lisp-and-die. (I'm ignoring the fact that without threads and callbacks on the PPC, the carbon integration is going to be lacking but Brian Mastenbrook has been working on callbacks and hopefully I, or better yet, somebody who knows what they're doing, will have a chance to take a look at threads and the required gencgc stuff one of these days. Now back to the original thread).

The call to purify fails, giving an error saying that 65536 not being an (integer 0 65535) or something similar. Purify (and perhaps other things?) attempts to create a compact environment that contains the same information as an existing environment in a more compact representation. There is a variable compact-info-env-entries-bits that is used to determine the size (in bits) of the compact-info-env-entries-index type, which, in turn, is used as the type of the compact-info-inv/cache-index and as the type of the simple-array compact-info-env/index.

As an environment gets more than 65535 entries, it becomes impossible to compact it do tue the size of the index into the underlying data structures. So the obvious thing to try is to boost the size of the index. I made compact-info-env-entries-bits 32 and have been using this for a few weeks (on sbcl 0.8.20.xx on PPC) with no problems.

Enough background. Now for some specifics:

Why is this necessary? Without this patch one can only have 65535 functions in an environment.

Here's a testcase that illustrates the problem:

(defparameter l nil)                                                                                             
(dotimes (i 65537)                                                                                               
  (setf l (cons (let ((g (gensym)) (z i))                                                                        
                  (setf (symbol-function g) #'(lambda () z)) g) l)))                                             
(purify)                                                                                                         

And the output is:

The value 65536 is not of type (UNSIGNED-BYTE 16).                                                               
   [Condition of type TYPE-ERROR]                                                                                
                                                                                                                

What are the impacts of the change? The compact-info-entries-index type itself is only used by the index and cache-index fields of the compact-info-env struct. These data structures hold indices into a table of the particular things being stored in this compact-info-env. Note that this change doesn't change the size of the things in the table, just the size of the index and the cached index into the table, AFAICT.

There is one potential problem. At some point a table-size gets calculated based on the number of names in the environment. We call primify to get a prime number size for the table. primify declares its argument x as an (unsigned-byte x) and then iterates over the odd numbers >= x until it finds a prime number. But positive-prime declares its argument to be a fixnum, so it's possible that we could have a table size that is > most-positive-fixnum which cause positive-primep to fail. I haven't run into this in practice, but we might want to watch it for this and/or come up with a better approach for getting the size of the table. There's a comment about using almost-primify from hash-table.lisp, but this must be an out-of-date comment, as I can't find this function in hash-table.lisp. But failing when we get larger than most-positive-fixnum is much better than failing when we hit 65536 entries in a compact info env.

I'm getting tired of flogging this horse, but I think it's important that this get fixed. If anyone disagrees with what I've proposed, I'd love to hear it.

Index: globaldb.lisp
===================================================================
RCS file: /cvsroot/sbcl/sbcl/src/compiler/globaldb.lisp,v
retrieving revision 1.36
diff -u -r1.36 globaldb.lisp
--- globaldb.lisp       12 Jun 2004 13:55:49 -0000      1.36
+++ globaldb.lisp       26 Mar 2005 15:03:13 -0000
@@ -476,7 +476,7 @@
 ;;;; compact info environments

 ;;; The upper limit on the size of the ENTRIES vector in a COMPACT-INFO-ENV.
-(def!constant compact-info-env-entries-bits 16)
+(def!constant compact-info-env-entries-bits 32)
 (deftype compact-info-entries-index () `(unsigned-byte ,compact-info-env-entries-bits))

 ;;; the type of the values in COMPACT-INFO-ENTRIES-INFO

So in my continued adventures in ffi-land, I decided it would be an interesting exercise to make a MOP metaclass that provided for transparent access to FFI structs. I took a first stab at this a while back using OpenMCL and it kinda worked. As part of my recent efforts to switch over to SBCL, I decided to pick this up again and see if I could make this work with UFFI. Sure enough, it works! And I've cleaned up the code a bit to create a package called meta-ffi which provides the MOP infrastructure and another package called vimage which uses the GCC-XML stuff and my GCC-XML-UFFI declarations to automatically parse the vImage (a MacOS framework for doing HW accelerated image manipulation) headers, generate an XML file of the appropriate declarations, generate a CL file with UFFI declarations and finally define a CL class which provides for transparent access to underlying structs. It's an awful lot of machinery, but it should be reusable in other contexts and might help write CL code that uses foreign libraries. Clearly this can be done without all of this machinery, but the point was to try to automate as much as possible. Now I can do something like this:

(defclass vimage-affine-transform ()
  ((a :accessor a :initarg :a :foreign-field '|a|)
   (b :accessor b :initarg :b :foreign-field '|b|)
   (c :accessor c :initarg :c :foreign-field '|c|)
   (d :accessor d :initarg :d :foreign-field '|d|)
   (tx :accessor tx :initarg :tx :foreign-field '|tx|)
   (ty :accessor ty :initarg :ty :foreign-field '|ty|))
  (:metaclass uffi-foreign-struct-class)
  (:foreign-type |vImage_AffineTransform|))

 (make-instance 'vimage::vimage-affine-transform
                 :a 1.0 :b 0.0 :c 0.0 :d 1.0 :tx 0.0 :ty 0.0)

The MOP stuff under the covers takes care of the foreign struct allocation and making sure that slot-value and setf (slot-value... do the right thing.

I think this is kinda cool, but I can't decide if it's really useful or just Brucio-cool. Time to do some more hacking to see if this really helps. I think it could prove useful for the development of Carbon apps in SBCL.

I'm sure there are more pressing matters in getting SBCL to 0.9, 1.0 and beyond, but I'd like to suggest that the SBCL_HOME/INSTALL_ROOT thing is too fragile and should be fixed. To require that INSTALL_ROOT be set at build and install and runtime seems overkill. I would prefer to see SBCL_HOME/INSTALL_ROOT be set at build and install time, leaving an sbcl executable that doesn't require the setting of SBCL_HOME. I think this will help make SBCL useable for new users. I understand that 1) this is a relatively trivial bug, 2) there is a number of simple (including some currently documented) workarounds and, 3) there are more important things to work on, but, nevertheless, I think it would help adoption of SBCL is this were to be fixed.

Ok, the DSL outage has been fixed. Now I can get back to work. I always think I'll be more productive without a net connection, but it's never the case.

In any event, the gcc-xml work continues. One problem is 64-bit support. C long long should be useable as args and return values, but UFFI doesn't support this. I'm not quite sure what to do here. Perhaps using SB-ALIEN and other native FFI interfaces might be appropriate for 64-bit stuff. It doesn't really matter if everything goes through UFFI or not, as long as it works, and it's probably easier to stick the 64-bit support in here directly rather than trying to convince KMR to support 64-bit types in UFFI, which seems like a losing battle. Might need some special C glue support too, which probably has no reason to be in UFFI.

A second problem is dealing with the huge number of declarations generated. Clearly processing the XML file and generating the lisp decls every time is not a good idea (nor is generating the .xml from the .h/.c every time) and compiling the lisp forms is probably not a good idea either. Loading a compiled fasl helps, but still yields a 30-60 second delay. Saving a lisp core might work, but I understand there are some issues WRT lisp cores on foreign libraries and I haven't worked through these yet. As a step in this direction, I tried save-lisp-and-die (SBCL's dump custom lisp core command) and things failed with some a 65536 is not an UNSIGNED-BYTE 16 error. Changing the element type of the compact-info-entries-index (or something like that) to be 32 bits fixed the problem, but I have no idea if this is correct much less wise, although it gets me one step closer to trying it out. More in a bit...

Work is continuing on generating UFFI-based lisp bindings to foreign libraries via gcc-xml. Now I can successfully parse all the GCC-XML declarations from Carbon.h. This (and some apparently undocumented (at least not officially) MacOS hackery) enabled me to do this:

So I've been working on automatically generating UFFI declarations from GCC-XML and it's almost there. The idea is that one feeds a C file (.c or .h) to GCC-XML which outputs an XML file containing XML elements for the declarations in the file, including function defintions, typedefs, structs, enums, etc... I feed this file into some code which parses the XML file and spits out a file of UFFI declarations, which is then eval'ed to produce the appropriate UFFI types and what not. This should make it easier to port code that uses OpenMCL's interface database to SBCL. Don't get me wrong, the interface database is very cool, but it is OpenMCL specific and it would be very nice to be able to develop Carbon applications, e.g., on SBCL. And there's nothing OS X specific about this; it should work on any platform with GCC-XML and a modern Lisp implementation supported by UFFI.

There are some potential issues:

1. Names: How should lisp names for functions, etc... be constructed? Should we attempt to map things like SuperCoolOpenGLBuffer_Fast to super-cool-open-gl-buffer-fast? Or should we just intern the C name yielding a function call like this (|EnableOpenGLMonkeyButter| ...) or whatever? I'm leaning towards the mixed case symbol approach which means that the symbols have to be enclosed in vertical bars, but I'd arguet that this is a feature as it makes more obvious which symbols refer to external declarations.
2. Packages: To which package should the C declarations go?
3. Callbacks: For full functionality of many C APIs we're going to need callbacks. My understanding is that SBCL still doesn't support them. Hopefully this will get fixed in the near future.

Anyway, hopefully I will have a release of this up in the next week or so. Or send me email if you're interested in seeing this sooner rather than later.

(setf (ref l :foo :ban :barry) 17)

(defun hash-table-to-plist (h &aux l)
  (if (hash-table-p h)
      (progn (maphash
              #'(lambda (key val)
                  (setf l (cons (hash-table-to-plist val)
                                (cons key l)))) h)
             (nreverse l))
      h))

(defun hash-ref (h &rest keys)                                                                                  
  (reduce #'(lambda (h k) (gethash k h)) keys :initial-value h))                                                

(defun %put-hash-ref (new-value h key &rest more-keys)
  ;; not quite Perl-style autovivification, but we do create
 ;; appropriate list structure for intermediate keys that
can't be found
  (unless (hash-table-p h) (setf h (make-hash-table :test 'equal)))
  (let* ((sub (gethash key h))
         (val (if more-keys
                  (apply #'%put-hash-ref new-value sub more-keys)
                  new-value)))
    (progn (setf (gethash key h) val) h)))

(define-setf-expander hash-ref (place &rest props
                                  &environment env)                                                             
  ;; %put-ref may cons new structure or mutate its argument.
  ;; all this magic is just so that we can
  ;; (let ((l nil)) (setf (ref l :foo :bar) t))
  (multiple-value-bind (temps values stores set get)
      (get-setf-expansion place env)
    (let ((newval (gensym))
          (ptemps (loop for i in props collect (gensym))))
      (values `(,@temps ,@ptemps )
              `(,@values ,@props )
              `(,newval)
              `(let ((,(car stores) (%put-hash-ref ,newval ,get ,@ptemps)))
                 ,set
                 ,newval)
              `(hash-ref ,get ,@ptemps)))))

CL-USER> (let ((l nil)) (setf (hash-ref l :foo :bar) 47) (hash-table-to-plist l))
(:FOO (:BAR 47))                                                                     

[I was going to send this to sbcl-devel, but I've already pestered the folks on the list with a couple trivial matters this week and I'm sure the folks have better things to do than listen to me whine about perceived (by me) weaknesses in SBCL without a patch to address them. So, here goes...]

I've been working on adding prepared query support and support for blobs to the CLSQL postgresql backend.

• clsql-cyrus-20050207.tgz

It works pretty well (i.e. it passes the few simple tests I wrote and works in the limited situations in which I tested it), but is currently not used at all by the oodml stuff, which would be nice, as it offers the potential to greatly speed up repeated accesses to the database

I'd like to get this stuff cleaned up and integrated into the CLSQL source tree, but it hasn't happened yet. Hopefully soon.

Ok, the wait is over! I've finally officially launched my own blog, cyrusharmon.org. I know, lame name. Maybe one day I'll thikn of something clever. Until that day, cyrusharmon.org will have to suffice as the central repository for my random musings, tales of lisp trials and tribulations, code snippets, and, occasionally a photograph or two. And maybe some links to interesting things on the web every now and then.

I know what you're thinking. What is this fantastic blog engine that is driving the awesome dynamic content at cyrusharmon.org? Well, it is a hacked up version of Brian Mastenbrook's cl-blog. A blog engine that runs in common lisp. This particulary hacked up version has some nice features, including a database backend for the content. Unfortunately, the comment system is still not there yet. One of these days, I'll get around to adding that. In the meantime, I hope to either merge my changes back into Brian's tree in the near future, or roll a release of my own. Watch this site for more information.

SBCL's sb-bsd-sockets and sb-posix packages directly export a number of symbols, instead of using the :export cluase in defpackage. This seems to cause a problem on subsequenct recompilation of the defpackage.lisp file as the symbols are exported by the package but not in the :exports clause. This patch fixes things these two packages.

Christope Rhodes kindly pointed out that the consquences of calling defpackage on existing packages that have been modified are undefined. This suggests that my proposed patch is probably the wrong way to handle this. Xophe suggests the following:

  (eval-when (:compile-toplevel :load-toplevel :execute)
    (unless (find-package "FOO")
      (make-package "FOO" :use ("BAR") ...)))

Sounds good to me, but this hasn't made it into the tree yet, AFAICT.