I’ll talk about what a lisp macro is, and what makes it unique in the world of programming, how it’s a technique only possible in lisp. I’ll then take you through an example.

So firstly, what’s a lisp macro, and why would you want to write one?

So, you may have seen lisp programs before, and you’ll recognise them instantly — Larry Wall, the inventor of Perl, said they had all the aesthetic appeal of a bowl of porridge mixed with toenail clippings;

(defun accumulate (combiner lst initial)
  (let ((accum initial))
    (dolist (i lst)
      (setf accum (funcall combiner accum i)))
    accum))

He has a point. They are butt-ugly. But hell, the best he came up with is Perl, so he can $_@++ right off. (I’m pretty sure that’s valid Perl, too )

It’s ugly, in an aesthetic way, but it’s amazingly practical. It’s got an engineering beauty to it. If you look at that snippet above, you’ll notice that the whole program is made out of exactly three types of symbols;

• open parenthesis: (
• close parenthesis: )
• symbols, like defun, accum and setf

All simple lisp programs are like this. Just brackets to group stuff together, and stuff that needs grouping. Compare that with C#, where you might find;

• parenthesis for;
  • function calls; print("hello")
  • special forms; using(OdbcConnection con = ...)
• semi-colon to end statements; int x = 1;
• curly brackets for;
  • code blocks; { /* code block */ }
  • array initialisers; string[] words = { “hello”, “world” };
• square brackets for array indexing; x[3] = 4;

and the list goes on. I gave up because there are too many to list.

So lisp has this seriously small syntactic footprint. You can have a thing, or a group of things in brackets. It’s simple. It’s so simple that you can start doing crazy stuff in lisp that you just can’t do otherwise. That crazy stuff goes by the name of macros.

I can write a program that takes a chunk of lisp (remember, just a thing or a list of things), cuts it up, and reassembles it. That creates new lisp code.

So imagine you do a lot of work on three-dimensional arrays. You find yourself, over and over, writing nested loops that say;

for x in range(100):
    for y in range(100):
        for z in range(100):
            # do something to matrix[x,y,z]

And frankly, you’re bored of typing it over and over. What you really want to do is something like;

for {x 100, y 100, z 100}:
    # do something to matrix[x,y,z]

You want a brand new bit of syntax for multiple-value looping. Can you add it to python? Nope. C? Nope. Java? Nope.

But now look at the lisp version;

I could, theoretically, write this

(domanytimes (x 100 y 100 z 100) 
    body)

and, because it’s just a list of stuff, I can chop and change that into this new bit of lisp;

(dotimes (x 100)
    (dotimes (y 100)
        (dotimes (z 100)
            body)))

I’ll show you how in a second, but notice what’s possible — I can write my own looping construct (domanytimes) and lisp will rewrite it into many simpler looping construct (the built-in dotimes).

Is that particularly special? Well, yeah. I’ve written new syntax. I’ve defined a new way of looping that is no different from the standard loops. I’ve basically added something new to the language. Lisp is now better at dealing with multi-dimensional loops. Try adding a new loop to ruby, or javascript. Make python understand

for x in range(100), y in range(100), z in range(100):
      # body here 

and you’ll find you can’t.

So I’ve made my version of lisp a bit better at handling loops. If I were writing database code, I could make lisp better at writing SQL statements or data access layers. C# recently got built-in DAL logic with LINQ, and it’s great, but only the C# team can write it. Whereas a lisper could write this sort of code;

(sql-select (ID NAME) from PROJECT where (DUEDATE > TODAY))

and it’s do basically the same thing as LINQ.

So that’s the why’s and wherefores. Here’s the how of the domanytimes macro.

domanytimes takes two parts; the loop variables (x 100 y 100 z 100) and whatever body you want to execute. We’re going to write a program that skims two elements from the front of the loop variables (say, x and 100) and uses them to write a built-in dotimes loop; so a program which converts

(domanytimes (x 100 y 100) body)

into

(dotimes (x 100) 
   (domanytimes (y 100) body))

and then again to give you

(dotimes (x 100)
    (dotimes (y 100)
        body))

Here’s the domanytimes macro, in all it’s eye-bleeding horror;

(defmacro domanytimes (loop-list &body body)
  "allows you to write (domanytimes (x 10 y 10) ...) 
  instead of (dotimes (x 10) (dotimes (y 10)) body ))"
  (if (eq (length loop-list) 0)
      ;; we have our form to execute
      `(progn ,@body)
      ;; we have more loops to arrange
      (let ((fst (car loop-list))
        (snd (cadr loop-list))
        (rst (cddr loop-list)))
    `(dotimes (,fst ,snd)
      (domanytimes ,rst ,@body)))))

There. Wasn’t that fun?

It looks nasty, I know. All lisp looks nasty. But it’s actually created something new in the language. As far as I understand it, lisp has survived for fifty years basically because the macro system lets you write macros which can add any new kind of syntax you like. You can write knock up a set of macros to implement OO, and suddenly lisp is OO. You can know up macros for manipulating lazy lists, and suddenly lisp has a lazy evaluation. You can knock up data access layer macros, and it’s got a version of LINQ. There seems to be nothing you can’t hack lisp into being.

And if you want to know how the hell that works, I’d recommend Practical Common Lisp, which is online and free.

At work, we’re designing a new product. It’s a process that involves non-technical users, IT managers, programming colleagues, me, and the computer. It involves a lot of talking, a lot of language. At one end, it’s making sure the users are happy with the system’s capabilities. While talking to them, we use non-technical language, diagrams, and demonstrations. Way at the other, it’s actually typing code.

This makes me think it might be useful to split the languages we’re talking into four;

User Langage: Non-technical, example-laden, visual, and concrete. This is where you get into general discussions, produce photoshop mockups of the system, draw rough-and-ready whiteboard pictures, write XP user stories, write user manuals.

Manager Language: Your manager needs to know the technical and algorithmic overview, but he doesn’t need to see every class that’s going into the design. This language is the language of high-level functional specs, some UML diagrams, conceptual diagrams.

Colleage Language: At this level, you’re going to have to assure yourself that the code you write and the code your colleagues write start meshing together. Precision starts being important, and a real burden. Probably where most of the real thinking goes on. I think one of the reasons Pair Programming works is because it forces you to get into a lot more of these types of discussion, which reduces the possibility that the team’s work won’t gel together.

Self Language: This is the idiosyncratic, outboard-brain style of writing that you get into to make sure you understand what you need to be doing. This can be personal pseudocode, notes-to-self, todo lists, or code comments.

Computer Language: The code itself.

I think a great deal of the actual design and thinking involved in creating new products goes on above the level of the computer language. When people discuss, say, “Python Vs Java” or “Ruby vs Lisp”, it’s valuable enough, but the choice of language probably isn’t that important in determining the success of the project. I think these language strata go some way to explaining why. Most of the communication goes on between people.

I’m struggling with this question because I write C#3 by day, and am learning python in the evenings. I’m only writing small python scripts at the moment and I’d like to write larger pieces, but I’m concerned about how easy it’ll be to make certain types of change.

For example. You’ve got 100,000 lines of code. You also have a logging function that’s looks like this;

void Log(string message)

And it’s called about 200 times in your code. You decide you need a severity; so you change the signature to

void Log(string message, LoggingSeverity severity) { .. }

Now, how long does it take to find all the calls to the Log() function that need to be updated? Under C#, about ten seconds. Once every call has been fixed, the code is almost certain to work correctly.

Consider, on the other hand, the python function

def log(message):
    ...

What happens if you change the signature to

def log(message, severity):
    ...

There is no way to tell where the log message is called. You’ve just introduced 200 bugs.

It’s made even worse by duck typing; maybe you have two loggers — a deployment logger which writes to a database, and a test logger which writes to stdout. You update the database logger so it has severity. Your tests continue to pass, but your deployed system will fail.

So it seems to me that static languages give you much more power to make changes to large codebases. I’d love to know if, and where, the mistakes are in my thinking.

German advert for milk, with the tagline “Hard types need hard bones… drink milk!”

link

Take, for example, this hoverboard;

Yes, that’s right. A hoverboard. For about seven grand, from these guys

Um, Hi, Rob.

I have two previous posts on lisp, A programming language only a mother could love, and lisp, the beautiful hydra.

Childhood Nightmares

When I’ve had time, I’ve been attempting the problems in Project Euler. Euler lays out a series of ~200 mathsy programming puzzles, things like ‘find the sum of all the even-valued terms in the Fibonacci sequence which do not exceed one million.’ I’m using them as a way of trying out the language. At my side, possibly the only decent beginner’s book, Practical Common Lisp. (It’s also free.)

So far, I’ve finished 2 problems. Hmm. I’m pretty sure I could have finished more of them, more quickly, in almost any other language.

That, I think, is at least in part that lisp doesn’t come naturally if you’re used to non-lisp languages, things that bear a resemblance to C or BASIC. I’m having to go right back to basics, learning to construct things in a new way. Here’s the code I used to solve problem 2;

(defun fibseq (max)
  (let ((result ())
        (n-1 1)
        (n-2 0)
        (term 0)
        (i 0))
    (loop
     (setf term (cond ((= i 0) 1)
                      ((= i 1) 1)
                      (T (+ n-1 n-2))))
     (setf n-2 n-1)
     (setf n-1 term)
     (setf i (1+ i))
     (when (> term max) (return))
     (push term result))
    (print result)
    result))

(defparameter allfib (fibseq 1000000))
(defparameter evenfib (remove-if-not #'evenp allfib))

; the answer!
(print (reduce #'+ evenfib))

I don’t print this here to demonstrate the clarity of lisp. In fact, the opposite. This looks like hell to me, at least right now. There is, no doubt, a far better way to do this. But I don’t know, and so I’m writing programs that are terribly inelegant. Nestled in the code above is a non-terminating loop with a break condition, because I couldn’t figure out how to do the equivalent of

while (n < 1000000)
{

Right now I feel like I did when I was ten, programming in BBC basic and using GOTOs. Hell, it knocks together working programs, right? But you don’t want to stay there too long. That feeling of childhood programming, though… there’s something compelling in it. Lisp tastes of nostalgia. Remember programming Logo? Or BASIC? Lisp is making me feel like that. At least for now.

Pretty soon, I hope to have mastered basic loops. Then I’ll be well on my way.

(setq subject (list ‘( ‘)))

Or, now I will talk about parentheses.

In my head, despite being entirely uncomfortable with while loops, I’m starting to see everything falling into a lisp syntax. The idea of just bracketing up your stuff into lists seems like a great first-draft syntax for everything — want to talk about data structures? write

(data
  (entity1 (attribute1 attribute2))
  (entity2 (attribute3 attriubte4)))

want to write pseudocode for a function call?

(func (param1 param2) ...)

want to write a todo list?

(do 
  (buy bread)
  (tidy (kitchen living-room bathroom))
  (get life))

Valentines day coming up?

 (get-count
     (make-list #'(lambda (i thee) 
                          (permute #'love i thee))))

Ok. Maybe not. (apologies to Elizabeth Browning there.)

But you get my point. I hope. Those brackets are just fine for everything. I’m starting to understand why hardcore lispers want to do everything in lisp. My brain is infected with brackets.

But for right now? I’m going to learn to do a for loop, and then we’ll see.

Fun oddness; Samurai Dog Armour

This suit of dog armor — identified by antique Japanese armor dealer Toraba.Com as the only known and certified authentic example of its kind — is believed to have been created for the pet of a wealthy, high-ranking and presumably eccentric samurai or daimyo (feudal lord) in the mid to late Edo period (mid-18th to mid-19th century)

link

Long-distance driving, especially somewhere you’ve never been before, can often be nasty. You have to juggle several things at once – planning the route, controlling the car, talking to your passengers, etc — and you have time pressure, too. For me, this means stress.

Two things, I think, made the journey much more pleasant;

1. I drove an automatic,
2. I used GPS to navigate.

This offloaded several tasks from my brain; I no longer had to plan a route; everything was handed to me in simple-to-follow left/right/straight on choices, and if I made a mistake (which I did a couple of times) the GPS just routed round it, giving me a new route which got me there.

The automatic gearbox also made things easier. That part of my brain that would normally be making sure I was in the right gear was no longer occupied.

Life was simpler, so life was sweeter.

Joel, of the ‘Joel on Software’ site, uses the analogy of automatic transmissions when he talks about some benefits for programmers (scroll to ‘Automatic Transmission Wins the Day’). The wider point really is that freeing up your mind from one type of detail is like juggling with one fewer ball; it’s much, much easier.

So this is my wider point; anything fully automatic (I mean really and truly don’t-make-me-think automatic) makes your life less stressful.

The flipside, of course, is control. You always sacrifice a little control to the system. You lose precise throttle control in an automatic car. You lose control of exactly what stocks you own if you buy an index-tracker ISA.

I think this scares people, but really shouldn’t — this level of detail will probably tire you, stress you, and give very little benefit over the automatic method.

I’ll finish with a fantastic quote from Alfred North Whitehead, co- author of the Principia Mathematica;

It is a profoundly erroneous truism, repeated by all copy books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like calvary charges in a battle–they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.