Interview about Lisp at SISCOG

My friend Rui was interviewed about Lisp and how we use it at SISCOG. The original interview is in Portuguese but you can read a translation via DeepL below:

SISCOG Engineering: get to know this cutting-edge Portuguese company

Find out how Lisp continues to drive innovation at SISCOG. Interview with Rui Patrocínio, Scheduling Team Leader at SISCOG

#1 How did you first get to know the Lisp programming language?

When I joined Técnico in 1999, the programming language taught in Introduction to Programming was Scheme, which is in the Lisp family. The curriculum closely followed the MIT 6.001 course (whose lectures from the 80s are on Youtube) and one of the best programming books ever for beginners and beyond: Structure and Interpretation of Computer Programs. The great advantage of learning to program with Scheme is that it is a very simple language, with a very simple syntax. All the effort goes into understanding the logic of what you're implementing, so there's no need to memorize the syntactical nuances of the language.

After that experience, Lisp reappeared as Common Lisp in the Artificial Intelligence course. Common Lisp is the current industrial version of the Lisp family of languages, which is what SISCOG uses on a daily basis. Common Lisp is a multi-paradigm language. Imperative, functional, object-oriented programming with very sophisticated meta-programming mechanisms available. As such, using the language as a whole implies some maturity and it's only natural that it should appear later in the curriculum of a computer engineering course.

I think it was the “power” of the Common Lisp language that made it particularly appealing to me and, despite the standard being from 1994, it is a very modern language. It should be noted that Guy Steele, one of the main figures behind the Common Lisp standard, was also heavily involved in the development of Scheme, C and Java, being hired by Sun at one point to improve Java. Here's a quote from him on a mailing list focused on the discussion of programming languages from the 1990s, talking about Java:

And you're right: we were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp. Aren't you happy? —Guy Steele

#2 How SISCOG uses Lisp in its products

We use Common Lisp in the vast majority of our software. Both desktop applications and backend parts of web applications are implemented in Common Lisp. We also use C++ to develop specific modules, but Common Lisp still has the most lines of code in our repositories. It's a very expressive language that compiles to machine code reasonably efficiently without much effort on the part of the programmer. So it's an easy choice for most scenarios.

It remains to be said that our products have been in operation for more than 30 years in various national and international companies, such as the London and Lisbon Underground, or the railways of the Netherlands or Canada. They are decision support software for the optimized planning of these transport operators' resources, namely time and space, materialized in timetables, vehicles and personnel. These products help to plan and manage these operational resources as quickly and efficiently as possible, providing gains and savings in various areas, as well as, for example, greater satisfaction on the part of workers thanks to shifts and work schedules that better meet their preferences. And all with Lisp as a base!

#3 What are the main advantages of the Lisp syntax compared to other programming languages, such as C++?

The great advantage of the syntax is also the thing you'll find most strange at first: the brackets. The fact that everything uses a prefixed syntax also makes everything quite uniform. This combination of parentheses and prefixed syntax is called s-expressions in Lisp. S-expressions are code and data at the same time, and this is what allows you to have macros (functions that take code as an argument and return code as a value) that extend the language transparently, as if they belonged to the standard language.

#4 Can you explain the concept of “S-expressions” and how they contribute to the clarity of LISP code?

Basically, “S-expressions” are one of two things:

• atomic expressions (e.g. the number 2024 or the string “hello world” or the symbol +)
• a list, typically in the format (operator arg1 arg2 ... argn)

It's this uniformity, as I said earlier, that makes everything simpler. It also helps a lot to be able to generate code programmatically because it's all about manipulating lists, for which Common Lisp has a very reasonable API. This is where the power of Lisp macros comes from.

At first you find it strange, then you understand it. [nice try, DeepL. Rui wrote "Primeiro estranha-se, depois entranha-se." which is quoting a famous Coca-Cola slogan written by Fernando Pessoa in the 1940s. Richard Zenith, in his biography Pessoa: An Experimental Life, attempts to translate it as "On the first day you drink it slow. On the fifth day you can't say no." —Luís]

#5 How does Lisp allow rapid prototyping using untyped variables?

Static typing in large codebases is widely advocated today because it allows more automatic tools to check for problems in the code. Common Lisp has dynamic typing, but allows optional type declaration. It's quite common to only declare types when it's necessary to “squeeze” more performance out of a code segment. The compiler, via type inference, provides some information about problems encountered when there are enough type declarations to extract relevant information. This allows us to “fight the compiler” only in segments of code where it is very relevant and not be making premature optimizations to the whole code.

#6 How does Lisp facilitate metaprogramming?

There are two basic “tools” for metaprogramming in Common Lisp: the Meta Object Protocol and the macros mentioned earlier.

Meta Object Protocol is basically the mechanism behind the implementation of the Common Lisp object system (called CLOS, Common Lisp Object System). Although it doesn't belong to the standard, it is available in most Common Lisp implementations and is sometimes useful, particularly when implementing utilities to inspect the code or improve our development environment. For example, it's possible to implement something that shows us the relationships between classes, what methods exist, etc. This is not something you do on a day-to-day basis, but it can be done by you without depending on the guts of a specific IDE.

Another interesting mechanism is macros. When CLOS was introduced, it was basically a set of macros on top of the Lisp of the time (historically there were a few more iterations - CLOS wasn't the first object system developed). In other words, with macros it's possible to extend the language to introduce most of the mechanisms and paradigms that exist in other languages (such as object-oriented programming) in a way that feels natural to a Lisp programmer.

This type of mechanism allows SISCOG to implement undo and redo mechanisms in our software in a way that is practically transparent to the programmer. This is done by extending the class declaration. The programmer notes for which attributes of each class a history needs to be kept and the end user is able to undo their operations and see these data variations instantly (with the typical implementation using Command Pattern, multi-level undo is not instantaneous as it is necessary to re-execute operations, which may not be trivial; the tradeoff is, of course, the memory spent).

#7 What is your opinion of Lisp's learning curve compared to other programming languages?

Anyone with experience in a few different paradigm languages can quickly program in Common Lisp without major problems. Most of the concepts are familiar. Our experience at SISCOG, particularly with more recent hires who had no contact with Lisp in college, is that people adapt quickly and within a week are programming and making minor corrections.

Obviously, there are parts of the language that are less common and you need more maturity to use them effectively. These include the Meta-object protocol and the macros mentioned earlier.

In this case, the need for greater maturity comes simply from the fact that we are extending the language. Designing a new language isn't easy and there are people whose career it is to do this (like Guy Steele mentioned earlier). Of course, this is also rare and the most common thing is to introduce macros for small functionalities that make the programmer more productive with Domain Specific Languages.

#8 Why is Lisp a popular choice for specialized areas such as artificial intelligence and natural language processing?

It's essentially for historical reasons. Lisp was born in that context, there have been several “classic” artificial intelligence systems implemented in Lisp and so it has continued to be used. The fact that it's very easy to start a project quickly, it's easy to iterate and progressively improve what's been done, also helps in a research context where we're more concerned with testing ideas than making the software “bulletproof”, which you can do progressively (treating errors, adding static typing, etc.) in Common Lisp.

#9 Can you give some examples of projects at SISCOG where Lisp has proved particularly advantageous?

SISCOG has relied on Lisp from the start. It's a very stable language, with good compilers and good performance, it's compact and has allowed us to go through the history of computer science across various platforms (Lisp machines, Unix, Windows) and maintain a code base with immense domain knowledge over the years. A company with software in production for over 30 years is not very common in the world and Lisp is clearly our little secret weapon.

#10 How does the Lisp development community compare with other programming language communities in terms of support and resources?

The Lisp community isn't very large, but what we lack in programmers, we make up for in enthusiasm. Paul Graham (of Y Combinators) was a great driving force behind the language in the late 1990s, early 2000s and Hacker News still talks about Common Lisp (and variants) on a regular basis. Many years ago there was a strong community on the old newsgroups, but that has largely moved on to Reddit and IRC. There are also a few annual conferences (e.g. European Lisp Symposium) where some of the most important members of the community usually gather. We don't have by far the largest number of libraries available and sometimes we have to implement things 'in house'. The community does, however, have enough to make high-quality software that is sold all over the world, as SISCOG has demonstrated.

#11 What are your predictions for the future of Lisp in the technology industry?

It seems to me that the future of a language depends a lot on fashions and the investments made in it. Google uses Lisp via the purchase of a company a few years ago (ITA Software, for flight search) and, as long as these large companies continue to invest, languages will thrive. At the moment, fashions have moved on to other platforms, but let's see what the future brings. Perhaps our little secret weapon will become less and less secret.

Manuel Simoni on CL's control flow primitives

Manuel Simoni dusts his Axis of Eval blog off and writes about Common Lisp's BLOCK / RETURN-FROM and UNWIND-PROTECT. A summary for non-Lispers.

Reddit 1.0

As many Lisp programmers may remember, Reddit was initially written in Common Lisp. Then, in late 2005, it was rewritten in Python. Much discussion ensued. Steve Huffman (aka spez, current CEO of Reddit) wrote about why they switched to Python. The late Aaron Swartz also wrote Rewriting Reddit, focusing on the Python side of things.

Last week, that original, Common Lisp version of Reddit was published on GitHub: reddit-archive/reddit1.0. There's no documentation, but we know from the articles above that it ran on CMUCL. reddit.asd is perhaps a good entry point, where we can see it used TBNL, CL-WHO and CLSQL, a common toolset at the time.

It's missing various bits of static HTML, CSS and JS, so I don't think you can actually run this website without a fair bit of scrapping and stitching from the Wayback Machine. The code is quite readable and clean. It's not terribly interesting, since it's a fairly simple website. Still, it's a website that grew to become the 6th most visited worldwide (with 542 million monthly visitors) and now has 230 employees, says Wikipedia, so it's nice to see what it looked like at the very beginning.

A Lisp REPL in your pocket

Thanks to Polos Ruetz, you can now play with Common Lisp directly on your Android phone. All you need to do is install CL REPL from the Google Play Store. CL REPL is one of the examples part of EQL5-Android which is built on top of EQL5, a project that marries ECL with Qt.

CL REPL

A Common Lisp REPL with command line and history, plus a simple editor with syntax highlighting, simple visual paren-matching, a file dialog for opening/saving files, and a simple debug dialog. It uses the ECL implementation for the Lisp side, and Qt5/QML for the UI. This is an open source project (see EQL5-Android).

(via @dk_jackdaniel)

Re: Querying plists

Zach's Querying plists blog post showcases a neat little querying DSL for plists. I couldn't shake the feeling that it looked an awful lot like pattern matching. I've often been impressed by optima, but I barely get to use it, so I thought I should try and see what querying plists looked like using pattern matching.

Here's what I came up with. Zach's example

(query-plists '(:and (:= :first-name "Zach") (:= :state "ME")
                     (:not (:= :last-name "Beane")))
              *people*)

becomes:

(remove-if-not (lambda-match ((plist :first-name "Zach" :state "ME"
                                     :last-name (not "Beane")) t))
               *people*)

It turned out more succinct than I initially expected! Also, it's trivially adaptable to other kinds of objects. E.g., given the following class:

(defclass person ()
  ((first-name :initarg :first-name)
   (last-name  :initarg :last-name)
   (state      :initarg :state)))

all we have to do is swap plist with the class name person and we're all set:

(remove-if-not (lambda-match ((person :first-name "Zach" :state "ME"
                                      :last-name (not "Beane")) t))
               *people*)

We can't quite define something exactly like Zach's query-plists because, AFAICT, optima's patterns are not first-class objects but perhaps we can cheat a little bit.

;; naming things is hard. :-/
(defmacro matchp (pattern) `(lambda-match (,pattern t)))
(defun filter (predicate list) (remove-if-not predicate list))

(filter (matchp (plist :first-name "Zach" :state "ME"
                       :last-name (not "Beane")))
        *people*)

Making this equally succinct when the query criteria are not constant is a challenge for another day and makes it clear that matchp is a lousy abstraction. ;-)

slime-macrostep

SLIME's just got a new contrib called slime-macrostep, courtesy of Jon Oddie who also wrote the underlying macrostep package.

And what is slime-macrostep? It's an interactive inline macro-expander. Have a look:

In this quick demo, using a CFFI example, I start by expanding the top-level WITH-FOREIGN-OBJECT form, then I expand the WITH-ALIEN form, but regret it and collapse it back. Then I proceed to expand everything else, including compiler macros!

A nice feature of slime-macrostep is that the expansions are annotated to show which forms are further expandable and macrostep-expand will jump automatically to the next expandable form. That's what makes it a stepper: pressing e repeadly will step through the macroexpansion. Plus, it expands macrolets!

If you'd like to try it out, please grab SLIME's bleeding edge (via MELPA or Git). It's enabled by default if you use the slime-fancy meta contrib. Feedback is most welcome.

Common Lisp Recipes: A Problem-Solution Approach

Edi Weitz, author of things like CL-PPCRE and Hunchentoot (amongst many other pieces of ediware), has just published a new book, “Common Lisp Recipes: A Problem-Solution Approach”. I've ordered mine. Get yours from Apress or a European Amazon store.

Common Lisp Recipes is a collection of solutions to problems and answers to questions you are likely to encounter when writing real-world applications in Common Lisp. Written by an author who has used Common Lisp in many successful commercial projects over more than a decade, this book is the first Common Lisp book which covers areas as diverse as web programming, databases, graphical user interfaces, communication with other programming languages, multi-processing, and mobile devices as well as debugging techniques and optimization, to name just a few. It is organized around specific problems or questions each followed by ready-to-use example solutions and clear explanations of the concepts involved, plus pointers to alternatives and more information. Each recipe can be read independently of the others and thus the book will hopefully earn a special place on your bookshelf as a reference work you always want to have within reach.

Common Lisp Recipes is aimed at programmers who are already familiar with Common Lisp to a certain extent but do not yet have the experience you typically only get from years of hacking in a specific computer language. It is written in a style that mixes hands-on no-frills pragmatism with precise information and prudent mentorship.

If you feel attracted to Common Lisp's mix of breathtaking features and down-to-earth utilitarianism, you'll also like this book.