Category Archives: Programming

Python vs C

A few days ago I wrote a post about how to convert a csv list to a table. I spent some time writing my program in C (for fun and for performance).

Now I have written an equivalent program in Python, and I am going to present the differences.
First the sourcecode: csv-list2table.c and csv-list2table.py.

About the C implementation
The C implementation uses only standard C library (stdlib.h, stdio.h, string.h). The good thing is that the program is very portable and easy to compile. The bad thing is that I do not have access to powerful datatypes (hash tables, balanced trees, sets). This basically means wasting time reinventing the wheel, and not getting a perfect wheel anyway. My estimation is that the C implementation is O(n) for suitable input (fairly sorted data) and O(n)Sqrt(n) for unsuitable data (reversely sorted). This is for fairly “square” matrices. The C implementation is not really optimized (perhaps a later excercise).

About the python implementation
The python implementation was written after the C implementation, and very nicely utilizes the python built in set and dict datatypes. Should be O(n) or O(n)log(n) depending on Python dict implementation.

The machine
The benchmarks are performed using the time command on an AMD Athlon II X2 250 machine with 4GB of RAM. The machine runs x64 Xubuntu 11.10.

The tests

C Python
Environment gcc 4.6.1, -O2 Python 2.7.2+
Lines of code
(excl help text)
355 lines 53 lines
Coding time 8h 1h
Input size Execution time
100×100
288kb
Sorted data: 0.014s
Reversed data: 0.027s
0.11s
200×200
1.2Mb
0.039s – 0.052s 0.18s
400×400
4.5Mb
0.16s – 0.25s 0.64s
800×800
18Mb
0.57s – 1.2s 2.6s
1600×1600
72Mb
2.4s – 7.8s 10s
3200×3200
288Mb
9.7s – 51s 41s
6400×6400
1.2Gb
40s – 337s Not enough RAM
RAM usage for
3200×3200
152Mb 1.4Gb

Conclusions
I draw the following conclusions:

  • Python is a very efficient language compared to C, when it comes to producing code fast, in this case about 10x faster
  • Python is impressively fast, even startup overhead is reasonable
  • Python can beat C on performance, when C programmer has not found optimal data type/algorithm. The Python datatypes makes it easy to write code that is fast even for large data
  • A well written C program uses 10% the RAM of Python, and is at least 5x faster. Especially the small RAM requirement is very powerful and valuable for many applications

It is tempting to optimize the C program to see if I can get 2x, 3x or 5x speedup.
It is also tempting to write a C program that uses Glib, and have access to well implemented data types. How would it compare?

CSV file: convert list to table/matrix

I found myself having data in CSV-files with three columns; two dimensions and a value. It could look like:

20080102,AAPL,194.84
20090102,AAPL,90.75
20100104,AAPL,214.04
20110103,AAPL,329.57
20120103,AAPL,411.23
20080102,MSFT,35.22
20090102,MSFT,20.33
20100104,MSFT,30.95
20110103,MSFT,27.98
20120103,MSFT,26.765

Data typically looks like this because it is very easy to output transactions on this format. That is very nice if you want to load it into a database. But for other purposes (like plotting a graph using LibreOffice Calc or even Excel) it would be much nicer with a table/matrix-layout:

,AAPL,MSFT
20080102,194.84,35.22
20090102,90.75,20.33
20100104,214.04,30.95
20110103,329.57,27.98
20120103,411.23,26.765

I could not find a standard tool for this. I thought about different options, and finally decided it was quite easy to just write a little program. So I did. You use it like this:

$ ./csv-list2table -t < list.csv > table.csv

There are a few things to think about:

  • The switches -t or -T decides if column 1 or 2 will be rows
  • Rows and columns are outputed sorted
  • Holes/missing values outputed as ,,
  • Comma is the only accepted delimiter
  • Input must have exactly 3 columns
  • Pre/post-process data with sed and cut

As the last item mentions, sed can fix a file with other delimiters than comma, and cut can pick the columns you need from a list with more data than you need.

Finally, the code written in standard C: csv-list2table.c.

Building should be trivial:

gcc -O2 -o csv-list2table csv-list2table.c

I dont think the code contains anything that should confuse any c-compiler on any reasonable platform… well, I have not tried to compile in Windows ;)

Build QT/Symbian Apps on Linux/Mac OS X

QT is a nice cross plattform development environment, but building for Symbian is a bit tricky on non Windows platforms. Or is it?

You can Google for it, and find information about Gnupoc, Gcce and building qt for ARM with tools that require WINE. Not trivial.

You can also find occational references to an experimental feature called “remote compiler” – sounds like you need a Window machine, doesn’t it?

No – it is really easy! You need a Nokia Developer account (which takes 5 minutes to set up, and costs nothing). Then you configure QtCreator to use the remote compiler – and it sends your code to Nokia, who compiles it, and returns you errors, warnings and binaries. Compilation-as-service? Fine with me!

To get it working, I suggest you install the full QT SDK (not just QtCreator), and you need to choose “Custom” installation, and under Experimental you pick Remote Compiler.

Next, when you have opened your project in QtCreator, you click the “Projects” button to the left. Now, in the top of your work area, there are boxes with build targets, and to the right of the targets a “+”-sign. Click it. Choose remote compiler, target platforms and other options. To deploy it to your phone, you need to choose to sign the application, and to use the smart installer.

Make Xcode 3.2.6 gcc work

Apple did something smart around Xcode 3.2.6 – probably to make it easy to configure gcc to use different SDKs. But, before figuring out what happened, it just brakes your compiler. With Xcode 3.2.6 (and maybe later versions):

sleipnir:~ gt$ cat hello.c 

#include 
int main() {
    printf("Hello world\n");
}

sleipnir:~ gt$ gcc -o hello hello.c
-bash: gcc: command not found
sleipnir:~ gt$ PATH=/Developer/usr/bin/:$PATH
sleipnir:~ gt$ gcc -o hello hello.c
hello.c:2:19: error: stdio.h: No such file or directory
hello.c: In function ‘main’:
hello.c:4: warning: incompatible implicit declaration of built-in function ‘printf’
sleipnir:~ gt$ gcc --sysroot /Developer/SDKs/MacOSX10.6.sdk  -o hello hello.c
sleipnir:~ gt$ 

Simple Encryption Library for QT (improved)

A while ago I posted a simple encryption library for QT on this blog. I have now made some improvements and publish a new version. Major improvements are:

  1. Improved performance
  2. Added non patented Serpent Algorithm
    (easy to include and build without RC5)

As before, SimpleQtCryptor should work on all QT platforms without modification. It has no dependencies to anything outside QT and just uses qt datatypes.

Performance – SimpleQtCryptor vs OpenSSL
Since encryption is just overhead to any program using it performance is important. I have tried to optimize SimpleQtCryptor as much as possible, without making the code hard to read or risking portability.

For practical purposes, its performance is comparable to that of OpenSSL. This is on an Athlon II X2 250 3GHz cpu running Ubuntu:

                                                                OPENSSL
RC5-32/CBC      RC5-64/CBC      RC5-64/CFB      Serpent/CFB     AES-128-CFB
-----------------------------------------------------------------------------
6.5s / 9.0s     3.5s / 6.1s     3.2s / 5.9s     16.4s / 17.6s   5.8s / 7.8s
6.3s / 9.5s     3.5s / 6.1s     3.1s / 5.7s     16.4s / 19.0s   6.0s / 8.3s
-----------------------------------------------------------------------------

Values are presented as user time/real time (including a sync command).
First line is encrypt, second line is decrypt. The data is a 512MB file.

The first 4 columns are SimpleQtCryptor using different algorithms and modes of operation. The last column is 128-bit AES for OpenSSL.

Performance – different machines
For different machines, benchmarks have been made (just algorithm, no streaming or I/O):

                                RC5-32  RC5-64  Serpent 
------------------------+-------------------------------
Athlon II X2 250 3GHz   | enc   4.6s    2.3s    14.6s   
512Mb data, Ubuntu      | dec   4.6s    2.3s    14.8s
------------------------+-------------------------------
Athlon II X2 250 3GHz   | enc   5.8s    3.2s    22.4s
512Mb data, Windows7 64 | dec   5.6s    3.2s    20.6s
------------------------+-------------------------------
Intel Core i5 2.3GHz    | enc   6.9s    3.4s    19.1s
512Mb data, Mac OS X    | dec   6.6s    3.3s    19.2s
------------------------+-------------------------------
Intel Pentium(M) 2.0Ghz | enc   10.9s   47s     35s
512Mb data, Windows7 32 | dec    9.8s   46s     36s
------------------------+-------------------------------
Intel Atom z520 1.33Ghz | enc   20.2s   83s     113s
512Mb data, Windows7 32 | dec   19.2s   84s     130s
------------------------+-------------------------------
PPC G4 866Mhz           | enc   20.4s   62s     85s
512Mb data, Mac OS X    | dec   32.9s   61s     83s
------------------------+-------------------------------
ARM,QNAP TS109 500Mhz   | enc   8.6s    30.4s   58.4s   
64Mb data, Debian       | dec   8.2s    26.4s   92.7s   
------------------------+-------------------------------

It is fascinating to see how different CPUs perform. A few comments:

  • 32 bit machines suffer more from the 64 bit version of RC5, than the other way around – stay with 32 bit unless you know most clients will be 64 bit
  • Serpent uses a precomputed SBOX table (16kb), and CPUs with little cache (Atom and especially the ARM) underperforms on Serpent
  • The Mac OS X G4 compiler fails to use the assembler right roll used in decryption (but finds left roll for encryption) which makes RC5-32 suffer (but not Serpent, for some reason).
  • See how much slower the program runs on Windows 7 than on Ubuntu (on the Athlon II X2). The Microsoft SDK and Microsoft compiler is used, and I expected it to outperform gcc, not the other way around. I wonder if this applies to 32-bit machines as well, but currently I have no dual boot 32 bit machine available.

Building
As usual, building is easy with QT:

 $ qmake
 $ make

Including in your project
If you want to include the SimpleQtCryptor in your project just copy these three files to your project:

  simpleqtcryptor.h
  simpleqtcryptor.cpp
  serpent_sbox.h (not necessary, but improves serpent performance 5-10 times)

You only need to #include simpleqtcryptor.h.

In the beginning of simpleqtcryptor.h there is a #define WITHRC5 that you can remove if you do not want RC5 in your program (for legal reasons).

The README file should tell you most things you need to know to include and use the library.

Disclaimer
Bad implementations of encryption algorithms is a very bad idea. I believe my implementation is correct and secure – I can not guarantee it. Implementation details (especially regarding modes of operations) make this library incompatible with other encryption libraries. You can not (expect to) decrypt data with any other implementation, even if they use the same encryption algorithms.

License
The source code for SimpleQtCryptor is here released under the GPLv3 license. If you need another license for the (not patent encumbered) Serpent part I will try to be flexible.

Background
The implementation of Serpent was originally coded in 2000, before the AES competition was resolved. Later Serpent lost to Rijndael, which became AES. In 2011 this Serpent implementation was ported to QT and optimized.

The implementation of RC5 was made in 2010 because I wanted simple encryption for QT, without being dependent on any non-qt-components. RC5 was chosen (despite it is patent protected) because it is a very simple algorithm to implement.

The command line tool
The library comes with a command line tool. The purposes of the command line tool is:

  • To demonstrate how to use the library in QT code
  • To facilitate self-test functionality – you can run this after building on your machine/OS/qt version to ensure compability with other builds.
  • To be used as a benchmark tool

If you want a simple command line encryption tool I strongly recommend you use OpenSSL instead!

This is how to use command line tool:

  SimpleQtCryptor -t testfile
  SimpleQtCryptor -b e|d rc532|rc564|spt Mb (benchmark Mb)
  SimpleQtCryptor -e OPTIONS
  SimpleQtCryptor -d OPTIONS
  SimpleQtCryptor -h
OPTIONS:
  -k SecretFile (preferred to -p)
  -p Secret (default = )
  -i IndataFile
  -o OutdataFile
  -rc5    : use native RC5 algorithm (default)
  -rc532  : use 32-bit RC5
  -rc564  : use 64-bit RC5
  -spt    : use Serpent algorithm
  -cbc    : CBC
  -cfb    : CFB (default)
  -n      : no header
  -v      : verbose

Download
Source code: SimpleQtCryptor-1.0.0.tgz
Command line tool for Windows: SimpleQtCryptor-1.0.0-x86.exe, SimpleQtCryptor-1.0.0-x64.exe

At this point I don’t supply any binaries for Linux or Mac OS X, however a Symbian version should show up some day.

Simple encryption library for QT (RC5)

!! Also check out the new improved library. !!

A while ago I did some QT programming. I think the QT API is very good and useful. I like things about C++ more than Java, but at the same time QT takes away most of the annoyances with C++. Especially, developing cross-platform GUI applications is very nice with QT.

One thing I missed was a simple encryption library. I mean, there is QCA, but how simple is it to use?

So, I did the forbidden thing, and implemented my own encryption library. Ok – it is for many reasons a very stupid thing to do. But it was fun, I did it anyway, it works, and I will share it with you ;)

Well, I did another forbidden thing; I based my library on RC5 – a very elegant and simple encryption algorithm, protected by patents (at least in the US, where I do not live). Anyway, it is of course allowed to make an open source implementation of RC5.

The downloadable package contains three things: the library, a command line encryption utility using the library, and a little command line “unit test” tool. The files are named as:

  • main.cpp – test utility
  • simpleqtrc5.* – the library
  • simpleqtrc5_test.* – the unit test tool

As always with QT, building is very easy:

  $ qmake
  $ make

I used QT 4.5 and QT 4.6. Perhaps you are fine with older versions as well.

I suggest you have a look in the main.cpp-file, or simpleqtrc5.h for examples, instructions and documentation.

The library of course uses QT datatypes, so you can use it very naturally from any QT code. The core is implemented in somewhat optimized C code, that only uses QT datatypes, so it should be 100% portable. There are no non-QT dependencies.

I have implemented both a 32 bit version and a 64 bit version of the algorithm. Of course, both versions work on any CPU, and you chose algorithm at runtime. 32 bit algorithm is faster on 32 bit cpu, and 64 bit algorithm is faster on 64 bit cpu. Maybe this is the only 64-bit implementation of RC5?

Performance is reasonable, as you can see in this example (Ubuntu 11.04, x64):

$ time md5sum 100Mb.bin
28a8c7a11327880877f21c78b7222273  100Mb.bin

real	0m0.238s
user	0m0.220s
sys	0m0.010s

$ time openssl enc -e -aes-128-cbc -k p4ssw0rd -in 100Mb.bin -out 100Mb.aes.enc

real	0m0.580s
user	0m0.510s
sys	0m0.060s

$ time openssl enc -d -aes-128-cbc -k p4ssw0rd -in 100Mb.aes.enc -out 100Mb.aes.dec

real	0m0.619s
user	0m0.460s
sys	0m0.150s

$ md5sum 100Mb.aes.dec
28a8c7a11327880877f21c78b7222273  100Mb.aes.dec

$ time ./SimpleQtRC5 -e -p p4ssw0rd -i 100Mb.bin -o 100Mb.rc5.enc

real	0m1.678s
user	0m1.540s
sys	0m0.120s

$ time ./SimpleQtRC5 -d -p p4ssw0rd -i 100Mb.rc5.enc -o 100Mb.rc5.dec

real	0m2.064s
user	0m1.970s
sys	0m0.090s

$ md5sum 100Mb.rc5.dec 
28a8c7a11327880877f21c78b7222273  100Mb.rc5.dec

$ ls -l 100Mb.*
-rw-r--r-- 1 freke freke 104857600 2011-05-04 19:16 100Mb.aes.dec
-rw-r--r-- 1 freke freke 104857632 2011-05-04 19:15 100Mb.aes.enc
-rw-r--r-- 1 freke freke 104857600 2011-05-04 19:10 100Mb.bin
-rw-r--r-- 1 freke freke 104857600 2011-05-04 19:19 100Mb.rc5.dec
-rw-r--r-- 1 freke freke 104857634 2011-05-04 19:19 100Mb.rc5.enc

Above you can see:

  1. Calculating md5 sum of 100Mb-file
  2. Encrypting using openssl/aes-128 in 0.6s
  3. Decrypting using openssl/aes-128 in 0.6s
  4. Calculating md5sum to confirm that decryption recovered original file
  5. Encrypting using QT/RC5 in 1.7s
  6. Decrypting using QT/RC5 in 2.1s
  7. Calculating md5sum to confirm that decryption recovered original file
  8. Listing 100Mb files

The test program is quite simple to use:

$ ./SimpleQtRC5 
SimpleRC5 (v0.0)
USAGE:
  SimpleRC5 -t testfile
  SimpleRC5 -e OPTIONS
  SimpleRC5 -d OPTIONS
  SimpleRC5 -h
OPTIONS:
  -k SecretFile (preferred to -p)
  -p Secret (default = )
  -i IndataFile
  -o OutdataFile
  -w32    : use 32-bit words
  -w64    : use 64-bit words
  -w      : use native CPU words (default)
  -cbc    : CBC
  -cfb    : CFB (default)
  -n      : no header
  -v      : verbose

The only thing to explain is that without the header, the program can not itself figure out what options to use to decrypt (32/64 bit or cbc/cfb). The header does not reveal anything at all.

!! Please download the new and improved library !!

You can download the source: SimpleQtRC5-0.1.tgz.

If you have any questions, suggestions or complaints just let me know! I really believe the library is stable and secure, and simple to use! I might document it better if anyone cares about it.

Building QT framework on Mac and Windows

When building programs with QT that you want do distribute to your friends, I found it preferable to compile the QT library myself.

On Mac OS X this was nice, because I could choose architecture combination for my binaries exactly as I wanted (PPC only, or PPC32 + x64, for example).

On Windows it was nice because I could download the latest Windows Platform SDK (which is free!) and build QT with it. That way, I could create both x86 and x64 Windows binaries that could be distributed along with no other files whatsoever.

However, building QT takes a long time. And if you are unlucky or too ambitious with your configure switches, it might not build at all. So, here are simply a few working configurations:

Mac OS 10.5, PPC, XCode 3.1.3, QT 4.5.3

# qt_conf_carbon-ppc_x86.sh
./configure -release -no-qt3support -nomake examples -nomake demos -arch ppc -arch x86 -platform macx-g++42 -sdk /Developer/SDKs/MacOSX10.5.sdk/ -prefix /Users/zo0ok/qt_dev/4.5.3-carbon-ppc_x86

# qt_conf_cocoa-ppc.sh
./configure -release -no-qt3support -nomake examples -nomake demos -arch ppc -cocoa -platform macx-g++42 -sdk /Developer/SDKs/MacOSX10.5.sdk/ -prefix /Users/zo0ok/qt_dev/4.5.3-cocoa-ppc

# qt_conf_cocoa-ppc_x64.sh
./configure -release -no-qt3support -nomake examples -nomake demos -arch ppc -arch x86_64 -cocoa -platform macx-g++42 -sdk /Developer/SDKs/MacOSX10.5.sdk/ -prefix /Users/zo0ok/qt_dev/4.5.3-cocoa-ppc_x64

# qt_conf_static-ppc.sh
./configure -static -release -arch ppc -sdk /Developer/SDKs/MacOSX10.5.sdk/ -no-framework -prefix /Users/zo0ok/qt_dev/4.5.3-static-ppc

# qt_conf_static-ppc_x86.sh
./configure -static -release -nomake examples -nomake demos -arch ppc -arch x86 -platform macx-g++42  -sdk /Developer/SDKs/MacOSX10.5.sdk/ -no-framework -prefix /Users/zo0ok/qt_dev/4.5.3-static-ppc_x86

Please refer to the QT documentation to understand the advantages of carbon vs cocoa. Note that all those configurations are made on (and for) a PPC target. You might just want x64 if you build for modern Macs.

Mac OS 10.6, x64, XCode 3.2.6, QT 4.7.3

$ ./configure -release -no-qt3support -nomake examples -nomake demos -arch x86_64 -cocoa -platform macx-g++42 -sdk /Developer/SDKs/MacOSX10.5.sdk/ -prefix /Users/zo0ok/qt_dev/4.7.3-cocoa-x64

For some reason I got linking error an hour into the building process when using the MacOSX10.6 sdk. But those of you who are cutting edge dont use 10.6 and 3.2.6 anyways.

Windows SDK 7.0, x64 OS, QT 4.6.1
If you install the (free) Windows SDK 7.0 you get all compilers and make tools you need (you can use Visual Studio C++ Express, but it does (did) not build for x64.

Now, in the Windows world, compiling for different target cpus is quite simple (but a bit confusing). When you build QT the build happens in two steps. In the first step (configure) some tools are compiled (qmake, and more). That qmake will be used to build the rest of qt in the second step.

If you “cross compile” on Windows you will first use x86 MS tools to build x64 QT tools. In the second step (nmake) x86 and x64 tools will end up being mixed and you get problems (linking problems if I remember correctly. So, the easiest way to get it right is to not cross compile but:

  1. Build x64 QT using native x64 Microsoft tools
  2. Build x86 QT using native x86 Microsoft tools

This is easy, on an x64 version of Windows, as it runs x86 programs just fine. I really have to put a screenshot here, because there is no other way to explain this Microsoft mess:

The executables you see (cl, the C compiler, for example) are the x86 tools generating x86 binaries. You use those tools if you invoke vcvars32.bat. But then there are four folders containing the same tools for different other combinations:

Tools bat-file Description
amd64 vcvars64.bat Create x64 on x64 (USE)
x86_amd64 vcvarsx86_amd64.bat Create x64 on x86 (avoid)

Dont even bother with the irrelevant ia64 stuff. When you have a shell with environment from the correct vcvars file, this is how to build QT (exactly the same for both versions):

> configure.exe -static -release -no-qt3support -platform win32-msvc2008
> nmake sub-src sub-tools

Note: on Windows it was best to build qt “in place” (no -prefix used). So, I suggest you install the qt source to two different folder: c:\qt-x86 and c:\qt-x64. When you are done you can make your own copies of the vcvars-files and add qt\bin-folder to the path.

On Windows it is clever to add to your .pro-file:

CONFIG += embed_manifest_exe

That way, the exe file is the only thing you need to distribute (well, people might need the right c/c++ runtimes installed).

Linux
Your friends running Linux can compile from source :)

Building a Common Lisp program (ECL)

LISP is easy – you just need to start up the interpreter and start playing. But what if you are dependent on libraries, and you want to compile a binary? If you come from another background, like I did, it is quite confusing in the beginning.

Everything below applies to ECL – I think most things will apply to other Common Lisp implementations as well. I build a little command line utility that converts things to/from Base64 (using a library for that).

ASDF
ASDF is a tool that handles dependencies between packages, and also controls your build process (like make). Every project is called a System. Yours to.

When you download lisp packages they typically come with an asd-file, and one or more lisp-files. Each package goes in its own directory, and ASDF needs to know about each package.

I did everything from scratch and installed ecl in /opt/ecl. I put the packages in /opt/ecl/packages (not standard at all).

Project files and building it
These are the files my project contains, and how to build.

kvaser@kvaser:~/lisp/simple-base64$ ls -l
-rwxr-xr-x 1 kvaser kvaser  337 Mar 13 10:18 build.lisp
-rw-r--r-- 1 kvaser kvaser  197 Mar 13 10:18 simple-base64.asd
-rwxr-xr-x 1 kvaser kvaser 1389 Mar 13 13:51 simple-base64.lisp
kvaser@kvaser:~/lisp/simple-base64$ ./build
  ... ...

The binaries (of my program, and all dependencies) end up ~/.cache/, so thats where you need to go to execute your program (or just make a symbolic link to the project directory).

simple-base64.asd

(in-package :asdf)

(defsystem :simple-base64
  :name "simple-base64"
  :author "Zo0ok"
  :version "0"

  :components((:file "simple-base64"))

  :depends-on (:s-base64 :flexi-streams))

:components points to my lisp-file(s).
:depends-on lists other systems that I depend on (the base64-library itself, and a stream library that turned out to be useful.

simple-base64.lisp
Here is the source code to the program itself. It is very non-Lispy, remember, I am new to Lisp and I dont know how to program Lisp with style.


(defun print-usage-and-quit ()
  (format *error-output* "Usage:~%")
  (format *error-output* "  ./simple-base64 -e PLAINDATA~%")
  (format *error-output* "  ./simple-base64 -d BASE64DATA~%")
  (format *error-output* "  ./simple-base64 -e < plaindata.file~%")
  (format *error-output* "  ./simple-base64 -d < base64data.file~%")
  (quit)
)

;;; MAIN starts here

(let ( (mode-op-enc NIL)
       (mode-src-stdin NIL)
       (input-stream NIL)
       (output-stream NIL) )
  (cond
    ( (= 2 (length si::*command-args*) )
      (setf mode-src-stdin T ) )
    ( (= 3 (length si::*command-args*) )
      (setf mode-src-stdin NIL ) )
    ( T
      (print-usage-and-quit) ) )
  (cond
    ( (string= "-d" (second si::*command-args*) )
      (setf mode-op-enc NIL) )
    ( (string= "-e" (second si::*command-args*) )
      (setf mode-op-enc T) )
    ( T
      (print-usage-and-quit) ) )

  (cond
    ( mode-src-stdin
      ( setf input-stream *standard-input* ))
    ( mode-op-enc
      ( setf input-stream (flexi-streams:make-in-memory-input-stream
                         (map 'vector #'char-code (third si::*command-args*)))))
    ( ( not mode-op-enc )
      ( setf input-stream (make-string-input-stream (third si::*command-args*))))
  )

  (if mode-op-enc
    (s-base64:encode-base64 input-stream *standard-output*)
    (s-base64:decode-base64 input-stream *standard-output*) )
)
   
(quit)

Notice that nowhere the systems I depend on are included, they are just used when needed.

build.lisp
Finally the build-script, a lisp program that uses asdf:

#!/opt/ecl/bin/ecl -shell

(require 'asdf)
(push (truename #P"/opt/ecl/packages/s-base64") asdf:*central-registry*)
(push (truename #P"/opt/ecl/packages/cl-trivial-gray-streams") asdf:*central-registry*)
(push (truename #P"/opt/ecl/packages/flexi-streams-1.0.7") asdf:*central-registry*)
(asdf:make-build :simple-base64 :type :program)

Note that the build-script is the place to put paths to systems I depend on. Also note that I have included cl-trivial-gray-streams, a system I dont use directly, but flexi-streams needs it so I need to tell where it is. Finally, this pushing paths to *central-registry* is supposed to be the "old way". But for now I was happy to find a way that works, and that I understand.

Conclusion
As usual, when something works it looks simple, but it is tricky to get all the details right in the first place. I believe this is a good starting point for a small lisp-project that depends on available libraries.

Trivial Gray Streams
The package Trivial Gray Streams caused problems. The standard package I downloaded did not work for ECL (complained it could not find the system). I ended up installing Trivial Gray Streams using Debian apt-get. It puts lisp packages in /usr/share/common-lisp, and that version worked.

This applies to ECL version 11.1.1 and Trivial Gray Streams from Debian 6.0. The version of Trivial Gray Streams that did not work was dated 2008-11-02.

Lisp on Debian/ARM

After reading Revenge of the Nerds I decided it was time to learn Lisp. Programming without some kind of real project is boring, so my plan is to write some web applications using jquery and Lisp (for the back end).

Since I have a Qnap TS-109 running 24×7 I thought it would make a good development machine and Lisp web server. It runs Debian 6.0, but running Lisp on it turned out to be a challenge.

Debian, Lisp and ASDF
Debian supports installing different implementations of (Common) Lisp. However, it seems to be tricky to find a version that installs a binary on Debian ARM.

Also, there is a package depency tool for lisp called ASDF. Lisp implementations should come with it.

GCL
The only Common Lisp that I managed to easily install (i.e. with apt-get) in Debain 6.0 ARM was GCL. But it is a version of GCL that is 5 years old, and it does not come with ASDF.

clisp
I spent much time trying to compile clisp, but in the end I ended up with:

  > ( / 6 3)
  2
  > ( / 5 2)
  Segmentation Fault

Not so fun. Significant parts of clisp is written in assembly (both a good thing and a bad thing), and I was really not able to figure out if it was supposed to work on ARM EABI at all, or just on the old ARM ABI. So after much struggle I gave up clisp.

ECL
I managed to compile ECL from source. Not completely without hassle though. It comes with libffi, but I ended up with compilation errors (the processor does not support…). So, I downloaded libffi, compiled it myself and installed it in /opt/libffi. That was no problem, but I ended up making a symbolic link to include myself:

kvaser@kvaser:/opt/libffi$ ls -l
total 8
lrwxrwxrwx 1 root root   40 Mar 11 16:52 include -> /opt/libffi/lib/libffi-3.0.10rc9/include
drwxr-xr-x 4 root root 4096 Mar 11 16:37 lib
drwxr-xr-x 4 root root 4096 Mar 11 16:37 share

Now I configured ecl with:

CPPFLAGS=-I/opt/libffi/include LDFLAGS=-L/opt/libffi/lib ./configure --prefix=/opt/ecl --with-dffi=auto

That worked, and compiling went fine until ecl_min could not be executed, because it could not find libffi.so.6. I tried to fix that a while, but finally ended up making another symbolic link:

kvaser@kvaser:/usr/lib$ ls -l libffi.so.6
lrwxrwxrwx 1 root root 31 Mar 11 19:56 libffi.so.6 -> /opt/libffi/lib/libffi.so.6.0.0

After that, I ran make again to finish compilation. It went fine.

ECL, ASDF and cl-who
Now, where to put the Lisp http library cl-who? I copied the asd-file and the lisp-files to the ecl library folder and ran ecl as root:

kvaser@kvaser:~/lisp/cl-who-0.11.1$ sudo cp cl-who.asd /opt/ecl/lib/ecl-11.1.1/
kvaser@kvaser:~/lisp/cl-who-0.11.1$ sudo cp *.lisp /opt/ecl/lib/ecl-11.1.1/
kvaser@kvaser:~$ sudo /opt/ecl/bin/ecl
  ... ...
> (require 'asdf)

;;; Loading #P"/opt/ecl/lib/ecl-11.1.1/asdf.fas"
;;; Loading #P"/opt/ecl/lib/ecl-11.1.1/cmp.fas"
("ASDF" "CMP")

> (asdf:operate 'asdf:load-op :cl-who)    
  ... ...

Now, cl-who is compiled and installed, ready to use. Next time, it does not need to be compiled.

Hello LISP
I wrote a little Hello World program:

kvaser@kvaser:~/lisp$ cat hello.lisp 
(format T "Hello Lisp~%")
(quit)
kvaser@kvaser:~/lisp$ /opt/ecl/bin/ecl -load hello.lisp 
;;; Loading "/home/kvaser/lisp/hello.lisp"
Hello Lisp

Quite good (except I already know the file was loaded and it disturbs my output, but whatever. How about compiling it?


kvaser@kvaser:~/lisp$ /opt/ecl/bin/ecl -compile hello.lisp 
;;; Loading #P"/opt/ecl/lib/ecl-11.1.1/cmp.fas"
;;;
;;; Compiling hello.lisp.
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0
;;;
;;; End of Pass 1.
;;; Note:
;;;   Invoking external command:
;;;   gcc -I. -I/opt/ecl/include/ -I/opt/libffi/include -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -fPIC -Dlinux -O2 -w -c hello.c -o hello.o 
;;; Note:
;;;   Invoking external command:
;;;   gcc -o hello.fas -L/opt/ecl/lib/ /home/kvaser/lisp/hello.o -Wl,--rpath,/opt/ecl/lib/ -shared -L/opt/libffi/lib -L/opt/libffi/lib -lecl -lgmp -lgc -lffi -ldl -lm 
;;; Finished compiling hello.lisp.
;;;
kvaser@kvaser:~/lisp$ ls
cl-who-0.11.1  cl-who.tar.gz  hello.fas  hello.lisp
kvaser@kvaser:~/lisp$ ./hello.fas 
Segmentation fault
kvaser@kvaser:~/lisp$ /opt/ecl/bin/ecl -load hello.fas 
;;; Loading "/home/kvaser/lisp/hello.fas"
Hello Lisp

Ok, how to make a standalone executable?

> (compile-file "hello.lisp" :system-p t)
  ... ...
#P"/home/kvaser/lisp/hello.o"

> (c:build-program "hello" :lisp-files '("hello.o"))
  ... ...
#P"hello"
> (quit)
kvaser@kvaser:~/lisp$ ls
hello  hello.fas  hello.lisp  hello.o
kvaser@kvaser:~/lisp$ time ./hello
Hello Lisp

real	0m3.084s
user	0m2.920s
sys	0m0.160s
kvaser@kvaser:~/lisp$ time /opt/ecl/bin/ecl -load hello.fas 
;;; Loading "/home/kvaser/lisp/hello.fas"
Hello Lisp

real	0m3.127s
user	0m3.060s
sys	0m0.080s
kvaser@kvaser:~/lisp$ time /opt/ecl/bin/ecl -load hello.lisp
;;; Loading "/home/kvaser/lisp/hello.lisp"
Hello Lisp

real	0m3.113s
user	0m2.960s
sys	0m0.160s
kvaser@kvaser:~/lisp$ 

Clearly, some overhead is involved in invoking ECL. I compared to C:

kvaser@kvaser:~/lisp$ cat hello.c 
#include 

int main(int argc, char **argv) {
	printf("Hello C\n");
}
kvaser@kvaser:~/lisp$ gcc -o hello_c hello.c 
kvaser@kvaser:~/lisp$ time ./hello_c 
Hello C

real	0m0.012s
user	0m0.010s
sys	0m0.000s
kvaser@kvaser:~/lisp$ 

So, I can not use this method for CGI programming right away – each call to the webserver will take at least 3 seconds.

(Re)building packages from source in Debian, Ubuntu

It is very easy to build a debian package from source. Typically you want to do this if you patched it yourself, but want it installed the “standard way”. This of course works with Ubuntu as well. In this example, we are rebuilding shotwell. Pay attention to when sudo is used.

First do:

  $ sudo apt-get build-dep shotwell
  $ mkdir tmpsrc
  $ cd tmpsrc
  $ apt-get source shotwell

Now it is time to patch the source of shotwell. Then:

  $ cd shotwell-0.7.2
  $ debuild -uc -us
  $ cd ..
  $ sudo dpkg -i shotwell_0.7.2-0ubuntu2_amd64.deb

Of course, the names may depend on versions, architectures etc.

NOTE: Debian 6.0, use dpkg-buildpackage instead of debuild.

How to make the next apt-get upgrade not touch your custom version?

  # echo "hold shotwell" | dpkg --set-selections
  # dpkg --get-selections | grep shotwell

Replace “hold” with “install” to enable upgrade again.