Automated Builds

Automated builds have been a part of life since, well, since software was built. In the UNIX world the tool of choice has been make. (The remainder of the programming world has grown up and learned better ways of doing things. Oops... I appear to have given away my bias early.)

Now for its time, make was an OK took. It sure beat build scripts and the nice touch of abstracting away dependencies made life a lot easier -- especially as project got more and more complicated. The problem I have with make is that it has not really appreciably changed since it was introduced ~30 years ago. It was created mostly to compile C and C-like languages and only works with languages that suit the C world's paradigm of source files, object files and assemblies of various types.

The real computing world has advanced in that time and make is holding us back.

Outside of the UNIX world, make is increasingly a rare sight. Its limitations clash pretty badly with modern IDEs and most build environments these days have automated mechanisms using language features to track dependencies, recompilation requirements, et al. Some of them make horribly-ugly makefiles as part of the process and others will allow you to export their format into a horribly-ugly makefile if, for some bizarre reason, you want to use makefiles in your build process.

The UNIX world is different. I find the UNIX world insanely backwards in most respects. The only reason I use a UNIX platform at all is because I will no longer give Uncle Bill my money for the dubious honour of allowing him and his cronies to decide where and when I get to use my computer. And even there, were there a credible option aside from Linux (and no, I will never call it "GNU/Linux" for any reason whatsoever!), I'd be ditching it like the rotting mass of user-hostility that it is. Still, I'm in the UNIX world right now, for better or worse, and I'll have to adapt...

...my environment to me. That is, after all, one of the things that separates us from most animals: we can force our environment to suit our needs with far greater capability.

Right now I'm building a process for producing a Linux distribution from the Linux From Scratch (LFS) process. The process, as provided in the book, is a very tedious one involving typing a lot of commands in by hand. (To be fair to them, this is actually their intent. The idea is that once you've gone through the process you will know your system and its architecture inside and out. I have no quarrel whatsoever with the hard work these people have put into that project!) There is, of course, tools for automating the process in the project, but the tools provided are, in my not-so-humble opinion, not very good.

A frequent error old-school hackers make when trying to be "user-friendly" is assuming "if it has a GUI it is user friendly". A few minutes spent with jhalfs, however, pretty much proves that a bad GUI can be worse than no GUI whatsoever. I have successfully gone through the manual process of building LFS three times now. It was tedious. There were problems where I made dumb mistakes that an automated process would not make. Still, in the end, I got it done. I cannot say the same for my various attempts to use jhalfs. I've never got a successful build out of it and most of the blame I lay on its architecture. It's a half-baked GUI that uses the (admittedly cool) technique of reading the book sources to find the files that it needs and then constructs a complex array of scripts and makefiles to do the build. The problem is that it....

Well, I won't go so far as to say "it doesn't work" because I'm sure that some people have got it working. This highlights, however, the usual problem of developers and testing. "It works for me" means precisely nothing. Someone too close to the problem will overlook flaws because, subconsciously, they know what to ignore and pass over and what to pay attention to. In the hands of someone unfamiliar with the product, however, the flaws will show up like blink tags in HTML.

So how am I adapting my environment? Well, part of the problem, I maintain, with the jhalfs process is that it's a mixture of so many different things. And part of this problem is that it relies on crappy old technology like make and bash. There's a large impedance mismatch between these two items that renders them subtly incompatible with each other so that it is very easy to make fragile errors between the two. So my solution is to use something which has no disjoint layer: Rake.

Rake is, well, Ruby. Or, rather, a Ruby library and program which uses Ruby's great facilities for building DSLs. Several languages have (or have had) this ability -- Lisp, Scheme, Dylan, even Haskell if you squint at it right (hint: the monads I've mentioned earlier...). Lisp/Scheme, however, forces you into the Lisp non-syntax to access the functionality (and I absolutely hate the fingernail parings of a typical Lisp program) and Dylan has the twin problem of being effectively a dead language and of having its macro system built on a different syntax structure (more Lisp-like) than its core language (more algebraic in its setup). It is, in effect, a completely different language. Haskell, too, has the ability to make DSLs through the use of its extremely rich monad architecture, but actually building one can be as complicated for the uninitiated (which is 99% of prospective users) as is just doing the repetitive, dull, error-prone tasks DSLs are designed for. This will change, of course, as Haskell matures and more people hack away at it, but for now it's not suited to the task.

Enter Ruby. Ruby is a great scripting language with features that make it, for me at any rate, a natural-feeling one. It's one of those rare languages I encounter that fades into the background while I'm working, allowing me to concentrate on the task at hand while the syntax and facilities just silently support what I'm doing. And its expressive, flexible syntax is ideally suited for making DSLs.

Rake is such a DSL and one that suffers from no impedance mismatch between its declarative aspects (like make) and its imperative/object-oriented/functional aspects (like, say, the hoary shell scripts that make typically drives). For my LFS building project, for example, I will start off by making, essentially, a classical makefile with Rake's syntax. Once this is working (and generalized sufficiently to work on any system with Ruby and Rake installed), I will begin to modify it so that it contains within it the user interface necessary to any sane complicated build environment. The key to this is that the user interface will not be an outside program manipulating scripts and relationship files in highly-breakable ways, it will instead be an integral part of the building file using the dynamic features of Ruby and Rake to adapt to the user's wants and the system's capabilities on the fly. Instead of clunky, difficult text file parsing and alteration there will instead be elegant and simple alterations of a single file through clear, consistent APIs.

Future blog entries will detail this process until, at the very end, my own automated system for building Linux systems from scratch will be published here as a tool for anybody else wanting to make a custom Linux build.

And wait 'till you see what I want to do to the ugliness that is a Linux initialization script....