The Full Feed

2008-04-29

UPDATE: There seems to be some interest in Idiopidae too and I’ve been neglecting the Idiopidae project page while working on Vellum. I’ve updated that page with information and updated install procedures, so please check it out again if nothing worked for you.

There’s a Vellum bug fix release 0.16 out you all can grab if you’re using it. Due to a slight change in the forall command where I stopped using raw regex and used the fnmatch instead, the unit tests for that command weren’t actually running. This meant that bugs found their way in and now the latest version of Vellum 0.16 should fix all of that. It fixes that and just adds a bunch of documentation really.

Speaking of documentation, I’m still writing the big Book of Vellum and now have the first draft of Appendix A where I do a complete walk through the Vellum code. You’ll notice that it matches the code I just changed two minutes ago in the forall command. This is the advantage of Idiopidae. Simply changing the code and re-running the book build gets me everything instantly updated.

Grab the updated copy (still without much editing yet) and read through if you haven’t already. I’ve also built this one with Emacs colors for all my friends from Reddit and YCombinator News since I know all you Lisp whores can’t stay away from your editor designed for a man with 8.5 fingers on each hand.

I’ll be rotating the color scheme each time I release vellum just to have fun with it.

You can install this latest drop of Vellum in all the usual ways, and you can also grab from Bazaar:

sudo easy_install zapps pygments idiopidae vellum
bzr branch http://zedshaw.com/repository/vellum/
Grab From Launchpad
Grab From Here

LGPL?

There’s also been a few good submissions from some folks with suggestions for typesetting improvements, Pygments patches and improvements, Vellum bug reports, and the PIDA project wants to use Vellum as their build specification system. They’ve asked if I’d change the license to LGPL so I thought I’d toss the question out and see how people feel about the licenses.

My main motivation for going with GPLv3 is to prevent the common situation where corporations take the work, modify it, and don’t give it back. I’m also pretty down on corporations in general these days, so I may even consider a dual license of BSD for non-profits and GPLv3 for corporations.

Any thoughts on the subject of licenses for Vellum are welcome.

Bazaar Problems

If you just updated your Bazaar and you use Paramiko, you’ll find that Bazaar blows up because the Paramiko folks removed a function (in a fucking minor version release at at that). The easiest fix for me was to simply remove the offending package and then do:

sudo easy_install paramiko==1.7.2

You can review the bug report at their launchpad site for more information.

2008-04-26

UPDATE: I forgot to mention that this is my first LaTeX text ever and that it’s based on one of my first Python projects and that I wrote what you find below in about 3 days while dorking around on Vellum and Idiopidae. Since I’m both a Python and LaTeX newb please feel free to school me. Better yet, if you think the typesetting sucks, then show me your samples. See if you can beat this one sent to me by Kashif Rasul. I still consider what he’s created as the bar to get over.

I think I’ve gone insane recently because I’ve been obsessed with making the documentation for Vellum so great angels will cry tears of gold. I’m treating it as a practice book for the big book I’m writing for A/W by writing a fully LaTeX typeset PDF about Vellum using all the tools I’ve created for more modern literate programming.

The big “feature” of the book is insanely well typeset code with color. The typesetting is so good that everyone I’ve showed it to seems shocked by it. I’m not sure if that’s because they think it’s ridiculous to document such a small project that well, or because it’s way better than many other books. I’ll let you people tell me what you think too.

I tried to make it fit what many people use for their editor looks, but without making the book hard to read. This turned out to be easy as hell with Idiopidae and Pygments and in the process I learned quite a lot about TeX and LaTeX.

The end result is good so far, but I’m looking to improve it even further with:

I had to avoid floats because they would shuffle the code around making it hard to talk about it. Instead I just inject the code right there and fake out a float. What I want is to have code which is broken between pages cleaned up so that it either has a “blah continued…” marker and/or just jumps to the next page. That’s probably going to be hard as hell so it might have to be a manual decision after the whole thing is written and positioned.
I’d like smaller snippets to exist in a side float or in a two-column page so you can read the code and the discussion of it side-by-side. I really good example of this is the discussion of the grammar for Vellum in “Appendix A: Parser”.
Play more with the boxes and outlines for the code sections to see how their status in the prose can be made more or less significant.
Figure out how to use Bazaar to sequentially version code and then get Idiopidae to understand the version to use in a discussion. This would probably fall flat with source code since that just changes, but it would be handy for some examples.
Fix up the Pygments line numbering. Right now they’re kind of annoying and interfere with the reader’s ability to cut-paste the code.

The Vellum book is a work in progress, but check out how it looks already in PDF form. Definitely look at Appendix A to see how well the source highlight works using Idiopidae to snarf in live code from the Vellum source.

Vellum and Idiopidae have made creating the book for Vellum incredibly easy. I now don’t worry about how the code is included or formatted, and I can access it using logical sections. I just tell Idiopidae the file and section and it makes all the moves. A bit of LaTeX glue to wrap it in something pretty and I’m off.

I can also do some really interesting things with sample outputs and incrementally built source files in the book. When you write a book about code you typically want to start with a simple example of some source file, and then build it up to more complex levels as the prose progresses. Traditionally this meant generating a series of individual files each with the slight modifications. Problem is this means maintaining all those little files for just one chapter.

With Idiopidae I can actually put multiple files into one larger file, and put the code snippets for the progressive discussion into different sections. I hadn’t thought of this until I did it yesterday, so I may try to formalize this in Idiopidae sometime this weekend.

There are a few things I want to add to Idiopidae which will probably involve a rewrite. If you use it, don’t get too attached to the syntax. What I’m thinking is that I’d like a feature to run commands, clean the output up, and then inject that into the book. I’d also like Idiopidae to keep track of md5sums for all the sections it includes. If the md5sum of a section changes then it’ll report an error assuming that you have to change the text around that section to match the code. After you change it, there’ll be a command line option to tell Idiopidae to update the md5sums.

I think this “md5sum tracking” feature could really help keep books about code in sync with the code. The assumption is, if the code changes, then the prose probably has to change as well.

Another feature I want in Idiopidae is to use the Pygments parser of ctags on the files so that it can find logical sections based on the already structure source. If all you need to do is include a series of functions, then having to sprinkle the source with export comments that match the function names is stupid.

As I think about these features though, what I find is a lot of overlap between how Idiopidae works at building books and how Vellum automates building anything. It might be worth looking at Idiopidae reusing Vellum’s parser and build engine so that you can write a nice little description of all the pieces that come together to make a book. It would be a higher level description of the book and designed to also embed Vellum build tasks inside import statements so that you can run code and have the output injected.

Check out how the book looks so far (grammar errors and spelling included) and let me know what you think. My plan is to wrap all these little tools up into one nice tool that helps programmers write better documentation, or at least helps them find someone who can write it without much fuss.

Knuth On Literate Programming

The ideas in Idiopidae are nothing new, as Donald Knuth has been actively promoting Literate Programming for over two decades. The main problem with LP though is its insistence on combining the code and the prose into a single file. There’s serious technical and social issues with this design choice, but in this very good interview with Knuth he mentions a quote that summarizes things quite well:

Jon Bentley probably hit the nail on the head when he once was asked why literate programming hasn’t taken the whole world by storm. He observed that a small percentage of the world’s population is good at programming, and a small percentage is good at writing; apparently I am asking everybody to be in both subsets.

However, this doesn’t cover the absolute hardest technical problem with combining code and prose in the same source file: syntax escape overload.

When you use most LP systems you’re taking the source or two whole languages with completely different lexical and semantic meaning, and then trying to cram them together. This means juggling in your head two different syntax structures and how to merge them together with escape sequences. The technical difficulty of this probably borders on an grammar that would rival Perl or Ruby’s.

This is a major technical hurdle for both people implementing such systems and for the users. As an implementer, I have to create a parser or some hack that can accurately handle both languages. I also have to figure out how the escape sequences are given to the users, how they can encode elements of one language in the other (say for giving constants a bold font). This is a huge task, so no wonder why all the literate tools out there also do a shitty job at finally typesetting the resulting code.

For the users they have to use their brain in ways that don’t make sense for either a programmer or a writer. A programmer has to work in terms of structure and algorithmics with the hardest problem on many projects being finding things of interest. A writer has to work on the creativity of the prose, entertainment, whether it reads so that it makes sense, and how best to keep the reader engaged.

When I code I don’t give a damn if the reader is engaged, I care if he can understand what’s written, but I’ll be damned if I’m supposed to entertain some dork who’s paid to know this stuff.

These flaws then extend to every other aspect of computer science. When people do LP they typically keep all the code and prose in one giant file, maybe a few medium sized ones if you’re lucky. This is a nightmare for revision control since people on a team will constantly be conflicting with other team members. This makes it hard to break out the work, share it with others, hell even find where something is in the source.

Can you imagine trying to grep for a normal C idiom but having to modify what you look for to handle TeX escaping just in case it shows up in a spot that’s inside TeX?

In the end, LP seems to have missed out on the biggest advancement in software development since it started: compilers. A compiler no longer expects you to keep your source in one file, but instead has insanely good (or in C’s case, okey-dokey) support for inclusion, exporting, combination, and sharing of code. Modularity and encapsulation into libraries, modules, objects, and functions has all been supported by compilers since they started being used to create larger projects.

Yet, LP almost misses this and does an “inverse compiler” You write the final combined document and then it writes what you’d normally include in the document. Insane.

It makes much more sense and works a hell of a lot better with modern tools (“modern” being after 1980) if the way programmers do Literate Programming works the way they do regular programming.

But, TeX is still pretty damn sweet.

2008-04-18

While working on a more complex build I decided to make recursive imports work and clean up the syntax for imports in Vellum

Now, your imports are more explicit (just the way Python people like it) and more consistent:

imports [
    recipe(from 'scripts/testing' as 'testing')
    recipe(from 'scripts/dist' as 'dist')
    recipe(from 'scripts/sample' as 'sample')
    module(from 'vellum.commands')
]

This is the import stanza from Vellum’s build.vel file. What you see is there’s two type of things you can import, recipes and modules. Recipes are just other Vellum scripts that contain common tasks you need. Modules are Python modules that have code you need in the form of commands.

I pushed the version of Vellum that has this as version 0.14-rev85 so it’s on the Cheese Shop and in the bzr and launchpad. I would avoid it right now unless you’re interested in the latest code as it currently doesn’t load from the ~/.vellum/ directory yet so your common stuff will break.

I’ll do another release later today that brings that back.

Python Is My make

I’ve finally figured out a huge difference between what I’m doing with Vellum and what make does that turns certain types of programmers off from Vellum.

When you work with a compiled programming language like Java, C, or C++ the build tool is an essential part of your daily working. You can’t do anything without make happily building and compiling your software so you can run it.

Without make you can’t test, run, analyze, debug, or do anything. The entire chain of build junk in a compiled language is useless until you follow a large number of steps to finally produce an executable. Even if you have just one .c file you still need one additional compile step before you can run it.

(NOTE: Although tcc makes this possible, very few C coders use it.)

Another necessity is autotools. Without autoconf and friends you can’t build your software on multiple systems and still use advanced features. This makes autotools another essential thing, but only if you use a language like C that has portability problems.

In a dynamic language like Python or Ruby I don’t have this problem at all. There is no need for a make to run my program. I just run it. The interpreter is my make and does all the stuff I need to make the code I wrote work. I only violate this when I write a C extension, and even then both languages have tools that help me do that consistently.

I don’t need complex steps, or linkers, or loaders or gold from Google to happily work all day long. I don’t need things like autoconf because Python is portable unless I’m writing C extensions, and even then ctypes lets me avoid a lot of that for quite a while.

When I code, I need to run unit tests, but even that is handled for me by nose so there’s not much for me to do manually. I don’t have a build phase, I don’t have build errors, I just have code that runs or doesn’t.

I can even use Python as a debugger, a console, the works. I don’t need fancy builders and test harnesses just to see if a few lines of code work. With a compiled language I’ve had to create entire Makefile or Ant builds taking hours just to try one idea.

In essence, for programmers using compiled languages make is their Python.

For Python programmers, make is nothing more than a complicated if-statement with lots of magic.

This is why, when I wrote Vellum, it didn’t really need to know how to handle the build junk of a .c file. Vellum is higher level than this because Python already does all the crap that make does. What I needed was a tool to help automate the things Python doesn’t do like packaging, distribution, test automation, bzr management, and code generation.

To a programmer who’s entire life belongs to make this seems retarded. On the surface it seems so deficient because it can’t automagically figure out that depending on a .o file means to obviously build a .c file (unless it’s .c++, or .cpp, or .C, or .d) using cc (or gcc, or g++, or …).

Vellum obviously can do those things, assuming someone decides to write the commands needed for such magic, but why bother? I don’t do much with those languages these days, and when I do, why I’ll just write the bare minimum make necessary to build the C code and then have Vellum run that.

No point re-inventing 30 years of mythological C building lore just to keep a bunch of people happy who spend 50% of their day waiting for a program to make their program executable.

What I’ve found is that people who use make are very focused on how they can use Vellum to do what make already does. What they miss is that Vellum, unlike make, can be extended with Python to do anything you need.

2008-04-15

I worked on Vellum today after waking up from jetlag and Poland. I feel like it’s near ready for actual use by people. I even managed to polish it off with a nice little command line option for dumping the commands a build spec uses including their documentation. Check out this Pastie clip that shows it off.

What you see here is first the vellum/commands.py file with the commands that you can use. This file is actually loaded the same way your own commands could be loaded, by just saying what to import as a regular Python module. What Vellum does is look in this file for anything that is a function and then makes those into commands.

Next is the actual build.vel file that makes those commands active in the modules section.

Now, the nice part is right after that you see the results of the vellum -C command line option. It literally just pulls out the documentation for each function and prints it as the command’s documentation. If you give -C any command names it just prints those out.

This means you can write Python functions for commands you want, then put them in a module in ~/.vellum/modules or even just make a regular Python project out of them and install in the normal way. Once you do that you can use those in your build specs.

Check out the project page for information on getting this and using it. I’ll have more extensive documentation on it soon.

620 Lines Of Python

Using this awesome CLOC program I figured out that Vellum is only 733 lines of code, but 620 if you take out the parser.py file generated from parser.g. That’s pretty fucking amazing considering all the shit this thing can do already.

Think about this for a second: In less than a month I managed to create a build tool that parses it’s own build format with dynamic scoped imports, loads additional Python modules to give you a command structure, and then runs arbitrarily sized shell or Python code according to the dependency structure. It’s also got a decent test suite, is fully self-documenting, and has comments on almost every function.

That says a lot for Python. Combine my efficiency working on Vellum with how quickly I cranked out Idiopidae and Python is really proving itself as a great workhorse language.

I also really enjoyed the interplay between people who wanted a Python only syntax and those who liked YAML syntax. What I got out of that exchange was what I consider a very nice setup for Vellum where users have a nice simple format for the build file that’s high-level, but can easily write new commands in regular Python modules for more complex work.

I don’t think I could have made that leap if it weren’t for the people who were talking to me about the benefits of both needs.

Next up for Vellum is a more extensive set of documentation for it so everyone else can learn to use it, and then a way to distribute recipes. I’ll probably use the Launchpad project for the recipes.

Let me know if you grab Vellum and start getting it working. It should just take a simple sudo easy_install vellum to get it now.

2008-04-13

Update: I totally fucked up this post. I corrected the URLs but just wait until tonight when I drop another release with some goodies.

I’m still in Poland enjoying RuPy 2008 but thought I’d put out a release of Vellum 0.11 for people to play with.

You can grab it from launchpad where you can get the bzr repo and download source tarballs.

You can also easy_install it, but this is new so let me know if it works. You should just have to do:

easy_install vellum

Finally, if you want the bleeding edge Vellum then track my bzr repo:

bzr branch http://zedshaw.com/repository/vellum/

Now, what’s in this release? It basically has all the features I want but isn’t well documented. There’s extensive tests and docstrings on every method, so a brave programmer could figure it out. For the most part the Vellum project page has samples of the new Vellum format, and there’s sample commands in the vellum/commands.py file in the source.

What you get in this release is the ability to write new commands for your build spec in pure python, plus tons of little clean-ups and bug fixes.

Try it out, and let me know if you can get it working. I’ll be writing a short manual for it pretty soon.

My RuPy Presentation In Bazaar

I’m putting up the full Bazaar repository for my RuPy presentation so other people can grab it and play with it. This presentation was done using all my tools. Vellum for building it. Zapps for Vellum. Idiopidae for code injection into the presentation.

The presentation was done using LaTeX Beamer to make this PDF presentation which was actually generated from this YAML file using a small Python script I wrote to generate the necessary (and repetitive) LaTeX.

I’ll be cleaning this up and writing about it when I write about RuPy sometime next week. So far RuPy has been a blast and I really like Poznan. Virgin Atlantic and Heathrow Airport however can die in the ocean.

2008-04-09

Well, well, well. Google comes out with their App Engine thing and what do you know, they use YAML for their application descriptions.

I don’t see any Pythonistas complaining about it publicly. Imagine that.

See, that pisses me off because several people said nobody would adopt a project if it didn’t use Python from stem to stern. I personally think if you can’t think in a few languages on a project then your a dumbass, but I obliged folks and wrote a Python parser for Vellum’s builds.

Then people said they didn’t like that format either.

This folks is the classic problem with programmers today. They absolutely refuse to learn anything new unless they can see that learning the new hotness will give them an immediate 200% boost in salary or get them hot honeys at the next conference.

I use a YAML format for a build file and it’s not good enough. I write a Python one and it’s still not good enough.

Google uses YAML and people can’t wait to blow Eric in the back bathroom as thanks for using Python.

Well, since that’s the case I’ve decided to just write my own damn format for Vellum. And it’s got strippers, and blackjack, and uh fewer symbols to deal with, and less typing. Wait, screw the strippers and blackjack, how about just less typing? Awesome. Since I can write a parser in about an hour I just did it. So there.

That’s right, Vellum just has its own syntax. It’s obvious Pythonistas can learn a new format when it’s tied to the handle of a massive pot of gold. My format is even cooler than YAML or Python, will give you a 300% boost in salary, and will eventually hand mold a Jenna Jameson Real Doll™ for you since you can’t get hot honeys no matter what language you learn.

Check out this abreviated grammar for the whole data format:

rule input: ( reference | COMMENT LINE)* ENDMARKER 
rule expr: atom | reference | structure
rule reference: NAME expr 
rule atom: NUMBER  | STRING  | SH LINE 
rule structure: LSQB elements? RSQB  | LPAR dictmaker? RPAR 
rule elements: (expr)+ 
rule dictmaker: (reference)+

That’s right! My format is only 7 productions and it can produce the same structures Python and YAML can (minus retarded shit like references). Take a look at the sample now and bask in my skills as the format makes you calm and clean. I was definitely channeling REBOL on this one.

And this only took me like 3 hours to craft and make Vellum use again (thank god for tests). That’s right, I’ve changed formats for the Vellum build file 4 times and I’ve written two parsers for it and I’m having fun so fuck you all. (sniff).

All Kidding Aside, GAE Is The Shit

Alright, I actually think it’s hilarious that Google came out with their stuff just a few weeks after PyCon2008 and didn’t seem to clue in any of the web framework developers. I was hanging out with the Turbo Gears and Pylons guys for most of PyCon2008 and they didn’t mention anything. I wonder how they feel about that?

I’d be a little pissed, but just for a few seconds before I started counting my mountains of cash cause Python is gonna be hot! I’ve already got two or three learning applications lined up and I’m even thinking of a slick little script to make cranking out simple GAE applications easier.

Yes, GAE is incredibly cool and I’m am so glad I jumped on the Python trail just before they released it. I can now take my new found fancy Python learnin’ and go do some real programming for a change. I don’t have an account because Google is a stingy bastard, but who cares. I can actually download their whole SDK and have fun tinkering on it. By the time I get done making something to share they should be open for business.

What I’m very impressed with is the deployment mechanism. After working on Mongrel for years and watching all the hosting companies barely make any headway with Ruby, it is so nice to see someone get it right. Just set it and forget it. None of this binding together 6 shitty services with duct tape like you do with AmazonWS.

Echo2 -> Campfire -> HuddleChat -> Utu

Everyone seems to like 37signals’ Campfire so much they came out swinging with claims of a ripoff from Google’s HuddleChat.

Well, silly boys and girls, rails-core ripped off the idea and probably most of the workings for Campfire from NextApp Echo2 ChatClient Demo. I know this because I was in the rails-core IRC channel and I showed them how cool this Echo2 framework was, including that chat demo. A few weeks later they had Campfire and since they say it took them two weeks to write it, I’m guessing they got lots of inspiration.

Sure, they probably won’t admit that, since they’re “innovators” you know, but it’s a really odd coincidence.

Who knows, maybe they were working on Campfire all along and they just used their psionic telepathy to will that I go find the Echo2 Chat demo and paste it into their IRC. Or, maybe I just remember it differently, who knows. My brain don’t work that well anyway.

However, there is a precedence. They also admitted stealing the mechanism for updating two divs at the same time from one Ajax call from the Nitro project. At the time this was the biggest barrier for things like RJS to work, and rather than figure it out themselves they stole it from poor George and never gave him credit.

Sure this simple Echo2 demo is a pale pathetic competitor to the glorious Campfire, but it also is just a demo that hasn’t changed in years. I actually dump it on Java application servers when I need to do some simple chat with people. It’s pretty damn useful, so I can see why 37s would take the time to rip it off.

Now, I will gladly admit that My Time With The Rails Core Cult was a major inspiration for the creation of Utu and that their Campfire actually did make me hate stupid people so much that I felt I needed to create a weapon of destruction for weeding them out. Probably the reason I haven’t worked on Utu in a while is that I stopped hanging out with the Ruby dickheads so I’m not so inclined to hate the flesh of all who type “fuck you” at me.

Therefor, I’ll freely admit that rails-core, you guys inspired me to find new ways to hate people over chat. Thank you.

2008-04-05

Latest Vellum (0.8) is out without YAML and with Python but also without all of Python. The motto for this release is:

You love your Python, but I don't trust your Python.

What this means is that Vellum now uses a simple small parser written with Zapps (you need that to install this release for now) that parses only enough real Python to describe a build.

This amounts to import statements, simple assignments, full strings, numbers, dicts, lists, tuples, and that’s all.

Since now the imports work by loading other build files recursively it will also find builds using . notation and pull them out of a ~/.vellum directory so you can have common build stuff available.

This release is still rough, with me cleaning up code that and I still need to implement the commands you can use, but otherwise try it out if you’re tracking the project.

You can get a release from Launchpad or from here and the Bazaar repo is still up at “http://www.zedshaw.com/repository/vellum/”.

If you’re interested in how the parser works, take a look at the vellum/parser.g file.

I’ll have another release shortly to clean things up.

2008-04-01

I blame Rage Against The Machine for today’s distraction and productivity. Between doing tons of chores and crap I managed to cram in three releases of Vellum giving it pretty big feature improvements while keeping the total code below 300 lines of Python.

Check out the Vellum project here, but I also setup a Launchpad project at https://launchpad.net/vellum for people to collaborate on the project with me. There you can download the latest, report bugs, and get at the latest code via Baazar if you can’t hit my bzr repository

The big changes that this release gives you is what I call “Strings in Python Clothes” where you can write a Python based build description, and rather than using the <<<, <—, or >>> syntax you can use functions of sh(), log(), given(), target(), and py().

All these functions do is return what you give them formatted into what Vellum wants for a build target description, but this seems to be a huge sticking point for Pythonistas. I mean, seriously I haven’t seen such a myopic dependence on arbitrary lexical elements since working with Java and C# guys. I actually received emails from people saying that if it didn’t have sh() and array syntax that nobody in Python would use it.

What I did like though was that everyone who disagreed with this syntax took the time to explain why and help me understand the culture of Python. I don’t agree with this attitude at all, being a language pragmatist and all, but I can understand it so I’ll help people out.

If you have insight into this weird obsession with everything being always in Python no matter what I’d love to hear it. From my perspective this is why so many programmers are incapable of keeping up with trends. This refusal to learn another language (or anything really) and the strange belief that an IDE will make you a better programmer (my two favorite stupidities) means that they stay slaves to what they have now and can’t formulate a perspective about what is possible.

For example, Vellum doesn’t require YAML at all, and it just uses PyYAML which is pure Python. Yet, other people seem to think PyYAML needs Syck, requires C, and other FUD they learned about two years ago. Each person had to be told (several times) that PyYAML was good for the task and they were wrong. This still lead to complaints about YAML’s syntax, even though it’s nearly the same, and then even though I pointed out they can do Python, the Python build syntax still wasn’t Python enough.

What they don’t realize is that there’s a reason I want YAML: it’s not executable. This is a huge deal when you want to distribute builds to other people but still want to keep things safer (notice I said safer not safe). Being able to grab a YAML description of the build and load it in dry-run mode with a guarantee that nothing will be run at all helps people who need to do automation of other projects.

Imagine this, with Vellum as it is you could start building other people’s projects, but before you do have it dry run and save that build output. The next time you build, do a diff on the dry-run. If the dry-run changes then don’t do it automatically, but require a human to approve it. With a simple diff of the previous build and the current build process you can trivially spot changes in the build process.

However, Python build descriptions can’t do this because they are Turing complete and a victim of the Halting Problem. In order to find out if the Python script could cause damage you’d have to solve this theoretically impossible to solve problem.

Alright, so why include Python build descriptions as an alternative? Because there’s some things that can’t be described with YAML. Things like a configuration script that analyzes the machine for the remaining build process. In this case, having a small config.py that gets imported into the larger build.yml solves the problem.

But also, it was trivial to add both and get the best of both worlds. People comfortable with YAML or who need safer builds can use YAML. People who hate YAML and need Python everywhere to be happy can use Python only builds. People who are pragmatic and want to just use whatever works best for the job can use both (like I do).

Everybody is happy hacking friends and Vellum is still small with a powerful punch.

2008-03-31

Just a quick announcement that I’ve managed to get Vellum nearly close to what I need for building all my stuff. It has been refactored to be funnier and also includes a simple “import” system that implements an idea of recipes. The recipes are still rough, but do work by taking the contents of another .yml (YAML) file and putting those contents into the master with a namespace.

Vellum also has a fairly complete unit test suite that exercises most of the system that I can test. Not sure if that’s how Python people do their testing, but it seems to be working good for me.

If you’re interested in trying it out, grab the latest 0.4 release or grab it from the bzr with:

bzr branch http://www.zedshaw.com/projects/vellum/

The funny refactoring I did was to break up the one large class into three smaller classes:

vellum.Press—Loads either YAML or Python build specs and does the “coagulate” needed to resolve imports.
vellum.Script—Responsible for taking the results of Press and building the internal executable data structures for the script.
vellum.Scribe—Runs a script you give it according to the options set.

Ok, that joke has been choked to death and probably won’t get any funnier, even if it ever was all that funny to begin with.

Now, here’s what you can do with Vellum’s new recipe stuff:

Put common tasks you have into other .yml (not .py yet) files.
Put those common files into projects you start so you don’t have to write them over and over again.
Have your primary build.yml load them, and then the tasks and options in the imported recipe file get put into a name space. This means if you have distutils.yml with a target of “clean” then your build.yml will have a target of “distutils:clean”.
You can put these files in other directories too and the slash will apply to the namespace name.
Since the main file can be a .py you can have it generate a .yml that gets loaded as part of the configuration process. This can be the start of an autotools style config system (but one that doesn’t suck).

I’ll just be adding two tiny little things this next week and then I’m gonna call Vellum 1.0 since it does most of what I need:

A path mechanism so that these common recipes can live in some central place and referenced easily. Then I’ll post up a bunch of recipes you can grab and install for common stuff.
Ability to have a recipe be a .py file so that they can get run like .yml files during the resolution.
Documentation and probably hosting the project on launchpad so people can contribute.

The Vellum project page has more information and a few more samples you can play with.

2008-03-29

Well it was a busy day for me in Python land:

Created a project for Vellum and bzr repository
Used Vellum to build Idiopidae, itself, my resume, and my web site.
Added the ability to have condition guards that are just evaluated Python expressions. You can see in the sample build.yml files in the Vellum source and project page.
Reorganized Idiopidae so it puts things into a good namespace rather than polluting. Also included a bunch of missing files.
The Vellum builds for my Python projects automate all the stuff you need, including setup.py runs, cleaning, bzr commits, and nosetests.
The -F option to Vellum will make it “force” all the conditionals, so that it runs every target no matter what.

Just have to add a good recipes mechanism and Vellum should be great. I’m leaning toward a template system that lets you put a recipe on one of your lines and the required options and that will generate new lines to be run. That or a simple way to just load other files and access that file’s tasks in a namespace. Or maybe both, gotta see what I need.

I was also talking with some friends, and they mentioned it’d be great if Vellum could distribute the builds to multiple machines in such a way that nobody needs to actually enter IP addresses. I’ve always wanted this too, but never thought of doing a build tool. Now that I have a fairly complete one, I think I’ll just try making this feature. I don’t really need something like this yet, but I can see having to run tons of tests on the code in my book and all my projects. Being able to push that out to the many machines in my closet will be nice.

What I’m thinking is using PyZeroConf or Avahi so that all the machines will announce themselves and their roles, and then a master pushes the Vellum script to them and runs targets using HTTP.

So far I’m really enjoying making stuff with Python. Zapps is very useful for the little parsers I want. Idiopidae is great for writing the docs and working good for my book so far. Now “Vellum” will help build my book and all the required software in the way I’ve always wanted these things built.