Vellum: A Simple Python Build Tool
NOTE: I’m working on a medium sized book about Vellum that will serve as the primary documentation and also try to teach you about the art of automating things. You can grab the unfinished PDF to follow my work and all the source to the book is in the Bazaar repository.Vellum is my project to create a nice build tool for Python, or really for anything you need to
automate in a consistent fashion. Vellum uses a simple format that takes no time to understand,
but also lets you easily write Python modules that it can load so you can extend what Vellum
does. Vellum tries to be safe whenever it can, and avoids complexity at all costs without
losing features.
The primary goal with Vellum is to be simple, safe, and explicit. No more magic chains of things happening that you can’t figure out.
Features
- You can write modules in Python and put them in your ~/.vellum/modules directory to get your own commands in Python when Vellum’s spec format isn’t capable enough.
- Builds are written in a safe and very simple format that is like “Python minus the crap”.
- Build files are not loaded with the Python interpreter and are actually parsed so they will (hopefully) be safer and easier to analyze.
- Options, imports, dependencies, and targets are separated into different sections to make the file simpler to debug and write.
- Call other targets out of order in those cases where the dependency graph has to be modified.
- Vellum has tons of debugging output and prints very good meaningful error messages when it can.
- Runs multiple or single lines of shell or Python code in a reliable way according to dependencies.
- Output formatted messages to the user.
- Test for true conditions to avoid running a target, which is more flexible than having target files that must match mtimes.
- A simple code generator system that will let you create templates for generating tasks of particular types.
- In addition to importing Python modules to make new commands, you can also import other build files to create master/child or more modular build systems.
- Imported build specs are automatically put into a namespace so that names don’t clash, and you can rename the imported namespace to anything you like.
- Since commands are functions, they become self-documenting in Vellum so that it’s easy to see what commands a build spec has as well as how to use it.
Download
You can get vellum either by using easy_install, from Launchpad, or through Bazaar.
From Launchpad
You can grab the latest version from the Launchpad project page
for the 0.0 series. There should be .tar.gz files, .egg packages, ChangeLogs, and sometimes samples.
With easy_install
Vellum is now registered in the Cheese Shop so you can
use easy_install to install it with one
command:
$ sudo easy_install zapps pygments idiopidae vellum
Once you do this it should be available. All the .vel files should also be included in the
distribution so you can check it out to see some sample. They’re also documented below.
You can also just install zapps and vellum if you don’t plan on building the
PDF book but it might be good to get all the gear to help me test it.
Tracking The Source
Your best way to track what I’m doing is to grab the Bazaar repo:
bzr branch http://www.zedshaw.com/repository/vellum/
Let me know if that don’t work for some reason. Feel free to send me patches, and it’s GPLv3 licensed for now.
Getting Started
Vellum takes a build.vel file, loads it, and runs any targets you give it
on the command line. Here’s the command line help:
Usage: vellum [options]
Options:
-h, --help show this help message and exit
-f FILENAME, --file=FILENAME
Build file to read the build recipe from (no .vel)
-q, --quiet Tell Vellum to shut up.
-d, --dry-run Dry run, printing what would happen
-k, --keep-going Don't stop, build no matter what
-T, --targets Display the list of targets and what they depend on
-F, --force Force all given conditions true so everything runs
-D, --dump Dump the build out to a fully coagulated build.
-s, --shell Run the vellum shell prompt.
-I, --install Create the ~/.vellum directories.
-v, --version Print the version/build number.
-C, --commands List all commands and their help. Give a name to see just
that one.
As of 0.13-rev79 this is the options, but normally you don’t use these. If your
build.vel had a target called build you’d do this:
$ vellum build
You can list as many targets as you like, and if the options stanza (mentioned below)
has a default setting then that is run if you don’t give any targets.
Vellum’s Build Spec Format
Vellum uses a simple format for it’s build file, and then modules you can write
in Python to implement custom commands. There is an overall philosophy to this
design that comes from years of dealing with builds written by other people:
- The format goes along with Lisp’s idea that code is data and creates an executable
description using simple lists, dicts, and name(value) expressions. You can think of
Vellum’s format as Lisp+dicts where every “function” takes only one argument (which
can be a list, or dict, or function that …) - The goal of the main build.vel and any recipes is to make something that is easy
to analyze (both by a person and by Vellum) without executing any code. This is
important for safe software distribution and distributed builds. - In order to provide flexiblity, Vellum let’s you create commands with Python but
loads them from trusted places. - Everything about the build format and the command modules is designed to let you
create reusable builds. This is why the dependencies are separated from the targets.
With build systems that attach the dependencies to the targets it’s too hard to just
reuse the targets in other builds. Breaking the dependencies out means that you can
pull in whole builds as components and reuse them, even with name spaces. It’s
also easier at a glance to see what every task does and what it will call. - Finally, unlike many other tools, Vellum does not force you to put all of the processing
logic into an entirely declarative structure. Some tasks are better specifie in an
imperative way, so Vellum allows you to specify dependencies at any point
in a target’s definition.
Here’s the main Vellum build.vel as your first sample:
# This is a working build spec, but it is also an example # so it consists of a lot more stuff than you would normally # find in a real project. imports [ recipe(from 'scripts/testing' as 'testing') recipe(from 'scripts/dist' as 'dist') recipe(from 'scripts/sample' as 'sample') recipe(from 'doc/book' as 'book') ] options( project "vellum" default 'tests' sudo 'sudo' version '0.17' website '../zedshaw.com/output/projects/vellum' bzr.revision '.bzr/branch/last-revision' version.file 'vellum/version.py' setup( name 'vellum' version '0.17' author 'Zed A. Shaw' description 'A flexible small make alternative for Python programmers.' author_email 'zedshaw@zedshaw.com' url 'http://www.zedshaw.com/projects/vellum' packages ['vellum'] scripts ['bin/vellum'] ) ) depends( build ['tests' 'version.gen' 'dist.install' 'dist.sdist'] commit ['dist.gen.setup' 'parser' 'dist.clean' 'book.clean'] tests ['parser' 'testing.run'] release ['build' 'dist.release' 'book.release' ] ) targets( commit [ $ bzr log --short > CHANGES $ bzr commit $ bzr push ] version.gen [ py [ |rev = open("%(bzr.revision)s").read().split() |ver = {"version": version, "rev": rev} |open("%(version.file)s", 'w').write( | "VERSION=" + repr(ver)) ] ] parser "zapps vellum/parser.g" dist $ cp doc/manual-final.pdf %(website)s cloc [ $ cloc --report-file=doc/test_cloc.txt --no3 --by-file tests $ cloc --report-file=doc/source_cloc.txt --no3 --by-file --force-lang=python,g vellum bin $ cat doc/*_cloc.txt ] )
Here you see that you get Python’s usual comment style, then you have a format that
consists of what looks like function calls with weird parameters in them and lists.
The file breaks down into a logical structure like this:
- imports — Other .vel files to load as “recipes” using import commands or “modules” that have Python definitions of new commands you want.
- options — A dict sets various options the rest of the file can use in commands as % replacements.
- depends — A simple dict that says which targets (or fake targets) need which other targets first.
- targets — A dict with the targets and what to run for each one as commands and shell.
The format is then very simple once you understand a few little grammar rules:
- Each element is a reference which consists of a name and an expression. Example: targets(commit “echo test”) is a reference that has the name “targets” and has a dict as the expression (which also contains a reference named “commit” that has an expression “echo test”).
- An expression can be anything else, even another reference. You get NUMBERs, STRINGs, lists, dicts (but with parenthesis), and a special thing called a “line string”.
- During processing Vellum assumes that a plain string is just a shell command. Since this is so common there’s a special string that works like Python’s comments called a “line string”. Any line that starts with $,>, or | is turned into a string preserving all space. This lets you easily do multi-line Python or shell commands and makes the file look nicer.
- The whole file is a dict, which means that order doesn’t matter for the above mentioned stanzas.
This means that you do one reference for each of the above stanzas, and then you put the required data
in each using the rules above.
Grammar (For Experts)
The grammar for Vellum is written for Zapps and is actually incredibly simple considering it
can encode quite an extensive data layout. If you take a look at this:
rule input: ( reference | COMMENT LINE)* ENDMARKER
rule reference: NAME expr
rule expr: atom | reference | structure
rule atom: NUMBER | STRING | SH LINE
rule structure:
LSQB elements? RSQB
| LPAR dictmaker? RPAR
rule elements: ( expr )+
rule dictmaker: ( reference )+
You should be able to figure out how things are structured. Start at the top
with input (that’s the whole file) then trace down through reference,
expr, atom, structure, elements, and dictmaker. You’ll notice
this is very similar to Python’s Grammar file and that’s on purpose. The
original Vellum format was just that, then I stripped out all the elements
which weren’t needed to encode Vellum’s build.
That’s why Vellum’s build format doesn’t need any commas, braces, semicolons, colons,
etc. Just as close to pure data as I could get things.
The interesting thing is that, when a reference is inside a list it’s simply
put in a list form like you’d expect, but when it’s placed inside a dict (using
( … ) format) the reference.name and reference.expression is
converted into a dict’s key=value pair. What this does is give you a simple
consistent way to build lists of anything or dicts of anything or do commands
that look like this:
gen(from "infile.txt" to "outfile.txt")
This looks like a function call, but really it’s just a reference named “gen” that has a dict which in Python
is: {"from": “infile.txt”s, “to”: “outfile.txt”}
Building Our First
With the structure in your head we can build a small vellum build to say “Hello World” in
different ways:
# Say that the hello target is the default. options( default 'hello' ) # Load the vellum commands (mandatory for now) imports [ module(from 'vellum.commands') ] depends( hello [ 'py.hello' 'sh.hello' 'line.hello' ] ) targets( py.hello py "print 'hello world'" sh.hello 'echo hello world' line.hello [ $ echo 'hello world with $' | echo 'hello world with |' > echo 'hello world with >' ] )
This is a really elaborate hello world demo, but it shows you all the main things you need:
- How to set a default target.
- How to import the vellum.commands so you can actually do stuff (this will change and just be a default later).
- How to specify the depends (just a reference with a list of strings).
- What targets can look like, including how they can just be a single command reference, a string, an array of line strings.
- What the different line strings look like with $, >, and | chars. These are handy, and I tend to use $ for shell commands and | for Python code that needs to be multi-line. You can also pass another list of line strings to the sh and py commands to do multi-line targets.
When we run this we get:
BUILDING: ['py.hello', 'sh.hello', 'line.hello', 'hello'] -->: py.hello py: print 'hello world' hello world -->: sh.hello sh: echo hello world hello world -->: line.hello sh: echo 'hello world with $' hello world with $ sh: echo 'hello world with |' hello world with | sh: echo 'hello world with >' hello world with > -->: hello
Which shows you Vellum transitioning into each target, printing what the command is, then actually doing it.
Available Commands
Vellum build specs have a decent set of commands already, and you can write your own (documented further down).
As of version 0.13-rev79 these are the commands and their full documentation which you can get
easily by running vellum -C and you can give just a single command to get that documentation:
given: Evaluates the list of strings or string as a Python expression (not statement) and if that expression is False stops processing this target. Read it as: given X is True continue. Usage: given 'os.path.exists("/etc/passwd")' unless: The inverse of given, this stops processing if the expression is True. Reads as: unless X is False continue. Usage: unless 'not os.path.exists("/etc/passwd")' log: Logs the string to the user. Usage: log "hi there user" py: Runs the given list of strings or string as a python statements. It doesn't stop, but if you raise an exception this will obviously stop the processing. You have access to all the options as globals. Usage: py 'print "hi"' needs: Indicates that before this target should continue, vellum needs to run the targets in the given list. If dependencies for a target don't fit in the main listing then use needs to put them in the target. Usage: needs ['clean', 'build', 'dist'] mkdirs: Takes a dict with "paths" and "mode" and then creates all of those paths each with the given mode. It will also expand user paths on each one so you can use the ~ shortcut. Usage: mkdirs(paths ["path1","path2"] mode 0700) sh: Runs the given list of strings or string as a shell command, aborting if the command exits with 0. Usage: sh 'echo "test"' install: Simple command that installs Vellum's ~/.vellum directory for you. You can also just use: vellum -I. WARNING: This might get replaced with something more like the install command. Usage: install forall: Iterates the commands in a do block over all the files matching a given regex recursively. You can put anything you'd put in a normal target in the do block to be executed, and when it is executed the 'as' variable is set to the full path of each file. This will also be in each task you transition to with the 'needs' command. Usage: forall(files ".*.py$" as "file" do [ ... ]) gen: Used to do simple code generation tasks. It expects a "from" and "to" argument in a dict then it loads from, string interpolates the whole thing against the expr merged with the options, and finally writes the results to "to". Usage: gen("from" somefile.txt "to" outfile.txt also "this")
Writing Your Own Commands
Vellum makes it easy for you to extend and modularize the build in two ways:
- Place alternative or common .vel files in ~/.vellum/ and then import them in the imports stanza.
- Write Python code as functions which become commands and place it in either the regular Python path or ~/.vellum/modules.
If you want to write a Python module to give it commands, you just have to follow two rules:
- The function takes a vellum.Scribe object and the expression that was passed in the build spec for you to process.
- Oddly, you return True if you want processing of the current target to stop. This is so you can just write the function and do nothing to have Vellum keep going (since the default Python return is None).
Once you have that down you’ll just need to look at the following examples of the main commands and read the documentation to vellum.Scribe.
from __future__ import with_statement # Copyright (C) 2008 Zed A. Shaw. Licensed under the terms of the GPLv3. import os import sys import fnmatch import subprocess def sh(scribe, expr): """ Runs the given list of strings or string as a shell command, aborting if the command exits with 0. Usage: sh 'echo "test"' """ formatted = scribe.interpolate("sh", "".join(expr)) scribe.log(" sh: %r" % formatted) if not scribe.option("dry_run"): retcode = subprocess.call(formatted, shell=True, stderr=1, stdout=1) if retcode != 0: scribe.die(expr) def py(scribe, expr): """ Runs the given list of strings or string as a python statements. It doesn't stop, but if you raise an exception this will obviously stop the processing. You have access to all the options as globals. Usage: py 'print "hi"' """ formatted = scribe.interpolate("py", "".join(expr)) scribe.log(" py: %r" % formatted) if not scribe.option("dry_run"): scribe.push_scope({"scribe": scribe, "script": scribe.script}) exec(formatted, globals(), scribe.options) scribe.pop_scope() def log(scribe, expr): """ Logs the string to the user. Usage: log "hi there user" """ scribe.log(" " + scribe.interpolate("log", expr)) def needs(scribe, expr): """ Indicates that before this target should continue, vellum needs to run the targets in the given list. If dependencies for a target don't fit in the main listing then use needs to put them in the target. Usage: needs ['clean', 'build', 'dist'] """ for target in expr: if not scribe.is_target(target): scribe.die(target, "target %s isn't in the targets list" % target) else: scribe.transition(target) def given(scribe, expr, name='given'): """ Evaluates the list of strings or string as a Python expression (not statement) and if that expression is False stops processing this target. Read it as: given X is True continue. Usage: given 'os.path.exists("/etc/passwd")' """ formatted = scribe.interpolate("given", "".join(expr)) scribe.log(" %s: %r" % (name, formatted)) if scribe.option("force"): return False try: return not eval(formatted) except Exception, err: scribe.die(expr, err) def unless(scribe, expr): """ The inverse of given, this stops processing if the expression is True. Reads as: unless X is False continue. Usage: unless 'not os.path.exists("/etc/passwd")' """ if scribe.option("force"): return False return not given(scribe, expr, "unless") def gen(scribe, input=None, output=None, **expr): """ Used to do simple code generation tasks. It expects a "from" and "to" argument in a dict then it loads from, string interpolates the whole thing against the expr merged with the options, and finally writes the results to "to". Usage: gen(input somefile.txt output outfile.txt other "variable") """ scribe.log("gen: input %s output %s" % (input, output) ) if not (input or output): scribe.die("gen", "You must give both input and output for gen.") if scribe.option("dry_run"): return expr.update(scribe.options) with open(input) as inp: with open(output,'w') as out: out.write(inp.read() % expr) def install(scribe, expr): """ Simple command that installs Vellum's ~/.vellum directory for you. You can also just use: vellum -I. WARNING: This might get replaced with something more like the install command. Usage: install """ # relies on mkdirs to honor dry_run mkdirs(scribe, paths=["~/.vellum/modules", "~/.vellum/recipes"], mode=0700) def mkdirs(scribe, paths=[], mode=0700): """ Takes a dict with "paths" and "mode" and then creates all of those paths each with the given mode. It will also expand user paths on each one so you can use the ~ shortcut. Usage: mkdirs(paths ["path1","path2"] mode 0700) """ assert isinstance(paths, list), "mkdirs expects a list as the expression" for dir in (os.path.expanduser(p) for p in paths): scribe.log(" mkdir: %s" % dir) if not (os.path.exists(dir) or scribe.option("dry_run")): os.makedirs(dir, mode=0700)
Getting More Help
Since Vellum is very new, feel free to email me (mentioned in the footer) and I’ll help you out.
Also assume that Vellum has bugs, but that it should work for most tasks. Feel free to post
any bug reports to the Launchpad project
License
Vellum is licensed under the GPLv3 but since it is a build tool that you normally don’t
distribute with your software it shouldn’t impact you. In general the GPL only applies
if you write something that links against Vellum and distribute that something. In fact
you could use a Vellum build spec and build your software with Vellum and keep your software
commercial or use any other license.
This is much the same way that GCC and GNU make works. Just using them does not mean that
your work is instantly GPLed. In more direct terms, the GPLv3 does not apply to any build
specs (.vel) files you write since those are parsed, and does not apply to anything you
build with Vellum (since building is not linking).
Now, if you write modules to load into vellum then those are linked into Vellum. You’ve
got a couple options:
- Since it’s a build system, just don’t release your Vellum build to customers. This is the norm when you distribute software anyway, so it should be fine.
- If you have to give them Vellum and modules you wrote so they can build it, then turn your build module(s) into a GPLv3 project and release then. You don’t have to release all of your softare, just the build modules you wrote. If you’re smart, you’ll make the build modules generic enough that it won’t matter if you release them.
- Vellum is a build tool, so if there’s any proprietary build modules, just convert them to a script and run them as a shell command.
The Distinction
When would you be required to GPLv3 your software? In the above scenarios you are safe to use Vellum for
nearly any task assuming you release your modules or never release your build. When you have to GPLv3
your software is if your software directly imports Vellum in order to operate. Don’t think you can
get around this, since, if your software does an “import vellum.*” it now has to be GPLv3.
The distinction is when Vellum is being used in your software lifecycle. Using it to build means
no GPLv3 (except for your modules). Using it to give your customers functionality (meaning, after the build)
means you have to GPLv3 your stuff.