Vellum: A Simple Python Build Tool

NOTE: I'm working on a medium sized book about Vellum that will serve as
the primary documentation and also try to teach you about the art of
automating things.  You can grab the unfinished PDF
to follow my work and all the source to the book is in the Bazaar repository.

Vellum is my project to create a nice build tool for Python, or really for anything you need to automate in a consistent fashion. Vellum uses a simple format that takes no time to understand, but also lets you easily write Python modules that it can load so you can extend what Vellum does. Vellum tries to be safe whenever it can, and avoids complexity at all costs without losing features.

The primary goal with Vellum is to be simple, safe, and explicit. No more magic chains of things happening that you can’t figure out.

Features

You can write modules in Python and put them in your ~/.vellum/modules directory to get your own commands in Python when Vellum’s spec format isn’t capable enough.
Builds are written in a safe and very simple format that is like “Python minus the crap”.
Build files are not loaded with the Python interpreter and are actually parsed so they will (hopefully) be safer and easier to analyze.
Options, imports, dependencies, and targets are separated into different sections to make the file simpler to debug and write.
Call other targets out of order in those cases where the dependency graph has to be modified.
Vellum has tons of debugging output and prints very good meaningful error messages when it can.
Runs multiple or single lines of shell or Python code in a reliable way according to dependencies.
Output formatted messages to the user.
Test for true conditions to avoid running a target, which is more flexible than having target files that must match mtimes.
A simple code generator system that will let you create templates for generating tasks of particular types.
In addition to importing Python modules to make new commands, you can also import other build files to create master/child or more modular build systems.
Imported build specs are automatically put into a namespace so that names don’t clash, and you can rename the imported namespace to anything you like.
Since commands are functions, they become self-documenting in Vellum so that it’s easy to see what commands a build spec has as well as how to use it.

Download

You can get vellum either by using easy_install, from Launchpad, or through Bazaar.

From Launchpad

You can grab the latest version from the Launchpad project page for the 0.0 series. There should be .tar.gz files, .egg packages, ChangeLogs, and sometimes samples.

With easy_install

Vellum is now registered in the Cheese Shop so you can use easy_install to install it with one command:

$ sudo easy_install zapps pygments idiopidae vellum

Once you do this it should be available. All the .vel files should also be included in the distribution so you can check it out to see some sample. They’re also documented below.

You can also just install zapps and vellum if you don’t plan on building the PDF book but it might be good to get all the gear to help me test it.

Tracking The Source

Your best way to track what I’m doing is to grab the Bazaar repo:

bzr branch http://www.zedshaw.com/repository/vellum/

Let me know if that don’t work for some reason. Feel free to send me patches, and it’s GPLv3 licensed for now.

Getting Started

Vellum takes a build.vel file, loads it, and runs any targets you give it on the command line. Here’s the command line help:

Usage: vellum [options]

Options:
  -h, --help            show this help message and exit
  -f FILENAME, --file=FILENAME
                        Build file to read the build recipe from (no .vel)
  -q, --quiet           Tell Vellum to shut up.
  -d, --dry-run         Dry run, printing what would happen
  -k, --keep-going      Don't stop, build no matter what
  -T, --targets         Display the list of targets and what they depend on
  -F, --force           Force all given conditions true so everything runs
  -D, --dump            Dump the build out to a fully coagulated build.
  -s, --shell           Run the vellum shell prompt.
  -I, --install         Create the ~/.vellum directories.
  -v, --version         Print the version/build number.
  -C, --commands        List all commands and their help. Give a name to see just
                        that one.

As of 0.13-rev79 this is the options, but normally you don’t use these. If your build.vel had a target called build you’d do this:

$ vellum build

You can list as many targets as you like, and if the options stanza (mentioned below) has a default setting then that is run if you don’t give any targets.

Vellum’s Build Spec Format

Vellum uses a simple format for it’s build file, and then modules you can write in Python to implement custom commands. There is an overall philosophy to this design that comes from years of dealing with builds written by other people:

The format goes along with Lisp’s idea that code is data and creates an executable description using simple lists, dicts, and name(value) expressions. You can think of Vellum’s format as Lisp+dicts where every “function” takes only one argument (which can be a list, or dict, or function that …)
The goal of the main build.vel and any recipes is to make something that is easy to analyze (both by a person and by Vellum) without executing any code. This is important for safe software distribution and distributed builds.
In order to provide flexiblity, Vellum let’s you create commands with Python but loads them from trusted places.
Everything about the build format and the command modules is designed to let you create reusable builds. This is why the dependencies are separated from the targets. With build systems that attach the dependencies to the targets it’s too hard to just reuse the targets in other builds. Breaking the dependencies out means that you can pull in whole builds as components and reuse them, even with name spaces. It’s also easier at a glance to see what every task does and what it will call.
Finally, unlike many other tools, Vellum does not force you to put all of the processing logic into an entirely declarative structure. Some tasks are better specifie in an imperative way, so Vellum allows you to specify dependencies at any point in a target’s definition.

Here’s the main Vellum build.vel as your first sample:

# This is a working build spec, but it is also an example
# so it consists of a lot more stuff than you would normally
# find in a real project.

imports [
    recipe(from 'scripts/testing' as 'testing')
    recipe(from 'scripts/dist' as 'dist')
    recipe(from 'scripts/sample' as 'sample')
    recipe(from 'doc/book' as 'book')
]

options(
        project "vellum"
        default 'tests'
        sudo 'sudo'
        version '0.17'
        website '../zedshaw.com/output/projects/vellum'
        bzr.revision '.bzr/branch/last-revision'
        version.file 'vellum/version.py'

        setup(
            name 'vellum'
            version '0.17'
            author 'Zed A. Shaw'
            description 'A flexible small make alternative for Python programmers.'
            author_email 'zedshaw@zedshaw.com'
            url 'http://www.zedshaw.com/projects/vellum'
            packages ['vellum']
            scripts ['bin/vellum']
        )
)

depends(
        build ['tests' 'version.gen' 'dist.install' 'dist.sdist']
        commit ['dist.gen.setup' 'parser' 'dist.clean' 'book.clean']
        tests ['parser' 'testing.run']
        release ['build' 'dist.release' 'book.release' ]
)

targets(
        commit [
            $ bzr log --short > CHANGES
            $ bzr commit
            $ bzr push
        ]

        version.gen [
            py [
                |rev = open("%(bzr.revision)s").read().split()
                |ver = {"version": version, "rev": rev}
                |open("%(version.file)s", 'w').write(
                |    "VERSION=" + repr(ver))
            ]
        ]

        parser "zapps vellum/parser.g"

        dist $ cp doc/manual-final.pdf %(website)s

        cloc [
           $ cloc --report-file=doc/test_cloc.txt --no3 --by-file tests
           $ cloc --report-file=doc/source_cloc.txt --no3 --by-file --force-lang=python,g vellum bin
           $ cat doc/*_cloc.txt
        ]
)

Here you see that you get Python’s usual comment style, then you have a format that consists of what looks like function calls with weird parameters in them and lists.

The file breaks down into a logical structure like this:

imports—Other .vel files to load as “recipes” using import commands or “modules” that have Python definitions of new commands you want.
options—A dict sets various options the rest of the file can use in commands as % replacements.
depends—A simple dict that says which targets (or fake targets) need which other targets first.
targets—A dict with the targets and what to run for each one as commands and shell.

The format is then very simple once you understand a few little grammar rules:

Each element is a reference which consists of a name and an expression. Example: targets(commit “echo test”) is a reference that has the name “targets” and has a dict as the expression (which also contains a reference named “commit” that has an expression “echo test”).
An expression can be anything else, even another reference. You get NUMBERs, STRINGs, lists, dicts (but with parenthesis), and a special thing called a “line string”.
During processing Vellum assumes that a plain string is just a shell command. Since this is so common there’s a special string that works like Python’s comments called a “line string”. Any line that starts with $,>, or | is turned into a string preserving all space. This lets you easily do multi-line Python or shell commands and makes the file look nicer.
The whole file is a dict, which means that order doesn’t matter for the above mentioned stanzas.

This means that you do one reference for each of the above stanzas, and then you put the required data in each using the rules above.

Grammar (For Experts)

The grammar for Vellum is written for Zapps and is actually incredibly simple considering it can encode quite an extensive data layout. If you take a look at this:

rule input: ( reference | COMMENT LINE)* ENDMARKER 
rule reference: NAME expr 
rule expr: atom | reference | structure 
rule atom: NUMBER  | STRING | SH LINE 
rule structure: 
    LSQB elements? RSQB  
    | LPAR dictmaker? RPAR 
rule elements: ( expr )+ 
rule dictmaker: ( reference )+

You should be able to figure out how things are structured. Start at the top with input (that’s the whole file) then trace down through reference, expr, atom, structure, elements, and dictmaker. You’ll notice this is very similar to Python’s Grammar file and that’s on purpose. The original Vellum format was just that, then I stripped out all the elements which weren’t needed to encode Vellum’s build.

That’s why Vellum’s build format doesn’t need any commas, braces, semicolons, colons, etc. Just as close to pure data as I could get things.

The interesting thing is that, when a reference is inside a list it’s simply put in a list form like you’d expect, but when it’s placed inside a dict (using ( ... ) format) the reference.name and reference.expression is converted into a dict’s key=value pair. What this does is give you a simple consistent way to build lists of anything or dicts of anything or do commands that look like this:

gen(from "infile.txt" to "outfile.txt")

This looks like a function call, but really it’s just a reference named “gen” that has a dict which in Python is: {“from”: “infile.txt”s, “to”: “outfile.txt”}

Building Our First

With the structure in your head we can build a small vellum build to say “Hello World” in different ways:

# Say that the hello target is the default.
options(
    default 'hello'
)

# Load the vellum commands (mandatory for now)
imports [
    module(from 'vellum.commands')
]

depends(
        hello [ 'py.hello' 'sh.hello' 'line.hello' ]
)

targets(
        py.hello py "print 'hello world'"

        sh.hello 'echo hello world'

        line.hello [
            $ echo 'hello world with $'
            | echo 'hello world with |'
            > echo 'hello world with >'
        ]
)

This is a really elaborate hello world demo, but it shows you all the main things you need:

How to set a default target.
How to import the vellum.commands so you can actually do stuff (this will change and just be a default later).
How to specify the depends (just a reference with a list of strings).
What targets can look like, including how they can just be a single command reference, a string, an array of line strings.
What the different line strings look like with $, >, and | chars. These are handy, and I tend to use $ for shell commands and | for Python code that needs to be multi-line. You can also pass another list of line strings to the sh and py commands to do multi-line targets.

When we run this we get:

BUILDING: ['py.hello', 'sh.hello', 'line.hello', 'hello']
-->: py.hello
py: print 'hello world'
hello world
-->: sh.hello
sh: echo hello world
hello world
-->: line.hello
sh: echo 'hello world with $'
hello world with $
sh: echo 'hello world with |'
hello world with |
sh: echo 'hello world with >'
hello world with >
-->: hello

Which shows you Vellum transitioning into each target, printing what the command is, then actually doing it.

Available Commands

Vellum build specs have a decent set of commands already, and you can write your own (documented further down).

As of version 0.13-rev79 these are the commands and their full documentation which you can get easily by running vellum -C and you can give just a single command to get that documentation:

given:
    Evaluates the list of strings or string as a Python
    expression (not statement) and if that expression is
    False stops processing this target.  Read it as:
      given X is True continue.

    Usage: given 'os.path.exists("/etc/passwd")'
    
unless:
    The inverse of given, this stops processing
    if the expression is True.  Reads as:
      unless X is False continue.

    Usage: unless 'not os.path.exists("/etc/passwd")'
    
log:
    Logs the string to the user.

    Usage: log "hi there user"
    
py:
    Runs the given list of strings or string as a python
    statements.  It doesn't stop, but if you raise an
    exception this will obviously stop the processing.
    You have access to all the options as globals.

    Usage: py 'print "hi"'
    
needs:
    Indicates that before this target should continue,
    vellum needs to run the targets in the given list.
    If dependencies for a target don't fit in the main
    listing then use needs to put them in the target.

    Usage: needs ['clean', 'build', 'dist']
    
mkdirs:
    Takes a dict with "paths" and "mode" and then creates all of those
    paths each with the given mode.  It will also expand user paths
    on each one so you can use the ~ shortcut.

    Usage:  mkdirs(paths ["path1","path2"] mode 0700)
    
sh:
    Runs the given list of strings or string as a shell
    command, aborting if the command exits with 0.

    Usage: sh 'echo "test"'
    
install:
    Simple command that installs Vellum's ~/.vellum 
    directory for you.  You can also just use: vellum -I.
    WARNING: This might get replaced with something more
    like the install command.

    Usage: install
    
forall:
    Iterates the commands in a do block over all the files
    matching a given regex recursively.  You can put anything
    you'd put in a normal target in the do block to be executed,
    and when it is executed the 'as' variable is set to the
    full path of each file.  This will also be in each task
    you transition to with the 'needs' command.

    Usage: forall(files ".*.py$" as "file" do [ ... ])
    
gen:
    Used to do simple code generation tasks.  It
    expects a "from" and "to" argument in a dict
    then it loads from, string interpolates the
    whole thing against the expr merged with the
    options, and finally writes the results to "to".

    Usage:  gen("from" somefile.txt "to" outfile.txt also "this")

Writing Your Own Commands

Vellum makes it easy for you to extend and modularize the build in two ways:

Place alternative or common .vel files in ~/.vellum/ and then import them in the imports stanza.
Write Python code as functions which become commands and place it in either the regular Python path or ~/.vellum/modules.

If you want to write a Python module to give it commands, you just have to follow two rules:

The function takes a vellum.Scribe object and the expression that was passed in the build spec for you to process.
Oddly, you return True if you want processing of the current target to stop. This is so you can just write the function and do nothing to have Vellum keep going (since the default Python return is None).

Once you have that down you’ll just need to look at the following examples of the main commands and read the documentation to vellum.Scribe.

from __future__ import with_statement
# Copyright (C) 2008 Zed A. Shaw.  Licensed under the terms of the GPLv3.

import os
import sys
import fnmatch
import subprocess

def sh(scribe, expr):
    """
    Runs the given list of strings or string as a shell
    command, aborting if the command exits with 0.

    Usage: sh 'echo "test"'
    """
    formatted = scribe.interpolate("sh", "".join(expr))
    scribe.log(" sh: %r" % formatted)
    if not scribe.option("dry_run"):
        retcode = subprocess.call(formatted, 
                shell=True, stderr=1, stdout=1)
                
        if retcode != 0: scribe.die(expr)

def py(scribe, expr):
    """
    Runs the given list of strings or string as a python
    statements.  It doesn't stop, but if you raise an
    exception this will obviously stop the processing.
    You have access to all the options as globals.

    Usage: py 'print "hi"'
    """
    formatted = scribe.interpolate("py", "".join(expr))
    scribe.log(" py: %r" % formatted)
    if not scribe.option("dry_run"):
        scribe.push_scope({"scribe": scribe, "script": scribe.script})
        exec(formatted, globals(), scribe.options)
        scribe.pop_scope()

def log(scribe, expr):
    """
    Logs the string to the user.

    Usage: log "hi there user"
    """
    scribe.log(" " + scribe.interpolate("log", expr))

def needs(scribe, expr):
    """
    Indicates that before this target should continue,
    vellum needs to run the targets in the given list.
    If dependencies for a target don't fit in the main
    listing then use needs to put them in the target.

    Usage: needs ['clean', 'build', 'dist']
    """
    for target in expr:
        if not scribe.is_target(target):
            scribe.die(target, "target %s isn't in the targets list" % target)
        else:
            scribe.transition(target)

def given(scribe, expr, name='given'):
    """
    Evaluates the list of strings or string as a Python
    expression (not statement) and if that expression is
    False stops processing this target.  Read it as:
      given X is True continue.

    Usage: given 'os.path.exists("/etc/passwd")'
    """
    formatted = scribe.interpolate("given", "".join(expr))
    scribe.log(" %s: %r" % (name, formatted))

    if scribe.option("force"): return False
    try:
        return not eval(formatted)
    except Exception, err:
        scribe.die(expr, err)

def unless(scribe, expr):
    """
    The inverse of given, this stops processing
    if the expression is True.  Reads as:
      unless X is False continue.

    Usage: unless 'not os.path.exists("/etc/passwd")'
    """
    if scribe.option("force"): return False
    return not given(scribe, expr, "unless")

def gen(scribe, input=None, output=None, **expr):
    """
    Used to do simple code generation tasks.  It
    expects a "from" and "to" argument in a dict
    then it loads from, string interpolates the
    whole thing against the expr merged with the
    options, and finally writes the results to "to".

    Usage:  gen(input somefile.txt output outfile.txt other "variable")
    """
    scribe.log("gen: input %s output %s" % (input, output) )
    if not (input or output): 
        scribe.die("gen", "You must give both input and output for gen.")

    if scribe.option("dry_run"): return
    expr.update(scribe.options)
    with open(input) as inp:
        with open(output,'w') as out:
            out.write(inp.read() % expr)

def install(scribe, expr):
    """
    Simple command that installs Vellum's ~/.vellum 
    directory for you.  You can also just use: vellum -I.
    WARNING: This might get replaced with something more
    like the install command.

    Usage: install
    """
    # relies on mkdirs to honor dry_run
    mkdirs(scribe, 
           paths=["~/.vellum/modules", "~/.vellum/recipes"], 
           mode=0700)

def mkdirs(scribe, paths=[], mode=0700):
    """
    Takes a dict with "paths" and "mode" and then creates all of those
    paths each with the given mode.  It will also expand user paths
    on each one so you can use the ~ shortcut.

    Usage:  mkdirs(paths ["path1","path2"] mode 0700)
    """
    assert isinstance(paths, list), "mkdirs expects a list as the expression"

    for dir in (os.path.expanduser(p) for p in paths):
        scribe.log(" mkdir: %s" % dir)
        if not (os.path.exists(dir) or scribe.option("dry_run")):
            os.makedirs(dir, mode=0700)

Getting More Help

Since Vellum is very new, feel free to email me (mentioned in the footer) and I’ll help you out.

Also assume that Vellum has bugs, but that it should work for most tasks. Feel free to post any bug reports to the Launchpad project

License

Vellum is licensed under the GPLv3 but since it is a build tool that you normally don’t distribute with your software it shouldn’t impact you. In general the GPL only applies if you write something that links against Vellum and distribute that something. In fact you could use a Vellum build spec and build your software with Vellum and keep your software commercial or use any other license.

This is much the same way that GCC and GNU make works. Just using them does not mean that your work is instantly GPLed. In more direct terms, the GPLv3 does not apply to any build specs (.vel) files you write since those are parsed, and does not apply to anything you build with Vellum (since building is not linking).

Now, if you write modules to load into vellum then those are linked into Vellum. You’ve got a couple options:

Since it’s a build system, just don’t release your Vellum build to customers. This is the norm when you distribute software anyway, so it should be fine.
If you have to give them Vellum and modules you wrote so they can build it, then turn your build module(s) into a GPLv3 project and release then. You don’t have to release all of your softare, just the build modules you wrote. If you’re smart, you’ll make the build modules generic enough that it won’t matter if you release them.
Vellum is a build tool, so if there’s any proprietary build modules, just convert them to a script and run them as a shell command.

The Distinction

When would you be required to GPLv3 your software? In the above scenarios you are safe to use Vellum for nearly any task assuming you release your modules or never release your build. When you have to GPLv3 your software is if your software directly imports Vellum in order to operate. Don’t think you can get around this, since, if your software does an “import vellum.*” it now has to be GPLv3.

The distinction is when Vellum is being used in your software lifecycle. Using it to build means no GPLv3 (except for your modules). Using it to give your customers functionality (meaning, after the build) means you have to GPLv3 your stuff.