The Full Feed
2008-07-20
RubyFringe was my last Ruby conference and it was the best conference to go out on. Everything about RubyFringe was great. It was well organized, contained eclectic talks, and supported the weirdness that’s usually hidden at the other conferences.
I performed four songs as my goodbye tribute to all the suckage and awesomeness I found in Ruby. They were a big hit, and hopefully pissed off all the right people in all the right ways. Read my full recap and you can hear the songs, download the tracks, and remix them under a CC license.
That’s right, I felt my performance was so incredibly insane that I’m pulling a NIN and letting you, the reader, fuck up my talent in anyway you can. I want to be the Star Wars Kid of Ruby haters as my last conference present to the Interwebs.
2008-06-25
I’ve been completely fed up with news/feed/rss/atom readers these days. I use Linux as my primary operating system, and I only have a few feeds that I want to rip through quick so I can get to reading the content. Yet, trying to find a reader that doesn’t suck donkey balls has been a chore.
After a long search through the world of open source feed readers, I finally found a great one named Newsbeuter that is what I want: “The Mutt of Feed Readers”. Apart from some build and dependency issues, it’s been great. All the features I need, none that I don’t, fast console operation, and it’s cut down on my information overload quite well.
However, in the process of trying to find a new reader I wrote up my experiences as a ranting product review of open source readers covering nearly all of them and just generally why most suck.
2008-06-21
Wanna know what all the Ruby vulnerabilities are? Or at least have a fun look at how to search through code for clues? It’s a blast.
I took the time to go through all the changes between Ruby 1.8.6 p114 and Ruby 1.8.6 p230 and analyzed the changes so that you can know what they are and how the defects are security defects.
My history with the Ruby guys hiding their shame in patches and never admitting that they fucked up important parts of Ruby goes pretty far back. I found some pretty bad memory leaks with Mongrel and during that time Ruby idiots would claim it was the, “OS holding the memory, not a leak.” Riiight. We now know much different, but I’ve never seen a public admission that things like the GC, IO, or Threads are completely fucked up. Well, apart from people who have to deal with them daily.
This list of vulnerabilities and the code is just what I did after working all day on my book and while bored. Ho hum. So hard. Have fun with them!
BTW, nobody in Ruby told me about these. I completely inferred them from just analyzing the source for common defects. And while I pick on the person who made the changes a bit, I’m not blaming him or claiming he’s complicit in anything. He’s probably just doing what he thinks needs to be done.
UPDATE: I added a little update to the post to clarify that I’m doing it through diffs and shit on purpose to show how it’s not that hard, even without the source repository. Read that section for the update.
2008-06-13
I’m dropping a large blog post on everyone to just say that I haven’t died, I’ve just been busy working on my book for A/W about Mongrel. I had contracted with them to do a book about deploying Mongrel, but then decided it wouldn’t be a very good book since we’d already done one about that topic and there wasn’t too much more to say.
Well the book is now about how I wrote Mongrel, kind of teaching you about writing a network server in Ruby and C but doing it by teaching you about Mongrel. The first part of the book is going slow, so I’ve had to hide out and not answer much email. It’s still murder writing a book like this though because I have to write it fast to meet a deadline. The book will cover things like writing parsers, efficient network code, HTTP the protocol, Ruby’s sockets, Event based programming, and even how Mongrel’s parser is used in other languages and servers.
You can think of it as “Zed’s Last Book of Ruby Tricks”. After that I’ll be working on my “Protocols and Performance” book which will be a more expanded and serious version of the same topic, not focused on Ruby.
Conferences
At the same time however I need to go to at least two more conferences. One I just finished was Ruby En Rails 2008 which was a blast to attend. Amsterdam is a fantastic city, there was tons of great people and man do they have a bunch of software shops popping up there. I think they have the hottest Rails market in Europe based on my limited experience. I’m glad they invited me too.
You can check out my presentation which was funny, but according to Obie very pointless.
I also met the Phusion Passenger guys and holy fucking crap are they on to something. If anyone is going to actually take on Mongrel in the hosting area it’s Passenger. The developers are super cool nice guys (unlike me) and even DHH likes their stuff. He really never liked anything I built, so hopefully those guys get more support. About the only thing keeping them from taking over is that they use forking so a few libraries that keep resources open will have serious problems. They’ll probably have to think up some kind of thing for that soon, but I think most Rails deployments could get pretty far with Passenger.
UPDATE: Hongli says: “Actually this has already been solved. 2 months ago, to be exact – before Phusion Passenger was released. :) Some libraries keep resources open, yes, but the only things that we have to worry about are open file descriptors. We used to close file descriptors upon forking, based on a black list of file descriptors to close, but we quickly found out that that doesn’t work very well and easily resulted in file descriptor leaks. So now we close all file descriptors, except those that are specified in a white list (i.e. stdin, stdout, stderr and a few others). Everything has been working flawlessly since then.”
For years everyone has tried to “beat mongrel” but just couldn’t pull it off. I personally wouldn’t care much since developers should use the best tool out there, but man the people trying to unseat Mongrel just suck at it. My favorite was how Charles declared Mongrel dead in 2007. I grabbed it and tried to upload an image when it crashed with a out-of-memory error. It took them like nine months to fix that shit. Here’s a clue, before you try to compete with someone who makes no money on their software and actually doesn’t work on it very much, at a bare minimum try to get the fucking protocol to allow an upload. Things might work better for you then.
Honestly though, it shouldn’t be that hard to beat Mongrel since Mongrel is crippled by Ruby. What the Phusion guys are pulling off is just using Apache to do the heavy lifting and then let their module do the work to stream out to Rails. It’s not a new idea, they’re just doing a great job marketing it and educating people while keeping things simple. The kicker is that they also have support for Rack and WSGI. Now that’s fucking sexy.
What is very new about Passenger is that even though it does forking, Hongli managed to clean up much of Ruby’s memory management so that each fork is small and clean. This should greatly reduce your memory footprint, I think an estimated 15-30%.
Announcing EaRing
There was also working software which came out of my talk that you might like to try. It’s called EaRing, and it’s the perfect accessory to every Ruby company. EaRing is a weirdo assembly language that’s almost like a scripting language. You hand it a sane assembler script and it compiles it on the fly and runs it. It’s fast and very dynamic and probably won’t work on any computer you have. Still, you can grab it from the EaRing project page and you can grab the source with bzr branch http://zedshaw.com/repository/earing/. I wrote it quickly in my spare time during the two weeks before the conference, so it’s not the greatest code ever.
My Last Ruby Conference
The next conference, and sadly my last Ruby conference will be Ruby Fringe in Toronto. This conference was apparently made in my honor/image and should prove to be pretty damn great, if it doesn’t get all the organizers and speakers passively aggressively killed by Chad Fowler.
Why is RubyFringe my last Ruby conference? Well my friends, I honestly don’t know why Ruby conferences keep inviting me. I ripped on tons of people and basically called everyone a bunch of ghetto fab whores. Of course I only meant the small percentage of people and the companies that are fucking up Ruby, but I said it and then walked away to go work on something else for a while. I figured that’d be it and then I could get back to more fun things.
Instead, I apparently said what a lot of programmers stuck at the bottom wanted to say but didn’t have the courage. I think now they look at me as the guy who can possibly call bullshit on all the goings on in the Ruby world, but honestly I’m not really that powerful. I basically have a huge nasty mouth and very little fear of reprisal. There’s nothing I can do to change the way things work, all I can do is write about how I feel about the industry and hope that some of the ideas sink in to the next batch of programmers.
The number of times I hear “Steak and Strippers” at conferences tells me I’m doing a good job at that through my blog.
If people want things to change, don’t invite me to a conference, invite other people who care about your language and have something new to say. One of the reasons I’m a draw at conferences is I never recycle a presentation and I try to write software for every single one. Whether the software is the presentation itself in Factor, Python build tools to make the presentation, or some nifty hack I wrote, it doesn’t matter. I always have new software to present on or with. If you want to keep things solid and real at conferences, make this demand of your speakers. They should not recycle talks, and they should present some kind of software.
I’m still going to write about how retarded the whole software industry is, and I’ll attend conferences that have nothing to do with Ruby. I’ll be at DefCon and another conference in Poland, and probably a few others, but none that have to do with Ruby. I’ll definitely be focusing on the dicks who think they’re at the top of the Ruby community and the companies trying to make a buck off the Ruby Proletariat.
But, I can’t justify going to Ruby conferences anymore. Sorry.
My Work At The Bear Is Done
Some people were wondering what happened at Bear Stearns. Well, Bear is no more. They were bought out in a single weekend by JPMorgan Chase for about 2 billion dollars. The building that Bear owns is worth 1.5 billion. Bear had been around for about 80 years, survived the Great Depression, World War II, Vietnam, and several market crashes, and was worth 400 billion around that time. To have them evaporate in a weekend is fucking scary as hell.
The dumbest thing is that I held out and waited around in NYC to go work at Bear. I had thought, man these fucking start-ups can’t pay me and flame out like crazy. I think I’ll go hide out in banking since they never go under.
Wow, that worked out great. It’s not really something you can predict, but the first day I went to work at Bear they announced their problems with subprime mortgages. The CEO lied to everyone and claimed it was only 4% of their business, but the truth is “much greater” as we found out later. I’m sure we’ll never know how much the idiots running Bear actually sunk into subprime mortgages, but I’m thinking it was probably closer to 50% of the company or more.
The various start-ups still aren’t that much better, and companies like Microsoft and Google aren’t very appealing. So I’m kind of waiting around and just focusing on writing and music now.
Now, what I got out of the deal was a super sweet severance package that sets me up for a short time, assuming I can keep my costs low. The deal to buy Bear closed on the 2nd of June, and the second I’m allowed I’ll spill more details about what everyone got and how it affected me.
If you get a job working at any company related to the financial industry, demand twice the market rate for your salary. Having a company worth close to 400 billion dollars die in a weekend is amazing. There is definitely some bad shit going to happen to our economy and I wouldn’t set foot in a finance institution to set the place on fire.
Rumor Control
UPDATE: Well that was fast. I originally said Jay Fields was spreading this rumor, but Jay says he’s never said this, although he did say it in several jabs at me at the RuPy2008 conference. I’ve edited to remove Jay and just replaced it with “Random ThoughtWorks Asshole” or RTWA. I’ll give him the benefit of the doubt and assume it’s just a random ThoughtWorker.
Apparently RTWA has been running his/her fat mouth so I think I’ll include a little bit of rumor control. Although it is kind of funny so I let it slide for a while, but now it’s just dumb.
RTWA’s been telling people that on the fateful Bear Stearns Ruby on Rails project (which ThoughtWorks fucked up) I wrote a text editor and made everyone use it.
Alright, now think about this for a second. First off, I am not that good. And, if I did write a text editor in a day (the part he keeps leaving off) do you really think it’d be good enough to make people use it? Also, if you know me I’m dead against making programmers use anything they don’t like. Why would I make them use a text editor I wrote if they didn’t like it? Shit I just wrote an assembler for fun and did a whole presentation on it to a packed crowd and I still don’t think it’s good enough. Right, I’m gonna make people use a fucking editor I wrote.
Here’s the real story. The ThoughtWorkers were insisting on using fucking Eclipse for the IDE and forcing everyone to use it. Eclipse is a nasty fucking piece of software, but it also meant that nobody learned how a real Rails application is laid out because Eclipse’s only real feature is a hot key to find files. After close to a month of watching programmers fight with Eclipse’s shitty text editor I said we have to find something new. Let’s do a shoot-out.
We tried every fucking editor there was. We even had a grid on the wall with all the features we wanted, and none of them met our goals. We basically wanted Textmate, but we were stuck on Linux so we were screwed. Now keep in mind that myself and another programmer were happily using Vim and Emacs when nobody was around or cared, and we’d crank out shitloads of great code with them. I’m a Vim whore so if I was going to force anyone to use an editor it’d be Vim. More on that later.
After trying every IDE, me and another developer decided to try every editor we could find. We went through about 20, and then I remembered J from way back in the day. J is this weird editor that is written in Java, but uses a Common Lisp system the author of J wrote to actually write the editor. Now, the funny thing is J is pretty small, but had all the features everyone wanted.
Problem was that J was bitrotted and needed a facelift. So, I grabbed the Substance Swing look and feel, and hacked it into the source in about an hour. I then fixed a few annoying bugs, and asked people to try it. Anyone who liked it just went and used it, everyone else suffered with Eclipse. I even created a Barbie Pink theme for one of the female developers to annoy her male arch nemesis. It was kind of fun.
Now, as we’d hit bugs I’d do a fix here and there, but really I spent maybe a day total working revising an editor that other programmers willingly used. Later, after everyone had been weened from their Eclipse addiction, people asked me to teach them Vim. Yes, that’s right, they asked me to teach them Vim. I actually didn’t want to because I wasn’t sure how it’d work for them, but they all liked it except for a few people who still used J or Emacs. I gave them one class in Vim and they were capable in a few hours. I am straight up not lying about that, and I’m still kind of weirded out by it, but that’s how it went down.
We ended up with some people using Vim, some using J, some using Emacs, and then a couple of remote developers who were forced to do lame Citrix were forced to use a terminal window through SSH. They tried Vim for a while, and I think they eventually used Nano before just giving up and hacking on some C# code for the project. Eclipse is horrible over Citrix.
Here’s the difference between my style and the ThoughtWorks style. ThoughtWorkers like RTWA hear this and think I’m crazy. “You wrote your own editor! OMG what about the bus factor?!” What they fail to see is that everyone got to experience a wide variety of editors and all learned something new, eventually deciding they liked Vim. By the end of the project everyone in the room could easily switch between editors and pairing wasn’t a problem at all. In my style, programmer choice leads to stronger programmers because they are exposed to new ideas and ways of doing things.
In the RTWA way, they are treated like factory workers or even just lowly assembly line widgets that all have to be the same so ThoughtWorks can rotate out newly minted trainees and train on the client’s dime. Keep that in mind when they want to “standardize” on a particular programmer tool. What you’re doing is removing any creativity and happiness a programmer derives from using her favorite tools.
The irony is, RTWA, one of the strongest advocates for everyone using the same shitty Eclipse editor, was walking around a conference in Poland insulting me claiming I forced people to use an editor I wrote. Use your brain when you hear things like that. I don’t have any reason to lie to people since I don’t make any money anyway. Always remember that I advocate programmers being allowed to use their favorite tools, and exposing them to new stuff they might not know about.
Another funny thing about the Vim experience: I also use Vimperator and the Awesome Window Manager among other fun tools. I never would force this environment on other people as it is very particular to how I work, but they would sit with me and watch me code, then ask me what I was using. I’d tell them and they’d go off and install it and then use it themselves. I’d give them quick training in it, and other people would go find similar tools that fit their needs.
Good tools don’t have to be forced on people.
And Remember, I’m An Asshole
Next, I think he’ll try to tell you how I ripped into one of their consultants, but I’m sure he’ll leave off how all of the ThoughtWorkers would constantly bully my employee because my guy was making them look bad. I sure as hell ripped into that turd fucking TWer when he started to rant at my guy yet again claiming the shitpile they wrote was “just like the 12 rails code bases I have on my laptop”. They would constantly lie about how their code was just like any other Rails project, which was total unmitigated bullshit.
Him and his TW buddies would constantly pick on my guy (who was 10 times the developer they ever could be) mostly because he’d speak out against their idiotic design decisions. Remember when you’re in charge that you first try to calmly get people to quit doing subversive social bullshit, but eventually you have to shut it down any way you can. That kind of psychological torture can drive some people to hurt themselves and can get you sued if you let it continue.
What amazed me is everyone was outraged at me telling this guy to fuck off with his “+1 Reputation Healing Laptop-of-12-code-bases” and shut his fucking mouth, but nobody ever jumped on him and his TW buddies for putting up signs insulting my guy. Yes, signs, only about a single employee that worked at Bear Stearns, their client. Nobody who thinks that’s the correct professional behavior for a consultant deserves to be listened to. It is evil, subversive, ruthless bullshit for no reason.
Here’s the truth, my guy was a very nice man who had some personal problems like we all do at times. It isn’t polite for me to go into what he was dealing with, but having a bunch of scrub coders try to tear him down daily while dealing with all the other bullshit in the company sure as hell wasn’t helping him. I tried my best to help him out, but it was just a fucked situation all around, and I wasn’t dealing with things very well either. About the only thing I could do was keep the TWers from hurting him further with their psychological warfare bullshit.
This is the thing they don’t tell you in business school. These dumb fuck sales tactics and shit like NLP hurt the decent and truly fragile people. The idea that all members of a team must be carbon clones removes their identities and forces the individuals who are intelligent out of a job for no real benefit to the software process. If my guy had been forced off the team or out of the job I’m not sure what would have happened to him. He was a very nice man and needed to leave on his own power, not forced out. I considered it my job to keep everyone else from pushing him out, even though I knew he wouldn’t stay very long.
He’s doing great now at a hot start-up making money, but I still feel very bad about how he was treated. Hopefully me telling people about it will make sure that nobody else gets treated poorly by those frat boys pretending to be programmers.
Finally, I Got A Sweet Guitar
I’ve been getting back into practicing my guitar and learning more about music. I kind of stopped for a while, but recently I went out and bought a very nice Schecter with a white body. At first I didn’t know it was marketed as a great Metal guitar. I mostly play Jazz and Blues, but when I played it I loved the way it felt.
What’s even better about this thing is that I got it for $500 off because the idiots at Guitar Center didn’t know that you have to replace the 9volt battery in it or the active EMG pick-ups don’t work right. I was playing it, and then did some research, and after I bought it I replaced the battery and it was a totally different guitar. Dumbfucks.
Of course I had to tinker with it and fucked up the tuning something fierce. I’ve never had a guitar with a floating bridge before, so when I went to tune it I knocked it all out of alignment. Took me forever to get that thing back, but now it sounds like it did when I bought it. Very crisp and easy to play.
I also bought this incredibly fun Ukulele that is electric/acoustic and has six strings. The two extra strings are sympathetic strings, and the jack for the Uke uses sensors to detect the string vibrations translating them into sound. What I do is I run it through my POD X3 or various other sound modulation systems to make some evil sounding things nobody should listen to sober.
Going Dark For A While
I have like three rants all piled up in my WIP directory ready to go, but I want to let them simmer and work on them when I have more space in my head. Right now I’ve got to focus on the book and then after that I’ll be publishing more stuff.
Feel free to email me if you have questions or are using the software on my site. I use most of the stuff here daily so if you track the Bazaar repository for each one then you can pull down any changes I make. I won’t announce them on the blog though, just track them if you’re interested.
See ya in a few months.
2008-04-29
UPDATE: There seems to be some interest in Idiopidae too and I’ve been neglecting the Idiopidae project page while working on Vellum. I’ve updated that page with information and updated install procedures, so please check it out again if nothing worked for you.
There’s a Vellum bug fix release 0.16 out you all can grab if you’re using it. Due to a slight change in the forall command where I stopped using raw regex and used the fnmatch instead, the unit tests for that command weren’t actually running. This meant that bugs found their way in and now the latest version of Vellum 0.16 should fix all of that. It fixes that and just adds a bunch of documentation really.
Speaking of documentation, I’m still writing the big Book of Vellum and now have the first draft of Appendix A where I do a complete walk through the Vellum code. You’ll notice that it matches the code I just changed two minutes ago in the forall command. This is the advantage of Idiopidae. Simply changing the code and re-running the book build gets me everything instantly updated.
Grab the updated copy (still without much editing yet) and read through if you haven’t already. I’ve also built this one with Emacs colors for all my friends from Reddit and YCombinator News since I know all you Lisp whores can’t stay away from your editor designed for a man with 8.5 fingers on each hand.
I’ll be rotating the color scheme each time I release vellum just to have fun with it.
You can install this latest drop of Vellum in all the usual ways, and you can also grab from Bazaar:
- sudo easy_install zapps pygments idiopidae vellum
- bzr branch http://zedshaw.com/repository/vellum/
- Grab From Launchpad
- Grab From Here
LGPL?
There’s also been a few good submissions from some folks with suggestions for typesetting improvements, Pygments patches and improvements, Vellum bug reports, and the PIDA project wants to use Vellum as their build specification system. They’ve asked if I’d change the license to LGPL so I thought I’d toss the question out and see how people feel about the licenses.
My main motivation for going with GPLv3 is to prevent the common situation where corporations take the work, modify it, and don’t give it back. I’m also pretty down on corporations in general these days, so I may even consider a dual license of BSD for non-profits and GPLv3 for corporations.
Any thoughts on the subject of licenses for Vellum are welcome.
Bazaar Problems
If you just updated your Bazaar and you use Paramiko, you’ll find that Bazaar blows up because the Paramiko folks removed a function (in a fucking minor version release at at that). The easiest fix for me was to simply remove the offending package and then do:
sudo easy_install paramiko==1.7.2You can review the bug report at their launchpad site for more information.
2008-04-26
UPDATE: I forgot to mention that this is my first LaTeX text ever and that it’s based on one of my first Python projects and that I wrote what you find below in about 3 days while dorking around on Vellum and Idiopidae. Since I’m both a Python and LaTeX newb please feel free to school me. Better yet, if you think the typesetting sucks, then show me your samples. See if you can beat this one sent to me by Kashif Rasul. I still consider what he’s created as the bar to get over.
I think I’ve gone insane recently because I’ve been obsessed with making the documentation for Vellum so great angels will cry tears of gold. I’m treating it as a practice book for the big book I’m writing for A/W by writing a fully LaTeX typeset PDF about Vellum using all the tools I’ve created for more modern literate programming.
The big “feature” of the book is insanely well typeset code with color. The typesetting is so good that everyone I’ve showed it to seems shocked by it. I’m not sure if that’s because they think it’s ridiculous to document such a small project that well, or because it’s way better than many other books. I’ll let you people tell me what you think too.
I tried to make it fit what many people use for their editor looks, but without making the book hard to read. This turned out to be easy as hell with Idiopidae and Pygments and in the process I learned quite a lot about TeX and LaTeX.
The end result is good so far, but I’m looking to improve it even further with:
- I had to avoid floats because they would shuffle the code around making it hard to talk about it. Instead I just inject the code right there and fake out a float. What I want is to have code which is broken between pages cleaned up so that it either has a “blah continued…” marker and/or just jumps to the next page. That’s probably going to be hard as hell so it might have to be a manual decision after the whole thing is written and positioned.
- I’d like smaller snippets to exist in a side float or in a two-column page so you can read the code and the discussion of it side-by-side. I really good example of this is the discussion of the grammar for Vellum in “Appendix A: Parser”.
- Play more with the boxes and outlines for the code sections to see how their status in the prose can be made more or less significant.
- Figure out how to use Bazaar to sequentially version code and then get Idiopidae to understand the version to use in a discussion. This would probably fall flat with source code since that just changes, but it would be handy for some examples.
- Fix up the Pygments line numbering. Right now they’re kind of annoying and interfere with the reader’s ability to cut-paste the code.
The Vellum book is a work in progress, but check out how it looks already in PDF form. Definitely look at Appendix A to see how well the source highlight works using Idiopidae to snarf in live code from the Vellum source.
Vellum and Idiopidae have made creating the book for Vellum incredibly easy. I now don’t worry about how the code is included or formatted, and I can access it using logical sections. I just tell Idiopidae the file and section and it makes all the moves. A bit of LaTeX glue to wrap it in something pretty and I’m off.
I can also do some really interesting things with sample outputs and incrementally built source files in the book. When you write a book about code you typically want to start with a simple example of some source file, and then build it up to more complex levels as the prose progresses. Traditionally this meant generating a series of individual files each with the slight modifications. Problem is this means maintaining all those little files for just one chapter.
With Idiopidae I can actually put multiple files into one larger file, and put the code snippets for the progressive discussion into different sections. I hadn’t thought of this until I did it yesterday, so I may try to formalize this in Idiopidae sometime this weekend.
There are a few things I want to add to Idiopidae which will probably involve a rewrite. If you use it, don’t get too attached to the syntax. What I’m thinking is that I’d like a feature to run commands, clean the output up, and then inject that into the book. I’d also like Idiopidae to keep track of md5sums for all the sections it includes. If the md5sum of a section changes then it’ll report an error assuming that you have to change the text around that section to match the code. After you change it, there’ll be a command line option to tell Idiopidae to update the md5sums.
I think this “md5sum tracking” feature could really help keep books about code in sync with the code. The assumption is, if the code changes, then the prose probably has to change as well.
Another feature I want in Idiopidae is to use the Pygments parser of ctags on the files so that it can find logical sections based on the already structure source. If all you need to do is include a series of functions, then having to sprinkle the source with export comments that match the function names is stupid.
As I think about these features though, what I find is a lot of overlap between how Idiopidae works at building books and how Vellum automates building anything. It might be worth looking at Idiopidae reusing Vellum’s parser and build engine so that you can write a nice little description of all the pieces that come together to make a book. It would be a higher level description of the book and designed to also embed Vellum build tasks inside import statements so that you can run code and have the output injected.
Check out how the book looks so far (grammar errors and spelling included) and let me know what you think. My plan is to wrap all these little tools up into one nice tool that helps programmers write better documentation, or at least helps them find someone who can write it without much fuss.
Knuth On Literate Programming
The ideas in Idiopidae are nothing new, as Donald Knuth has been actively promoting Literate Programming for over two decades. The main problem with LP though is its insistence on combining the code and the prose into a single file. There’s serious technical and social issues with this design choice, but in this very good interview with Knuth he mentions a quote that summarizes things quite well:
Jon Bentley probably hit the nail on the head when he once was asked why literate programming hasn’t taken the whole world by storm. He observed that a small percentage of the world’s population is good at programming, and a small percentage is good at writing; apparently I am asking everybody to be in both subsets.
However, this doesn’t cover the absolute hardest technical problem with combining code and prose in the same source file: syntax escape overload.
When you use most LP systems you’re taking the source or two whole languages with completely different lexical and semantic meaning, and then trying to cram them together. This means juggling in your head two different syntax structures and how to merge them together with escape sequences. The technical difficulty of this probably borders on an grammar that would rival Perl or Ruby’s.
This is a major technical hurdle for both people implementing such systems and for the users. As an implementer, I have to create a parser or some hack that can accurately handle both languages. I also have to figure out how the escape sequences are given to the users, how they can encode elements of one language in the other (say for giving constants a bold font). This is a huge task, so no wonder why all the literate tools out there also do a shitty job at finally typesetting the resulting code.
For the users they have to use their brain in ways that don’t make sense for either a programmer or a writer. A programmer has to work in terms of structure and algorithmics with the hardest problem on many projects being finding things of interest. A writer has to work on the creativity of the prose, entertainment, whether it reads so that it makes sense, and how best to keep the reader engaged.
When I code I don’t give a damn if the reader is engaged, I care if he can understand what’s written, but I’ll be damned if I’m supposed to entertain some dork who’s paid to know this stuff.
These flaws then extend to every other aspect of computer science. When people do LP they typically keep all the code and prose in one giant file, maybe a few medium sized ones if you’re lucky. This is a nightmare for revision control since people on a team will constantly be conflicting with other team members. This makes it hard to break out the work, share it with others, hell even find where something is in the source.
Can you imagine trying to grep for a normal C idiom but having to modify what you look for to handle TeX escaping just in case it shows up in a spot that’s inside TeX?
In the end, LP seems to have missed out on the biggest advancement in software development since it started: compilers. A compiler no longer expects you to keep your source in one file, but instead has insanely good (or in C’s case, okey-dokey) support for inclusion, exporting, combination, and sharing of code. Modularity and encapsulation into libraries, modules, objects, and functions has all been supported by compilers since they started being used to create larger projects.
Yet, LP almost misses this and does an “inverse compiler” You write the final combined document and then it writes what you’d normally include in the document. Insane.
It makes much more sense and works a hell of a lot better with modern tools (“modern” being after 1980) if the way programmers do Literate Programming works the way they do regular programming.
But, TeX is still pretty damn sweet.
2008-04-18
While working on a more complex build I decided to make recursive imports work and clean up the syntax for imports in Vellum
Now, your imports are more explicit (just the way Python people like it) and more consistent:
imports [
recipe(from 'scripts/testing' as 'testing')
recipe(from 'scripts/dist' as 'dist')
recipe(from 'scripts/sample' as 'sample')
module(from 'vellum.commands')
]
This is the import stanza from Vellum’s build.vel file. What you see is there’s two type of things you can import, recipes and modules. Recipes are just other Vellum scripts that contain common tasks you need. Modules are Python modules that have code you need in the form of commands.
I pushed the version of Vellum that has this as version 0.14-rev85 so it’s on the Cheese Shop and in the bzr and launchpad. I would avoid it right now unless you’re interested in the latest code as it currently doesn’t load from the ~/.vellum/ directory yet so your common stuff will break.
I’ll do another release later today that brings that back.
Python Is My make
I’ve finally figured out a huge difference between what I’m doing with Vellum and what make does that turns certain types of programmers off from Vellum.
When you work with a compiled programming language like Java, C, or C++ the build tool is an essential part of your daily working. You can’t do anything without make happily building and compiling your software so you can run it.
Without make you can’t test, run, analyze, debug, or do anything. The entire chain of build junk in a compiled language is useless until you follow a large number of steps to finally produce an executable. Even if you have just one .c file you still need one additional compile step before you can run it.
(NOTE: Although tcc makes this possible, very few C coders use it.)
Another necessity is autotools. Without autoconf and friends you can’t build your software on multiple systems and still use advanced features. This makes autotools another essential thing, but only if you use a language like C that has portability problems.
In a dynamic language like Python or Ruby I don’t have this problem at all. There is no need for a make to run my program. I just run it. The interpreter is my make and does all the stuff I need to make the code I wrote work. I only violate this when I write a C extension, and even then both languages have tools that help me do that consistently.
I don’t need complex steps, or linkers, or loaders or gold from Google to happily work all day long. I don’t need things like autoconf because Python is portable unless I’m writing C extensions, and even then ctypes lets me avoid a lot of that for quite a while.
When I code, I need to run unit tests, but even that is handled for me by nose so there’s not much for me to do manually. I don’t have a build phase, I don’t have build errors, I just have code that runs or doesn’t.
I can even use Python as a debugger, a console, the works. I don’t need fancy builders and test harnesses just to see if a few lines of code work. With a compiled language I’ve had to create entire Makefile or Ant builds taking hours just to try one idea.
In essence, for programmers using compiled languages make is their Python.
For Python programmers, make is nothing more than a complicated if-statement with lots of magic.
This is why, when I wrote Vellum, it didn’t really need to know how to handle the build junk of a .c file. Vellum is higher level than this because Python already does all the crap that make does. What I needed was a tool to help automate the things Python doesn’t do like packaging, distribution, test automation, bzr management, and code generation.
To a programmer who’s entire life belongs to make this seems retarded. On the surface it seems so deficient because it can’t automagically figure out that depending on a .o file means to obviously build a .c file (unless it’s .c++, or .cpp, or .C, or .d) using cc (or gcc, or g++, or …).
Vellum obviously can do those things, assuming someone decides to write the commands needed for such magic, but why bother? I don’t do much with those languages these days, and when I do, why I’ll just write the bare minimum make necessary to build the C code and then have Vellum run that.
No point re-inventing 30 years of mythological C building lore just to keep a bunch of people happy who spend 50% of their day waiting for a program to make their program executable.
What I’ve found is that people who use make are very focused on how they can use Vellum to do what make already does. What they miss is that Vellum, unlike make, can be extended with Python to do anything you need.
2008-04-15
I worked on Vellum today after waking up from jetlag and Poland. I feel like it’s near ready for actual use by people. I even managed to polish it off with a nice little command line option for dumping the commands a build spec uses including their documentation. Check out this Pastie clip that shows it off.
What you see here is first the vellum/commands.py file with the commands that you can use. This file is actually loaded the same way your own commands could be loaded, by just saying what to import as a regular Python module. What Vellum does is look in this file for anything that is a function and then makes those into commands.
Next is the actual build.vel file that makes those commands active in the modules section.
Now, the nice part is right after that you see the results of the vellum -C command line option. It literally just pulls out the documentation for each function and prints it as the command’s documentation. If you give -C any command names it just prints those out.
This means you can write Python functions for commands you want, then put them in a module in ~/.vellum/modules or even just make a regular Python project out of them and install in the normal way. Once you do that you can use those in your build specs.
Check out the project page for information on getting this and using it. I’ll have more extensive documentation on it soon.
620 Lines Of Python
Using this awesome CLOC program I figured out that Vellum is only 733 lines of code, but 620 if you take out the parser.py file generated from parser.g. That’s pretty fucking amazing considering all the shit this thing can do already.
Think about this for a second: In less than a month I managed to create a build tool that parses it’s own build format with dynamic scoped imports, loads additional Python modules to give you a command structure, and then runs arbitrarily sized shell or Python code according to the dependency structure. It’s also got a decent test suite, is fully self-documenting, and has comments on almost every function.
That says a lot for Python. Combine my efficiency working on Vellum with how quickly I cranked out Idiopidae and Python is really proving itself as a great workhorse language.
I also really enjoyed the interplay between people who wanted a Python only syntax and those who liked YAML syntax. What I got out of that exchange was what I consider a very nice setup for Vellum where users have a nice simple format for the build file that’s high-level, but can easily write new commands in regular Python modules for more complex work.
I don’t think I could have made that leap if it weren’t for the people who were talking to me about the benefits of both needs.
Next up for Vellum is a more extensive set of documentation for it so everyone else can learn to use it, and then a way to distribute recipes. I’ll probably use the Launchpad project for the recipes.
Let me know if you grab Vellum and start getting it working. It should just take a simple sudo easy_install vellum to get it now.
2008-04-13
Update: I totally fucked up this post. I corrected the URLs but just wait until tonight when I drop another release with some goodies.
I’m still in Poland enjoying RuPy 2008 but thought I’d put out a release of Vellum 0.11 for people to play with.
You can grab it from launchpad where you can get the bzr repo and download source tarballs.
You can also easy_install it, but this is new so let me know if it works. You should just have to do:
easy_install vellumFinally, if you want the bleeding edge Vellum then track my bzr repo:
bzr branch http://zedshaw.com/repository/vellum/Now, what’s in this release? It basically has all the features I want but isn’t well documented. There’s extensive tests and docstrings on every method, so a brave programmer could figure it out. For the most part the Vellum project page has samples of the new Vellum format, and there’s sample commands in the vellum/commands.py file in the source.
What you get in this release is the ability to write new commands for your build spec in pure python, plus tons of little clean-ups and bug fixes.
Try it out, and let me know if you can get it working. I’ll be writing a short manual for it pretty soon.
My RuPy Presentation In Bazaar
I’m putting up the full Bazaar repository for my RuPy presentation so other people can grab it and play with it. This presentation was done using all my tools. Vellum for building it. Zapps for Vellum. Idiopidae for code injection into the presentation.
The presentation was done using LaTeX Beamer to make this PDF presentation which was actually generated from this YAML file using a small Python script I wrote to generate the necessary (and repetitive) LaTeX.
I’ll be cleaning this up and writing about it when I write about RuPy sometime next week. So far RuPy has been a blast and I really like Poznan. Virgin Atlantic and Heathrow Airport however can die in the ocean.
2008-04-09
Well, well, well. Google comes out with their App Engine thing and what do you know, they use YAML for their application descriptions.
I don’t see any Pythonistas complaining about it publicly. Imagine that.
See, that pisses me off because several people said nobody would adopt a project if it didn’t use Python from stem to stern. I personally think if you can’t think in a few languages on a project then your a dumbass, but I obliged folks and wrote a Python parser for Vellum’s builds.
Then people said they didn’t like that format either.
This folks is the classic problem with programmers today. They absolutely refuse to learn anything new unless they can see that learning the new hotness will give them an immediate 200% boost in salary or get them hot honeys at the next conference.
I use a YAML format for a build file and it’s not good enough. I write a Python one and it’s still not good enough.
Google uses YAML and people can’t wait to blow Eric in the back bathroom as thanks for using Python.
Well, since that’s the case I’ve decided to just write my own damn format for Vellum. And it’s got strippers, and blackjack, and uh fewer symbols to deal with, and less typing. Wait, screw the strippers and blackjack, how about just less typing? Awesome. Since I can write a parser in about an hour I just did it. So there.
That’s right, Vellum just has its own syntax. It’s obvious Pythonistas can learn a new format when it’s tied to the handle of a massive pot of gold. My format is even cooler than YAML or Python, will give you a 300% boost in salary, and will eventually hand mold a Jenna Jameson Real Doll™ for you since you can’t get hot honeys no matter what language you learn.
Check out this abreviated grammar for the whole data format:
rule input: ( reference | COMMENT LINE)* ENDMARKER rule expr: atom | reference | structure rule reference: NAME expr rule atom: NUMBER | STRING | SH LINE rule structure: LSQB elements? RSQB | LPAR dictmaker? RPAR rule elements: (expr)+ rule dictmaker: (reference)+
That’s right! My format is only 7 productions and it can produce the same structures Python and YAML can (minus retarded shit like references). Take a look at the sample now and bask in my skills as the format makes you calm and clean. I was definitely channeling REBOL on this one.
And this only took me like 3 hours to craft and make Vellum use again (thank god for tests). That’s right, I’ve changed formats for the Vellum build file 4 times and I’ve written two parsers for it and I’m having fun so fuck you all. (sniff).
All Kidding Aside, GAE Is The Shit
Alright, I actually think it’s hilarious that Google came out with their stuff just a few weeks after PyCon2008 and didn’t seem to clue in any of the web framework developers. I was hanging out with the Turbo Gears and Pylons guys for most of PyCon2008 and they didn’t mention anything. I wonder how they feel about that?
I’d be a little pissed, but just for a few seconds before I started counting my mountains of cash cause Python is gonna be hot! I’ve already got two or three learning applications lined up and I’m even thinking of a slick little script to make cranking out simple GAE applications easier.
Yes, GAE is incredibly cool and I’m am so glad I jumped on the Python trail just before they released it. I can now take my new found fancy Python learnin’ and go do some real programming for a change. I don’t have an account because Google is a stingy bastard, but who cares. I can actually download their whole SDK and have fun tinkering on it. By the time I get done making something to share they should be open for business.
What I’m very impressed with is the deployment mechanism. After working on Mongrel for years and watching all the hosting companies barely make any headway with Ruby, it is so nice to see someone get it right. Just set it and forget it. None of this binding together 6 shitty services with duct tape like you do with AmazonWS.
Echo2 -> Campfire -> HuddleChat -> Utu
Everyone seems to like 37signals’ Campfire so much they came out swinging with claims of a ripoff from Google’s HuddleChat.
Well, silly boys and girls, rails-core ripped off the idea and probably most of the workings for Campfire from NextApp Echo2 ChatClient Demo. I know this because I was in the rails-core IRC channel and I showed them how cool this Echo2 framework was, including that chat demo. A few weeks later they had Campfire and since they say it took them two weeks to write it, I’m guessing they got lots of inspiration.
Sure, they probably won’t admit that, since they’re “innovators” you know, but it’s a really odd coincidence.
Who knows, maybe they were working on Campfire all along and they just used their psionic telepathy to will that I go find the Echo2 Chat demo and paste it into their IRC. Or, maybe I just remember it differently, who knows. My brain don’t work that well anyway.
However, there is a precedence. They also admitted stealing the mechanism for updating two divs at the same time from one Ajax call from the Nitro project. At the time this was the biggest barrier for things like RJS to work, and rather than figure it out themselves they stole it from poor George and never gave him credit.
Sure this simple Echo2 demo is a pale pathetic competitor to the glorious Campfire, but it also is just a demo that hasn’t changed in years. I actually dump it on Java application servers when I need to do some simple chat with people. It’s pretty damn useful, so I can see why 37s would take the time to rip it off.
Now, I will gladly admit that My Time With The Rails Core Cult was a major inspiration for the creation of Utu and that their Campfire actually did make me hate stupid people so much that I felt I needed to create a weapon of destruction for weeding them out. Probably the reason I haven’t worked on Utu in a while is that I stopped hanging out with the Ruby dickheads so I’m not so inclined to hate the flesh of all who type “fuck you” at me.
Therefor, I’ll freely admit that rails-core, you guys inspired me to find new ways to hate people over chat. Thank you.