When I was fresh out of college, my first job was working for the OpenGroup, and my command line looked something like this:

In 2009, fired by the white heat of years of cutting edge research and development, my command line looks a lot more like this:

To be fair, we've now got transparency, anti-aliasing, our shells now do better completion. But that's about it for changes. Considering how much the rest of the industry has changed, that's shockingly glacial.

For the past week or so, I've had the privilege of working on the Bespin command line, and I think there's a lot that we can do to make things better.

This is what my Bespin command line looks like now:

This post is all about how we can make command lines better.

The core of the unix command line philosophy seems to be: small programs and pipe. There’s a lot to be said for loose coupling but there are some drawbacks as well...

Zsh

I love Zsh, and love the fancy completion that it does. Basic file completion has been around since ksh, IIRC:

$ ls [TAB]
Applications/  Mesh/       Projects/
Desktop/       Movies/     Public/
Documents/     Music/      Sites/
Downloads/     Parallels/  backup/
Library/       Pictures/   bin/

But zsh can complete against more than just files:

$ telnet [TAB]
192.168.0.1
192.168.0.12
download.directwebremoting.org
download.dojotoolkit.org
getahead.org
hg.mozilla.com
localhost
people.mozilla.com
sandbox.dojotoolkit.org
svn.directwebremoting.org

Zsh knows that you don’t telnet to a file, so it completes against hosts (Does anyone know where that host list came from BTW?)

Zsh also knows about other commands:

$ git [TAB]
add                 -- add paths to the index
apply               -- apply patch on a git index file and a work tree
applymbox           -- apply patches from a mailbox
applypatch          -- apply one patch extracted from an e-mail
archimport          -- import an Arch repository into git
bisect              -- find the change that introduced a bug
branch              -- create and show branches
checkout            -- checkout and switch to a branch
cherry              -- find commits not merged upstream
cherry-pick         -- cherry-pick the effect of an existing commit
clone               -- clones a repository into a new directory
commit              -- record changes to the repository
...

That list of commands is the result of a 'git help' command, but that's not what I typed. Zsh has been doing something funky behind my back when I pressed [TAB].

Zsh can do different things depending on where in a command you are:

$ git add [TAB]
Applications/  Mesh/       Projects/
Desktop/       Movies/     Public/
Documents/     Music/      Sites/
Downloads/     Parallels/  backup/
Library/       Pictures/   bin/

My point, is that, whilst all of this is very clever, the implementation is also insane.

In order to make this work, the zsh guys have had to reverse engineer every single unix command (or at least all most of the ones that I use). And any time those commands change, then zsh is going to have to change too.

It's very useful, and I'm glad that zsh has done it, and they probably didn't have much of a choice, but this way of going about things is not the Best Idea ever.

I'm sure somewhere the bash guys are working on the same feature themselves, or maybe they've already done it. I don’t care - it just strengthens the point about the lunacy of the situation.

The Unix philosophy of small programs and pipe has a lot to be said for it, but there are serious drawbacks too.

Here's some of the ways that we're innovating when it comes to the Bespin command line, and benefiting from being about to integrate things better.

GUI integration

Traditionally you've had a set of shells (bash, zsh, etc) that work in a set of terminal emulators (xterm, terminal, gnome-terminal, konsole, etc) But there's so much more you could do if you had one program do both.

In Bespin:

• If you click on an old command, it gets copied to the prompt (the blue highlight is a mouseover).
• If you double click it's re-executed.
• Output is not restricted to ascii in a grid. We (predictably) use HTML.
• Error output is visually different to normal output
• You can minimize or maximize output (the help output has been minimized above) - I hate it when a vcs diff operation takes up 4k lines of scrollback obliterating what I wanted to see.

There's no reason why you couldn't do this in Unix (or even Windows). I don't think you’d want such a gui-shell as your login shell, but since its use is very much from within a GUI, that's OK - you’d probably launch it from the GUI anyway.

In Bespin we don't have pipes yet, but would it be neat to be able to inspect a pipeline visually as it's working, I'm sure it's possible with a gui-shell.

Linking Output to Commands

We link the output from a command to the command that created it. The obvious effect is that output from slow running commands no longer gets confused with what's happening now.

The 'slow' command is just something I hacked up to demonstrate asynchronous behavior.

A nice upshot of this is that we can implicitly add & to the end of every command, and you no longer have to wait for anything.

In essence, each command has it’s own separate stdout, which goes to it’s own div. Maybe there is a reason why unix shells don’t do something similar. Would it mean shells being curses based or something?

Command Meta-Data

With the exception of the reverse-engineering of the command line structure done by zsh, the integration between commands and the shell is limited. In Bespin, a command is a structure like this

{
    name: 'createproject',
    takes: ['projectname'],
    preview: 'create a new project',
    usage: '[newprojectname]',
    execute: function(commandLine, projectname) {
        /* code to do the work */
    }
}

This means that we can provide help as the user is typing. The idea is that TAB will guess what you want to do at all times.

This ties in a lot with GUI integration too. If the command line knows that the next parameter is a file, when the user presses TAB without typing anything to complete on, we can offer a dialog to allow selection of a file, and so on. This isn’t about forcing the user to switch to the mouse, but it is about allowing better feedback from the system about what options are available.

We've only really got started with this part - there’s a lot more work to do. I hope we'll be able to do some of the funky GUI integration that Ubiquity has been pioneering.

Attention Guides

There's a lot to look at on any command line, I want to find ways to keep the most important bits in front of your eyes at all times. I'd like to expire older commands more intelligently than a standard terminal (oldest dies first), Commands are more important if:

• They are still in progress
• They keep being re-executed
• They are waiting for input
• etc

We've only basically started on this right now, with the ability to hide/show output, but there's potential for it to make the command line lots more usable.

Plug-ins

Linking everything together is easy if you make one big monolithic lump. That's not the plan - we're evolving a plug-in strategy as we go, and planning on making as much of Bespin itself use the plug-in system. The key to coupling is to get the right level of coupling.

Status Messages

My other integration obsession is about status messages. There's a huge amount we could do to integrate the familiar concept of "what are you doing now" with commit messages, bug trackers, chat, and team tracking. More soon.

What other things could we do to trick out the command line?

The DWR project has a new lead developer - We've just closed a unanimous vote that declares David Marginian to be the new lead.

David had been around DWR since the beginning, and has been consistently one of the most helpful people on the mailing list, and has contributed to many areas, most particularly the documentation.

Q: Why? Because David is already doing a better job of leading DWR than I am, and because for a while I've had too many projects on and been trying to do a better job of less things.

Q: Am I gone for good? No, I certainly don't intend to disappear, and I've still got a lot of love for DWR - this is mostly just me admitting that I'm not the best man for the job, and that DWR deserves better.

Q: What are you doing instead? Quite a bit of Bespin. Oh, and like everyone else, I'm excusing my lack of blog activity to the whole 140 char thing. It's a cheap excuse, but if everyone else can use it...

I'd like to thank the David, and the whole DWR community for helping make DWR a success.

Yeah, so Bespin a cool editor, and it makes use of lots of bleeding edge tech, (and it's got several annoying bugs - sorry about those). But I think Bespin has potential way beyond just looking nice ...

Open source development: You hear about project X, use it, and it’s nice, but to use a worn out phrase - there’s an itch to scratch ...

So what do you do?

Open Source Development Before Bespin

You search the project website for the directions to the source control system, you check that you’re up to date with cvs/svn/git/hg/whatever and checkout the source. Next you need to figure out how to build the project with ant/make/maven/paver/rake/whatever, and maybe download new libraries to make the compile clean and you then configure your IDE or editor to the new project. You can then go about editing the code. When you’re done with that you might need to figure out xunit/yunix/zunit to run the unit tests. Then you have to join a mailing list to work out how to send your changes in, which might be a patch or a bundle or a zip or direct commit access. You might discover that you need to sign a CLA so the project knows who you are, so you mess about with printing/signing/scanning/emailing/whatever.

And then you’re done. With complex projects there could be lost more to it.

While you’re doing this you have no idea what the other developers are doing. Maybe they’re working on a totally different branch. Maybe they’ve done something similar already. Maybe they’re making changes that mean you should try a different approach. It would be good to know because you might be wasting your time.

Open Source Development After Bespin

You visit Bespin, and elect to edit project X. When you’re done, you click submit, and the project owners get your change.

Plus you can see what the project owners are doing with the source, and how your changes fit in.

The Power of Bespin

For me, what I find interesting about Bespin isn't the tech, it's the lowering of the barriers to entry and the socializing of open development.

Virtually no software is perfect, and there are billions of people that are affected by the imperfections. There are hundreds of ways to contribute to any development project, and millions of people who could use their skills to help if only we could find ways to get the sources of help to the problems needing work

This application of huge resource to complex problems is what the Internet has always been good at, but so far we've not really done much to help software development. I hope that Bespin can become part of the solution.

This isn’t going to become a Bespin blog (that’s here or we're on twitter). Irregular DWR/Ajax/Security programming will return to my blog and twitter feed shortly.

I’m very excited to be going to work for Mozilla Labs.

I have very much enjoyed my time working at SitePen - they are doing awesome work with Dojo and DWR, but with the main development on DWR 3 being done, it's time to move on. SitePen has some amazing talent, and it's been a huge privilege for me to have been able to share the ride, so I'm very grateful to Dylan and everyone at SitePen for allowing me to be part of it. I'm not going to be ignoring DWR either, so between me and the excellent team of Mike, Jose, David, Lance and many others I'm sure it has a long future.

My new role will be working with Ben, Dion and Kevin on Bespin.

You might be able to get hints about Bespin from postings that are around already. Oh, and the name - that might be a clue too. Needless to say I’m excited about the project. More soon.

Recently:

• Aaron Boodman - "I'm a big believer in silent autoupdates"
• PPK/Quirksmode - "I hate automatic updating"

Something Aaron didn’t mention - What happens when a silent auto-update breaks something? Anyone that's had to do battle with the demo-pixies silently changing something just before an important demo hates the idea that something might be silently altering their rendering engine as they are opening their laptop on-stage.

PPK's issue basically comes down to testing older versions - how can I test version X-1, when it's just been replaced by version X. On the one hand with silent auto-update virtually everyone will be on version X, so X-1 isn’t important, however I don't think that removes all need for testing on X-1.

Solution?

We should have silent auto-update. It makes the Internet safer, and we badly need that. But we should also have a system that keeps old versions around for some period of time with a way for users to revert. Maybe there is then an entry in the help menu that allows users to say 'it broke take me back', and maybe a preference that configures the number of old versions left lying around.

I have a suspicion that many corporates turn auto-update off in order to test the updates batches before internal release (or to put it another way - it's a cost saving) Maybe with undoable silent autoupdate they would allow the system to work without getting in the way?

I'm a big believer in undoable silent autoupdates.