Introducing ActiveShell: A Shell for the Web

Monday, January 16, 2012

For a few weeks I've been working on a project called ActiveShell, a shell for the Web. Like the Unix shell, ActiveShell is a way to give textual commands which produce some output or effect. Unlike the Unix shell, an ActiveShell session exists in the browser. The Web supports increasingly powerful applications, and we can do better than interfaces designed when hardware text terminals were the state of the art. We can reduce wasted effort by saving more output and all command history, and soften the learning curve by providing discoverability based on context. Rather than supporting only the local machine, or only remote SSH servers, we can support any service available over the network, such as social networking APIs, email, FTP, or database servers, through the same consistent interface.

This is an opportunity to design a better shell for the development environment of the future, which I believe will be the Web itself. Consumer computing devices are becoming less general and open, designed for consumption more than creation. Those born today may grow up without access to the kinds of flexible tools that sparked so many young people's interest in programming during the era of desktop computing. Bringing developer tools to the Web means bringing the full power of general-purpose computing to a new generation.

ActiveShell is currently at the prototype stage. I am focused on extending the prototype for specific practical use cases, while resolving any design issues around extension points that would be difficult to change later. Once this is done, new interface features, verbs, and ports can be created and shared by the community of users. A sysadmin might use ActiveShell to deploy to multiple servers; a student to follow an interactive programming tutorial; a programmer to fix a bug and commit the fix back to the revision control system; a researcher to load a sample data set, run some exploratory commands, and eventually queue a job on a cluster to run that same analysis over the full data set and publish the output to a Web server. These use cases require different capabilities and access to different kinds of remote systems, which is why verbs (which process data) and ports (which mediate I/O with the rest of the Web) are extension points. What remains constant is the interaction model of sequential, text-based commands, and the consistent interface through which data of any kind can be explored and manipulated.

There are many benefits to a Unix-like uniform interface between small, composable programs. In Unix, unfortunately, the use of plain text streams requires inconvenient parsing and escaping at every turn. We can do better by passing structured data, with a JSON-like small set of primitive data types and collections over them: strings, numbers, Boolean values, and lists and maps. String arguments are always quoted, using C-like string literal syntax. Command syntax is minimalist and allows verbs to appear before, after, or among arguments, which gives greater scope to autocompletion and hinting. Most of the ambiguity and ugliness of Unix shell command syntax is easily eliminated.

ActiveShell supports the ad-hoc re-use of the output of any earlier command. In Unix, command output is written to the terminal but otherwise lost, unless it was explicitly redirected to another command or to a file. This made more sense in the 1970s than it does now. Easy re-use of the output of previous commands encourages exploratory programming and data analysis. This ad-hoc routing creates a relationship between command inputs and outputs, which begins to erase the difference between using a shell and writing a shell script. After calculating a value by an ad-hoc sequence of commands, the entire calculation of that value and its dependency tree is known to the shell and can be used to make things simple that should be simple, such as re-applying the same complex calculation to new input data. A process that was performed on one value can be applied to a set of values without specifying the process again. The shell can be asked to "save the sequence of steps which took inputs A, B, C and gave the output X", and the result is then effectively an ad-hoc shell script taking three arguments, which can be saved as a new verb or even exported as a library function for use outside of the shell. Such scripts, and the command history itself, are stored in a parsed representation, so the command syntax can be freely changed without breaking backward compatibility with existing sessions or stored scripts (this has held back Unix shell syntax considerably).

Comfortable exploratory programming is also encouraged by providing a safe environment where such side-effects as accidentally erasing a filesystem are simply not possible. An environment where "rm -rf *" is even possible must be approached with caution, which hinders exploration and learning. ActiveShell begins with safe, reversible operations. Any side-effects outside of this safe environment must occur through ports, which are the means of communication with the world outside of the shell. An open port represents any external system or service, such as an IMAP email account, blog publication API, social networking account, version control system, or the filesystem of a remote server. Data comes into the shell session or goes out into the world only through a port. A port must be explicitly opened by the user, and if no dangerous ports are open, no dangerous side-effects are possible. Any service available over the network is made available to the shell by writing a new port implementation, which may run entirely in the client or may have a server-side component.

Hopefully this brief introduction has given you some idea of what the ActiveShell project is all about, and why I'm excited to be working on it. In my next post, I'll introduce the prototype, with a series of screenshots illustrating a simple task.

For further updates, follow me on twitter or github. Comments welcome here or on hacker news. What would you use a Web-based shell for, if it could do anything you would want?

Thanks to Hugh FD Jackson, Gary Katsevman, Devin Samarin, and Connor Lane Smith for their comments on earlier drafts.