Introducing ActiveShell: A Shell for the Web

For a few weeks I've been working on a project called ActiveShell, a shell for the Web. Like the Unix shell, ActiveShell is a way to give textual commands which produce some output or effect. Unlike the Unix shell, an ActiveShell session exists in the browser. The Web supports increasingly powerful applications, and we can do better than interfaces designed when hardware text terminals were the state of the art. We can reduce wasted effort by saving more output and all command history, and soften the learning curve by providing discoverability based on context. Rather than supporting only the local machine, or only remote SSH servers, we can support any service available over the network, such as social networking APIs, email, FTP, or database servers, through the same consistent interface.

This is an opportunity to design a better shell for the development environment of the future, which I believe will be the Web itself. Consumer computing devices are becoming less general and open, designed for consumption more than creation. Those born today may grow up without access to the kinds of flexible tools that sparked so many young people's interest in programming during the era of desktop computing. Bringing developer tools to the Web means bringing the full power of general-purpose computing to a new generation.

ActiveShell is currently at the prototype stage. I am focused on extending the prototype for specific practical use cases, while resolving any design issues around extension points that would be difficult to change later. Once this is done, new interface features, verbs, and ports can be created and shared by the community of users. A sysadmin might use ActiveShell to deploy to multiple servers; a student to follow an interactive programming tutorial; a programmer to fix a bug and commit the fix back to the revision control system; a researcher to load a sample data set, run some exploratory commands, and eventually queue a job on a cluster to run that same analysis over the full data set and publish the output to a Web server. These use cases require different capabilities and access to different kinds of remote systems, which is why verbs (which process data) and ports (which mediate I/O with the rest of the Web) are extension points. What remains constant is the interaction model of sequential, text-based commands, and the consistent interface through which data of any kind can be explored and manipulated.

There are many benefits to a Unix-like uniform interface between small, composable programs. In Unix, unfortunately, the use of plain text streams requires inconvenient parsing and escaping at every turn. We can do better by passing structured data, with a JSON-like small set of primitive data types and collections over them: strings, numbers, Boolean values, and lists and maps. String arguments are always quoted, using C-like string literal syntax. Command syntax is minimalist and allows verbs to appear before, after, or among arguments, which gives greater scope to autocompletion and hinting. Most of the ambiguity and ugliness of Unix shell command syntax is easily eliminated.

ActiveShell supports the ad-hoc re-use of the output of any earlier command. In Unix, command output is written to the terminal but otherwise lost, unless it was explicitly redirected to another command or to a file. This made more sense in the 1970s than it does now. Easy re-use of the output of previous commands encourages exploratory programming and data analysis. This ad-hoc routing creates a relationship between command inputs and outputs, which begins to erase the difference between using a shell and writing a shell script. After calculating a value by an ad-hoc sequence of commands, the entire calculation of that value and its dependency tree is known to the shell and can be used to make things simple that should be simple, such as re-applying the same complex calculation to new input data. A process that was performed on one value can be applied to a set of values without specifying the process again. The shell can be asked to "save the sequence of steps which took inputs A, B, C and gave the output X", and the result is then effectively an ad-hoc shell script taking three arguments, which can be saved as a new verb or even exported as a library function for use outside of the shell. Such scripts, and the command history itself, are stored in a parsed representation, so the command syntax can be freely changed without breaking backward compatibility with existing sessions or stored scripts (this has held back Unix shell syntax considerably).

Comfortable exploratory programming is also encouraged by providing a safe environment where such side-effects as accidentally erasing a filesystem are simply not possible. An environment where "rm -rf *" is even possible must be approached with caution, which hinders exploration and learning. ActiveShell begins with safe, reversible operations. Any side-effects outside of this safe environment must occur through ports, which are the means of communication with the world outside of the shell. An open port represents any external system or service, such as an IMAP email account, blog publication API, social networking account, version control system, or the filesystem of a remote server. Data comes into the shell session or goes out into the world only through a port. A port must be explicitly opened by the user, and if no dangerous ports are open, no dangerous side-effects are possible. Any service available over the network is made available to the shell by writing a new port implementation, which may run entirely in the client or may have a server-side component.

Hopefully this brief introduction has given you some idea of what the ActiveShell project is all about, and why I'm excited to be working on it. In my next post, I'll introduce the prototype, with a series of screenshots illustrating a simple task.

For further updates, follow me on twitter or github. Comments welcome here or on hacker news. What would you use a Web-based shell for, if it could do anything you would want?

Thanks to Hugh FD Jackson, Gary Katsevman, Devin Samarin, and Connor Lane Smith for their comments on earlier drafts.

  1. Ben Werdmuller

    Cool! For me, the obvious question is: can you pipe commands and direct output to other web services? That's the real power of the shell, and would turn this into an incredible piece of API glue.

    1. inimino

      Yep! Piping will use something a little different from Unix syntax. (The next post should have a worked example.) Then to send your final output to a web service you'll just point a port at it.

      1. Anonymous

        Why use a different syntax for pipes? I think the power in this is that I should be able to quickly adapt to the shell based on existing knowledge.

        google "hello world" | grep "python"

        1. inimino

          The syntax is evolving one feature at a time, and I'm not sure where it will end up, but at the moment I think references to previous command outputs are both more intuitive and more powerful than traditional pipe syntax. I'll say more about this in the next post.

  2. Elijah

    Sounds interesting. When will the code be posted to GitHub?

  3. Three Pipe Problem

    Check out the chickenfoot project.

  4. Anonymous

    you should look at yahoo pipes and the data tools they have. you might discover the wheel already exists and is generally ignored by people who should know better.

    1. inimino

      I don't think we're building the same wheel. Can I install Yahoo Pipes on my own server? Can I re-use their code, or the pipes created by the community, in my own non-pipe project?

  5. richard

    sounds great! I want to add a shell to a webgl game i am making, i'd really like to open the game up to be programmable from within the game and this sort of power/ease-of-use would be awesome! imagine importing all sorts of fun data and working on it within the context of a 3D game!

  6. Daaron

    I like the idea. When I think web, I'd use it for scraping, rest-ful communication, and some cloud APIs. Especially on scraping, I'd imagine something like zombie.js on a command line with the ability to create virtual screen shots. Every shell seems to come with it's own language, but how about JavaScript? It could even be built in JavaScript too...

  7. Lars Kemmann

    This looks very exciting.

    How would I use it? Ideally, this is the sort of tool I'd like to use for setting up continuous integration and automated deployment of my software between different servers. So I'd be interested in making scripts that can work with systems like trac, mantis, svn, mercurial, and ftp across different operating systems (primarily Windows in my case), both on a machine/network and across the Internet.

    From what I can see, this is fundamentally related to Microsoft's PowerShell, another replacement for text-based shells. It is a shell for (.NET) objects, much like ActiveShell is a shell for (JSON-like) objects. In fact, I wonder if you could use the JSON serialization mechanism in .NET to interoperate with ps cmdlets ("verbs")? It would be great to only invest in a single scripting language and have it work cross-platform.

    1. inimino

      Lars, that's one of the major use cases I'm interested in to drive this forward as a practical tool.

      You'd need ports that provide access to trac, svn, FTP, etc, and you'd eventually want a way to run scripts outside of a browser tab, with cron-like scheduling, maybe by just running the JavaScript verb and port implementations on the server side with node.js.

      You're right about the similarity to PowerShell. Interoperating with cmdlets on the local machine is an idea that I hadn't considered, but certainly would be a compelling feature. Probably something would need to be installed on the Windows host to provide a bridge between the browser and PowerShell.

  8. Nathan Hartley

    Cool. I will have to keep my eye on this project.

    I am a fan of, a simplified version of this concept, and Powershell, who's piping of full .Net objects is very slick. I would love to see these two concepts merged together in an elegant way.

  9. Ivan

    Sounds interesting, looking forward to the post on prototype.

  10. Andreas

    Good luck. I would like to read more about it!