Spindle

So I looked at Mashups, and I looked at my own personal gripes about application development on the web, and I started trying to find a solution. I began at the language level, because that is the level at which I work (as a programmer), and also the level at which I think.

The problem space of creating a nice application development platform is one with a lot of answers. The problem of securely distributed computing with untrusted sources less so. So that’s the part I’m approaching first.

Erlang, Gears, Isolation

Mashups are happening on the web now (not securely) and Gears is an attempt to fix, but let’s be realistic. We cannot keep putting band-aids on this. (Well. We can. I just hope we don’t.) Without straying too far from what has made it successful on the web, what else is out there to see what works?

Erlang comes to mind. (I know I’ve said this stuff before in previous posts, but I thought it would be good to refresh). The message passing model works well for concurrency, but also for security. Both are going to be important for us. There is no way to avoid concurrency in a system built for network communications. The internet is one fat concurrency issue. The single threaded JavaScript model doesn’t work, and asynchronous callbacks are clumsy. Additionally, isolating processes with no shared state and a message passing model is a great way to keep untrusted sources from wreaking havoc.  The smart people at Google recognized this and created Gears workers. The question you may be asking is, “Well if Gears does this, why do we need a whole new language?” I’ll tell you. JavaScript was never built to use a message passing model. Using strings to pass messages around sucks. The lack of message pattern matching is a pain. It still doesn’t solve the major problem of accessing the dom. And asynchronous callbacks are still clumsy.

So let’s say we put Erlang in the browser. Would that actually work? Doubtful. Erlang unmodified has plenty of it’s own problems, or rather, it has some design choices that would not work well for the browser. For one thing, it’s strict immutability and almost purely functional syntax would not fly in the general public. For another, it just has no concept of a sandbox. It’s all or nothing.

The Sandbox

Many languages have implementations that operate in a sandbox. Java prides itself on sandboxing. It has those marvelous confirmation dialogs. JavaScript is also sandboxed. In both of these cases the sandboxing is really only protecting the user and the system from the program. What both of them lack is a way of protecting one part of the program from another part of the program.

The first concept when conceiving of Spindle was that it must have the fundamental concept of a sandbox. A barrier protecting one part of an application from another. A sandbox is more than a process. A sandbox is basically defined as the place where code lives. No code is shared between sandboxes. A function in one sandbox is not available in the other unless it is explicitly loaded in that one as well. It goes without saying that they do not share state. A sandbox can spawn one or more lightweight processes to actually execute code. These processes share a code base. This way code does not have to be loaded in duplicate into each process. The only communication between sandboxes is through messaging. In this way, it is the same as between processes.

Breaking up data from behavior

So I was thinking about this message thing. I thought about my own experience passing json between client and server. Luckily for my team, we built a system from automatically marshalling and unmarshalling so we never dealt with the raw json. It seemed to me that if messages were basically json objects, that should be built in for free.

So I thought that it might be a good idea to clearly divide the data from the behavior. An object has certain “properties”. When turned into message format, these properties become the data structure a la json. When accepting a message object, it should be just as easy to flip it back.

Imagine you send a Person object from one sandbox to another.  Or even from server to client. Using json notation, lets say it just looks like:

 {"firstName":"Russell", "lastName":"Leggett"}

But you actually have a Person class (or prototype) with some functions. Instead of having to create a new Person object and transfer the data, it would be nice to be able to ‘imbue’ the object with Person functionality. Like so:
personVar <- Person

Now this might seem like something as simple as copying functions over like many utilities do, but there’s more to it. It would be more like swapping out the prototype on the fly.

This thought process brought me around to the idea that the specification for functions should stay away from properties. This breaks the fundamental concept of prototypes, but hey, I’m not trying to build JS2.

Those were my main initial thoughts going into Spindle. There are a lot more ideas I’ve come up with since that starting point, and I’ll get to those next.

Advertisements

Mashups

So I tried to think about the web from the perspective of an application platform. Having built what we have, what could we do differently if we could do it from scratch? Clearly, security is one of the biggest problems. It is a problem with existing applications, but many of those security holes can be protected against with effort. The even harder security problems are the ones that have no solution. The kind that are attempting to be solved by Google Gears. Mashups have become a popular idea that cannot meet their true potential due to insecure connections between domains/owners/code bases.

In a recent presentation by Douglas Crockford, he talks about moving the web forward. There is a lot of great stuff here. There always is. But there are two major things that I wanted to address.

  1. “The next great leap [in software] might realize the dream of assembling software like Lego.” He further stated that that leap was being realized already through Mashups.
  2. Mr. Crockford goes on to describe how the current web technologies (both open AND closed) are far too insecure for anything but the most trivial Mashups. He then lays out how the web can move forward:
    • Safe JavaScript subsets (Caja, Cajita, ADsafe)
    • Communicating Vats (Gears)
    • Secure Programming Language (?????)

Interestingly enough, I recently saw something on InfoQ about “Lego” software in a presentation called, “The Lego Hypothesis” by James Noble. His presentation is worth a listen, but it is long and rambling and a little hard to summarize as easily as Douglas Crockford’s. However, the subject matter was relative. He discusses the history and feasibility of the dream of Lego block software. The same concept as described by Crockford in his presentation that he believes will be the “next great leap”. Noble demonstrated the complication in the Lego dream. Mostly that it’s a lot more complicated than plugging different parts together. You cannot build a complete application out of simple reusable parts. Some things have far too many dependencies to be simply abstracted into a reusable plug interface. In contemporary programming the best we can hope for is to glue together what we can to reuse.  Near the end of the presentation, he did in fact point out how Mashups have the right idea and that more and more software will go in that direction.

I agree with Crockford and Noble. The fact is that modern software and the web are fully intertwined. There is no going back to the dark ages of isolated machines. It’s not just about RIA anymore. And the fact is, as we move more and more towards networked information, we will need the ability to integrate between parties that have to operate under mutual suspicion. Crockfords third point about how the web can move forward has some question marks next to it, but I hope that Spindle can be a possible solution. A language with the goals of secure distributed computing.

I know I keep putting this off. I started this post with the intent of actually describing Spindle, but I guess it’ll have to wait till next time.

A Web Application Platform

The Layers

Ok, so it’s another ambitious future looking post. I’ve been thinking a lot about the architecture of a browser platform that fits what I’ve got in mind. Looking at the HTML5 mailing list largely confirms my thought process. I noticed a few weeks ago a thread in the list regarding text in the canvas element.

> I still think by introducing the drawString() method into Canvas we are
> opening the same can of worms that was open in SVG.
>
> If we go that way we will need a drawParagraph() method to draw multi
> line strings or blocks of text with wrapping and a bounding width. We
> also need to be able to stylize the text, i.e. changing the font-weight
> / color / font-style … of any word.
>
> The list goes on and on … and HTML and CSS already cover it all.
>
> The HTMLElement.drawElement() method should be no problem to implement
> since userAgents already do render HTMLElements.
>
> Having it return an ImageData object will make it insanely simple to
> manipulate in Canvas. The text elements/contents can easily be in the
> fall back content of the Canvas tag thus keeping it accessible. Getting
> the bounding box of an HTMLElement is no problem either in JavaScript.
> And applying gradients and patterns can be done using a fillRect() with
> the appropriate globalCompositeOperation.
>
> Everything (almost) is there. Let’s not re-invent a square wheel.

Let me just summarize what I think is important here. Text on the canvas highlights something very key, the browser is in many ways a specialized graphics engine. Under the hood it is capable of a lot of things, but through html we are given just a small subset. SVG and Canvas are also subsets of the capabilities built for other purposes. Really, one can think of each one of these, in addition to CSS as Domain-Specific languages (well, Canvas is really more an API) that specialize in accessing certain portions of the browsers abilities, and there is a lot of cross-over. SVG is markup based vector graphics and Canvas is command based, but they can both draw arbitrary shapes and create complex graphics. And if you look at what Webkit has been up too, you can see that they’re pushing CSS to the next level as well.  They’ve got support for CSS gradients, canvas drawing, reflections, and masks.

Cutting Through The Layers

I am reminded of the blind men and the elephant. Each man feels a different part and thinks it is a different animal, because they do not see the whole. Each of these languages gives us a piece of the elephant, but wouldn’t it be nice to leverage the whole damn thing? I am a huge proponent of domain-specific languages, but they can’t work in a bubble. Imagine defining rails without ruby. Maybe rails can cut it for the 80%, but you need the general purpose language when you have advanced logic rails doesn’t account for.

But Keeping Them

Let me stop for a second and once more reiterate that the concerns of a web application may not be the concerns of a web page. Advancing the abilities of html and css alone would go a long way for those concerned only about documents. The document aspect of html and css are extremely important and should not be marginalized. Html and css should live on as specifications independent of the browser. That is the beauty of the open web. It is more accessible than just through an application on your pc at home. Content on the internet is accessible through any number of devices, and the specifications that we’ve built for the internet can live on without it. JavaScript as a programming language, html and css as a document format, and the whole ball of wax used for various widget platforms.

Bringing It All Together

So what do we do with this disparate collection of specifications that overlap and work with each other in various ways? For developers targeting the mainstream, it would be most advantageous to have a single, solid development platform. This is the draw of Flash and other plugins. So, for the sake of argument, let’s say we start with that. This hypothetical development platform would be designed to be completely on par with (or I dare say better than) the offerings of other plugins/RIA platforms. The open web provided the seeds of innovation that have spurned the next wave of software. It should not be relegated to the back seat when RIAs become the norm.

I come from a Java world (I know, I know). While it is not perfect, it has a history of multiple implementations on multiple platforms by multiple vendors with a high degree of compatibility. The write once, run anywhere promise used to be something of a joke, but now it holds true more than any other platforms I can think of.

I don’t want the open web to become Java. Far from it. I simply think it is a technology that took a similar problem and came up with a fairly successful solution.

So What Am I Suggesting?

What we as open web application developers need is a true Web Application Platform.  The same way that Java, Mac, Windows etc. provide a complete platform for robust applications, we need something capable of similar capabilities, but solving a slightly different problem. The Web Application Platform needs to be safe, loadable from the web without installation, and fluidly communicate with the web while taking advantage of the power of the local machine. I want to be able to dig deep and have access to painting apis, layout managers, and low level loading apis. HTML and CSS are high level abstractions that can be layered over this, and rarely should the lower level stuff be needed, but to truly be powerful, it needs to be there. Adobe Air is getting close to this type of power, but clearly it is proprietary, and is heavily dependent on flash. I just want something more.

Posted in ideas. Tags: , , . 1 Comment »

*Web* Developers vs Web *Developers*

Preface

I’m not trying to make the distinction a hierarchical one.  There is a whole spectrum of equally valid web developers.  Some come from a software engineering background and work mostly on the server, some don’t know much about programming, but they are whiz-tastic at crafting semantic markup and css.  Don’t forget the thousands of n00bs, the WYSIWYG users, and everything in between.  The web is for everyone. That is one of the most important aspects of the Open Web.

Simplification

Let’s just break things down for the sake of a simplified discussion. The people developing for the web (whether it be sites, applications, or something in between) either have a background in programming or they don’t.  Looking at the origins of the Open Web technologies, specifically HTML, it was intended as a way of formatting and linking documents in a very simple way. No programming experience needed. As that changed, web technologies became more complex and programmers were needed.

So, following logic:

  1. Is it important to continue supporting both programmers and non-programmers?
  2. Can we have a single, unified model that supports both at once?
  3. If we had to choose a group as the highest priority, which would it be?

sigh – ok let’s give it a shot.  Yes. Hopefully. The programmer group (don’t hurt me).  Basically, my argument would be this – you can’t squeeze blood from a stone.

XYZ

Lets say there is a set of functionality available to language X. [When I say language, I mean its syntax, but also its libraries and system apis.] X is tedious to work with, but represents all possible functionality.

Y is a language built on X that can accomplish 90% of what X can do, but in a way that is much easier to work with.  It still takes a trained user to work with Y, but they can get a lot accomplished.

Finally, language Z is a domain specific language built on Y that is simplified for a specific use that takes very little training to use. It is a lot more forgiving to its users, and while it can only accomplish 20% of what Y can do, it can do 80% of what the users can possibly want.

I’m sure you already picked up on what I’m putting down, but I’ll elaborate anyway.  XYZ is a process that happens all over the place with great success.  In fact, all modern computing is built on this very process. Machine instructions are like X.  Anything the computer does must eventually boil down to it.  But we build abstraction layers on top of it because its the only way to build scaleable software. Then we add more layers as it makes sense.  One important thing to remember, though, is that you always build on any layer.  In the example, Z built on Y and Y built on X, but maybe language W also builds on X.  Maybe W has a different idea of what would be the important 90%, and what the syntax should look like.  Maybe T, U, and V are all small languages like Z, but they just want to cover a different 20%.  The problem is, it’s very hard to reverse the order and build Y on top of Z instead of Z on top of Y.

Enough letters already!

I know, sorry.  Let’s apply this logic to the Open Web.  Right now, the basic building blocks that we have available are HTML, CSS, and JavaScript.  These were all built with the end user in mind. At the time of creation, the end user was not software engineers, and the purpose was documents, not applications.  I guess what I’m trying to say is that right now, we might be trying to build Y on top of Z.  Throw in a lot of workarounds and hacks and we’re getting there, but there’s just no substitution for a more solid foundation.  I will present just a couple of examples.

  1. Security.  I cannot stress this one enough. We need to be ready for secure mashups. They are coming and they are important. Having a global space is a big problem.  Not just javascript objects, but also the DOM.  And the DOM is not very secure.  We need the foundation to ALLOW better security.  Even if the higher level opts not to use it, if it is impossible to achieve at the base level, it is impossible to achieve at a higher level.
  2. That damned file input! This is exactly the kind of constraint that is still worked around through hacks, but it seriously needs improvement.  The point is, without lower level control, there is nothing you can really do about it.
I’m going to end the post here.  These thought are stewing around in my brain, and I’ll post something else when the they’ve simmered long enough.

Gears as a prototype platform

So the blog has been silent for a week.  I was hoping to not let that happen so early on, but I only have so much free time, and I have to divide it between getting things done and talking about getting things done. So here’s the scoop on what I’ve been working on:

I’ve been researching a lot of different areas to at least get some goals and rudimentary plans together.  In general, I’ve been trying to find all of the areas that can be used in some way to prototype the behavior I’m looking for.  The main code bases that I’m looking into are Gears and Webkit.  I’ve also been spending a lot of time with HTML5 and SVG.  At this point, I’m just a little overwhelmed.  The scope is obviously huge and I’m not sure where to start, but I think I’ve got some good ideas.

The First Step

As only one guy for now, I just want to get the minimum effort going that is required to get the project off of the ground.  I think the easiest way to start is to begin with javascript language improvements.  The easiest way to begin that is to build a translator.  So here’s the plan: I’m mostly a Java/JavaScript programmer and the easiest tool for me to use is ANTLR. It’s extremely fast to build grammars with and I think translating to existing JavaScript a la GWT will be a very fast approach.  Aspects of the new javascript such as actor style concurrency can be simulated using the Gears worker pool.

At first, I will just translate on the server, but the next step can be to create a new Gears module that will perform the translation on the client.  When a good working model is established the next steps can be decided from there.

Future steps

At some point in the future, a visual component of the plugin will be required.  Gears has nothing of the kind right now.  This will be a much bigger challenge than just having another language runtime.

While I don’t want to just settle for what is available, I do think that building off of existing technologies is much more realistic than starting completely from scratch.  I think that SVG could be a very good building block for the next evolution of the open web.  HMTL5 is currently planning on directly including it, so its support should increase dramatically over the coming months.  What we need is a better way of leveraging it.  sXBL is a start at componentization, but I think we need a more cohesive approach.

Starting from scratch would be extremely silly.  At the moment, I think that webkit has a very good base. It’s been proposed half-jokingly before, but I am saying non-jokingly now – I think some form of webkit as a plugin could work.  That is for a later date, though…

Help!

I don’t really expect anyone to pour time into some crazy scheme with no code to show for it yet, but if are interested and want to help, check out the group.  When I scrape some code together I’ll start a google code project.

Let’s Get Crackin’!

I recently had a conversation with Brad Neuberg about the concept of using a plugin to have an Open Web competitor.  Brad suggested that this was precisely what Google Gears was trying to do (sort of).  In a recent post of his (which has since sparked a conversation across the blogosphere), Brad discussed the definition of the term Open Web and its importance, but also how Gears can help to push the web forward. In our conversation, he asked, “If you were to add functionality to Gears that doesn’t enhance the web’s existing technologies, but rather creates new ones that live in the browser through Gears what would these look like?” The following was my response:

………………………………………………………

1. I think data binding needs to be built in – with mechanisms for formatting and validation (and if you mention XBL, I’ll respond with a blank stare.  Seriously? Stuff like that is the reason Flex and Silverlight are looking so good right now.)

2. HTML5 and SVG can get us most of the way there when it comes to visuals, but its not very friendly for application development because there’s no abstraction layer.  There is no way to compose or componentize a set of elements and attibutes together and then use it as a custom tag for example.  JavaScript frameworks have gone a long way towards simulating this, but it needs to be built in, and it needs to be easy. I would propose an addition to the HTML syntax that basically allows for templating in a very component oriented way.  These templates would be incorporated into the binding syntax, so that they could “re-template” when needed without having to make explicit DHTML calls. I would also advocate for staying with xml syntax.

3. CSS is a really cool idea, but it needs some re-evaluation.  First of all, it needs better layout management.  The hoops required currently are a little obscene.  Even adding a “layout-manager” property with a few possibilities would go a long way.  I would also definitely add CSS variables that, again, could be integrated with the binding layer.  CSS expressions were definitely a mistake, but variables would be extremely powerful.  For some ideas, just look at what webkit is doing with their css transformations and animations.  It would be so much simpler to just put a variable in.  Then a simple easing library could be used to change the variable over time to create animations. Finally, I think that CSS could be better incorporated into the new component model.  For example, it would be helpful to be able to scope rules to components, and allow custom tags to be selected.

4. I’m in the group of people that isn’t totally gung ho about JS2.  I understand the motivation, but its looking like a bit of a kitchen sink language. I think JS needs some improvements, but I’m actually looking at Erlang for inspiration instead of Java and Python.  In my vision of a future JavaScript, I see a few things.  First of all, I think there are some functional language features that would be good to add considering JavaScript is a lready a very functional language. I would like to add overloaded functions that use matching and guards for differentiation. I would also like to steal some aspects of the big Erlang feature of concurrent processes.  Here, I think, is a perfect convergence with Gears.  The Gears worker pools are a lot like Erlang processes (which I’m guessing you knew).  No shared state, separate process, and no access to the dom.  As I’m sure you know, this can be extremely helpful when trying to stay secure doing mashups, offload intesive operations from the main thread, and communicate with the server. Additionally, I think that there needs to be a little more in the way of modularizing code, allowing private data members, and facilitating better code reuse.  I think prototypal inheritance and mixins are definitely better than classes for the language, and I’d like to add some more syntax to encourage them.
………………………………………………………

It’s great to talk about the deficiencies in the technologies we have, but to really move forward, we need a conversation for what can come next. And by next I guess I mean the next next.  HTML5, CSS3, and JS2 are all really great things.  I encourage people to be as involved in those as possible, but I still have a lot of doubts about its ability to stay pragmatic, on track, and happen quickly.

If you could snap your fingers and have a plugin with a wide install base built for web applications (as opposed to web sites), what would it look like?  You can leave a comment, or even better, take a look at the group.