This post lightly explores the problem of “automatically” backing up a git repository to subversion. Why would anyone want to do this? Well, if your organization has a policy that all code must make it into subversion, but your team is interested in leveraging git in a deeper way than just by using git-svn as a sexy subversion client, then you’ll find yourself pondering the question of repository synchronization.

For our purposes, we’re guided by the following requirements:

1. An at least periodic (< 24 hour backup) of code must exist in subversion — While it’s not strictly necessary to have a perfect commit history stored in subversion, all code should be present and buildable with minimal fuss.
2. Code must exist in git with full history.
3. Branching & tagging must remain usable and should properly translated between svn and git
4. To the extent possible, non-linear git histories should be handled
5. it is not required that all features of git are usable (i.e., sub-modules), but there should be viable workarounds for each feature we sacrifice.
6. it should be possible to have an external git/public mirror of source code.
7. There should be a way to take changes committed to forks of the public mirror to be merged back into the system.

Given the requirements, their are at least two flows that make sense:

• Subversion Primary is an arrangement where all commits are made into subversion (using the client of your choice, possibly git-svn) and “read only” git mirrors of the subversion repository support the other requirements.
• The Dumb Subversion Backup flips the order around and says that there is a single git repo that is the source of truth, which is automatically backed up to subversion (perhaps in a lossy fashion).

The remainder of this document will discuss these two approaches.

The Subversion Primary

In this scenario, all commits funnel first through the subversion repository. There is an automated process that polls or pushes changes from subversion into an internal git staging repository. That staging repository uses git-svn to map the complete subversion commit history into git, and performs some processing to deal with mapping branches and tags among other things.

The internal staging repository can then mirror to an external repository which is visible to all for code review and contributions. External contributors can fork, change, and send pull requests in any form they like. Internal developers can then manage pulling commits into the main svn repository.

Key Features

The Subversion Primary is a straightforward solution that has the following characteristics:

• Internal devs interact only with subversion and can use the client tool of their choice
• Downtime that affects the internal git repo or the public git repo do not affect internal developers
• All git repositories are ephemeral, and their loss or corruption is Not A Disaster.
• External developers are insulated from the fact that SVN is in use, except for the git-svn artifacts attached to commit messages

Challenges

There are a couple obstacles to surmount in this arrangement, and we’ll briefly explore them here.

Username Mapping

One problem that exists with git-svn is that when commits are pulled from subversion the ‘author’ field is meaningless. Here’s an example:

In order to map internal svn usernames (i.e. lloydh) to meaningful author entries (i.e. “Lloyd Hilaiel ”) we can use mechanisms built into git-svn. Simply create an authors.txt file that looks something like:

having created this file, use the -A argument on all git svn operations:

Mapping SVN branches and tags to git

Another problem that exists is how SVN branches and tag appear in git. Subsequent to a git svn clone you can view the remote branches with git branch -r. You’ll see something like this:

Because subversion has no proper notion of tags, you’ll notice that the tags set in subversion are branches in git. If you want your published git repository to look reasonable to an average git user, it’d be nice to turn these tags into proper git branches. To solve this problem I wrote a small shell script designed to be run after fetching changes from SVN.

Accepting contributions

Given the flexibility of git, there are many tactics for merging changes forked off of the public mirror back into subversion’s linear view of the world: the least sophisticated approach would be to manually do the merge using diff and her friend patch and perhaps add a little --fuzz. A much nicer way would be to cherry-pick or rebase changes directly into a git repository that is svn cloned from the original, and then to dcommit back.

This latter approach seems like it would be much faster and less error prone, however there’s at least one gotcha: In order to simplify the merging of changes, one should ensure that commits are identical between the public repository and the version that’s cloned from SVN. This means when an internal developer clones a git repository from SVN, they should use the same authors.txt file as is used by the internal git repo (which pulls from subversion and publishes to external). Additionally, the internal developers clone should refer to the subversion repository in the same way as the internal staging git repo does, specifically hostname must be the same in both cases (two cases where they might diverge include using ssh tunnels and aliases, or using DNS shorthands or ip addresses).

Once commits are bitwise identical in the developers svn clone and the public mirror, it’s possible to use the advanced merging and rebasing features of git in a straightforward way to absorb external changes and then dcommit them to SVN. Even if some circumstances arise and it’s not possible to get bitwise identical commits, it’s still possible to fetch an entire remote branch and cherry-pick individual commits, an approach which is perhaps trivially better than manual diffing and patching.

NOTE: In order to artificially join histories without bitwise identical commits (messages or authors), one might try the “ours” merge strategy. I’ve no personal experience with this approach.

I think whatever strategy you end up employing, the key, is that the person or people on the team that are most in love with git should be the ones integrating external contributions.

The Dumb Subversion Backup

In this arrangement, rather than all commits flowing through subversion, its simply a mechanism for the backup of the git repository. The key idea here is that there is a single git repository that is backed up to subversion. While other requirements then require the ability to publish the repo externally and pull contributions internally, these tasks can be accomplished in a thousand different ways leveraging the built-in features of git. Restated, the only element of this arrangement worth discussing is the challenge of actually synchronizing changes from git to subversion.

Key Features

• All developers interact with git, while subversion is a hidden implementation detail
• The internal team is on the hook to set up and maintain a more complex system than with the Subversion Primary arrangement
• Inevitably, there will be information stored in the many git clones that will cannot be represented in subversion, so restoration from subversion backup would inevitably imply loss of some information.
• External developers see no artifacts which suggest that subversion is used as a backing store

Challenges

While this arrangement undeniably requires more systems administration, most of it is straightforward. The setup of the main internal git repo will probably require the installation of some scripts to allow multiple users to push changes over ssh to the same repository, like gitosis or gitolite.

The real challenge in this setup, however, is given a git repository, how do you back it up into subversion? The range of solutions includes:

1. periodically take snapshots of the main branch of development (think, literally, a git archive), and overlay them on the current subversion view of the world. commit. prosper.
2. the same as #1, but do so for all branches present in the git repository (automatically bring branches into existence in SVN as they’re created in git)
3. #1 or #2 but on a per commit basis, with log message (and author?) preservation
4. invent a system that will actually use git svn dcommit to perform the commits, squashing non-linear histories into a linear history as it goes.

So, I’ve seen #1 in place on a successful, vibrant, large scale project. Depending on the intent of the policy that requires a backup to SVN (and how much you really care), this may or may not fit the bill.

Options 3 and 4 are what are really interesting. These options would seem to be possible with a little bit of well-thought-out scripting on top of the existing facilities of git-svn, and hold the potential to capture a significant subset of the information that’s represented in the git original — preserving commit messages, authorship, and performing admirably when smashing non-linear histories into something subversion can understand.

The path to implement option 3 would be similar to fairly well documented tactics for moving a single git branch with a (mostly) linear history into subversion, but would have to take into account multiple branches. Specifically, some areas of difficulty would seem to be:

1. keeping cruft out of the public git repository – when dcommitting using git-svn, the branch that you dcommit from is changed as meta-data is embedded into the commits. In order to use git-svn you would need to design a mechanism that could merge changes from the branch where a commit lands, into a branch which has been associated with a subversion branch, and dcommit from there.
2. incremental commits — An extension of #1, commits are made into the internal git repository, it would be required that you then move these commits into the branch from where you can dcommit. Doing so will require that you establish a merge base (common ancestor) between the pristine actual branch in git, and the git branch that’s associated with subversion.
3. non-linearity and git merges — Whenever you have a merge commit in git (a commit with more than one ancestor), you create an instance of non-linearity where git-svn will preserve changes but loose commit history. There may be cases where git-svn will simply break (!?)
4. sub-modules — git sub-modules (which point to remote git repositories) cannot be mapped onto svn:externals (which point to remote subversion repositories), so there would need to be a mechanism that could either fetch and commit the remote sub-module, or
5. changing history — in git it’s very possible to change the history of a remote repository. Subversion would get really pissed and the whole mechanism would probably go to hell.

Hairy, no?

Non-linearity and git-svn

Above it was mentioned that git-svn will attempt to collapse non-linear histories in git into something linear in subversion. A concrete example of this in action follows: This is your commit history in git:

This is your git commit history once dcommitted via git-svn

When looking at the commits, changes 9197078, 7a1ecfa, and d49b6d4 in the former non-linear history are all combined into 4090f22 once they’ve been routed through git-svn.

Now What?

I’ve implemented a Subversion Primary system myself, with great success, and I’ve seen the “periodic snapshot” approach used effectively to implement a Dumb Subversion Backup. While it seems like one could go further to backup more information from git incrementally into subversion, I fear the resultant system might be too brittle and high maintenance to make any sense. What do you think? Have you ever seen an automatic git –> subversion backup that didn’t suck?

(photo cred – williac)

Looking Back — Goodbye Yahoo!

Six years ago, when the little startup I worked for was swallowed up by a big purple cow, I was convinced that I would be gone just as soon as possible — I had a deep skepticism of large companies and more generally any large organization of people. There is no way I would have been able to predict that half a decade later I would still be at Yahoo!, and more interestingly, that I would be so proud of the colleagues and friends I’d gained and the accomplishments we shared.

Specifically, working on BrowserPlus has been an amazing experience. With a small team of wonderfully adept people supported by a diverse network of folks all over the company, we realized an idea, scaled it to tens of millions of users speaking over 30 different languages, and figured out how to surmount any obstacle that came our way: from managing internal company dynamics — to dealing with (not so) subtle changes in the behavior of the browsers upon which we built — to tough design challenges which raised legal, privacy, security, and engineering issues. This project continues to be technically interesting because it lies at the intersection of many different disciplines, and I believe it is important in that it demonstrates a real opportunity to accelerate the development of the web platform by making it much more inclusive and organic.

While these couple weeks at Yahoo! will be my last, I leave with nothing but love for the people I’ve met at Yahoo!, and have enormous hope for the future of BrowserPlus.

Looking Inward…

Recently I decided it was time to find a new mountain. A decision which led to several months wondering what would be the most meaningful and rewarding thing I could do with myself professionally. Having spent all that time sitting and thinking, I didn’t emerge with a clear answer, however I did come to a couple conclusions:

1. I’m most happy when I’m able to create stuff with beauty and utility (and I’ll often trade some units of the former to achieve more of the latter).
2. There’s nothing quite as fun as doing #1 with other people.
3.  #2 becomes a lot more likely when you give the stuff in #1 away.
4. An exception to #2 might be when other people themselves are empowered to do #1 because of stuff you’ve made.

Decrypted and rearranged, what that really means is that I’m interested in spending more time creating. Creating open source platforms, tools, and toys which allow me to work with, learn from, and empower folks. That’s all. Why create? Well _why once said it well:

When you don’t create things, you become defined by your tastes rather than ability. Your tastes only narrow and exclude people. so create.

— Why the Lucky Stiff

Looking Forward — ‘lo liz

When an opportunity popped up to join Mozilla Labs I was instantly intrigued. The mission of mozilla resonates well with my recipe for happiness, they believe that “openness, innovation, and opportunity are key to the continued health of the Internet”. Openness? That’s #2 and #3 above. Innovation? #1. And opportunity? Clearly, #4. Heeey, will ya look at that? We’re a perfect match! Further, the focus and structure of the labs group seems like it matches my natural way of being — keep several experimental projects of different scope going simultaneously, pursue them for some number of months, collaborate constantly, learn, and move on.

But what’s so much more important to me than how well I feel the organization I’m joining fits me, are the people. I’ve been talking to and learning from members of the labs team and mozilla from a distance for a while now, and am jazzed at the opportunity to finally get to play with them directly.

So with that, I’m off to buy a bigger saddle.

1. Has more footnotes than any other implementation. Evar.

(screencap from pluploader)

(originally posted on the Yahoo Developer Network)

In recent years, we've seen increased energy put into web extensibility platforms. These platforms let distributed developers collaborate to produce new kinds of interactive features on websites and in the web browser itself. Because these platforms frequently enable data-sharing between multiple distinct organizations, and often sit between two completely different security domains (desktop vs. web), the security and privacy issues that arise are complex and interesting. This post explores some of that complexity: both the current state of platforms that extend the web and their associated security challenges.

Plugins, Extensions, and Mashups: A Primer

The extension platforms of the web today exist at every level of the technology stack. First there are mashup platforms, which include Facebook Apps, Google Gadgets, and YAP Apps, built on Yahoo!'s Application Platform. All these platforms are abstractions that allow developers to embed content within a host site. Often this content has a level of dynamism and interactivity that exceeds what is expected of "content" — so they're called "Applications" or "Gadgets."

A deeper type of extensibility occurs in web replacement technologies. Examples include JavaFX, Adobe Flash, Microsoft Silverlight, and Google Chrome Frame¹. These technologies leverage exisiting hooks in browsers to replace native rendering and scripting technologies from the browser vendors with new environments that claim to offer a variety of benefits.

In addition to web replacement technologies, there's the related area of web augmentation technologies. These use hooks exposed by browser vendors to bolt on native code, however, unlike replacement technologies, these tools attempt to expose new scriptable facilities to browser-based JavaScript. (Google) Gears, (Yahoo!) BrowserPlus, Mozilla Geode², and Loki are some prominent recent examples of such technologies. At the fringe of this category, there are several projects that abstract different replacement and augmentation technologies to provide feature focused abstractions. Prominent examples include include SVG Web for cross browser SVG support and the YUI Uploader for in-browser content upload with in-page feedback.

The existence of both replacement technologies and augmentation technologies for the web is made possible by the browser vendors themselves, who generally expose two different ways of extending the browser.

Browser plugins were originally intended to allow 3rd parties to add support to the browser so as to handle new types of content. They have since evolved into a rich means of embedding scriptable code into browsers, code that may (optionally) render in the browser's content area. (Microsoft calls these plugins Content Extensions, while other browser vendors have converged on the NPAPI architecture). A second, much more fragmented means of extending modern browsers is to write a browser extension (referred to as a "Browser Helper Object" by our friends in Redmond). Extensions typically have browser lifetime, as opposed to page lifetime. They also have programmatic access to manipulate the user interface of the browser.

In terms of Browser Extension environments, the most recent developments include Mozilla JetPack and Google Chrome Extensions. Both JetPack and Chrome employ web technologies to broaden their target developer audience. In Chrome, an extension is "essentially [a] web page", while with JetPack, "anyone who knows HTML, CSS, and JavaScript" can create an extension. Because of the large numbers of distributed developers authoring software which teeters between the world of unrestricted code running on your desktop and the sandboxed code running in your browser, these projects raise unique security issues. Our own work with BrowserPlus addresses a closely related set of challenges and will be used throughout the article as a point of comparison.

Sandboxing vs. Code Review

In designing an extension system where developers contribute code, there's a key upfront decision that has a very deep effect on the platform — To what extent should plugins be trusted? Should plugin authors be forced to attain the approval of some body of reviewers, or do we instead rely on a software sandbox to mitigate the potential harm that could be done by untrusted code from unknown authors (and hence relax the review requirement)? The former approach can minimize upfront investment in platform development, while the latter can reduce delays in publishing 3rd party code.

In addition to increasing complexity, the sandbox-based approach constrains the creativity of developers who use the platform. The universe of what is possible is pre-defined by the platform authors, which is contrary to the idea of an extension system. Thanks to Atul Varma's recent overview of JetPack development, I now can identify this tension as Jonathan Zittrain's "Generative Dilemma".

At first glance, the platform developer might feel stuck with a binary decision between generativity vs. self service (the ability of plugin authors to publish immediately without oversight or review). A little more thought, however, reveals that in reality there's a spectrum with several interesting intermediate choices. The first hybrid to consider: a sandbox with a set of additional capabilities or permissions that may be requested by code running therein.

Building a Capable Sandbox

In many ways, browser extensions can be considered conceptual peers of the browser: They share direct access to the end user and to the resources of the machine within which they run. Additionally, given the wide target audience of modern extension systems, developers with widely varied levels of experience and grasp of security issues are empowered to author and quickly publish extensions. Providing extensions with system access to permit generativity creates a tension with the requirement that extensions run in a locked-down environment to preserve reviewless publishing and prevent accidental or malicious end-user compromise. We seem to be converging on a capability-based security model, where sandboxed code may explicitly or implicitly³ request enhanced "capabilities" or "permissions." Several of the projects mentioned here have mechanisms which allow an extension to express its required permissions:

1. Chrome extensions require explicit expression of capabilities (or "permissions") in a manifest.
2. In JetPack, it may eventually be possible to automatically determine the capabilities required by a given JetPack, based on the dependency graph of "superpowers" it uses (see below)
3. In BrowserPlus, as in Chrome, permissions are explicitly stated in a manifest.

As different projects attempt to build capabilities-based systems, an interesting opportunity for collaboration arises. Users are going to learn to some extent what "writing to a temporary location on your disk" means. They will also learn more about "using location," and maybe also about "capturing video using a webcam." As new platforms emerge that ask the user more and more questions, it would be beneficial if these platforms shared similar capability lists, perhaps even a similar language or visual vocabulary.

From Capabilities to Meaningful Decisions

While we all seem to agree that an enumerable list of capabilities is a prerequisite to intelligent interface design, it's not clear that we've yet determined how to consolidate this information in a user interface that allows meaningful decisions. This is not for a lack of thought: Over a year ago, Dion Almaer suggested a "nuanced question" combined with good post-decision feedback. Since then, he's summarized ideas from Mozilla folks regarding a "layer [of] social trust on top of the technical security." More recently, there's been some initial discussion of a stop-light style representation of risk, which if correctly applied could distill a complex decision into a series of simple questions, (for example, Do you trust yahoo.com?) which iteratively becomes more ominous as the stakes (and the implied risk) get higher.

Better Responses Through Fewer Questions

One approach employed by BrowserPlus attempts to minimize explicit prompting and leverages mechanisms of implicit consent instead ⁴ by carefully crafting the API between trusted and untrusted code. One such example is file selection, where the act of navigating an operating system-supplied file picker dialog indicates implicitly that the site should be allowed to read the contents of selected file(s). Another example where this technique might be applied involves webcam access. Rather than asking the user if a site may use the webcam, pop up a window displaying the view from the webcam, allowing the user to capture a picture inside that frame (outside the control of the page), and finally after capturing "share" the capture with the page.

Architecting APIs and UI in this way can somewhat alleviate the need for explicit up-front questions and allow in-context kinesthetic decision making to occur. This approach is not without fault however, as it can also confine the freedom of UI designers. Eliminating needless questions assumes that only by asking fewer questions of higher quality, will we truly empower the user to make meaningful decisions.

Permission Prompts in Practice

In addition to ongoing thought experiments in meaningful user prompting, there are plenty of instances in practice where user questions have been built to varying degrees of success.

Facebook combines "social trust" (the rating) with a generic bit of language about how an app can poke at your stuff:

My Yahoo! takes a language-laden approach to the problem:

BrowserPlus uses a lot of screen real estate to convey a fairly literal translation of capabilities into human language:

Google Gears adds a visual representation of a capability and a prominent display of who is asking:

Language Choice and the Generativity Continuum

The choice of implementation language adds another dimension to the problem of security in web extension systems. As might be expected, native code (C, C++, or Objective C) will always afford maximal access to the system, yielding the ultimate in generativity. Certain features cannot be accessed from a higher level language — some good examples include access to physical position information from a motion sensor⁵ or access to a wifi card to extract nearby wifi base stations with associated signal strength. The benefit to working at such a low level is, in turn, this freedom. Ultimately, attempts to introduce sandboxing into an extension system that supports native code run the risk of introducing extra developer facing complexity without giving much in return⁶. In general, the choice of implementation language can bound where an extension system falls on the generativity continuum:

Languages like Lua or JavaScript land on the other side of the extreme and apply well to a system where automatic publishing is important. In the case of JavaScript, there is no common defined standard library⁷, so this makes it possible to introduce new protected file system APIs (for instance) without surprising a developer. This same reasoning is what places Ruby and Python in the middle of the continuum. While it would be possible to embed Ruby and shoehorn in a capability-enforced access system (or perhaps use a modified version of the "Safe" mechanism that already exists in Ruby), it would yield a system sufficiently different from expectations that it might feel clumsy or confusing to developers.

While none of these claims are without exception, it is true that the two newest browser extension platforms today are built upon JavaScript and are sandboxed and capabilities-based. Older plugin and extension systems tend to leverage native code and either use code review or don't even attempt to address the issue of security in any granular way.

Case Studies in Extension Security

So far we've enumerated several pertinent decisions that extension platform authors must make:

• Are submissions sandboxed or reviewed?
• What language are extensions written in?
• Is the system capability based? If so, how is this implemented?

As it turns out, the two newest browser extension platforms avoid straight answers to these questions. Given the tradeoffs with each decision, both systems are hybrids that attempt to capture the benefits of reviewless publishing, yet still preserve the generativity of native code. They leverage JavaScript to provide an authoring environment that's accessible to a majority of the world's developers, but allow the sprinkling of native code in controlled ways to preserve generativity. It's deeply interesting to look at how these platforms straddle some of the trickiest questions.

Google Chrome's Extensions

Aaron Boodman, working on extensions for Google Chrome, recently spoke about where Chrome can be found on the generativity continuum. Chrome uses a hybrid approach where "webby" extensions may optionally include native code bundled as scriptable NPAPI plugins. Extensions which are based solely on the Chrome extension environment's JavaScript APIs need no review and are "enabled automatically." Extensions that include unsandboxed native code are subject to additional review. In Aaron's words:

We're really proud of the fact that in the Google Chrome extension gallery we enable extensions automatically, there's no review period. But we make an exception for NPAPI because when you get to native code a lot of the security mechanisms built into the extension system can no longer apply. Once you have native code running it can modify the registry or make permanent modifications to your system... So we have additional review for NPAPI extensions.

This strategy seems reasonable. One would hope that a lion's share of extensions are developed without reliance on native code. The set of extensions that bundle native code can then be periodically examined. Those which leverage native code to do things that may be interesting to a large number of extensions developers and can be considered for uplifting into the core platform.

Mozilla's JetPack

One interesting thing that the Chrome model lacks is a means for sharing privileged functionality between extensions without actually merging it into the core extension platform. JetPack, on the other hand, is taking a much more ambitious initial approach to building a platform that can more rapidly evolve. Atul recently summarized the design goals of JetPack's capability-based security model. They aim to "allow anyone to create capabilities that securely expose privileged functionality." The JetPack world might end up looking something like this:

The key difference from Chrome extensions is the existence of a middle tier consisting of community contributed bundles. These provide JavaScript APIs to access system resources not otherwise exposed by the core extension system. By breaking out potential platform features into a separate layer you empower the community to not only imply the features that they'd like to see in the plugin platform, but to go and build them. This is an exciting idea. It will be interesting to see how this middle tier of superpowers is embraced by the community of extension developers, as well as to learn the details of how the review process is structured.

Today's Extensions as Tomorrow's Web

Web extensions today are a dynamic and fascinating area at the intersection of several disciplines. This post attempts to give a taste of some of the considerations and tradeoffs involved in the architecture of such systems. Hopefully I've sparked your interest to learn more or perhaps even join the fray. A renewed interest in more open software platforms is no surprise, and the potential to harness the creativity of developers of the world is inspiring.

One area, however, that dampens the joy of web extension platforms covered here is that they're limited in scope — vendor locked and browser specific. While we'll hopefully see increased (and appropriate) cross-vendor collaboration, it's reasonable to conclude that because "browser extensions" are about extending web browsers — and because different web browsers can have vastly different architectures — browser extension APIs will be slow to converge.

Imagine, however, an extension system which struck a balance between simplicity, generativity, and security, and was pertinent to the whole darn web. This is the promise of Web Augmentation technologies. Distributed Extensibility has been a topic of discussion in terms of HTML5 for a while now, but the conversation thus far has failed to cover a distributed way to explore new standard JavaScript APIs (the stuff that so many emergent HTML5 standards are made of). A significant obstacle to this conversation is the problem of helping users understand and securly navigate a dynamically evolving web.

The current work being done to secure extension platforms logically extends to web augmentation platforms. The degree to which we can collectively refine these permissioning models may become the key limiting factor in how far we can migrate our current desktop experience to the web.

Lloyd Hilaiel
BrowserPlus Hacker

A graphical pontification on how the web actually works...

Recently I proposed orderly, an idea for a small microlanguage on top of JSONSchema — something easier to read and write.

There’s been some great feedback which I find encouraging. In response I’ve set up orderly-json.org and started a project on github which will host the specification, the reference implementation, and all of the contents of the json-orderly.org site.

recent changes

Given initial feedback and a bunch more thought about orderly, theres a couple significant early changes that have taken place.

repetition quantifiers get curly, enumeration values go square

in v-1 (yes, negative one) the following code would have specified an integer property with a range of allowable values:

integer foo[0,10];

And the following, would specify an integer property @bar@ with allowable values 0-10.

integer bar[0,10]

uh. oops? We need a way outta this mess. The current v0 specification moves range specification to after the type (suggested by Toby A Inkster, and changes syntax from square braces to curly braces. We’re re-using “repetition quantifiers” from regular expressions here, and while the analogy isn’t perfect, nor is the square brace analogy where storage allocation is the expectation. Enumeration values, in turn, have more exclusive access to square braces, which is nice because this is the JSON representation of arrays. Here’s an example of a property @weight@ which must be in a certain numeric range:

number{0.02, 0.98} weight;

While possible values are represented after the type using JSON style array notation:

string os ["osx", "win32", "linux", "freebsd"];

An enumeration property on JSONSchema is nothing more than an array of potential values. And a more beautiful seque into the next point, I couldn’t have engineered..

Orderly syntax is a superset of JSON

JSON is wholly contained within the syntax of orderly, and the reason for this is simple. Because orderly must be able to represent default values of any object of arbitrary complexity. The bad news is this means it’s going to be a little more work to parse, but the good news is there’s plenty of precedent to leverages (numerical representation, etc).

What’s next?

As the language starts to firm up, I’ve resolved to take popular APIs and try to describe them with orderly, see how far we get. Today’s target is the BrowserPlus JSON WSAPI that returns a list of available services, here’s a subset of the data we’re trying to describe:

[
  {
    "name": "PublishSubscribe",
    "versionString": "1.0.0",
    "os": "ind",
    "size": 5114,
    "documentation": "A cross document message service that allows JavaScript to send and receive messages between web pages within one or more browsers (cross document + cross process).",
    "CoreletType": "dependent",
    "CoreletRequires": {
      "Name": "RubyInterpreter",
      "Version": "4",
      "Minversion": "4.2.5"
    }
  },  
  {
    "name": "Uploader",
    "versionString": "3.2.6",
    "os": "osx",
    "size": 279378,
    "documentation": "This service lets you upload files faster and easier than before.",
    "CoreletType": "standalone",
    "CoreletAPIVersion": 4
  },
  {
    "name": "RubyInterpreter",
    "versionString": "4.2.6",
    "os": "osx",
    "size": 1691095,
    "documentation": "Allows other services to be written in Ruby.",
    "CoreletType": "provider",
    "CoreletAPIVersion": 4
  }
]

And here’s concocted orderly to describe this data:

# A schema describing the data returned from the BrowserPlus services
# API at http://browserplus.yahoo.com/api/v3/corelets/osx
array {
  object {
    string name;
    string versionString;
    string os [ "ind", "osx", "win32" ];
    integer size;
    string documentation;
    string CoreletType [ "standalone", "dependent", "provider" ];
    # if CoreletType is "standalone" or "provider", then
    # CoreletAPIVersion must be present
    integer CoreletAPIVersion ?;
    # if CoreletType is "dependent", then CoreletRequires must be present
    object {
      string Name;
      string Version;
      string Minversion;
    } CoreletRequires ?;
  };
};

Not bad. Some semantics of this ad-hoc WSAPI cannot be captured in orderly nor it seems the JSONSchema underneath, but I think I’m ok with that. You?

lloyd