Yahoo Query Language and Open Tables

I've been looking at the Yahoo Open Data Tables and Query Language documentation. This is truly amazing stuff! It provides a service API that accesses many well known data sources (many are Yahoo) and transforms the data into XML or JSON. The data sources can be external URLs that provide XML and Yahoo does the fetch, parse, extract and transform that you want. You can provide a definition of some other external data source and they will hook it into their unified API fetch/query/transform service.

Some of the data sources are Flickr, local listings, geo location info, web search, image search, news search, weather and so on. One stop shopping for lots of great data.

Their console http://developer.yahoo.com/yql/console/ is a great way to see what's possible.

This is what I've wanted for many years. A long time ago I wanted to build a service that would provide "XML data sources" (I even registered xmldatasource.com) for everything available on the Web - now it looks like Yahoo has actually done it.

Let's hope they keep this data access service open to all.

WebHooks

This looks interesting - in a 'teach people how the Web really works' kind of way. WebHooks is a catch phrase for Web application development where notification are sent from theh source to the listener via HTTP POST, rather than the other way around via polling (which, as some have said, doesn't scale).

Somewhat related to my earlier post on how HTTP can be used as an alternate to XMPP/Jabber in a publish/subscribe scenario without too much problem or angst.

REST and pub/sub

It's unfortunate that technologists continue to propagate serious mistakes like "[...] its also clear that REST and its inherent polling mechanism isn't the best way of building a user notification system [...]"

REST is about state transfer - and event notifications are also state transfer.
As for HTTP, it isn't only "polling" - anyone that has posted a blog entry knows that. The 'client' can 'post' updates to the 'server' - exactly the same as event notifications via XMPP. The great thing about XMPP is the federated multi-hop capability with 'trust' built-in. Just like email, only with everyone using settings for very low latency delivery.

There have been multiple publish/subscribe over HTTP mechanism (comet, mod_pubsub, KnowNow, etc) over the years.

Oh, the irony of shallow WSDL

I'm looking into the API for the Hi5 social network and unfortunately found some WSDL. They also have a REST API, but it's documentation appears auto-generated from WSDL that nobody actually filled in. Somewhat useless.

Normally I wouldn't post about WSDL, but I couldn't pass up the irony of the WSDL documentation for the authentication service API having an HTML form describing how to authenticate the user. If that's not irony, I don't know what is.

More REST and Erlang

Man, I wish I had the time to dig into these posts about Erlang, Yaws and RESTful services.

Cross domain web apps

This post about cross domain Web apps is spot on. The growth of the current Web was due to millions of silos - no data shared between sites. Over the past few years technology like iframes, ajax and other styles of 'mashups' have made it possible to create applications composed from cross-domain services. I believe this will drive the next major growth curve of the Web.

Steve Vinoski’s Blog

Steve Vinoski is one cool dude. I've always had the utmost respect for IONA and even after leaving a while back, Steve continues to show the attitude of professional engineering that garnered that respect. With his measured explanation of his view of REST and Dare's theory and practice post I think I'm ready to walk away from cooling embers of the dying REST .vs. SOAP flame war. Time to unsubscribe from rest-discuss.

From Steve's blog, here are some good quotes:

People seem to get really upset when I say that the static typing benefits of popular imperative languages are greatly exaggerated, and when I say that developing real, working systems in dynamic languages is not only possible, it’s preferable.

Either way, no interface definition language is ever going to keep you or some other real live person from having to figure out what the service actually does and how to actually use it, and then coding your client accordingly.

Remember, REST is an example of applying well-chosen constraints to achieve desired architectural properties for a broad class of distributed systems, and so that’s what its constraints are all about.

Either way, anti-REST folks commonly claim that REST’s success is due only to the fact that there’s a human-driven browser in the mix, but that’s one of the dumbest things I’ve ever heard.

Having a solid thinker like Steve Vinoski blogging makes the Web a better place. Can't wait to hear the details of what he's working on.

Nielsen on Generic Commands

I ran across this interesting bit from Jakob Nielsen's UseIt site about user interface design. It's about using the same few commands in a UI and is an interesting twin to the 'uniform interface' aspect of REST. Although a 'user interface' and a 'network interface' are very different beasts I can't help but wonder if there's some fundamental reason for the similarity.

Summary: Applications can give users access to a richer feature set by using the same few commands to achieve many related functions.

In application design, there's a tension between power and simplicity: Users want the ability to get a lot done, but they don't want to take the time to learn lots of complicated features.

OpenSocial - what? no logo?

Everybody has been commenting on the news of the week - Google and MySpace spinning out their own system for applets within social networks called OpenSocial.

It seems everyone has missed the biggest gap in the OpenSocial system - they have no logo! Rather than talk about the disappointing lack of open content in their 'extension' to Atom for profile data (anyone heard of hCard or FOAF?) or the lack of anything innovative like client-side includes (that might actually lead to social networks being spidered by a big search engine) I've decided to contribute my l33t grafx skillz to the community in the form of an OpenSocial logo. It includes the mandatory Web 2.0/startup color scheme of blue and orange. I need to figure out how to include the RSS radio waves into the design somehow...

or this...

Next, a motto : "Where do I belong today?"

HTTP errors - a photoset on Flickr

(From BB)

HTTP - a protocol so simple, even a cartoonist can understand it. (and why haven't you bought a shirt yet?)

404 Arrgghh - (insert pirate code joke here)

Object serialization raises it's insidious head

Funny post from Dare Obasanjo decrying the horrible state of affairs created by the company he works for.

This part is about object serialization and version incompatibility.

However, the insidious thing is that the failure wasn’t because their application was improperly coded to fail if it saw a fruit it didn’t know, it was because the platform they built on was statically typed. Specifically, the Web Services platform automatically converted the XML to objects by looking at our WSDL file (i.e. the interface definition language which stated up front which types are returned by our service) . So this meant that any time new types were added to our service, our WSDL file would be updated and any application invoking our service which was built on a Web services platform that performed such XML<->object mapping and was statically typed would need to be recompiled. Yes, recompiled.

Insidious indeed! Who in their right mind could have predicted such things!

And this...

It’s sad that as an industry we built a technology on an eXtensible Markup Language (XML) and our first instinct was to make it as inflexible as technology that is two decades old which was never meant to scale to a global network like the World Wide Web.

Who's this industry of which you speak, Kemosabe?

Getting all meta

Dare Obasanjo seemingly always has time to create detailed and thoughtful posts, I've gone from being annoyed at early posts to reading them regularly.
His latest two posts are worth reading as well, but I can't help but comment on his strawman characterization of ReST.

I should probably start out by pointing out that the title of this post is a lie. By definition, RESTful protocols can not be truly SQL-like because they depend on Uniform Resource Identifiers (URIs aka URLs) for identifying resources. URIs on the Web are really just URLs and URLs are really just hierarchical paths to a particular resource similar to the paths on your local file system (e.g. /users/mark/bobapples, A:\Temp\car.jpeg). Fundamentally URIs identify a single resource or aset of resources. On the other hand, SQL is primarily about dealing with relational data which meansyou write queries that span multiple tables (i.e. resources). A syntax for addressing single resources (i.e. URLs/URIs) is fundamentally incompatible with a query language that operates over multiple resources. This was one ofthe primary reasons the W3C created XQuery even though we already had XPath.

That said, being able to perform sorting, filtering, and aggregate operations over a single set of resources via a URI is extremely useful and is a fundamental aspect of the Web today. As Sam Ruby points out in his blog post Etymology, a search results page is fundamentally RESTful even though its URI identifies a query as opposed to a specific resource or set of resources [although you could get meta and say it identifies the set of resources that meet your search criteria].

I don't think there's any harm in the mis-characterization of ReST and the detail of the post about the two query systems from Google and Microsoft is rather interesting, but I was surprised at his misunderstanding since I thought he knew better. Dare does have an out - as any good program manager does - by saying "... you could get meta...", so I will take that opportunity and get meta.

• "URIs on the Web are really just URLs and URLs are really just hierarchical paths to a particular resource [...]" - no, they are not really just hierarchical paths. They are an identifier. Sometimes they are a 'path', but usually not.
  
• "Fundamentally URIs identify a single resource or a set of resources." - always just a single resource.
  
• "A syntax for addressing single resources (i.e. URLs/URIs) is fundamentally incompatible with a query language that operates over multiple resources." - this is core missed direction in the explanation of ReST. The term 'resource' as applied within RDBMS isn't the same as 'resource' as applied within ReST. A ReST 'resource' is the view from the outside, and datbase 'resources' are the data elements on the inside. Read up on Data on the Outside vs. Data on the Inside by Pat Helland (not exactly relevant to this post, but a catchy title).
  
• "a search results page is fundamentally RESTful even though its URI identifies a query as opposed to a specific resource or set of resources [although you could get meta and say it identifies the set of resources that meet your search criteria]." - the URI actually does identify the results (as Dare suggests) and not the query. The URI is the query. This means you can operate on the set of data that are the results - you could delete them, retrieve them, replace them, etc.
  

I'm not suggesting that URI are better than or a replacement for queries, just that they are different. They do different things and are both very useful, and when you think in terms of "a URI is a query" you think of the processing of the implementation and miss out on some aspects of Web architecture which naturally arise when you think of "a URI identifies the results".

REST - it's inevitable

A couple weeks ago I was having lunch with a friend at Amazon - who coincidentally used to work at MS with the data access team and is a brilliant architect and engineer - and we talked about REST and how he was helping use REST concepts in refactoring some core back-end services. I made the observation that I was never stressed about how long it has taken for REST to achieve common industry understanding and acceptance - I just said "It's inevitable". I didn't realize just how short it would be for inevitable to show up.

It looks like REST has taken Redmond by storm. First I read that Mark Baker did some consulting with Microsoft (I missed the chance to have dinner with him when he was in town due to email snafu - major bummer). Then I read Dare Obasanjo's post that says "REST is totally sweeping Microsoft."

We passed the tipping point quite a while back, but it's still good to see pragmatic architectural sensibilities take root finally.

Amazon.com Widgets and hypertext

It looks like Typepad and Amazon are collaborating to help bloggers link to Amazon products. They have three widgets outlined, and one of them - the quick linker widget - uses custom attributes on an HTML anchor tag to make it easier to reference a set of products. The 'old school' would just use a URI, but those are hard to construct, hard to type and the wizards slow people down, blah blah blah. This not-quite-micro-formats approach is more understandable and more forgiving for hand-crafted markup. They define a new 'type' attribute for the anchor with the value "amzn". Then there are several more attributes like 'search' or 'category' or you can create a direct link via an 'asin' attribute. I assume some snippet of javascript would scan the page after it was loaded and construct the URI on the fly and set the href attribute on these anchors.
This is a creative solution to the "how do you construct a URI" problem, but it does leave spiders out in the cold, breaking the hyperlinking which defines the Web.

However, there is a simple approach they could use which would make a fully declarative and locally described document work and continue to allow auto-discover of hyperlinks to work - add a 'meta' tag to the head of the document with the URI template that corresponds to the 'type' attribute. I think the URI template proposal may need to do a bit of work related to optional or conditional patterns, or the URI template could stay static and the type="amzn" could change to be type="amzn-direct-link" or some other more qualified value.

Social Content Services and REST

This article from Leigh Dodds about Connecting Social Content Services is an extremely well written summary of several social content services, a greatly pragmatic description of REST and an extremely useful review of the service APIs from those social content services.

If you need a quick list of aspects for engineers to consider when building a ReST based system (the list is derived from Joe Gregorio's site), try this:

• How does the API use URLs to identify resources?
• What HTTP methods does the API support?
• What status codes are returned?
• How are users and applications authenticated to the API?
• Are resources linked to one another (use of hypermedia)?
• What data formats does the API support?
  

This review notes that nearly all services do not use hypermedia which I think is unfortunate but understandable. I've always had a problem resolving the desire to be flexible in allowing the internal data identifiers to be used in many situations and the desire to be trivally easy for clients to access other resources by simply using links - the mashup problem. One issue I have is that the server-side software that generates the representation might not know all the possible resources made available by sibling services. Think of a US postal zip-code - if you have a service that provides weather based on zip code, should that representation also be responsible for linking to all other services - either provided by your system or some other server - that could potentially take in a zip-code? My approach is to return both direct links to known resources (tagged appropriately) as well as the short-form of the identifier, the plain zip-code for example. Microformats sort of do this, but it's a style that isn't well applied by data services.

One comment I have on this review of RESTful services is the author mistakes "URL" for only the portion before the query terms. This results in judging these services to be less RESTful than they actually are. Use of query terms does not mean there is one 'controller' with parameters, it actually increases the number of addressable resources which is good.

37Signals and Google Web Accelerator

From the rest-discuss list, here's a post from 37 Signals that missing the point about the Google Web Accelerator fetching URLs.

This wouldn’t be much of a problem on the public web since it’s pretty tough to be destructive on public web pages, but web apps, with their admin links here and there, can be considerably damaged. If you have a web app, it might be worth returning a 403 when the HTTP_X_MOZ is set to “prefetch” header is sent. This will keep Web Accelerator from clicking destructive links.

I like this "destructive links". A major point of the Web is that links are not destructive.

I think the better approach is to not use GET to modify or remove data. That's simply unnecessary and against the word and spirit of HTTP.

Remember, a link is not a widget. You can't use a simple anchor element as if it were the same as a push button. Whenever an HTML page has an anchor tag, it is explicitly advertising that the URL is safe to retrieve. By sprinkling your HTML with these landmines, it's the author that made the mistake, not the browser or web accelerator.

If you are writing web apps, it might be worth more to think about what's happening and why, rather than hack a workaround that only occasionally avoids a bug introduced by your own application. Fix the bug, and no workaround is necessary. Again - a link is not a widget.

Benjamin's REST Tutorial

This is a really good engineering document that gives pragmatic design and implementation guidance for building RESTful systems: Benjamins REST Tutorial:
My favorite quote:

"You are starting to see network effects as programs with similar schemas and content types can 'just talk to each other'. They are running out of things to disagree on, and are running out of software that needs to be written to support the differences."

Norm on Names

From a post from Mark Baker, there's a fine post from Norm Walsh on names that you all should read - my favorite quote:

Contrary to what you may believe, there is nothing about the “http” URI scheme that requires use of the “http” protocol. And even where the “http” protocol is used, there's nothing about it that requires access to any particular machine.

On your desktop, your web browser may return things from its cache without ever hitting the web.

Erlang - the next buzz in web conversations

This post by Patrick Logan points to another post talking about Erlang and scalability. I've been hearing a lot about Erlang from the REST community recently and I've been thinking about all the things I could build with yet another web server that supposedly supports over 60,000 concurrent connections.

RESTful Interface Description

The other day, a friend from Oz asked about IDL (interface definition language) for RESTful services (it turns out WSDL 2.0 has binding definitions that support all HTTP methods) and now there is this post from Phil Windley Crying Out for a RESTful Service Interface Description Language:

The only way that we’ll get to a place where Web 2.0 apps are more easily integrated is when we have a service interface description language and other metadata standards for RESTful services.

At the top of the post he links to another blog by Dave Rosenberg that says in part "To me the big opportunity of Web 2.0 development is the ability to create a better user experience based on features etc."

Following Phil's post is a comment starting with "What we need is a [...]".

Here's a suggestion - pick two services you would like to integrate, something that would result in real tangible value that you would actually use day-to-day, and try actually implementing these difficult integrations (and some are tricky) and write about the effort and the problems. That real effort with real implementation will surface the real problems that need solving. Crying out "you should do this" or "people need that" is just so much wishful theory talking.

In theory there is no difference between theory and practice. In practice there is.