The Spreadsheet as Mashup Fabric

The title of Joe Gregorio's recent post - The Spreadsheet as Mashup Fabric - really caught my eye. For the past two days I've been thinking about the spreadsheet as a simplifying tool for building event driven web applications - accepting events, summarizing many messages, routing and republishing the results to a new set of subscribers.

I've just started thinking about this so I don't have any actual situations it might be useful for, but the idea of opening up a compact, rich and information dense document in a very responsive interactive application for ad-hoc applications and messaging components sounds interesting.

Sort of like message-driven beans on acid.

What's old is new again

Check out this short article from 1998 on making more responsive web applications using - get this - javascript.

We can create a more responsive experience if we look at the available technology in a different way. In particular, we are going to forget the assumption that a Web page is loaded directly from a Web server. Instead, we are going to use the request/response structure of the Web in a slightly different way.

I wonder how far back we can see this approach being used or clearly talked about.

37Signals and Google Web Accelerator

From the rest-discuss list, here's a post from 37 Signals that missing the point about the Google Web Accelerator fetching URLs.

This wouldn’t be much of a problem on the public web since it’s pretty tough to be destructive on public web pages, but web apps, with their admin links here and there, can be considerably damaged. If you have a web app, it might be worth returning a 403 when the HTTP_X_MOZ is set to “prefetch” header is sent. This will keep Web Accelerator from clicking destructive links.

I like this "destructive links". A major point of the Web is that links are not destructive.

I think the better approach is to not use GET to modify or remove data. That's simply unnecessary and against the word and spirit of HTTP.

Remember, a link is not a widget. You can't use a simple anchor element as if it were the same as a push button. Whenever an HTML page has an anchor tag, it is explicitly advertising that the URL is safe to retrieve. By sprinkling your HTML with these landmines, it's the author that made the mistake, not the browser or web accelerator.

If you are writing web apps, it might be worth more to think about what's happening and why, rather than hack a workaround that only occasionally avoids a bug introduced by your own application. Fix the bug, and no workaround is necessary. Again - a link is not a widget.

New Direction is new theme for Democratic plan

Is it just me or does the phrase
'New Direction' sound just like 'nude erection'?

I have a bad feeling about this...

Participation Inequality: Lurkers vs. Contributors in Internet Communities

A few years ago I had stopped reading Jakob Nielsen's Alertbox even though I had been reading it for a long time before then and had enjoyed his fact filled articles and analysis. Just the other day I popped back in to check out his recent postings and this one caught my eye - Participation Inequality: Lurkers vs. Contributors in Internet Communities - as I'm looking into communities, social aspects of the Web and such.

The article gives some history and data that covers the basics of a 90-9-1 rule with respect to participation - most people just lurk, some interact occasionally and a few are very active. Many people know this and if you've read Clay Shirky's "Power Laws, Weblogs and Inequality" article you'll be familiar with the idea. Although there have been comments elsewhere that Jakob doesn't "get" RSS (and his alertbox site doesn't have an RSS feed), I think Jakob does understand the implications of RSS and this post provides some value by highlighting some issues and suggesting how to design useful systems in the context of participation inequality.

Search. Search engine results pages (SERP) are mainly sorted based on how many other sites link to each destination. When 0.1% of users do most of the linking, we risk having search relevance get ever more out of whack with what's useful for the remaining 99.9% of users. Search engines need to rely more on behavioral data gathered across samples that better represent users, which is why they are building Internet access services.

Make participation a side effect. Even better, let users participate with zero effort by making their contributions a side effect of something else they're doing. For example, Amazon's "people who bought this book, bought these other books" recommendations are a side effect of people buying books. You don't have to do anything special to have your book preferences entered into the system. Will Hill coined the term read wear for this type of effect: the simple activity of reading (or using) something will "wear" it down and thus leave its marks -- just like a cookbook will automatically fall open to the recipe you prepare the most.

(Bummer - Jakob Nielsen had a 'User Experience 2006' in Seattle a few weeks ago and I missed it...but maybe I can catch some of it on a blog somewhere)

Added: Here is another good article from Jakob Nielsen about traffic log analysis.

The top 10 queries accounted for 10% of the total traffic, so each one of these queries is obviously more important than those that brought only one visitor. Taken together, however, the single-use queries accounted for three times as much traffic as the top 10 queries. This statistic shows the folly of focusing search engine optimization solely on a few high-performing queries. Your site must be found when users enter relevant queries -- and the possibilities are typically vast.

Abundance .vs. Scarcity

This is a good post by Dare Obasanjo commenting on economies of abundance .vs. scarcity with respect to the digital abundance of Internet and the scarcity of time and attention with respect to human intelligence. The Economy of Abundance and Other Fairy Tales.
I first heard the phrase 'economoy of abundance' many years ago in a story called "Manna" by Lee Correy (pseudonym of G. Harry Stine) - a neat science fiction story. I wonder where it first came into use...

Most successful Web companies today are exploiting the scarcity of attention and time that plagues all humans. In a world where there a hundred million websites the problem isn't lack of content, it is finding the right content.

WS-Notification approved by OASIS

And in unrelated news, OASIS members have approved WS-Notification v1.3

OASIS, the international standards consortium, today announced that its members have approved WS-Notification version 1.3 as an OASIS Standard, a status that signifies the highest level of ratification. WS-Notification defines a pattern-based approach for disseminating information amongst Web services.

I haven't read the relevant specifications, but should make the effort. Nah... I'll just go for a bike ride in the forest instead.

Benjamin's REST Tutorial

This is a really good engineering document that gives pragmatic design and implementation guidance for building RESTful systems: Benjamins REST Tutorial:
My favorite quote:

"You are starting to see network effects as programs with similar schemas and content types can 'just talk to each other'. They are running out of things to disagree on, and are running out of software that needs to be written to support the differences."

Taking a break from Amazon

I've decided to take a break from Amazon - tomorrow is my last day. Officially I'm taking time off as a leave of absence, but at this point I'm not sure of the odds of returning full time. However, I have talked with a few groups and there is some mighty interesting work coming up - one possibility in the Web Services group and another in a 'new business' sort of group.

The past three and a half years have been great and full of action, but that action has me pretty much burned out. I was really, really lucky to start in the group that I did - the product catalog group. After a couple years I wound up managing the team that is responsible for the primary storage of everything offered for sale on Amazon's platform - a couple hundred million listings. The reason I enjoyed this area is that while there are many things that Amazon does, one of the core areas is allowing sellers to offer thing for sale. There is both the satisfaction of scaling to huge numbers, and the satisfaction of enabling the 'little guy' to list stuff and make some money, possibly even make a small business. There are so many people that created a business or run their business based mostly on Amazon that it's both humbling and humanizing - believe me, we all take it very seriously.

The people I've interacted with at Amazon are the best I've ever worked with, and I've been at several companies over the past fifteen years. The really amazing thing about Amazon is that everyone is uniformly smart - not just engineering people, but everyone. That alone makes things so much easier.

All the Data. All the Time

Remember when nobody had a web server, commodity hardware, gigabit ethernet or scalable storage? There was (and still is) a huge industry building special purpose computing machines for special application - CAD, electronic design, video and so on.

Then some innovative folks realized that access to a hard drive would become slower than access over a network and the slide away from direct attached storage to network attached storage began.

Directly or indirectly, that shift was echoed in this year's launch of Amazon's S3 web service - storage on a dime (well, on fifteen cents).

Today I had an amazing suprise when I got home from work. My brother has been preparing to paint the exterior of our house and while in the area he goes bargain hunting, both on Craigslist and Live Expo (I'll refrain from my normal rant about local listings at this point.) In the back of his pickup was a huge black cabinet - an Auspex NetServer 2000. This is a 700GB storage server that was wicked cool back in the day. It could have two additional storage nodes attached to get up to 4.5TB - all for the measly sum of $70,000. And it's in my brothers pickup. And he snagged it for free. For free. Of course, it took three people to heft it into the truck and we have absolutely no idea what to do with it, but my gawd this is cool.

Among available NAS designs, only the Auspex 4Front NS2000 (Auspex NetServer 2000) series of content servers uses both approaches, making it the most advanced NAS product design available. In this design, each of three I/O nodes has two Intel processors. Each processor runs specialized real-time software called the DataXpress kernel. One I/O node processor, called the Network Processor (NP), manages highly reliable customized software that controls all network protocol and caching functions. The other I/O node processor, called the File and Storage Processor or FSP, handles file system processing and storage processing. This design, called Functional Multiprocessing (FMP), allows network processing for a second I/O to occur in parallel with data retrieval for the first I/O. The FMP design provides advantages over both nonparallel single processor and Symmetric Multipro-cessing (SMP) designs that handle I/O activity in serial.

This whole discussion of a processing node sounds very cool - kernels, multi-processors, parallel operations - very advanced.

If anybody wants some of that high tech magic, let me know because I have a couple hundred pounds of it in a pickup in my driveway.

Norm on Names

From a post from Mark Baker, there's a fine post from Norm Walsh on names that you all should read - my favorite quote:

Contrary to what you may believe, there is nothing about the “http” URI scheme that requires use of the “http” protocol. And even where the “http” protocol is used, there's nothing about it that requires access to any particular machine.

On your desktop, your web browser may return things from its cache without ever hitting the web.

Erlang - the next buzz in web conversations

This post by Patrick Logan points to another post talking about Erlang and scalability. I've been hearing a lot about Erlang from the REST community recently and I've been thinking about all the things I could build with yet another web server that supposedly supports over 60,000 concurrent connections.

Rock 'Em Sock 'Em Robots

Ah, the good old days of Rock 'Em Sock 'Em Robots. When I was a kid this was a favorite game and it looks like Mattel has a new re-issue of the game.

The interesting thing about this particular page on Amazon is that this game is offered by three separate sellers, and Amazon is one of them. Before today, this was not possible due to the exclusivity contract Amazon was operating under. My team was only peripherally involved with the effort to make this happen, but watching the speed and professionalism of all the teams drilling into all the details, crossing the T's and dotting the I's, really made me proud. Not only did the people here work quickly and competently, but the software platform worked as designed.

Another interesting thing about that page, the customer reviews have multiple categories of stars - durability, fun, educational and overall - I never noticed that. Those are exactly the areas I look into when reviewing toys and games. Pretty cool.

Let the Rock 'Em Sock 'Em Robots competition begin.

RESTful Interface Description

The other day, a friend from Oz asked about IDL (interface definition language) for RESTful services (it turns out WSDL 2.0 has binding definitions that support all HTTP methods) and now there is this post from Phil Windley Crying Out for a RESTful Service Interface Description Language:

The only way that we’ll get to a place where Web 2.0 apps are more easily integrated is when we have a service interface description language and other metadata standards for RESTful services.

At the top of the post he links to another blog by Dave Rosenberg that says in part "To me the big opportunity of Web 2.0 development is the ability to create a better user experience based on features etc."

Following Phil's post is a comment starting with "What we need is a [...]".

Here's a suggestion - pick two services you would like to integrate, something that would result in real tangible value that you would actually use day-to-day, and try actually implementing these difficult integrations (and some are tricky) and write about the effort and the problems. That real effort with real implementation will surface the real problems that need solving. Crying out "you should do this" or "people need that" is just so much wishful theory talking.

In theory there is no difference between theory and practice. In practice there is.

The Web Is a Pipe

From The Web Is a Pipe:

Opportunity opens up when we use HTTP to connect our server infrastructure components together. If you use HTTP between your front-end web server and your back-end application server, suddenly you gain the ability to swap Apache for Lighttpd on demand without worrying about the FastCGI bits. Your application server won’t notice the difference. You can replace either Apache or Lighttpd with Pound or Pen. You can even replace them with some sort of hardware load balancer solution if that floats your boat.

Even better, when you use HTTP as the glue, you suddenly can use a whole host of tools to probe the various parts of your application. You can use curl to probe just your application server. Or even point a browser at it, assuming that you have a clear path through your firewalls and what not. And, if you’re really l33t, you can do a manual telnet and make just the request you need—with all the right headers—to simulate exactly a problematic client request.

Exactly. Using a web server as your application server means you can replace your application server with yet another web server.

HDR Coconut

big coconut
Originally uploaded by Haiku Garry.

I stumbled across this surreal photo of a coconut and was intrigued by the tag "HDR". What is "HDR" I thought? It turns out that HDR means high dynamic range. This is more than merely 32bits per color channel per pixel, it usually means capturing the full dynamic range. Most cameras can't do that, so the trick is to take three or more exposures and blend them together - called tone mapping or exposure mapping. Flickr is full of amazingly artistic photos with rich and deep colors. Many looks not quite right, almost as if they are a bit - but only a little bit - beyond current capabilities of computer graphics lighting models.
The HDR pools on Flickr have a lot of abandoned heavy iron cars from America's motor past, many photos of solitary houses, of disorderly mechanical/industrial buildings and one photo stream of an apparently abandoned castle.

Kaboodle

I found Kaboodle through a Google search alert (for 'social network shopping'), and for some reason I think the idea is really cool. It is sort of similar to Ta-Da lists from 37signals but more of a rich media list maker - wishlists, compare product data from different sites (very useful when hunting for a digital camera), assembling details on places to go for vacation (I could have used this last year...). Lots of uses come to mind.

The color scheme is orange and blue, like all good startups, but the site still looks fresh.

S3 support Virtual Hosting

This is good news - S3 supports the Host header in HTTP requests. I had been meaning to write about the security holes is storing different people's data within the same domain - as soon as two people host javascript, then 'cross site scripting' becomes possible within one host domain. This enhancement allows folks to trivially avoid that problem (assuming people want to host HTML and Javascript on S3 - not it's advertised purpose).

ACM Interview with Werner Vogels

I finally found time to read the ACM interview with Werner Vogels about Amazon and service oriented architecture (conducted by Jim Grey, no less). He talks a bit about Amazon history and a bit about how services affect development teams as well as the runtime and operational benefits.

Here's one of several 'lessons learned' that Werner recounts:

A second lesson is probably that by prohibiting direct database access by clients, you can make scaling and reliability improvements to your service state without involving your clients.

Interesting - since a database is also a service, what is the essential difference between direct 'database' access and direct 'service' access that improves the situation?

Another interesting comment:

Other lessons are related to how you access services: If you want to be able to aggregate services easily, if you want to insert advanced infrastructure techniques such as decentralized request routing or distributed request tracking, you need a single unified service-access mechanism.

Hmmm, the single unified access mechanism sounds like a protocol. I'm in the middle of building a new service and haven't been considering this - I wonder if I'm going to get in trouble. But I guess that's okay, since my business card does say Sr. Troublemaker...


Now this is the part that I like - as it involves my stuff!

About a million small and larger businesses sell on the Amazon platform. For example, if you go to one of the book pages, you will find that item is also available new or used from some of our many partners. These can be very small independent bookshops or larger retail operators that want to sell on our platform.

And onto REST...

Do we see that customers who develop applications using AWS care about REST or SOAP? Absolutely not! A small group of REST evangelists continue to use the Amazon Web Services numbers to drive that distinction, but we find that developers really just want to build their applications using the easiest toolkit they can find. They are not interested in what goes on the wire or how request URLs get constructed; they just want to build their applications.

Amber stones in the Northwest

Amber stones
Originally uploaded by dierken.

I've always been fascinated with minerals and gems, so this last Thursday I was very excited to go on a school field trip with my son to hunt for amber near Tiger Mountain.
There is a geologigist - Geology Bob - that conducts field trips for groups and schools to hunt for rocks, crystals, fossils and other cool things. From what I could hear (my ears are plugged up from a cold), back in the 70's he discovered a field of amber stones here in western Washington, which is astounding - there are only five places in North America that have amber deposits. Because he has been involved in local geology for quite a long time, he has received a permit to go onto state park land to conduct these geology tours for educational purposes. We learned about how amber is formed - ancient tree sap sinks into a swamp, gets covered in sand and mud, is pushed down into the Earth a quarter mile for ten million years, and then somehow surfaces again. It appears there are two earthquake faults in the area that are pushing together and squeezing material up to the surface. This area has coal, fossils and suprisingly some amber. Digging is fairly easy and when the shool kids find the shiny clear flakes and pebbles, they just holler with excitement - it was very fun. The class was full of third and fourth graders, and some of them were already planning on selling chunks of coal and real amber for a milllllion dollars on eBay. I just love that wild enthusiasm. They weren't even very disappointed to discover they would be lucky to get one or two cents - they figured they would need a hundred pounds, or maybe even five hundred pounds...