Kinetic

Mobile and Web job trends

2011-12-13T22:58:00.000-08:00

Here's a great graph from indeed.com (resume trends) showing mobile and HTML5 job trends - all are surging strong. The second graph shows RoR compared to mobile - the hip web app framework isn't the new hotness any longer.

MongoDB replica sets - high level overview

2011-09-05T13:45:00.001-07:00

Here is a very very brief overview of MongoDB replica sets and a tip to enable read access to read-only replica slaves.
http://www.codypowell.com/taods/2011/08/a-cloud-hosting-architecture-for-mongodb.html

The full definition from the MongoDB site is here http://www.mongodb.org/display/DOCS/Replica+Sets

Browser geolocation APIs

2011-03-29T23:15:00.000-07:00

Many mobile web browsers provide access to the current geo location via JavaScript (see the W3C spec). It's very easy to use but there are a couple gotchas to be aware of. First, not all browsers support the API so you will need to take that into consideration when designing your user experience. Next, requesting the geo location from the browser will prompt the viewer to approve the request. On every page view. This is very annoying. You should store the location data away in a cookie and only periodically request updated location information. Another cool function is that the geolocation API allows your code to be notified as the location moves - perhaps your visitors take the bus or use their mobile devices while riding a bike. This is done with callbacks which is very compatible with client development and makes total sense.

Here is some sample script showing how you could use this geolocation API in your mobile or location aware web apps.


function onLocationUpdated(position)
{
  // do something useful
    savePosition(position);
  createCookie("s_geo","on",3600);
  updateLocationDisplay(position);
}

// request location
if (navigator.geolocation && !readCookie("s_geo"))
{
  navigator.geolocation.getCurrentPosition(onLocationUpdated);

  var watchID = navigator.geolocation.watchPosition(
  onLocationUpdated, null, {
    enableHighAccuracy : true,
    timeout : 30000
  });
}

geonote.org - sharing the world around you

2011-03-29T22:44:00.000-07:00

Over the past month I've put together a mobile friendly web app which lets people share notes about the places they visit. Building the basic web app for storing and sharing notes about a place was pretty straightforward, but like any new application meant to be social the biggest problem is the empty room syndrome - if there is nothing to see, most people just wander off. It takes a special person to start sharing in an empty space.

Rather than try to build up functionality and features to attract a crowd, it seemed that showing information that already exists would be a good way to bootstrap the app. Since I originally envisioned this app as something like Wikipedia for places, but more of an open medium that people can use for any purpose they can put it to, I first thought to look at ways to index Wikipedia entries by their geo location. I quickly found that other folks had already done the indexing and provided an API - geonames.org Pulling this data in was pretty easy, they have a simple HTTP API that returns XML, which geonote.org simply formats into a mobile friendly display. Once there was a web app for sharing notes and viewing 'atlas' pages (the Wikipedia entries), I went in search of other location based APIs and found several great ones.

Here's the list of geo location APIs I've used so far

GeoNames.org (Wikipedia entries and more) - http://www.geonames.org/export/ws-overview.html
Flickr.com (Photos) - http://www.flickr.com/services/api/
Plancast.com (Events) - http://groups.google.com/group/plancast-api/web/overview?pli=1
Hunch (Recommendations) - http://hunch.com/developers/v1/
Twitter (chitter chatter) - http://apiwiki.twitter.com/Twitter-API-Documentation

The Plancast crew especially was extremely helpful. Their forum described upcoming support for searching by latitude and longitude, but it had not been released at the time. After posting a comment they were able to build and release that feature in only a few days (on a weekend too!)
One of the most intriguing APIs was the Hunch API for recommendations. Although it has a lot of power, it requires a Twitter username to provide personalized recommendations and the geonote.org app is too simple to try to do real Twitter authentication integration. I'm sure to revisit the Hunch API though.

Mobile webapps and the JQuery Mobile library

2011-03-13T13:45:00.000-07:00

Recently I've been experimenting with geo location APIs and mobile friendly web applications. Building a native mobile application felt like it would have too steep a learning curve for the miniscule amount of time I have so I looked at what mobile browsers can deliver with just HTML, CSS and JavaScript. It turns out to be pretty easy to build a good looking mobile web application from scratch and I found the JQuery Mobile framework works well to style pages with a native look and feel.
You can see the results at http://geonote.org/places/plans for a 'from scratch' look and http://m.geonote.org/places/plans for the JQuery Mobile look.

The first thing to take to heart is the spartan look of mobile web apps. There simply isn't room for multiple crowded top nav and side nav bars or for the data dense (but information poor) layouts of most sites. Take a look at a sample page from AllRecipes (which is a great site) - http://allrecipes.com/Cook/SHORECOOK/Photo.aspx?photoID=602783 - there are nav bars for site section, tabs, breadcrumbs, sub-page navigation and so on. Not to mention a right nav bar with even more links. These are all useful I'm sure, but for a mobile web app you need to start from a blank page and work you way up and consider the information value of each pixel used. (Every pixel is sacred, every pixel is great. If any pixel is wasted, Tufte gets quite irate.) Another way to think of this is to consider each link as an internal advertisement for a page the user doesn't want to visit. There is a name for unwanted links on a page put there for commerical gain and that is 'spam'. Don't let your designs become link spammy.

Next, you will want to have a way to preview your web app on a mobile device. If you have a modern phone then you can use it's browser and point it to your local dev environment, but another way is to use an iframe wrapped in a phone mockup. Here's the one I use http://geonote.org/html/iphone/ There may be better mobile browser emulators but I didn't spend much time looking for something once I had the iframe based "emulator" working.

Building pages for the 'from scratch' look follows the typical web app development path - you can use most any framework you are comfortable with, but be careful with approaches that are 'client heavy'. You'll want the smallest HTML, few images and the least number of resources downloaded for rendering each page.
Many scripting libraries have a way to package only the necessary modules into a single resource - this cuts down on the network time needed to get the page rendered. Personally, I avoid client libraries since they are mostly meant for whiz-bang interactivity and on a mobile device the interaction feels better when it is as direct as possible. Common web app performance advice applies here - caching is your friend, the network is not.

The JQuery Mobile look was the most interesting part of building the UI for this site. I was really looking forward to getting a native look and feel for free. Although the library is currently in Alpha 3 stage it's very usable and I haven't run into any bugs in my limited testing. The JQuery Mobile library changes how you think of browser based pages. Not only does it try to use Ajax for most things it also introduces "compound pages" which results in an ever-growing DOM with 'sub pages' or panels that are shown and hidden during screen navigation. This allows for JQuery to perform the animated transitions between screens that give the hip 'mobile look' which is so captivating.

The downside to using an Ajax approach is the use of local anchors (the part of a URL after the '#' character) for tracking state. While this is certanly a popular and Ajaxy way of doing things it does have it's problems. If you aren't familiar with the details it really mucks up how you work when building pages and causes things to simply not work and breaks the page (requiring the user to manually refresh the page). I still don't have forms working and had to disable the Ajax loading of some pages due to this hash-based URL trickery. You will need to rigorously test all pages and transitions between all pages to ensure that it actually works.

Another downside to using JQuery Mobile is that the user interaction is noticably slower than just a simple HTML and CSS page. It is almost not "interactive", which is not a good thing for client applications. There is a lot of promise though and I haven't even looked at the built-in capabilities of JQuery Mobile for wider screen devices like tablets.

Non-blocking operations and deferred execution with node.js

2010-08-15T17:17:00.000-07:00

If you write high volume server applications with high concurrency or low latency requirements you have probably heard about node.js This is a relatively easy to understand system that came out in 2009 and has some pretty amazing characteristics. An early presentation by the main author is here - http://s3.amazonaws.com/four.livejournal/20091117/jsconf.pdf

Node.js is an environment for writing Javascript based server applications with a big twist - all IO operations are non-blocking. This non-blocking aspect introduces a concurrency model that may be new to most developers but enables node.js applications to scale to a huge number of concurrent operations - it scales like crazy.

Using non-blocking operations means code that would normally wait for data from a disk file or from a network connection does not wait and waste CPU cycles - your code returns control to the runtime environment and will be called later when the data actually is available. This allows the runtime environment to execute some other code whose data is ready at the moment and gains efficiency by avoiding context switches. This also means there is a single thread accessing data and no synchronization or semaphores are needed to prevent corruption of data due to concurrent access, making your application even more efficient.

Although writing applications in Javascript makes node.js very approachable, the use of non-blocking operations isn't very common in most server applications and results in code that looks similar but is oddly different from what is familiar to most developers. For example, consider a simple program that reads data from a file and processes that data. In a typical procedural program the steps would be :

file = open("filname");
 read(file,buffer);
 close(file);
 do_something(buffer);

This pseudo-code example is easy to understand and probably familiar to most developers. The step-by-step sequence of operations is the way most languages work and how most application logic is described. However, in a non-blocking version the open() function returns immediately - even though the file is not yet open. This introduces some challenges.

file = open("filename");

 // the 'file' is not yet open! what to do?
 read(file,buffer);
 close(file);
 do_something(buffer);

If the open() function were a blocking operation, the runtime environment would defer execution of the remaining sequence of operations until the data was available and then pick up where it left off. In node.js the way that code after a non-blocking operation is paused and picked up later is through the use of callback functions. All the steps listed after using the open() function are bundled into a new function and that bundle of steps is passed as a parameter to the open() function itself. The open() function will return immediately and your code has the choice of doing some work unrelated to the data that is not yet available or simply returning control to the runtime environment by exiting the current function.
When the data for the opened file actually does become available your callback function is invoked by the runtime and your bundle of steps will then proceed.

open("filename",function (f) {
 read(f,buffer);
 close(f);
 do_something(buffer);
});

The parameters to the callback function are defined by the non-blocking operation. In node.js opening files uses a callback that provides an error object (in case opening the file fails) and a file descriptor that can be used to actually read data. In node.js most callback functions have an error object and a list of parameters with the desired data.

In the non-blocking example above you may have noticed the read(f,buffer) function call and guessed that this might be a non-blocking operation. This requires an additional callback function holding the remaining sequence of operations to execute once the data is read into a buffer.

open("filename",function (f) {
 read(f,buffer, function(err,count) {
  close(f);
  do_something(buffer);
 });
});

Some people feel this is a natural way to structure your code. Those people would be wrong.

Here is an actual node.js example of reading from a file

var fs=require('fs'),
 sys=require('sys');


 fs.open("sample.txt",'r',0666,function(err,fd) {
  fs.read(fd,10000,null,'utf8',function(err,str,count) {
   fs.close(fd);
   sys.puts(str);
  });
 });

Although this may appear a bit complex for such a simple task, and you can imagine what happens with more complex application logic, the benefit of this approach becomes more apparent when thinking about more interesting situations. For example, consider reading from two files and merging the contents. Normally a program would read one file, then read another file, then merge the results. The total time taken would be the sum of the time to read each file. With non-blocking operations, reading both files can be started at the same time and the total time taken would only be the longest time to read either of the two files.

Hiring a Sr Engineer at the Rubicon Project

2010-01-15T16:30:00.001-08:00

Hey everybody - I'm looking to hire a few engineers and thought I'd send out a note to let you all know. The Rubicon Project is truly an /awesome/ company to work for and the work we are doing is really exciting, challenging, very high scale and fun! It's like a startup - with benefits. So if you are ready to take charge of some big technology or know someone that is up to it, please shoot me an email. I've included the obligatory job description below. The position is in Seattle by the way.

Mike

Sr Software Engineer

the Rubicon Project is looking for several senior software engineers to help build out new products and features for the Data Intelligence area of our cutting edge online advertising platform. We are looking for people with experience building and operating large-scale, high-traffic Web applications and customer facing Web services. If you are an extremely productive contributor with a get-it-done attitude, work well in a highly collaborative team and want to work in an environment where software engineers are not just cubicle coders but full participants in shaping the product and the business then this job is for you. Serious experience with the following technologies is desired - Linux, Apache, HAProxy, memcached, memcacheq, Java, JSON, Tokyo Tyrant, MongoDB and MySQL.

Posted via email from Kinetic

Sunset BBQ

2009-12-24T10:10:00.001-08:00

The view while barbecuing some chicken from a local market. (Not pictured - the Mirror Pond Ale I had, the local brews have all been disappointing)

Posted via email from Kinetic

Evening on Maui

2009-12-21T20:12:00.001-08:00

We've finally settled into our condo for the week. Spent the day snorkeling and wandering. Things are wonderfully quiet.

Posted via email from Kinetic

Holiday cookies

2009-12-17T22:55:00.001-08:00

My kitchen is a hazard - it's full of Christmas cookies.

Posted via email from Kinetic

Algorithmic (almost) content creation

2009-12-17T11:40:00.001-08:00

This article from Wired on Demand Media and their demand-based creation and delivery of 'content' is an important movement on the Web (and off the Web too).

The choice quote is :

Instead of trying to raise the market value of online content to match the cost of producing it — perhaps an impossible proposition — the secret is to cut costs until they match the market value.

The costs to be cut are the costs of creation (manufacturing). The delivery costs are already nearly zero. Currently Demand Media is generating answers to unfulfilled questions using 'crowd sourcing' and blending media assets like video and photos and quickly written text. I wonder if someday even the text could be auto-generated.

I'm sure in the next six months we'll see a blooming of clones - 'DemandMedia for FooBar' style.

Quite a while ago I had thought about what it would take to build a content site with heavy automation on the gathering, review and approval of content. But I had not thought of optimizing that process based on audience demand. Quite clever really.

update
Just found this post on ReadWriteWeb from a writer that previously worked with DemandMedia - required reading to see things from the viewpoint of someone actually creating DemandMedia content.

Choice quote:

They [writers] appear to be overwhelmingly women, often with children, often English majors or journalism students, looking for a way to do what they love and make a little money at it.

Compare those demographics to Wikipedia: more than 80% male, more than 65% single, more than 85% without children, around 70% under the age of 30.

Sunrise over Rainier

2009-12-07T07:34:00.001-08:00

A beautiful day to be flying

Posted via email from Kinetic

A gift from St. Nicholas

2009-12-06T18:40:00.001-08:00

Look what I found in my shoe from St. Nicholas (or Sinterklaas if you prefer).

Posted via email from Kinetic

The Fables of Aesop

2009-11-18T22:36:00.001-08:00

I like to collect folk tales and old books - especially ones with good illustrations. Here are some scans from a book originally copyright in 1894 (the edition I have was printed in 1917).

See and download the full gallery on posterous

Posted via email from Kinetic

Making the Web faster - SPDY

2009-11-16T19:53:00.000-08:00

Those crafty people at Google are doing some cool work to "make the Web faster". The first I had heard of this initiative it turned out to be how to make "pages" faster - a decent thing, but fairly well known. But recently some folks over there have started to look at the actual underlying issues with the gears grinding out the Web - mainly networking latency. Trying to improve the network protocol of the Web is a tricky thing - lots of people (and egos) can get involved. Surprisingly their effort seems to be off to a good start and everybody is taking it at face value and being supporting and questioning things in a positive way.

One really cool thing mentioned in their whitepaper isn't a direct 'latency' thing - it's about 'server push'. If they can really make this happen a whole knew world of application development would open up.

To enable the server to initiate communications with the client and push data to the client whenever possible.

IE and heinous "operation aborted" error

2009-11-06T12:20:00.000-08:00

We ran into a heinous bug in IE regarding using Javascript to modify the DOM while the page is loading. It turns out that IE6 and IE7 will show a modal error dialog and then clear the page when the user dismisses the error message. On IE8 it was fixed to merely stop rendering the page at that point. How helpful.

You can find out more here on an MDSN blog

If you are unable to defer Javascript execution until after the page finishes loading, the following snippet may work in your use case.


var tags = document.getElementsByTagName("*");
tags[tags.length-1].parentNode.appendChild(n);

Coffee at Zokas in Kirkland

2009-10-20T15:30:00.001-07:00

Haven't been to the new coffee place in Kirkland. They have the largest single block of wood table I've ever seen

Posted via email from Kinetic

The UnderTown in Pt Townsend

2009-10-12T17:29:00.001-07:00

This past weekend we headed out of town to visit Pt. Townsend on the Peninsula. The weather couldn't have been better for this time of year - blue sky and sunny from the time we arrive to when we left on Sunday. We did a little walking around the beaches and forests of Ft Worden doing some geocaching and after a dinner we were looking for a cool place to hang out. Rinneke spotted this brightly lit stairway going down underground into who knows where. We could hear music drifting up so we went down. It turned out to be the UnderTown, a coffee/wine bar and they had live music on Saturday night.
It was a great way to relax, have a warm drink and spend some time together. If you are ever in Pt Townsend check it out.

See and download the full gallery on posterous

Posted via email from Kinetic

Tokyo Tyrant tuning parameters

2009-09-29T21:40:00.000-07:00

We've been working with Tokyo Tyrant for some large scale key-value lookups and the performance has been very nice, but has degraded over time. I've been poking around the various options to try to improve the performance and although there is documentation of various options, the pages are hard to read and figure out what's what. So I thought I'd collect them here for reference. I'll describe the results of tuning and tweaking in a future post.

The most recent authoritative references are here:

Tokyo Tyrant (actually Tokyo Cabinet – the storage engine) supports various types of storage – B+ Tree indexing, hash index, etc. This is configured by setting the filename or file extension to a particular value:

If the name is "*", the database will be an in-memory hash database.
If it is "+", the database will be an in-memory tree database.
If its suffix is ".tch", the database will be a hash database.
If its suffix is ".tcb", the database will be a B+ tree database.
If its suffix is ".tcf", the database will be a fixed-length database.
If its suffix is ".tct", the database will be a table database.

Each has its own set of options and while different flavors of storage may accept the same option name (like bnum), the optimal value likely should be different across storage types.
Tuning parameters can trail the filename, separated by "#". Each parameter is composed of the name and the value, separated by "=". For example, "casket.tch#bnum=1000000#opts=ld" means that the name of the database file is "casket.tch", and the bucket array size is 1000000, and the options are large and deflate.

For disk-based storage, several tuning parameters specify the on-disk layout while others specify memory and caching settings. Changing the on-disk layout requires scanning and re-writing the database data file which requires exclusive access to the file – which means taking the database offline. This scanning and re-writing process is done via tools provided with the distribution (ex: tchmgr and tcbmgr). Changing the memory and caching settings only requires a restart of Tokyo Tyrant.

We've been working only with on-disk storage via the hash and B+ Tree database engines. For a hash database the tuning parameters for the on-disk layout is limited to the size of the bucket array and the size of an element in the bucket array (choosing 'large' gets you 64-bit addressing and addressable data greater than 2GB). When a hash database file is first created, space is allocated on disk for the full bucket array. For example a database with 100M bucket size and 'large' option would start out at around 800MB. This region of the data file is accessed via memory mapped IO. There is an additional 'extra mapped memory' setting which default to 64MB – I'm not sure what this is used for, but for performance more memory is always better.

For a B+ Tree database, there are additional tuning parameters for the structure of the B+ Tree – how many members (links to child nodes) in an interior non-leaf node and how many members in a leaf node. Records are not stored in the B-Tree leaf nodes, but within 'pages'. The leaf nodes point to these pages and each page holds multiple records and is accessed via an internal hash database (and since this is a B+ Tree the records within a page are of course stored in sorted order). There is also a parameter for the bucket size of this internal hash database. One subtle detail is that the bucket size for a B+Tree database is the number of pages, not the number of elements (records) being stored – so this would likely be a smaller number than a hash database for the same number of records.

I've not yet figured out how the dfunit tuning parameter works or what impact that has on a running server, but it looks interesting.

bnum: the number of buckets
capnum: the capacity number of records
capsiz: the capacity size of using memory. Note - records spilled the capacity are removed by the storing order.

capnum: the capacity number of records
capsiz: the capacity size of using memory. Note - records spilled the capacity are removed by the storing order.

opts: "l" of large option (the size of the database can be larger than 2GB by using 64-bit bucket array.), "d" of Deflate option (each record is compressed with Deflate encoding), "b" of BZIP2 option, "t" of TCBS option
bnum: number of elements of the bucket array. If it is not more than 0, the default value is specified. The default value is 131071 (128K). Suggested size of the bucket array is about from 0.5 to 4 times of the number of all records to be stored.
rcnum: maximum number of records to be cached. If it is not more than 0, the record cache is disabled. It is disabled by default.
xmsiz: size of the extra mapped memory. If it is not more than 0, the extra mapped memory is disabled. The default size is 67108864 (64MB).
apow: size of record alignment by power of 2. If it is negative, the default value is specified. The default value is 4 standing for 2^4=16.
fpow: maximum number of elements of the free block pool by power of 2. If it is negative, the default value is specified. The default value is 10 standing for 2^10=1024.
dfunit: unit step number of auto defragmentation. If it is not more than 0, the auto defragmentation is disabled. It is disabled by default.
mode: "w" of writer, "r" of reader,"c" of creating,"t" of truncating ,"e" of no locking,"f" of non-blocking lock

opts: "l" of large option,"d" of Deflate option,"b" of BZIP2 option,"t" of TCBS option
bnum: number of elements of the bucket array. If it is not more than 0, the default value is specified. The default value is 32749 (32K). Suggested size of the bucket array is about from 1 to 4 times of the number of all pages to be stored.
nmemb: number of members in each non-leaf page. If it is not more than 0, the default value is specified. The default value is 256.
ncnum: maximum number of non-leaf nodes to be cached. If it is not more than 0, the default value is specified. The default value is 512.
lmemb: number of members in each leaf page. If it is not more than 0, the default value is specified. The default value is 128.
lcnum: maximum number of leaf nodes to be cached. If it is not more than 0, the default value is specified. The default value is 1024.
apow: size of record alignment by power of 2. If it is negative, the default value is specified. The default value is 8 standing for 2^8=256.
fpow: maximum number of elements of the free block pool by power of 2. If it is negative, the default value is specified. The default value is 10 standing for 2^10=1024.
xmsiz: size of the extra mapped memory. If it is not more than 0, the extra mapped memory is disabled. It is disabled by default.
dfunit: unit step number of auto defragmentation. If it is not more than 0, the auto defragmentation is disabled. It is disabled by default.
mode: "w" of writer, "r" of reader,"c" of creating,"t" of truncating ,"e" of no locking,"f" of non-blocking lock

width: width of the value of each record. If it is not more than 0, the default value is specified. The default value is 255.
limsiz: limit size of the database file. If it is not more than 0, the default value is specified. The default value is 268435456 (256MB).
mode: "w" of writer, "r" of reader,"c" of creating,"t" of truncating ,"e" of no locking,"f" of non-blocking lock

opts: "l" of large option,"d" of Deflate option,"b" of BZIP2 option,"t" of TCBS option
idx: specifies the column name of an index and its type separated by ":"
bnum: number of elements of the bucket array. If it is not more than 0, the default value is specified. The default value is 131071. Suggested size of the bucket array is about from 0.5 to 4 times of the number of all records to be stored.
rcnum: maximum number of records to be cached. If it is not more than 0, the record cache is disabled. It is disabled by default.
lcnum: maximum number of leaf nodes to be cached. If it is not more than 0, the default value is specified. The default value is 4096.
ncnum: maximum number of non-leaf nodes to be cached. If it is not more than 0, the default value is specified. The default value is 512.
xmsiz: size of the extra mapped memory. If it is not more than 0, the extra mapped memory is disabled. The default size is 67108864.
apow: size of record alignment by power of 2. If it is negative, the default value is specified. The default value is 4 standing for 2^4=16.
fpow: maximum number of elements of the free block pool by power of 2. If it is negative, the default value is specified. The default value is 10 standing for 2^10=1024.
dfunit: unit step number of auto defragmentation. If it is not more than 0, the auto defragmentation is disabled. It is disabled by default.
mode: "w" of writer, "r" of reader,"c" of creating,"t" of truncating ,"e" of no locking,"f" of non-blocking lock

Working hard is overrated

2009-09-25T22:36:00.000-07:00

Very insightful post about startups and hard work from someone who has been there.

We agreed that a lot of what we then considered "working hard" was actually "freaking out". Freaking out included panicking, working on things just to be working on something, not knowing what we were doing, fearing failure, worrying about things we needn't have worried about, thinking about fund raising rather than product building, building too many features, getting distracted by competitors, being at the office since just being there seemed productive even if it wasn't -- and other time-consuming activities.

Much more important than working hard is knowing how to find the right thing to work on. Paying attention to what is going on in the world. Seeing patterns. Seeing things as they are rather than how you want them to be. Being able to read what people want. Putting yourself in the right place where information is flowing freely and interesting new juxtapositions can be seen. But you can save yourself a lot of time by working on the right thing. Working hard, even, if that's what you like to do.

Korean BBQ with Rubicon team

2009-09-25T08:43:00.001-07:00

This past Wednesday several of the Rubicon Project engineering team went out for dinner at a Korean BBQ in LA. It was a good mix of a working meeting - talking about engineering practices and development in general - and good food and drink. The meat was all very tasty and only at the end did I find out what some of it was. I had never had beef tongue before - I always swore I wouldn't taste anything that could taste me back - but it was all really good, especially the soju (a lot like vodka).

See and download the full gallery on posterous

Posted via email from Kinetic

PubSubHubBub - feed futures

2009-09-20T16:39:00.000-07:00

Cool - Bob Wyman is involved in the PubSubHubBub discussion group. In this post he hints at content-based routing - not just topic based routing - being possible in the future with PSHB. It's time to find some excuse to use this new PSHB technology at my day job.

For instance, while today we think mostly about "topic-based" distribution -- i.e. subscribing to known feeds by name, in the future, people might like to subscribe to "concepts" or "words" that appear in the content of updates. Rather than saying "Tell me whenever Tom's feed changes!", you might like to say: "Tell me whenever any feed mentions PSHB." In that case, down stream systems are going to want to have the content (not just a notification of change) in order to match updates to subscriptions.

Real-time web, take 2

2009-09-18T15:15:00.000-07:00

Bernard Lunn has a good post over on ReadWriteWeb putting the recent PubSubHubBub/RSSCloud news into context. Very funny that he calls KnowNow a "blow out", but I think he correctly identified their issue being a focus on the enterprise market (when that market had fairly established solutions).

Wish I hadn't been so busy over the past two years and could have worked on helping build PubSubHubBub-style technology.

Swimming at the beach

2009-09-09T23:17:00.001-07:00

After a few months of travel to LA I finally took time to go swimming in the surf. We headed north from Santa Monica and wound up near Malibu.

Posted via email from Kinetic

Zeeks pizza after the game

2009-09-05T19:07:00.001-07:00

The boys won two of three games so we are out for pizza and (root) beer.

Posted via email from Kinetic

Tractor Tavern

2009-09-03T23:32:00.001-07:00

Most excellent West Texas bluegrassy rocking music with The Dead Horses

Posted via email from Kinetic

Empire State Building

2009-08-23T08:14:00.001-07:00

View up, from top of Empire State Building.
It took just one year to build the building!

Posted via email from Kinetic

Attitash mountain

2009-08-13T14:55:00.001-07:00

This is what you do with a ski resort in the summer - add sled rides and water slides!

Posted via email from Kinetic

In a conference call, from the hammock

2009-08-03T15:13:00.001-07:00

Posted via email from Kinetic

Faster MySQL B-Tree performance

2009-08-02T15:04:00.000-07:00

In our ongoing quest for low-cost, high-performance solutions for our platform I've run across TokuDB from TokuTek. This is a MySQL storage engine that uses a different indexing technology that makes updates to an index faster. Making index maintainence faster means being able to have more indices and that allows for richer queries and more interesting applications and analytics.

For one part of our system, I'm interested simply in faster inserts to the index. Having more indices for each table would help in other areas but I haven't started working on those yet. The table in question manages a many-to-one mapping of tokens (text) to internal database row IDs (integers). Essentially, it's a simple key/value lookup table. Currently a single machine does 150 inserts per second, which is fairly abysmal for a table with only 200M records.
This system uses MySQL and the InnoDB storage engine which uses a B-Tree indexing approach. Since the lookups are simply key/value lookups there really is no need for a B-Tree - a hash indexing scheme would work, but unfortunately InnoDB does not support that. We may migrate to Tokyo Tyrant and Tokyo Cabinet since that supports hash indexing and also seems to have better concurrency support (many non-blocking concurrent requests). The keys inserted are almost sequential, but from what I've read the sequential nature of the inserts should help (although the bottom-up building of a B-tree could be optimized by detecting that the key causing a page split is greater than all the values in the page being split - I think Oracle does this).

While looking for hash based indexing for MySQL I found TokuDB which uses the sexy phrase 'fractal tree' for their indexing. Their site has a few introductory whitepapers but a for deeper understanding I wanted to get to the theoretical foundation of their technology.

I soon found that there are several meanings to 'fractal tree' data indexing

fpb+ tree. Fractal pre-fetching b+ trees, embeds cache-optimized trees within disk-optimized trees. Unrelated to TokuDB
Fractal Trees (TM) from TokuTek. Uses 'cache oblivious' algorithms to improve B-tree usage.

One of their founders was kind enough to post some links on the mostly useful MySQL performance blog - here are the links (however, a few seem to be missing)

The core algorithms seem to be for 'cache oblivious' b-tree structures, based on a description from a masters thesis from 1999. Maybe not all things 20th century suck (postscript files excluded).
Although I haven't finished reading all the papers, what I like about this approach is the way they model algorithm performance as the cost to transfer blocks of data between layers of storage and also that they consider multiple layers. The most important transfer costs being between disk and memory, but including considerations like an OS managed disk cache is good. This models the multi-layer caching that nearly all large scale systems use. We are big fans of measuring algorithm performance and selecting the right tool for the right job.

As an aside, one of the comments on the MySQL performance blog pointed me to InfoBright, a column-oriented store, which might be useful in some of our analytics systems for ad-hoc reporting.

Cannon Beach

2009-07-26T16:03:00.001-07:00

Last week we spent a great weekend down at Cannon Beach, Oregon. Although there was ocean mist in the mornings, that burned off fairly quickly. The sunsets were fantastic and the kids (and our niece and nephew) had a blast.

See and download the full gallery on posterous

Posted via email from Kinetic

At the Ocean

2009-07-19T20:39:00.001-07:00

Cannon Beach, Oregon with family.

Posted via email from Kinetic

Bikers invade Seaside

2009-07-18T16:10:00.001-07:00

Tons of big Harley hogs in town - very cool!

Posted via email from Kinetic

Hiking to Lake 22

2009-07-17T23:08:00.001-07:00

It has a funny name but good old Lake 22 has a great trail with giant old growth cedars following a stream up to a beautiful alpine lake.
This is a fairly popular hike so there were more people than I'd like but it was good to see so many people with kids and dogs out and enjoying the mountains.

See and download the full gallery on posterous

Posted via email from Kinetic

More rock and roll

2009-07-08T23:25:00.001-07:00

These guys kick ass compared to the previous band. Love the Boy Scout shirt on the lead singer. The guy on the right had a '59 Les Paul goldtop. Wicked.

Turns out, their name is 'Join the Fight'.

Posted via email from Kinetic

Butterfly in the garden

2009-06-25T19:44:00.001-07:00

I saw this butterfly flittering between the flowers and ran over to capture it with the camera. It was really big, bigger than I had thought was normal - I wonder what kind of caterpiller turns into this kind of butterfly.

Posted via email from Kinetic

More rock and roll at Dakota

2009-06-17T23:09:00.001-07:00

Another great night at the Dakota.
This band is Ambrose - any band that starts with a cowbell is okay by me.

Posted via email from Kinetic

Waiting in the sun for conference call to start

2009-06-15T15:04:00.001-07:00

Posted via email from Kinetic

More great live music

2009-06-10T22:25:00.001-07:00

Wonderful Aussie accent and wicked slide guitar too

Posted via email from Kinetic

Live music at the Dakota

2009-06-10T21:14:00.001-07:00

Really enjoying the live music here in Santa Monica

Posted via email from Kinetic

Fishing

2009-06-06T14:55:00.001-07:00

Fishing at the north end of Lake Washington. Nothing here but nice to be outside.

Posted via email from Kinetic

Wasting time in the sun

2009-05-30T14:58:00.001-07:00

Ready to lay in the hot hot sun and listen to the birds in the forest. probably will fall asleep and burn the other side of my legs.

Posted via email from Kinetic

Walking at Juanita Bay park

2009-05-24T12:24:00.001-07:00

We haven't been to Juanita Bay park in a long time and after a nice sushi dinner in Kirkland for Rinneke's birthday we headed down there.
It's just as picturesque as I remembered, but it had changed a bit -a few less trees, the beaver dam is gone and I didn't see any turtles this time. Still a log of red-winged blackbirds though.

Posted via email from Kinetic

Fostering kittens

2009-05-23T15:02:00.001-07:00

We picked up four kittens today that need a foster home for the next few days before they have surgery and then can find a permanent home.
One of our other two cats is very curious (but probably just looking for soft kitten food) and the other cat just hisses as it walks by the room with the kittens.
If you would like to foster kittens - and it's pretty easy - just contact Homeward Pets in Woodinville.

Posted via email from Kinetic

Real-time Web just around the corner

2009-05-22T13:38:00.000-07:00

The ReadWriteWeb blog has a good post about gathering momentum for a resurgence of interest in real-time search and notifications. I don't think the examples he points to will push it into the mainstream - that functionality has been around in many forms for many years (I even built searchalert.net seven or eight years ago to do that). I do think something will happen, but I'm not sure what application of this technology will make it to the big time.

The Real Time Web is coming so fast we've hardly had any time to think about it yet. So let's do that, shall we? The two hottest technologies online, Twitter and Facebook, are fast integrating real-time delivery of activity streams to their users. Paul Buchheit, the man who built the first versions of both Gmail and Adsense, says the real time web is going to be the next big thing. Buchheit's FriendFeed is a key point of innovation in real time. Social media ping server Gnip promised to turn everything online into Instant Messaging-style XMPP feeds, and though that's been put on hold in favor of more immediately clear value - we've still got our fingers crossed.

Beers at Fado

2009-05-20T17:39:00.001-07:00

Having a quiet time at Fados and having a Fat Tire.

Posted via email from Kinetic

The end of the road

2009-05-20T12:47:00.001-07:00

Out taking the dog-in-law for a walk and liking how everything is that glowing spring green color I love so much. It's like there is this whole new beginning just around the corner

Posted via email from Kinetic

Another great weekend at Orkila

2009-05-18T22:17:00.001-07:00

We had another great weekend at Camp Orkila on Orcas Island. The weather
couldn't have been better and the sunsets were beautiful as always. The best
part was being able to go out on the water in kayaks - something we've
wanted to do for years. We'll be back next year but that will likely be our
last year with the Y-Westerners program.

Posted via email from Mike's posterous

Having a campfire on the beach

2009-05-17T13:50:00.001-07:00

Campfires on the beach are one of my favorite things about our campouts. Our local band of hooligans enjoyed the smores and chased each other til it was too dark to see.

Posted via email from Mike's posterous

Cliff climbing on Freeman Island

2009-05-17T13:44:00.001-07:00

On the way back from our kayak trip we stopped at Freeman Island. There is a geocache at the top of the cliff. But there was a nest of bees too, so I didn't do the climb this year.

Posted via email from Mike's posterous

Kayaks ready to go in the Sound

2009-05-17T13:08:00.001-07:00

After six years of camp on Orcas we finally had the chance to take out some kayaks for a quick afternoon trip.
There were seals sunning on rocks at Doughty Point but no other wildlife.
We were soaked by a freak tsunami caused by a quake that measured 2 on the (Ron) Richter scale).

Posted via email from Mike's posterous

Life and Death in the Forest

2009-05-16T10:09:00.001-07:00

A game where some are herbivores, some are omnivores and some are carnivores.
You make your choice then run like crazy.

Posted via email from Mike's posterous

On our way to Orcas Island

2009-05-15T11:40:00.001-07:00

This weekend Stephan and I will be going to Camp Orkila for our annual Y-Guides campout. Hopefully we'll be doing kayaks this time.

Posted via email from Mike's posterous

Above the Clouds whitepaper

2009-04-23T22:25:00.000-07:00

Here's a whitepaper on Cloud Computing from the UC Berkeley RAD Lab. - just what everyone has been waiting for, a whitepaper on Cloud Computing.

In part this describes obstacles and opportunities. My personal favorite :

Obstacle #6 : Scalable Storage
Opportunity #6 : Invent Scalable Store

That's right, we finally have the go-ahead to Invent Scalable Store.

The paper gets better the more you read. Another great quote:

Google Search is effectively the dial tone of the Internet: if people went to Google for search and it wasn’t available, they would think the Internet was down

When not showing an ad is better

2009-03-26T09:13:00.000-07:00

The Mike on Ads blog had an interesting post a while back referencing some research by Yahoo about how not showing ads might be better for you.

Instead of showing crappy CPA offers the publisher should show either nothing at all, or some relevant site content. Show a snippet of the friend-feed, or maybe a list of 'online friends'. Show "interesting related links", or "new photos posted"… it doesn’t really matter. Show something that is of interest to the user. The point of the exercise is to train the user to start looking at this specific space again.

[...]

If this is obviously so good, why is nobody doing it? Well there’s only one small insignificant problem… Publishers have no way of identifying the top 20% of impressions. You see, especially on social networking sites a huge portion of that 20% are impressions that are sold behaviorally via ad-networks and exchanges. Those ad-networks and exchanges need to see the full 100% to be able to cherry-pick the 10% that are valuable to them thereby making it quite difficult for the publisher to “not show ads” on worthless impressions.

Showing something other than ads when there is no money involved is a great idea. Unfortunately, most traditional ad networks have no interest or capability to do this. Even 'behavioral' targeting folks aren't in a position because they have only a few 'behaviors' rather than a full tagcloud of interests.

Our Others Online affinity profiling system has behaviors, interests and a measure of the commercial value of those interests which means we can power ad units that know when its the right time to show an ad, or whether it's better to show relevant news or other content.

A clean desktop. The hard way.

2009-03-17T15:11:00.001-07:00

My hard drive has been complaining recently and yesterday gave up the ghost. I found a few decent replacement drives online, but needed a new one right now. We had a gift certificate to Staples and what do you know - they had a reasonable 500GB SATA drive for $80. I called them last night just before 9pm - not only were they helpful late at night, but they put one behind the counter for me. Nice people there.

This morning I backed up all my files to my Linux server here at home and swapped in the new drive. I re-installed Win XP Pro and my machine was now a blank, default machine with no drivers for all the hardware plugged into it - like the big monitor, the speakers, power management, etc. I finally figured out Dell's horrible user interface for installing drivers and soon was back to a running machine with a clean desktop (first time in several years).

The only oddity was the clock was off by an hour. The old Win XP install didn't know about Congress changing when Daylight Savings Time started.

After restoring all my files, I needed to update Win XP and install all the old applications again. Not too hard, but time consuming.

Here's the order that I restored things

Dell drivers
Firefox browser
iTunes (so I could have music while restoring everything else)
PuTTY
PasswordSafe
Windows XP service pack 3
Microsoft Office 2003
Java 6 SDK
Eclipse
Apache 2.2
Tomcat 6
DBVisualizer
TortoiseSVN shell
GTalk

Yahoo Query Language and Open Tables

2009-03-02T11:34:00.000-08:00

I've been looking at the Yahoo Open Data Tables and Query Language documentation. This is truly amazing stuff! It provides a service API that accesses many well known data sources (many are Yahoo) and transforms the data into XML or JSON. The data sources can be external URLs that provide XML and Yahoo does the fetch, parse, extract and transform that you want. You can provide a definition of some other external data source and they will hook it into their unified API fetch/query/transform service.

Some of the data sources are Flickr, local listings, geo location info, web search, image search, news search, weather and so on. One stop shopping for lots of great data.

Their console http://developer.yahoo.com/yql/console/ is a great way to see what's possible.

This is what I've wanted for many years. A long time ago I wanted to build a service that would provide "XML data sources" (I even registered xmldatasource.com) for everything available on the Web - now it looks like Yahoo has actually done it.

Let's hope they keep this data access service open to all.

Mining the Twitter Stream

2009-02-19T21:38:00.000-08:00

From the Data Mining: Text Mining, Visualization and Social Media blog -
The Business of Mining the Twitter Stream

Good post about mining the Twitter stream which hints that existing social media mining companies may already be too established to be replaced by newcomers. This is fast moving area that almost looks like a microcosm of the 'innovators dilemma' - except there is no large, fossilized ecosystem in place.

SDB adds aggregate function

2009-02-19T21:28:00.000-08:00

Looks like Amazon has added an aggregate function to SDB - basically count(*).

Amazon Web Services Blog: New Count Function

I wonder why there isn't a mass quantity of blogs with performance data for various queries against various types and sizes of data sets.

CouchDB: Jeremy Zawodny's impressions

2009-02-10T21:13:00.000-08:00

I haven't done any work with CouchDB other than read through documents, so I hope Jeremy continues to post what he learns.

Playing With CouchDB: First Impressions (by Jeremy Zawodny)

Scalable, reliable key-value lookup service

2009-01-29T17:22:00.000-08:00

Here's a great summary of most of the available key-value storage services from Richard Jones of last.FM - great stuff.

From what I can tell, Scalaris is only memory-resident at the moment and doesn’t persist data to disk. This makes it entirely impractical to actually run a service like Wikipedia on Scalaris for real - but it sounds like they tackled the hard problems first, and persisting to disk should be a walk in the park after you rolled your own version of Chord and made Paxos your bitch.

I'm leaning towards Voldemort, but need to look into this more.

McSweeney's : Fire: The Next Sharp Stick?

2009-01-05T23:02:00.000-08:00

Good gravy, this gave me a great laugh!

Fire: The Next Sharp Stick?

You've got to read the whole thing, but here's an excerpt

ONE: Hairy One, Maker of Fire. Maker of Fire, the Hairy One.

MAKER: My pleasure, Hairy One. I've followed your work with Ten Men for a long time. It's a remarkable firm.

HAIRY ONE: So you're the one with the fire?

MAKER: Yes.

HAIRY ONE: Is it here?

MAKER: Well, no.

HAIRY ONE: Where is it?

MAKER: Well, in a sense, Hairy One, fire is everywhere. Rather than being an object, say, like your sharp stick, it's really a process, so it can't really be said to exist anywhere. In a sense, fire exists in its own imaginary, virtual space, where we can only talk about what is not fire and what might become fire.

HAIRY ONE: Whoa whoa whoa! English, please!

I Hate Computers - Log entry #762

2008-12-31T16:13:00.000-08:00

Here are some helpful tips, if you ever find yourself using 'computers', especially the Windows species.

Never ever disable the "VgaSave" video adapter. Ever. This is the fallback software used by Windows to display things on the screen if no other video driver works. If this is disabled, Windows will start but your screen will show nothing but the finest shade of black. If you fail to follow this advice, be prepared to follow these instruction to re-enable vgasave service from the Windows recovery console. Not that it worked for me, but hey, good luck with that.

If you perform a 'repair' installation while your screen shows nothing but the black darkness which has enveloped the heart of every poor Windows operating system developer, hoping for your video adapter driver to be repaired, do not (no not ever) turn the power off. Not even once, just for fun.

If you happen to boot up your Windows computer and it fails to start due to 'Registry is corrupted' or some such, and you choose to re-install Windows rather than becoming a Tibetan monk (who would likely have fewer problems than a Windows user with a failing disk drive, even considering the Chinese government's approach to freedom), be prepared for a long stretch of file recovery and application re-installation. I recommend Fat Tire Amber Ale from the New Belgium Brewing Company.

If you happen to have not followed these tips and you have re-installed Windows and you now have a default user account with a default green and deceptively bucolic grassy field, where before you had many files and folders and possibly (if you are reading this) insipid desktop wallpaper, here is an actual, useful tip : you can change the settings for this newly created user account to use the folders and settings of your previous user account. This won't solve all your problems, but really, do you think that's even possible? Even with advice from me (and I'm composed of nearly 100% pure awesome). Here is what you do :
- use the regedit program (and if you don't know what that is, give up now) to view HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList
- Look for a sub-entry that has a really long value with an entry of ProfileImagePath that points to the fairly empty and useless 'new user account' (e.g. "%SystemDrive%\Documents and Settings\Mike.NAUTILUS". Change the value to point to the location of the old and wondrous user account (e.g. "%SystemDrive%\Documents and Settings\Mike").
- Go back to the ProfileList registry key, and update the "AllUsersProfile" and "DefaultUserProfile" settings - you'll probably want to continue to use the old folder that had settings for all the old applications you had previously installed and which are most likely useless, since you've re-installed Windows.
- Log out, then log back in. Hopefully you now see your old desktop. In either case, remember - Fat Tire Amber Ale from the New Belgium Brewing Company.
- To be honest, you could just copy the files from the old user account folder into the folder for the new user account. But I'm lazy. Just be aware that by pointing the new user account to the old user folder, there may be permission issues with accessing some of the folders or files. I haven't seen that happen but it seems a likely next failure.

Catalina - point-of-sale ad network

2008-12-09T21:24:00.001-08:00

I wonder if they support cookies...

Armed with two years of purchase data for 80 million individual consumers, Catalina Marketing is this week launching a new in-store ad network called the Pointer Media Network.

The information comes from frequent shopper cards covering most of the nation's supermarket chains, thousands of drugstores and other retailers.

[...]

Pointer Media works like this: Catalina has installed color printers at the checkout counters of close to 50,000 stores around the country that are linked to the company's massive database of consumer purchases. When a shopper's order is rung up, the printer instantly creates a print ad on a receipt-size piece of paper based on the unique purchasing history of that shopper. The ad is handed to the shopper along with the receipt for the current purchase.

SpaceX reaches orbit

2008-09-28T21:29:00.000-07:00

SpaceX became the first private company to launch a liquid fueled rocket into Earth orbit. Totally awesome! This was their fourth launch of the Falcon 1 configuration and was carrying a test payload. Their third launch two months ago didn't achieve orbit and actually was carrying paying payload - oops.

This is a huge step on the road to space.

SpaceX - Flight 3 of the Falcon 1

2008-08-02T16:38:00.000-07:00

SpaceX is launching their Falcon 1 today from Kwajalein - about 1 hour to go before liftoff, if things go according to plan.

Here's a live webcast - http://www.spacex.com/webcast.php

WebHooks

2008-07-30T14:58:00.000-07:00

This looks interesting - in a 'teach people how the Web really works' kind of way. WebHooks is a catch phrase for Web application development where notification are sent from theh source to the listener via HTTP POST, rather than the other way around via polling (which, as some have said, doesn't scale).

Somewhat related to my earlier post on how HTTP can be used as an alternate to XMPP/Jabber in a publish/subscribe scenario without too much problem or angst.

MySQL and LAST_INSERT_ID()

2008-07-29T14:39:00.000-07:00

We've recently migrated our software servers from a dedicated server environment (BlueGecko - really good people and service) to Amazon's EC2 'virtual compute cloud' environment.

The new system has a very nice performance monitoring capability based on Ganglia that gives us visibility into the performance of service requests as well as details on more fine-grained functions that take place within each request. We can now see not only the number of requests or functions but also the average duration and the time it took for 50%, 95% or 99% of the requests to complete in a five minute interval. This percentile breakdown gives a quick feel for how 'spiky' performance is and how common outliers are.

So far, things have gone very well but there was one function of the system that seemed to suddenly have terrible performance. I was able to quickly see which area of the code was involved, which pointed me to some SQL statements.

As our system adds new anonymous profiles, the code retrieves the unique number assigned by MySQL using the special query that MySQL provides:

SELECT LAST_INSERT_ID() AS user_id FROM oo_anon_user

However, it turns out that this SQL is actually incorrect - the extra "FROM oo_anon_user" caused the database to return every single record from the table back to the client, on every request that created a new record. This of course took some time.

The correct syntax to retrieve an auto-increment field from a MySQL table is

SELECT LAST_INSERT_ID() AS user_id

Remember to omit any FROM clause.

It's much much much faster now.

REST and pub/sub

2008-07-24T14:24:00.000-07:00

It's unfortunate that technologists continue to propagate serious mistakes like "[...] its also clear that REST and its inherent polling mechanism isn't the best way of building a user notification system [...]"

REST is about state transfer - and event notifications are also state transfer.
As for HTTP, it isn't only "polling" - anyone that has posted a blog entry knows that. The 'client' can 'post' updates to the 'server' - exactly the same as event notifications via XMPP. The great thing about XMPP is the federated multi-hop capability with 'trust' built-in. Just like email, only with everyone using settings for very low latency delivery.

There have been multiple publish/subscribe over HTTP mechanism (comet, mod_pubsub, KnowNow, etc) over the years.

Twhirl Adds Identi.ca

2008-07-21T16:39:00.000-07:00

I don't really follow the social/friend feed streaming application space, but this post about Twhirl from
ReadWriteWeb had a quote that caught my eye (emphasis mine):

However, for the regular user, always on social networking doesn't have to be a source of stress - it just means that when you go online, socializing with others just becomes part of the overall experience of being on the internet.

MoonBoy

2008-06-01T18:54:00.001-07:00

We've been doing GeoCaching for a number of years, enjoying learning about new parks and finding Geocaches on our trips. Earlier this year we created our own Geocache near our house and placed a Travel Bug in it - MoonBoy. His goal was to view a space shuttle launch and to someday travel on a Space Shuttle.

Amazingly, he's made his first goal and was an observer to this weekend's launch of STS. This is totally cool! Here are the photos on the GeoCache site

Oh, the irony of shallow WSDL

2008-05-12T14:56:00.000-07:00

I'm looking into the API for the Hi5 social network and unfortunately found some WSDL. They also have a REST API, but it's documentation appears auto-generated from WSDL that nobody actually filled in. Somewhat useless.

Normally I wouldn't post about WSDL, but I couldn't pass up the irony of the WSDL documentation for the authentication service API having an HTML form describing how to authenticate the user. If that's not irony, I don't know what is.

Ultimate Twitter revenue model - chatbots??

2008-05-09T15:18:00.000-07:00

From ReadWrite Web

"Essentially, this would entail Twitter parsing over the Tweets of a given user, as well as the Tweets of the users he/she is following. Common keywords, themes, and phrases are then pulled from this data and associated with that user. As a result, highly-targeted ads can be displayed based on the user's network of content ("web design", for example). These simple text ads would look very similar to regular Tweets, but would be clearly marked as "Sponsored Content"."

I think chatbots haven't work for a reason - people want to chat not shop.

Reading RWW and other pundit blogs that describe "how the future will work" reminds of reading Popular Science as a kid and gazing in wonder at the flying cars and transparent house soon to be built.

Web scale pubsub

2008-05-06T13:41:00.001-07:00

Looks like people are now starting to talk about a de-centralized, Web native pubsub system. Now is the time, if applications are going to avoid being based on a single company's service. Too bad pubsub.com and KnowNow aren't driving this.

CouchDB

2008-04-24T21:42:00.000-07:00

I'd heard the CouchDB name but but didn't realize that it's an HTTP-accessed, Erlang-implemented, schema-free, distributed datatabase - and it supports Javascript for defining views. Time to dive in!

From Apache CouchDB: The CouchDB Project:

"Apache CouchDB is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API. Among other features, it provides robust, incremental replication with bi-directional conflict detection and resolution, and is queryable and indexable using a table-oriented view engine with JavaScript acting as the default view definition language."

What programmers like

2008-04-21T22:17:00.000-07:00

Quote of the day, found on the High Scalability blog while looking into Amazon's SimpleDB

"Programmers like problems they can solve with more programming."

Online services and availability

2008-04-20T21:32:00.001-07:00

I've been working with the design and development of online services for quite some time and am very familiar with what 'high availability' means, but today I had the opportunity to learn first hand how the lack of reliability can directly impact people. This is a minor example, to be sure but as more 'software as a service' companies are built out more people will be affected in ways that they don't quite understand.

My daughter attends a Middle School that uses the online curriculum from Key Curriculum Press - I have no idea if the content is decent or not (but knowing the teachers at the school I'd bet it's good) and in order to access specific materials a login process has to take place and then access to some PDFs become available. This weekend, the login service was unavailable and so my daughter could not work on that part of her homework - very annoying.

The problem is likely something as simple as a server misconfiguration or a host being down (I can't imagine there's too much horsepower required for this sort of service), but as a 'customer' we are unable to do anything. There is no 24x7 tech support or online contact information. I could make a phone call, but that would only work between 8am and 5pm (PST), and I'm pretty sure kids do homework outside of those hours.

I've sent an email to their president and we'll see what kind of response that gets.

Update - I received a nice email from their president and she also sent a follow up email with more details. Very nice!

MyOpenID for Your Domain

2008-04-19T19:57:00.000-07:00

This post from ReadWriteWeb about
MyOpenID for Your Domain is interesting. I've looked into OpenID several times in the past but it didn't have enough immediate value to get me to do anything about it. Now that more sites are 'supporting' OpenID and with the simple to use MyOpenID provider maybe I'll set something up. It would be interesting to see decentralized profiles and authentication, it seems a shame that everyone is getting locked into a few social networks for hosting and managing personal profiles and shared contact info.

Yaws and reverse proxy

2008-04-16T20:10:00.000-07:00

It seems that there are a lot of situations where efficient connection handling in an HTTP reverse proxy is needed to help with scaling and keeping costs down, and my first guess was that Erlang would be the way to go.

So I just searched to see if there was a reverse proxy written in Erlang and sure enough Yaws has one.

Next, I need to find some folks that have actually used it and ask some questions. Especially for persistent connections (accepting a request but sending no response until there is data available, like in a message queue or pubsub scenario).

Of Liquidity, Competition and Platforms

2008-04-14T23:29:00.000-07:00

This post by Bob Wyman "Of Liquidity, Competition and Platforms" reminds me why I subscribe to tech blogs - occasionally you get a deep thinker that clearly and concisely describes something that likely took them a long time to figure out.

"As many others have said, much of what people are building today is 'features' not products. As long as that is the case, raw economics is the real problem with the software business. Competition strengthens the platform builders while eviscerating the component builders... The rich get richer and the builders of innovative features must be satisfied with the 'personal rewards' of doing a job well unless they get lucky in the buy-out lottery game."

With the various cloud computing efforts well underway, and with each providing a different approach as a 'platform', it would be good for developers and entrepreneurs to think about which road to travel.

More REST and Erlang

2008-04-10T15:29:00.000-07:00

Man, I wish I had the time to dig into these posts about Erlang, Yaws and RESTful services.

Cross domain web apps

2008-04-10T14:45:00.000-07:00

This post about cross domain Web apps is spot on. The growth of the current Web was due to millions of silos - no data shared between sites. Over the past few years technology like iframes, ajax and other styles of 'mashups' have made it possible to create applications composed from cross-domain services. I believe this will drive the next major growth curve of the Web.

Elephant on a Bicycle

2008-04-04T16:26:00.000-07:00

Here's an interesting blog on China from someone living there. The photos are both beautiful and intriguing.

Elephant on a Bicycle

An example of the writing (which I really like)

I'm now in Xining, Qinghai, in a youth hostel 15 floors up - which you think would offer remarkable views of the mountains that crowd the city on all sides. But it doesn't, this is China. 15 floors up generally just presents you with worse pollution, as it seems to procrastinate in mid-air limbo.

I'm in a quiet little room with two computers. Beside me sits a young monk of about 18, who has rather rashly decided to accessorise his austere crimson robes with a pink feather boa. I couldn't make that up. Furthermore, he is currently downloading provocative pictures of Nicole Kidman; certainly harder to find fault with this inevitable phase of teenage experimentation.

Xining city is a fascinating stopover because it offers a unique diversity. It is still predominantly Han Chinese, in both it's populace and it's blindfolded embrace of modernity - noisy, dirty, difficult to look at, pulsing
with energy. But alongside this, it contains a deeply religious aspect from the presence of size-able pockets of Islamic and Tibetan communities. In places there even exists a meditative quiet - a stillness - which amidst the commercial bustle and the perpetual foot-race of human progress can seem almost revolutionary. This is especially apparent in the areas surrounding the great mosque where I spent most of the day after befriending a young Koranic scholar named Farooq.

Party like it's 1994

2008-03-21T19:48:00.000-07:00

You learn something new every day. Although sometimes it takes 5,110 days.
From Steve Vinoski's blog - here's
A Note on Distributed Computing from Sun in 1994.

We argue that objects that interact in a distributed system need to be dealt with in ways that are intrinsically different from objects that interact in a single address space. These differences are required because distributed systems require that the programmer be aware of latency, have a different model of memory access, and take into account issues of concurrency and partial failure.

What Sucks About Erlang

2008-03-11T23:22:00.000-07:00

Very funny yet enlightening post about Erlang. And I still want to build a real system with it.

From Damien Katz: What Sucks About Erlang:

"The only purpose of the true -> ok line is to give it an else condition to match. That weird taste in the back of your throat? It's probably vomit."

Seattle's nuclear reactor

2008-03-01T09:45:00.000-08:00

I knew that some universities have or had nuclear reactors for research, but I didn't know they could look this beautiful.

From Crosscut:

Abby Martin hopes that, whatever its fate, the Nuclear Reactor Building finally gets public acknowledgment of its role in history, and credit for its architectural originality. That would be in keeping with its original intent. It's hot core may be gone, but it can still teach us something.

No more ugly desktop software

2008-02-25T14:10:00.000-08:00

From
ReadWriteWeb it seems Mr Kirkpatrick is all a quiver with Adobe AIR :

AIR lets developers use Adobe Flash, Adobe Flex, HTML and AJAX to create desktop apps. That means no more ugly desktop software!

No more ugly desktop software. Do you really think it was the software that was the problem before? Unless you're part of the short-attention-span-journalism industry, it'll take more than drop shadows and bouncy icons to get past ugly software.

Dare Obasanjo aka Carnage4Life - How "View Source" Broke the Web

2008-02-22T21:49:00.000-08:00

I haven't been following the Web blogging elite recently (really, 'view source' broke the Web??), but this post - How "View Source" Broke the Web - made me think of Bob Dylan.

Broken lines, broken strings,
Broken threads, broken springs,
Broken idols, broken heads,
People sleeping in broken beds.
Ain't no use jiving
Ain't no use joking
Everything is broken.

SpaceX signs more business, builds more engines

2008-02-02T17:23:00.000-08:00

It looks like SpaceX is continuing to make good progress, both with engineering and with business.
From the SpaceX status page:

"For the last few months, optional activities such as website updates have gone on the back burner while we finished the regeneratively cooled Merlin 1C engine, got the Falcon 9 first stage integrated, proof tested and fired, signed up our first GTO (geostationary transfer orbit) customer - Avanti Communications Group, and took our COTS system past the big CDR milestone."

Signing a contract for placement into a geostationary orbit is huge! I'm really hoping these folks don't spend all their capital on development of bigger engines without establishing the operational business that will generate revenue. They seem to be looking to raise $100M in the first half of 2008, hopefully that will happen and keep this company going to profitability.

Yowza! MicroHoo in the future?

2008-02-01T08:19:00.000-08:00

Looks like Microsoft is serious about buying advertising's future - they have offered $44B for Yahoo.
Favorite punditry (from Stowe Boyd)

"Personally, I think the Microsoft and Yahoo matchup is like two tired swimmers who bump into each other and then wind up drowning each other in their scramble to survive. But Yahoo will be the first to go under in this embrace."

This one from iMe on Twitter is good too:

"Microsoft and Yahoo! Its like a blind man trying to lead a deaf guide dog...."

Stonebraker on MapReduce

2008-01-19T18:03:00.000-08:00

I've too busy lately to post on all the exciting things happening in the database world - more people getting into column-oriented storage, Amazon's SimpleDB service, consumer database Web services like blist and LongJump, Sun buying MySQL. But I couldn't pass up commenting on Joe Gregorio's post on Stonebraker on MapReduce.
It seems everybody is panning Stonebraker's evaluation of MapReduce as incorrectly comparing it to a DBMS. One commenter even said (ironic comment of the year) Michael Stonebraker should learn what a DMBMS is.

The point, which I'm sure someone has pointed out, can be found by looking at the summary of their findings. Here are a few:
- A sub-optimal implementation, in that it uses brute force instead of indexing
- Missing most of the features that are routinely included in current DBMS
- Incompatible with all of the tools DBMS users have come to depend on

These read like a quote from The Innovator's Dilemma. People enjoy the benefits of MapReduce for large scale data processing because of these points. The lack of support for these DBMS features and tools are the reason it scales like a mother fucker. That is the feature people want. And it does it 10x better and cheaper than anything else. Those other things simply don't matter to them. It's a new audience and a new market.

Animation, Pivot and Maya

2007-12-09T19:23:00.000-08:00

Over the past few months I have been volunteering in my son's gradeschool class to teach art and animation using computers. The class has several computers in the room and the school has a good computer lab with lots of equipment, and of course most of the kids have a computer at home - quite a change from when I was growing up.

Before I started, I did a little research into available programs for art, sound and animation. There aren't a lot of solid applications freely available, but here are the ones that I found to be interesting:

ArtRage - beautiful digital oil painting

Pivot - stick figure animation (very addictive)

Scratch - visual programming of sprites from MIT (essentially visual logo)

Microsoft Movie Maker - simple track-based video and audio compositing

I found one sound effects generator but haven't found the perfect sound effects program appropriate for these kids. Yesterday I found Reaper, a full featured audio track and effects program - it looks wicked cool, but complicated for gradeschoolers.

The kids have amazed me by picking it all up really quickly. Several of the kids already knew about the Pivot cell animation program, and others have already made stop motion animation with Legos, digital cameras and Microsoft Movie Maker. Normally I would have expected the boys to be the ones more into working with the computer, but in this class everybody is very excited and has been diving in. All the kids are trying their hand at different skills - some are more comfortable creating a background painting in ArtRage, but still make a go of animation in Pivot. Others quickly grabbed some background photos off the Web and brought them into Paint.net and faded them out to make a good background.

The teacher has told me that she has to shoo them out during recess because they'd rather work on their animations than play outside! Makes me feel so proud...

Right now I have them working on a project in teams of two. To explain why working in teams is a good thing to learn, I told them of time I visited Pixar and learned how they used teams - an 'artist' and an 'engineer'. They got a kick out of my describing walking the halls and looking at pencil sketches of a cowboy and a spaceman - I thought the movie idea was great but didn't know if it would have mass appeal. I still regret not having the gumption to ask for one of those sketches - I was so in awe of meeting Ed Catmull at the time, it was hard to speak! I'm sure I made a fool out of myself...

Recently I've thought about introducing the class to 3D graphics, but it can be very time consuming to create the models and I haven't found a good animation program for use with pre-existing models. I did recently download the personal edition of Maya, and maybe there's something in there but I'm guessing it'll be too much for the kids.

Edgeio shutting down

2007-12-07T15:46:00.000-08:00

Looks like Edgeio is shutting down. That's too bad, I liked the idea of supporting decentralized listings of offers. I suppose being a centralized aggregator wasn't the way to go.

Good quotes from Michael Arrington's (a co-founder) comments:

# Andrew
December 7th, 2007 at 12:31 am
what exactly did you spend 5 million dollars on?

# Michael Arrington
December 7th, 2007 at 12:32 am
Andrew - parties, scotch, hookers, blow. you know, the usual.

The Old Web = Ten Million Catalogs

2007-11-19T19:23:00.000-08:00

Stowe Boyd always has something interesting and thought provoking to share on his /Message blog. If you are interested in the intersection of technology and society, I highly recommend giving a little attention.

Today's post is
The Old Web = Ten Million Catalogs (part of his The Social Web: What's The New Web Worth series) which looks back at the past ten years of the Web to draw out the distinction of what the new Social Web has become.

These services are based on a catalog metaphor, where sellers can offer goods or services, and buyers (generally consumers) can find them and acquire them. The volume and low overhead of online services hollowed out the markets in most areas thet they touched, for example, sideswiping brick-and-mortar bookstores, blowing up the travel agent business, and strongly cratering the head hunter marketplace.

I like where this is going - pointing out a 'catalog metaphor' for Web services helps me describe to people how my current company isn't just a directory of people - it's a way to add a social dimension to the Web, centered on people not pages.

Steve Vinoski’s Blog

2007-11-17T21:01:00.000-08:00

Steve Vinoski is one cool dude. I've always had the utmost respect for IONA and even after leaving a while back, Steve continues to show the attitude of professional engineering that garnered that respect. With his measured explanation of his view of REST and Dare's theory and practice post I think I'm ready to walk away from cooling embers of the dying REST .vs. SOAP flame war. Time to unsubscribe from rest-discuss.

From Steve's blog, here are some good quotes:

People seem to get really upset when I say that the static typing benefits of popular imperative languages are greatly exaggerated, and when I say that developing real, working systems in dynamic languages is not only possible, it’s preferable.

Either way, no interface definition language is ever going to keep you or some other real live person from having to figure out what the service actually does and how to actually use it, and then coding your client accordingly.

Remember, REST is an example of applying well-chosen constraints to achieve desired architectural properties for a broad class of distributed systems, and so that’s what its constraints are all about.

Either way, anti-REST folks commonly claim that REST’s success is due only to the fact that there’s a human-driven browser in the mix, but that’s one of the dumbest things I’ve ever heard.

Having a solid thinker like Steve Vinoski blogging makes the Web a better place. Can't wait to hear the details of what he's working on.

Amazon PR: Neither Open Nor Social

2007-11-15T21:25:00.000-08:00

Looks like there was a bit of a messaging snafu between Marshall Kirkpatrick and Amazon and the results are not pretty - ouch. There's messaging then there's messaging. Let's see what the 'go-forward strategy' is...

Is this just communication gone awry, or is there a culture clash looming?

From Read/WriteWeb
- Amazon PR: Neither Open Nor Social

Did you know that there's been no RSS feeds for top selling items in categories at Amazon.com? Well, there is now - and they were so excited that they figured it out, that they wrote it up in a press release.

Nielsen on Generic Commands

2007-11-04T22:10:00.000-08:00

I ran across this interesting bit from Jakob Nielsen's UseIt site about user interface design. It's about using the same few commands in a UI and is an interesting twin to the 'uniform interface' aspect of REST. Although a 'user interface' and a 'network interface' are very different beasts I can't help but wonder if there's some fundamental reason for the similarity.

Summary: Applications can give users access to a richer feature set by using the same few commands to achieve many related functions.

In application design, there's a tension between power and simplicity: Users want the ability to get a lot done, but they don't want to take the time to learn lots of complicated features.

OpenSocial - what? no logo?

2007-11-04T11:17:00.000-08:00

Everybody has been commenting on the news of the week - Google and MySpace spinning out their own system for applets within social networks called OpenSocial.

It seems everyone has missed the biggest gap in the OpenSocial system - they have no logo! Rather than talk about the disappointing lack of open content in their 'extension' to Atom for profile data (anyone heard of hCard or FOAF?) or the lack of anything innovative like client-side includes (that might actually lead to social networks being spidered by a big search engine) I've decided to contribute my l33t grafx skillz to the community in the form of an OpenSocial logo. It includes the mandatory Web 2.0/startup color scheme of blue and orange. I need to figure out how to include the RSS radio waves into the design somehow...

or this...

Next, a motto : "Where do I belong today?"

Armadillo Aerospace almost wins prize

2007-10-28T19:12:00.000-07:00

So close... Armadillo Aerospace had a couple good flights but tipped over just before touchdown and the next day caught on fire on takeoff. They were this close to winning the Level 1 prize in the Lunar Lander Challenge.

More about the Lunar Lander Challenge over here.

Friday humor

2007-10-05T16:05:00.001-07:00

Gotta love inner-thought-humor on a sunny Friday.

You want four million users by DECEMBER?? You have four hundred active licenses for your product currently! Nothing - and I mean NOTHING - is going to add four zeros to the end of that number in three months short of hiring Arthur Anderson to handle the bookkeeping.

Wait... First you wanted to clone Digg... Then you wanted to "add the social aspects of Facebook to it," and NOW you want it to be Wikipedia? Where the HELL did you spend your morning? In the "Web 2.0 Company Names to Memorize" symposium sponsored by the local Linux Enthusiasts club?

Uh... Four million active users means minimum 20,000 concurrent users at any given moment, and you want to do all of this on ONE co-located virtual server in India? On .Net and MS SQL Server? Honestly? You really, really think that's how it will go? In that case, can I punch you? Please? I mean, I only ask because you seem like the type of person who'd ponder the question and then just blurt out "Yes," and I've been dying to hit something since I pressed "1" to join your conference.