Data-oriented Networking

August 1st, 2002

Internet Engineering Task Force
Expires March 2003

		       Data-oriented networking

			   by Brent Baccala
			     August, 2002

This document is an Internet-Draft and is subject to all provisions of
Section 10 of RFC2026.

Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups.  Note that other
groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time.  It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at

The list of Internet-Draft Shadow Directories can be accessed at


   Differentiates between connection-oriented and data-oriented
   networking, identifies the advantages of data-oriented networks,
   argues that Internet web architecture is becoming more
   data-oriented, and suggests ways of encouraging and accelerating
   this trend.

Contemporary Internet architecture is heavily connection-oriented.  IP
underlies almost all Internet operations, and its fundamental
operation is to deliver a data packet to an endpoint.  TCP uses IP to
sequence streams of data packets to those endpoints; higher-level
services, such as HTTP, are built using TCP.  All of these operations
are based upon the underlying IP addresses, which identify specific
machines and devices.  Even UDP operations are connection-oriented in
the sense that UDP addresses identify a specific machine on the
Internet with which a connection (even just a single packet) must be
established.  Note that I use the term connection-oriented in a
somewhat different sense than the traditional distinction between
connection-oriented and connection less protocols.

More recently, Uniform Resource Locators (URLs) have emerged as the
dominant means for users to identify web resources.  The distinction
is not merely one of introducing a new protocol with new terminology,
either.  URLs are used to name blocks of data, not network devices.
Especially with the advent of caching, it's now clear that a web
browser may not have to make any network connections at all in order
to retrieve and display a web page.  "Retrieving" a URL differs
significantly from opening an HTTP session, since an HTTP session
implies a network connection to a named device, while accessing a URL
implies only that its associated data (stored, perhaps, on a local
disk) is made available.  HTTP, SMTP, ssh, and other TCP-based
protocols are inherently connection-oriented, while the URL is
inherently data-oriented.

The Internet is moving away from a connection-oriented model and
becoming more data-oriented.  Since the original Internet design was
modeled, at least loosely, after a telephone system, all of its
original protocols were connection-oriented.  Increasingly, we're
becoming aware that often a user is not interested in connecting to
such-and-such a computer, but rather in retrieving a specific piece of
data.  Since such operations are so common, Internet architects need
to recognize the distinction between connection-oriented and
data-oriented operations and design networks to support both well.
Data-oriented models will not replace connection-oriented models;
sometimes, you'll still want to make the telephone call.  Rather, the
pros and cons of each need to be understand, so both can be
incorporated into the Internet of the 21st century.

To understand the emergence of data-oriented networking, it is useful
to consider the historical development of the Internet.  Initially,
the driving application for what became the Internet was email,
allowing virtually instantaneous communications over large distances,
FTP and TELNET were second and third.  FTP provided file transfer and
a rudimentary publication system; TELNET extended the 1970s command
line interface over the network, letting people "log in" over the net,
thus allowing remote use and management of computer systems.

Even in these early years of the Internet, the network was becoming
more data-oriented than a cursory examination of its protocols would
suggest.  FTP archive sites, such as uunet and wuarchive, began
assembling collections of useful data, including programs, protocol
documents, and mailing list archives in central locations.  Other
sites began mirroring the archives, so that retrieving a particular
program, for example, did not require a connection to a centralized
server for the program, but only a connection to a convenient mirror
site.  The practice continues to this day.  Of course, accessing the
mirror sites required using the connection-oriented protocols, and the
process of finding a mirror site or archive that contained the
particular program you wanted remained largely a manual process.  It
still does.

A significant change occurred during the 1980s - the appearance of
graphical user interfaces (GUIs) in personal computers by the end of
the decade.  In the early to mid 90s, the world wide web extended the
GUI over the network, much as TELNET had extended the command line
interface over the net.  More than anything else, the web represents a
global GUI, a means of providing the commonly accepted point-and-click
interface to users around the world.

It is impossible to understate the impact of the web.  The GUI was a
critical technology that made computers more accessible to the average
person.  No longer did you need to type cryptic instructions at a
command prompt.  To open a file, represented by a colorful icon, just
move a pointer to it and click.  Yet until the web, you still needed
to use the old command-line interface to use the network.  Your
desktop PC might use a GUI, but connecting to another computer
generally meant a venture into TELNET or FTP.  The web extended the
GUI metaphor over the network.  Instead of learning FTP commands to
retrieve a file, you could just browse to a web site and click on an

Other technologies could have provided a network GUI, but not as well
as HTML and HTTP.  X Windows certainly was designed specifically with
network GUI applications in mind, but provided so little security that
using it to "browse" potentially untrusted sites was never realistic.
AT&T's Virtual Network Computing (VNC) is similar to X Windows, and is
designed so that its effects can be confined to a single window.  With
some extensions, it could be used as the basis for a network GUI.
However, both X Windows and VNC share a single common major flaw -
they are connection-oriented protocols that presuppose a real-time
link between client and server.  The user types on the keyboard or
clicks on a link, then the client transmits this input to the server,
which processes the input and sends new information to the client,
which redraws the screen.  X Windows has never been widely used over
the global Internet, because the bandwidth and delay requirements for
interactive operation are more stringent than the network can
typically provide.  VNC is very useful for using GUI systems remotely,
but still doesn't provide the performance of local software.

The present HTML/HTTP-based design of the web does have one
overwhelming advantage over X Windows / VNC, however.  The web is
data-oriented, not connection-oriented, or is at least more so than
conventional protocols.  A web page is completely defined by a block
of HTML, which is downloaded in a single operation.  Highlighting of
links, typing into fill-in forms, scrolling - all are handled locally
by the client.  Rather than requiring a connection to remain open to
communicate mouse and keyboard events back to the server, the entire
behavior of the page is described in the HTML.

The advent of web caches changes this paradigm subtly, but
significantly.  In a cache environment, the primitive operation in
displaying a web page is no longer an end-to-end connection to the web
server, but the delivery of a named block of data, specifically the
HTML source code of a web page, identified by its URL.  The presence
of a particular DNS name in the URL does not imply that a connection
will be made to that host to complete the request.  If a local cache
has a copy of the URL, it will simply be delivered, without any wide
area operations.  Only if the required data is missing from the local
caches will network connections be opened to retrieve the data.

Experience with web caches demonstrates that data-oriented networks
provide several benefits.  First, the bandwidth requirements of a
heavily cached, data-oriented network is much less than a
connection-oriented network.  Connection-oriented protocols such as X
windows, VNC, and TELNET presuppose a real-time connection between
client and server, and in fact could not operate without such a
connection, since the protocols do not specify how various user
events, such as keyclicks, should be handled.  All the protocols do is
to relay the events across the network, where the server decides how
to handle them, then sends new information back to the client in
response.  A data-oriented network, which specifies the entire
behavior of the web page in a block of HTML, does not require a
real-time connection to the server.  Having retrieved the data to
describe a web page, the connection can be severed and the user can
browse through the page, scrolling, filling out forms, watching
animations, all without any active network connection.  Only when the
user moves to another web page is a connection required to retrieve
the data describing the new page.  Furthermore, since the data
describing the pages is completely self-contained, no connection to
the original server is required at all if a copy of the web page can
be found.  A copy, stored anywhere on the network, works as well as
the original.  As the network becomes more data-oriented, fewer and
briefer connections are required to carry out various operations,
reducing overall network load.

A data-oriented network is also more resilient to failures and
partitions than a connection-oriented network.  Consider the
possibility of a major network failure, such as the hypothetical
nuclear strike that originally motivated the Defense Department to
build the packet-based network that evolved into the Internet.  Modern
routing protocols would probably do a fairly good job of rerouting
connections around failed switching nodes, probably in a matter of
minutes, but what if the destination server itself were destroyed?
The connection would be lost, and no clear fallback path presents
itself.  The obvious solution is to have backup copies of the server's
data stored in other locations, but creating and then finding these
backups is currently done by hand.  Existing routing protocols can
reroute connections, but are woefully inadequate for rerouting data.

A more mundane, but far more common scenario is the partitioned
network.  Simply operating in a remote area may dictate long periods
of operation without network connectivity.  In such an environment,
it'd be convenient to drop any information that might be needed on a
set of CD-ROMs.  That works fine until the first search page comes up
that connects to a specialized program on the web server, or a CGI
script that presents dynamic content, or an image map.  Solutions have
been developed to put web sites on CD-ROMs - none of them standard,
most of them incomplete.  A more data-oriented design, that didn't
depend on connections to a server, would be far better suited to such

HTML, the workhorse protocol of the web, was never designed with use
as a network GUI in mind, even though this is the role it has evolved
into.  It's the HyperText Markup Language (HTML), and hypertext is not
a GUI.  Hypertext is text that includes hyperlinks.  Perhaps we can
expand the definition somewhat into a "hyperdocument" that can include
colors, diagrams, pictures, and even animation.  A GUI is much more
than a hyperdocument, however.  A GUI is a complete user interface
that provides the human front end to a program.  Not only can it
include dialog boxes, pull down menus and complex mouse interactions,
but more than anything else it provides the interface to a program,
which could perform any arbitrary task, and is thus not just a
document.  The program could be a document browser, a document editor,
a remote login client, a language translator, a simulation package,
anything.  What was pioneered by Xerox PARC, deployed by Apple Lisa,
marketed by Macintosh and brought with such stunning success to the
masses by Microsoft was not hypertext, but the GUI.  The GUI is what
we are trying to extend across the network, not hypertext, and thus
HTML just isn't very well suited for the task.

Since it wasn't designed to provide a network GUI, HTML doesn't
provide the right primitives for the task it has been asked to
perform, and thus we've seen a long series of alterations and
enhancements.  First there was HTML 2, then HTML 3, then HTML 4, now
HTML with Cascading Style Sheets, soon XHTML, plus Java applets,
Javascript, CGI scripts, servlets, etc, etc...  The fact that HTML has
had to change so much, and that the changes require network-wide
software updates, is a warning sign that the protocol is poorly
designed.  The problem is that HTML has been conscripted as a network
GUI, though, to this day, it has never been clearly designed with this
goal in mind.  Part of what is needed is a replacement for HTML
specifically designed to act as a network GUI.

In addition, one of the great challenges to a data-oriented model is
dynamic pages.  Presently, web caching protocols provide means for
including meta information, in either HTTP or HTML, that inhibits
caching on dynamic pages, and thus forces a connection back to the
origin server.  While this works, it breaks the data-oriented metaphor
we'd like to transition towards.  To maintain the flexibility of
dynamic content in a data-oriented network, we need to lose the
end-to-end connection requirement and this seems to imply caching the
programs that generate the dynamic web pages.  While cryptographic
techniques for verifying the integrity of data have been developed and
are increasingly widely deployed, no techniques are known for
verifying the integrity of program execution on an untrusted host,
such as a web cache.  Baring a technological breakthrough, it seems
impossible for a cache to reliably run the programs required to
generate dynamic content.  The only remaining solution is to cache the
programs themselves (in the form of data), and let the clients run the
programs and generate the dynamic content themselves.  Thus, another
part of what's needed is a standard for transporting and storing
programs in the form of data.

An important change in moving to a more data-oriented network would be
to replace HTML with a standard specifically designed to provide a
data-oriented network GUI.  The features of this new protocol:

  1) It must be data-oriented, not connection-oriented.  Thus, the
     protocol must define a data format that can describe GUI behavior
     on a remote system.  HTML already basically does this.

  2) It must programmatic.  The whole point is to eliminate the server
     and replace it with a standard, network-deliverable specification
     of the GUI behavior.  The exact behaviors of GUI interfaces
     vary dramatically, and simply providing an escape mechanism
     to connect back to the server violates the data-oriented
     design goal.  Thus, the protocol must implement powerful enough
     primitives to describe arbitrary GUI behaviors without escape
     mechanisms, i.e. it must support arbitrary programming constructs.

  3) It must be secure.  Since the program may be untrusted to
     the client, it must be limited from performing arbitrary
     operations on the client system.

  4) It must provide GUI primitives, and cleanly interact with other
     GUI applications, such as window managers, and provide features
     such as drag-and-drop functionality between windows.

  5) It must provide backwards compatibility with the existing web.

The programmatic requirement fairly well dictates some kind of
virtual machine architecture, and the obvious candidate is therefore
Java, but Java may or may not be the best choice.  Netscape began
work on a 100% Java web browser, but abandoned this effort in 1998.
Commenting on the demise of "Javagator", Marc Andreesen quipped - "it's
slower, it will crash more and have fewer features... it will simplify
your life".

This misses the point.  We're not trying to build an Java-based HTML
web browser that would simply achieve cross-platform operability.  The
goal is to build a web browser that, as its primary metaphor, presents
arbitrary Java-based GUIs to the user.  HTML could be displayed using
a Java HTML browser.  The difference is that the web site designer
controls the GUI by providing the Java engine for the client to use
for displaying any particular page.  Switching to a different web site
(or web page) might load a different GUI for interacting with that
site's HTML, or XML, or whatever.  Unlike Andreesen's "Javagator", the
choose of GUI is under control of the web server, not tied into a
Java/HTML web browser.

For example, consider if a web site wants to allow users to edit its
HTML pages in a controlled way.  Currently, you have a few choices,
none completely satisfactory.  First, you could put your HTML in an
HTML textbox, and allow the user to edit it directly, clicking a
submit button to commit it and see what the page will actually look
like.  Alternately, you could allow the HTML to be edited with
Netscape Composer or some third party HTML editor on the client,
accepting the HTML back from the client in a POST operation.  This
provides the server very little control over exactly what the user can
and can't do to the page.  Since parts of the page might be
automatically generated, this isn't satisfactory, nor do we really
know much about this unspecified "third party editor".  On the other
hand, with a Java browser, the web site could simply provide a
modified HTML engine that would allow the user to edit the page, in a
manner completely specified by the web designer, prohibiting
modifications to parts of the page automatically generated, and
allowing special cases, such as spreadsheet tables with the page, to
be handled specially.

Another advantage to this proposal is that it provides a solution to a
problem plaguing XML - how do you actually display to the user the
information you've encoded in XML?  This is left glaringly unaddressed
by the XML standards, the solution seeming to be that you either use a
custom application capable of manipulating the particular XML data
structures, or present the data in two different formats - XHTML for
humans and XML for machines.  A Java-based web browser addresses this
problem.  You ship only one format - XML - along with a Java
application that parses and presents it to the user.

On the other hand, let's keep Andressen's criticism in mind. Java may
not be suitable for such a protocol, for either technical or political
reasons.  The speed issues seem to be largely addressed by the current
generation of Just-In-Time (JIT) Java runtimes, but whatever the
standard is, it should be an RFC-published, IETF standard-track
protocol, and if the intellectual property issues around Java preclude
this, then something else needs to replace it.  Alternatives include
Parrot, the yet unfinished Perl 6 runtime, and Microsoft's .NET
architecture, based around a virtual machine architecture recently
adopted as ECMA standard ECMA-335.

PDF also deserves consideration.  Though it lacks the generality to
provide a network GUI, its presentation handling is vastly superior to
HTML's, giving the document author complete control over page layout,
and allowing the user to zoom the document to any size for easy
viewing.  It is also easier to render than HTML, since its page layout
is more straightforward for the browser to understand.

A definite metaphor shift is required.  Rather than viewing HTML as
the primary standard defining the web, the primary standard must
become Java or something like it, that provides full programmability.
Browsing a web page becomes downloading and running the code that
defines that page's behavior, rather than downloading and displaying
HTML, that might contain an embedded applet.

Backwards compatability can be provided along the lines of HotJava,
Sun's proprietary Java-based web browser, which implements HTML in
Java.  To display an HTML page, Java classes are loaded which parse
the HTML and display it within a Java application.  The browser
provides little more than a Java runtime that can download arbitrary
Java and run it in a controlled environment.  Initially, 99% of the
pages would be HTML, viewed using a standard (and cached) HTML engine
coded in Java.

Notwithstanding the creeping featurism present in Java, adopting this
approach would avoid the creeping featurism so grossly apparent in web
browsers.  Even the casual observer will note that mail, news, and
teleconferencing are simply bloat that results in multi-megabyte
"kitchen sink" browsers.  Will the next release of Netscape, one might
ask, contain an Emacs mail editor with its embedded LISP dialect?  And
if not, why not?  Only because the majority of users wouldn't use
Emacs to edit their mail?  Why should we all be forced to use one type
of email browser?  Why should we have Netscape's email browser
packaged into our web browser if we don't use it?  Like the constant
versioning of HTML, the shear size of modern browsers is a warning
sign that the web architecture is fundamentally flawed.  A careful
attempt to standardize "network Java" would hopefully result in
smaller, more powerful browsers that don't have to be upgraded every
time W3C revs HTML; you simply update the Java GUI on those particular
sites that are taking advantage of the newer features.

Another tremendous advantage is the increased flexibility provided to
web designers.  HTML took a big step in this direction with Cascading
Style Sheets, but CSS doesn't provide the power of a full GUI.  For
example, if a web page designer wanted to, he could publish an HTML
page with a custom Java handler that allowed certain parts of the HTML
text to be selectively edited by the user.  This simply can't be done
using CSS.

Network-deliverable, data-oriented GUIs aren't a panacea, or course.
For starters, one of the advantages of the present model is that all
web pages have more or less the same behavior (since they are all
viewed with the same GUI).  The "Back" and "Forward" buttons are
always in the same place, the history function always works the same
way, you click on a link and basically the same thing happens as
happens on any other page.  Providing the web designer with the
ability to load a custom GUI changes all that.  Standards need to be
developed for finding and respecting user preferences concerning the
appearance of toolbars, the sizing of fonts, the operation of links.
The maturing Java standards have already come a long way towards
addressing issues such as drag-and-drop that would have to be
effectively implemented in any network GUI.

Hurdles need be crossed before we can reach a point where web
designers can depend on Java-specific features.  One possibility would
be to migrate by presenting newer web pages to older browsers using a
Java applet embedded in the web page.  Performance might suffer, but
clever design would hopefully make it tolerable.  For starters,
consider that the web page data presented to the applet need not be
the source HTML, but could be a processed version with page layout
already done.  Newer, Java-only browsers should be leaner and faster.

In summary, I recommend the following steps:

1. Recognize the importance of data-oriented design, as opposed to
   connection-oriented design.  Break the dependence on special server
   configurations and realize that the client has to do almost all the
   work in a scalable, cached, redundant web architecture.

2. Migrate the web towards being based on a model of a network GUI,
   rather than a massively enhanced hypertext system.

3. Select a standard for the network delivery of executable content,
   Java being the most likely candidate

4. Develop a Java-based HTML browser along the lines of HotJava, but
   completely open, allowing existing HTML-based websites to be
   browsed via Java.  Provide an applet version that allows web
   designers to specify a custom Java applet to browse their HTML
   sites using conventional web browsers.

5. Develop a lean, fully Java-based web browser, with Multivalent
   being the most obvious candidate.

6. Recognize the transient nature of HTML/HTTP and specify their
   operation in terms of a generic API, based on the network
   executable content standard (probably Java), for finding and
   delivering the GUI presented by a specified URL.

With all the inertia built up behind the present web design, one needs
to question the wisdom of abandoning HTML and completely re-vamping
the structure of the web, even if a migration path is in place.  The
promise of data-oriented networking is a leaner, more reliable, more
efficient network.  Ultimately, if this analysis is correct, then the
cost of migrating away from a flawed design will ultimately be less
than the cost of constantly shoehorning it.  However, in his essay,
"The Rise of 'Worse is Better'", Richard Gabriel suggests "that it is
often undesirable to go for the right thing first.  It is better to
get half of the right thing available so that it spreads like a
virus."  Bearing this in mind, I've proposed an alternate set of
recommendations, aimed at something more immediately practical, in a
companion essay, "Standardized caching of dynamic web content".  At
the same time, I think it's time to take another look at the
Java-based web browser, and to seriously ask if Java isn't a better
choice than HTML for a network standard GUI.


Hypertext Markup Language (HTML) / Extensible Markup Language (XML)

   HTML has a long and sordid history.  HTML version 2, specified in
   RFC 1866, was one of the earliest (1995) documented HTML versions.
   Later revisions added tables (RFC 1942), applets (HTML 3.2),
   JavaScript and Cascading Style Sheets (HTML 4.01).

Universal Resource Locators

   RFC 1630 - Universal Resource Identifiers in WWW

   RFC 1737 - Functional Requirements for Uniform Resource Names, a
      short document notable for its high-level overview of URN

   RFC 1738 - Uniform Resource Locators, a technical document of more
      importance to programmers than architects


   International standard X.509 (not available on-line)

DNS Security

   RFC 2535 - Domain Name System Security Extensions

   RFC 2536 - DSA KEYs and SIGs in the Domain Name System

   RFC 2538 - Storing Certificates in the Domain Name System

   RFC 3110 - RSA/SHA-1 SIGs and RSA KEYs in the Domain Name System


   Postscript specification

   PDF Reference: Adobe portable document format version 1.4
   (ISBN 0-201-75839-3)

   Ghostscript - freely available Postscript interpreter that also
      reads and writes PDF and thus can be used to convert PS to PDF

   Multivalent (see below) includes a Java PDF viewer

   html2ps - a largely illegible Perl script written by Jan Karrman
      to convert HTML to Postscript.  Yes, it can be done.

X Windows

   Developed by an MIT-lead consortium, X Windows is one of the most
   successful network GUIs

VNC (Virtual Network Computing)

   Similar in concept to X Windows, but radically different in
   design - an absurdly simple protocol combined with various
   compression techniques to achieve decent WAN performance


   Java Virtual Machine (JVM) specification

   Bill Venner's excellent Under the Hood series for JavaWorld
   is a better starting point than the spec for understanding JVM.
   He also has written a book - Inside the Java Virtual Machine
   (McGraw-Hill; ISBN 0-07-913248-0)

   Java 2 language reference

   Java languages page

   Criticism of Java

Other virtual machines

   Perl 5 runtime

   Parrot - Perl 6 runtime

   Microsoft's .NET architecture includes the Common Language
   Infrastucture, based around a virtual machine, now adopted as

Various Java-based web browsers

   HotJava, Sun's Java browser, but with binary-only licensing

   Multivalent, an open-source web browser written totally in Java,
   with an extension API to add "behaviors" similar to applets

   NetBeans, an attempt to develop a "fully functional Java browser"

   Jazilla, a now defunct attempt to carry the "Javagator" project
   forward under an open source banner

Java Servlets/WARs

   "Tomcat is the servlet container that is used in the official
    Reference Implementation for the Java Servlet and JavaServer Pages

   Java Servlets - server-side Java API (CGI-inspired; heavily
   HTTP-based) The Java servlet specification includes a chapter
   specifying the WAR (Web Application Archive) file format, an
   extension of ZIP/JAR


   RFC 3040 - Internet Web Replication and Caching Taxonomy
      broad overview of caching technology

   RFC 2186 - Internet Cache Protocol (ICP), version 2

   RFC 2187 - Application of ICP

   Squid software

   NLANR web caching project

   Various collections of resources for web caching
Anonymous surfing – VPN, Proxies, Tor and JonDo IETF Web Intermediaries working group (webi) IETF Web Replication and Caching working group (wrec) RFC 3143 - Known HTTP Proxy/Caching problems Cache Array Routing Protocol (CARP) - used by Squid RFC 2756 - Hypertext Caching Protocol (HTCP) - use by Squid Napster and its variants Napster, the original peer-to-peer file sharing service, has been fraught with legal difficulties, having recently entered bankruptcy Napster's protocol lives on, even if the service is dead. It's basically a centralized directory with distributed data Gnutella has emerged as the leading post-Napster protocol, employing both a distributed directory and distributed data Several popular clients use the Gnutella network and protocol Other proprietary peer-to-peer systems Other free peer-to-peer systems Richard Gabriel, "The Rise of 'Worse is Better'" Brent Baccala, "Standardized caching of dynamic web content"

On Napster

January 1st, 2001, a website facilitating the on-line exchange of digital music, has been highly publicized by the mass media. The legal wrangling over copyright issues has overshadowed other, equally legitimate questions. Is Napster for real, or is it just hype? Are the issues it presents purely legal, or are there technical lessons to be learned, too? What does Napster reveal about the future of the Internet?

First, we need to recognize that Napster represents a real technological advance. It is one of the newest and most prominent examples of a directory service. Directory services are based on the realization that centralized data stores tend to generate performance bottlenecks. All the data being served to the clients has to come from a centralized server or a handful of centralized servers. Throwing bandwidth at the problem is sometimes realistic, but a better solution is to design more efficient networks. Distributing data sources across the network is a major, emerging technique for achieving greater efficiency.

For example, one of the reasons we currently lack decent video-on-demand services are the bandwidth requirements of video. It’s simply not feasible to construct a centralized server to feed two hour movies to a million people. The bandwidth requirements are too great; the centralized server becomes too much of a bottleneck. A Napster-esk solution would be to have thousands of video servers, each capable of serving perhaps a dozen video streams, spread all over the network. Due to the current bandwidth demands of video, this is still unrealistic, but similar schemes are immediately plausible for books, software, and websites in general.

In fact, it’s reasonable to suppose that at least 90% of the present Internet’s traffic is unnecessary. The net is young and rapidly evolving. The protocols currently in use are inefficient, some more so than others. As the network continues to mature, it will become more efficient, and the bandwidth requirements of particular applications will decrease. The present boom in bandwidth demand is driven by new users and new applications. At some point, most people will be “connected”, and the uses of the network will stabilize. From that point onward, improvements in network design will begin to drive bandwidth requirements downward. The network will be most inefficient while it is young, so we can expect bandwidth requirements to peak at some point, I’d estimate within the next two decades, and then begin heading down.

Directory services such as Napster will be instrumental in reducing demand for network bandwidth. Other keys to more effectively using bandwidth are compression, caching, and multicast, all of which are in their infancy. Many issues remain to be addressed, for example, server selection. Napster currently presents the user with a list of servers for each song, one of which is manually selected to download the song from. Developing automated techniques for server selection will be an important step forward in making this technology more seamless, and therefore more attractive for other applications.

Security deserves special mention, since distributing data across the net would seem to seriously compromise security, but this is probably not so. Encrypting the data allows it to be distributed even to insecure servers, which could serve the data, but couldn’t read it. Then, the centralized directory would provide a key that could be used to decrypt and read the data. Controlling access to the key would control access to the data. Typically block cipher keys are only a few dozen bytes long, so access to a 100KB file could be granted by a directory server in less than 1KB – a 100-to-1 savings in centralized bandwidth requirements. The authenticity of the data could be verified by X.509 certificates – placed in the directory, of course.

While Napster represents a real advance over older, more centralized, techniques, this doesn’t mean that the current protocol can’t be improved. Let me outline how I’d redesign Napster, if I were given the task:

  1. Use LDAP. The Lightweight Directory Access Protocol (LDAP) has become an accepted standard for directory service. Furthermore, a “pure” directory service, such as Napster, doesn’t require any special handling on the part of the directory server. All the server has to do is register directory entries, then fed them back out again in response to search requests. A standard LDAP server, such as OpenLDAP, could be used unmodified.
  2. Define and publish a standard schema. LDAP, and directory access protocols in general, use “schemas” to define the format of directory entries. In Napster’s case, a standard schema would probably include a “Song” class, defining artist, title, and year, and perhaps an “Album” class, listing all the tracks on a particular album. The “Song” class could then be extended (subclassed) into a “NetSong” class that would also include URLs where the song can be accessed. Using a standard, published schema would clearly define the directory structure, and make it easier to reuse the directory for new applications.
  3. Use HTTP or FTP. Just as there’s no need to create a custom directory service, there’s no need to invent new file transfer methods, either. Specifying a URL in the directory entry, using one of the standard methods, “http:” or “ftp:”, should suffice. Of course, most “clients” aren’t set up to be “servers”. In the present computing environment, Napster would be quite hard to configure if it relied on an external web or FTP server, and much more complex if it included an entire web server within it. The “peer-to-peer” paradigm ultimately implies that a machine can be simultaneously both a client and a server, and must be configured to act as both. This obviously contrasts with Microsoft’s policy of separate “client” and “server” operating system packages (the “server” usually being much more expensive), but free software hasn’t solved this problem completely, either. How exactly does an arbitrary software program go about registering itself with the local web server in order to share files?

Napster isn’t the first directory based system to be deployed on the Internet, but it is one of the newest and most exciting. If the government and economic leaders can be persuaded to surrender a measure of control, its decentralized nature may pave the way to a more distributed and more efficient network.

Wireless Internet

September 1st, 2000

For a long time, conventional wisdom held that the telephone system was a natural monopoly, or at least a natural oligopoly, because of the need for a large physical infrastructure, namely, the telephone wires. In recent years, though, the widespread deployment of cellular telephones illustrates that this is not necessarily true. Clearly, cell phone networks are capable of handling significant traffic loads and delivering near-landline quality of service. The emergence of all-digital cellular telephones, such as PCS, shows that data can be effectively transferred using wireless, and leads me to wonder; could we build a totally wireless Internet?

I’m not the only person to ask this question. The Ricochet Network has been in operation for several years now, the PalmVII handheld offers wireless Internet access, and companies such as Intel are promoting the Mobile Data Initiative. Yet these efforts are largely technical in nature, while I perceive wireless as changing the fundamental rules of the game. Without dependence on a landline infrastructure, is telecommunications really a natural oligopoly anymore? Could we build a free wireless digital communications system?

Decentralization, not privatization,

should be the buzzword of freedom.

First, what do I mean by the term “free”? Primarily two things – first, the absence of recurring charges, such as monthly or per-minute fees, and second, an open, non-proprietary infrastructure free from patent or regulatory barriers to entry. The only fee I’m prepared to concede is the initial purchase price of the equipment itself. The telephone will ideally become like a computer or a microwave oven – once you purchase it, you can continue using it free of charge.

The capitalist model would be for the communication system to be controlled by companies, driven by competition to improve service, but with access limited by economic barriers, i.e, the phone gets switched off if you don’t pay your bill. The socialist model would ensure access by having the government control the phone system, as well everything else. A better solution than either would be to eliminate the large institutions entirely, by eliminating the centralized infrastructure. Decentralization, not privatization, should be the buzzword of freedom.

A decentralized communications infrastructure would have to be based primarily, if not totally, on wireless radio technology. Any non-wireless service, such as the existing telephone system, would require landlines connecting to a central office, implying right-of-ways, centralized ownership of the lines, and consequent dependence on large institutions, either governments or corporations. Only a wireless scheme would allow devices to communicate directly, without any centralized infrastructure.

Current wireless technology (i.e, cell phones) still rely on centralized infrastructure. Sadly, they’re designed to rely on it. A cell phone communicates exclusively with a radio tower, which then relays the call to its destination, typically over landlines. As the cell phone moves, it switches from one tower to another, but never communicates directly with another cell phone. I can’t make a phone call from one car to the next without going through a radio tower.

What’s needed is a new kind of wireless infrastructure – one where the telephones and computers are designed to communicate directly with each other, without relying on a phone company’s switches. Clearly, if I have such a wireless telephone, and my next door neighbor has such a telephone, they can directly connect and we can talk without any dependence on third parties, and consequently without any recuring charges.

Yet, the big question remains – can I call from Maryland to California with such a telephone? Hopefully, the answer is yes, and the key lies in the routing technology of the Internet. A typical Internet connection will be relayed through a dozen or more routers. No single connection exists between the source and the destination, but by patching together dozens of connections, a path can be traced through the network for the data to flow through. Network engineers have spent literally decades developing the software technology to find these paths quickly. Theoretically, there is no single point of failure in the system, since the routers can change the data paths on-the-fly if some part of the network fails. The keys to making it work are the adherence to open standards, such as TCP/IP, and the availability of multiple redundant paths through the network.

A wireless infrastructure can be built on similar technology. If I can call my neighbor’s telephone directly, and my neighbor’s telephone can reach the grocery store’s telephone directly, they I can call the grocery store by relaying the data through my neighbor’s telephone. If adequate bandwidth is designed into the system, my neighbor’s telephone can relay the data without any impairment to her service. She can be talking to her hairdresser without even knowing that her phone is relaying my conversation with the grocery clerk.

What stands in the way of building such a free, national digital communications infrastructure?

First, the presence of standards. Just as English is the standard used by the author and readers of this document, and TCP/IP is the standard used by the Internet devices that relay the document, standards are required for the wireless devices to communicate. IEEE recently standardized a wireless data LAN (802.11) capable of handling 1 to 2 Mbps. To put this into perspective, an uncompressed voice conversation requires 64 Kbps. Thus, a 1 Mbps circuit could handle 15 such conversations. Not only can I talk to the grocery store while my neighbor talks to the hairdresser, but a dozen other people can use the same circuit with any service impairment. Newer compression techniques can improve this performance by a factor of ten.

IEEE 802.11 is a good start, but the power limitations imposed by FCC regulations may impede its use for any but short-range applications. However, Metrocom’s Ricochet network demonstrates that this might not be a show stopper. Working in conjunction with power companies, Metrocom pioneered the novel approach of putting low-power radio repeaters on existing utility poles. The repeaters communicated directly with each other, eliminated the need for a landline data connection; only power was required, which was readily available on the pole. A similar approach could be used to build a network that would provide 802.11 coverage to an entire metropolitan area.

Also, the existing 802.11 devices aren’t very sophisticated in their design. They’re designed to be cheap, not effective. Their single most glaring problem is their antenna design. Existing 802.11 transceivers use mainly low-gain, omnidirectional antennas, although Raytheon recently announced the availability of an 802.11 PCMCIA card with a jack for connecting an external, hopeful better, antenna. Improved antennas will probably take one of two forms. Adaptive arrays are preferred by the military, and justly so, but are complex and expensive. Directional arrays, typified by TV aerials, are simpler and therefore cheaper, but must be physically pointed at their destination. One possible scenario would be for routers to use the more expensive adaptive arrays, and for end systems to use mechanically steered antennas. In my opinion, the development of improved 802.11 devices is the single most important advance needed today.

Second, an initial infrastructure is required. A 802.11 telephone would be a popular item if everyone else had one, but initially few people would possess such devices, making it difficult if not impossible to route a connection through such a sparse matrix. Philanthropies could be formed to build infrastructures. A Ricochet-type network could be deployed in cooperation with power companies, who might be persuaded to donate the relatively small amounts of electricity the routers would consume. After an initial investment in the (hopefully) rugged and scalable pole-top devices, the entire network could be managed from a central location. At this point, the network would provide 802.11 service to an entire metropolitan area, jump-starting the service. As more and more people bought these devices, each capable of relaying traffic on its own, the dependence on the initial infrastructure would diminish, hopefully to the point where the pole-top devices wouldn’t need replacement when they started to fail.

Furthermore, users would want to call telephones on the conventional phone network, requiring some sort of gateway. A solution to this chicken-and-egg problem would be to provide mechanisms for some fee-based services. Thus, a service provider could construct a network that would, for a monthly fee, interconnect its users and provide gateway service to the existing phone network. It’s possible that the only fee the provider would need to charge would be for the gateway service – initially, almost all connections would go through the gateway, since few people would have the new phones and most calls would be relayed onto the existing phone system. As the wireless network became more and more widely deployed, more and more destinations would go wireless, and the reliance on gateway systems would diminish.

In short, wireless is in its infancy. This exciting new technology offers great possibilities not just to expand existing phone and data networks, but to break down the old service models and replace them with newer, more decentralized designs. The oft-touted idea of the free phone call might even become a reality.

Smart Cars

August 1st, 2000

Roughly 40,000 Americans die every year in automobile accidents. Thousands more are maimed and injured. Many of these accidents are not caused by alcohol, speed, or reckless driving, but by simple driver error. For example, even if drunk driving were totally eliminated, cutting car fatalities roughly in half, automobile accidents would still be the leading cause of accidental death in this country.

Let me illustrate this point by personal example. My two most serious car accidents both occured while I was stone cold sober. One accident happened because I was tired and fell asleep at the wheel. Fortunately, it was just after dawn, there were few cars on the highway, and though my car spun out of control at around 60 mph, it didn’t flip over and came to a stop in a drainage ditch without hitting anything or anyone. In the second accident, I was making a left turn at a traffic light with a green arrow. An elderly gentleman ran the opposing stop light and broadsided me. Though I was not at fault, I can honestly say that if I’d bothered to look around, I’d have seen the other car and been able to stop in time. Instead, I was coming home from work, doing the same thing I did every other day at 5 PM, probably more concerned about finding a decent song on the radio, so when the light turned green, I just hit the gas and went.

Automobiles are simply dangerous. Air bags, lower speed limits, and mandatory seatbelt laws have reduced the fatalities but do not offer any ultimate solutions. Undoubtedly, drivers need to constantly remind themselves how dangerous this everyday activity is, and reduce distractions such as fatigue, intoxication, and car phones. Operating an automobile requires long periods of boring, repetitive work, interrupted rarely by unannounced moments requiring instant, intensive concentration. Humans by nature are not well suited to this kind of task, but it’s just the kind of thing computers excel at, so to significantly reduce automobile fatalities, computerizing the operation of cars has to be seriously considered.

Proposals for computerized automobiles have been around for some time. TRW prepared an story imagining the possibility of a computerized automobile system by 2012. In 1997, a modified stretch of California’s Interstate 15 served as the testbed for a series of demonstrations with automated cars and buses. The U.S. Department of Transportation has an Intelligent Transportation Systems Joint Program Office that coordinates many of these efforts. The Intelligent Transportation Society of America (ITS America) organizes conferences, maintains a website, and publishes a regular newsletter. Unfortunately, the goal is still distant, and many current ITS efforts are focused on programs such as more sophisticated traffic signals and alleviating congestion with automatic toll collection. The DOT-funded National Automated Highway Systems Consortium, which oversaw the I-15 tests and intended to develop a prototype system by 2002, has been terminated.

The highway in California was modified by placing magnetics in the roadway, which the vehicles then followed. Much important work has also been done on vehicles operating on unmodified roadways. CMU’s Robotics Institute developed a series of vehicles (the Navlabs) which drove from Pittsburg, PA to San Diego, CA under computer control for 98% of the trip. Navlab would make an excellent starting point for a smart car of the future, since CMU has already developed a controller system to operate the car, a standard API for operating the controller, and a simulation environment for testing new controller programs.

Some look to large corporations like GM or Toyota to design the smart car of the future. Others expect the initiative to come from the U.S. federal government. According to the U.S. Department of Transportation’s Intelligent Vehicle Initiative Governance Structure, the Enabling R&D group is only open to “vehicle OEMs with a World Manufacturer Identifier… and will require contribution of substantial financial resources” As a free software aficionado, I’d rather see a initiative to build an open, co-operative system in which governments, large companies, small organizations, and individuals can all contribute. The ability of the Internet community to develop complex, open source software systems, such as Linux, demonstrates the feasibility of using the Internet as a basis for collaboration.

To minimize the infrastructure requirements, the system would have to interoperate with ordinary cars on unmodified highways, basically Navlab’s approach to the problem. Much of the hardware required to support an automated car is already available:

Computer platform. The modern laptop computer seems well-suited to support a future smart car. It offers ample processing power and disk space, can operate off 12 VDC power, and is well standardized. PCMCIA cards provide a convenient and standard hardware interface. Slight modifications, such as a detached display to be placed on the dashboard, wouldn’t be difficult to implement.

Radiolocation. GPS (Global Positioning System) can’t provide enough accuracy to locate a car within a lane, but can locate a car within a dozen meters or so. Furthermore, GPS technology is mature, readily available at low cost, and easily integrated with existing computer technology. For example, Premier Electronics markets the SatNav GPS Receiver, a PCMCIA GPS receiver.

Communications. For long range communications, cellular telephones and cellular modems are expensive, but well understood and widely deployed. Upon locating itself with GPS, the car’s computer could dial into a server and download a database for the surrounding area. For short range communications, between nearby vehicles and traffic signals, the IEEE 802.11 wireless LAN standard is newer, but available as off-the-shelf, unlicensed products that work well within roughly a hundred meters.

Video capture. Small video cameras are commonly connected to video capture cards, providing a ready base for visual sensor systems. Navlab’s No Hands Across America demonstration relied heavily on their video-based RALPH system to follow highway markings. PCMCIA video capture cards can be connected to off-the-shelf miniature CCD cameras to provide a readily available video capability.

Radar. No really adequate automobile-based radar system exists today. Advances in microstrip fabrication technology, such as the ready, cheap availability of dielectric resonant oscillators, allows gigahertz-wavelength devices to be fabricated on a conventional PC board. It should now be possible to mass produce a low cost (sub $500) radar system to mount on an automobile and scan for other cars and pedestrians within a hundred meters. If you don’t believe this, check out my essay Guardian Alert: How it works for a description of a simple radar system and an outline of how it could be adapted for vehicular use.

Vehicle control. The development of Linux would have been impossible without the IBM PC – a standard, widely available hardware platform that software designers across the planet had ready access to. Likewise, to build an open source smart car, a standard software API needs to be developed for issuing commands like “drive forward at 20 mph”, “right turn 10 degrees”, “stop”. At least one model hardware implementation needs to be made readily available in kit form, probably using a USB serial interface, and simple enough that an average auto mechanic could install it on an automatic transmission car.

The most important thing now is to collect together the available technology, publish it on a website, and launch a collaborative effort to synergize the talents of the Internet community. An open-source version of CMU’s Navlab would make an excellent start.

Personal Robotics

July 1st, 2000

We stand today at the culmination of the industrial revolution. For the last four centuries, rapid advances in science have fueled industrial society. In the twentieth century, industrialization found perhaps its greatest expression in Henry Ford’s assembly line. Mass production affects almost every facet of modern life. Our food is mass produced in meat plants, commercial bakeries, and canaries. Our clothing is shipped by the ton from factories in China and Taiwan. Certainly all the amenities of our lives – our stereos, TVs, and microwave ovens – roll off assembly lines by the truck load.

Yet we’ve paid a price for industrialization. Our entire society has become a giant assembly line. Each factory’s small assembly line is but one piece in a larger assembly line, interconnected by trucks and trains. Oil refineries feed chemical plants feeding ceramics factories feeding automobile plants. Most seriously of all, the individual has been reduced to a cog, and money is the oil that lubricates the machine. Introduce too much friction, and you’re removed, discarded, and replaced. Pop music reverberates with themes of alienation and depression. In the U.S., almost as many people kill themselves every year as die from automobile accidents. People feel overwhelmed by a mass society dominated by big business and cold economics.

These problems aren’t just the “way the world is”, either. They’re the way our world is; they’re the nature of our industrial civilization. Other civilizations granted more autonomy to the individual. American Indians raised their children to be self-sufficient by teaching them how to recognize wild edibles, how to build bows and arrows from saplings and sinew, how to fashion tepees from animal hides. By the time they were twenty, young Indians could literally walk out into the woods and take care of themselves. In feudal Europe, people may have been more dependent than the early Americans, but society was predominately rural, and the small towns that dotted the landscape were largely self-sufficient. I’m not suggesting we go back to living by bow and arrow, but by studying the nature of other civilizations, we can better understand the advantages and shortcomings of our own.

Over the course of the industrial revolution, many have noted these problems. The socialist solution was complete centralization in the hands of government, either voluntarily (the Utopian Socialists) or violently (the Communists). “Blow up the factories!” was the Luddite cry, and the transcendentalists urged us to return to nature. A few decades ago in this country, many people looked to a government welfare system that latter collapsed in a mass of red ink and red tape. In fact, for one reason or another, all these solutions failed. We may read Thoreau’s poetry, but few of us are ready to move to a pond and live in a log cabin.

Today, we’re presented with another solution, that hopefully will fare better than its predecessors. It goes by the name of post-industrialism, and is commonly associated with our computer technology. What is post-industrialism, and how can it solve our problems? Let me illustrate with an example.

Consider an author, who writes a book and then desires to publish it. In the industrial model, a printing factory is needed to mass produce the book. In fact, several factories are needed. The printing presses require paper, which is made in a paper mill (factory); inks, solvents and glues, made in chemical plants (factories); and an elaborate transportation system of trains, planes, and trucks to transport the raw materials to the presses and the finished books to distributors. Several hundred or even a few thousand people may be required, directly and indirectly, just to get a single book published. Of course, the only way to justify this much effort is to produce not just one book, but millions, the so-called “economics of scale”. Most authors are therefore dependent on a publisher, who is unlikely to go to all this trouble unless he thinks the author’s book can turn a profit. This is the industrial model.

In the post-industrial model, the author writes the book using a computer, prepares it in electronic form, and uploads it to a web site. Now anyone with an Internet connection can read the book. No factories required. The author is no longer dependent on a publisher, is much freer in what he can write, and has achieved a real degree of liberation. Economics of scale have been replaced with an economics of information, letting one individual reach an entire planet.

Of course, this is no panacea. For starters, the author is dependent on a new kind of infrastructure – the data network used to deliver the bits and bytes, and its routers, switches, and servers, all of which, incidentally, are still made in factories. The world may be going post-industrial, but is still heavily industrialized.

Furthermore, while the Internet may liberate the author, it’s hard to see how you can download a stereo, a bed, or a car. Yet consider how a car gets built, at least in Japan’s robot-dominated plants. The robots are operated by computers, which are controlled by software, which can be downloaded across a data network, even if the cars can’t be. The cars can’t be transported across fiber optics, but the “smarts” that drive their construction can be.

Robots are today where computers were 25 years ago. They’re huge, hulking machines that sit on factory floors, consume massive resources and can only be afforded by large corporations and governments. A former president of IBM once remarked that he foresaw a world market for five computers. Then came the PC revolution of the 1980s, when computers came out of the basements and landed on the desktops. So we’re on the verge of a “PR” revolution today – a Personal Robotics revolution, which will bring the robots off the factory floor and put them in our homes and on our desktops.

Ultimately, you might have small robots to perform tasks like cooking breakfast, larger robots that could construct a stereo or repair a microwave oven, and really big robots, perhaps one or two in a town, that could build an automobile. Specialized robots could perform tasks such as micro chip fabrication or casting industrial ceramics. The software needed to performs all of these tasks could be downloaded via data networks. Just as an author can write a book today, and with a few keystrokes make it available to the world, so tomorrow an engineer could design a better mousetrap and then “ship” it in a second to be replicated by robots in every time zone. Gardening robots could raise our food, letting us feed ourselves without being dependent on Safeway, and giving us the freedom Thoreau dreamed of, while our machines handle the hoes.

The combination of personal computers and personal robots offer an awesome potential to break down the assembly line and put the means of production into the hands of the individual. However, the computer industry has sadly demonstrated its ability to centralize and control technology through copyright restrictions and secret source code. Instead of downloading a motorcycle repair program with a few clicks, you may be presented with a e-commerce form requesting your credit card number first.

Fortunately, the free software community has demonstrated that determined individuals, through sacrifice and hard work, can build open systems to replace and improve on proprietary ones. Free software developers need to take the lead in advancing this new robotic technology, preventing it from becoming another weapon to control people’s lives, and instead fulfilling the promise of post-industrialism to liberate mankind from the assembly line.

How do we get there from here? Much of robotic hardware technology is commonly available today. Video capture cards, which can give a computer “eyes” when connected to a video camera, have been on the market for several years. While arms and grabber hands might not be as common, the mechanics behind their construction is fairly simple and well understood. Here’s a short list of major milestones:

  1. Standard robot manipulator arm. Just as a PC hardware standard was needed before Linux could be written, a standard robot arm is needed to facilitate cooperation between software developers. Arm must be able to perform a set of benchmark tasks under direct manual control. Mechanical design and construction details published on-line.
  2. Record-and-playback programming interface. Robot arm can repeat programmed tasks. Able to deal with slight variations in position and orientation of objects.
  3. Specialized robot work cells. Tasks such as micro chip fabrication will require specialized support hardware such as vacuum chambers. Other specialized tasks include chemical processing, and fabrication of plastics and ceramics. Other specialized work cells will be developed as needed to achieve later milestones.
  4. Self-replication. Robot arm can build a working duplicate of itself using specialized work cells. A major milestone, comparable to a compiler being able to compile itself.
  5. Work cell replication. Robot arm can build all specialized work cells needed for its self-replication.
  6. Computer replication. Robot arm can build a computer capable of controlling it.

Having achieved these milestones, we’ll have constructed a robotic system able to duplicate both itself and the computer needed to control it. Further refinement of this technology will allow ever simpler raw materials to be input into the construction process. A robot capable of building a duplicate of itself will no doubt be sophisticated enough to be used for many other tasks, and provide a starting point for tasks it’s yet incapable of doing.

Science fiction writers have imagined such possibilities for decades, just as they once imagined men flying to the moon. If the Apollo space program is any indication, we may only be waiting for a Jack Kennedy to lead us forward.