Connected: An Internet Encyclopedia
Downloading Connected

Up: Connected: An Internet Encyclopedia
Up: Project Documentation
Prev: Project Description
Next: Installation

Downloading Connected

Downloading Connected The entire encyclopedia is available for download; see Licensing Information. Be warned - the whole thing is over 300 MB. I don't give exact figures here since they change as the website is updated. For your disk space and bandwidth, you get over 5000 Web pages, including all published RFCs and pre-built indices for full-text searching. You can also download individual HTMLized RFCs (follow the links on the RFC's index page). Although the encyclopedia is available via ftp (, and of course HTTP, the recommended download protocol is currently rsync. Once you've download all or part of the encyclopedia, see the Installation Instructions.

Download option A: rsync

This is currently the recommended way, due to the relative efficiency of the rsync algorithm, which breaks files into sections and sends only those sections that have changed. Rsync is designed to synchronize two sites together; in particular, it handles service disconnections well and will just pick up where it left off. It can be slow to start, however, since it begins by downloading a complete file list. RedHat Linux RPMs and source are available here. For Windows 9x and NT users, I've provided a Windows version of rsync here.

There are presently four rsync modules available on; more may have become available since this was written. Run rsync without any options except the rsync URL to get a list of modules:

Each of the modules is independent of the others. If you just want my original work (the Topical Core and Course, mainly), fetch CIE. If you want all the RFCs, fetch RFC; they'll go into a subdirectory called RFC/Orig, because that's where they appear on the website. If you plan to do full text searches on the Encyclopdia or RFCs, you'll need the appropriate index modules.

To download a module, give rsync the -r (recursive) switch, the module's full URL, and the destination directory you want the files to appear in. I also recommend the -v (verbose), -z (compress), and -t (preserve timestamps) switches, and if you're downloading the search engine indices, you might want to use the --partial switch to keep partially transferred index files in case of a lost connection. For example, if you want to download the Topical Core (part of the CIE package) and put it in a directory called freesoft, use:

Download option B: mirror

mirror is a Perl script that uses FTP to perform its transfers. RPMs are available from If you're using Microsoft Windows, you'll need a Windows version of Perl, which you can get from here; be warned - it's big. Here's a sample configuration file and crontab entry to use with mirror. Once the local directory has been set in the configuration file, the program can be run manually:

Download option C: manual FTP

The entire web site is available via FTP, and the Encyclopedia appears in the /www/CIE directory. The FTP server is currently running WU-FTPD and supports FTP conversion, where the FTP server performs special functions according to prefixes added to existing file and directory names. Supported FTP conversions:

.ZOn-the-fly file compression (compress)
.gzOn-the-fly file compression (gzip)
.zipOn-the-fly compression and archiving of directories (PkZip, WinZip)
.tarOn-the-fly archiving of directories (GNU tar)
.tar.ZOn-the-fly compression and archiving of directories (GNU tar)
.tar.gzOn-the-fly compression and archiving of directories (GNU tar)
.tgzOn-the-fly compressions and archiving of directories (GNU tar)

The special prefixes do not appear in directory lists, since they do not exist as actual files. For example, if the file rfc-index.txt appears in a directory listing, a compressed version can be downloaded using the FTP command "GET rfc-index.txt.gz". Please do not use this feature to download the entire website, since this creates a lot of load on the FTP server and if the connection breaks, the whole process will have to be started again the next time. Use rsync or mirror instead.

Other software

If you use other mirroring or download software, just follow these simple rules:

Next: Installation

Connected: An Internet Encyclopedia
Downloading Connected