One Terabyte of Kilobyte Age

What you get by downloading the Geocities Torrent is not actually an “archive”. It contains many 7zip archive files, but how the data therein is organized is not fit to make statistical analysis. The Geocities Torrent tells the story of a great disaster and salvation, also in its structure.

For example, to simply answer the question “How many Geocities accounts are contained in the torrent?” or “What was the most used divider image in a certain neighborhood?”, counting index.html files is not enough. For this, we need to know the original directory structure on the Geocities server, and since Yahoo! didn’t give anybody access to it directly, we have to rely on the information about it that was available to the Archive Team via HTTP during the time when they made the copy. Also, the Archive Team had to pack the data for optimal distribution, which worked very well, and created an almost entertaining downloading experience.¹ But the big amount of symbolic links makes it difficult to do even simple counting.

Users do not like case-sensitive file systems

Geocities used the powerful Apache web server on an unknown Unix-like operating system.² User account names, neighborhood names plus directory and file names were stored case-sensitive on there, meaning that the file “Hello.html” is different from “hello.html” or “heLLo.html”. Traditionally, most users do not understand why there should be a difference for the same name written in a different case. “Consumer” operating systems (aka Windows and Macintosh) do not distinguish case in the file system. Most users of Geocities didn’t care for case when putting links in their HTML code, for example they could link to http://www.geocities.com/bob/dogs/ when the actual file name on the Geocities server would call for a link like http://www.geocities.com/Bob/Dogs/.

Apparently, Geocities followed two strategies for easing their users’ pain with case-sensitivity:

Symbolic Links by Geocities

They created symbolic links in their file system that pointed from Bob to bob.³ This means that when looking into the directorly bob, it will always contain the same content as the directory Bob, and vice-versa. Symbolic links are a powerful file system feature, however it is very easy to create train wrecks with for example directories that contain a link to themselves: an infinite loop in the file system. What’s worse, when looking at a site through a browser, symbolic links can not be distinguished from a real file or directory. So both Bob and bob would exist as if there were two users instead of one. And the Archive Team of course hadto save both variations, because, without looking inside of each directory, they wouldn’t know if there maybe was another user that went with the same name in lowercase.⁴

There are many ruins of symbolic links to be found in the torrent, especially of the type that creates infinite loops.

mod_speling

There is a plugin for the Apache web server, mod_speling, which tries to correct wrongly typed URLs and redirects the browser to the actual URL with the correct case. It appears like at some point the Geocities server was equipped with this module — otherwise it would be a miracle how all this could function in general. However, the mirror tool wget used by the Archive Team to copy Geocities, will still save the file under the original request name. So if you ask wget to copy bob, it will be redirected to Bob and save what is found there, but still locally give it the name bob.⁵ And again it would result in a potentially duplicate file.

This is neither the fault of Archive Team⁶, the wget developers or Geocities. HTTP and HTML were designed in a certain way, but when millions of users are let loose on a technically well-defined standard, unpredictable things happen.

Where the Archive Team detected duplicate downloads, they replaced them with symbolic links in their copy’s file system. While this makes browsing the data much easier, it also leads to problems about deciding for what operation which type of symbolic link has to be followed. If for example a symlink makes a whole sub-neighborhood exist twice in two different spellings, this symlink should be ignored. A symlink to an user’s account that is stored in YAHOOIDS should be followed though. It would be possible to develop a logic that takes all of this into account, but it will be prone to errors, resulting in some research operations having to be repeated when bugs in research scripts are found. And each run can take ages! So it seems like a good investment to fix the file system before going any further.

Fixing

1. Most analysis on the Geocities Torrent will have to be conducted through HTTP and an Apache webserver running mod_speling. Redirects will have to be taken into account.
2. All symbolic links have to be resolved. Steps:
   1. Use the command find . -type l to catch the first level of symbolic links.
   2. Use readlink to determine where symlinks are pointing and replace them with the original files (first rm the symlink, then mv the original to the symlink’s location).
   3. Repeat steps C and B until no symbolic links are left.

   Of course, every round of found symlinks has to be examined manually for infinite loops or obvious traps.

3. Find directories and with “almost equal” names, e.g. names that would be found by mod_speling. Compare the pairs’ contents. If the contents are equal, decide which is the original and delete the other. If the contents are partly different, merge the contents and keep only one version. If the content is different, keep both versions. (Probably should be done using diff.)

Each of these operations takes from hours to days. So please bear with us for a while :)