Upgrading by Deep Dependency

The technical analog to “Management by Shiny Object” would have to be “Upgrading by Deep Dependency”. I am usually very quick to speak highly of Gentoo Linux as both a Linux distribution and general philosophy, but I have recently been stung by a requirement to upgrade a cascade of seemingly unrelated packages that ultimately “broke” my system in the performance of what I thought was a simple application update…an update intended to resolve a minor bug. Little did I know that I would very soon be begging for the minor annoyance of that bug to return just so I could have my life back. I’ve been a Gentoo user for at least 5 years, so I know that if I am frustrated, other less dedicated users must be leaving the distribution in droves.

For those that don’t know, Gentoo is a Linux distrubtion that considers itself “bleeding edge”. Instead of feeding the ceaseless cycle of patching old code with security and features (like many other “stable” trees in other distributions), Gentoo developers focus on making the integration of the latest and greatest code stable for their users. New code from external projects with security and features already built-in is tested and released much more quickly in this usage paradigm. This is both fun and exciting, as it exposes people like myself to the newest code, while keeping us all ahead of major code exploits. The drawback, as you might imagine, is that things don’t always work as expected, leading one down a road of discovery that might never have been intended.

Now, you might be asking yourself, “Well, you’ve just summed up the benefits and challenges of using Gentoo, what the heck are you complaining about”? The answer is quite simple, really. I don’t mind working things out with a package, or two, or three, or even handful of interrelated packages…so long as the process doesn’t break my entire system, and prevent me from getting real work done. Call me crazy, but “bleeding edge” or not, core things like the X Windows system, window managers, etc., that are critical to everything else, should be STABLE when they are marked “stable”.

To give you an idea of my last few days’ worth of experiential learning:

• I wanted to fix an odditiy in Amarok by upgrading to an “unstable” version
• …which required an update to the KDE base packages and libraries, which were marked as “stable”
• …which required an update to the “new” modular Xorgpackages, also marked as “stable”

After accepting the cascade of upgrades as the cost of doing business, I proceeded. The builds went well. I followed the detailed instructions for the smooth upgrade to the modular Xorg packages from the monolithic installation I had already running. After a couple of hours, everything was good…but due to some X library dependencies, I had to perform what is called a “reverse dependency rebuild” to make sure packages that were built against my previous installation of Xorg continued to function. On the list was my Mozilla Firefox and OpenOffice.org applications, which, by themselves, took the better part of an 8 hour day to recompile from source. Again, I thought, just the cost of doing business…no big deal.

I was delighted when, after performing a reboot (customary after a large package integration…otherwise my system stays up for months), my system presented me with a normal login screen. Login worked, and I set-out to begin working again. After all of this, my first order of business was to check on the application which prompted my upgrade in the first place. I launced Amarok, saw the application frame for a while while it attempted to load everything…then nothing. The app died…like, “you never launched me” died. OK. No biggie. I just figured I’d come back to that later, and would launch Firefox to check-in with email, etc.

While Firefox launched successfully, I quickly recognized that it was working very, very slowly. Any time I transitioned from one tab to another, the CPU utilization shot up to 100% for up to 5 seconds while the new tab content was loaded into view. That ain’t right. So I began to dig. I tried combinations of packages marked as “unstable” in Gentoo’s tree, changed USE flags, rebuilt entire subsections of the machine…all with relatively disappointing results.

After about two full days of effort, I managed to get Amarok working with a version of KDE libraries and base applications that is still marked “unstable”. It locks-up on me after a while, but allows me to continue to work with my music library on my iPod as I had been previously.

As for Firefox, I finally found an obscure reference to an environment variable that had previously, running the same version of Firefox with an Xorg system one rev back, been unnecessary. Now, it seems you must disable Pango before starting Firefox by setting the following in one of your start-up profile scripts:

MOZ_DISABLE_PANGO=1

While Firefox performance has improved, it is still not up to the speeds I was enjoying before this merry journey. I continue to try to troubleshoot, but for now, it works well enough.

Thankfully, this experience did not cripple me. I have other systems on which to perform work. It did set some interesting facts straight for me that had not made complete sense before. I read somewhere in the Gentoo Forums that in a recent user survey (data from the survey itself could not be found with a simple Google search), less than 20% of Gentoo users use the distribution on production-level systems. Well, duh! Production-level systems cannot be down for prolonged periods of time to debug what should be a relatively straight-forward process when dealing with system packages that are marked “stable”…which is true for ANY distribution.

I continue as a Gentoo user, but fear the distribution will eventually be permanently marked as an elitist, fringe version of the Linux OS that is seldom suitable for deployment into production environments. Perhaps the dark corners of the Linux fringe is Gentoo’s market, and data centers will have to look to less flexible, but stable distributions like Scientific Linux or Ubuntu for their needs. Given the time I’ve spent with Gentoo, I would never have hoped for this outcome, but if you ask me, Gentoo hasn’t been quite the same since its founder and former chief architect, Daniel Robbins, left the project. But that’s another story…