Alex Hudson

Thoughts on Technology, Product, & Strategy

The quality of Fedora releases

Scott James Remnant blogged his ideas about how to improve the quality of Ubuntu releases recently, triggering some discussion at LWN about the topic. I offered some opinions about Ubuntu which are not terribly interesting because I don’t get to use it often; however, I did also write about Fedora based on the last couple months’ experience of Fedora 15 & 16.

Before I get to that, at roughly the same time, Doug Ledford was posting his thoughts about the “critical path” process – essentially, saying it was broken. I’m pretty sure he will find vociferous agreement with his views, based on previous feedback, but not (alas) from me.

I made a number of claims about Fedora in the LWN comments, that it is essentially unusable for anyone non-expert. I stand by this: if you use Fedora, and care about things continuing to work on an ongoing basis, you have to be entirely au fait with:

  • upgrading your distribution every six months;
  • rolling back an update (including config) when (not if) things break;
  • knowing how to distinguish which part of the stack is broken (and oh boy, this isn’t easy).

People complain about Windows “rotting” over time and getting slower, and slower. Fedora is worse than this: it works great until you get an update which blows out something critical. At that point it stops working: maybe you stop receiving updates, maybe you can’t boot it, maybe you can’t log in. I haven’t had a release yet, that I can recall, that something crucial wasn’t broken.

Of course, in Fedora, we have the critpath process, which is supposed to stop this kind of thing. And this is what Doug is complaining about, because the process which puts roadblocks in the way of updates to crucial packages naturally gets in the way of “fixes”.

This is where I depart with Doug, I suspect. I have sympathy for his situation, particularly in F16. But the point of critpath is that it does cause pain.

The issue with critpath isn’t that it gets in the way; it’s that it highlights release problems. When major issues turn up in critical packages and get through to release, it hurts – it cannot be fixed quickly. And here’s my point of disagreement: we shouldn’t allow more direct fix releases just to avoid the pain, we should address the root cause of the problem – the release of bad code.

(Interlude: as a point of order, it should probably be made clear that the specific issue Doug is facing is a lack of karma on a specific release branch with a package which, although critical, is not necessarily widely used. That’s an obvious bummer, and I’m not sure is terribly instructive about the process as a whole because of that).

There have been a variety of other solutions proposed. AutoQA is something I have a huge amount of time for, and in particular would help reduce the incidence of obvious brown paper bag bugs. It’s not a solution itself, though. Equally, there will always be stuff which evades testing – hardware support in device drivers being an obvious case in point.

I am extremely jealous, though, of the quality Debian achieves, and I say this as a Fedora user. It’s stable, it’s easily upgradable from release to release, and generally the use:surprise ratio is reassuringly high. To a large extent, I think they illustrate that the specific processes don’t really matter a huge amount: what matters is actually getting to a point where maintainers don’t release bad packages often. And to me, that is the point of critpath: to encourage packages of sufficient quality that the pain is generally avoided. Simply short-cutting the process doesn’t encourage that; it just encourages more of the same in Fedora.


Who can program?


Speculation on Google’s “Dart”


  1. I’ve always been a huge fan of fedora, as described on paper. But every time I try, I find myself feeling relatively let down. The “cram as much mostly-stable stuff down your throats every 6 months” mindset just feels unfinished, almost like they try to do too much during the rawhide phase, and then rush to get it ironed out. Then again, this comes down to target audience. I don’t even think justifying broken updates with “fedora’s market is for the more advanced users” even fits 🙂 Surely develops know how shared libraries work too?

  2. I added this comment on LWN as well but you are wrong in claiming that Fedora 15 has Pino as default. It was dropped from the default list

  3. Alex

    Yep, I was wrong about that, but as I noted on LWN – it was dropped on the day that should have been final release. I’m glad it did get finally removed, but it was such a regression from pino in F14 that it never, ever should have made it into Rawhide, let alone F15. F14 pino was usable, bar problems with twitter.

  4. The comment about Debian make me wonder ( and I do not say that with bad intention ) why do people not use it more.

    I mean, on the mageia project, we had a huge ( even if not as huge as the discussion on fedora-devel ) discussion about releases processes, and some people told us that “yes, people want something that doesn’t break, something that can be upgraded, and that just work”. And I agree that’s desirable features for a quite significant group of people, so the feedback is perfectly valid. And based on my own experience, that’s exactly what debian deliver, yet no one recommend them. The question is why ?

  5. Alex

    I think the issue with Debian is two-fold; I have some experience of this because years ago I was primarily a Debian user (I then switched to Ubuntu when that came out, and then switched from Ubuntu to Fedora when it became clear how un-community the Ubuntu development process was going to be).

    The first issue is the timeliness of releases. Debian have basically more or less fixed this, and have a compelling two-year schedule which they’ve stuck to so far. Before that, though, the stable release was more unpredictable and had arrived a couple of times at almost exactly the wrong moment – so that, for desktop users, the new release was basically old software from the moment it arrived.

    The second issue is really one more of accessibility. Ubuntu is seen as “user friendly Debian”, and I really don’t think that’s fair. But there still isn’t as much emphasis on making things really usable, and a visitor to the Debian homepage – although it’s much improved – is going to be put off by the complexity compared to the other major projects.

    For a technical user, I think Debian is far and away the most friendly distro (except perhaps for the hated split exim4 config 😉 ) – but for a non-Debianite, there is a fair barrier there. And, to be honest, I know they don’t take marketing that seriously, but their image is *awful*. That cheesy space-craft – I hate to criticise what someone has worked hard on, but it makes the thing look like a toy.

  6. I think the update process in Fedora has improved a lot over the last couple of years – I haven’t hard an update break my system for a long time. Maybe I’ve just been lucky!

  7. Alex,

    I don’t think you can assert that it shouldn’t be allowed in Rawhide. You don’t know the background and you didn’t really ask either so I am not going to bore you down with the details.

  8. pug

    Even on its unstable branch Debian is usually way behind Fedora regarding software currentness. Many latent bugs are not uncovered until the software gets its first testing exposure by a broader user base. For many critical low-level components in our stack this most likely happens in a Fedora development release. When other distributions adopt those components many bugs tend to be weeded out already.

    I use Fedora especially because it is bleeding edge. This is what Fedora is known for and what makes it unique. Being on the tip of free software development has advantages for me. For this, I’m ready to put up with occasional breakage.

    If maintaining older Fedora releases means less manpower for making advancements, then as far as I’m concerned, drop those releases.

    The nice thing is, we have numerous fine distributions available. On some boxes I actually use Debian because I think its a better fit for those. There’s no reason why a single distribution has to cater to all kinds of users for all possible fields of use. Fedora once was known for delivering new software versions not only on new releases but also as updates without the need for running a whole rolling release distribution. The new update policy kind of killed that “unique selling point”.

    In my opinion the restrictions on pushing updates actually make the situation worse for users. Fixes to real problems don’t get pushed in time, therefore exposing users to known bugs for a longer period of time. Heck, even critical security updates don’t get the neccessary testing and are delayed, sometimes even for weeks. What could be possibly worse than being vulnerable to remote exploitation? This is just bad and needs to be fixed.

    As long as there is no solution to those problems, yes, it seems as the update policy just gets in the ways of maintainers trying to fix actual problems that users care of.

  9. Alex

    pug, I just don’t agree that all those factors you talk about are in opposition. But that aside, I honestly don’t think the “Fedora as one big testing process” is actually what people want.

    If you go to the Fedora homepage, it says “Fedora is a fast, stable, and powerful operating system” – and at the moment, it’s not living up to that. There are something like 33 million IP addresses that collect Fedora updates, implying something close to that number of installs.

    Are you really claiming that “bleeding edge, occasional breakage, but that’s fine” attitude is representative of those 33 million-odd users?

    I would rather suspect that it is entirely a minority view.

Leave a Reply