Wednesday, December 16, 2009

udisks and gnome-disk-utility - past, present and future

The past year or so I’ve been working a lot on gnome-disk-utility and DeviceKit-disks^Wudisks and I haven’t really blogged anything about it. Time to fix that!

First of all, given that the main DeviceKit daemon was killed by libudev (if you are using GLib you want GUdev instead!), we (that is, myself and other contributors) decided to rename the project - most people were cool with that; see this message for more reasons. One improvement in this area is that we’re committing to limited ABI stability - hopefully this will make things easier on distributors.

Most of the work in the udisks project been triggered by features we want in the Palimpsest Disk Utility - which is the most prominent part of gnome-disk-utility - the other part includes various desktop shell integration bits that I don’t have room to cover in this post.

On the Disk Utility side, we’re now using an user interface that scales better with many disks - in the old user interface, we used to show all partitions as children of each disk in the tree view. In the new user interface, we’re using a grid to convey the partitioning:

Also see bgo #579211

While this interface isn’t optimal for the laptop or the machine-with-only-a-single-disk cases, it works a lot better for machines with many disks

Bunch of disks

The Disk Utility also have somewhat complete support for MD RAID (aka Software RAID on Linux) - we’ll show you running arrays and also arrays not yet running. There’s a way to add/remove components and currently I’m working on creating the UI for Online Capacity Expansion (e.g. dynamically expand a RAID-5 array from 4 to 8 drives).

The SMART stuff has been revamped and there is now a way to turn off the desktop shell notifications. As it turns out a lot of our current users are like “I’m fine with using a disk that is about to fail” - kinda shocking but there you go. Another change is that we don’t consider a disk to be failing if there’s only a couple of bad sectors - the disk will have to have many bad sectors to be considered failing. This helps with all the false positives the SMART warnings generated in e.g. Fedora 11. Anyway, it seems like the desktop notifications are useful - more than a handful of people have pinged me on IRC thanking me and Lennart for this feature. Woot!

We also now have rudimentary support for showing the SAS topology - this is really useful since machines with many disks typically use dedicated disk enclosure shelves. Currently we only show SAS Expanders but the plan is to peek (and poke!) SES-2 Enclosures (and in the future also SGPIO ones) to get information about bay numbering and to toggle the FAILURE and LOCATE leds. Ideally we want to turn on the FAILURE led if a member of a RAID array is kicked or if SMART status indicates that the disk is about to fail. Ideally, we also want to do (sensible!) things like starting a RAID rebuild if a kicked RAID member is replaced by a uninitialized and same-size disk. As always, we have to be really careful about automatically doing policy things like this - especially on Linux where it is hard to make any assumptions on how the system is used.

The goal of the udisks effort is to be useful to users. For example, if you do video editing or a lot of photography, you care about IO performance. So I added a way to easily benchmark drives and RAID arrays:


This feature was inspired by various similar Windows programs (a pastime on big storage forums such as this one involves trading benchmark screenshots) and the zcav program. It’s pretty cute, actually; I learned that my Intel 80GB G1 SSD only gives me 140MB/s on my Lenovo Thinkpad X61 while I get a full 270MB/s on my workstation (connected via SAS). Looks like the X61 is only using SATA 150 MB/s - a search on the tubes confirmed this. Gee! This feature is also useful when experimenting with RAID setups - for example, a 10K 36.7GB SAS disk benchmarks like this but when you put three of them together in Software RAID-5 it turns that writing is really slow. I know it’s supposed to be slow when writing to software RAID… but, really, this slow? Maybe I need to tweak some kernel tunables (which would be bad - we should be fast out of the box etc etc).

Finally, I did a really cute hack a few weeks ago - I made Palimpsest use D-Bus over TCP/IP over SSH to speak to udisks:

Bunch of servers

With the way this works we leverage SSH for authentication and Avahi for Service Discovery. The feature is still a bit rough - it needs some UI additions (such as a disconnect button) and we probably want to replace the stock OpenSSH dialog with something, uhm, less scary. There’s also the question of authorization - right now you have to connect as root in order to modify things (udisks uses polkit for authorization) - we need to figure out how we want this to work. Right now I’m thinking udisks will ship some polkit configuration bits (a .pkla file) and a udisks-admin group that administrators can add users to. We’ll see.

I’m kinda excited about the remote connectivity bits - it means that the Disk Utility is now very useful for headless server setups and said servers won’t have to run any graphical UI in order to be managed through the Disk Utility. We probably want to make the Disk Utility (e.g. Palimpsest) run on Win32 and OS X too (for users not running Linux on the desktop) for remote connectivity to e.g. Linux boxes running udisks. It shouldn’t be too hard to do this as we mainly depend on GTK+ and D-Bus.

There’s a bunch of plans for both udisks and Palimpsest. We want to add support (meaning: udisks API and Palimpsest User Interface bits) for the following things: LVM, iSCSI (both target and initiator), Enclosure Management (as mentioned above), Event logging (e.g. “raid5:md2: read error corrected”), Multi-path, Btrfs and many other useful storage technologies. There’s also a few rough edges and missing features (e.g. file system resizing) but we’ll get to that in due time.

FWIW, it’s important to realize that udisks and the Disk Utility complements, but does not replace, existing command-line tools. This means that you can go ahead and keep using your command line tools and scripts and the udisks/GNOME/Palimpsest/etc. stack will react to changes. For example, you can start/stop RAID arrays using mdadm(8), partition your disks using parted(8), create filesystems using mkfs(8), mount/umount filesystems and so on - the UI is kept up to date. While it’s approximately ten times harder (with the 10x figure being 65% accurate) to write software this way, it’s really the only way to do things on something like Linux where our set of users is as diverse as it is.

Tuesday, September 29, 2009

Plumbers 2009

Had a great time at Plumbers Conference last week in lovely Portland, OR. On the Friday on the conference, Kay and I gave a talk, aptly named “Replugging the Modern Desktop”. The talk covered most of the hal-ectomy that has been happening and included a couple of demos. The slides are here. UPDATE: People on IRC are telling me that the fd.o servers are not reachable, I’ve put up another copy of the slides here.

Monday, June 1, 2009

GUdev, gobject-introspection

I just committed gudev-1.0 to udev-extras. GUdev isn’t a particulary big library, it’s just a thin wrapper around libudev with integration bits for the GLib main loop and support for GObject Introspection.

While GObject introspection is very neat, see this GUdev example using Seed, I must admit the whole thing feels… half-finished?

For example, we seem to be doing the wrong thing for default annotations in some cases. Another example is the fact that gjs lacks a print() function, Seed can’t handle GStrv properties (works in gjs), lack of support for string arrays in gjs (works in Seed), neither Seed nor gjs supports Shebang support (gjs bug, Seed bug).

So all in all, trying to make my new small library introspectable.. was kinda.. much more work than I expected.. and most of the time was spent with the Seed and gjs runtimes trying to figure out what was wrong.

Anyway, I don’t want to sound all negative.. with all the work going on in this area, it looks like it’s going to rock once corner cases like mentioned above are ironed out. Rock on!

Monday, May 4, 2009

Storage handling in GNOME

The GNOME 2.26 release in Fedora 11 will ship with a completely different stack for handling storage devices. The plan is to land all this work in the upstream GNOME 2.28 release and most of that work is done already.

Basically, we’ve switched away from one daemon to another and in the process we gained a few new features.

OMG, the disk is dying - using libatasmart

including letting the user inspect what his disk is trying to tell him. Kudos goes to Lennart for adding USB support, fixing issues with false positives in libatasmart and just for providing a really good library. And it looks like it’s working.

To really make this 100% useful, we probably want some kind of directory where system vendors shipping GNOME can drop an XML file and then the user can click a button “Order new disk” and get on with their life. Of course, then we need to handle the much harder problem of backup and data migration, but, hey, we want to solve that anyway.

So I also wrote this Palimpsest Disk Utility application

Kitchen sink of disk information

which also handles things like RAID devices and support for managing things like LVM and btrfs is planned as well. We’re also looking at changing the UI to be a bit more sane - something like this perhaps

Clean, leaner, simpler? Tell us in bug #579211!

Tomas, Matthias and I also worked on a simple formatting tool suitable for replacing the venerable gfloppy tool

Formatting tool

you can see the whole series of screenshots to see how the user interaction is supposed to be like.

Finally, I just merged a series of patches to GVfs for using this new machinery

OMG, pretty RAID icon!

Building this on top of a modern storage daemon, as opposed to HAL, means that your file manager (and any other app using GIO), is instantly updated as you repartition or reformat your disks. It even works if you are using command-line tools, etc. fdisk(8) or mkfs(8) - geek comfort FTW!

We might want some extensions to GIO to make working with RAID and encrypted devices (and in the future LVM) easier (such as unlock/lock/start/stop buttons in the file manager sidepane) or maybe we’re just going to rely on the system auto-assembling such things and leave it in Palimpsest. Decisions, decisions.

Another geek-comfort I added, is the ability to show user mountable fstab entries, that’s bug #536292. I was a bit reluctant to do this, I think instead we want native support in GVfs for nfs:// yet still use the kernel filesystem driver though some kind of setuid helper. Then we’d also fix NFS servers to publish the fact they exist using Avahi and then people can discover it in Nautilus’ computer:// window. Just cleaner and easier, I think. Combined with support for bookmarks on the GIO level, this should be a dream come true for people deplying systems in the enterprise.

Most of this work is part of the bigger HAL-ectomy - my good friend Kay has a good post outlining the plans for this.