tag:blogger.com,1999:blog-52688474174179533492024-02-29T01:48:39.016-05:00inactivity log for davidzWhat's that? Chicken?davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.comBlogger123125tag:blogger.com,1999:blog-5268847417417953349.post-1999171025584687322012-08-02T17:09:00.000-04:002012-08-02T17:19:19.398-04:00Disks improvements in GNOME 3.6Just a short exposition of new features and changes in the Disks and udisks releases that will ship with GNOME 3.6<br />
<ul>
<li>There are now configuration dialogs for power management for ATA disks, see these <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/03-settings-standby-and-apm.png">two</a> <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/05-settings-standby-and-aam.png">screenshots</a>. Implementation-wise, it's backed by udisks configuration files, see the <a href="http://udisks.freedesktop.org/docs/latest/udisks.8.html">udisks man page</a> for details. In the future we may add more disk and block device settings (for example, for controlling write-back/write-thru caching)</li>
<li>The Disks application will now show a <i>Zzz</i> icon if a disk is in standby mode. It looks like <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/04-zzz-icon.png">this</a>. You can also manually put a disk into <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/07-disk-standby-now.png">standby mode</a> and also <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/06-disk-resume-now.png">wake it up</a></li>
<li>There is now a way to erase a disk by filling it with zeroes. If the disk supports the ATA SECURE ERASE command (all modern ATA hard disks from the last decade or so), we support that as well. The erase type is selected from a <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/08-erase-options.png">combo box</a>. In the erase confirmation dialog we also now offer guidance for all three cases since each of them have interesting implications (<a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/09-erase-quick-warning.png">quick format</a>, <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/10-erase-full-warning.png">overwrite</a>, <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/11-erase-ata-secure-warning.png">ATA secure erase</a>). The link in the last warning points the user to the <a href="https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase">ATA Secure Erase page</a> on the Linux ATA Wiki. Hopefully this guidance is useful for people donating/selling old disks/devices (e.g. remind them to wipe)</li>
<li>Long-running jobs are now <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/12-job-progress.png">displayed</a> in the Disks user interface along with a way to cancel the job as well</li>
<li>There's a new "Disk Image Mounter" application for attaching (you've guessed it), <a href="http://en.wikipedia.org/wiki/Disk_image">disk images</a>. It comes with MIME type associations and in Fedora 18 it is the <a href="https://bugzilla.redhat.com/show_bug.cgi?id=820403">default application for ISO files</a>. This means that if you <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/14-disk-image-mounter-app.png">double click an ISO file in Files</a>, this application is invoked and it simply just sets up the loop device.</li>
<li>Combined with this <a href="http://git.gnome.org/browse/gvfs/commit/?id=88ffd85e73fec8611386530962fb35451e28571c">GVfs change</a> the desktop bits will Do The Right Thing(tm) when the user sets up a loop device (don't worry, we don't do anything automatic if your shell script calls <a href="http://linux.die.net/man/8/losetup">losetup(8)</a> - it only affect loop devices set up via the <a href="http://udisks.freedesktop.org/docs/latest/gdbus-org.freedesktop.UDisks2.Manager.html#gdbus-method-org-freedesktop-UDisks2-Manager.LoopSetup">udisks D-Bus interface</a>). For example, if the image contains an installable OS, the desktop will go on to <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/15-gvfs-loop-device-handling.png">prompt the user to install the OS</a> in the disk image. Note that the machinery is exactly the same as for physical media / devices - e.g. it works with partitioned disk images, the user is prompted to unlock LUKS volumes in the disk image etc. For example, for a ISO image of a Video DVD, the desktop will suggest opening whatever app is registered for x-content/video-dvd (normally <a href="http://projects.gnome.org/totem/">Totem</a> but could also be a 3rd party DVD player claiming to support that MIME type)</li>
<li>The GVfs bits will also toggle the autoclear flag for the loop device when it mounts / unlocks the loop device (or a partition on the loop device) - this is so the disk image is detached when the last user stops using it. This flag can also be toggled from the Disks application, see this <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/13-loop-autoclear.png">screenshot</a></li>
<li>After a <a href="https://bugzilla.gnome.org/show_bug.cgi?id=674161">good and healthy discussion</a>, the desktop shell now offers to save the LUKS passphrases in the keyring again (removed in 3.4, back in 3.6!). So it was a natural thing to make the Disks application read it when <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/01-passphrase-from-keyring.png">unlocking the volume</a></li>
<li>Speaking of passphrases, Disks is now using <a href="https://fedorahosted.org/libpwquality/">libpwquality</a> and <a href="http://developer.gnome.org/gtk3/unstable/GtkLevelBar.html">GtkLevelBar</a>. It looks like <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/02-libpwquality-and-gtklevelbar.png">this</a></li>
<li>By popular demand, the benchmark feature that was lost in the Disks/udisks <a href="http://davidz25.blogspot.com/2012/03/simpler-faster-better.html">rewrite that landed in GNOME 3.4</a> is now back - it now looks like <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/16-benchmark.png">this</a>. Not only is it back, it's back in <i>style</i>: write benchmarks are now non-destructive (just use ECC memory, please) and the graph is updated while the benchmark is underway. You can also <a href="http://people.gnome.org/~davidz/gnome-disks-3.6-shots/18-benchmark-new.png">configure how many samples to make etc.</a>. Also, benchmarking isn't constrained to whole disks anymore - you can now do it on any volume or block device (for example this <a href="http://people.freedesktop.org/~david/disks-benchmark-md-raid-6-16x1TB-SATA-7200rpm.png">RAID-6 array</a>). It's a handy little feature</li>
</ul>
<div>
Of course, the GNOME Disks changes are just the user-visible bits. The <i>"lower half"</i> of the Disks application (basically invisible to end users) is called <a href="http://www.freedesktop.org/wiki/Software/udisks">udisks</a> and happens to be shared with other user interfaces including KDE and all the other user interfaces built on top of GLib/GTK+. For udisks changes, see its <a href="http://cgit.freedesktop.org/udisks">git repository</a> for details.</div>
<h2>
Looking forward to GNOME 3.8</h2>
<div>
Some of the things I'm planning to work on in the future (hopefully some of it will land in GNOME 3.8) include</div>
<div>
<ul>
<li><a href="https://bugs.freedesktop.org/show_bug.cgi?id=51783">Filesystem/partition resizing</a> - not that interesting of a feature on its own, but incredibly useful in an installer setup where you want to dual-boot with another OS. It shouldn't be that hard to implement since the functionality more or less exists and the main job here is a) add a couple of methods to the <a href="http://udisks.freedesktop.org/docs/latest/gdbus-org.freedesktop.UDisks2.Filesystem.html">Filesystem</a> and <a href="http://udisks.freedesktop.org/docs/latest/gdbus-org.freedesktop.UDisks2.Partition.html">Partition</a> D-Bus interfaces; and b) design and implement the UI in Disks. I'm planning to collaborate on the udisks bits with the KDE developer who filed the bug</li>
<li><a href="https://raid.wiki.kernel.org/index.php/Linux_Raid">Linux MD Raid</a> support in udisks and Disks. We actually had somewhat decent support for this in the old version of udisks/Disks (see <a href="http://people.freedesktop.org/~david/nautilus-palimpsest-assembly-shots/palimpsest-mdraid-1.png">these</a> <a href="http://people.freedesktop.org/~david/gdu-mdraid-expansion/gdu-mdraid-expand-3.png">screenshots</a> <a href="http://people.freedesktop.org/~david/palimpsest-newui-raid.png">for</a> <a href="http://people.freedesktop.org/~david/md-raid-create.png">example</a>) but it was removed in the rewrite as it was implemented in a hurry without a lot attention to design. Turns out there's still quite a bit of interest from several parties to add back this functionality plus also supporting <a href="https://raid.wiki.kernel.org/index.php/RAID_setup#External_Metadata">external metadata and firmware/platform features</a> in a meaningful way. To keep complexity down, we'll probably only allow creating arrays from whole disks as this is the recommended way to do RAID (users can still use <a href="http://linux.die.net/man/8/mdadm">mdadm(8)</a> to create arrays on partitions if they so desire)</li>
<li>Simple <a href="http://en.wikipedia.org/wiki/ISCSI">iSCSI</a> initiator support - there's already git branches for this in the <a href="http://git.gnome.org/browse/gnome-disk-utility/log/?h=wip/iscsi">gnome-disk-utility</a> and <a href="http://cgit.freedesktop.org/udisks/log/?h=wip/iscsi">udisks</a> repos but it's probably somewhat out of date by now. Anyway, the idea is really simple - just provide a simple front-end to the most common features provided by <a href="http://www.open-iscsi.org/docs/README">Linux Open-iSCSI</a>. Some old screenshots showing the work I've already done are <a href="http://people.freedesktop.org/~david/palimpsest-iscsi-wip.png">here</a> and <a href="http://people.freedesktop.org/~david/palimpsest-iscsi-wip-2.png">here</a></li>
</ul>
</div>
<div>
Now an obvious question comes to mind: why should Disks support complicated stuff like MD-RAID and iSCSI? Honestly, for the last couple of years, I've been bouncing back and forth between wanting / not wanting complicated and advanced features like this. However, recent discussions in the GNOME community has put me on a trajectory where I'm more and more convinced that GNOME should try harder to cater to, uh, <a href="http://en.wikipedia.org/wiki/Workstation">workstation</a> users (for want of a better word).<br />
<br />
If workstations are part of GNOME's mission statement then it follows that you want to enable users to, say, easily set up a beefy workstation with eight disks in a RAID configuration so <a href="https://live.gnome.org/Jhbuild/">rebuilding the OS</a> goes faster, easily connect to the lab SAN for if you're into <a href="http://www.youtube.com/watch?v=Y6ljFaKRTrI">science</a> and so on. Things like that. No matter how good, user-friendly and well-documented commands and configuration files are, I don't think saying "just use the command-line or edit this or that file in /etc" is a super-satisfying answer.</div>
<div>
<br /></div>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com15tag:blogger.com,1999:blog-5268847417417953349.post-88520234761185322152012-06-04T16:39:00.001-04:002012-06-04T16:45:13.682-04:00Authorization Rules in polkitFor the past couple of weeks, I've been working on rewriting the part of <a href="http://www.freedesktop.org/wiki/Software/polkit">polkit</a> that actually makes the authorization decision: the polkit authority.<br />
<h2>
Some history</h2>
In its current implementation, the so-called <i><a href="http://www.freedesktop.org/software/polkit/docs/0.105/pklocalauthority.8.html">polkit local authority</a> </i>ended up being one of these things that never really worked well for the (relatively small subset of) users with a need to configure it. First of all the local authority was never really supposed to be the main polkit authority... back then, we envisioned that in an <a href="http://en.wikipedia.org/wiki/Enterprise_software">Enterprise setting</a>, you'd have something like <a href="http://freeipa.org/page/Main_Page">FreeIPA</a> to handle permissions and authorization and this would in turn provide a <a href="http://www.freedesktop.org/software/polkit/docs/0.105/PolkitBackendAuthority.html">polkit backend</a> responsible for answering authorization questions from applications (sadly this haven't materialized yet but with this new work it could be we're a lot closer).<br />
<br />
Second of all, the local authority had some serious usability problems, in fact so many that I early on <a href="https://bugs.freedesktop.org/show_bug.cgi?id=30181">re-purposed a bug for the rewrite</a>. I think I basically concluded that it's just too hard to even do simple things and the key-file based format and priority rule scheme for the <i>.pkla</i> file format wasn't really helping. It's also really hard to test <i>.pkla</i> files.<br />
<h2>
Defining the problem</h2>
After thinking about this problem on and off for a while (while working on other things, mostly udisks/GNOME Disks and all the GDBus stuff), I identified the following requirements:<br />
<br />
<ol>
<li>There is no one-size-fits-all - for example some admins want to allow all users in a group to do the action xyz, while other admins want the opposite (forbid users in a group to do the action xyz). Similarly, some admins want black-lists (<i>"allow anything but action xyz"</i>) and some want white-lists (<i>"only allow actions xyz, abc, ..."</i>). Some admins even want to have the result depend on the time (<i>"don't allow xyz on school nights"</i>).</li>
<li>It would be good to have more information than just the <i>action</i> available when making a decision. For example for mounting or formatting a disk, it would be nice to have the device file or <a href="http://en.wikipedia.org/wiki/World_Wide_Name">serial number or name</a> and e.g. make the decision depend on this. For connecting to a wireless access point, having access to its <a href="http://en.wikipedia.org/wiki/ESSID">ESSID</a> would be great (to only allow connections to ESSID's in a whitelist) and so on.</li>
<li>Some admins may want to use external programs so there should be some facility available for this. Although this can't be the primary interface because forking a process every time someone calls <a href="http://www.freedesktop.org/software/polkit/docs/master/eggdbus-interface-org.freedesktop.PolicyKit1.Authority.html#eggdbus-method-org.freedesktop.PolicyKit1.Authority.CheckAuthorization">CheckAuthorization()</a> is a recipe for disaster [1].</li>
<li>Ideally, it should be easy test authorization rules - after all, this has to do with security so being able to easily (and automatically) test that your rules Does The Right Thing(tm) would be, uhm, great</li>
</ol>
<br />
A lot of these requirements are actually similar to the requirements you have for <a href="http://www.freedesktop.org/software/systemd/man/udev.html">udev rules</a>. And a lot of the constraints are similar as well, especially the fork-fest part.<br />
<br />
Another thing that's important to identify is who is going to end up using this. The answer here is mostly "Enterprise Admin" although in reality a lot hobbyist end up using it because, well, they like to tinker around. Some users of Linux distributions with broken defaults (see below) may also need to undo the damage done by the distribution so they can actually use their computer. All in all, one can probably assume that the target user here is relatively skilled, ie. can read the provided documentation and is at least capable of copy-pasting some snippet from a website into a file in <span style="font-family: 'Courier New', Courier, monospace;">/etc</span> as root and check that it has an effect.<br />
<br />
<span style="font-size: x-small;">[1] : History lesson: before we had <span style="font-family: 'Courier New', Courier, monospace;">udevd(8)</span>, the kernel forked <span style="font-family: 'Courier New', Courier, monospace;">/sbin/hotplug</span> for every hotplug event. And <span style="font-family: 'Courier New', Courier, monospace;">/sbin/hotplug</span>, being a shell script and all, itself forked another ten shell scripts or so in </span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">/etc/hotplug.d/</span><span style="font-size: x-small;">and these forked other shell-scripts and... the result was that hotplugging a USB hub full of devices could easily take </span><i style="font-size: small;">minutes</i><span style="font-size: x-small;"> because tens of thousands of <span style="font-family: 'Courier New', Courier, monospace;">/bin/sh</span>-instances were forked. Awesometown. Today <span style="font-family: 'Courier New', Courier, monospace;">udevd(8)</span> does the same in less than a second without forking a lot of extraneous processes.</span><br />
<h2>
Just embed JavaScript</h2>
The first requirement clearly indicates that we want some kind of programming language. However, <a href="http://programmers.stackexchange.com/questions/62727/when-is-it-reasonable-to-create-my-own-programming-language">inventing your own programming language</a> is rarely a good idea so I decided to just embed a <a href="http://en.wikipedia.org/wiki/JavaScript_engine">JavaScript interpreter</a> (specifically <a href="http://en.wikipedia.org/wiki/SpiderMonkey">SpiderMonkey</a>) and try that out.<br />
<br />
For the second requirement, I just exposed information we already have to the rules engine. I also rewrote the <a href="http://www.freedesktop.org/software/polkit/docs/master/polkit-apps.html">"Writing polkit applications"</a> chapter to mention that mechanisms should use this feature as well as a ton of other advice. [2]<br />
<br />
For the third requirement, spawning programs, I added a simple <b>polkit.spawn()</b> method.<br />
<br />
The fourth requirement, testing, is fulfilled by just observing that the polkit authorization rules are JavaScript files that the user can test any way they want by trivially mocking the <i><b>Polkit</b></i>, <i><b>Action</b></i> and <i><b>Subject</b></i> types via <a href="http://en.wikipedia.org/wiki/Duck_typing">Duck Typing</a> in their favorite server-side JavaScript environment (<a href="https://live.gnome.org/Gjs">gjs</a>, <a href="http://nodejs.org/">node.js</a>, <a href="https://live.gnome.org/Seed">seed</a> etc.).<br />
<br />
After a couple of prototyping attempts, I ended up with something that isn't too awful - see for yourself in the <a href="http://www.freedesktop.org/software/polkit/docs/master/polkit.8.html">polkit(8) man page</a> in the AUTHORIZATION RULES section. I also ended up adding a number of tests for this, including a test to ensure that even runaway-scripts are terminated. Over all, I'm very satisfied with the result.<br />
<br />
<span style="font-size: x-small;">[2] : I also included <a href="https://plus.google.com/102150693225130002912/posts/1vyfmNCYpi5">common-sense advice</a> like "don't ask for a root password for adding printer queues" in this section</span><br />
<h2>
Wait, isn't embedding a JS interpreter inherently dangerous?</h2>
Embedding a JS interpreter is actually perfectly safe. First of all, it all runs inside the <b>polkitd(8)</b> system daemon (which runs without privileges, see below). Second of all, all data passed to the rules are either trusted or from a process that is trusted (except where designated untrusted, e.g. <a href="http://www.freedesktop.org/software/polkit/docs/master/pkexec.1.html">pkexec(1)</a>'s <span style="font-family: 'Courier New', Courier, monospace;">command_line</span> variable). Third, being an interpreted language, we can actually sensibly terminate runaway scripts.<br />
<h2>
Hmm, OK, but you are bloating Linux anyway. You suck.</h2>
<div>
The only new dependency here is libmozjs185.so.1 which in turn depends on the C++ runtime and <a href="http://en.wikipedia.org/wiki/NSPR">NSPR</a> (which <a href="http://en.wikipedia.org/wiki/Network_Security_Services">NSS</a> also depends on and most Linux installs have this library). Note that you also already need a JS interpreter for <a href="http://en.wikipedia.org/wiki/Proxy_auto-config">proxy server auto-configuration</a>. It's also possible (or will be, at least) to just not install <b>polkitd(8)</b> at all (<a href="http://0pointer.de/blog/projects/three-levels-of-off.html">or disable/mask it using systemd</a>) while still having the client-side libraries installed.</div>
<br />
<h2>
Other features</h2>
Apart from the new authorization rules engine, I also made the <b>polkitd(8)</b> system daemon run as the unprivileged <b>polkitd</b> user instead of root (much safer for obvious reasons). The main reason why this hadn't been done before had to do with the fact that <b>polkitd(8)</b> was loading backends via an extension system and we didn't really know if some future backend had to run as root. Since I decided <a href="http://cgit.freedesktop.org/polkit/commit/?id=0f830c76048229895164837f8ce01869d88a2616">nuke</a> the extension system, we no longer need to make such assumptions so changing it was straightforward.<br />
<br />
A while ago, I also added the <a href="http://www.freedesktop.org/software/polkit/docs/latest/pkttyagent.1.html">pkttyagent(1)</a> command on request from Lennart for optional use in <a href="http://www.freedesktop.org/software/systemd/man/systemctl.html">systemctl(1)</a>. Optional here means that it's a soft dependency - systemctl(1) works fine even when polkit is not installed.<br />
<br />
<h2>
Next steps</h2>
I've been doing all this work on the <a href="http://cgit.freedesktop.org/polkit/log/?h=wip/js-rule-files">wip/js-rule-files branch</a> and today I merged this branch to the <a href="http://cgit.freedesktop.org/polkit">master branch</a>. I plan to do a new 0.106 release shortly and put it in what will end up being Fedora 18.<br />
<br />
Once that's available, I plan to file bugs against the most important mechanisms (such as NetworkManager) so they can start exporting variables to be used in authorization rules and also properly document this.<br />
<br />davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com25tag:blogger.com,1999:blog-5268847417417953349.post-24252301268931911382012-03-07T15:14:00.001-05:002012-03-07T16:43:31.281-05:00Simpler, Faster, BetterFor the past year or so, I've been working on and off on the next version of <a href="http://udisks.freedesktop.org/docs/latest/udisks.8.html">udisks</a> and what is now known as the <a href="https://live.gnome.org/Design/Apps/Disks">Disks</a> GNOME application (née <a href="http://en.wikipedia.org/wiki/Palimpsest_Disk_Utility">palimpsest</a>):<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/01-palimpsest2-overview.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="360" src="http://people.freedesktop.org/~david/palimpsest2-2012-03/01-palimpsest2-overview.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Hello Shiny!</i></td></tr>
</tbody></table>
<br />
This code will ship in the upcoming <a href="http://www.gnome.org/">GNOME</a> 3.4 and <a href="http://fedoraproject.org/">Fedora</a> 17 releases.<br />
<br />
The motivation for rewriting udisks and palimpsest hadn't really anything to do with the user-experience per se - the main motivation was to port udisks/palimpsest code from <a href="http://dbus.freedesktop.org/doc/dbus-glib/index.html">dbus-glib</a> to use all the D-Bus work I've been doing the last couple of years, e.g.<br />
<br />
<ul>
<li>Use new <a href="http://developer.gnome.org/gio/unstable/gdbus-lowlevel.html">GDBus</a> <a href="http://developer.gnome.org/gio/unstable/gdbus-convenience.html">routines</a> from GLib</li>
<li>Use the <a href="http://dbus.freedesktop.org/doc/dbus-specification.html#standard-interfaces-objectmanager">org.freedesktop.DBus.ObjectManager interface</a></li>
<li>Use the <a href="http://dbus.freedesktop.org/doc/dbus-specification.html#standard-interfaces-properties">org.freedesktop.DBus.Properties::PropertiesChanged signal</a></li>
<li>Use <a href="http://developer.gnome.org/gio/unstable/gdbus-codegen.html">gdbus-codegen(1)</a></li>
</ul>
Since everything had to be rewritten to use new interfaces, I figured that while I was at it, I'd redesign the UI part as well. A big thanks goes to <a href="http://blogs.gnome.org/mccann/">Jon McCann</a> for helping me design the application and enduring my complicated descriptions of how storage works on Linux.<br />
<br />
<span style="font-size: large;"><b>Cleanups</b></span><br />
<br />
There's a couple of new user-visible features in the code but before delving into specifics I want to mention that we've also removed a couple of features. Why? Because the <a href="http://people.freedesktop.org/~david/gdu-multipath-component-selected-2.png">old interface was</a> <a href="http://people.freedesktop.org/~david/palimpsest-md-raid-components-on-mp.png">quickly becoming</a> <a href="http://people.freedesktop.org/~david/palimpsest-lvm2-pvs-on-mp.png">a mess</a> and it certainly wasn't very <a href="https://live.gnome.org/Design/Apps">GNOME3-ish</a> neither in the way it looks nor the simplicity you'd expect. If there is one mantra that has driven this effort, it may very well be "don't try to expose everything in a desktop user interface" and I think this is something that extends to the greater GNOME 3 effort as well.<br />
<br />
For example, the extent to which Disks support <a href="http://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)">LVM</a> (and <a href="http://en.wikipedia.org/wiki/Mdadm">MD RAID</a> for that matter) is now that we only show the <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/02-palimpsest2-fake-block-device.png">mapped devices</a> (if the VG / array is running) and, except for e.g. showing the user-friendly (!) name <i>/dev/vg_x61/lv_root</i> instead of <i>/dev/dm-2</i>, that's pretty much it. You can still, of course, manipulate the device (such as changing the file system label) and its contents as if it was any other device (see below), but we no longer provide any UI to configure LVM or MD RAID itself - you are expected to use the <a href="http://tldp.org/HOWTO/LVM-HOWTO/">command-line tools</a> to do so yourself.<br />
<br />
Another important goal of Disks that hasn't changed is the realization that we're only one tool among many - we don't pretend to own your machine and that we're the only tool you'll ever use. That's why you'd e.g. see device names in the user interface (without being primary identifiers) so you can copy/paste that to the terminal and use command-line tools on the device. Additionally, thanks to the <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/gudev/GUdevClient.html#GUdevClient-uevent">notifications</a> from <a href="http://en.wikipedia.org/wiki/Udev">udev</a>, the Disks application will update its UI in response to changes you do from e.g. the command-line or other applications - for example, if you mkfs(8) a new filesystem, or add a new partition using parted(8).<br />
<br />
<span style="font-size: large;"><b>Common operations</b></span><br />
<br />
Fundamentally, the Disks end-user interface hasn't changed much (it's still basically a tree-view on the left and volume grid on the right), however instead of <a href="http://people.freedesktop.org/~david/gdu-ata-port-number.png">lots of buttons</a>, most operations are initiated by accessing a popup-menu either for the <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/05-palimpsest2-volume-menu.png">volume</a> or <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/06-palimpsest2-drive-menu.png">drive</a> in question. Common operations like <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/08-palimpsest2-format.png">formatting</a> <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/09-palimpsest2-format-type.png">a</a> <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/10-palimpsest2-format-encrypted.png">device</a>, <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/11-palimpsest2-change-passphrase.png">changing the passphrase of an encrypted device</a>, <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/12-palimpsest2-create-partition.png">creating a new partition</a>, <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/13-palimpsest2-format-disk.png">formatting a</a> <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/14-palimpsest2-format-disk-type.png">disk</a>, <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/21-palimpsest2-ata-smart.png">checking ATA</a> <a href="http://people.freedesktop.org/~david/gdu2-self-test-failed.png">SMART data</a> are pretty much the same, although the UI has been cleaned up some. Pretty basic stuff, really, no big changes here.<br />
<br />
<span style="font-size: large;"><b>Mount and encryption options</b></span><br />
<br />
A common theme since I started working on storage in desktop Linux (some eight years ago), is that some people are extreme control-freaks about where a device is mounted and with what options. Most of the time you won't need this (the desktop shell will automount the device somewhere in <i>/media</i>) but there's a couple of important uses for this. One common example, is that people are using their computer to serve media files to their TV or Media Jukebox over the LAN. For this to work you often need to make sure the device is mounted in say, <i>/mnt/foo (</i>because <i>/mnt/foo</i> is what you added to the config file for the media server) and that this happens before the media server is started.<br />
<br />
We've been trying all sorts of things over the years (the most complex being the now deprecated gnome-mount program reading mount options from e.g. GConf) and while some of these things have worked great in theory, they were just way too complex; our users just didn't have the time nor the inclination to figure out how to use our software... and I don't blame them... my experience is that if the effort more complex than editing a single text file with emacs(1) or vi(1) you've lost most people.<br />
<br />
(In retrospect, realizing when your software is too complex for people to use takes much longer than you think - worse, I suspect a lot of developers don't realize just how over-engineered their software is until it's too late.)<br />
<br />
So, the way it has worked the past few years in GNOME is that if you want specific configuration for mounting a specific disk, I've been telling people to just add an entry to <i>/etc/fstab</i> and use one of the symlinks in <i>/dev/disk</i>. Works great. So it was only natural to actually add some UI for this and the result is this dialog<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/03-palimpsest2-edit-fstab.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="360" src="http://people.freedesktop.org/~david/palimpsest2-2012-03/03-palimpsest2-edit-fstab.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Editing an entry in the /etc/fstab file</i></td></tr>
</tbody></table>
<br />
which basically correspond to the fields in the <i>/etc/fstab</i> file.<br />
<br />
A nice touch here (I think) is that the "Identify As" combo box allows you to select any of the <i>/dev/disk</i> symlinks that currently point to the device, for example, you can make the given mount options <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/03-palimpsest2-edit-fstab-by-path.png">apply to any disk connected to the given USB port</a> (actually, partition 1 from said disk). This is of course nothing new, you've always been able to use any <i>/dev/disk</i> symlink in the <i>/etc/fstab</i> file but I bet most people don't have time or inclination to find out exactly what link to use so most just end up using e.g. <i>/dev/sdb1</i> because that works right now for them. Again, the good old "if it's more complex than editing a single file" line of thought.<br />
<br />
(My secret hope here is that even old Unix neck-beards will want to use such UI dialogs instead of editing the <i>/etc/fstab</i> file manually ... or at the very least realize how such a combo box adds value before they flame the tool to death :-) ...)<br />
<br />
As for <i>/etc/fstab</i>, we also have <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/04-palimpsest2-edit-crypttab.png">similar UI</a> for the <i>/etc/crypttab</i> file except that this is a bit more complex insofar that we allow managing passphrase-files to e.g. automatically unlock a device on boot-up... you may need encrypted <i>/etc</i> for this to be secure - this of course depends on your <a href="http://en.wikipedia.org/wiki/Threat_model">threat model</a> but it could be you have provisions for securely shredding all content in <i>/etc/luks-keys</i> when needed etc. etc., I don't know :-)<br />
<br />
Anyway, the nice thing about using standardized files such as <i>/etc/fstab</i> and <i>/etc/crypttab</i> for this, is interoperability with the rest of the OS - in particular, you can <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/03-palimpsest2-edit-fstab-boot.png">edit the mount options e.g. the boot file-system</a> from the Disks application. Additionally, things like <a href="http://www.freedesktop.org/wiki/Software/systemd">systemd</a> will actually mount devices referenced in the <i>/etc/fstab</i> file at start-up (unless the noauto option is used) which is something you really want for the media server use-case mentioned above.<br />
<br />
<span style="font-size: large;"><b>Disk images and loop devices</b></span><br />
<br />
Another new feature has to do with <a href="http://en.wikipedia.org/wiki/Disk_image">disk images</a> - the Disks application now allows you to <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/15-palimpsest2-create-disk-image.png">create</a> and <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/20-palimpsest2-restore-disk-image.png">restore</a> disk images (<a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/16-palimpsest2-create-disk-image-copying.png">including showing the speed and estimated time left</a>) which is handy especially e.g. from a rescue live cd (it's basically just a GUI for <a href="http://en.wikipedia.org/wiki/Dd_(Unix)">dd(1)</a>). Additionally, there is UI for <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/17-palimpsest2-attach-disk-image.png">attaching disk images</a> and <a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/18-palimpsest2-loop-device.png">using them</a> and, <a href="http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=e03c8dd14915fabc101aa495828d58598dc5af98">thanks to my friend Kay Sievers</a>, it even works with partitioned disk images<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/19-palimpsest2-partitioned-loop-device.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="360" src="http://people.freedesktop.org/~david/palimpsest2-2012-03/19-palimpsest2-partitioned-loop-device.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Loop device, with partitions, oh my!</i></td></tr>
</tbody></table>
<br />
which I think is pretty handy.<br />
<br />
<span style="font-size: large;"><b>Desktop integration</b></span><br />
<br />
Another important part of the equation is what the user sees when it comes to storage devices in general (e.g. in general, not only when using the Disks application). This is largely the domain of the <a href="https://live.gnome.org/GnomeShell/Design">Desktop Shell</a>, the <a href="https://live.gnome.org/Design/Apps/Files">Files application</a> (née Nautilus) and the <a href="http://developer.gnome.org/gtk3/unstable/GtkFileChooserDialog.html">file chooser</a>. These pieces all rely on the <a href="http://developer.gnome.org/gio/unstable/GVolumeMonitor.html">GLib Volume Monitor APIs</a> which is a high-level abstraction with a stable API/ABI that applications can use (among other things) to figure out what volumes to present in their user interfaces.<br />
<br />
In a nutshell, the volumes to show is a complex mixture of physical storage devices (say, mountable filesystems residing on a plugged-in USB stick), devices backed by GVfs backends (say, digital cameras and iPod that are not block devices), GVfs network backends (e.g. smb:// network mounts) and non-storage mounts such as e.g. NFS mounts, FUSE mounts and fstab entries representing things that <i>can</i> be mounted but may not already be mounted.<br />
<br />
Now, the way GVfs works is pretty complex (there are multiple volume monitor daemons and storage backends instances) but the important point is that the volume monitor responsible for device and fstab mounts has been updated to use udisks2, the same storage daemon used by the Disks application. In addition to returning GDrive, GVolume and GMount objects for storage devices, this volume monitor also return GVolume entries for <i>/etc/fstab</i> entries so you can e.g. mount a NFS server by clicking an icon in Files (again, a common user request), like this<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://people.freedesktop.org/~david/palimpsest2-2012-03/22-palimpsest2-desktop-integration.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="360" src="http://people.freedesktop.org/~david/palimpsest2-2012-03/22-palimpsest2-desktop-integration.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Desktop Integration. Bringing it all together</i></td></tr>
</tbody></table>
<br />
Note that the name, icon and desktop visibility of the GVolume instance representing the NFS mount is controlled by mount options in the <i>/etc/fstab</i> file - see <a href="http://git.gnome.org/browse/gvfs/tree/monitor/udisks2/what-is-shown.txt">this write-up</a> for more information.<br />
<br />davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com28tag:blogger.com,1999:blog-5268847417417953349.post-24412807720824170312011-09-22T16:44:00.004-04:002011-09-22T16:59:52.537-04:00New D-Bus features in GLib 2.30For the upcoming <a href="http://en.wikipedia.org/wiki/GLib">GLib</a> 2.30 release, there's a couple of new features to make it even easier to use <a href="http://en.wikipedia.org/wiki/D-Bus">D-Bus</a>.<br />
<h1>
C Code Generator</h1>
GLib 2.30 ships with a command called <a href="http://developer.gnome.org/gio/unstable/gdbus-codegen.html">gdbus-codegen(1)</a> which can be used to generate C code. The command is similar in spirit to the <a href="http://dbus.freedesktop.org/doc/dbus-glib/dbus-binding-tool.html">dbus-binding-tool(1)</a> command but it targets the D-Bus implementation added to GLib 2.26 (often informally referred to as GDBus). The command's manual page and <a href="http://developer.gnome.org/gio/unstable/ch29.html">migration documentation</a> is the authoritative source of documentation, but from a 50,000 feet view what the tool does is simply to map a D-Bus interface (described by XML) to a couple of <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html">GObject</a>-based types (including all D-Bus methods, signals and properties). It of course supports the <a href="http://dbus.freedesktop.org/doc/dbus-specification.html#standard-interfaces-properties">PropertiesChanged signal</a> added in D-Bus spec 0.14.<br />
<br />
Even though the tool is targeting C programmers (most higher-level languages are a lot more dynamic than C so objects can be exported via e.g. language meta-data such as <a href="http://en.wikipedia.org/wiki/Java_annotation">annotations</a>), it's useful to note that the generated code is 100% annotated so it can be used from e.g. JavaScript through <a href="https://live.gnome.org/GObjectIntrospection">GObject Introspection</a> - for example the <a href="https://live.gnome.org/Design/Apps/Documents">GNOME Documents</a> application (written in JavaScript) is consuming the <a href="http://people.freedesktop.org/~david/goa-20110512/">GNOME Online Accounts</a> client-library which is 99% generated by the gdbus-codegen(1) command.<br />
<br />
In addition to just generating code, the gdbus-codegen(1) command can also generate very nice Docbook documentation (<a href="http://people.freedesktop.org/~david/goa-20110512/ch02.html">example 1</a>, <a href="http://people.freedesktop.org/~david/udisks2-20110719/ch02.html">example 2</a>) for D-Bus interfaces - in this respect it's useful even for non-GNOME applications in the same way a lot of libraries like libblkid and libudev are already using <a href="https://live.gnome.org/DocumentationProject/GtkDoc">gtk-doc</a> for their C library documentation.<br />
<br />
<h1>
Object Manager</h1>
Another part of GLib 2.30 is support for the <a href="http://dbus.freedesktop.org/doc/dbus-specification.html#standard-interfaces-objectmanager">org.freedesktop.DBus.ObjectManager</a> D-Bus interface - in fact, the implementation in GLib, that is, the newly added <a href="http://developer.gnome.org/gio/unstable/GDBusObject.html">GDBusObject</a>{<a href="http://developer.gnome.org/gio/unstable/GDBusObjectProxy.html">Proxy</a>,<a href="http://developer.gnome.org/gio/unstable/GDBusObjectSkeleton.html">Skeleton</a>}and <a href="http://developer.gnome.org/gio/unstable/GDBusObjectManager.html">GDBusObjectManager</a>{<a href="http://developer.gnome.org/gio/unstable/GDBusObjectManagerClient.html">Client</a>,<a href="http://developer.gnome.org/gio/unstable/GDBusObjectManagerServer.html">Server</a>} types, is actually what drove me to <a href="https://bugs.freedesktop.org/show_bug.cgi?id=34869">propose this</a> as a standard interface instead of just doing a GLib-only thing (with shoutouts to my homeboys <a href="http://smcv.pseudorandom.co.uk/">smcv</a> and <a href="http://blog.verbum.org/">walters</a> for excellent review and feedback).<br />
<br />
In a nutshell, the org.freedesktop.DBus.ObjectManager interface is basically a formalization of what each and every non-trivial D-Bus service is already doing: offering some kind of GetAll() method (to return all objects) and signals ::Foo{Added,Removed} (to convey changes) on its Manager interface in its own special way (<a href="http://projects.gnome.org/NetworkManager/developers/api/09/spec.html#org.freedesktop.NetworkManager">example 1</a>, <a href="http://hal.freedesktop.org/docs/udisks/UDisks.html">example 2</a>). It first sounds weird to standardize such a simple thing as object enumeration and change signals but if you think about all the possible <a href="http://developer.gnome.org/gio/unstable/GDBusObjectManagerClient.html#GDBusObjectManagerClient.description">edge cases and race-conditions</a> then using a well-tested implementation just makes everything so much easier. Additionally, with the way the interface is defined and the <a href="https://bugs.freedesktop.org/show_bug.cgi?id=34870">newly added path_namespace match rule</a>, two method invocations and a single round-trip is all it takes for a client to grab the state from the service - this is a huge win compared to existing services that typically first retrieve a list of object paths and then gets properties for each object path (and only if you are lucky it does the latter in parallel).<br />
<br />
When combined with the gdbus-codegen(1) command (which can also generate specialized GDBusObject types for use with GDBusObjectManager) you can start writing service-specific code <b>right away</b> and not worry about horrible implementation details like marshaling, too many round trips or race conditions. It just works out of the box as you'd expect it to.davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com1tag:blogger.com,1999:blog-5268847417417953349.post-22422617229198728342011-07-06T12:58:00.008-04:002011-07-06T17:33:56.084-04:00Writing a C library, intro, conclusion and errata<i>This is a series of blog-posts about best practices for writing C libraries. See below for each part and the topics covered.</i><br />
<h1>Table of contents</h1>The entire series about best practices for writing C libraries covered 15 topics and was written over five parts posted over the course of approximately one week. Feel free to hotlink directly to each topic but please keep in mind that the content (like any other content on this blog) is copyrighted by its author and may not be reproduced without his consent (if you are friendly towards free software, like e.g. <a href="http://lwn.net/">LWN</a>, just ask and I will probably give you permission):<br />
<ul><li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html">Part one</a></li>
<ul><li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html#base-libraries">Base libraries</a></li>
<li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html#library-init">Library initialization and shutdown</a></li>
<li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html#memory-mgmt">Memory management</a></li>
<li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html#multiple-threads-and-processes">Multiple threads and processes</a></li>
</ul><li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-2.html">Part two</a></li>
<ul><li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-2.html#event-handling">Event handling and the main loop</a></li>
<li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-2.html#io">Synchronous and asynchronous I/O</a></li>
<li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-3.html#modularity">Modularity and namespaces</a></li>
</ul><li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-3.html">Part three</a></li>
<ul><li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-3.html#error-handling">Error handling</a></li>
<li><a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-3.html#encapsulation-and-oo">Encapsulation and OO design</a></li>
</ul><li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-4.html">Part four</a></li>
<ul><li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-4.html#helpers-and-daemons">Helpers and daemons</a></li>
<li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-4.html#testing">Testing</a></li>
</ul><li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-5.html">Part five</a></li>
<ul><li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-5.html#api-design">API design</a></li>
<li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-5.html#documentation">Documentation</a></li>
<li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-5.html#language-bindings">Language bindings</a></li>
<li><a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-5.html#abi-api-versioning">ABI, API and versioning</a></li>
</ul></ul><h1>Topics not covered</h1>Some topics relevant for writing a C library isn't (yet?) covered in this series either because I'm not an expert on the topic, the topic is still in development or for other reasons:<br />
<ul><li><b>Networking</b><br />
You would think IP networking is easy but it's really not and the low-level APIs that are part of <a href="http://en.wikipedia.org/wiki/POSIX">POSIX</a> (e.g. <a href="http://en.wikipedia.org/wiki/Berkeley_sockets">BSD Sockets</a>) are not really that helpful since they only do part of what you need. Difficult things here include <a href="http://en.wikipedia.org/wiki/Domain_Name_System">name resolution</a>, <a href="http://en.wikipedia.org/wiki/SRV_record">service resolution</a>, <a href="http://en.wikipedia.org/wiki/Proxy_server">proxy server handling</a>, <a href="http://blogs.gnome.org/dcbw/2011/06/14/networkmanager-and-dual-stack-addressing/">dual-stack addressing</a> and <a href="http://en.wikipedia.org/wiki/Transport_Layer_Security">transport security</a> (including handling <a href="http://en.wikipedia.org/wiki/Digital_certificate">certificates</a> for authentication).<br />
<br />
If you are using modern GLib networking primitives (such as <a href="http://developer.gnome.org/gio/unstable/GSocketClient.html">GSocketClient</a> or <a href="http://developer.gnome.org/gio/unstable/GSocketService.html">GSocketService</a>) all of these problems are taken care of for you without you having to do much work; if not, well, talking to people (or at least, read the blogs) such as <a href="http://danw.mysterion.org/">Dan Winship</a>, <a href="http://blogs.gnome.org/dcbw/">Dan Williams</a> or <a href="http://0pointer.de/blog">Lennart Poettering</a> is probably your best bet.<br />
<br />
</li>
<li><b>Build systems</b><br />
This is a topic that continues to make me sad so I decided to not really cover it in the series because the best guidance I can give is to just copy/paste whatever other projects are doing - see e.g. the GLib source tree for how to nicely integrate unit testing (see Makefile.decl) and documentation (see docs/reference sub-directories) into the build system (<a href="http://git.gnome.org/browse/glib/tree/">link</a>).<br />
<br />
Ideally we would have a single great IDE for developing Linux libraries and applications (integrating testing, documentation, distribution, package building and so on - see e.g. Sami's <a href="http://wagiaalla.com/2011/05/17/devhelp-support-in-eclipse-through-libhover/">libhover video</a>) but even if we did, most existing Linux programmers probably wouldn't use it because they are so used to e.g. emacs or vi (<a href="http://en.wikipedia.org/wiki/If_you_build_it,_they_will_come">if you build it, they will come</a>?). There's a couple of initiatives in this area including <a href="http://www.eclipse.org/cdt/">Eclipse CDT</a>, <a href="http://www.anjuta.org/">Anjuta</a>, <a href="http://kdevelop.org/">KDevelop</a> and <a href="http://monodevelop.com/">MonoDevelop</a>.<br />
<br />
</li>
<li><b>Bundling libraries/resources</b><br />
The traditional way of distributing applications on Linux is through so-called <a href="http://en.wikipedia.org/wiki/Linux_distribution">Linux distributions</a> - the four most well-known being <a href="http://www.debian.org/">Debian</a>, <a href="http://fedoraproject.org/">Fedora</a>, <a href="http://www.opensuse.org/en/">openSUSE</a> and <a href="http://www.ubuntu.com/">Ubuntu</a> (in alphabetical order!). These guys, basically, take your source code, compile it against some version of other software it depends on (usually a different version than you, the developer, used), and then ship binary packages to users using dynamic linking.<br />
<br />
There's a couple of problems with this legacy model of distributing software (this list is not exhaustive): <b>a) </b>it can take up to one or two distribution release cycles (6-12 months) before your software is available to end users; and <b>b)</b> user X can't give a copy of the software to user Y - he can only tell him where to get it (it might not be available on user Y's distro); and <b>c) </b>it's all a <a href="http://en.wiktionary.org/wiki/hodgepodge">hodgepodge</a> of version skew e.g. the final product that your users are using is, most likely, using different versions of different libraries so who knows if it works; and <b>d) </b>the software is sometimes changed in ways that you, the original author, wasn't expecting or does not approve of (for example, by removing credits); and <b>e)</b> the distribution might not forward you bug reports or may forward you bug reports that is caused by downstream patches; and <b>f)</b> there's a <a href="http://en.wikipedia.org/wiki/Peer_pressure">peer pressure</a> to not depend on too new libraries because distributions want to ship your software in old versions of their OS - for example, <a href="http://www.getfirefox.net/">Mozilla</a> wants to be able to run on a system with just GTK+ 2.8 installed (and hence won't use features in GTK+ 2.10 or later except for using <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/dlsym.3.html">dlopen()</a>-techniques), similar for e.g. <a href="http://www.google.com/chrome/">Google Chrome</a> (maybe with a newer GTK+ version though). These problems are virtually unknown to developers on other platforms such as <a href="http://en.wikipedia.org/wiki/Microsoft_Windows">Microsoft Windows</a>, <a href="http://en.wikipedia.org/wiki/Mac_OS_X">Mac OS X</a> or even some of the smartphone platforms such as <a href="http://en.wikipedia.org/wiki/IOS_(Apple)">iOS</a> or <a href="http://en.wikipedia.org/wiki/Android_(operating_system)">Android</a> - they all have fancy tools that bundles things up nicely so the developers won't have to worry about such things.<br />
<br />
There's a couple of interesting initiatives in this area see e.g. <a href="https://github.com/rubenv/bockbuild#readme">bockbuild</a>, <a href="http://blogs.gnome.org/alexl/2007/08/07/experiments-with-runtime-less-app-bundles/">glick</a> and the proposal to add a <a href="http://mail.gnome.org/archives/gtk-devel-list/2011-April/msg00079.html">resource-system</a> to GLib. Note that it's very very hard to do this properly since it depends not only on fixing a lot of libraries so they are <a href="https://bugzilla.gnome.org/show_bug.cgi?id=615903">relocatable</a>, it also depends on identifying exactly what kind of run-time requirements each library in question has. The latter includes the <a href="http://en.wikipedia.org/wiki/Kernel_(computing)">kernel</a>/<a href="http://en.wikipedia.org/wiki/Udev">udev</a> version, the <a href="http://en.wikipedia.org/wiki/C_standard_library">libc</a> version (unless bundled or statically linked), the <a href="http://en.wikipedia.org/wiki/X_Window_System">X11 server</a> version (and its extensions such as e.g. <a href="http://en.wikipedia.org/wiki/XRender">RENDER</a>) version, the presence of one or more <a href="http://en.wikipedia.org/wiki/D-Bus">message buses</a> and so on. With modern techniques such as <a href="http://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure">direct rendering</a> this becomes even harder if you want to take advantage of hardware acceleration since you must assume that the host OS is providing recent enough versions of e.g. <a href="http://en.wikipedia.org/wiki/OpenGL">OpenGL</a> or <a href="http://en.wikipedia.org/wiki/Cairo_(graphics)">cairo</a> libraries (since you don't want to bundle hardware drivers). And even after all this, you still need to deal with how each distribution patches core components. In some circumstances it might end up being easier to just ship a <a href="http://en.wikipedia.org/wiki/Just_enough_operating_system">kernel+runtime</a> along with the application, <a href="http://en.wikipedia.org/wiki/Virtualization">virtualized</a>.</li>
</ul>The way the series is set up is so it can be extended at a later point - so if there is a demand for one or more popular topics about writing a C library, I might write another blog entry and add it to this page as it's considered the canonical location for the entire series.<br />
<h1>Errata</h1>Please send me feedback and I will fix up the section in question and credit you here (I already have a couple of corrections lined up that I will add later).davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com10tag:blogger.com,1999:blog-5268847417417953349.post-66594813201702018912011-07-05T15:31:00.006-04:002011-07-06T13:00:26.512-04:00Writing a C library, part 5<i>This is part five in a <a href="http://davidz25.blogspot.com/2011/07/writing-c-library-intro-conclusion-and.html">series of blog-posts</a> about best practices for writing C libraries. Previous installments: <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html">part one</a>, <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-2.html">part two</a>, <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-3.html">part three</a>, <a href="http://davidz25.blogspot.com/2011/07/writing-c-library-part-4.html">part four</a>.</i><br />
<a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=6659481320170201891" name="api-design"></a><br />
<h1>API design</h1>A C library is, almost by definition, something that offers an <a href="http://en.wikipedia.org/wiki/Application_programming_interface">API</a> that is used in applications. Often an API can't be changed in incompatible ways (it can, however, be extended) so it is usually important to get right the first time because if you don't, you and your users will have to live with your mistakes for a long time.<br />
<br />
This section is not a a full-blown guide to API design as there's a lot of literature, courses and presentations available on the subject - see e.g. <a href="http://cworth.org/~cworth/papers/guadec_2006/">Designing a library that's easy to use</a> - but we will mention the most important principles and a couple of examples of good and bad API design.<br />
<br />
The main goals when it comes to API design is, of course, to make the API easy to use - this include <a href="http://stackoverflow.com/questions/143701/what-is-the-worst-class-variable-function-name-you-have-ever-encountered">choosing good names</a> for types, functions and constants. Be careful of abbreviations - <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/atof.3.html">atof</a> might be quick to type but it's not exactly clear that the function parses a C string and returns a double (no, not a float as the name suggests). Typically <a href="http://en.wikipedia.org/wiki/Noun">nouns</a> are used for types and while <a href="http://en.wikipedia.org/wiki/Verb">verbs</a> are used for methods.<br />
<br />
Another thing to keep in mind is the number of function arguments - ideally each function should take only a few arguments so it's easy to remember how to use it. For example, no-one probably ever remembers exactly what arguments to pass to <a href="http://developer.gnome.org/glib/unstable/glib-Spawning-Processes.html#g-spawn-async-with-pipes">g_spawn_async_with_pipes()</a> so programmers end up looking up the docs, <a href="http://www.codinghorror.com/blog/2006/09/the-multi-tasking-myth.html">breaking the rhythm</a>. A better approach (which is yet to be implemented in GLib), would be to create a new type, <a href="https://bugzilla.gnome.org/show_bug.cgi?id=646635#c96">let's call it GProcess</a>, with methods to set what you'd otherwise pass as arguments and then a method to spawn the actual program. Not only is this easier to use, it is also extensible as adding a method to a type doesn't break API while adding an argument to an existing function/method does. An example of such an API is libudev's <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/libudev-udev-enumerate.html">udev_enumerate</a> API - for example, at the time udev starting dealing with <a href="http://en.wikipedia.org/wiki/Tag_(metadata)">device tags</a>, the <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/libudev-udev-enumerate.html">udev_enumerate</a> type gained the <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/libudev-udev-enumerate.html#udev-enumerate-add-match-tag">add_match_tag()</a> method.<br />
<br />
If using constants, it is often useful to use the <a href="http://en.wikipedia.org/wiki/Enumerated_type">C enum type</a> since the compiler can warn if a switch statement isn't handling all cases. Generally avoid boolean types in functions and use <a href="http://en.wikipedia.org/wiki/Flag_(computing)">flag enumerations</a> instead - this has two advantages: first of all, it's sometimes easier to read foo_do_stuff(foo, FOO_FLAGS_FROBNICATOR) than foo_do_stuff(foo, TRUE) since the reader does not have to expend mental energy on remembering if TRUE translates into whether the frobnicator is to be used or not. Second, it means that several booleans arguments can be passed in one parameter so hard-to-use functions like e.g. <a href="http://developer.gnome.org/gtk3/unstable/GtkBox.html#gtk-box-pack-start">gtk_box_pack_start()</a> can be avoided (most programmers can't remember if the <i>expand</i> or <i>fill</i> boolean comes first). Additionally, this technique allows adding new flags without breaking API.<br />
<br />
Often the compiler can help - for example, C functions can be annotated with all kinds of <a href="http://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html">gcc-specific annotations</a> that will cause warnings if the user is not using the function correctly. If using, GLib, some of these annotations are available as macros prefixed with G_GNUC, the most important ones being <a href="http://developer.gnome.org/glib/unstable/glib-Miscellaneous-Macros.html#G-GNUC-CONST:CAPS">G_GNUC_CONST</a>, <a href="http://developer.gnome.org/glib/unstable/glib-Miscellaneous-Macros.html#G-GNUC-PURE:CAPS">G_GNUC_PURE</a>, <a href="http://developer.gnome.org/glib/unstable/glib-Miscellaneous-Macros.html#G-GNUC-MALLOC:CAPS">G_GNUC_MALLOC</a>, <a href="http://developer.gnome.org/glib/unstable/glib-Miscellaneous-Macros.html#G-GNUC-DEPRECATED-FOR:CAPS">G_GNUC_DEPRECATED_FOR</a>, <a href="http://developer.gnome.org/glib/unstable/glib-Miscellaneous-Macros.html#G-GNUC-PRINTF:CAPS">G_GNUC_PRINTF</a> and <a href="http://developer.gnome.org/glib/unstable/glib-Miscellaneous-Macros.html#G-GNUC-NULL-TERMINATED:CAPS">G_GNUC_NULL_TERMINATED</a>.<br />
<h2>Checklist</h2><ul><li>Choose good type and function names (favor expressiveness over length).</li>
<li>Keep the number of arguments to functions down (consider introducing helper types).</li>
<li>Use the type system / compiler to your advantage instead of fighting it (enums, flags, compiler annotations).</li>
</ul><a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=6659481320170201891" name="documentation"></a><br />
<h1>Documentation</h1>If your library is very simple, the best documentation might just be a nicely formatted C header file with inline comments. Often it's not that simple and people using your library might expect richer and cross-referenced documentation complete with code samples.<br />
<br />
Many C libraries, including those in GLib and GNOME itself, use inline documentation tags that can be read by tools such as <a href="http://www.gtk.org/gtk-doc/">gtk-doc</a> or <a href="http://en.wikipedia.org/wiki/Doxygen">Doxygen</a>. Note that gtk-doc works just fine even on low-level non-GLib-using libraries - see e.g. <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/">libudev</a> and <a href="http://www.kernel.org/pub/linux/utils/util-linux/libblkid-docs/">libblkid</a> API documentation.<br />
<br />
If used with a GLib library, gtk-doc uses the GLib type system to <a href="http://developer.gnome.org/gio/unstable/gio-hierarchy.html">draw type hierarchies</a> and show type-specific things like <a href="http://developer.gnome.org/gtk3/unstable/GtkWidget.html#GtkWidget.properties">properties</a> and <a href="http://developer.gnome.org/gtk3/unstable/GtkWidget.html#GtkWidget.signals">signals</a>. gtk-doc can also easily integrate with any tool producing <a href="http://docbook.org/">Docbook</a> documentation such as <a href="http://en.wikipedia.org/wiki/Man_page">manual pages</a> or e.g. <a href="http://developer.gnome.org/gio/unstable/gdbus-codegen.html">gdbus-codegen(1)</a> when used to generate docs describing D-Bus interfaces (<a href="http://people.freedesktop.org/~david/udisks2-20110628/">example with C API docs, D-Bus docs and man pages</a>).<br />
<h2>Checklist</h2><ul><li>Decide what level of documentation is needed (HTML, pdf, man pages, etc.).</li>
<li>Try to use standard tools such as Doxygen or gtk-doc.</li>
<li>If shipping commands/daemons/helpers (e.g. anything showing up in <a href="http://en.wikipedia.org/wiki/Ps_(Unix)">ps(1)</a> output), consider shipping man pages for those as well.</li>
</ul><a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=6659481320170201891" name="language-bindings"></a><br />
<h1>Language bindings</h1>C libraries are increasingly used from higher-level languages such as <a href="http://en.wikipedia.org/wiki/Python_(programming_language)">Python</a> or <a href="http://en.wikipedia.org/wiki/JavaScript">JavaScript</a> through a so-called <a href="http://en.wikipedia.org/wiki/Language_binding">language binding</a> - for example, this is what allows the <a href="https://live.gnome.org/GnomeShell/">Desktop Shell</a> in <a href="http://www.gnome.org/">GNOME 3</a> to be written entirely in JavaScript while still using C libraries such as <a href="http://en.wikipedia.org/wiki/GLib">GLib</a>, <a href="http://www.clutter-project.org/">Clutter</a> and <a href="http://git.gnome.org/browse/mutter/">Mutter</a> underneath.<br />
<br />
It's outside the scope of this article to go into detail on language bindings (however a lot of the advice given in this series does apply - see also: <a href="https://live.gnome.org/GObjectIntrospection/WritingBindingableAPIs">Writing Bindable APIs</a>) but it's worth pointing out that the goal of the <a href="https://live.gnome.org/GObjectIntrospection">GObject Introspection</a> project (which is what is used in GNOME's Shell) is aiming for 100% coverage of GLib libraries assuming the library is properly <a href="https://live.gnome.org/GObjectIntrospection/Annotations">annotated</a>. For example, this applies to the <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/gudev/">GUdev library</a> (a thin wrapper on top of the <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/">libudev library</a>) can be used from any language that supports GObject Introspection (<a href="http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=extras/gudev/gjs-example.js;h=5586fd6a61fcd8cfcadd7dc79e25be3e8b210cce;hb=4c748b05a3bd0c36245e71b24dc45e2884f80376">JS example</a>).<br />
<br />
GObject Intropspection is interesting because if someone adds GObject Introspection support to a new language, X, then the GNOME platform (and a lot of the underlying Linux plumbing as well cf. GUdev) is now suddenly available from that language without any work.<br />
<h2>Checklist</h2><ul><li>Make sure your API is easily bindable (avoid C-isms such as <a href="http://en.wikipedia.org/wiki/Variadic_function">variadic functions</a>).</li>
<li>If using GLib, set up GObject Introspection and ship GIR/typelibs (<a href="http://blog.verbum.org/2008/10/06/dear-gobjectc-library-authors/">notes</a>).</li>
<li>If writing a complicated application, consider writing parts of it in C and parts of it in a higher-level language.</li>
</ul><a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=6659481320170201891" name="abi-api-versioning"></a><br />
<h1>ABI, API and versioning</h1>While the API of a library describes how the programmer use it, the <a href="http://en.wikipedia.org/wiki/Application_binary_interface">ABI</a> describes how the API is mapped onto the target machine the library is running on. Roughly, a (shared) library is said to be compatible with a previous version if a recompile is not needed. The ABI involves a lot of factors including <a href="http://en.wikipedia.org/wiki/Data_structure_alignment">data alignment rules</a>, <a href="http://en.wikipedia.org/wiki/Calling_convention">calling conventions</a>, <a href="http://en.wikipedia.org/wiki/Executable_and_Linkable_Format">file formats</a> and other things that are not suitable to cover in this series; the important thing to know about when writing C libraries is how (and if) the ABI changes when the API changes. Specifically, since some changes (such as adding a new function) are backwards compatible, the interesting question is what kind of API changes result in non-backwards-compatible ABI changes.<br />
<br />
Assuming all other factors like calling convention are constant, the rule of thumb about compatibility on the ABI level basically boils down to a very short list of allowed API changes:<br />
<ul><li>you may add new C functions; and</li>
<li>you may add parameters to a function only if it doesn't cause a memory/resource leak; and</li>
<li>you may add a return value to a function returning <a href="http://en.wikipedia.org/wiki/Void_type">void</a> only if it doesn't cause a memory leak; and</li>
<li>modifiers such as <a href="http://en.wikipedia.org/wiki/Const-correctness">const</a> may be added / removed at will since they are not part of the ABI in C</li>
</ul>The latter is an example of a change that breaks the API (causing compiler warnings when compiling existing programs that used to compile without warnings) but preserve the ABI (still allowing any previously compiled program to run) - see e.g. <a href="http://git.gnome.org/browse/glib/commit/?id=cdd04f36cac838f92826cbf415868100a4007741">this GLib commit</a> for a concrete example (note that this can't be done in C++ because of how <a href="http://en.wikipedia.org/wiki/Name_mangling">name mangling</a> work).<br />
<br />
In general, you may not extend C structs that the user can allocate on the stack or embed in another C structure which is why <a href="http://en.wikipedia.org/wiki/Opaque_data_type">opaque data types</a> are often used since they can be extended without the user knowing. In case the data type is not opaque, an often used technique is to add padding to structs (<a href="http://git.gnome.org/browse/glib/tree/gio/gvolumemonitor.h?id=2.29.10#n123">example</a>) and use it when adding a new virtual method or signal function pointer (<a href="http://git.gnome.org/browse/glib/commit/?id=7a5a2be92b0502deb171830020124bc6c4b9f65e">example</a>). Other types, such as enumeration types, may normally be extended with new constants but existing constants may not be changed unless explicitly allowed.<br />
<br />
The semantics of a function, e.g. its <a href="http://en.wikipedia.org/wiki/Side_effect_(computer_science)">side effect</a>, is usually considered part of the ABI. For example, if the purpose of a function is to print diagnostics on <a href="http://en.wikipedia.org/wiki/Standard_output">standard output</a> and it stops doing it in a later version of the library, one could argue it's an ABI break even when existing programs are able to call the function and return to the caller just fine possibly even returning the same value.<br />
<br />
On Linux, <a href="http://en.wikipedia.org/wiki/Shared_library#Shared_libraries">shared libraries</a> (similar to <a href="http://en.wikipedia.org/wiki/Dynamic-link_library">DLL</a>s on Windows) use the so-called <a href="http://en.wikipedia.org/wiki/Soname">soname</a> to maintain and provide backwards-compatibility as well as allowing having multiple incompatible run-time versions installed at the same time. The latter is achieved by increasing the major version number of a library every time a backwards-incompatible change is made. Additionally, other fields of the soname have other (complex) rules associated (<a href="http://sourceware.org/autobook/autobook/autobook_91.html#SEC91">more info</a>).<br />
<br />
One solution to managing non-backwards-compatible ABI changes without bumping the so-number is <a href="http://www.akkadia.org/drepper/symbol-versioning">symbol versioning</a> - however, apart from being hard to use, it only applies to functions and not e.g. higher-level run-time data structures like e.g. signals, properties and types registered with the GLib type-system.<br />
<br />
It is often desirable to have multiple incompatible versions of libraries and their associated development tools installed at the same time (and in the same prefix) - for example, both version 2 and 3 of <a href="http://www.gtk.org/">GTK+</a>. To easily achieve this, many libraries (including GLib and up) include the major version number (which is what is bumped exactly when non-backwards-compatible changes are made) in the library name as well as names of tools and so on - see the <a href="http://ometer.com/parallel.html">Parallel Installation</a> essay for more information.<br />
<br />
Some libraries, especially when they are in their early stages of development, specifically gives no ABI guarantees (and thus, does not manage their soname when incompatible changes are made). Often, to better manage expectations, such unstable libraries require that the user defines a macro acknowledging this (<a href="http://git.gnome.org/browse/gnome-online-accounts/tree/src/goa/goa.h?id=3.1.1#n26">example</a>). Once the library is baked, this requirement is then removed and normal ABI stability rules starts applying (<a href="http://cgit.freedesktop.org/dbus/dbus/commit/?id=a947713c354462d1235e6ef3d8d3929bf9f080d4">example</a>).<br />
<br />
Related to versioning, it's important to mention that in order for your library to be easy to use, it is absolutely crucial that it includes pkg-config files along with the header files and other development files (<a href="http://www.freedesktop.org/wiki/Software/pkg-config">more information</a>).<br />
<h2>Checklist</h2><ul><li>Decide what ABI guarantees to give if any (and when)</li>
<li>Make sure your users understand the ABI guarantees (being explicit is good)</li>
<li>If possible, make it possible to have multiple incompatible versions of your library and tools installed at the same time (e.g. include the major version number in the library name)</li>
</ul>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com6tag:blogger.com,1999:blog-5268847417417953349.post-83815010828492902902011-07-01T15:19:00.002-04:002011-07-06T13:00:11.981-04:00Writing a C library, part 4<i>This is part four in a <a href="http://davidz25.blogspot.com/2011/07/writing-c-library-intro-conclusion-and.html">series of blog-posts</a> about best practices for writing C libraries. Previous installments: <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html">part one</a>, <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-2.html">part two</a>, <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-3.html">part three</a>.</i><br />
<a href="" name="helpers-and-daemons"></a><br />
<h1>Helpers and daemons</h1>Occasionally it's useful for a program or library to call upon an external process to do its bidding. There are many reasons for doing this - for example, the code you want to use<br />
<ul><li>might not be easily used from C - it could be written in say, <a href="http://en.wikipedia.org/wiki/Python_(programming_language)">python</a> or, gosh, <a href="http://en.wikipedia.org/wiki/Bash_(Unix_shell)">bash</a>; or</li>
<li>could mess with <a href="http://www.kernel.org/doc/man-pages/online/pages/man7/signal.7.html">signal handlers</a> or other global process state; or</li>
<li>is not thread-safe or leaking or just bloated; or</li>
<li>its error handling is incompatible with how your library does things; or</li>
<li>the code needs elevated privileges; or</li>
<li>you have a bad feeling about the library but it's not worth (or (politically) feasible) to re-implement the functionality yourself. </li>
</ul>There are three main ways of doing this.<br />
<br />
The first one is to just call <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/fork.2.html">fork(2)</a> and start using the new library in the child process - this usually doesn't work because chances are that you are already using libraries that cannot be reliably used after the fork() call as discussed in <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html">previously</a> (additionally, a lot of unnecessary <a href="http://en.wikipedia.org/wiki/Copy-on-write">COW</a> might be happen if the parent process has a lot of writable pages mapped). If portability to Windows is a concern, this is also a non-starter as Windows does not have fork() or any meaningful equivalent that is as efficient.<br />
<br />
The second way is to write a small helper program and distribute the helper along with your library. This also uses fork() but the difference is that one of the <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/exec.3.html">exec(3)</a> functions is called immediately in the child process so all previous process state is cleaned up when the process image is replaced (except for file descriptors as they are inherited across exec() so be wary of undesired leaks). If using GLib, there's a couple of (portable) useful <a href="http://developer.gnome.org/glib/unstable/glib-Spawning-Processes.html">utility functions</a> to do this (including support for <a href="http://developer.gnome.org/glib/unstable/glib-Spawning-Processes.html#GSpawnFlags">automatically closing file descriptors</a>).<br />
<br />
The third way is to have your process <a href="http://en.wikipedia.org/wiki/Inter-process_communication">communicate</a> with a long-lived helper process (a socalled <a href="http://en.wikipedia.org/wiki/Daemon_(computing)">daemon</a> or background process). The helper daemon can be launched either by <a href="http://dbus.freedesktop.org/doc/dbus-daemon.1.html">dbus-daemon(1)</a> (if you are using <a href="http://en.wikipedia.org/wiki/D-Bus">D-Bus</a> as the IPC mechanism), <a href="http://www.freedesktop.org/wiki/Software/systemd">systemd</a> if you are using e.g. <a href="http://www.kernel.org/doc/man-pages/online/pages/man7/unix.7.html">Unix domain sockets</a>, an init script (<a href="http://linux.die.net/man/8/uuidd">uuidd(8)</a> used to do this - wasteful if your library is not going to get used) or by the library itself.<br />
<br />
Helper daemons usually serve multiple instances of library users, however it is sometimes desirable to have a helper daemon instance per library user instance. Note that having a library spawn a long-lived process by itself is usually a bad idea because the environment and other inherited process state might be wrong (or even insecure) - see <a href="http://0pointer.de/blog/projects/systemd.html">Rethinking PID 1</a> for more details on why a good, known, minimal and secure working environment is desirable. Another thing that is horribly difficult to get right (or, rather, horribly easy to get wrong) is <a href="http://en.wikipedia.org/wiki/Singleton_pattern">uniqueness</a> - e.g. you want at most one instance of your helper daemon - see <a href="http://blog.verbum.org/2008/04/26/how-to-share-state-with-applications-for-free-unix-based-desktops/">Colin's notes for details and how D-Bus can be used</a> and note that things like <a href="http://developer.gnome.org/gio/unstable/GApplication.html#GApplication.description">GApplication</a> has built-in support for uniqueness. Also, in a system-level daemon, note that you might need to set things like the <a href="http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=include/linux/sched.h;h=781abd13767302cce2ba08db01a211e3178de0d9;hb=61c4f2c81c61f73549928dfd9f3e8f26aa36a8cf#l1374">loginuid</a> (<a href="http://cgit.freedesktop.org/accountsservice/commit/?id=224b7e93a27a1ab5cf2eec2f56bc3adafd02e7af">example of how to do this</a>) so things like <a href="http://people.redhat.com/sgrubb/audit/">auditing</a> work when rendering service for a client (this is related to the Windows concept known as <a href="http://technet.microsoft.com/en-us/library/cc961980.aspx">impersonation</a>).<br />
<br />
As an example, GLib's <a href="http://code.google.com/p/libproxy/">libproxy</a>-based <a href="http://developer.gnome.org/gio/unstable/GProxy.html">GProxy</a> implementation uses a helper daemon because dealing with <a href="http://en.wikipedia.org/wiki/Proxy_server">proxy servers</a> involves a interpreting <a href="http://en.wikipedia.org/wiki/JavaScript">JavaScript</a> (!) and initializing a JS interpreter from every process wanting to make a connection is too much overhead not to mention the pollution caused (<a href="http://git.gnome.org/browse/glib-networking/tree/proxy/libproxy/glibpacrunner.c?id=2.28.7#n128">source</a>, <a href="http://git.gnome.org/browse/glib-networking/tree/proxy/libproxy/org.gtk.GLib.PACRunner.service.in?id=2.28.7">D-Bus activation file</a> - also note <a href="http://git.gnome.org/browse/glib-networking/tree/proxy/gnome/gproxyresolvergnome.c?id=2.28.7#n310">how the helper daemon is activated</a> by simply creating a <a href="http://developer.gnome.org/gio/unstable/GDBusProxy.html#GDBusProxy.description">D-Bus proxy</a>).<br />
<br />
If the helper needs to run with elevated privileges, a framework like <a href="http://hal.freedesktop.org/docs/polkit/polkit.8.html">PolicyKit</a> is convenient to use (for checking whether the process using your library is authorized) since it nicely integrates with the <a href="http://davidz25.blogspot.com/2011/02/gnome-3-authorization.html">desktop shell</a> (and also <a href="http://davidz25.blogspot.com/2011/02/policykit-textual-authentication.html">console/ssh logins</a>). If your library is just using a short-lived helper program, it's even simpler: just use the <a href="http://hal.freedesktop.org/docs/polkit/pkexec.1.html">pkexec(1)</a> command to launch your helper (<a href="http://git.gnome.org/browse/gnome-power-manager/tree/src/gpm-backlight-helper.c?id=GNOME_POWER_MANAGER_3_0_2#n138">example</a>, <a href="http://git.gnome.org/browse/gnome-power-manager/tree/policy/org.gnome.power.policy.in2?id=GNOME_POWER_MANAGER_3_0_2">policy file</a>).<br />
<br />
As an aside (since this write-up is about C libraries, not <a href="http://en.wikipedia.org/wiki/Software_architecture">software architecture</a>), many subsystems in today's Linux desktop are implemented as a system-level daemons (often running <a href="http://en.wikipedia.org/wiki/Superuser">privileged</a>) with the primary API being a D-Bus API (<a href="http://people.freedesktop.org/~david/udisks2-20110628/ref-dbus.html">example</a>) and a C library to access the functionality either not existing at all (applications then use generic D-Bus libraries or tools like <a href="http://developer.gnome.org/gio/unstable/gdbus.html">gdbus(1)</a> or <a href="http://dbus.freedesktop.org/doc/dbus-send.1.html">dbus-send(1)</a>) or mostly <a href="http://developer.gnome.org/gio/unstable/gdbus-codegen.html">generated</a> from the <a href="http://en.wikipedia.org/wiki/Interface_description_language">IDL</a>-like <a href="http://cgit.freedesktop.org/udisks/tree/data/org.freedesktop.UDisks2.xml?h=gdbus-port&id=9363381604e807a3b4d2c57b7940cb785c259dcb">D-Bus XML</a> definition files (<a href="http://people.freedesktop.org/~david/udisks2-20110628/ref-library.html">example</a>). It's useful to contrast this approach to libraries using helpers since one is more or less upside down compared to the other.<br />
<h2>Checklist</h2><ul><li>Identify when a helper program or helper daemon is needed</li>
<li>If possible, use <a href="http://www.freedesktop.org/wiki/Software/dbus">D-Bus</a> (or similar) for activation / uniqueness of helper daemons.</li>
<li>Communicating with a helper via the D-Bus protocol (instead of using a custom binary protocol) adds a layer of safety because message contents are checked.</li>
<li>Using D-Bus through a <a href="http://dbus.freedesktop.org/doc/dbus-specification.html#message-bus">message bus</a> router (instead of peer-to-peer connections) adds yet another layer of safety since the two processes are connected through an intermediate router process (a <a href="http://dbus.freedesktop.org/doc/dbus-daemon.1.html">dbus-daemon(1)</a> instance) which will also validate messages and disconnects processes sending garbage.</li>
<li>Hence, if the helper is privileged (meaning that it must <b>a)</b> treat the unprivileged application/library using it as untrusted and potentially compromised; and <b>b)</b> validate all data to it - see <a href="http://www.dwheeler.com/secure-programs/">Wheeler's Secure Programming notes</a> for details), activating a helper daemon on the D-Bus system bus is often a better idea than using a <a href="http://en.wikipedia.org/wiki/Setuid">setuid root</a> helper program spawned yourself.</li>
<li>If possible, in particular if you are writing code that is used on the Linux desktop, use <a href="http://hal.freedesktop.org/docs/polkit/polkit.8.html">PolicyKit</a> (or similar) in privileged code to check if unprivileged code is authorized to carry out the requested operation.</li>
</ul><a href="" name="testing"></a><br />
<h1>Testing</h1>A sign of maturity is when a library or application comes with a <a href="http://en.wikipedia.org/wiki/Test_suite">test suite</a>; a good test suite is also incredible useful for ensuring mostly bug-free releases and, more importantly, ensuring that the maintainer is comfortable putting releases out without loosing too much sleep or sanity. Discussing specifics of testing is out of the scope for a series on writing C libraries, but it's worth pointing to the <a href="http://developer.gnome.org/glib/unstable/glib-Testing.html#glib-Testing.description">GLib test framework</a>, how it's used (<a href="http://git.gnome.org/browse/glib/tree/glib/tests/gvariant.c?id=2.29.8#n4040">example</a>, <a href="http://git.gnome.org/browse/glib/tree/gio/tests/gdbus-connection.c?id=2.29.8#n120">example</a> and <a href="http://git.gnome.org/browse/glib/tree/glib/tests/bitlock.c?id=2.29.8">example</a>) and how this is used by e.g. the <a href="http://build.gnome.org/">GNOME buildbots</a>.<br />
<br />
One metric for measuring how good a test suite is (or at least how <i>extensive</i> it is), is determining how much of the code it covers - for this, the <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a> tool can be used - see <a href="http://people.redhat.com/berrange/dbus-coverage/">notes on how this is used in D-Bus</a>. Specifically, if the test suite does not cover some edge case, the code paths for handling said edge case will appear as never being executed. Or if the code base handles <a href="http://en.wikipedia.org/wiki/Out_of_memory">OOM</a> but the test suite isn't set up to handle it (for example, <a href="http://cgit.freedesktop.org/dbus/dbus/tree/dbus/dbus-internals.c?id=dbus-1.5.2#n952">by failing each allocation</a>) the code-paths for handling OOM should appear as untested.<br />
<br />
Innovative approaches to testing can often help - for example, <a href="http://www.mozilla.org/">Mozilla</a> employ a technique known as <a href="http://weblogs.mozillazine.org/roc/archives/2008/12/reftests.html">reftests</a> (see also: <a href="http://mail.gnome.org/archives/gtk-devel-list/2011-May/msg00003.html">notes on GTK+ reftests</a>) while the <a href="http://fedoraproject.org/wiki/Dracut">Dracut</a> <a href="http://git.kernel.org/?p=boot/dracut/dracut.git;a=tree;f=test;h=11e80f7337f55aebd0e0c85ebb853c2e85f090e8;hb=829c2585547d86c84cbb3113c73b5dd7fcb7b30d">test suite</a> employs <a href="http://en.wikipedia.org/wiki/Virtual_machine">VMs</a> for both <a href="http://en.wikipedia.org/wiki/SCSI_initiator_and_target">client and server</a> to <a href="http://git.kernel.org/?p=boot/dracut/dracut.git;a=blob;f=test/TEST-30-ISCSI/test.sh;h=32c0d5772cbbbaa91f00478d3ff6383710978923;hb=829c2585547d86c84cbb3113c73b5dd7fcb7b30d">test that</a> booting from <a href="http://en.wikipedia.org/wiki/ISCSI">iSCSI</a> work.<br />
<h2>Checklist</h2><ul><li>Start writing a test suite as early as possible.</li>
<li>Use tools like gcov to ascertain how good the test suite is.</li>
<li>Run the test suite often - ideally integrate it into the build system ('make check'), release procedures, version control etc.</li>
</ul>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com4tag:blogger.com,1999:blog-5268847417417953349.post-9629587019660192862011-06-29T18:12:00.003-04:002011-07-06T12:59:54.931-04:00Writing a C library, part 3<i>This is part three in a <a href="http://davidz25.blogspot.com/2011/07/writing-c-library-intro-conclusion-and.html">series of blog-posts</a> about best practices for writing C libraries. Previous installments: <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html">part one</a>, <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-2.html">part two</a>.</i><br />
<a href="" name="modularity"></a><br />
<h1>Modularity and namespaces</h1>The C programming language does not support the concept of <a href="http://en.wikipedia.org/wiki/Namespace_(computer_science)">namespaces</a> (as used in e.g. C++ or Python) so it is usually emulated simply by using naming conventions. The main reason of namespaces is to avoid <a href="http://en.wikipedia.org/wiki/Naming_collision">naming collisions</a> - consider both libwoot and libkool providing a function called get_all_objects() - which one should be used if a program links to both libraries? Namespacing is an important part of a naming strategy and applies to variables, function names, type names (including structs, unions, enums and typedefs) and macros.<br />
<br />
The standard convention is to use a short identifier, e.g. for <a href="http://projects.gnome.org/NetworkManager/developers/libnm-glib/09/index.html">libnm-glib</a> you will see nm_ and NM being used, for <a href="http://docs.clutter-project.org/docs/clutter/unstable/">Clutter</a> it's clutter and Clutter and for <a href="http://hal.freedesktop.org/docs/polkit/ref-authentication-agent-api.html">libpolkit-agent-1</a>, it's polkit_agent and PolkitAgent. For libraries that don't use <a href="http://en.wikipedia.org/wiki/CamelCase">CamelCase</a> for its types, the same prefix is normally used for functions and types - for example, <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/">libudev</a> the prefix used is simply udev.<br />
<br />
Code that isn't using namespaces properly is not only hard to integrate into other libraries and applications (the chance of symbol collisions is high), there's also a chance that it will collide with future additions to the standard C library or POSIX standards.<br />
<br />
One benefit of using namespaces in C (one that ironically is not present in a language with proper support for namespaces), is that it's a lot easier to pinpoint what the code is doing by just looking at a fragment of the source code - e.g. when you see an item being added to a container, you are usually not in doubt whether the programmer meant to invoke GtkContainer's <a href="http://developer.gnome.org/gtk3/unstable/GtkContainer.html#gtk-container-add">add()</a> method or ClutterContainer's <a href="http://docs.clutter-project.org/docs/clutter/unstable/ClutterContainer.html#clutter-container-add">add()</a> method because of how C namespacing forces the programmer to be explicit, for better or worse.<br />
<br />
In addition to choosing a good naming strategy, note that the visibility of what symbols (typically variables and functions) a library export can be fine-tuned, see <a href="http://gcc.gnu.org/wiki/Visibility">these notes</a> for why this is desirable.<br />
<br />
On the topic of naming, it is usually a good idea to avoid C++ keywords (such as "class") for variable names, at least in header files that you except C++ code to include using e.g. <a href="http://en.wikipedia.org/wiki/Compatibility_of_C_and_C%2B%2B#Linking_C_and_C.2B.2B_code">extern "C"</a>. Additionally, generally avoid names of functions in the C standard library / POSIX for variable names such as "interface" or "index" because these functions can (and on Linux, actually is) be defined as macros.<br />
<h2>Checklist</h2><ul><li>Choose a naming convention - and stick to it.</li>
<li>Do not export symbols that are not public API.</li>
</ul><a href="" name="error-handling"></a><br />
<h1>Error handling</h1>If there's one statement that adequately describes error handling in the C programming language, it's perhaps that it's something that people rarely agree on. Most programmers, however, would agree that errors can be broken down into two categories <b>1) </b>programmer errors; <b>2)</b> run-time errors.<br />
<br />
A <i>programmer error</i> is when the programmer isn't using a function correctly - e.g. passing a non-<a href="http://en.wikipedia.org/wiki/UTF-8">UTF-8</a> string to a function expecting a valid UTF-8 string such as <a href="http://developer.gnome.org/glib/unstable/glib-GVariant.html#g-variant-new-string">g_variant_new_string()</a> (if unsure, validate with <a href="http://developer.gnome.org/glib/unstable/glib-Unicode-Manipulation.html#g-utf8-validate">g_utf8_validate()</a> before calling the function) or passing an invalid D-Bus name to <a href="http://developer.gnome.org/gio/unstable/gio-Owning-Bus-Names.html#g-bus-own-name">g_bus_own_name()</a> (if unsure, validate with <a href="http://developer.gnome.org/gio/unstable/gio-D-Bus-Utilities.html#g-dbus-is-name">g_dbus_is_name()</a> and <a href="http://developer.gnome.org/gio/unstable/gio-D-Bus-Utilities.html#g-dbus-is-unique-name">g_dbus_is_unique_name()</a> before calling).<br />
<br />
Most libraries have undefined behavior in the presence of being used incorrectly - in the GLib case the macros <a href="http://developer.gnome.org/glib/unstable/glib-Warnings-and-Assertions.html#g-return-if-fail">g_return_if_fail()</a> / <a href="http://developer.gnome.org/glib/unstable/glib-Warnings-and-Assertions.html#g-return-val-if-fail">g_return_val_if_fail()</a> are used, see e.g. the <a href="http://git.gnome.org/browse/glib/tree/glib/gvariant.c?id=2.29.8#n1116">checks in g_variant_new_string()</a> and the <a href="http://git.gnome.org/browse/glib/tree/gio/gdbusnameowning.c?id=2.29.8#n596">checks in g_dbus_own_name()</a>. Additionally, for performance, these checks can be disabled by defining the macro G_DISABLE_CHECKS when building either GLib itself or applications using GLib (but usually aren't). Not all parameters may be checked, however, and the check might not cover all cases because checks can be expensive. Combined with the G_DEBUG flag, it's even easy to trap this in debugger by running the program in an environment where <a href="http://developer.gnome.org/glib/unstable/glib-running.html">G_DEBUG=fatal-warnings</a>.<br />
<br />
Having g_return_if_fail()-style checks is usually a trade-off - for example, GLib didn't initially have the UTF-8 check in g_variant_new_string() - it was only added when it became apparent that a considerable amount of users passed non-UTF-8 data which caused errors in unrelated code that was extremely hard to track down - see the <a href="http://git.gnome.org/browse/glib/commit/glib/gvariant.c?id=5e6f762d61db1a5c64bd1d33e5ba112755106581">commit message</a> for details. If this cost is unacceptable, the programmer can easily use the <a href="http://developer.gnome.org/glib/unstable/glib-GVariant.html#g-variant-new-from-data">g_variant_new_from_data()</a> function passing TRUE as the <i>trusted</i> parameter.<br />
<br />
Even with a library doing proper parameter validation (to catch programmer errors early on), if you pass garbage to a function you usually end up with <a href="http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html">undefined behavior</a> and undefined behavior can mean <i>anything</i> including formatting your hard disk or evaporating all booze in a five-mile radius (oh noz). That's why some libraries simply calls <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/abort.3.html">abort()</a> instead of carrying on pretending nothing happened. In general, a C library can <b>never</b> guarantee that it won't blow up no matter what data is passed - for example the user may pass a pointer to invalid data and, boom, <a href="http://en.wikipedia.org/wiki/SIGSEGV">SIGSEGV</a> is raised when the library tries to accesses it. Of course the library <i>could</i> try to recover, <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/longjmp.3.html">longjmp(3)</a> style, but since it's a library it can't mess around with process-wide state like signal handlers. Unfortunately, even smart people sometime <a href="http://lwn.net/Articles/449517/">fail to realize that the caller has a responsibility</a> and instead blames the library instead of its user (for the record, libdbus-1 is fine which is why <a href="http://www.freedesktop.org/wiki/Software/systemd">process 1</a> is able to use it without any problems). In most cases, problems like these are solved by just throwing documentation at the problem.<br />
<br />
To conclude, when it comes to programmer errors, one key take away is that it's a good idea to document exactly what kind of input a function accepts. As the saying goes, <a href="http://cs.nyu.edu/~dodis/quotes.html">"trust is good, control is better"</a>, it is also a good idea to verify that the programmer gets it right by using g_return_if_fail() style checks (and possibly provide API that does no such checks). Also, if your code does any kinds of checks, make sure that the functions used for checking (if non-trivial) are public so e.g. language bindings have a chance to validate input before calling the function (see also: <a href="http://lists.freedesktop.org/archives/dbus/2007-October/008679.html">notes on errors in libdbus</a>).<br />
<br />
A <i>run-time</i> error is e.g. if <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/fopen.3.html">fopen(3)</a> returns NULL (for example the file to be opened does not exist or the calling process is not privileged to open it), <a href="http://developer.gnome.org/gio/unstable/GSocketClient.html#g-socket-client-connect">g_socket_client_connect()</a> returns FALSE (the network might not be up) or <a href="http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-try-malloc">g_try_malloc()</a> returns NULL (might not have enough address space for a 8GiB array). By definition, run-time errors are recoverable although the code you are using might treat some (like <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/malloc.3.html">malloc(3)</a> failing) as irrecoverable because handling some run-time errors (such as <a href="http://en.wikipedia.org/wiki/Out_of_memory">OOM)</a> would complicate the API not only on the function level (possibly by taking an error parameter), but also by requiring <a href="http://en.wikipedia.org/wiki/Atomicity_(database_systems)">transactional semantics</a> (e.g. rollback) on most data types (see also: <a href="http://blog.ometer.com/2008/02/04/out-of-memory-handling-d-bus-experience/">write-up on why handling OOM is hard</a> and a <a href="http://lwn.net/Articles/449481/">good explanation of Linux's overcommit feature</a>).<br />
<br />
For simple libraries just using <a href="http://en.wikipedia.org/wiki/Errno">libc's errno</a> is often simplest approach to handling run-time errors (since it's thread-safe and every C programmer knows it) but note that some functions including <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/asprintf.3.html">asprintf(3)</a> does not set errno to ENOMEM if e.g. failing to allocate memory. If you are basing your code on a library like GLib, use its native error type, e.g. <a href="http://developer.gnome.org/glib/unstable/glib-Error-Reporting.html#glib-Error-Reporting.description">GError</a>, for run-time errors. An interesting approach to handling errors <a href="http://www.cairographics.org/manual/cairo-Error-handling.html">is the one</a> used by the <a href="http://www.cairographics.org/">cairo 2D graphics library</a> where (non-trivial) object instances track the error state (see e.g. <a href="http://www.cairographics.org/manual/cairo-cairo-t.html#cairo-status">cairo_status()</a> and <a href="http://www.cairographics.org/manual/cairo-cairo-device-t.html#cairo-device-status">cairo_device_status()</a>). There are many many other ways to convey run-time errors - as always, the important thing when writing a C library is to be consistent.<br />
<h2>Checklist</h2><ul><li>Document valid and invalid value ranges for parameters (if any) and provide facilities to validate parameters (unless trivial) for programmers and language bindings</li>
<li>Try to validate incoming parameters at public API boundaries</li>
<li>Establish a policy on how to deal with programmer errors (e.g. undefined behavior or abort()).</li>
<li>Establish a policy on how to deal with run-time errors (e.g. use errno or GError)</li>
<li>Ensure the way you handle run-time errors map to common <a href="http://en.wikipedia.org/wiki/Exception_handling">exception handling</a> systems.</li>
</ul><a href="" name="encapsulation-and-oo"></a><br />
<h1>Encapsulation and OO design</h1>While C as programming language does not have built-in support for <a href="http://en.wikipedia.org/wiki/Object-oriented_programming">object-oriented programming</a> lots of C programmers use C that way - in many ways it's almost hard <i>not</i> to. In fact, many C programmers regard the simplicity of C (compared to, say, <a href="http://en.wikipedia.org/wiki/C%2B%2B">C++</a>) as a feature insofar that you are not bound to any one <a href="http://en.wikipedia.org/wiki/Object_model">object model</a> - for example, the kernel uses <a href="http://lwn.net/Articles/444910/">various</a> <a href="http://lwn.net/Articles/446317/">OO techniques</a> and the GLib/GTK+ stack has its own dynamic type system called <a href="http://developer.gnome.org/gobject/unstable/pt01.html">GType</a> on which the <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#gobject-The-Base-Object-Type.description">GObject base class</a> (that many classes are derived from) is built.<br />
<br />
There's of course a price to pay for defining your own object model - it typically involves more typing (identifiers are longer) and, especially for GObject, involves actual function calls to <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#g-object-class-install-property">register properties</a>, <a href="http://developer.gnome.org/gobject/unstable/gobject-Type-Information.html#g-type-class-add-private">add private instance data</a> and so on (<a href="http://git.gnome.org/browse/clutter/tree/clutter/clutter-actor.c?id=1.7.2#n3786">example</a>). On the other hand, such a dynamic type system often offer some level of <a href="http://en.wikipedia.org/wiki/Reflection_(computer_programming)">type introspection</a> so it's possible to easy link the property for <a href="http://developer.gnome.org/gtk3/unstable/GtkToggleButton.html#GtkToggleButton--active">whether a check-button widget is active</a> with <a href="http://developer.gnome.org/gtk3/unstable/GtkEntry.html#GtkEntry--visibility">whether an text-entry widget should use password mode</a> using the <a href="http://developer.gnome.org/gobject/unstable/GBinding.html#g-object-bind-property">g_object_bind_property()</a> function (<a href="http://people.freedesktop.org/~david/palimpsest-entry-checkbutton-binding.png">screenshot</a>). <a href="http://en.wikipedia.org/wiki/Polymorphism_in_object-oriented_programming">Polymorphism</a> in GObject is provided by embedding a <a href="http://en.wikipedia.org/wiki/Vtable">virtual method table</a> in the class struct (<a href="http://git.gnome.org/browse/gnome-online-accounts/tree/src/goabackend/goaprovider.h?id=3.1.0#n57">example</a>) and providing a C functions that uses the function pointer (<a href="http://git.gnome.org/browse/gnome-online-accounts/tree/src/goabackend/goaprovider.c?id=3.1.0#n313">example</a>) - note that derived types can access the class struct to chain up (<a href="http://git.gnome.org/browse/gnome-online-accounts/tree/src/goabackend/goagoogleprovider.c?id=3.1.0#n245">example</a>).<br />
<br />
One important feature of object-oriented design in C is that it usually promotes <a href="http://en.wikipedia.org/wiki/Encapsulation_(object-oriented_programming)">encapsulation</a> and data hiding through the use of <a href="http://en.wikipedia.org/wiki/Opaque_data_type">opaque data types</a> - this is desirable as it allows extending the data type (e.g. adding more properties or methods) without breaking or requiring a recompile existing programs using the library (a future installment will discuss <a href="http://en.wikipedia.org/wiki/Application_programming_interface">API</a> and <a href="http://en.wikipedia.org/wiki/Application_binary_interface">ABI</a> and what it means wrt. API design). In an opaque data type, fields that would usually be in the <a href="http://en.wikipedia.org/wiki/Struct_(C_programming_language)">C struct</a> are hidden from the user and instead are made available via a getter (<a href="http://developer.gnome.org/gio/unstable/GApplication.html#g-application-get-flags">example</a>) and/or setter (<a href="http://developer.gnome.org/gio/unstable/GApplication.html#g-application-set-flags">example</a>) - additionally, if the object model support properties, the member may also be made available as a property (<a href="http://developer.gnome.org/gio/unstable/GApplication.html#GApplication--flags">example</a>) - for example, this is useful for <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#GObject-notify">notifying when the property changes</a>.<br />
<br />
Of course, not every single data structure need to be a full-blown GObject - for example, in some cases data hiding might not be desirable (sometimes it's awkward to use a C getter function) or maybe it's too slow to do from an inner loop (direct struct access is without a doubt faster). Also, for simple data structures it is sometimes desirable to initialize struct instances directly in the code.<br />
<br />
Even when a full-blown object model (like GType and GObject) isn't used, it's never a bad idea to use opaque data structures and getters/setters. As an interesting alternative to this, note that some libraries explicitly <a href="http://git.0pointer.de/?p=libatasmart.git;a=blob;f=atasmart.h;h=aa4e339cb4bfa882c7d1b5608663d000d0166ba0;hb=HEAD#l106">allows extending a C structure without considering it an ABI change</a> - while there's no easy way to enforce this (the user may allocate the structure on the stack), at least the library author can always tell the programmer that he shouldn't have done so (which may or may not be useful).<br />
<h2>Checklist</h2><ul><li>Establish an object model for your library (if applicable).</li>
<li>Hide as many implementation details as is practical without impacting performance</li>
<li>Ensure that you can extend your library and types without breaking API or ABI.</li>
<li>If possible, build on top of an established and well-understood object system (such as the GLib one)</li>
</ul>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com8tag:blogger.com,1999:blog-5268847417417953349.post-78069451387003990402011-06-28T16:15:00.003-04:002011-07-06T12:59:38.510-04:00Writing a C library, part 2<i>This is part two in a <a href="http://davidz25.blogspot.com/2011/07/writing-c-library-intro-conclusion-and.html">series of blog-posts</a> about best practices for writing C libraries. Previous installments: <a href="http://davidz25.blogspot.com/2011/06/writing-c-library-part-1.html">part one</a>.</i><br />
<a href="" name="event-handling"></a><br />
<h1>Event handling and the main loop</h1>Event-driven applications, especially <a href="http://en.wikipedia.org/wiki/Graphical_user_interface">GUI</a> applications, are often built around the idea of a "main loop" that intercepts and dispatches all sorts of events such as key/button presses, incoming <a href="http://en.wikipedia.org/wiki/Inter-process_communication">IPC</a> calls, mouse movements, timers and file/socket I/O and so on. The main loop typically "call back" into the application whenever an event happens.<br />
<br />
For example, the GLib/Gtk+ library stack is built around the <a href="http://developer.gnome.org/glib/unstable/glib-The-Main-Event-Loop.html#glib-The-Main-Event-Loop.description">GMainContext, GMainLoop and GSource</a> types and other library stacks provide similar abstractions. Many kernel and systems-level programmers often look funny at GUI programmers when they utter the word "main loop" - much the same way GUI programmers stare confused at kernel programmers when they say <a href="http://www.win.tue.nl/~aeb/linux/lk/lk-13.html#ss13.2">put() or get()</a> something. The truth is that a main-loop is really a well-known concept with a different name: it's basically an abstraction of OS primitives such as <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/select.2.html">select(2)</a>, <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/poll.2.html">poll(2)</a> or equivalent on e.g. <a href="http://en.wikipedia.org/wiki/Windows_API">Windows</a>.<br />
<br />
It is important to note that a multi-threaded application may run different main loops in different threads to ensure that callbacks happen in the right threads - in GLib this is achieved by using the <a href="http://developer.gnome.org/glib/unstable/glib-The-Main-Event-Loop.html#g-main-context-push-thread-default">g_main_context_push_thread_default()</a> function which records the main loop for the current thread in <a href="http://en.wikipedia.org/wiki/Thread-local_storage">thread-local storage</a>. This variable is in turn read when starting an asynchronous operation (such as <a href="http://developer.gnome.org/gio/unstable/GDBusConnection.html#g-dbus-connection-call">g_dbus_connection_call()</a> or <a href="http://developer.gnome.org/gio/unstable/GInputStream.html#g-input-stream-read-async">g_input_stream_read_async()</a>) to ensure that the passed callback function is invoked in a thread running a main loop for the context set with g_main_context_push_thread_default() earlier.<br />
<br />
Some main loops, for example the GLib one, allows creating recursive main loops and this is used to implement GtkDialog's <a href="http://developer.gnome.org/gtk3/unstable/GtkDialog.html#gtk-dialog-run">run() method</a>. While this indeed appears to block the calling thread, it is important to note that events are <i>still</i> being processed (to e.g. process input events and redraw animations). Specifically, this means that the functions (plural since applies to everything in the call stack) that brought up the dialog might end getting called again (from a callback). Thus, when using functions like gtk_dialog_run() you need to ensure that your functions are either <a href="http://en.wikipedia.org/wiki/Reentrant_(subroutine)">re-entrant</a> or that they are guaranteed to not get called when the dialog is showing (typically achieved by making the dialog <a href="http://en.wikipedia.org/wiki/Modal_dialog">modal</a> so the UI action triggering the display of the dialog can't be accessed). Because of pitfalls like this, you <b>must</b> clearly document if a function is using a recursive main loop.<br />
<br />
Note that main loops are not a GUI-only concept - a lot of <a href="http://en.wikipedia.org/wiki/Daemon_(computing)">daemons</a> (e.g. background process without any GUI) are built around this concept since it nicely integrates events from any source whether they are file descriptor based or synthetic such as timers or logging events. In fact, a considerable part of the system-level software on a modern Linux system is built on top of GLib and uses its main event loop abstraction to dispatch events - most of the time such daemons sit idle in one or more main loops and wait for <a href="http://en.wikipedia.org/wiki/D-Bus">D-Bus</a> <a href="http://developer.gnome.org/gio/unstable/GDBusConnection.html#g-dbus-connection-register-object">messages to arrive</a> (to service a client), a <a href="http://developer.gnome.org/glib/unstable/glib-The-Main-Event-Loop.html#g-timeout-add">timeout to fire</a> (maybe to kick off periodic house-keeping tasks) or <a href="http://developer.gnome.org/glib/unstable/glib-The-Main-Event-Loop.html#g-child-watch-add">a child process to terminate</a> (when using a helper program or process to do work).<br />
<br />
Now that we've explained what a main loop is (or rather, what the <i>idea</i> of a main loop is), let's look at why this matters if you a writing a C library. First of all, if your library doesn't need to deliver events to users, you don't need to worry about main loops. Most libraries, however, are not that simple - for example, <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/">libudev</a> delivers events when devices are plugged or changed, <a href="http://live.gnome.org/NetworkManager">NetworkManager</a> wants to inform of changes in networking and so on.<br />
<br />
If your library is using GLib, it is often suitable to just require that the user runs the GLib main loop (if the application is using another main loop, it can either integrate the GLib main loop (<a href="http://labs.qt.nokia.com/2006/02/24/qt-and-glib/">like Qt does</a>) or run it in a separate thread) and use <a href="http://developer.gnome.org/glib/unstable/glib-The-Main-Event-Loop.html#g-main-context-get-thread-default">g_main_context_get_thread_default()</a> when setting up a callback. This is the way many GObject-based libraries, such as <a href="http://hal.freedesktop.org/docs/polkit/">libpolkit-gobject-1</a>, <a href="http://projects.gnome.org/NetworkManager/developers/libnm-glib/09/index.html">libnm-glib</a> or <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/gudev/">libgudev</a> work - for example, callbacks connected to the <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/gudev/GUdevClient.html#GUdevClient-uevent">GUdevClient::uevent</a> signal will be called in what was the <a href="http://developer.gnome.org/glib/unstable/glib-The-Main-Event-Loop.html#g-main-context-push-thread-default">thread-default main loop</a> when the object was constructed. For a shared resource, such as a <a href="http://developer.gnome.org/gio/unstable/GDBusConnection.html#g-bus-get-sync">message bus connection</a>, a good policy is that callbacks happen in what was the the thread-default main loop when the method was called (see e.g. <a href="http://developer.gnome.org/gio/unstable/GDBusConnection.html#g-dbus-connection-signal-subscribe">g_dbus_connection_signal_subscribe()</a> where this is the case) since applications or libraries have no absolute control of when the shared object was created. In any case, functions dealing with callbacks <b>must</b> always document in what context the callback happens in.<br />
<br />
On the other hand, if your library is not using GLib, a good way to provide notification is simply to <b>a)</b> provide a file descriptor that e.g. turns readable when there are events to process; and <b>b)</b> provide a function that processes events (and possibly invoke callback functions registered by the user). A good example of this is libudev's <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/libudev-udev-monitor.html#udev-monitor-get-fd">udev_monitor_get_fd()</a> and <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/libudev-udev-monitor.html#udev-monitor-receive-device">udev_monitor_receive_device()</a> functions. This way the application (or the library using your library) can easily control what thread the event is handled in. As an example of how libudev is integrated into the GLib main loop, see <a href="http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=extras/gudev/gudevclient.c;h=97b951adcd421e559c4a2d7b3b822eb95dd01f1d;hb=HEAD#l228">here</a> and <a href="http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;f=extras/gudev/gudevclient.c;h=97b951adcd421e559c4a2d7b3b822eb95dd01f1d;hb=HEAD#l88">here</a>. In the libudev case, the returned file descriptor is the underlying <a href="http://en.wikipedia.org/wiki/Netlink">netlink socket</a> used to receive the event from <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev/udevd.html">udevd</a> (via the kernel); in the case that there are no natural file descriptor (could be the event is only happening in response to a certain entry in, say, a log file), your library could use <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/pipe.2.html">pipe(2)</a> (or <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/eventfd.2.html">eventfd(2)</a> if on Linux) and use a private worker thread to signal the other end.<br />
<br />
If your library provide callback functions, make sure they take <i>user_data</i> arguments so the user can easily associate callbacks with other objects and data. If the scope of your callback is undefined (e.g. may fire more than once or if there is no way to disconnect the callback), also provide a way to free the <i>user_data</i> pointer when it is no longer needed - otherwise the application will leak data that needs to be freed later. See <a href="http://developer.gnome.org/gio/unstable/gio-Watching-Bus-Names.html#g-bus-watch-name">g_bus_watch_name()</a> and for an example.<br />
<h2>Checklist</h2><ul><li>Provide APIs for main loop integration</li>
<li>Make sure callback functions take <i>user_data</i> arguments (possibly with free an accompanying free function)</li>
</ul><a href="" name="io"></a><br />
<h1>Synchronous and Asynchronous I/O</h1>It is important for users of a library to know if calling a function involves doing <a href="http://en.wikipedia.org/wiki/Asynchronous_I/O">synchronous I/O</a> (also called blocking I/O). For example, an application with an user interface need to be responsive to user input and may even need to update the user interface <a href="http://en.wikipedia.org/wiki/Vertical_blanking_interval">every frame</a> for smooth animations (e.g. 60 times a second). To avoid unresponsive applications and jerky animations, its UI thread must <b>never</b> call any functions that does any synchronous I/O.<br />
<br />
Note that even loading a file from local disk may block for a very long amount of time - sometimes tens of seconds. For example the file may not be in the page cache and the hard disk with the file system may be powered down - or the file may be in the users home directory which could be on a network filesystem such as <a href="http://en.wikipedia.org/wiki/Network_File_System_(protocol)">NFS</a>. Other examples of blocking IO includes local IPC such as <a href="http://en.wikipedia.org/wiki/D-Bus">D-Bus</a> or <a href="http://en.wikipedia.org/wiki/Unix_domain_socket">Unix domain sockets</a>.<br />
<br />
If an operation is known to take a long time to complete (synchronous or otherwise), it is often nice if it is possible to easily cancel the it (perhaps from another thread). For example, see the <a href="http://developer.gnome.org/gio/unstable/GCancellable.html#GCancellable.description">GCancellable</a> type in the GLib stack. Another nicety (although easily implemented via the GCancellable type) is a way to set a timeout for potentially long-running operations - see e.g. <a href="http://developer.gnome.org/gio/unstable/GDBusConnection.html#g-dbus-connection-send-message-with-reply">g_dbus_connection_send_message_with_reply()</a> and <a href="http://developer.gnome.org/gio/unstable/GDBusProxy.html#g-dbus-proxy-call">g_dbus_proxy_call()</a> and note how the latter has an object-wide timeout so the timeout only has to be set once.<br />
<br />
Some libraries provide both synchronous and asynchronous versions of a function where the former blocks the calling thread and the latter doesn't. Typically asynchronous I/O is implemented using worker threads (where the worker thread is doing synchronous I/O) but it could also involve communicating with another process via IPC (e.g. D-Bus) or even TCP/IP. For example, in the <a href="http://developer.gnome.org/gio/unstable/">libgio-2.0</a> case asynchronous file I/O is implemented via synchronous IO (e.g. basically <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/read.2.html">read(2)</a> and <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/write.2.html">write(2)</a> calls) in worker threads (using a <a href="http://developer.gnome.org/glib/unstable/glib-Thread-Pools.html">GThreadPool</a>) simply because the Linux kernel does not (yet?) provide an adequate way to do asynchronous I/O suitable for libraries (see also: <a href="http://davmac.org/davpage/linux/async-io.html">colorful notes about Asynchronous I/O</a>). On the upside, this is mostly an implementation detail and the libgio-2.0 implementation can migrate to a non-threaded approach should such a mechanism be made available in the future.<br />
<br />
Asynchronous I/O typically involves callbacks (or at least some kind of event notification) and thus involves a main loop. If a library provides functions for this, it should clearly state what thread the callback will happen in, and whether it requires the application to run a (specific kind of) main loop - see the previous section about main loops for details.<br />
<br />
If a library is thread-safe, it is often easier for the application itself to just use the synchronous version of a function in a worker-thread - if using GLib, <a href="http://developer.gnome.org/gio/unstable/gio-GIOScheduler.html#g-io-scheduler-push-job">g_io_scheduler_push_job()</a> is the right way to do that.<br />
<br />
In some cases synchronous I/O is implemented by using a recursive main loop (typically by using the asynchronous form of the function) - this should be avoided as it typically causes all kinds of problems because of reentrancy and events being processed while waiting for the supposedly synchronous operation to complete. As always, clearly document what your code is doing.<br />
<br />
Some libraries, such as those in the GLib stack, use a <a href="http://developer.gnome.org/gio/unstable/GSimpleAsyncResult.html#GSimpleAsyncResult.description">consistent pattern</a> for asynchronous I/O for all of its functions involving the <a href="http://developer.gnome.org/gio/unstable/GAsyncResult.html">GAsyncResult</a> / <a href="http://developer.gnome.org/gio/unstable/GSimpleAsyncResult.html">GSimpleAsyncResult</a>, <a href="http://developer.gnome.org/gio/unstable/GAsyncResult.html#GAsyncReadyCallback">GAsyncReadyCallback</a> and <a href="http://developer.gnome.org/gio/unstable/GCancellable.html">GCancellable</a> types - this makes it a lot easier both for programmers and for higher-level language bindings especially since important things like life-cycles are part of this model (for example, you are guaranteed that the callback will always happen, even on cancellation, timeout or error).<br />
<h2>Checklist</h2><ul><li>Clearly document if a function does any synchronous I/O</li>
<li>Ideally suffix synchronous functions it with _sync() so it's easy to inspect large code-trees using e.g. <a href="http://www.gnu.org/software/grep/manual/grep.html">grep(1)</a> </li>
<li>Consider if an operation needs to be available in synchronous or asychronous form or both.</li>
<li>Point to both synchronous and asynchronous functions in your API documentation.</li>
<li>If possible, use an established model (such as the <a href="http://developer.gnome.org/gio/unstable/GSimpleAsyncResult.html#GSimpleAsyncResult.description">GIO model</a>) for I/O instead of rolling your own</li>
</ul>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com1tag:blogger.com,1999:blog-5268847417417953349.post-56497842879478142622011-06-27T15:49:00.008-04:002011-07-06T12:59:19.542-04:00Writing a C library, part 1<i>This is part one in a <a href="http://davidz25.blogspot.com/2011/07/writing-c-library-intro-conclusion-and.html">series of blog-posts</a> about best practices for writing C libraries.</i><br />
<a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=5649784287947814262" name="base-libraries"></a><br />
<h1>Base libraries</h1>Since <a href="http://en.wikipedia.org/wiki/C_library">libc</a> is a fairly low-level set of libraries, there exists higher-level libraries to make C programming a more pleasant experience including libraries in the <a href="http://en.wikipedia.org/wiki/GLib">GLib</a> and and <a href="http://en.wikipedia.org/wiki/GTK%2B">GTK+</a> stack. Even while the following is going to be somewhat GLib- and GTK+-centric, these notes are written to be useful for any C code whether it's based on libc, GLib or other libraries such as <a href="http://en.wikipedia.org/wiki/NSPR">NSPR</a>, <a href="http://en.wikipedia.org/wiki/Apache_Portable_Runtime">APR</a> or <a href="http://talloc.samba.org/talloc/doc/html/index.html">some of the Samba libraries</a>. <br />
<br />
Most programmers would agree that it's usually a bad idea to implement basic data-types such as string handling, memory allocation, lists, arrays, hash-tables or queues yourself just because you <i>can</i> - it only makes code harder to read and harder to maintain by others. This is where C libraries such as GLib and GTK+ come into play - these libraries provides much of this out of the box. Plus, when you end up needing non-trivial utility functions (and chances are you will) for, say, <a href="http://developer.gnome.org/glib/unstable/glib-Unicode-Manipulation.html">Unicode manipulation</a>, <a href="http://developer.gnome.org/pango/stable/pango-Scripts-and-Languages.html">rendering complex scripts</a>, <a href="http://developer.gnome.org/gio/unstable/GDBusConnection.html">D-Bus support</a> or <a href="http://developer.gnome.org/glib/unstable/glib-Data-Checksums.html#glib-Data-Checksums.description">calculating checksums</a>, ask yourself (or worse: wait until your manager or peers ask you) if the decision to avoid a well-tested and well-maintained library was a good decision.<br />
<br />
In particular, for things like <a href="http://en.wikipedia.org/wiki/Cryptography">cryptography</a>, it is usually a <a href="http://stackoverflow.com/questions/1914257/when-can-you-trust-yourself-to-implement-cryptography-based-solutions">bad idea</a> to implement it yourself (however <a href="http://diovo.com/2009/02/wrote-your-own-encryption-algorithm-duh/">inventing your own algorithm</a> is worse); instead, it is better to use an existing well-tested library such as <a href="http://en.wikipedia.org/wiki/Network_Security_Services">NSS</a> (and even if you do, <a href="http://www.codinghorror.com/blog/2009/05/why-isnt-my-encryption-encrypting.html">be careful of using the library correctly</a>). Specifically, said library may even be <a href="http://en.wikipedia.org/wiki/FIPS_140">FIPS-140</a> certified which is a requirement if you want to do business with the <a href="http://en.wikipedia.org/wiki/US_federal_government">US government</a>.<br />
<br />
Similarly, while it’s more efficient to use e.g. <a href="http://www.kernel.org/doc/man-pages/online/pages/man4/epoll.4.html">epoll </a>than <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/poll.2.html">poll</a> for event notification, maybe it doesn't matter if your application or library is only handling on the order of ten file descriptors. On the other hand, if you know that you are going to handle thousands of file descriptors, you can still use e.g. GLib for the bulk of your library or application - just use epoll from dedicated threads. Ditto, if you need O(1) removal from a list, maybe don’t use a <a href="http://developer.gnome.org/glib/unstable/glib-Doubly-Linked-Lists.html">GList</a> - use an <a href="http://git.ozlabs.org/?p=ccan;a=tree;f=ccan/list">embedded list </a>instead.<br />
<br />
Above all, no matter what libraries or code you end up using, make sure you have at least a rudimentary understanding of the relevant data-types, concepts and implementation details. For example, with GLib it is very easy to use high-level constructs such as <a href="http://developer.gnome.org/glib/unstable/glib-Hash-Tables.html">GHashTable</a>, <a href="http://developer.gnome.org/glib/unstable/glib-The-Main-Event-Loop.html#g-timeout-add">g_timeout_add()</a> or <a href="http://developer.gnome.org/glib/unstable/glib-File-Utilities.html#g-file-set-contents">g_file_set_contents()</a> without knowing how things are implemented or what a file descriptor really is. For example, when saving data, you want to do so atomically (to avoid data-loss) and just knowing that g_file_set_contents() Does The Right Thing(tm) is often enough (often just reading the API docs will tell you what you need to know). Additionally make sure you understand both the <a href="http://en.wikipedia.org/wiki/Algorithmic_complexity">algorithmic complexity</a> of the data-types you end up using and how they <a href="http://www.futurechips.org/tips-for-power-coders/modern-programmers-learn-hardware.html">work on modern hardware</a>.<br />
<br />
Finally, try not to get caught up in religious discussions about “bloated” libraries with <a href="http://xkcd.com/386/">random people on the Internet</a> - it’s usually not a good a use of time and resources.<br />
<h2>Checklist</h2><ul><li>Don’t reinvent basic data-types (unless performance is a concern).</li>
<li>Don’t avoid standard libraries just because they are portable.</li>
<li>Be wary of using multiple libraries with overlapping functionality.</li>
<li>To the extent where it’s possible, keep library usage as a private implementation detail.</li>
<li>Use the right tool for the right job - don’t waste time on religious discussions.</li>
</ul><a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=5649784287947814262" name="library-init"></a><br />
<h1>Library initialization and shutdown</h1>Some libraries requires that an function, typically called foo_init(), is called before other functions in the library is called - this function typically initializes global variables and data structures used by the library. Additionally, libraries may also offer a shutdown function, typically called foo_shutdown() (forms such as foo_cleanup(), foo_fini(), foo_exit() and the grammatically dubious foo_deinit() have also been observed in the wild), to release all resources used by the library. The main reason for having a shutdown() function is to play nicer with <a href="http://en.wikipedia.org/wiki/Valgrind">Valgrind</a> (for finding memory leaks) or to release all resources when using <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/dlsym.3.html">dlopen() and friends</a>.<br />
<br />
In general, library initialization and shutdown routines should be avoided since they might cause interference between two unrelated libraries in the dependency chain of an application; e.g. if you don’t call them from where they are used, you are possible forcing the application to call a init() function in main(), just because some library deep down in the dependency chain is using the library without initializing it.<br />
<br />
However, without a library initialization routine, every function in the library would have to call the (internal) initialization routine which is not always practical and may also be a performance concern. In reality, the check only has to be done in a couple of functions since most functions in a library depends on an object or struct obtained from e.g. other functions in the library. So in reality, the check only has to be done in _new() functions and functions not operating on an object.<br />
<br />
For example, every program using the GLib type system <a href="https://bugzilla.gnome.org/show_bug.cgi?id=628041">has to call g_type_init()</a> and this includes libraries based on libgobject-2.0 such as <a href="http://hal.freedesktop.org/docs/polkit/">libpolkit-gobject-1</a> - e.g. if you don’t call <a href="http://developer.gnome.org/gobject/unstable/gobject-Type-Information.html#g-type-init">g_type_init()</a> prior to calling <a href="http://hal.freedesktop.org/docs/polkit/PolkitAuthority.html#polkit-authority-get-sync">polkit_authority_get_sync()</a> then your program will probably segfault. Naturally this is something most people new to the GLib stack gets wrong and you can’t really blame them - if anything, g_type_init() is a great poster-child of why init() functions should be avoided if possible.<br />
<br />
One reason for library initialization routine has to do with library configuration, either app-specific configuration (e.g. the application using the library might want to force a specific behavior) or end-user specific (by manipulating argc and argv) - for example, see <a href="http://developer.gnome.org/gtk3/unstable/gtk3-General.html#gtk-init">gtk_init()</a>. The best solution to this problem is of course to avoid configuration, but in the cases where it’s not possible it is often better to use e.g. environment variables to control behavior - see e.g. the <a href="http://developer.gnome.org/gtk3/unstable/gtk-running.html">environment variables supported by libgtk-3.0</a> and the <a href="http://developer.gnome.org/gio/unstable/ch03.html">environment variables supported by libgio-2.0</a> for examples.<br />
<br />
If your library does have an initialization routine, do make sure that it is idempotent and thread-safe, e.g. that it can be called multiple times and from multiple threads at the same time. If your library also has a shutdown routine, make sure that some kind of “initialization count” is used so the library is only shutdown once all users of it have called its shutdown() routine. Also, if possible, ensure that your library init/shutdown routines calls the init/shutdown routines for libraries that it depends on.<br />
<br />
Often, a library's init() and shutdown() functions can be removed by introducing a <i>context object</i> - this also fixes the problem of global state (which is undesirable and often break multiple library users in the same process), locking (which can then be per context instance) and callbacks / notification (which can call back / post events to separate threads). For example, see <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/">libudev's</a> <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/libudev-udev-monitor.html">struct udev_monitor</a>.<br />
<h2>Checklist</h2><ul><li>Avoid init() / shutdown() routines - if you can’t avoid them, do make sure they are idempotent, thread-safe and reference-counted.</li>
<li>Use environment variables for library initialization parameters, not argc and argv.</li>
<li>You can easily have two unrelated library users in the same process - often without the main application knowing about the library at all. Make sure your library can handle that.</li>
<li>Avoid <a href="http://stackoverflow.com/questions/6082232/c-doing-something-when-the-program-exits">unsafe API like atexit(3)</a> and, if portability is a concern, unportable constructs like library constructors and destructors (e.g. gcc’s __attribute__ ((constructor)) and __attribute__ ((destructor))).</li>
</ul><a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=5649784287947814262" name="memory-mgmt"></a><br />
<h1>Memory management</h1>It is good practice to provide a matching free() function or each kind of allocated object that your API returns. If your library uses reference counting, it is often more appropriate to use the suffix _unref instead of _free. An example of this in the GLib/GTK+ stack the functions used are <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#g-object-new">g_object_new()</a>, <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#g-object-ref">g_object_ref()</a> and <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#g-object-unref">g_object_unref()</a> that operate on instances of the <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html">GObject type</a> (including derived types). Similarly, for the <a href="http://developer.gnome.org/gtk3/unstable/GtkTextIter.html">GtkTextIter</a> type, the relevant functions are <a href="http://developer.gnome.org/gtk3/unstable/GtkTextIter.html#gtk-text-iter-copy">gtk_text_iter_copy()</a> and <a href="http://developer.gnome.org/gtk3/unstable/GtkTextIter.html#gtk-text-iter-free">gtk_text_iter_free()</a>. Also, note that some objects may be <a href="http://en.wikipedia.org/wiki/Stack_allocation">stack-allocated</a> (such as GtkTextIter) while others (such as GObject) can only be <a href="http://en.wikipedia.org/wiki/Dynamic_memory_allocation">heap-allocated</a>.<br />
<br />
Note that some object-oriented libraries with the concept of derived types may require the app to use the unref() method from a base type - for example, an instance of a <a href="http://developer.gnome.org/gtk3/unstable/GtkButton.html">GtkButton</a> must be released with g_object_unref() because GtkButton is also a GObject. Additionally, some libraries have the concept of floating references (see e.g. <a href="http://developer.gnome.org/gobject/unstable/gobject-The-Base-Object-Type.html#GInitiallyUnowned">GInitiallyUnowned</a>, <a href="http://developer.gnome.org/gtk3/unstable/GtkWidget.html#GtkWidget.object-hierarchy">GtkWidget</a> and <a href="http://developer.gnome.org/glib/unstable/glib-GVariant.html">GVariant</a>) - this can make it more more convenient to use the type system from C since it e.g. allows using the g_variant_new() constructor in place of a parameter like in the example code for <a href="http://developer.gnome.org/gio/unstable/GDBusProxy.html#g-dbus-proxy-call-sync">g_dbus_proxy_call_sync()</a> without leaking any references.<br />
<br />
Unless it’s self-evident, all functions should have documentation explaining how parameters are managed. It is often a good idea to try to force some kind of consistency on the API. For example, in the GLib stack the general rule is that the caller owns parameters passed to a function (so the function need to take a reference or make a copy if the parameter is used after the function returns) and that the callee owns the returned parameters (so the caller needs to make a copy or increase the reference count) unless the function can be called from multiple threads (in which case the caller needs to free the returned object).<br />
<br />
Note that thread-safety often dictates what the API looks like - for example, for a thread-safe object pool, the lookup() function (returning an object) must return a reference (that the caller must unref()) because the returned object could be removed from another thread just after lookup() returns - one such example is <a href="http://developer.gnome.org/gio/unstable/GDBusObjectManager.html#g-dbus-object-manager-get-object">g_dbus_object_manager_get_object()</a>.<br />
<br />
If you implement reference counting for an object or struct, make sure it is using <a href="http://en.wikipedia.org/wiki/Atomic_operation">atomic operations</a> or otherwise protect the reference count from being modified simultaneously by multiple threads.<br />
<br />
If a function is returning a pointer to memory that the caller isn’t supposed to free or unref, it is often necessary to document for how long the pointer is valid - for example the documentation for the <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/getenv.3.html">getenv()</a> C library function says “The string pointed to by the return value of getenv() may be statically allocated, and can be modified by a subsequent call to getenv(), putenv(3), setenv(3), or unsetenv(3).”. This is useful information because it shows that care should be taken if the result from getenv() is used by multiple threads; also this kind of API can never work in a multi-threaded application and the only reason it works is that applications or libraries normally don’t modify the environment.<br />
<br />
It is often advantageous for an application to not worry about out-of-memory conditions and instead just call <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/abort.3.html">abort()</a> if the underlying allocator signals an out-of-memory condition. This holds true for most libraries as well since it allows a simpler and better API and huge code-footprint reductions. If you do decide to worry about OOM in your library, do make sure that you test all code-paths or your effort will very likely have been in vain. On the other hand, if you know your library is going to be used in e.g. process 1 (the init process) or other critical processes, then not handling OOM is not an option.<br />
<h2>Checklist</h2><ul><li>Provide a free() or unref() function for each type your library introduces.</li>
<li>Ensure that memory handling consistent across your library.</li>
<li>Note that multi-threading may impose certain kinds of API.</li>
<li>Make sure the documentation is clear on how memory is managed.</li>
<li>Abort on OOM unless there are very good reasons for handling OOM.</li>
</ul><h1>Multiple Threads and Processes</h1><a href="http://www.blogger.com/post-edit.g?blogID=5268847417417953349&postID=5649784287947814262" name="multiple-threads-and-processes"></a><br />
A library should clearly document if and how it can be used from multiple threads. There are often multiple levels of thread-safety involved - if the library has a concept of objects and a pool of objects (as most libraries do), the enumeration and management of the pool might be thread safe while applications are supposed to provide their own locking when operating on a single object from multiple threads, concurrently.<br />
<br />
If you are providing a function performing synchronous I/O, it is often a good idea to make it thread-safe so an application can safely use it from a helper thread<br />
<br />
If your library is using threads internally, be wary of manipulating process-wide state, such as the current directory, locale, etc. Doing so from your private worker thread will have unexpected consequences for the application using your library.<br />
<br />
A library should always use thread-safe functions (e.g. <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/getpwnam.3.html">getpwnam_r()</a> rather than getpwnam()) and avoid libraries and code that is not thread-safe. If you can’t do this, clearly state that your library isn’t thread-safe so applications can use it from a dedicated helper process instead if they need thread-safety.<br />
<br />
It is also important to document if your library is using threads internally, e.g. for a pool of worker threads. Even though you think of the thread as a private implementation detail, its existence can affect users of your library; e.g. <a href="http://www.kernel.org/doc/man-pages/online/pages/man7/signal.7.html">Unix signals</a> might need to be handled differently in the the presence of threads, and there are extra complications when forking a threaded application.<br />
<br />
If your library has interfaces involving resources that can be inherited over <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/fork.2.html">fork()</a>, such as file descriptors, locks, memory obtained from <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/mmap.2.html">mmap()</a>, etc, you should try to establish a clear policy for how an application can use your library before/after a fork. Often, the simplest policy is the best: start using nontrivial libraries only after the fork, or offer a way to reinitialize the library in the forked process. For file descriptors, using <a href="http://www.kernel.org/doc/man-pages/online/pages/man2/fcntl.2.html">FD_CLOEXEC</a> is a good idea. In reality most libraries have undefined behavior after the fork() call, so the only safe thing to do is to call the <a href="http://www.kernel.org/doc/man-pages/online/pages/man3/exec.3.html">exec()</a> function.<br />
<h2>Checklist</h2><ul><li>Document if and how the library can be used from multiple threads.</li>
<li>Document what steps need to be taken after fork() or if the library is now unusable.</li>
<li>Document if the library is creating private worker threads.</li>
</ul>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com8tag:blogger.com,1999:blog-5268847417417953349.post-12319100823330531842011-04-27T11:52:00.001-04:002011-04-27T11:54:48.974-04:00GNOME Online AccountsThe past couple of weeks I've been looking into how to make <a href="http://www.gnome.org">GNOME</a> work with online accounts. I sent <a href="http://mail.gnome.org/archives/desktop-devel-list/2011-April/msg00107.html">some notes</a> about to d-d-l about this last week. I'm now at a point where I have working code. And with working code, I of course mean a working screencast (<a href="http://youtu.be/Y1ZTdRB5ZII?hd=1">Youtube Video</a>):
<iframe title="YouTube video player" width="960" height="750" src="http://www.youtube.com/embed/Y1ZTdRB5ZII?hd=1" frameborder="0" allowfullscreen></iframe>
OK, I mentioned working code and I wasn't kidding — it's stashed in <a href="http://cgit.freedesktop.org/~david/goa/">this git repo</a> for now. I expect to move it to <a href="http://git.gnome.org/browse">git.gnome.org</a> assuming we decide to use this in GNOME. More importantly, I've uploaded <a href="http://people.freedesktop.org/~david/goa-20110427/">the docs</a> here. Oh, and to build this, you will need GLib from the <a href="http://git.gnome.org/browse/glib/log/?h=gdbus-codegen">gdbus-codegen branch</a> — fear not, this will be <a href="https://bugzilla.gnome.org/show_bug.cgi?id=647577">merged to mainline</a> soon!davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com9tag:blogger.com,1999:blog-5268847417417953349.post-75926130523955690852011-04-06T17:46:00.000-04:002011-04-06T17:46:25.733-04:00GNOME 3.0 is outYay, yay, yay, <a href="http://www.gnome.org/">GNOME</a> 3.0 is out!
<p>
I'm especially proud of being part of this release, as I was involved in coding up two very user-visible pieces - the <a href="http://davidz25.blogspot.com/2011/02/gnome-3-authorization.html">Authorization Dialog</a> and the <a href="http://davidz25.blogspot.com/2011/01/gnome-3-calendar.html">Calendar Drop-down</a>. Not only is the shell super-duper hot looking (making GNOME 2.x vintage desktops look like Windows 95 more than ever), it is a true pleasure to use. One that grows on you. One that makes you never go back.
<p>
The best part about this release is probably how re-energized everyone working on GNOME 3 is. It is important to realize that 3.0 is only the beginning and that we have tons of stuff lined up for <a href="http://mail.gnome.org/archives/gnome-shell-list/2011-April/msg00073.html">3.0.x</a>, <a href="http://mail.gnome.org/archives/desktop-devel-list/2011-March/msg00109.html">3.2</a> and beyond. I'm especially psyched about the prospect of making the desktop shell <a href="http://mail.gnome.org/archives/desktop-devel-list/2011-March/msg00117.html">integrate nicely</a> with the many <a href="http://en.wikipedia.org/wiki/Web_application">Web Applications</a> that are in use today. Myself? I'm already busy working on improvements for 3.2 - for the authorization dialog we want to present better strings to the user (e.g. including name of app in the dialog and what object the authorization is about) and for the calendar we want <a href="https://live.gnome.org/GnomeShell/Design/Whiteboards/DateNTime">Locations (and possibly weather too)</a> in the drop-down. And ideally an easier way to make the shell show e.g. your Google Calendars if that is your calendar application of choice.
<p>
<center><a title="Help promote GNOME 3!" href="https://live.gnome.org/ThreePointZero/Promote"><img border="0" alt="I am GNOME" src="http://www.gnome.org/wp-content/uploads/2011/04/iamgnome.png"/></a></center>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com1tag:blogger.com,1999:blog-5268847417417953349.post-7520412469238489162011-02-22T12:52:00.000-05:002011-02-22T12:52:12.301-05:00PolicyKit textual authenticationRelated to yesterday's <a href="http://davidz25.blogspot.com/2011/02/gnome-3-authorization.html">bling post</a>, I wanted to mention that PolicyKit also has a textual authentication agent (<a href="http://www.youtube.com/watch?v=7riJd5DpwUA">video</a>):
<p>
<iframe title="YouTube video player" width="480" height="375" src="http://www.youtube.com/embed/7riJd5DpwUA?rel=0&hd=1" frameborder="0" allowfullscreen></iframe>
<p>
This is useful in <a href="http://hal.freedesktop.org/docs/polkit/polkit-apps.html">PolicyKit applications</a> running in a non-graphical environment. It's simple to use: simply use <a href="http://hal.freedesktop.org/docs/polkit/PolkitAgentTextListener.html">PolkitAgentTextListener</a> if the <a href="http://hal.freedesktop.org/docs/polkit/PolkitAuthorizationResult.html">response from PolicyKit</a> indicates that authorization can be obtained <a href="http://hal.freedesktop.org/docs/polkit/PolkitAuthorizationResult.html#polkit-authorization-result-get-is-challenge">if more information is provided</a>. <a href="http://cgit.freedesktop.org/PolicyKit/diff/src/programs/pkexec.c?id=42177383585e1e01cd6150f891176afcd4538a82">Here</a> are the changes that was needed for <a href="http://hal.freedesktop.org/docs/polkit/pkexec.1.html">pkexec(1)</a> command. An obvious candidate for this would be the <a href="http://linux.die.net/man/1/pkcon">pkcon(1)</a> command.davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com1tag:blogger.com,1999:blog-5268847417417953349.post-48785415986163775372011-02-21T15:20:00.001-05:002011-02-21T15:32:17.604-05:00GNOME 3 AuthorizationThe past couple of days I've been working on implementing the GNOME 3 <a href="https://live.gnome.org/GnomeShell/Design/Whiteboards/AuthorizationDialog">Authorization Dialog</a>. The implementation uses <a href="http://hal.freedesktop.org/docs/polkit/polkit.8.html">PolicyKit</a> to achieve this and is conceptually very simple: the Shell process itself just implements the <a href="http://hal.freedesktop.org/docs/polkit/eggdbus-interface-org.freedesktop.PolicyKit1.AuthenticationAgent.html">org.freedesktop.PolicyKit1.AuthenticationAgent</a> interface to handle authentication requests from the system-wide authority (<a href="http://www.youtube.com/watch?v=AaCxqhHpqFk">video</a>).
<p>
<iframe title="YouTube video player" width="480" height="375" src="http://www.youtube.com/embed/AaCxqhHpqFk?rel=0&hd=1" frameborder="0" allowfullscreen></iframe>
<p>
For authentication, the Shell is using the <a href="http://hal.freedesktop.org/docs/polkit/PolkitAgentSession.html">existing code</a> that in turn uses <a href="http://en.wikipedia.org/wiki/Pluggable_Authentication_Modules">PAM</a>. This means that the Authorization Dialog will work out-of-the-box with e.g. <a href="http://reactivated.net/fprint/wiki/Main_Page">fingerprint readers</a>, provided, of course, that the PAM stack has been set up correctly (<a href="http://www.youtube.com/watch?v=1IvDf0YSQVk">video</a>).
<p>
<iframe title="YouTube video player" width="480" height="375" src="http://www.youtube.com/embed/1IvDf0YSQVk?rel=0&hd=1" frameborder="0" allowfullscreen></iframe>
<p>
Additionally, the code has been tested with <a href="http://linux.die.net/man/8/pam_rps">pam_rps</a> - a PAM module whereby you authenticate by proving that you are smart enough to win in <a href="http://en.wikipedia.org/wiki/Rock_paper_scissors">Rock-paper-scissors</a>. The idea is that if your application works with pam_rps, then there's a good chance that it will work with any PAM module (<a href="http://www.youtube.com/watch?v=pBX5mrReHYQ">video</a>).
<p>
<iframe title="YouTube video player" width="480" height="375" src="http://www.youtube.com/embed/pBX5mrReHYQ?rel=0&hd=1" frameborder="0" allowfullscreen></iframe>
<p>
One major difference - <a href="https://bugzilla.gnome.org/show_bug.cgi?id=596260#c4">and something I've been wanting to do for a while</a> - is that the dialogs are now system-modal. In addition to fixing a lot of focus problems, this also makes the dialogs <i>look</i> more <a href="http://en.wiktionary.org/wiki/trustworthy">trustworthy</a> since they are part of the actual shell. They're also way prettier.
<p>
(Bonus chatter: Of course, there's an emphasis on <i>look</i> in the above paragraph because any random app can still go full-screen to fool the user to give them their password. To do this in a more secure way, we need to use the <a href="http://en.wikipedia.org/wiki/Secure_attention_key">Secure Attention Key</a> and run the dialog in another security context etc. It's an open question whether it's even worth it to do this. But I digress.)
<p>
You can find the current code drop in <a href="https://bugzilla.gnome.org/show_bug.cgi?id=642886">this bug</a>.davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com7tag:blogger.com,1999:blog-5268847417417953349.post-80337591713514512012011-01-31T13:01:00.000-05:002011-01-31T13:01:49.428-05:00GNOME 3 CalendarFrom <a href="https://live.gnome.org/GnomeShell/Design/Whiteboards/DateNTime">design</a> through <a href="https://bugzilla.gnome.org/show_bug.cgi?id=632109">review</a> to <a href="http://git.gnome.org/browse/gnome-shell/commit/?id=885b6ffaef055291f01c139837e73b81a26fedde">code</a>, I am proud to present the new GNOME calendar panel:
<div class="separator" style="clear: both; text-align: center;">
<a href="http://people.freedesktop.org/~david/calendar.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="196" width="400" src="http://people.freedesktop.org/~david/calendar-thumb.png"/></a></div>
Thanks to everyone involved!davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com19tag:blogger.com,1999:blog-5268847417417953349.post-81560647304100075782010-11-30T15:03:00.003-05:002010-11-30T17:12:26.321-05:00GNOME 3 Calendar IntegrationI'm actually not supposed to be working this week (the first screenshot contains a subtle hint why), but here's a couple of screenshots of the upcoming calendar integration for the next major GNOME release:
<p>
<center><a href="http://people.freedesktop.org/~david/gnome3-calendar-shell-1.png"><img alt="Today" border="0" src="http://people.freedesktop.org/~david/gnome3-calendar-shell-1-thumb.png" /></a> <br>
<i>Click to embiggen</i>
</center>
<p>
Working in Javascript is really weird compared to C. But so far I think it's weird in a good way.
<p>
<center><a href="http://people.freedesktop.org/~david/gnome3-calendar-shell-2.png"><img alt="Today" border="0" src="http://people.freedesktop.org/~david/gnome3-calendar-shell-2-thumb.png" /></a> <br>
<i>Click to embiggen</i></center>
<p>
Here's the <a href="http://git.gnome.org/browse/gnome-shell/log/?h=datetime">git branch</a> with this code and here's the <a href="https://bugzilla.gnome.org/show_bug.cgi?id=632109">bug</a>. There's still some work left to make the code look and act like the <a href="http://live.gnome.org/GnomeShell/Design/Whiteboards/DateNTime">mockups</a>. I'm really hoping we can land the bulk of this code before Christmas - the biggest blocker right now is getting events from Evolution which is stressing the capabilities of the <a href="http://live.gnome.org/Gjs">gjs runtime</a> and <a href="http://live.gnome.org/GObjectIntrospection">GObject Introspection</a>. To make forward progress on the UI side, I had to switch to a in-process event source with fake events.davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com5tag:blogger.com,1999:blog-5268847417417953349.post-23907930327334275892010-08-18T17:58:00.000-04:002010-08-18T17:58:33.673-04:00gdbus(1) bash completionIt seems to be a well-kept secret that the new <a href="http://library.gnome.org/devel/gio/unstable/gdbus.html">gdbus(1)</a> command available in the latest GLib releases does bash completion. So here's a quick screencast demonstrating this very useful feature!<br />
<br />
<iframe class="youtube-player" type="text/html" width="480" height="385" src="http://www.youtube.com/embed/uByoaLM-Bro?hl=en_US" frameborder="0"></iframe><br />
<br />
Enjoy!davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com2tag:blogger.com,1999:blog-5268847417417953349.post-10741696056290485762010-05-19T00:07:00.001-04:002010-05-19T00:09:18.073-04:00This blog has moved...I'm no longer running my own server (good riddance!) so <a href="http://blog.fubar.dk/">http://blog.fubar.dk</a> does not work anymore (except it being a CNAME to here but I'm not sure how well that works). If you're seeing this, you're in the <a href="http://davidz25.blogspot.com/">right place</a>. The new setup has comments enabled (an often requested feature) and it looks like it's easier to use and stuff. Hopefully it'll make me blog some more. We'll see!davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com1tag:blogger.com,1999:blog-5268847417417953349.post-4634071175651284132009-12-16T11:16:00.000-05:002010-04-19T20:21:58.161-04:00udisks and gnome-disk-utility - past, present and future<p>The past year or so I’ve been working a lot on <a href="http://git.gnome.org/cgit/gnome-disk-utility">gnome-disk-utility</a> and DeviceKit-disks^W<a href="http://cgit.freedesktop.org/DeviceKit/DeviceKit-disks/">udisks</a> and I haven’t really blogged anything about it. Time to fix that!</p><br/> <p>First of all, given that the main <a href="http://cgit.freedesktop.org/DeviceKit/DeviceKit/">DeviceKit daemon</a> was killed by <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/libudev/">libudev</a> (if you are using GLib you want <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/gudev/">GUdev</a> instead!), we (that is, myself and other contributors) decided to <a href="http://lists.freedesktop.org/archives/devkit-devel/2009-December/000567.html">rename the project</a> - most people were cool with that; see <a href="http://lists.freedesktop.org/archives/devkit-devel/2009-December/000644.html">this message</a> for more reasons. One improvement in this area is that we’re committing to limited <a href="http://cgit.freedesktop.org/DeviceKit/DeviceKit-disks/tree/README#n17">ABI stability</a> - hopefully this will make things easier on distributors.</p><br/> <p>Most of the work in the udisks project been triggered by features we want in the Palimpsest Disk Utility - which is the most prominent part of gnome-disk-utility - the other part includes <a href="http://people.freedesktop.org/~david/blog-entry-20091216/computer.png">various</a> <a href="http://people.freedesktop.org/~david/blog-entry-20091216/gdu-format.png">desktop</a> <a href="http://people.freedesktop.org/~david/blog-entry-20091216/gdu-luks-unlock.png">shell</a> <a href="http://people.freedesktop.org/~david/blog-entry-20091216/gnome-ata-smart-fail.png">integration</a> <a href="http://people.freedesktop.org/~david/blog-entry-20091216/nautilus-mdraid-2.png">bits</a> that I don’t have room to cover in this post.</p><br/> <p>On the Disk Utility side, we’re now using an user interface that scales better with many disks - in the <a href="http://people.freedesktop.org/~david/blog-entry-20091216/old-ui.png">old user interface</a>, we used to show all partitions as children of each disk in the tree view. In the new user interface, we’re using a grid to convey the partitioning:</p><br/> <p><a href="http://people.freedesktop.org/~david/blog-entry-20091216/partitioning.png"><img src="http://people.freedesktop.org/~david/blog-entry-20091216/partitioning-thumb.png"></a> <br /> <i>Also see <a href="http://bugzilla.gnome.org/show_bug.cgi?id=579211">bgo #579211</a></i></p><br/> <p>While this interface isn’t optimal for the laptop or the machine-with-only-a-single-disk cases, it works a lot better for machines with many disks</p><br/> <p><a href="http://people.freedesktop.org/~david/blog-entry-20091216/palimpsest1.png"><img src="http://people.freedesktop.org/~david/blog-entry-20091216/palimpsest1-thumb.png"></a> <br /> <i>Bunch of disks</i></p><br/> <p>The Disk Utility also have somewhat complete support for <a href="http://en.wikipedia.org/wiki/Mdadm">MD RAID</a> (aka Software RAID on Linux) - we’ll show you <a href="http://people.freedesktop.org/~david/blog-entry-20091216/raid1.png">running arrays</a> and also arrays <a href="http://people.freedesktop.org/~david/blog-entry-20091216/raid2.png">not yet running</a>. There’s a way to <a href="http://people.freedesktop.org/~david/blog-entry-20091216/raid3.png">add/remove</a> components and currently I’m working on creating the UI for Online Capacity Expansion (e.g. dynamically expand a RAID-5 array from 4 to 8 drives).</p><br/> <p>The <a href="http://en.wikipedia.org/wiki/S.M.A.R.T.">SMART</a> stuff has been revamped and there is now <a href="http://people.freedesktop.org/~david/blog-entry-20091216/smart1.png">a way</a> to turn off the desktop shell notifications. As it turns out a lot of our current users are like “I’m fine with using a disk that is about to fail” - kinda shocking but there you go. Another change is that we don’t consider a disk to be failing if there’s only a couple of bad sectors - the disk will have to have <a href="http://git.0pointer.de/?p=libatasmart.git;a=commitdiff;h=b7f3834654cb05f5a8aae6b2381d548f49d72987">many bad sectors</a> to be considered failing. This helps with all the <i>false positives</i> the SMART warnings generated in e.g. Fedora 11. Anyway, it seems like the desktop notifications are useful - more than a handful of people have pinged me on IRC thanking me and <a href="http://0pointer.de/">Lennart</a> for this feature. Woot!</p><br/> <p>We also now have rudimentary support for showing the <a href="http://en.wikipedia.org/wiki/Serial_attached_SCSI">SAS</a> topology - this is really useful since machines with many disks typically use dedicated <a href="http://images.google.com/images?hl=en&um=1&sa=1&q=rack-mount+disk+enclosure&aq=f&oq=&aqi=&start=0">disk enclosure shelves</a>. Currently we only show <a href="http://en.wikipedia.org/wiki/Serial_attached_SCSI#SAS_Expanders">SAS Expanders</a> but the plan is to peek (and poke!) <a href="http://en.wikipedia.org/wiki/SES-2_Enclosure_Management">SES-2 Enclosures</a> (and in the future also <a href="http://en.wikipedia.org/wiki/SGPIO">SGPIO</a> ones) to get information about bay numbering and to toggle the FAILURE and LOCATE leds. Ideally we want to turn on the FAILURE led if a member of a RAID array is kicked or if SMART status indicates that the disk is about to fail. Ideally, we also want to do (sensible!) things like starting a RAID rebuild if a kicked RAID member is replaced by a uninitialized and same-size disk. As always, we have to be really careful about automatically doing policy things like this - especially on Linux where it is hard to make any assumptions on how the system is used.</p><br/> <p>The goal of the udisks effort is to be useful to users. For example, if you do video editing or a lot of photography, you care about IO performance. So I added a way to easily benchmark drives and RAID arrays:</p><br/> <p><a href="http://people.freedesktop.org/~david/blog-entry-20091216/benchmark2.png"><img src="http://people.freedesktop.org/~david/blog-entry-20091216/benchmark2-thumb.png"></a> <br /> <i>RAID-6 FTW!</i></p><br/> <p>This feature was inspired by various similar Windows programs (a pastime on <i>big storage</i> forums such as <a href="http://www.hardforum.com/forumdisplay.php?f=29">this one</a> involves trading benchmark screenshots) and the <a href="http://www.coker.com.au/bonnie++/zcav/">zcav</a> program. It’s pretty cute, actually; I learned that my Intel 80GB G1 SSD <a href="http://people.freedesktop.org/~david/Screenshot-80%20GB%20Solid-State%20Disk%20(ATA%20INTEL%20SSDSA2MH080G1GC)%20–%20Benchmark.png">only gives me 140MB/s</a> on my Lenovo Thinkpad X61 while I get a <a href="http://people.freedesktop.org/~david/benchmark-early-ui/benchmark-ssd.png">full 270MB/s</a> on my workstation (connected via SAS). Looks like the X61 is only using SATA 150 MB/s - a search on the tubes confirmed this. Gee! This feature is also useful when experimenting with RAID setups - for example, a 10K 36.7GB SAS disk <a href="http://people.freedesktop.org/~david/blog-entry-20091216/benchmark3.png">benchmarks like this</a> but when you put three of them together in Software RAID-5 it turns that <a href="http://people.freedesktop.org/~david/blog-entry-20091216/benchmark-raid5-oh-my-why-is-writing-so-slow.png">writing is really slow</a>. I know it’s supposed to be slow when writing to software RAID… but, really, this slow? Maybe I need to tweak some kernel tunables (which would be bad - we should be fast out of the box etc etc).</p><br/> <p>Finally, I did a really cute hack a few weeks ago - I made Palimpsest use <a href="http://en.wikipedia.org/wiki/D-Bus">D-Bus</a> over <a href="http://en.wikipedia.org/wiki/Internet_Protocol_Suite">TCP/IP</a> over <a href="http://en.wikipedia.org/wiki/Secure_Shell">SSH</a> to speak to udisks:</p><br/> <p><a href="http://people.freedesktop.org/~david/blog-entry-20091216/gdu-multiple-servers.png"><img src="http://people.freedesktop.org/~david/blog-entry-20091216/gdu-multiple-servers-thumb.png"></a> <br /> <i>Bunch of servers</i></p><br/> <p>With the <a href="http://git.gnome.org/cgit/gnome-disk-utility/tree/src/gdu/gdu-ssh-bridge.c#n41">way this works</a> we leverage SSH for authentication and <a href="http://avahi.org/">Avahi</a> for <a href="http://en.wikipedia.org/wiki/DNS-SD">Service Discovery</a>. The feature is still a bit rough - it needs some UI additions (such as a disconnect button) and we probably want to replace the <a href="http://people.freedesktop.org/~david/blog-entry-20091216/gdu-remote-server-openssh-auth-wtf.png">stock OpenSSH dialog</a> with something, uhm, less scary. There’s also the question of authorization - right now you have to connect as <a href="http://en.wikipedia.org/wiki/Superuser">root</a> in order to modify things (udisks uses <a href="http://cgit.freedesktop.org/PolicyKit">polkit</a> for authorization) - we need to figure out how we want this to work. Right now I’m thinking udisks will ship some polkit configuration bits (a <tt>.pkla</tt> file) and a <tt>udisks-admin</tt> group that administrators can add users to. We’ll see.</p><br/> <p>I’m kinda excited about the remote connectivity bits - it means that the Disk Utility is now very useful for headless server setups and said servers won’t have to run any graphical UI in order to be managed through the Disk Utility. We probably want to make the Disk Utility (e.g. Palimpsest) run on Win32 and OS X too (for users not running Linux on the desktop) for remote connectivity to e.g. Linux boxes running udisks. It shouldn’t be too hard to do this as we mainly depend on <a href="http://www.gtk.org/">GTK+</a> and D-Bus.</p><br/> <p>There’s a bunch of plans for both udisks and Palimpsest. We want to add support (meaning: udisks API and Palimpsest User Interface bits) for the following things: <a href="http://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)">LVM</a>, <a href="http://en.wikipedia.org/wiki/ISCSI">iSCSI</a> (both target and initiator), Enclosure Management (as mentioned above), Event logging (e.g. “raid5:md2: read error corrected”), <a href="http://en.wikipedia.org/wiki/Multipath_I/O">Multi-path</a>, <a href="http://en.wikipedia.org/wiki/Btrfs">Btrfs</a> and many other useful storage technologies. There’s also a few rough edges and missing features (e.g. file system resizing) but we’ll get to that in due time.</p><br/> <p>FWIW, it’s important to realize that udisks and the Disk Utility complements, <i>but does not replace</i>, existing command-line tools. This means that you can go ahead and keep using your command line tools and scripts and the udisks/GNOME/Palimpsest/etc. stack will react to changes. For example, you can start/stop RAID arrays using <tt>mdadm(8)</tt>, partition your disks using <tt>parted(8)</tt>, create filesystems using <tt>mkfs(8)</tt>, mount/umount filesystems and so on - the UI is kept up to date. While it’s approximately ten times harder (<a href="http://www.btinternet.com/~se16/hgb/statjoke.htm">with the 10x figure being 65% accurate</a>) to write software this way, it’s really the only way to do things on something like Linux where our set of users is as diverse as it is.</p>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com4tag:blogger.com,1999:blog-5268847417417953349.post-6738578964445376992009-09-29T11:34:00.000-04:002010-04-19T20:21:58.148-04:00Plumbers 2009<p>Had a great time at <a href="http://linuxplumbersconf.org/2009/">Plumbers Conference</a> last week in lovely Portland, OR. On the Friday on the conference, <a href="http://vrfy.org/">Kay</a> and I gave a talk, aptly named “Replugging the Modern Desktop”. The talk covered most of the <a href="https://wiki.ubuntu.com/Halsectomy">hal-ectomy</a> that has been happening and included a couple of demos. The slides are <a href="http://people.freedesktop.org/~david/talks/Plumbers-2009-Sievers-Zeuthen-Replugging-The-Modern-Desktop.pdf">here</a>. <b>UPDATE:</b> People on IRC are telling me that the fd.o servers are not reachable, I’ve put up another copy of the slides <a href="http://people.redhat.com/~davidz/Plumbers-2009-Sievers-Zeuthen-Replugging-The-Modern-Desktop.pdf">here</a>.<br/></p>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com0tag:blogger.com,1999:blog-5268847417417953349.post-25044976381178408732009-06-01T14:02:00.000-04:002010-04-19T20:21:58.139-04:00GUdev, gobject-introspection<p>I just committed <a href="http://git.kernel.org/?p=linux/hotplug/udev-extras.git;a=commit;h=28f6667f07ce429cc7b5e9796d5490ec9d4748e2">gudev-1.0</a> to <a href="http://git.kernel.org/?p=linux/hotplug/udev-extras.git;a=summary">udev-extras</a>. GUdev isn’t a particulary big library, it’s just a <a href="http://people.freedesktop.org/~david/gudev/">thin wrapper</a> around libudev with integration bits for the GLib main loop and support for <a href="http://live.gnome.org/GObjectIntrospection/">GObject Introspection</a>.</p><br/> <p>While GObject introspection is very neat, see this <a href="http://git.kernel.org/?p=linux/hotplug/udev-extras.git;a=blob;f=gudev/example.js;h=93aa118e1af2361e62e06d52a02b2b8be4ff5e90;hb=HEAD">GUdev example</a> using <a href="http://live.gnome.org/Seed/">Seed</a>, I must admit the whole thing feels… half-finished?</p><br/> <p>For example, we seem to be doing <a href="http://bugzilla.gnome.org/show_bug.cgi?id=584453">the wrong thing</a> for default annotations in some cases. Another example is the fact that <a href="http://live.gnome.org/Gjs/">gjs</a> <a href="http://bugzilla.gnome.org/show_bug.cgi?id=584558">lacks a print() function</a>, Seed <a href="http://bugzilla.gnome.org/show_bug.cgi?id=584321">can’t handle GStrv</a> properties (works in gjs), <a href="http://bugzilla.gnome.org/show_bug.cgi?id=560567">lack of support for string arrays</a> in gjs (works in Seed), neither Seed nor gjs supports <a href="http://en.wikipedia.org/wiki/Shebang_%28Unix%29">Shebang</a> support (<a href="http://bugzilla.gnome.org/show_bug.cgi?id=584560">gjs bug</a>, <a href="http://bugzilla.gnome.org/show_bug.cgi?id=584561">Seed bug</a>).</p><br/> <p>So all in all, trying to make my new small library introspectable.. was kinda.. much more work than I expected.. and most of the time was spent with the Seed and gjs runtimes trying to figure out what was wrong. </p><br/> <p>Anyway, I don’t want to sound all negative.. with all the work going on in this area, it looks like it’s going to rock once corner cases like mentioned above are ironed out. Rock on!</p>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com0tag:blogger.com,1999:blog-5268847417417953349.post-51947471832391559302009-05-04T05:40:00.000-04:002010-04-19T20:21:58.123-04:00Storage handling in GNOME<p>The GNOME 2.26 release in Fedora 11 will ship with a completely different stack for handling storage devices. The plan is to land all this work in the upstream GNOME 2.28 release and most of that work is done <a href="http://mail.gnome.org/archives/desktop-devel-list/2009-May/msg00007.html">already</a>.</p><br/> <p>Basically, we’ve switched away from <a href="http://www.freedesktop.org/wiki/Software/hal">one</a> daemon to <a href="http://hal.freedesktop.org/docs/DeviceKit-disks/">another</a> and in the process we gained a few new features.</p><br/> <p><br/><a href="http://people.freedesktop.org/~david/gdu-ata-smart-notification.png"> <img border="0" src="http://people.freedesktop.org/~david/gdu-ata-smart-notification-thumb.png" /> </a><br/><br/><i>OMG, the disk is dying - using <a href="http://0pointer.de/blog/projects/being-smart.html">libatasmart</a></i><br/></p><br/> <p>including letting the user <a href="http://people.freedesktop.org/~david/gdu-smart.png">inspect</a> what his disk is trying to tell him. Kudos goes to <a href="http://0pointer.de/lennart/">Lennart</a> for adding USB support, fixing issues with false positives in libatasmart and just for providing a really good library. And it looks like it’s <a href="https://www.redhat.com/archives/fedora-test-list/2009-April/msg01648.html">working</a>.</p><br/> <p>To really make this 100% useful, we probably want some kind of directory where system vendors shipping GNOME can drop an XML file and then the user can click a button “Order new disk” and get on with their life. Of course, then we need to handle the much harder problem of <a href="http://en.wikipedia.org/wiki/Backup">backup</a> and <a href="http://en.wikipedia.org/wiki/Data_migration">data migration</a>, but, hey, we want to solve that anyway.</p><br/> <p>So I also wrote this Palimpsest Disk Utility application<br/><br/><a href="http://people.freedesktop.org/~david/gdu-unmount-dialog.png"> <img border="0" src="http://people.freedesktop.org/~david/gdu-unmount-dialog-thumb.png" /> </a><br/><br/><i>Kitchen sink of disk information</i><br/></p><br/> <p>which also handles things like <a href="http://people.freedesktop.org/~david/gdu-raid5.png">RAID devices</a> and support for managing things like <a href="http://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linux%29">LVM</a> and <a href="http://btrfs.wiki.kernel.org/index.php/Main_Page">btrfs</a> is planned as well. We’re also looking at <a href="http://bugzilla.gnome.org/show_bug.cgi?id=579211">changing</a> the UI to be a bit more sane - something like this perhaps</p><br/> <p><br/><a href="http://people.freedesktop.org/~david/grid-3.png"> <img border="0" src="http://people.freedesktop.org/~david/grid-3-thumb.png" /> </a><br/><br/><i>Clean, leaner, simpler? Tell us in <a href="http://bugzilla.gnome.org/show_bug.cgi?id=579211">bug #579211</a>!</i><br/></p><br/> <p>Tomas, Matthias and I also worked on a simple formatting tool suitable for replacing the venerable <a href="http://library.gnome.org/users/gfloppy/stable/usage.html.en">gfloppy</a> tool</p><br/> <p><br/><a href="http://people.freedesktop.org/~david/gdu-format-tool-3.png"> <img border="0" src="http://people.freedesktop.org/~david/gdu-format-tool-3-thumb.png" /> </a><br/><br/><i>Formatting tool</i><br/></p><br/> <p>you <a href="http://people.freedesktop.org/~david/gdu-format-tool-1.png">can</a> <a href="http://people.freedesktop.org/~david/gdu-format-tool-2.png">see</a> <a href="http://people.freedesktop.org/~david/gdu-format-tool-3.png">the</a> <a href="http://people.freedesktop.org/~david/gdu-format-tool-4.png">whole</a> <a href="http://people.freedesktop.org/~david/gdu-format-tool-5.png">series</a> <a href="http://people.freedesktop.org/~david/gdu-format-tool-6.png">of</a> screenshots to see how the user interaction is supposed to be like.</p><br/> <p>Finally, I just merged a series of patches to GVfs for using this new machinery</p><br/> <p><br/><a href="http://people.freedesktop.org/~david/dkd-gdu-gvfs.png"> <img border="0" src="http://people.freedesktop.org/~david/dkd-gdu-gvfs-thumb.png" /> </a><br/><br/><i>OMG, pretty RAID icon!</i><br/></p><br/> <p>Building this on top of a modern storage daemon, as opposed to HAL, means that your file manager (and any other app using GIO), is instantly updated as you repartition or reformat your disks. It even works if you are using command-line tools, etc. <tt>fdisk</tt>(8) or <tt>mkfs</tt>(8) - geek comfort FTW!</p><br/> <p>We might want some extensions to GIO to make working with RAID and encrypted devices (and in the future LVM) easier (such as unlock/lock/start/stop buttons in the file manager sidepane) or maybe we’re just going to rely on the system auto-assembling such things and leave it in Palimpsest. Decisions, decisions.</p><br/> <p>Another geek-comfort I added, is the ability to show user mountable fstab entries, that’s <a href="http://bugzilla.gnome.org/show_bug.cgi?id=536292">bug #536292</a>. I was a bit reluctant to do this, I think instead we want native support in GVfs for <tt>nfs://</tt> yet still use the kernel filesystem driver though some kind of setuid helper. Then we’d also fix NFS servers to publish the fact they exist using <a href="http://avahi.org/">Avahi</a> and then people can discover it in Nautilus’ <tt>computer://</tt> window. Just cleaner and easier, I think. Combined with support for <a href="http://bugzilla.gnome.org/show_bug.cgi?id=556040">bookmarks on the GIO level</a>, this should be a dream come true for people deplying systems in the enterprise.</p><br/> <p>Most of this work is part of the bigger HAL-ectomy - my good friend Kay has a good post <a href="http://lists.freedesktop.org/archives/devkit-devel/2009-April/000140.html">outlining the plans</a> for this.</p>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com0tag:blogger.com,1999:blog-5268847417417953349.post-11480408760872424462008-10-01T12:24:00.000-04:002010-04-19T20:21:58.115-04:00Getting GVfs and FUSE right<p>One of the cool things about the new <a href="http://library.gnome.org/devel/gio/unstable/">IO library (GIO)</a> in the <a href="http://www.gnome.org/">GNOME</a> stack is that it’s <a href="http://library.gnome.org/devel/gio/unstable/extending.html">extensible</a>. One set of extensions for GIO is provided by the <a href="http://en.wikipedia.org/wiki/GVFS">GVfs</a> package. The main (but not only) purpose of GVfs is to provide access to <a href="http://en.wikipedia.org/wiki/Virtual_file_system">Virtual File Systems</a> by having the file system backend (e.g. file system driver) running in a separate daemon process in user space - sort of similar to what the <a href="http://en.wikipedia.org/wiki/Filesystem_in_Userspace">Filesystems in Userspace (FUSE)</a> framework provides on Linux and, nowadays, most UNIX operating systems. GVfs currently includes backends for <a href="http://en.wikipedia.org/wiki/SSH_file_transfer_protocol">SFTP</a>, <a href="http://en.wikipedia.org/wiki/Ftp">FTP</a>, <a href="http://en.wikipedia.org/wiki/OBEX">OBEX for Bluetooth</a>, <a href="http://en.wikipedia.org/wiki/Gphoto2">Digital cameras and MTP devices (through libgphoto2)</a>, <a href="http://en.wikipedia.org/wiki/Compact_Disc_Digital_Audio">Compact Disc Audio</a>, <a href="http://en.wikipedia.org/wiki/Webdav">Webdav</a>, <a href="http://en.wikipedia.org/wiki/Http">HTTP</a> and <a href="http://en.wikipedia.org/wiki/Archive_file">archive files (through libarchive)</a> to name the most important ones.</p><br/> <p>One key difference between FUSE and GVfs, is that GVfs file system drivers are not constrained by the rather old-fashioned, limited and <a href="http://en.wikipedia.org/wiki/Design_by_committee">design-by-committee</a> POSIX API. Instead, GIO provides (and <a href="http://svn.gnome.org/viewvc/gvfs/trunk/daemon/gvfsbackend.h?revision=1922&view=markup">exposes</a> for implementation to file system drivers) a modern and powerful API that provides <a href="http://library.gnome.org/devel/gio/unstable/GFile.html#g-file-load-contents-async">asynchronous IO</a>, <a href="http://library.gnome.org/devel/gio/unstable/GFile.html#g-file-monitor">file monitoring (like inotify)</a> and <a href="http://library.gnome.org/devel/gio/unstable/GFile.html#g-file-replace-contents">high-level</a> <a href="http://library.gnome.org/devel/gio/unstable/GFile.html#g-file-move">operations</a> as part of the standard API. The GIO API is at the right level of the stack, it’s part of the <a href="http://en.wikipedia.org/wiki/Glib">GLib</a> tarball. This means that any application that today uses <a href="http://en.wikipedia.org/wiki/GTK+">GTK+</a> has access to GIO. This is a pretty big deal, if you depend on GTK+ 2.14 on newer, you’ll also depend on GIO.</p><br/> <p>However, since it’s rather naive (not to mention arrogant) to think that people are going to be using only GTK+ applications, GVfs also ships with a FUSE daemon. This daemon provides a POSIX interface to the virtual file systems provided by GVfs. All the fancy high-level operations available when using GIO won’t be available (there’s no POSIX equivalent function to map to) of course, but the basic POSIX stuff (e.g. open(2), read(2) etc.) will work just fine. The way it works is that the GVfs FUSE daemon provides a mount in <tt>$HOME/.gvfs</tt> where each top-level directory represents a GVfs mount:</p><br/> <p><code><br/>$ ls -1 $HOME/.gvfs<br/>gphoto2 mount on usb%3A004,002<br/>sftp on people.freedesktop.org<br/>sftp on quad.local<br/></code></p><br/> <p>For example, this means all the good old command line tools still work</p><br/> <p><code><br/>$ stat /home/davidz/.gvfs/sftp\ on\ quad.local/home/davidz/.bashrc<br/> File: `/home/davidz/.gvfs/sftp on quad.local/home/davidz/.bashrc'<br/> Size: 313 Blocks: 0 IO Block: 4096 regular file<br/>Device: 14h/20d Inode: 44 Links: 1<br/>Access: (0600/-rw-------) Uid: ( 500/ davidz) Gid: ( 500/ davidz)<br/>Access: 2008-10-01 14:35:28.000000000 -0400<br/>Modify: 2008-04-25 11:55:02.000000000 -0400<br/>Change: 2008-04-25 11:55:02.000000000 -0400<br/></code><br/><code><br/>$ convert /home/davidz/.gvfs/gphoto2\ mount\ on\ usb%3A004\,002/DCIM/100NIKON/DSCN0001.JPG -dither out.jpg<br/></code></p><br/> <p>and so forth. Lots of <a href="http://en.wikipedia.org/wiki/Command-line_interface">command-line</a> people like this, at least they always like to complain when we didn’t have this back in the <a href="http://en.wikipedia.org/wiki/GnomeVFS">GnomeVFS</a> days.</p><br/> <p>But it gets better. Today I committed a <a href="http://bugzilla.gnome.org/show_bug.cgi?id=528670">patch to GIO</a> to ensure that GIO applications launching applications (such as the <a href="http://en.wikipedia.org/wiki/Nautilus_(file_manager)">Nautilus file manager</a> or the <a href="http://en.wikipedia.org/wiki/Evolution_(software)">Evolution mailer</a>) will always pass a FUSE path instead of the GIO URI. One implication of this is that if you launch a non-GIO application (such as <a href="http://en.wikipedia.org/wiki/Mplayer">mplayer</a> or a <a href="http://en.wikipedia.org/wiki/KDE">KDE</a> application) from Nautilus, that application will Just Work(tm) even if the file lives on a GVfs share.</p><br/> <p>I think this is a pretty big deal; now it <b>just doesn’t matter</b> what VFS system of the week is used in the application, it’s all POSIX as far as applications are concerned. Sure, there’s a small performance hit by having to go through the GVfs FUSE daemon (a couple of extra context switches) but for run-of-the-mill desktop applications this is insignificant. Also, as explained <a href="http://bugzilla.gnome.org/show_bug.cgi?id=530654#c8">here</a> it’s currently not a very smart idea to pass an URI to an application; there’s the problem we don’t have proper <a href="http://en.wikipedia.org/wiki/Request_for_Comments">standards</a> defining the URI’s we use; there’s also the problem of <a href="http://en.wikipedia.org/wiki/Single_sign_on">having to authenticate over and over again</a>.</p><br/> <p>Finally, if the application itself is using GIO, <a href="http://bugzilla.gnome.org/show_bug.cgi?id=530654">this patch</a> ensures we map back to a GIO URI and, thus, we bypass the FUSE daemon. This is best illustrated by the following screenshot:</p><br/> <p><br/><a href="http://people.freedesktop.org/~david/hadess-as-a-gvfs-example.png"><br/><img src="http://people.freedesktop.org/~david/hadess-as-a-gvfs-example-thumb.png" alt="Bastien in VFS heaven"></a><br/><br /><i>Bastien in VFS heaven</i><br/></p><br/> <p>As you can see from the terminal window, <a href="http://en.wikipedia.org/wiki/Eye_of_gnome">Eye of GNOME</a> is being passed a FUSE path from Nautilus. But thanks to the aforementioned patch, this is getting mapped back to a GIO URI as shown in the properties dialog.</p><br/> <p>This feature won’t be available in GNOME until 2.26 (and GLib 2.20) - we’re concerned some broken applications might be examining the URI before it’s <a href="http://svn.gnome.org/viewvc/glib/trunk/gio/gappinfo.c?r1=7554&r2=7566&view=patch">mapped back</a> so we’re <a href="http://bugzilla.gnome.org/show_bug.cgi?id=528670#c20">reluctant</a> to provide it in GLib 2.18 / GNOME 2.24 for now. FWIW, we’ve been shipping patches for this feature since <a href="http://en.wikipedia.org/wiki/Fedora_(operating_system)">Fedora 9</a> and the feature will be in Fedora 10 as well. Other vendors are encouraged to ship these patches as well.<br/></p>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com0tag:blogger.com,1999:blog-5268847417417953349.post-37564714047980364102008-08-19T08:32:00.000-04:002010-04-19T20:21:58.109-04:00DeviceKit presentations<p>Earlier today <a href="http://blogs.gnome.org/hughsie/">Richard</a> and I did a DeviceKit (the set of projects replacing HAL) presentation for some other group here at <a href="http://www.redhat.com">Red Hat</a>. You can see the slides <a href="http://lists.freedesktop.org/archives/devkit-devel/2008-August/000040.html">here</a> and <a href="http://lists.freedesktop.org/archives/devkit-devel/2008-August/000041.html">here</a>. </p><br/> <p>Perhaps of interest to the <a href="http://www.gnome.org">GNOME</a> community, there’s also screenshots of Palimpsest, an upcoming Disk Utility library and application for GNOME. Most of this is already available in Fedora’s development branch, dubbed Rawhide, but won’t be installed by default in Fedora 10. Right now I’m busy with <a href="http://www.freedesktop.org/wiki/Software/PolicyKit">PolicyKit</a> stuff but the plan is definitely to get a gnome-disk-utility mailing list going soon and get this stuff integrated throughout GNOME (I’ve already made sure it’s easy to plug into <a href="http://en.wikipedia.org/wiki/GVFS">gvfs</a> for example). </p><br/> <p>For example, I learned the other day that my disk <a href="http://people.freedesktop.org/~david/palimpsest-my-disk-is-failing.png">is</a> <a href="http://people.freedesktop.org/~david/palimpsest-fail-2.png">failing</a> (actually, as <a href="http://mjg59.livejournal.com/">mjg59</a> pointed out, it’s wrong to use the word FAILING since it’s Old-Age. Easily fixed.). Now, it would definitely be useful to have a notification bubble indicating this. This is pretty trivial to write using the <a href="http://hal.freedesktop.org/docs/DeviceKit-disks/">DeviceKit-disks API</a>; just monitor the <a href="http://hal.freedesktop.org/docs/DeviceKit-disks/Device.html#Device:drive-smart-is-failing">org.freedesktop.DeviceKit.Disks.Device:drive-smart-is-failing</a> property. Of course the gnome-disk-utility libraries (there’s one at the <a href="http://en.wikipedia.org/wiki/GLib">GLib</a> level and one at the <a href="http://en.wikipedia.org/wiki/GTK+">GTK+</a> level) wraps this nicely. In fact the gnome-disk-utility library at the GTK+ level should probably provide the code for doing this status icon. Something to discuss. Hence why a mailing list and more community involvement in the gnome-disk-utility project is needed.</p><br/> <p>Anyway, the goal is to port the most of the Fedora desktop to use DeviceKit instead of HAL for the Fedora 11 time frame.</p>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com0tag:blogger.com,1999:blog-5268847417417953349.post-68721701546994056292008-08-06T13:50:00.000-04:002010-04-19T20:21:58.103-04:00Resolution Independent GTK+<p>Got one of them big laptops where you feel tempted to use a looking glass because the pixels are so tiny? Ever feel cheated when you adjust the font size but the rest of the UI looks like crap in comparison? Or maybe ever felt dirty when hard coding pixel values in your application?</p><br/> <p><br/><a href="http://people.freedesktop.org/~david/file-chooser-em20.png"><br/><img><br/></a></p><br/> <p><br/><i>there’s also a <a href="http://people.freedesktop.org/~david/file-chooser-em5.png">tiny</a> and a <a href="http://people.freedesktop.org/~david/file-chooser-em10.png">normal</a> version…<br/></i></p><br/></p><br/> <p>Today I <a href="http://mail.gnome.org/archives/gtk-devel-list/2008-August/msg00044.html">sent</a> a patch to <a href="http://mail.gnome.org/mailman/listinfo/gtk-devel-list">gtk-devel-list</a> adding <a href="http://en.wikipedia.org/wiki/Resolution_independence">Resolution Independence</a> to the <a href="http://en.wikipedia.org/wiki/GTK+">GTK+</a> toolkit. Let’s see how that goes.<br/></p>davidzhttp://www.blogger.com/profile/18166813552495508964noreply@blogger.com0