Federico's Blog

  1. Loving Gitlab.gnome.org, and getting notifications

    - gitlab, gnome

    I'm loving gitlab.gnome.org. It has been only a couple of weeks since librsvg moved to gitlab, and I've already received and merged two merge requests. (Isn't it a bit weird that Github uses "pull request" and Everyone(tm) knows the PR acronym, but Gitlab uses "merge request"?)

    Notifications about merge requests

    One thing to note if your GNOME project has moved to Gitlab: if you want to get notified of incoming merge requests, you need to tell Gitlab that you want to "Watch" that project, instead of using one of the default notification settings. Thanks to Carlos Soriano for making me aware of this.

    Notifications from Github's mirror

    The github mirror of git.gnome.org is configured so that pull requests are automatically closed, since currently there is no way to notify the upstream maintainers when someone creates a pull request in the mirror (this is super-unfriendly by default, but at least submitters get notified that their PR would not be looked at by anyone, by default).

    If you have a Github account, you can Watch the project in question to get notified — the bot will close the pull request, but you will get notified, and then you can check it by hand, review it as appropriate, or redirect the submitter to gitlab.gnome.org instead.

  2. Librsvg 2.40.20 is released

    - gnome, librsvg, rust

    Today I released librsvg 2.40.20. This will be the last release in the 2.40.x series, which is deprecated effectively immediately.

    People and distros are strongly encouraged to switch to librsvg 2.41.x as soon as possible. This is the version that is implemented in a mixture of C and Rust. It is 100% API and ABI compatible with 2.40.x, so it is a drop-in replacement for it. If you or your distro can compile Firefox 57, you can probably build librsvg-2.41.x without problems.

    Some statistics

    Here are a few runs of loc — a tool to count lines of code — when run on librsvg. The output is trimmed by hand to only include C and Rust files.

    This is 2.40.20:
    -------------------------------------------------------
     Language      Files   Lines   Blank   Comment    Code
    -------------------------------------------------------
     C                41   20972    3438      2100   15434
     C/C++ Header     27    2377     452       625    1300
    
    This is 2.41.latest (the master branch):
    -------------------------------------------------------
     Language      Files   Lines   Blank   Comment    Code
    -------------------------------------------------------
     C                34   17253    3024      1892   12337
     C/C++ Header     23    2327     501       624    1202
     Rust             38   11254    1873       675    8706
    
    And this is 2.41.latest *without unit tests*, 
    just "real source code":
    -------------------------------------------------------
     Language      Files   Lines   Blank   Comment    Code
    -------------------------------------------------------
     C                34   17253    3024      1892   12337
     C/C++ Header     23    2327     501       624    1202
     Rust             38    9340    1513       610    7217
    

    Summary

    Not counting blank lines nor comments:

    • The C-only version has 16734 lines of C code.

    • The C-only version has no unit tests, just some integration tests.

    • The Rust-and-C version has 13539 lines of C code, 7217 lines of Rust code, and 1489 lines of unit tests in Rust.

    As for the integration tests:

    • The C-only version has 64 integration tests.

    • The Rust-and-C version has 130 integration tests.

    The Rust-and-C version supports a few more SVG features, and it is A LOT more robust and spec-compliant with the SVG features that were supported in the C-only version.

    The C sources in librsvg are shrinking steadily. It would be incredibly awesome if someone could run some git filter-branch magic with the loc tool and generate some pretty graphs of source lines vs. commits over time.

  3. Librsvg moves to Gitlab

    - gnome, librsvg

    Librsvg now lives in GNOME's Gitlab instance. You can access it here.

    Gitlab allows workflows similar to Github: you can create an account there, fork the librsvg repository, file bug reports, create merge requests... Hopefully this will make it nicer for contributors.

    In the meantime, feel free to take a look!

    This is a huge improvement for GNOME's development infrastructure. Thanks to Carlos Soriano, Andrea Veri, Philip Chimento, Alberto Ruiz, and all the people that made the move to Gitlab possible.

  4. A mini-rant on the lack of string slices in C

    - librsvg, rust

    Porting of librsvg to Rust goes on. Yesterday I started porting the C code that implements SVG's <text> family of elements. I have also been replacing the little parsers in librsvg with Rust code.

    And these days, the lack of string slices in C is bothering me a lot.

    What if...

    It feels like it should be easy to just write something like

    typedef struct {
        const char *ptr;
        size_t len;
    } StringSlice;
    

    And then a whole family of functions. The starting point, where you slice a whole string:

    StringSlice
    make_slice_from_string (const char *s)
    {
        StringSlice slice;
    
        assert (s != NULL);
    
        slice.ptr = s;
        slice.len = strlen (s);
        return slice;
    }
    

    But that wouldn't keep track of the lifetime of the original string. Okay, this is C, so you are used to keeping track of that yourself.

    Onwards. Substrings?

    StringSlice
    make_sub_slice(StringSlice slice, size_t start, size_t len)
    {
        StringSlice sub;
    
        assert (len <= slice.len);
        assert (start <= slice.len - len);  /* Not "start + len <= slice.len" or it can overflow. */
                                            /* The subtraction can't underflow because of the previous assert */
        sub.ptr = slice.ptr + start;
        sub.len = len;
        return sub;
    }
    

    Then you could write a million wrappers for g_strsplit() and friends, or equivalents to them, to give you slices instead of C strings. But then:

    • You have to keep track of lifetimes yourself.

    • You have to wrap every function that returns a plain "char *"...

    • ... and every function that takes a plain "char *" as an argument, without a length parameter, because...

    • You CANNOT take slice.ptr and pass it to a function that just expects a plain "char *", because your slice does not include a nul terminator (the '\0 byte at the end of a C string). This is what kills the whole plan.

    Even if you had a helper library that implements C string slices like that, you would have a mismatch every time you needed to call a C function that expects a conventional C string in the form of a "char *". You need to put a nul terminator somewhere, and if you only have a slice, you need to allocate memory, copy the slice into it, and slap a 0 byte at the end. Then you can pass that to a function that expects a normal C string.

    There is hacky C code that needs to pass a substring to another function, so it overwrites the byte after the substring with a 0, passes the substring, and overwrites the byte back. This is horrible, and doesn't work with strings that live in read-only memory. But that's the best that C lets you do.

    I'm very happy with string slices in Rust, which work exactly like the StringSlice above, but &str is actually at the language level and everything knows how to handle it.

    The glib-rs crate has conversion traits to go from Rust strings or slices into C, and vice-versa. We alredy saw some of those in the blog post about conversions in Glib-rs.

    Sizes of things

    Rust uses usize to specify the size of things; it's an unsigned integer; 32 bits on 32-bit machines, and 64 bits on 64-bit machines; it's like C's size_t.

    In the Glib/C world, we have an assortment of types to represent the sizes of things:

    • gsize, the same as size_t. This is an unsigned integer; it's okay.

    • gssize, a signed integer of the same size as gsize. This is okay if used to represent a negative offset, and really funky in the Glib functions like g_string_new_len (const char *str, gssize len), where len == -1 means "call strlen(str) for me because I'm too lazy to compute the length myself".

    • int - broken, as in libxml2, but we can't change the API. On 64-bit machines, an int to specify a length means you can't pass objects bigger than 2 GB.

    • long - marginally better than int, since it has a better chance of actually being the same size as size_t, but still funky. Probably okay for negative offsets; problematic for sizes which should really be unsigned.

    • etc.

    I'm not sure how old size_t is in the C standard library, but it can't have been there since the beginning of time — otherwise people wouldn't have been using int to specify the sizes of things.

  5. Code Hospitality

    - software with living structure

    Recently on the Greater than Code podcast there was an episode called "Code Hospitality", by Nadia Odunayo.

    Nadia talks about thinking of how to make people comfortable in your code and in your team/organization/etc., and does it in terms of thinking about host/guest relationships. Have you ever stayed in an AirBnB where the host carefully prepares some "welcome instructions" for you, or puts little notes in their apartment to orient/guide you, or gives you basic guidance around their city's transportation system? We can think in similar ways of how to make people comfortable with code bases.

    This of course hit me on so many levels, because in the past I've written about analogies between software and urbanism/architecture. Software that has the Quality Without A Name talks about Christopher Alexander's architecture/urbanism patterns in the context of software, based on Richard Gabriel's ideas, and Nikos Salingaros's formalization of the design process. Legacy Systems as Old Cities talks about how GNOME evolved parts of its user-visible software, and makes an analogy with cities that evolve over time instead of being torn down and rebuilt, based on urbanism ideas by Jane Jacobs, and architecture/construction ideas by Stewart Brand.

    I definitely intend to do some thinking on Nadia's ideas for Code Hospitality and try to connect them with this.

    In the meantime, I've just rewritten the README in gnome-class to make it suitable as an introduction to hacking there.

  6. Rust+GNOME Hackfest in Berlin, 2017

    - Berlin, gnome, hackfests, rust

    Last weekend I was in Berlin for the second Rust+GNOME Hackfest, kindly hosted at the Kinvolk office. This is in a great location, half a block away from the Kottbusser Tor station, right at the entrance of the trendy Kreuzberg neighborhood — full of interesting people, incredible graffitti, and good, diverse food.

    Rug of Kottbusser Tor

    My goals for the hackfest

    Over the past weeks I had been converting gnome-class from the old lalrpop-based parser into the new Procedural Macros framework for Rust, or proc-macro2 for short. To do this the parser for the gnome-class mini-language needs to be rewritten from being specified in a lalrpop grammar, to using Rust's syn crate.

    Syn is a parser for Rust source code, written as a set of nom combinator parser macros. For gnome-class we want to extend the Rust language with a few conveniences to be able to specify GObject classes/subclasses, methods, signals, properties, interfaces, and all the goodies that GObject Introspection would expect.

    During the hackfest, Alex Crichton, from the Rust core team, kindly took over my baby steps in compiler writing and made everything much more functional. It was invaluable to have him there to reason about macro hygiene (we are generating an unhygienic macro!), bugs in the quoting system, and general Rust-iness of the whole thing.

    I was also able to talk to Sebastian Dröge about his work in writing GObjects in Rust by hand, for GStreamer, and what sort of things gnome-class could make easier. Sebastian knows GObject very well, and has been doing awesome work in making it easy to derive GObjects by hand in Rust, without lots of boilerplate — something with which gnome-class can certainly help.

    I was also looking forward to talking again with Guillaume Gomez, one of the maintainers of gtk-rs, and who does so much work in the Rust ecosystem that I can't believe he has time for it all.

    Graffitti heads

    Extend the Rust language for GObject? Like Vala?

    Yeah, pretty much.

    Except that instead of a wholly new language, we use Rust as-is, and we just add syntactic constructs that make it easy to write GObjects without boilerplate. For example, this works right now:

    #![feature(proc_macro)]
    
    extern crate gobject_gen;
    
    #[macro_use]
    extern crate glib;
    use gobject_gen::gobject_gen;
    
    gobject_gen! {
        // Derives from GObject
        class One {
        }
    
        impl One {
            // non-virtual method
            pub fn one(&self) -> u32 {
                1
            }
    
            virtual fn get(&self) -> u32 {
                1
            }
        }
    
        // Inherits from our other class
        class Two: One {
        }
    
        impl One for Two {
            // overrides the virtual method
            // maybe we should use "override" instead of "virtual" here?
            virtual fn get(&self) -> u32 {
                2
            }
        }
    }
    
    #[test]
    fn test() {
        let one = One::new();
        let two = Two::new();
    
        assert!(one.one() == 1);
        assert!(one.get() == 1);
        assert!(two.one() == 1);
        assert!(two.get() == 2);
    }
    

    This generates a little boatload of generated code, including a good number of unsafe calls to GObject functions like g_type_register_static_simple(). It also creates all the traits and paraphernalia that Glib-rs would create for the Rust binding of a normal GObject written in C.

    The idea is that from the outside world, your generated GObject classes are indistinguishable from GObjects implemented in C.

    The idea is to write GObject libraries in a better language than C, which can then be consumed from language bindings.

    Current status of gnome-class

    Up to about two weeks before the hackfest, the syntax for this mini-language was totally ad-hoc and limited. After a very productive discussion on the mailing list, we came up with a better syntax that definitely looks more Rust-like. It is also easier to implement, since the Rust parser in syn can be mostly reused as-is, or pruned down for the parts where we only support GObject-like methods, and not all the Rust bells and whistles (generics, lifetimes, trait bounds).

    Gnome-class supports deriving classes directly from the basic GObject, or from other GObject subclasses in the style of glib-rs.

    You can define virtual and non-virtual methods. You can override virtual methods from your superclasses.

    Not all argument types are supported. In the end we should support argument types which are convertible from Rust to C types. We need to finish figuring out the annotations for ownership transfer of references.

    We don't support GObject signals yet; I think that's my next task.

    We don't support GObject properties yet.

    We don't support defining new GType interfaces yet, but it is planned. It should be easy to support implementing existing interfaces, as it is pretty much the same as implementing a subclass.

    The best way to see what works right now is probably to look at the examples, which also work as tests.

    Digression on macro hygiene

    Rust macros are hygienic, unlike C macros which work just through textual substitution. That is, names declared inside Rust macros will not clash with names in the calling code.

    One peculiar thing about gnome-class is that the user gives us a few names, like a class name Foo and some things inside it, say, a method name bar, and a signal baz and a property qux. From there we want to generate a bunch of boilerplate for GObject registration and implementaiton. Some of the generated names in that boilerplate would be

    Foo              // base name
    FooClass         // generated name for the class struct
    Foo::bar()       // A method
    Foo::emit_baz()  // Generated from the signal name
    Foo::set_qux()   // Generated property setter
    foo_bar()        // Generated C function for a method call
    foo_get_type()   // Generated C function that all GObjects have
    

    However, if we want to actually generate those names inside our gnome-class macro and make them visible to the caller, we need to do so unhygienically. Alex started started a very interesting discussion on macro hygiene, so expect some news in the Rust world soon.

    TL;DR: there is a difference between a code generator, which gnome-class mostly intends to be, and a macro system which is just an aid in typing repetitive code.

    Fuck wars

    People for whom to to be thankful

    During the hackfest, Nirbheek has been porting librsvg from Autotools to the Meson build system, and dealing with Rust peculiarities along the way. This is exactly what I needed! Thanks, Nirbheek!

    Sebastian answered many of my questions about GObject internals and how to use them from the Rust side.

    Zeeshan took us to a bunch of good restaurants. Korean, ramen, Greek, excellent pizza... My stomach is definitely thankful.

    Berlin

    I love Berlin. It is a cosmopolitan, progressive, LGBTQ-friendly city, with lots of things to do, vast distances to be traveled, with good public transport and bike lanes, diverse food to be eaten along the way...

    But damnit, it's also cold at this time of the year. I don't think the weather was ever above 10°C while we were there, and mostly in a constant state of not-quite-rain. This is much different from the Berlin in the summer that I knew!

    Hackers at Kimchi Princess

    This is my third time visiting Berlin. The first one was during the Desktop Summit in 2011, and the second one was when my family and I visited the city two years ago. It is a city that I would definitely like to know better.

    Thanks to the GNOME Foundation...

    ... for sponsoring my travel and accomodation during the hackfest.

    Sponsored by the GNOME Foundation

  7. Compilation notifications in Emacs

    - emacs

    Here is a little Emacs Lisp snippet that I've started using. It makes Emacs pop up a desktop-wide notification when a compilation finishes, i.e. after "M-x compile" is done. Let's see if that keeps me from wasting time in the web when I launch a compilation.

    (setq compilation-finish-functions
          (append compilation-finish-functions
              '(fmq-compilation-finish)))
    
    (defun fmq-compilation-finish (buffer status)
      (call-process "notify-send" nil nil nil
            "-t" "0"
            "-i" "emacs"
            "Compilation finished in Emacs"
            status))
    
  8. How glib-rs works, part 3: Boxed types

    - gnome, rust

    (First part of the series, with index to all the articles)

    Now let's get on and see how glib-rs handles boxed types.

    Boxed types?

    Let's say you are given a sealed cardboard box with something, but you can't know what's inside. You can just pass it on to someone else, or burn it. And since computers are magic duplication machines, you may want to copy the box and its contents... and maybe some day you will get around to opening it.

    That's a boxed type. You get a pointer to something, who knows what's inside. You can just pass it on to someone else, burn it — I mean, free it — or since computers are magic, copy the pointer and whatever it points to.

    That's exactly the API for boxed types.

    typedef gpointer (*GBoxedCopyFunc) (gpointer boxed);
    typedef void (*GBoxedFreeFunc) (gpointer boxed);
    
    GType g_boxed_type_register_static (const gchar   *name,
                                        GBoxedCopyFunc boxed_copy,
                                        GBoxedFreeFunc boxed_free);
    

    Simple copying, simple freeing

    Imagine you have a color...

    typedef struct {
        guchar r;
        guchar g;
        guchar b;
    } Color;
    

    If you had a pointer to a Color, how would you copy it? Easy:

    Color *copy_color (Color *a)
    {
        Color *b = g_new (Color, 1);
        *b = *a;
        return b;
    }
    

    That is, allocate a new Color, and essentially memcpy() the contents.

    And to free it? A simple g_free() works — there are no internal things that need to be freed individually.

    Complex copying, complex freeing

    And if we had a color with a name?

    typedef struct {
        guchar r;
        guchar g;
        guchar b;
        char *name;
    } ColorWithName;
    

    We can't just *a = *b here, as we actually need to copy the string name. Okay:

    ColorWithName *copy_color_with_name (ColorWithName *a)
    {
        ColorWithName *b = g_new (ColorWithName, 1);
        b->r = a->r;
        b->g = a->g;
        b->b = a->b;
        b->name = g_strdup (a->name);
        return b;
    }
    

    The corresponding free_color_with_name() would g_free(b->name) and then g_free(b), of course.

    Glib-rs and boxed types

    Let's look at this by parts. First, a BoxedMemoryManager trait to define the basic API to manage the memory of boxed types. This is what defines the copy and free functions, like above.

    pub trait BoxedMemoryManager<T>: 'static {
        unsafe fn copy(ptr: *const T) -> *mut T;
        unsafe fn free(ptr: *mut T);
    }
    

    Second, the actual representation of a Boxed type:

    pub struct Boxed<T: 'static, MM: BoxedMemoryManager<T>> {
        inner: AnyBox<T>,
        _dummy: PhantomData<MM>,
    }
    

    This struct is generic over T, the actual type that we will be wrapping, and MM, something which must implement the BoxedMemoryManager trait.

    Inside, it stores inner, an AnyBox, which we will see shortly. The _dummy: PhantomData<MM> is a Rust-ism to indicate that although this struct doesn't actually store a memory manager, it acts as if it does — it does not concern us here.

    The actual representation of boxed data

    Let's look at that AnyBox that is stored inside a Boxed:

    enum AnyBox<T> {
        Native(Box<T>),
        ForeignOwned(*mut T),
        ForeignBorrowed(*mut T),
    }
    

    We have three cases:

    • Native(Box<T>) - this boxed value T comes from Rust itself, so we know everything about it!

    • ForeignOwned(*mut T) - this boxed value T came from the outside, but we own it now. We will have to free it when we are done with it.

    • ForeignBorrowed(*mut T) - this boxed value T came from the outside, but we are just borrowing it temporarily: we don't want to free it when we are done with it.

    For example, if we look at the implementation of the Drop trait for the Boxed struct, we will indeed see that it calls the BoxedMemoryManager::free() only if we have a ForeignOwned value:

    impl<T: 'static, MM: BoxedMemoryManager<T>> Drop for Boxed<T, MM> {
        fn drop(&mut self) {
            unsafe {
                if let AnyBox::ForeignOwned(ptr) = self.inner {
                    MM::free(ptr);
                }
            }
        }
    }
    

    If we had a Native(Box<T>) value, it means it came from Rust itself, and Rust knows how to Drop its own Box<T> (i.e. a chunk of memory allocated in the heap).

    But for external resources, we must tell Rust how to manage them. Again: in the case where the Rust side owns the reference to the external boxed data, we have a ForeignOwned and Drop it by free()ing it; in the case where the Rust side is just borrowing the data temporarily, we have a ForeignBorrowed and don't touch it when we are done.

    Copying

    When do we have to copy a boxed value? For example, when we transfer from Rust to Glib with full transfer of ownership, i.e. the to_glib_full() pattern that we saw before. This is how that trait method is implemented for Boxed:

    impl<'a, T: 'static, MM: BoxedMemoryManager<T>> ToGlibPtr<'a, *const T> for Boxed<T, MM> {
        fn to_glib_full(&self) -> *const T {
            use self::AnyBox::*;
            let ptr = match self.inner {
                Native(ref b) => &**b as *const T,
                ForeignOwned(p) | ForeignBorrowed(p) => p as *const T,
            };
            unsafe { MM::copy(ptr) }
        }
    }
    

    See the MM:copy(ptr) in the last line? That's where the copy happens. The lines above just get the appropriate pointer to the data data from the AnyBox and cast it.

    There is extra boilerplate in boxed.rs which you can look at; it's mostly a bunch of trait implementations to copy the boxed data at the appropriate times (e.g. the FromGlibPtrNone trait), also an implementation of the Deref trait to get to the contents of a Boxed / AnyBox easily, etc. The trait implementations are there just to make it as convenient as possible to handle Boxed types.

    Who implements BoxedMemoryManager?

    Up to now, we have seen things like the implementation of Drop for Boxed, which uses BoxedMemoryManager::free(), and the implementation of ToGlibPtr which uses ::copy().

    But those are just the trait's "abstract" methods, so to speak. What actually implements them?

    Glib-rs has a general-purpose macro to wrap Glib types. It can wrap boxed types, shared pointer types, and GObjects. For now we will just look at boxed types.

    Glib-rs comes with a macro, glib_wrapper!(), that can be used in different ways. You can use it to automatically write the boilerplate for a boxed type like this:

    glib_wrapper! {
        pub struct Color(Boxed<ffi::Color>);
    
        match fn {
            copy => |ptr| ffi::color_copy(mut_override(ptr)),
            free => |ptr| ffi::color_free(ptr),
            get_type => || ffi::color_get_type(),
        }
    }
    

    This expands to an internal glib_boxed_wrapper!() macro that does a few things. We will only look at particularly interesting bits.

    First, the macro creates a newtype around a tuple with 1) the actual data type you want to box, and 2) a memory manager. In the example above, the newtype would be called Color, and it would wrap an ffi:Color (say, a C struct).

            pub struct $name(Boxed<$ffi_name, MemoryManager>);
    

    Aha! And that MemoryManager? The macro defines it as a zero-sized type:

            pub struct MemoryManager;
    

    Then it implements the BoxedMemoryManager trait for that MemoryManager struct:

            impl BoxedMemoryManager<$ffi_name> for MemoryManager {
                #[inline]
                unsafe fn copy($copy_arg: *const $ffi_name) -> *mut $ffi_name {
                    $copy_expr
                }
    
                #[inline]
                unsafe fn free($free_arg: *mut $ffi_name) {
                    $free_expr
                }
            }
    

    There! This is where the copy/free methods are implemented, based on the bits of code with which you invoked the macro. In the call to glib_wrapper!() we had this:

            copy => |ptr| ffi::color_copy(mut_override(ptr)),
            free => |ptr| ffi::color_free(ptr),
    

    In the impl aboe, the $copy_expr will expand to ffi::color_copy(mut_override(ptr)) and $free_expr will expand to ffi::color_free(ptr), which defines our implementation of a memory manager for our Color boxed type.

    Zero-sized what?

    Within the macro's definition, let's look again at the definitions of our boxed type and the memory manager object that actually implements the BoxedMemoryManager trait. Here is what the macro would expand to with our Color example:

            pub struct Color(Boxed<ffi::Color, MemoryManager>);
    
            pub struct MemoryManager;
    
            impl BoxedMemoryManager<ffi::Color> for MemoryManager {
                unsafe fn copy(...) -> *mut ffi::Color { ... }
                unsafe fn free(...) { ... }
            }
    

    Here, MemoryManager is a zero-sized type. This means it doesn't take up any space in the Color tuple! When a Color is allocated in the heap, it is really as if it contained an ffi::Color (the C struct we are wrapping) and nothing else.

    All the knowledge about how to copy/free ffi::Color lives only in the compiler thanks to the trait implementation. When the compiler expands all the macros and monomorphizes all the generic functions, the calls to ffi::color_copy() and ffi::color_free() will be inlined at the appropriate spots. There is no need to have auxiliary structures taking up space in the heap, just to store function pointers to the copy/free functions, or anything like that.

    Next up

    You may have seen that our example call to glib_wrapper!() also passed in a ffi::color_get_type() function. We haven't talked about how glib-rs wraps Glib's GType, GValue, and all of that. We are getting closer and closer to being able to wrap GObject.

    Stay tuned!

  9. Initial posts about librsvg's C to Rust conversion

    - librsvg, rust

    The initial articles about librsvg's conversion to Rust are in my old blog, so they may be a bit hard to find from this new blog. Here is a list of those posts, just so they are easier to find:

    Within this new blog, you can look for articles with the librsvg tag.

  10. The Magic of GObject Introspection

    - gnome, gobject-introspection, rust

    Before continuing with the glib-rs architecture, let's take a detour and look at GObject Introspection. Although it can seem like an obscure part of the GNOME platform, it is an absolutely vital part of it: it is what lets people write GNOME applications in any language.

    Let's start with a bit of history.

    Brief history of language bindings in GNOME

    When we started GNOME in 1997, we didn't want to write all of it in C. We had some inspiration from elsewhere.

    Prehistory: GIMP and the Procedural Database

    There was already good precedent for software written in a combination of programming languages. Emacs, the flagship text editor of the GNU project, was written with a relatively small core in C, and the majority of the program in Emacs Lisp.

    In similar fashion, we were very influenced by the design of the GIMP, which was very innovative at that time. The GIMP has a large core written in C. However, it supports plug-ins or scripts written in a variety of languages. Initially the only scripting language available for the GIMP was Scheme.

    The GIMP's plug-ins and scripts run as separate processes, so they don't have immediate access to the data of the image being edited, or to the core functions of the program like "paint with a brush at this location". To let plug-ins and scripts access these data and these functions, the GIMP has what it calls a Procedural Database (PDB). This is a list of functions that the core program or plug-ins wish to export. For example, there are functions like gimp-scale-image and gimp-move-layer. Once these functions are registered in the PDB, any part of the program or plug-ins can call them. Scripts are often written to automate common tasks — for example, when one wants to adjust the contrast of photos and scale them in bulk. Scripts can call functions in the PDB easily, irrespective of the programming language they are written in.

    We wanted to write GNOME's core libraries in C, and write a similar Procedural Database to allow those libraries to be called from any programming language. Eventually it turned out that a PDB was not necessary, and there were better ways to go about enabling different programming languages.

    Enabling sane memory management

    GTK+ started out with a very simple scheme for memory management: a container owned its child widgets, and so on recursively. When you freed a container, it would be responsible for freeing its children.

    However, consider what happens when a widget needs to hold a reference to another widget that is not one of its children. For example, a GtkLabel with an underlined mnemonic ("_N_ame:") needs to have a reference to the GtkEntry that should be focused when you press Alt-N. In the very earliest versions of GTK+, how to do this was undefined: C programmers were already used to having shared pointers everywhere, and they were used to being responsible for managing their memory.

    Of course, this was prone to bugs. If you have something like

    typedef struct {
        GtkWidget parent;
    
        char *label_string;
        GtkWidget *widget_to_focus;
    } GtkLabel;
    

    then if you are writing the destructor, you may simply want to

    static void
    gtk_label_free (GtkLabel *label)
    {
        g_free (label_string);
        gtk_widget_free (widget_to_focus);          /* oops, we don't own this */
    
        free_parent_instance (&label->parent);
    }
    

    Say you have a GtkBox with the label and its associated GtkEntry. Then, freeing the GtkBox would recursively free the label with that gtk_label_free(), and then the entry with its own function. But by the time the entry gets freed, the line gtk_widget_free (widget_to_focus) has already freed the entry, and we get a double-free bug!

    Madness!

    That is, we had no idea what we were doing. Or rather, our understanding of widgets had not evolved to the point of acknowledging that a widget tree is not a simply tree, but rather a directed graph of container-child relationships, plus random-widget-to-random-widget relationships. And of course, other parts of the program which are not even widget implementations may need to keep references to widgets and free them or not as appropriate.

    I think Marius Vollmer was the first person to start formalizing this. He came from the world of GNU Guile, a Scheme interpreter, and so he already knew how garbage collection and seas of shared references ought to work.

    Marius implemented reference-counting for GTK+ — that's where gtk_object_ref() and gtk_object_unref() come from; they eventually got moved to the base GObject class, so we now have g_object_ref() and g_object_unref() and a host of functions to have weak references, notification of destruction, and all the things required to keep garbage collectors happy.

    The first language bindings

    The very first language bindings were written by hand. The GTK+ API was small, and it seemed feasible to take

    void gtk_widget_show (GtkWidget *widget);
    void gtk_widget_hide (GtkWidget *widget);
    
    void gtk_container_add (GtkContainer *container, GtkWidget *child);
    void gtk_container_remove (GtkContainer *container, GtkWidget *child);
    

    and just wrap those functions in various languages, by hand, on an as-needed basis.

    Of course, there is a lot of duplication when doing things that way. As the C API grows, one needs to do more and more manual work to keep up with it.

    Also, C structs with public fields are problematic. If we had

    typedef struct {
        guchar r;
        guchar g;
        guchar b;
    } GdkColor;
    

    and we expect program code to fill in a GdkColor by hand and pass it to a drawing function like

    void gdk_set_foreground_color (GdkDrawingContext *gc, GdkColor *color);
    

    then it is no problem to do that in C:

    GdkColor magenta = { 255, 0, 255 };
    
    gdk_set_foreground_color (gc, &magenta);
    

    But to do that in a high level language? You don't have access to C struct fields! And back then, libffi wasn't generally available.

    Authors of language bindings had to write some glue code, in C, by hand, to let people access a C struct and then pass it on to GTK+. For example, for Python, they would need to write something like

    PyObject *
    make_wrapped_gdk_color (PyObject *args, PyObject *kwargs)
    {
        GdkColor *g_color;
        PyObject *py_color;
    
        g_color = g_new (GdkColor, 1);
        /* ... fill in g_color->r, g, b from the Python args */
    
        py_color = wrap_g_color (g_color);
        return py_color;
    }
    

    Writing that by hand is an incredible amount of drudgery.

    What language bindings needed was a description of the API in a machine-readable format, so that the glue code could be written by a code generator.

    The first API descriptions

    I don't remember if it was the GNU Guile people, or the PyGTK people, who started to write descriptions of the GNOME API by hand. For ease of parsing, it was done in a Scheme-like dialect. A description may look like

    (class GtkWidget
           ;;; void gtk_widget_show (GtkWidget *widget);
           (method show
                   (args nil)
                   (retval nil))
    
           ;;; void gtk_widget_hide (GtkWidget *widget);
           (method hide
                   (args nil)
                   (retval nil)))
    
    (class GtkContainer
           ;;; void gtk_container_add (GtkContainer *container, GtkWidget *child);
           (method add
                   (args GtkWidget)
                   (retval nil)))
    
    (struct GdkColor
            (field r (type 'guchar))
            (field g (type 'guchar))
            (field b (type 'guchar))) 
    

    Again, writing those descriptions by hand (and keeping up with the C API) was a lot of work, but the glue code to implement the binding could be done mostly automatically. The generated code may need subsequent tweaks by hand to deal with details that the Scheme-like descriptions didn't contemplate, but it was better than writing everything by hand.

    Glib gets a real type system

    Tim Janik took over the parts of Glib that implement objects/signals/types, and added a lot of things to create a good type system for C. This is where things like GType, GValue, GParamSpec, and fundamental types come from.

    For example, a GType is an identifier for a type, and a GValue is a type plus, well, a value of that type. You can ask a GValue, "are you an int? are you a GObject?".

    You can register new types: for example, there would be code in Gdk that registers a new GType for GdkColor, so you can ask a value, "are you a color?".

    Registering a type involves telling the GObject system things like how to copy values of that type, and how to free them. For GdkColor this may be just g_new() / g_free(); for reference-counted objects it may be g_object_ref() / g_object_unref().

    Objects can be queried about some of their properties

    A widget can tell you when you press a mouse button mouse on it: it will emit the button-press-event signal. When GtkWidget's implementation registers this signal, it calls something like

        g_signal_new ("button-press-event",
            gtk_widget_get_type(), /* type of object for which this signal is being created */
            ...
            G_TYPE_BOOLEAN,  /* type of return value */
            1,               /* number of arguments */
            GDK_TYPE_EVENT); /* type of first and only argument */
    

    This tells GObject that GtkWidget will have a signal called button-press-event, with a return type of G_TYPE_BOOLEAN, and with a single argument of type GDK_TYPE_EVENT. This lets GObject do the appropriate marshalling of arguments when the signal is emitted.

    But also! You can query the signal for its argument types! You can run g_signal_query(), which will then tell you all the details of the signal: its name, return type, argument types, etc. A language binding could run g_signal_query() and generate a description of the signal automatically to the Scheme-like description language. And then generate the binding from that.

    Not all of an object's properties can be queried

    Unfortunately, although GObject signals and properties can be queried, methods can't be. C doesn't have classes with methods, and GObject does not really have any provisions to implement them.

    Conventionally, for a static method one would just do

    void
    gtk_widget_set_flags (GtkWidget *widget, GtkWidgetFlags flags)
    {
        /* modify a struct field within "widget" or whatever */
        /* repaint or something */
    }
    

    And for a virtual method one would put a function pointer in the class structure, and provide a convenient way to call it:

    typedef struct {
        GtkObjectClass parent_class;
    
        void (* draw) (GtkWidget *widget, cairo_t *cr);
    } GtkWidgetClass;
    
    void
    gtk_widget_draw (GtkWidget *widget, cairo_t *cr)
    {
        GtkWidgetClass *klass = find_widget_class (widget);
    
        (* klass->draw) (widget, cr);
    }
    

    And GObject has no idea about this method — there is no way to query it; it just exists in C-space.

    Now, historically, GTK+'s header files have been written in a very consistent style. It is quite possible to write a tool that will take a header file like

    /* gtkwidget.h */
    typedef struct {
        GtkObject parent_class;
    
        void (* draw) (GtkWidget *widget, cairo_t *cr);
    } GtkWidgetClass;
    
    void gtk_widget_set_flags (GtkWidget *widget, GtkWidgetFlags flags);
    void gtk_widget_draw (GtkWidget *widget, cairo_t *cr);
    

    and parse it, even if it is with a simple parser that does not completely understand the C language, and have heuristics like

    • Is there a class_name_foo() function prototype with no corresponding foo field in the Class structure? It's probably a static method.

    • Is there a class_name_bar() function with a bar field in the Class structure? It's probably a virtual method.

    • Etc.

    And in fact, that's what we had. C header files would get parsed with those heuristics, and the Scheme-like description files would get generated.

    Scheme-like descriptions get reused, kind of

    Language binding authors started reusing the Scheme-like descriptions. Sometimes they would cannibalize the descriptions from PyGTK, or Guile (again, I don't remember where the canonical version was maintained) and use them as they were.

    Other times they would copy the files, modify them by hand some more, and then use them to generate their language binding.

    C being hostile

    From just reading/parsing a C function prototype, you cannot know certain things. If one function argument is of type Foo *, does it mean:

    • the function gets a pointer to something which it should not modify ("in" parameter)

    • the function gets a pointer to uninitialized data which it will set ("out" parameter)

    • the function gets a pointer to initialized data which it will use and modify ("inout" parameter)

    • the function will copy that pointer and hold a reference to the pointed data, and not free it when it's done

    • the function will take over the ownership of the pointed data, and free it when it's done

    • etc.

    Sometimes people would include these annotations in the Scheme-like description language. But wouldn't it be better if those annotations came from the C code itself?

    GObject Introspection appears

    For GNOME 3, we wanted a unified solution for language bindings:

    • Have a single way to extract the machine-readable descriptions of the C API.

    • Have every language binding be automatically generated from those descriptions.

    • In the descriptions, have all the information necessary to generate a correct language binding...

    • ... including documentation.

    We had to do a lot of work to accomplish this. For example:

    • Remove C-isms from the public API. Varargs functions, those that have foo (int x, ...), can't be easily described and called from other languages. Instead, have something like foov (int x, int num_args, GValue *args_array) that can be easily consumed by other languages.

    • Add annotations throughout the code so that the ad-hoc C parser can know about in/out/inout arguments, and whether pointer arguments are borrowed references or a full transfership of ownership.

    • Take the in-line documentation comments and store them as part of the machine-readable description of the API.

    • When compiling a library, automatically do all the things like g_signal_query() and spit out machine-readable descriptions of those parts of the API.

    So, GObject Introspection is all of those things.

    Annotations

    If you have looked at the C code for a GNOME library, you may have seen something like this:

    /**
     * gtk_widget_get_parent:
     * @widget: a #GtkWidget
     *
     * Returns the parent container of @widget.
     *
     * Returns: (transfer none) (nullable): the parent container of @widget, or %NULL
     **/
    GtkWidget *
    gtk_widget_get_parent (GtkWidget *widget)
    {
        ...
    }
    

    See that "(transfer none) (nullable)" in the documentation comments? The (transfer none) means that the return value is a pointer whose ownership does not get transferred to the caller, i.e. the widget retains ownership. Finally, the (nullable) indicates that the function can return NULL, when the widget has no parent.

    A language binding will then use this information as follows:

    • It will not unref() the parent widget when it is done with it.

    • It will deal with a NULL pointer in a special way, instead of assuming that references are not null.

    Every now and then someone discovers a public function which is lacking an annotation of that sort — for GNOME's purposes this is a bug; fortunately, it is easy to add that annotation to the C sources and regenerate the machine-readable descriptions.

    Machine-readable descriptions, or repository files

    So, what do those machine-readable descriptions actually look like? They moved away from a Scheme-like language and got turned into XML, because early XXIst century.

    The machine-readable descriptions are called GObject Introspection Repository files, or GIR for short.

    Let's look at some parts of Gtk-3.0.gir, which your distro may put in /usr/share/gir-1.0/Gtk-3.0.gir.

    <repository version="1.2" ...>
    
      <namespace name="Gtk"
                 version="3.0"
                 shared-library="libgtk-3.so.0,libgdk-3.so.0"
                 c:identifier-prefixes="Gtk"
                 c:symbol-prefixes="gtk">
    

    For the toplevel "Gtk" namespace, this is what the .so library is called. All identifiers have "Gtk" or "gtk" prefixes.

    A class with methods and a signal

    Let's look at the description for GtkEntry...

        <class name="Entry"
               c:symbol-prefix="entry"
               c:type="GtkEntry"
               parent="Widget"
               glib:type-name="GtkEntry"
               glib:get-type="gtk_entry_get_type"
               glib:type-struct="EntryClass">
    
          <doc xml:space="preserve">The #GtkEntry widget is a single line text entry
    widget. A fairly large set of key bindings are supported
    by default. If the entered text is longer than the allocation
    ...
           </doc>
    

    This is the start of the description for GtkEntry. We already know that everything is prefixed with "Gtk", so the name is just given as "Entry". Its parent class is Widget and the function which registers it against the GObject type system is gtk_entry_get_type.

    Also, there are the toplevel documentation comments for the Entry class.

    Onwards!

          <implements name="Atk.ImplementorIface"/>
          <implements name="Buildable"/>
          <implements name="CellEditable"/>
          <implements name="Editable"/>
    

    GObject classes can implement various interfaces; this is the list that GtkEntry supports.

    Next, let's look at a single method:

          <method name="get_text" c:identifier="gtk_entry_get_text">
            <doc xml:space="preserve">Retrieves the contents of the entry widget. ... </doc>
    
            <return-value transfer-ownership="none">
              <type name="utf8" c:type="const gchar*"/>
            </return-value>
    
            <parameters>
              <instance-parameter name="entry" transfer-ownership="none">
                <type name="Entry" c:type="GtkEntry*"/>
              </instance-parameter>
            </parameters>
          </method>
    

    The method get_text and its corresponding C symbol. Its return value is an UTF-8 encoded string, and ownership of the memory for that string is not transferred to the caller.

    The method takes a single parameter which is the entry instance itself.

    Now, let's look at a signal:

          <glib:signal name="activate" when="last" action="1">
            <doc xml:space="preserve">The ::activate signal is emitted when the user hits
    the Enter key. ...</doc>
    
            <return-value transfer-ownership="none">
              <type name="none" c:type="void"/>
            </return-value>
          </glib:signal>
    
        </class>
    

    The "activate" signal takes no arguments, and has a return value of type void, i.e. no return value.

    A struct with public fields

    The following comes from Gdk-3.0.gir; it's the description for GdkRectangle.

        <record name="Rectangle"
                c:type="GdkRectangle"
                glib:type-name="GdkRectangle"
                glib:get-type="gdk_rectangle_get_type"
                c:symbol-prefix="rectangle">
    
          <field name="x" writable="1">
            <type name="gint" c:type="int"/>
          </field>
          <field name="y" writable="1">
            <type name="gint" c:type="int"/>
          </field>
          <field name="width" writable="1">
            <type name="gint" c:type="int"/>
          </field>
          <field name="height" writable="1">
            <type name="gint" c:type="int"/>
          </field>
    
        </record>
    

    So that's the x/y/width/height fields in the struct, in the same order as they are defined in the C code.

    And so on. The idea is for the whole API exported by a GObject library to be describable by that format. If something can't be described, it's a bug in the library, or a bug in the format.

    Making language bindings start up quickly: typelib files

    As we saw, the GIR files are the XML descriptions of GObject APIs. Dynamic languages like Python would prefer to generate the language binding on the fly, as needed, instead of pre-generating a huge binding.

    However, GTK+ is a big API: Gtk-3.0.gir is 7 MB of XML. Parsing all of that just to be able to generate gtk_widget_show() on the fly would be too slow. Also, there are GTK+'s dependencies: Atk, Gdk, Cairo, etc. You don't want to parse everything just to start up!

    So, we have an extra step that compiles the GIR files down to binary .typelib files. For example, /usr/lib64/girepository-1.0/Gtk-3.0.typelib is about 600 KB on my machine. Those files get mmap()ed for fast access, and can be shared between processes.

    How dynamic language bindings use typelib files

    GObject Introspection comes with a library that language binding implementors can use to consume those .typelib files. The libgirepository library has functions like "list all the classes available in this namespace", or "call this function with these values for arguments, and give me back the return value here".

    Internally, libgirepository uses libffi to actually call the C functions in the dynamically-linked libraries.

    So, when you write foo.py and do

    import gi
    gi.require_version('Gtk', '3.0')
    from gi.repository import Gtk
    win = Gtk.Window()
    

    what happens is that pygobject calls libgirepository to mmap() the .typelib, and sees that the constructor for Gtk.Window is a C function called gtk_window_new(). After seeing how that function wants to be called, it calls the function using libffi, wraps the result with a PyObject, and that's what you get on the Python side.

    Static languages

    A static language like Rust prefers to have the whole language binding pre-generated. This is what the various crates in gtk-rs do.

    The gir crate takes a .gir file (i.e. the XML descriptions) and does two things:

    • Reconstructs the C function prototypes and C struct declarations, but in a way Rust can understand them. This gets output to the sys crate.

    • Creates idiomatic Rust code for the language binding. This gets output to the various crates; for example, the gtk one.

    When reconstructing the C structs and prototypes, we get stuff like

    #[repr(C)]
    pub struct GtkWidget {
        pub parent_instance: gobject::GInitiallyUnowned,
        pub priv_: *mut GtkWidgetPrivate,
    }
    
    extern "C" {
        pub fn gtk_entry_new() -> *mut GtkWidget;
    }
    

    And the idiomatic bindings? Stay tuned!

« Page 3 / 4 »