-
How I added a test suite for GtkFileChooserButton
GtkFileChooserButton is a quirky little beast. It is the widget you use when you have something like a dialog box to configure options, and you need a way to ask the user for a file or a folder.
Jim Cape wrote the original version of GtkFileChooserButton a long time ago, as a reasonably simple wrapper around GtkFileChooserDialog. You clicked on something like a GtkButton, which would bring up a GtkFileChooserDialog, and when the dialog was dismissed by the user, the button would then show the name of the selected file. Simple.
And then, things got more complicated. The underlying file chooser dialog became asynchronous. We replaced Gnome-VFS with GIO. GTK3 happened.
While I've kept GtkFileChooserDialog mostly in check (a debatable statement, given the number of bugs it has...), I've never really paid much attention to GtkFileChooserButton. It's a wrapper, so it should just keep working, I thought. But bugs appeared as the underlying infrastructure changed, and every time I fixed one, other new bugs would appear.
I never did a good job of testing all the subtleties of GtkFileChooserButton. So, this happened:
Resurrecting the tests
A long time ago I had a litle, ad-hoc, automated test suite for GtkFileChooserDialog in general. This was before GTestUtils got included in GLib. The code for the file chooser's tests got adapted to use GTestUtils as a testing framework, but since the tests failed, that code was left in the gtk+ source tree but never built.
A few weeks ago, when
I felt shark teeth in my buttocksthe extent of the brokenness of GtkFileChooserButton started to become clear, I started resurrecting the old tests.My home-grown testing program initially looked like this:
static gboolean test_filechooser_open (void) { ... return passed; } static gboolean test_filechooser_save (void) { ... return passed; } static gboolean test_filechooser (void) { passed = test_file_chooser_open (); passed = passed && test_file_chooser_save (); passed = passed && test_file_chooser_select_folder (); passed = passed && ...; return passed; } int main (int argc, char **argv) { gtk_init (&argc, &argv); if (test_filechooser ()) return 0; else return 1; }This had several shortcomings. First, since the tests were hardcoded as a chain of "&&" operations, all the tests had to be run every time. You couldn't select each test individually. It was hard to see which test failed when one did.
When the tests were intially converted to use GTestUtils, it looked more or less like this:
static void test_filechooser_open (void) { ... g_assert (passed); } static void test_filechooser_save (void) { ... g_assert (passed); } int main (int argc, char **argv) { gtk_test_init (&argc, &argv); ... g_test_add_func ("/GtkFileChooser/open", test_filechooser_open); g_test_add_func ("/GtkFileChooser/save", test_filechooser_save); ... }So, all the "return passed" became "g_assert (passed)" and each test got a human-readable name. With this you can pass an argument like "-p /GtkFileChooser/save" to the test program, and GTestUtils will just run that test. Also, when a test fails, you get its name printed on the console. GTestUtils is not the nicest testing framework out there, but it's much better than my old setup.
Tests for GtkFileChooserButton
My original tests weren't very useful for GtkFileChooserButton, so I added new ones. After writing some tests by hand with similar code between each other, I figured out that they are all a slightly different subset of these:
- Create a GtkFileChooserButton in either OPEN or SELECT_FOLDER mode.
- Set a starting filename or folder.
- Invoke the underlying GtkFileChooserDialog.
- Change the filename or folder of the GtkFileChooserButton while the dialog is active.
- Confirm the dialog or cancel it.
- Look at the final state of the button.
I replaced the hand-written tests with a generic "test everything" function, that checks the externally-visible state of GtkFileChooserButton after each of those key stages. The tests run many variations of the steps above. Some don't set up a starting filename, and others do. Some use the underlying dialog, some don't. Some confirm the dialog as if you pressed "OK"; some cancel the dialog, and thus GtkFileChooserButton needs to go back to its initial state.
The tests failed at first, of course, due to bugs in GtkFileChooserButton. But it was having the tests that forced me to understand exactly why the widget was failing in subtle ways. Having a way to test each assumption and every change made me get a deep understanding of how GtkFileChooserButton works very quickly. I now regret not having written those tests since the beginning.
What was hard to test
GtkFileChooserButton can work in two modes: one for GTK_FILE_CHOOSER_ACTION_OPEN, where it lets you select a single file, and one for GTK_FILE_CHOOSER_ACTION_SELECT_FOLDER, where it lets you select a single folder.
In OPEN mode, the button looks like an actual GtkButton that shows the basename of the currently-selected file. When clicked, it brings up a proper GtkFileChooserDialog. When the user dismisses the dialog, the button updates its contents to show the newly-selected filename.
However, in SELECT_FOLDER mode, the button actually uses a GtkComboBox. When clicked, it brings up a combo-box menu of Places (standard folders plus bookmarks). The menu also has an "Other..." item, which you can select to get a proper GtkFileChooserDialog to select an arbitrary folder. When the user dismisses the dialog, the combo box must be updated to show the newly-selected folder.
I wanted to test some things:
-
When you call GtkFileChooser's methods, for example to select a file, does the GUI update accordingly? While you can easily call gtk_file_chooser_select_file(), there is no direct way of asking, "what text is displayed in this widget?".
-
When the user presses on the button in OPEN mode, or in the combo box in SELECT_FOLDER mode, does the selection change accordingly? Testing the selection is a simple matter of calling gtk_file_chooser_get_filename(), but the first part of the question requires digging in the widget hierarchy and telling widgets to do things as if the user were driving them.
GUI testing projects like LDTP can more or less simulate a user driving the GUI by using accessibility interfaces. When a test wants to ask, "what text is this widget displaying?", it wants just to get the string, without caring whether the widget in question is a button, a combo box, an entry, or a label — and the a11y interfaces make this easy. Similarly, they make it easy to say, "activate this combo box, and select the Nth item", something which would be really cumbersome (and fragile) to do by simulating mouse clicks and grabs.
Since I had never used the accessibility interfaces to do anything, I ran Accerciser to see how to do operations on the GUI "by hand" with the accessibility APIs. For example, once you have a widget you want to query, you can get its accessible representation with gtk_widget_get_accessible() and then do atk_object_get_name() to see the text that the widget is displaying. Bingo! There is no need to see if the widget in question is a GtkButton, or a GtkLabel, or anything else.
So, currently, the tests for GtkFileChooserButton call ATK by hand and are written in C, as is the rest of the GTK+ source tree. I'd much rather have something higher-level like LDTP in place, which already has utilities to dig for widgets and activate them and dig for strings, all in various high-level languages.
Advancing tests based on catching signals instead of sleeping
The file chooser in GTK+ is highly asynchronous. When you call gtk_file_chooser_select_file(), that function will return quickly and the file chooser will start selecting that file asynchronously: it will load the directory, select the file, figure out its icon...
Regretfully, one anti-pattern in the file chooser's API is that while it is easy to know when an async operation starts, it is hard to know when it actually finishes. Although internally GIO has the right kind of async API (you start the async operation, you pass a callback to be called when the operation finishes, the callback gets called with a success or error value, you can cancel a running operation), the file chooser was originally not designed to be async. It had a synchronous, blocking API. When it was converted to be async, the original signals like selection-changed remained there, but we made no provision for those signals to return error values ("the file you wanted to select was not found"), or for operations to be cancelled.
Normally this is not a problem, and you can consider gtk_file_chooser_select_file() to be fire-and-forget — you call it when you are initializing your file chooser, and you only query the final filename when the user closes the file chooser. You never query it in between, and if the file chooser was not able to select the file your program wanted, it's not a big deal.
However, querying things in between is exactly what GtkFileChooserButton's test suite wants to do. Since my testing code could not know when the various operations were finished, it just had things like a sleep_in_main_loop() function that would wait for a fraction of a second before continuing. In theory, this would let the file chooser catch up. This is both fragile and slow: it is fragile if the tests are being run on a slow machine (or a slow VM), and it is slow because there is so much sleeping being done in dozens of tests.
To compound that, GtkFileChooserButton wasn't always proxying the correct signals from the underlying GtkFileChooserDialog. So, after doing an operation in the test code, all the code could do was wait for a bit before continuing.
I replaced the "sleep here" functions with a few others that actually check if certain signals have been emitted by the GtkFileChooserButton — signals that indicate that the file chooser has really loaded the file in question and that the async operations are finished. In the tests, there are calls to a function like signal_watcher_expect (watcher, "selection-changed"). The signal watcher notices if the signal has been emitted yet, and if not, waits for up to a predefined maximum amount of time before deciding that the signal is just not being emitted at all.
After doing that all the tests failed, of course — the signals weren't being emitted at the right times! So I set to fixing that. This made me re-understand the code of GtkFileChooserButton, and let me clean it up substantially.
After fixing things, the tests are able to run as fast as the file chooser button can confirm that things have happened. There is no sleeping anymore.
The tests took 71 seconds with the sleeps; they take only 21 seconds with signals only. Big win! And now GtkFileChooserButton's signals are actually reliable, and tests can be reliably run on any kind of machine. I hope.
Postscript, and help wanted
Finally, I would like to apologize to the people for whom GtkFileChooserButton has been so broken in the past. I hope that the tests will help keep things in check from now on.
The tests run out of the box in "make check" in GTK+ master, because accessibility is always on in Gnome 3. However, the test code does not work at all in the gtk-2-24 branch. I don't know how to enable a11y for the test program, or even to enable it for all GTK2 apps running in the desktop. If anyone knows how to do this, I would be very thankful to know.
-
Dirk-Jan C. Binnema sent a couple of items to let Yasnippet (screencast) expand g_return_*_if_fail() when you type.
Alex Murray pointed out that DevHelp ships devhelp.el, by Richard Hult, to let you open the reference docs for a library function with a single keystroke.
Thanks, and keep the ideas coming, people!
git clone https://github.com/federicomenaquintero/gnome-emacs-utils