World Domination as an Optimization Hack

Report from the GNOME Deployments, July 2006

Federico Mena-Quintero


Table of Contents

What
Why
How
When
Who
The questions
Classifying the replies
Administration tools
Performance
Performance on thin clients
Printing problems in general
Integration
Old versions of GNOME
Usability
Navigation
Lots of printers
Session management
Localization
Differences between the Real World and hackers
Thin clients
NFS
Customization and lockdown
Other considerations
Acknowledgments
A. References to the original data

Abstract

Across the world, there are many large to small deployments of GNOME. The large deployments account for the largest part of our user base. This paper presents an informal study of the requirements that those deployments have, based on feedback which they provided about their particular needs. By fixing the most common problems which the deployments are experiencing, we will make GNOME more attractive for future deployments, and we'll get more users faster.

What

The main user of GNOME is the average citizen in a government office, a public school, or a community center in a random town.

A "GNOME deployment" is wherever someone has installed GNOME on various machines for several users. Deployments vary in size from the very small, such as a small 10-person company with a few computers running GNU/Linux and GNOME, to the huge deployments in the public sector in India (500,000 users, 35,000 computers) and Spain (200,000 users, 115,000 computers). The large deployments account for the largest part of GNOME's user base. These are our main users!

In the past, the GNOME hackers have been more or less isolated from the end users of the software they produce. We hackers write code, either voluntarily or for a living, but we seldom get feedback from users: is our software usable? Are users finding problems which we never envisioned? Are system administrators having trouble setting up GNOME for thousands on machines? Does GNOME run well on the kind of machines that the deployments use — not overpowered gaming machines or latest-model laptops, but low-cost hardware, thin clients, and networked home directories?

This paper presents the results of an informal study conducted in June and July 2006, designed to be the first step in consciously catering to the needs of those large deployments.

Why

In short: world domination, fast.

During GUADEC 2005 in Stuttgart, Jeff Waugh presented the "10x10 vision". 10x10 means that it should be GNOME's goal to capture 10% of the global desktop market by 2010[1] . As of 2005, GNU/Linux's market share on the desktop is around 2.6% of the global computer population (thus GNOME's is even smaller), so achieving 10x10 is a tall order.

One way to help 10x10 happen is to make GNOME as friendly as possible to the kind of large deployments that are being set up across the world. If we analyze the problems they have right now and fix them, then GNOME will become more attractive to future deployments.

Everyone should read the article on eliminating barriers to entry, by Joel Spolsky.

Also, growing our user base eventually means that contributors will spring up from it. A technically-oriented kid in the third world will become interested in the software that his computer is running, and he will want to know the internals.

How

We can do this as an optimization hack.

How do you approach a performance problem in software? First, you find out what is wrong by careful analysis. Then, you make measurements or benchmarks. You fix the main problem. You make the measurements again to see if your fix helped. You repeat the process until you meet your performance requirements.

What we will be optimizing here is the friendliness of GNOME to its deployments.

First, we will ask the deployments what their problems are. This will set a nice precedent for GNOME getting in contact with its users, instead of just "pushing out software" with no feedback on how well it works for them.

Second, we will gather the responses and classify them. People will report problems of different kinds: usability problems, issues particular to thin clients, problems with administration tools. We'll see which problems are the most important.

Third, we will fix those problems. Some are a matter of programming, and some others require setting up a mini-community of people to solve them. For example, we'll see that nobody maintains older versions of GNOME, and yet people keep using them — this requires maintainers for those old versions, similar to the way old branches of the Linux kernel are actively maintained (or at least kept alive).

Fourth, we will advertise those fixes. They will make life easier for the existing deployments, and also make GNOME more friendly to new potential ones. Large parts of the world are getting computers and Internet connections: we want them to choose free software and GNOME.

Fifth, we will run these studies periodically, perhaps every year. This will let us see whether our fixes have actually fixed people's problems. Gradually, GNOME will become something that no one in his right mind would not want to have, and we will have achieved world domination. Mwahahahaha.

When

The questions to the deployments were sent out on June 09, 2006. The last responses were tallied around the beginning of July 2006.

Who

Like in all of free software, you can help if you care.

Fixing the problems which the deployments have reported is not a single-person job.

  • We'll need usability testers to crack the usability problems which people still experience.

  • We'll need X hackers to fix some performance problems.

  • We'll need tool builders to improve our tools for system administrators.

  • We'll need hackers of the core platform to improve our lockdown infrastructure.

  • We'll need web hackers to set up another round of questions next year, and a way to gather the replies.

The questions

The following questions were posted on several widely-read forums:[2]

  1. Where is your deployment located?

  2. Is it a government-sponsored deployment, or private?

  3. Approximately how many users do you have?

  4. Approximately how many computers do you have?

  5. When did you first deploy GNOME? What version did you use then?

  6. What version of GNOME are you using now? How often do you upgrade?

  7. Please describe your setup. Is it a big server and thin clients? Fat clients, centrally administered? Fat clients, no central administration?

  8. What software do your users use?

  9. Do you have a customized desktop for your users?

  10. How locked down are your users' desktops?

  11. Do your users have individual identities / home directories, or do they share a few accounts?

  12. Where do the users keep their data?

  13. What made you choose GNOME? Was it an indirect decision (i.e. did you pick a certain distribution, and GNOME happened to come with it)?

  14. What are your top 5 annoyances with GNOME on your deployment? What would make your life easier? What would have made your life easier when deploying GNOME?

  15. What would make your users' lives easier?

  16. What do you and your users like about GNOME? Are there any particular things you want to highlight?

  17. Would you mind your answers being made public? Or would you rather have us keep them private?

  18. Thank you! Any final comments?

The replies came in by email. The format of the answers is intentionally free-form, so that people would be able to reply with their real problems, instead of being constrained to reply within a pre-defined set of problems.

Classifying the replies

I put the basic data from the deployments in a spreadsheet: location of the deployment, number of users, number of client and server computers, NFS versus local home directories, thin vs. fat clients. A pointer to the spreadsheet is available in Appendix A, References to the original data.

Then, I created a number of categories for the problems which the deployments reported, and tried to match each problem to one of those categories. This is is not a perfect categorization, nor is it the only possible one.

In total, 18 deployments replied. This a small number to be a good statistical sample, but at least it's a starting point.

Figure 1, “Percentage of deployments that had each kind of problem” shows a histogram of how many deployments experienced each kind of problem. The following sections describe some of the most frequent problems.

Figure 1. Percentage of deployments that had each kind of problem

Percentage of deployments that had each kind of problem

Administration tools

Since the replies were sent in by the people in charge of the deployments, it is not surprising that they requested better tools for system administrators. GNOME has recently started to cater to them with tools like Sabayon, to create customized desktops, and Pessulus, to set up the lockdown parameters.

Having good tools for system administrators will make us look very good for potential deployments. These people need all the help they can get: imagine an overworked system administrator charged with the task of installing 1,000 machines and configuring them with GNOME.

Maybe a networked gconf, centrally managed would have made our life easier, instead of addressing all the needs using a cluster-ssh approach. A simplified and well-integrated user, group, shares and printer central management would have helped our users.” (Christopher Gabriel, South Tyrol, Italy, 15,000 users)

Lack of documentation about gnome configuration files, menu, syntax.” (Damien Chambe, Saint Etienne, France, 500 users)

"Continual hiding of settings". This came from one of the other administrators who is frustrated that things are moved around like a supermarket without clear reference with where the moved (or hidden) thing has gone. For a vivid example see this bug: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=190692 Not obvious how to preconfigure evolution. Scattered documentation. ” (Sitsofe Wheeler, Swansea University, United Kingdom, 80 users).

Managing settings systemwide for all users. Gnome is good with this by Sabayon but it can be pretty hard with applications like Gimp or Openoffice. A bit of help there would make an admins life much easier. My top 5 would all be more work on sabayon because its so incredibly good at what it does. It should just support more applications and be easier to use.” (Daniel Hedblom, Sollefteå, Sweden, 3,500 users)

It would be nice if there was a system-wide notification option, so we could send /etc/motd to every user logging in or something like that. Sabayon and pessulus need to be developed further, there needs to be a GUI to deploy sabayon profiles, and it would be nice to be able to use pessulus for every option on the desktop. [...] There's no real equivalent to a Group Policy Object like there is in active directory. It'd be nice to have the same level of control of our fat clients as we do with our thin clients.” (Jorge Castro, Oakland University, United States, 2500 users)

An application like kwrited. No, seriously, GNOMErs on my company don't get notified by shutdown +nn or wall unless they have open terminals.” (Manuel Amador, Guayaquil, Ecuador, 6 users)

Sabayon is poor, it was not very well documented and it would only create a config for 800x600 desktop where we wanted 1024x768. This resulted in the wastebin and clear desktop we wanted on the far right of the panel ending up in the middle somewhere.” (Mark Johnson, United Kingdom, 1,500 users).

Difficulty in customising the menus.” (James Mathew, Free Software Foundation of India, 500,000 users)

Currently GNOME doesn't support a tool for administration centraliced of certificates (ex: x.509,..). The Junta of Andalucia gives to each person a personal certificate that is used to sign official documents, sing taxes, sing califications of students,.... Until now we have to add the same certificate to evolution, to the web browser and the office application too.” (Alfredo Matas, Andalucía, Spain, 200,000 users)

Tools to create a default profile, set up gconf defaults, etc. Documentation for the same - not just the tool, but the underlying setup. Preferably not tools that rely on ageing and somewhat flaky software like Xnest.” (Craig Ringer, Perth, Australia, 30 users)

Performance

Optimizing for performance and low memory consumption is particularly important for large deployments. They are usually cash-strapped and have low-powered machines; or if they use thin clients, they want to maximize the number of simultaneous users that can share a server.

Memory occupation, because on one server, we run 50 gnome users. With 4GB of memory, sometimes it is short.” (Damien Chambe, Saint Etienne, France, 500 users)

Evolution IMAP performance is horrible, I had to replace it with thunderbird. [...] Initial testing shows that the 2.14 desktop takes up MUCH less resources than 2.12. Even a few kilobytes of memory shared per applet becomes significant when serving multiple users off of one machine. Keep up the performance work!” (Jorge Castro, Oakland University, United States, 2500 users)

Users' mention that startup time is slow. Many of the people who switch their desktop do it for reasons of speed (i.e. XFCE is faster). [...] WebDav with graphical file browsing is slow.” (Sitsofe Wheeler, Swansea University, United Kingdom, 80 users)

And finally improve the performance. Most of pcs are pentium celeron 500mhz with 256 of ram.” (Alfredo Matas, Andalucía, Spain, 200,000 users)

Performance on thin clients

Hackers who work at home or in their offices, in their own personal computers, are not used to seeing the kind of performance problems that happen on thin clients.

Some thin clients are X terminals, and the main problem is to reduce the number of round-trips to the X server: latency in the connection is the main problem. Some other thin clients use a VNC-like setup, and there the problem has to do with minimizing the size of the areas that get repainted.

The fade out to quit is very slow with remote X, unable to find a way to disable it.” (Damien Chambe, Saint Etienne, France, 500 users)

Detection of remote displays and automatic disabling of some graphical features that require excessive server round trips. Bandwidth is nowhere near as important. Good examples include the metacity minimize/maximize animations and the log out fade.” (Craig Ringer, Perth, Australia, 30 users)

Printing problems in general

Printing is still painful in GNU/Linux. Programs do not come up with the right paper size, drivers are not installed for common printers, and printing is just not reliable.

Not always easy to default all GNOME programs to default to the correct paper size.” (Sitsofe Wheeler, Swansea University, United Kingdom, 80 users).

The gnome printing system (cups-frontend) is really nice to look at but far from useful. we cannot manage to print "number-up=2" from the interfaces. We only manage by setting these options in the commandline.” (Klaus Ita, University of Vienna, Austria, 800 users)

Printing is routinely broken, specifically gnome-cups-icon, it leaks, and takes up too much CPU, so I remove the package. Users can still print, they just get no notification or any kind of management of their jobs.” (Jorge Castro, Oakland University, United States, 2500 users)

My daughter gets confused when she uses the machine at home, I need Linux for Work, she needs Windows XP for school. There has been many times where i've been busy using the computer for work and she's had an assignment to print and it's easier to drive her to a professional printer than it is to attempt to get Linux to print her document at home.” (requested privacy, 300 users)

Integration

Some integration problems are about making various pieces of the software smarter when they work together. This is a very broad category, but in general anything that can reduce the amount of bookkeeping that users and administrators have to do is appreciated.

Applications often require or are enhanced with the notification area. Some software (like TomBoy) requires it by default. The problem is that ordinary users have no idea what it means, or how to add it. GNOME should automatically add this if software asks for it..or just always have it turned on.” (Dave Richards, City of Largo, United States, 800 users)

Firefox doesn't use GNOME settings.” (Sitsofe Wheeler, Swansea University, United Kingdom, 80 users).

Old versions of GNOME

GNOME hackers stop caring about GNOME 2.x when 2.(x+2) comes out. This creates a big problem for deployments: they have to rely on what is essentially unmaintained software, and they do not have the technical resources to backport patches themselves.

On the other hand, the GNOME community as it exists at the time of this writing does not have enough resources to maintain older versions of GNOME. This should be perfectly possible: there are maintainers for old versions of the Linux kernel, and there is enough interest in those version that patches get updated.

Upgrading to a newer GNOME version is very difficult for a deployment. Unlike our API and ABI stability guarantees, we have no such guarantees or recommendations for things like GConf keys. With a new version, a program may decide to change the way its configuration keys work, or to completely change which configuration keys it uses. Often the program does not provide a way to automatically migrate old settings: the author assumes that "people never upgrade", that everything is "a new install", and thus existing installations do break down when the old keys are not preserved. If you upgrade from GNOME 2.8 to GNOME 2.14, many programs will break because part of your configuration data in ~/.gconf is meaningless in the new version.

A related problem is when a deployment has different versions of GNOME installed for different client machines, but users have their home directories available on all of them. There may be machines with GNOME 2.6, machines with GNOME 2.10, and machines with GNOME 2.14. Several things can happen: old versions break when they get an unexpected configuration value from a newer version of the program; newer versions break or "look unconfigured" if the only thing that is present is the configuration from an older version.

Users see this as very annoying behavior: their panel's configuration breaks down because applets change their configuration values all the time, programs stop working, and they have no option but to request that an admin "fix it" by erasing big parts of their ~/.gnome2 or ~/.gconf.

I told you that if I had to pick one thing as an 'admin', it's the over-aggresive (in my view) desire to open a new release and leave behind the stable version. It's very hard to deploy and test when people are using bleeding edge libraries.” (Dave Richards, City of Largo, United States, 800 users)

Upgrade between versions sometimes gets broken. we have to 'mv .gnom* __tmp/' in order to make accounts work again.” (Klaus Ita, University of Vienna, Austria, 800 users)

gconf is fragile. Since we deploy multiple distros, we make each distro use it's own .gconf directory, like .gconf-rhel3, .gconf-rhel4, .gconf-ubuntu, etc. Since the users use NFS homes, if they all use .gconf, they clobber each other, which results in broken desktop icons, etc. etc. ” (Jorge Castro, Oakland University, United States, 2500 users)

Usability

We are doing things right with the push for usability since GNOME 2.0. Several people mentioned that the usability of GNOME was one of the big deciding factors for their deployment.

However, there are remaining problems. The usability teams at Sun and Novell have ensured that the basic desktop is usable, that people are able to find the programs they need, and that basic operations are easy to discover. However, we now need to do usability tests on the day-to-day work which people do. Can we simplify their workflow? Can we find operations that are common yet cumbersome?

Nautilus was a big advantage - users who tried KDE found themselves unable to effectively work with Konqueror.” (Craig Ringer, Perth, Australia, 30 users)

It's Simple. Once you tell them "the menu is on the top left instead of the bottom right", they tend to explore and learn things on their own.” (Jorge Castro, Oakland University, United States, 2500 users)

To this day, 99% of our users have no idea that you can create shortcuts in the left edge of the GNOME file manager. Nothing on the screen indicates that you can do this.” (Dave Richards, City of Largo, United States, 800 users)

logout button (they never find it). Inconsistency which buttons cancels an action (on windows always the right one irc). ” (Maximilian Attems, Technical University of Vienna, Austria, 30 users)

Navigation

These are problems with navigating the file system. While the basic usability of the desktop is acceptable, people have real trouble navigating their own home directories and operating with their files. Common problems include the following:

  • A user types a document and hits File/Save. He simply changes the "Untitled" name to "Letter to mom", and hits the Save button. However, the user doesn't notice the actual folder where the file was saved. The next day, he has no idea where to look for that file. Is it in ~/Documents, in ~/, in ~/Desktop? Or did the program happen to pick a totally unrelated folder for the Save dialog?

  • People spend too much time hunting down files which they just worked on. You receive a mail attachment, and you save it to a folder on your home directory. A few hours later, you need to open that file. Do you remember where you put it? The computer should be able to help you: since you were working with that file just a little while ago, it is likely that you'll need that file again soon. New user interfaces like Gimmie may be the right way to solve this problem: Gimmie is a replacement for the GNOME Panel which always shows you your most recently-used files. Programs need to be made to save this information as appropriate.

  • Programs pick completely different default folders for their Open and Save dialogs. Some use the current working directory of the program; some always use ~/Documents; some use the last directory on which you worked within that program. The result is that file dialogs are completely unpredictable. One option is to make file dialogs automatically show the last folder on which you worked, regardless of program. However, this needs to be tested on users to see how well it works.

One has to remember that these are everyday people who are having trouble, not hackers. People don't understand hierarchical file systems at first, and simply put everything in the default location, or in ~/Desktop, or in ~/Documents. Once they realize that this approach does not scale well to more than a few files, they may learn to create folders and subfolders. Then the problem becomes as described above: the default location in file dialogs becomes unpredictable, and users forget which folder they used to put certain files.

Right now, the desktop does very little to help users maintain their working set of files, which is the set files that they have used most recently. These should just be available everywhere, immediately. For files which you no longer use, but you may want to consult occasionally, there are solutions like Beagle searching.

The biggest issue by far in computers with low and medium skilled works is file management. They do not understand where they are saving documents, and how to later find it.” (Dave Richards, City of Largo, United States, 800 users)

The save file window is sometimes hard to understand for them if they want to save a document on a particular place [...] Unable to redirect the default directory to save documents in gnome applications, and change the "personal folder" icon to a subfolder of "home".” (Damien Chambe, Saint Etienne, France, 500 users)

Lots of printers

Large networks are likely to have many printers available. Since these get automatically discovered by GNOME, the Print dialog may show hundreds of printers, and the user is expected to find and choose one! There is no way to find or define the printers that are physically closest to each computer.

We have 75 printers. The printer selection dialogs are almost never tested for that many printers, and you have to scroll through lots of printers to find what you want..sometimes with big ugly icons that take up too much space. Then,often the printer setting doesn't stick from session to session.” (Dave Richards, City of Largo, United States, 800 users)

It's nice that cups broadcasts printers and it's easy to find and get to print, but with a department of nearly 40 printers, users routinely print to the wrong printer. It would be nice if as an administrator I could specify "these clients in this room default to this specific printer" or "they can only print to this printer".” (Jorge Castro, Oakland University, United States, 2500 users)

Session management

Users routinely leave labs without logging out. We disable the lock screen so people just can't lock workstations and leave them useless. It would be nice if there was an option to logout users after an idle period.” (Jorge Castro, Oakland University, United States, 2500 users)

People also want software to remember where it was closed, what size it was saved in and what view was last used.” (Dave Richards, City of Largo, United States, 800 users)

Localization

These are problems with poor localization for the user's local language. We are doing really well in this respect, and the fabulous efforts of the GNOME translators have paid off. The only deployment which reported localization problems was for Malayalam in India (500,000 users!). GNOME is truly worldwide in this respect, and works well.

It's nice to observe a professor giving a lab, and seeing a few student's desktops in their native languages, especially RTL languages, and watch them follow along the instruction. That's just f****ing cool. ” (Jorge Castro, Oakland University, United States, 2500 users)

Differences between the Real World and hackers

GNOME is used in the real world in a way that is very different from what we hackers are used to in our personal computers.

Thin clients

Most hackers have a computer which they own: a high-powered desktop machine or laptop, or both. All their applications and data are stored and run on that machine. The user has all of the machine's resources at his disposal. However, this is a problem once you want to have a thousand users, because a thousand high-powered machines are very expensive and they'll inevitably be underutilized. Also, administering a thousand machines is a royal pain.

However, 39% of the deployments in this study use thin clients as a way to save money in hardware and in administration. A setup of thin clients is when you have a fat server running applications simultaneously for all your users, and a bunch of low-powered machines which simply present a display to each user, and let him type on a keyboard. That is, the machine at which the user sits to work is not the machine that is running the software. The user may sit at any of the machines available, and his data and applications will be available there; they share the hardware available on the server.

NFS

At home or in their office, hackers normally keep all their personal data on their local hard drive. This means that file locking always works, operations on user files are reasonably fast, and backups are a big headache.

Among the deployments, however, local home directories are more or less rare. 72% of the deployments use home directories mounted over NFS. The assumptions in that kind of environment are very different to the ones we are accustomed.

Customization and lockdown

50% of the deployments do not use a GNOME desktop "as shipped", but they include their own customizations: launchers for programs, panel menu items, a desktop background suitable for the deployment, and branding of the desktop to use specific logos and images. Most hackers are happy to use a stock GNOME desktop, or they carry over the personal customizations of several years from their home directories.

33% of the deployments reported that they use the lockdown features of GNOME. Often, a system administrator wants to disable certain parts of the desktop's functionality for the users. For example, on a computer with shared accounts, the sysadmin may not want users to change the configuration of the panels or the desktop background. Hackers normally don't use the lockdown features at all, since they want to be able to tweak everything in their desktops.

Even the basic GDM setup needs to be centralized with the other tools for system administrators. A sysadmin will want to change things like "users should not have a Shut Down command in their desktop". It would be good to have this option in the same place where the custom desktops are defined.

Other considerations

Only 33% of the deployments use the lockdown features. These need to be more widely publicized. Right now applications do lockdown on an ad-hoc basis. There are some GConf keys which are not widely known, and application writers need to know, for example, that /desktop/gnome/lockdown/disable_command_line is the key used to control whether users should be able to use terminals and shell command lines. We need a formal lockdown API, with high-level functions like is_printing_enabled() or can_users_change_the_desktop_background() (the latter is so that Nautilus and F-spot can simply not display the "set image as background" menu option appropriately).

In general, we need to make very good publicity of the lockdown and customization tools for system administrators. It would be convenient to do usability tests of these tools, too! There needs to be a section of www.gnome.org which tells you what steps you need to take to deploy GNOME effectively.

The graphs above show the number of deployments that had a particular problem. Another way to graph this is by the number of users that have each problem, based on which deployments reported the problem — this has the disadvantage that the really big deployments (those with hundreds of thousands of users) hide all the other problems, even if they are common.

Future studies like this one may want to add some questions to the ones that got included here:

  • What kind of hardware do you have? What is the average CPU/memory for your clients and your servers?

  • How often do you upgrade that hardware?

  • If you use thin clients, do you use a Citrix/VNC-like setup or X terminals?

In the future we need to train local people to do the kind of usability tests that would help us know what problems the actual users find when using the desktop.

We need an infrastructure to do this kind of studies periodically. A web poll with free-form questions would work well.

It is very important that we nurture our contacts within the deployments! We can't simply ask them a few questions each year and then disappear. We need to make sure that our fixes address their problems; the deployments themselves may even be able to "lend" their technical people to GNOME to share their code and solutions.

Finally, we need to find all the deployments that we don't know about. GNOME is used in many places, and we would like to know who needs our help.

Acknowledgments

Federico Mena-Quintero compiled the replies and put together a spreadsheet with the raw data.

John Williams, a statistician, very kindly dissected the spreadsheet data with his ninja skills. Federico only knows how to use =SUM() after all, which is not enough for this kind of study.

Many thanks to the people who replied to the questions for deployments! Without your input and your users, we would be writing software without a purpose.

Finally, a big cheer and many thanks to all the GNOME hackers who care about their users and write awesome software!

A. References to the original data