Pulling the plug on dpkg and rpm

Why are we here?

• Free Software is fun
• A lot of people use our software, for work and play.
• "our software": General purpose FOSS systems: Debian/Fedora/Gentoo etc.

But we have embarassing flaws

• Upgrades can break your system, not atomic
• Too easy to regress, and then no path to revert
• Upgrading Firefox via dpkg/rpm breaks the running instance

Designers building GNOME is bad

• The original problem I was trying to solve: Designers having to build GNOME from source on their machine in order to see what they're designing.

Increment all the versions! (forever)

• RPM and dpkg both define "upgrade" to be increase in version number
• Hacking around this (RPM epoch, Debian ~really) are too painful and ugly
• I believe this semantic deeply influcences our culture; kernel often reverts commits, rare to see discussed in "distributions"
• Oh also, if it breaks, you're screwed.

You don't actually run Debian/Fedora/Gentoo...

• You run a filesystem tree resulting from installation and upgrade of a set of packages.

OSTree: "git for operating system binaries"

• Research project that has borne a lot of fruit
• OSTree/VCS - git for hardlinks, plus management of /etc, GRUB/initramfs
• OSTree/Ostbuild - take input set of git repositores, continually feeds multiple trees
• Designed to deploy into /ostree
• Does not affect your distribution - just takes up disk space, attempts to minimize most costly attribute - your time

Boot into chroots, with bind mounts outside

• Rather than operating system living in /, you boot into a chroot
• Bind mounts set up:
• /home -> /home
• /var -> /ostree/var
• /etc -> /ostree/deploy/trees/X-etc
• /usr (read-only)

Demo

$ ls -al /ostree/
total 40
drwxr-xr-x.  8 root root 4096 Aug 25 13:26 .
dr-xr-xr-x. 25 root root 4096 Aug 26 17:36 ..
lrwxrwxrwx.  1 root root  100 Aug 25 13:26 current -> deploy/trees/gnomeos-3.6-i686-devel-b49228c929cfa31a30d8f98191abfadced58b3836ea8a85989f877174e5e2b04
drwxr-xr-x.  3 root root 4096 Aug 25 13:24 deploy
drwxr-xr-x.  8 root root 4096 Jul  8 10:52 modules
lrwxrwxrwx.  1 root root  100 Aug 24 16:05 previous -> deploy/trees/gnomeos-3.6-i686-devel-b49228c929cfa31a30d8f98191abfadced58b3836ea8a85989f877174e5e2b04
drwxr-xr-x.  7 root root 4096 Aug 15 17:37 repo
drwxr-xr-x.  7 root root 4096 Jun  7 18:20 var
$ ls -al /ostree/deploy/trees/
total 16
drwxr-xr-x.  4 root root 4096 Aug 25 13:26 .
drwxr-xr-x.  3 root root 4096 Aug 25 13:24 ..
drwxr-sr-x. 18 root root 4096 Aug 25 13:24 gnomeos-3.6-i686-devel-b49228c929cfa31a30d8f98191abfadced58b3836ea8a85989f877174e5e2b04
drwxr-xr-x. 45 root root 4096 Aug 25 13:26 gnomeos-3.6-i686-devel-b49228c929cfa31a30d8f98191abfadced58b3836ea8a85989f877174e5e2b04-etc

Benefits

• Fully atomic upgrades - Check out new tree, swap symbolic link
• Bisect across operating system builds; "my sound broke in the last week"
• Ensure that everyone is running same set of bits that have been integration tested

Downloading new revisions

# cd /ostree
# ostree-pull --repo=repo gnome --depth=3
Fetching http://ostree.gnome.org/repo/refs/heads/trees/gnomeos-3.6-i686-runtime
Fetching http://ostree.gnome.org/repo/refs/heads/trees/gnomeos-3.6-i686-devel
Fetching http://ostree.gnome.org/repo/objects/55/48dbac3ed5ece0fee08936ae238bdd8c0e67d05df8d825ffc26e0861e0569b.commit
Fetching http://ostree.gnome.org/repo/objects/9f/26d6902cadd26a91d6970243e061e65a001c07d15be6c6bf9b46c670fd2ec1.dirtree
...
162 content objects to fetch
...
remote gnome/trees/gnomeos-3.6-i686-runtime is now 5548dbac3ed5ece0fee08936ae238bdd8c0e67d05df8d825ffc26e0861e0569b
remote gnome/trees/gnomeos-3.6-i686-devel is now aa07f4674d47e0e89c034a91433a95d333fe2d2d28adddcb4b2ea5731e97152e
# ostree --repo=repo diff trees/gnomeos-3.6-i686-devel
M    /contents.json
M    /usr/libexec/gdm-session-worker
M    /usr/libexec/gdm-simple-slave
M    /usr/share/locale/lt/LC_MESSAGES/gnome-panel-3.0.mo
#

Upgrading - deploying a new revision

# ls -al /ostree/trees/gnomeos-3.6-i686-devel
 lrwxrwxrwx. 1 root root 87 Aug 29 16:50 /ostree/trees/gnomeos-3.6-i686-devel -> gnomeos-3.6-i686-devel-aa07f4674d47e0e89c034a91433a95d333fe2d2d28adddcb4b2ea5731e97152e
# ostadmin deploy trees/gnomeos-3.6-i686-devel
ostree-run-triggers: Starting: 0001ldconfig.trigger
ostree-run-triggers: Completed 0001ldconfig.trigger in 4.83 seconds
...
ostree-run-triggers: Starting: 0080gtk+.trigger
ostree-run-triggers: Completed 0080gtk+.trigger in 43.93 seconds
ostree-checkout: Rev trees/gnomeos-3.6-i686-devel checked out as aa07f4674d47e0e89c034a91433a95d333fe2d2d28adddcb4b2ea5731e97152e
Already have OSTree entry in grub config
/ostree/current set to trees/gnomeos-3.6-i686-devel

Configuration propagates forwards in time (git branch)

• Each tree has its own -etc
• When deploying new tree:
• Compute diff between current etc and current default etc
• Apply diff to new etc
• New feature: Can choose which etc you want at boot time

/lib/passwd

• Patched glibc/nss/nss_files/files-XXX.c to read from both /etc/passwd and /lib/passwd
• /lib/passwd contains gdm,polkitd
• /etc/passwd contains walters

Build system

• OpenEmbedded/Poky used to build initial chroot with gcc/bash/libc (currently, cross built from RHEL6)
• gobject-introspection can't cross build, therefore must switch over to same-arch build
• "ostbuild" pulls from git repositories, puts results into OSTree roots
• Two roots: gnomeos-3.6-i686-runtime, gnomeos-3.6-i686-devel, but could be as many as desired
• Entire system builds as non-root; powered by bitbake/pseudo (OE Core) and linux-user-chroot (ostbuild)

Continuous integration. Finally!

• After a GDM commit, can have binaries available within 30 seconds.
• Buildroot filesystem layout measured in seconds - like updates, just hardlinking
• Except for gtk-update-icon-cache/gconf =(

Demo

  "components": [
   		{"src": "gnome:gtk-doc-stub",
		 "component": "devel",
		 "noarch": true},

		{"src": "git:git://github.com/atgreen/libffi.git",
		 "tag": "v3.0.10"},

		{"src": "gnome:glib",
                 "config-opts": ["--disable-modular-tests"]},
  ...

Alternatives/Technologies

• BTRFS: Can easily snapshot /; harder to just do /usr, doesn't answer /etc, and really wants good userspace management tools (e.g. OpenSUSE Snapper)
• Conary: Like subversion; can rollback, but not atomic, and hacky
• Virtualization: Useful for developers; not realistic for default system
• NixOS: Extremely influential for me. However, not realistic to rebuild everything when libc changes.
• Other <noun> Linux and their wrappers for make/tar/wget: Pretty much all the same, most difficult problem was probably finding an as-yet-unused noun from dictionary.

Issues

• I'm not actually running the system myself actively, need to make /etc work
• No built in application story; I'll probably just build Emacs and Epiphany from source myself for now...
• To what degree should we enable third-party application developers, versus the encouragement to be FOSS that the "distribution" model provides?
• Also need a story for "system extensions" like nVidia driver; fake up an RPM provides database?

Feeding back into dpkg/rpm

• Support using OSTree underneath zypper/yum/rpm and apt-get/dpkg
• Modify RPM to have a "build serial", allow going backwards
• Modify RPM to inspect source code and Do The Right Thing (build API, distutils)
• Make spec files more declarative: "Run adduser with this name"
• Add Application tag to RPMs - don't have %post scripts or setuid binaries, can't conflict/obsolete, gets installed to globally-unique dir, have exactly one .desktop file, etc. Basically zeroinstall type integration.

Quick recap

• You can download and try builds of GNOME minutes after changes are committed, with total safety and a high degree of efficiency.

Questions?