Stuff Michael Meeks is doing

This is my (in)activity log. You might like to visit Collabora Productivity a subsidiary of Collabora focusing on LibreOffice support and services for whom I work. Also if you have the time to read this sort of stuff you could enlighten yourself by going to Unraveling Wittgenstein's net or if you are feeling objectionable perhaps here. Failing that, there are all manner of interesting things to read on the LibreOffice Planet news feed.

Older items: 2021: ( J F M A M J J A S O N D ), 2019, 2019, 2018, 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999, legacy html


2015-12-31 Thursday.

2015-12-30 Wednesday.

2015-12-29 Tuesday.

2015-12-28 Monday.

2015-12-27 Sunday.

2015-12-26 Saturday.

2015-12-25 Friday.

2015-12-24 Thursday.

2015-12-23 Wednesday.

2015-12-22 Tuesday.

2015-12-21 Monday.

2015-12-20 Sunday.

2015-12-19 Saturday.

2015-12-18 Friday.

2015-12-17 Thursday.

2015-12-16 Wednesday.

Collabora Online Developer Edition (CODE)

Today we release an easy way to get stuck into playing with LibreOffice online alongside ownCloud - please do checkout the CODE page and have a play. The purpose of my blog here is to credit the people involved in the development so far: currently all of the core work is by Collabora - that's something we hope that making it easier to get involved will improve.

Some background

The history of the code we're showing today goes a long way back, but the specific pieces necessary for this work are rather newer. The initial work done around adapting LibreOfficeKit to implement tiled viewing of documents (for Android) whereby LibreOffice provides a simple API to produce pixels from abstract document regions was implemented by a Smoose and Collabora partnership.

Follow-on work to get basic editing functionality exposed via the LibreOfficeKit API was funded by The Document Foundation - which enabled basic editing on Android.

Then, after a hiatus, IceWarp weighed in and invested in the implementation of LibreOffice Online with Collabora; huge cudos to them. Its great to work alongside them.

Naturally a fundamental piece of bringing LibreOffice to the web, is LibreOffice itself. Many thanks to all those who have contributed to making it the success it is today.

Hard work from many Collaborans

Having credited those funding and investing in the work alongside Collabora, its well worth calling out the engineers that put their sweat and tears (hopefully no blood) into the work. Splitting down who did what here is somewhat complicated, since there are four main pieces, and some of the work overlaps several of these, but lets try; unless stated all the work is from Collabora:

Web Services Daemon

LibreOfficeKit

This API is provided by the LibreOffice core, and is used by both Android, LibreOffice Online, and GNOME Documents to do both tiled rendering and adding editing support.

JavaScript / leaflet

Shared front-end / backend bits

There are some chunks of work that are sufficiently wide-ranging that they involve significant work in both the client Javascript, and the LibreOfficeKit implementation, as well as touching the Web Services Daemon.

OwnCloud integration

Thanks again to all of our engineers who made this happen, and to all those whose work we built it upon.

Conclusion

CODE, as is true of all Free Software, builds heavily on previous work done by many. Having said that, Collabora through the skill and dedication of our engineers, have clearly contributed huge amounts of expertise and effort to architect and implement this exciting new development in Cloud based productivity. We thank all our partners and customers who have worked alongside us to enable that. To play with it now and get involved please checkout the CODE.

2015-12-15 Tuesday.

2015-12-14 Monday.

2015-12-13 Sunday.

2015-12-12 Saturday.

2015-12-11 Friday.

2015-12-10 Thursday.

2015-12-09 Wednesday.

2015-12-08 Tuesday.

2015-12-07 Monday.

2015-12-06 Sunday.

2015-12-05 Saturday.

2015-12-04 Friday.

2015-12-03 Thursday.

2015-12-02 Wednesday.

2015-12-01 Tuesday.

2015-11-30 Monday.

2015-11-29 Sunday.

2015-11-28 Saturday.

2015-11-27 Friday.

2015-11-26 Thursday

2015-11-25 Wednesday

2015-11-24 Tuesday

2015-11-23 Monday

2015-11-22 Sunday

2015-11-21 Saturday

2015-11-20 Friday

2015-11-19 Thursday

2015-11-18 Wednesday

2015-11-17 Tuesday

2015-11-16 Monday

2015-11-15 Sunday

2015-11-14 Saturday

2015-11-13 Friday

2015-11-12 Thursday

2015-11-11 Wednesday.

2015-11-10 Tuesday.

2015-11-09 Monday.

2015-11-08 Sunday.

2015-11-07 Saturday.

2015-11-06 Friday.

2015-11-05 Thursday.

2015-11-04 Wednesday.

2015-11-03 Tuesday.

2015-11-02 Monday.

2015-11-01 Sunday.

2015-10-31 Saturday.

2015-10-30 Friday.

2015-10-29 Thursday.

2015-10-28 Wednesday.

2015-10-27 Tuesday.

2015-10-26 Monday.

2015-10-25 Sunday.

2015-10-24 Saturday.

2015-10-23 Friday.

2015-10-22 Thursday.

2015-10-21 Wednesday.

2015-10-20 Tuesday.

2015-10-19 Monday.

2015-10-18 Sunday.

2015-10-17 Saturday.

2015-10-16 Friday.

2015-10-15 Thursday.

2015-10-14 Wednesday.

2015-10-13 Tuesday.

2015-10-12 Monday.

2015-10-11 Sunday.

2015-10-10 Saturday.

2015-10-09 Friday.

2015-10-08 Thursday.

2015-10-07 Wednesday.

2015-10-06 Tuesday.

2015-10-05 Monday.

2015-10-04 Sunday.

2015-10-03 Saturday.

2015-10-02 Friday.

2015-10-01 Thursday.

2015-09-30 Wednesday.

2015-09-29 Tuesday.

2015-09-28 Monday.

2015-09-27 Sunday.

2015-09-26 Saturday.

2015-09-25 Friday.

2015-09-24 Thursday.

2015-09-23 Wednesday.

2015-09-22 Tuesday.

2015-09-21 Monday.

2015-09-20 Sunday.

2015-09-19 Saturday.

2015-09-18 Friday.

2015-09-17 Thursday.

2015-09-16 Wednesday.

2015-09-15 Tuesday.

2015-09-14 Monday.

2015-09-13 Sunday.

2015-09-12 Saturday.

2015-09-11 Friday.

2015-09-10 Thursday.

2015-09-09 Wednesday.

2015-09-08 Tuesday.

2015-09-07 Monday.

2015-09-06 Sunday.

2015-09-05 Saturday.

2015-09-04 Friday.

2015-09-03 Thursday.

2015-09-02 Wednesday.

2015-09-01 Tuesday.

2015-08-31 Monday.

2015-08-30 Sunday.

2015-08-29 Saturday.

2015-08-28 Friday.

2015-08-27 Thursday.

2015-08-26 Wednesday.

2015-08-25 Tuesday.

2015-08-24 Monday.

2015-08-23 Sunday.

2015-08-22 Saturday.

2015-08-21 Friday.

2015-08-20 Thursday.

2015-08-19 Wednesday.

2015-08-18 Tuesday.

2015-08-17 Monday.

2015-08-16 Sunday.

2015-08-15 Saturday.

2015-08-14 Friday.

2015-08-13 Thursday.

2015-08-12 Wednesday.

2015-08-11 Tuesday.

2015-08-10 Monday.

2015-08-09 Sunday.

2015-08-08 Saturday.

2015-08-07 Friday.

2015-08-06 Thursday.

2015-08-05 Wednesday.

LibreOffice under the hood: progress to 5.0

Today we release LibreOffice 5.0.0, a new foundation for ongoing work over the next months and years. It also has a fine suite of new features for people to enjoy - you can read and enjoy all the great news about the user visible features from so many great hackers, but there are, as always, many contributors whose work is primarily behind the scenes, and a lot of work that is more technical than user-facing. That work is, of course, still vitally important to the project. It can be hard to extract those from around eleven thousand commits since LibreOffice 4.4 was branched, so let me try to expand:

VCL - Toolkit Improvements

One of the largest areas of work in LibreOffice 5.0 is in the VCL toolkit, the graphics toolkit LibreOffice uses for all the widgets and rendering. 5.0 means modernizing and improving several aspects of it and bringing them into line with other cross-platform toolkits.

Mainloop / idle handling

This is a rather major change that landed in 5.0, and is a vital under-pinning to the ongoing attempts to make VCL and LibreOffice more efficient and performant, thanks to Jennifer Liebel and Tobias Madl (interview). The essential problem with our previous approach to deciding what to do next in LibreOffice (eg. should I do some more word-counting ? or process some deferred window re-sizing work ? or re-paint a windows' contents ?) was decided by a rather arbitrary set of random number of millisecond timeouts eg. 30ms for a re-paint, 50ms for a re-size - which was not only race prone, but also horribly inefficient - there being no solid basis to these pseudo-random numbers.

Thankfully in LibreOffice 5.0 we have a new 'idle' concept that prioritizes tasks we want to get completed and allows them to be executed in order at top speed. This combined with Jan Holesovsky (Collabora)'s work to ensure we can queue sub 10ms timeouts on Windows means we finally have a reasonably useful mainloop.

This has also helped us to find some power-draining bad behavior that was previously less visible - since frequently executed (say every 30ms) shortish tasks that wastefully woke the CPU without making any progress, now cause a 100% CPU spike - and can be addressed. Thanks to Ashod Nakashian for attacking several of these.

Lifecycle re-work (VclPtr)

For much of its lifetime, VCL widget lifecycle was a bit of a mystery, even to VCL itself. Widgets could be heap allocated, stack allocated, or be members of other widgets. If heap allocated they could be wrapped in various flavours of shared pointers. As such predicting when a widgets would be destroyed, and/or following its lifecycle across the code was non-trivial. Inside VCL we often used dog-tags: special listeners that would turn null when an object was destroyed to try to avoid referencing an object involved in several back-to-back callbacks. Unfortunately this support was rather incomplete, and lots of code would end up deferring deleting heap allocated widgets until idle in an attempt to avoid problems.

In an attempt to solve all of this mess, we now have a single smart pointer type: VclPtr to reference-count all Window (and OutputDevice) sub-classes, which are now always heap allocated. This gives a consistent lifecycle mechanism, which is even documented. We moved to a 'dispose' mechanism to break reference-cycles, replacing the previous explicit or implicit 'delete' mechanism, and have made lots of methods safe to call even on disposed widgets. This should, in the end provide predictable lifecycle, and much less fragile destruction code paths making it easier to safely re-factor code. In the meantime we continue to iron out problems, thanks to Noel Grandin (Peralex) for his invaluable help to me with this work, and Caolan McNamara (RedHat) and Julien Nabet among others for helping to fixup some of the aftermath. It is hoped that (ultimately) nearly all long-lived VCL types will use a similar lifecycle mechanism. This work was made possible by Caolan's huge re-factor to use VclBuilder for all dialogs.

Modern rendering: RenderContext

A bold attempt to switch the code-base from immediate rendering to deferred rendering was initiated. LibreOffice previously rendered what is seen on the screen in one of two ways - either immediately: ie. when you press an 'A' it tries to nail the pixels for 'A' immediately to the screen; or - via. a very deferred (30+ms delay) idle rendering = callback.

This situation is really non-ideal for modern rendering hardware and APIs - where we want to ensure the scene is fully and perfectly painted as a whole before showing it on-screen. Happily with the new idle handling work, there is no longer a hard-coded delay before deferred rendering can occur; so we started the task of removing immediate rendering, and replacing it with deferred rendering. This means replacing explicit rendering calls with area invalidation to queue this area for later re-rendering. In many cases this can remove any visible flickering and other intermediate rendering artifacts as the UI refreshes. Many thanks to Tomaž Vajngerl (Collabora), Miklos Vajna (Collabora), with help and fixing from Krisztian Pinter, Noel Grandin (Peralex), Jan Holesovsky (Collabora), Caolán McNamara (RedHat), Laszlo Nemeth (Collabora)

Gtk3 backend: Wayland

An very rough, initial gtk3 port was hacked together long ago by yours truly to prototype LibreOffice online via gdk-broadway. However thanks to Caolán McNamara (RedHat) who has done the 80% of the hard work to finish this, giving us a polished and complete VCL backend for gtk3. His blog entry focuses on the importance of this for running LibreOffice natively under wayland - the previous gtk2 backend was heavily tied to raw X11 rendering, while the new gtk3 backend uses CPU rendering via the VCL headless backend, of which more below.

OpenGL rendering improvements

The OpenGL rendering backend also significantly matured in this version, allowing us to talk directly to the hardware to accelerate much of our rendering, with large numbers of bug fixes and improvements. Many thanks to Louis-Francis Ratté-Boulianne (Collabora), Markus Mohrhard, Luboš Luňák (Collabora), Tomaž Vajngerl (Collabora), Jan Holesovsky (Collabora), Tor Lillqvist (Collabora), Chris Sherlock and others . It is hoped that with the ongoing bug-fixing here, that this can be enabled by default as a late feature, after suitable review, for LibreOffice 5.0.1 or at the outside 5.0.2.

LibreOfficeKit Improvements

LibreOfficeKit provides an easy way to re-use the rendering, file-format and now editing core from LibreOffice. In the last six months it has gone from being primaily useful for file format conversion, to being the foundation of LibreOffice on Android, and Online.

Headless rendering improvements

LibreOfficeKit re-uses our headless rendering backend, which allows us to render documents without underlying OS assistance, ie. without X11, Windows, OS/X etc. A number of performance and other rendering fixes were implemented here as part of the gtk3 and online work (headless rendering is also used on Android while our GL backend is maturing for that platform). Thanks to Caolán McNamara (RedHat) and Michael Meeks (Collabora).

Android editing extensions

Android editing builds on top of the LibreOfficeKit editing features, and provides the user with the Android equivalent of the gtktiledviewer feature list, like native cursor, text and graphic selection, resizing and more. Thanks to The Document Foundation & their generous donors these significant API extensions and core work are thanks to Miklos Vajna, Tor Lillqvist, Andrzej Hunt, Siqi Liu, Mihai Varga, Tomaž Vajngerl and Jan Holesovsky all of Collabora, as well as work from Pranav Kant (GSOC), and cleanups from Stephan Bergmann (RedHat)..

LibreOffice Online bits

LibreOfficeKit (alongside an adapted leaflet) is the basis for the new work targetting LibreOffice at the Cloud, checkout the code and a presentation. Huge amounts of tangled heavy lifting here were done thanks to: Tor Lillqvist, Mihai Varga, Jan Holesovsky, Henry Castro and Miklos Vajna, all of Collabora. With thanks to IceWarp for funding this work.

Conversion performance improvements

LibreOfficeKit provides a nice simple, clean API for loading and saving (ie. converting) documents. Thanks to Laszlo Nemeth (Collabora) and Mihai Varga (Collabora) we now have a new filter attribute: SkipImages to allow a significant acceleration for the use-case of converting any file type to HTML. This is really useful for re-using the wide range of LibreOffice filters to do document text indexing - giving a very significant speedup for large and complex documents. Another vital win here was to avoid doing an accurate word-count before export (for document statistics). Document conversion to text with this option should be significantly quicker for certain documents.

Build / platform improvements

Clawing back compilation time

With increasing template use in headers, compile times have taken a turn for the slower, thanks to Michael Stahl (Red Hat) who created a nice script bin/includebloat script to locate the largest and most problematic headers to be removed. As an example dropping boost/utility.hpp from several places removes ~830Mb of boost/preprocessor/seq/fold_left.hpp pre-processing.

Win64 porting action

The 5.0 release debuts a Win64 build - with many thanks to David Ostrovsky (CIB) with help from Thorsten Behrens (CIB), Norbert Thiebaud, Stephan Bergmann (RedHat) and others fixing and cleaning up a number of nasty platform-specific corner-cases across the suite. While we have had many 64bit platforms for years, the Windows LLP64 model can create issues.

Code quality work

Work is ongoing around code quality in many areas, with 120 or so cppcheck fixes thanks to Caolán McNamara (RedHat), Michael Weghorn, Julien Nabet, Noel Grandin (Peralex), and others. along with the daily commits to build without any compile warnings -Werror -Wall -Wextra etc. on many platforms with thanks primarily to Tor Lillqvist (Collabora) and Caolán McNamara (Red Hat) - this category of problems however is shrinking thanks to the increasing use of CI.

Coverity at ~zero

Having hit nearly zero coverity issues Caolán McNamara (RedHat) (with some help from others) does an awesome job of keeping the count at (or nearly at) zero each week with ~360 commits this cycle. We routinely have a few new issues in each build and fix a few others, the total being currently two issues (of 6+ million lines analyzed). Hopefully keeping the numbers at zero is a reasonably achiveable goal:

Graph of coverity static checking issues
PVS-Studio

The company OOO "Program Verification Systems" develops the PVS-Studio static analysis tool and made results of a one-time analysis run available to LibreOffice developers. Dozens of reported issues were fixed by Caolán McNamara (RedHat), Michael Stahl (RedHat), David Tardon (RedHat), and Markus Mohrhard. You can read more about that (with cartoon) here.

Import and export testing

The new TDF donor funded crash-testing hardware combined with a significant effort from Caolán McNamara (RedHat), Michael Stahl (RedHat), Markus Mohrhard and several others we have got the number of (paranoid) assertions and/or crashes on import of our significant bugzilla document corpus (of 75k+ dodgy bug documents) down to effectively zero. It's wonderful to be able to catch commits that cause regressions here and nail them within days on master, before they have a chance to escape into the user-base.

Graph of import crash-testing results

Graph of export crash-testing results

Ongoing work here is to compile the crash-testing binaries with Address Sanitizer as well as starting to fuzz various document types and expanding the set of input file-types.

Clang plugins / checkers

We have continued to add to our clang compiler plugins; a quick git grep for 'Registration' in compilerplugins shows that we've gone from 38 to 59 in the last six months (double the growth of last release). These are used to check for all manner of nasty gotchas, and also to automatically re-write various problematic bits of code. Many are run automatically by tinderboxes to catch badness. Thanks to: Stephan Bergmann (Red Hat) and Noel Grandin (Peralex) for their hard work on these checkers this cycle.

The new plugins do all sorts of things, and usually come complete with a set of relevant fixes for the underlying code; here are some examples:

Other sets of cleanups were also clang assisted such as Noel's attack on cleaning up, making consistent and nicely scoping our enumerations. Stephan's drive to detect and remove implicit bool conversion, switching many inline methods from sal_Bool (really an unsigned char) to a true 'bool' whever possible, and several other helpful plugins.

Unit testing

We also built and executed more unit tests with LibreOffice 5.0 to avoid regressions as we change the code. Grepping for the relevant TEST and ASSERT macros we continue to grow the number of unit tests:

Graph of number of unit tests and assertions
Our ideal is that every bug that is fixed gets a unit test to stop it ever recurring. With around 800 commits, and over seventy committers to the unit tests in 5.0 it is hard to list everyone involved here, apologies for that; what follows is a sorted list of those with over 10x commits to the relevant qa/ directories: Miklos Vajna (Collabora), Markus Mohrhard, Caolán McNamara (RedHat) Stephan Bergmann (RedHat), Noel Grandin (Peralex), Michael Meeks (Collabora), Michael Stahl (RedHat), Zolnai Tamás, Tor Lillqvist (Collabora), Bjoern Michaelsen (Canonical), Eike Rathke (RedHat), Takeshi Abe, Andras Timar (Collabora), PriyankaGaikwad (Synerzip)

Windows Testing

While we have had a subset of unit tests that we run at compile time on Windows, our larger battery of make check tests has been hindered by strange thread-affine behavior on Windows related to handling various Window and event resources. Thanks to various locking, and inter- thread messaging fixes from Michael Stahl (RedHat), and Stephan Bergmann (Redhat) we now have far more robust and reliable unit testing on Windows.

QA / bugzilla

One metric we watch in the ESC call is who is in the top ten in the freedesktop Weekly bug summary. Here is a list of the people who have appeared more than five times in the weekly list of top bug closers in order of frequency of appearance: Adolfo Jayme, Beluga, Caolán McNamara (RedHat), raal, Julien Nabet, Jean-Baptiste Faure, Markus Mohrhard, m.a.riosv, Gordo, V Stuart Foote, Eike Rathke (RedHat), Andras Timar (Collabora), Alex Thurgood, Yousuf (Jay) Philips, Miklos Vajna (Collabora), Joel Madero, Cor Nouws, Michael Stahl (RedHat), Michael Meeks (Collabora), Matthew Francis, David Tardon (Redhat), tommy27, Timur, Robinson Tryon (qubit) (TDF). And thanks to the many others that helped to close and triage so many bugs for this release.

Jenkins / CI

Thanks to Norbert Thiebaud - we now have some rather excellent Jenkins / CI integration with gerrit, to allow us to test-build all incoming patches across our three major platforms. Using CI to test patches before pushing them to master has become another valuable tool to increase the quality of master (and thus its accessibility to casual builders), and to allow those without access to Windows & Mac devices to check their code builds there. Thanks to ByteMark and TDF donors we hope to have even more, fast hardware to throw at the CI build farm soon making this an even more attractive route to test submitted code. With over 25,000 builds from 13 build slaves since the beginning of the year (which compares favourably with the around 11,000 commits, it is hoped that with enough hardware we can compile and run tests vs. all incoming commits in future without introducing excessive latency.

Also for the next development cycle we have enabled tests beyond those run during compile. We enable a slew of extra assertions in a dbgutil build and run make check at least on Linux to apply a much larger set of extra tests to each individual commit.

Expanded bibisect

In this cycle we expanded the great Bi(nary)Bisect(ion) repositories - which contain thousands of compressed pre-built binaries to allow end-users to quickly ascertain almost down to a single commit that introduced a regression long after the date - to include Mac and Windows builds for the 5.0 epoch (ie. the range from the 4.4 branch to 5.0 branching. The 5.1 epoch is being built and refreshed reasonably regularly. Many thanks to Norbert Thiebaud, Matthew Jay Francis & Robinson Tryon (qubit) (TDF)

Code cleanup

Code that is dirty should be cleaned up - so we did a lot of that left & right:

Upgrading to (a) C++11 subset

In the 5.0 release we started to move more aggressively to the subset of C++11 we can now use with our updated compiler baselined. Features such as variadic templates, simpler initializations, and more. Work also involved removing deprecated std:: functions such as std::ptr_fun using std::any_of & std::none_of and other newer constructs such as auto. Thanks goes to many hackers cleaning the code including Stephan Bergmann (RedHat), Takeshi Abe, Nathan Yee, Bjoern Michaelsen (Canonical) and others.

Framework Cleanup

Thanks to Maxim Monastirsky we saved many hundreds lines of duplicate code from the framework, by creating nice generic controllers that could be controlled via small, clean XML configuration descriptions - great to see such cleanups.

Expanding integer id types

A number of legacy structures in LibreOffice have used 16bit indicees, and stored / serialied these to various structures for many years. This can cause problems with very large mail merged - such as those in-use at Munich City. Thanks to Katarina Behrens (CIB) - Writer in 5.0 allows more than 64k of: Page Descriptions, Sections and Style Names.

Ongoing German Comment redux

We continued to make progress, but somehow the last ~5000 lines of comment persistently appear to defy translation. Answers by E-mail postcard from German speakers much appreciated. Many thanks to: Michael Weghorn, Michael Jaumann (Munich), Daniel Sikeler (Munich), Albert Thuswaldner, Christian M. Heller, Philipp Weissenbacher. There are now only the following eight modules left to do: include, reportdesign, rsc, sc, sfx2, stoc, svx, sw

Graph of remaining lines of German comment to translate
std:: containers

A systematic set of improvements to our usage of the std:: containers has been going on through the code. Things like avoiding inheritance from std::vector, changing std::deque to std::vector and starting to use the newer C++ constructs for iteration like for (auto& it : aTheContainer) { ... }. There are many people to credit here, thanks to Stephan Bergmann (Red Hat), Takeshi Abe, Tor Lillqvist (Collabora), Caolán McNamara (Red Hat), Michaël Lefèvre, and many others.

Writer

Thanks to Bjoern Michaelsen (Canonical) we have had a few key, long desired writer cleanups in 5.0. These include:

writerfilter's resourcemodel

The resourcemodel building block of writerfilter (that handles Writer’s DOCX and RTF import in LibreOffice) was basically a bucket of old and unused stuff. The few still needed pieces from it are now moved into the relevant mapper/tokenizer/filter parts, and the rest is now removed. You can read more detail thanks to Miklos Vajna (Collabora).

Other wins

We had a number of other wins that are somewhat difficult to categorize, but well worth noting:

OOXML vs. MS Office 2007

MS Office 2007 has an unhelpfully different set of default values for many of its attributes - ie. the same XML (with an attribute ommitted) can produce different results in Office 2007 and later versions. Clearly this is a little irritating. Thanks to Markus Mohrhard for adding some infrastructure (and a set of fixes) for known problematic attributes in this regard. This should improve our interoperability with the zoo of documents out there.

Android - file-system abstraction

Thanks to TDF's donors and Jacobo Aragunde Pérez (Igalia) we implemented an abstract file-system API for Android - to allow arbitrary file-system backends to be plugged in (in a separate thread). An example OwnCloud backend was implemented to show-case this.

Base bits

Thanks to Matthew Nicholls we removed a couple of thousand lines of redundant wrappers in svx's dbtoolsclient - which was duplicated elsewhere in connectivity. Great to see this much cruft leave the code-base.

Getting involved

I hope you get the idea that more developers continue to find a home at LibreOffice and work together to complete some rather significant work both under the hood, and also on the surface. If you want to get involved there are plenty of great people to meet and work alongside. As you can see individuals make a huge impact to the diversity of LibreOffice (the colour legends on the right should be read left to right, top to bottom, which maps to top down in the chart):

Graph showing individual code committers per month

And also in terms of diversity of code commits, we love to see the unaffiliated volunteers contribution by volume, though clearly the volume and balance changes with the season, release cycle, and volunteers vacation / business plans:

Graph of number of commits per month by affiliation

Naturally we maintain a list of small, bite-sized tasks which you can use to get involved at our Easy Hacks page, with simple build / setup instructions. It is extremely easy to build LibreOffice, each easy-hack should have code pointers and be a nicely self contained task that is easy to solve. In addition some of them are really nice-to-have features or performance improvements. Please do consider getting stuck in with something.

Another thing that really helps is running pre-release builds and reporting bugs just grab and install a pre-release and you're ready to contribute alongside the rest of the development team.

Conclusion

LibreOffice 5.0 is a great new foundation for building the next series of releases which will incrementally improve not only features, but also the foundation of the Free Software office suite. It is of course not perfect yet, this is the first in a long series of monthly 5.0.x releases, and six monthly 5.x releases which will bring a stream of bug fixes and quality improvements over the next months and years.

I hope you enjoy LibreOffice 5.0.0, thanks for reading, don't forget to checkout the user visible feature page and thank you for supporting LibreOffice.

Raw data for many of the above graphs is available.

2015-08-04 Tuesday.

2015-08-03 Monday.

2015-08-02 Sunday.

2015-08-01 Saturday.

2015-07-31 Friday.

2015-07-30 Thursday.

2015-07-29 Wednesday.

2015-07-28 Tuesday.

2015-07-27 Monday.

2015-07-26 Sunday.

2015-07-25 Saturday.

2015-07-24 Friday.

2015-07-23 Thursday.

2015-07-22 Wednesday.

2015-07-21 Tuesday.

2015-07-20 Monday.

2015-07-19 Sunday.

2015-07-18 Saturday.

2015-07-17 Friday.

2015-07-16 Thursday.

2015-07-15 Wednesday.

2015-07-14 Tuesday.

2015-07-13 Monday.

2015-07-12 Sunday.

2015-07-11 Saturday.

2015-07-10 Friday.

2015-07-09 Thursday.

2015-07-08 Wednesday.

2015-07-07 Tuesday.

2015-07-06 Monday.

2015-07-05 Sunday.

2015-07-04 Saturday.

2015-07-03 Friday.

2015-07-02 Thursday.

2015-07-01 Wednesday.

2015-06-30 Tuesday.

2015-06-29 Monday.

2015-06-28 Sunday.

2015-06-27 Saturday.

2015-06-26 Friday.

2015-06-25 Thursday.

2015-06-24 Wednesday.

2015-06-23 Tuesday.

2015-06-22 Monday.

2015-06-21 Sunday.

2015-06-20 Saturday.

  • Up lateish; fixed a LibreOffice bug, pushed last night's fix; breakfast. Carpet & floor-boards up to see if there is a gas-connection near the fireplace, eventually located one. Out to buy and fit a new toilet seat - now with magic dash-pot to retard closing: hopefully more robust than the previous version.
  • Emily over in the afternoon, watched Heaven is for real - spiritual candy-floss; enjoyable - while building your life on something more rocky. Late-night bug triage / chasing.

2015-06-19 Friday.

  • Into Cambridge, caught up with Tracie & Lucy. Interesting customer meeting; picnic lunch in the Botanical Gardens with the office guys; sync. with Niall, train home, picked up babes; more mail.
  • Watched the last Potter movie with M. Up late chasing a LibreOffice bug to its ancient roots.

2015-06-18 Thursday.

  • Breakfast in bed; and a fine crop of birthday presents: primarily pretty clothes for J. and Sci-fi books to read to the kids with a crate of craft ciders. Mail chew, etc. out for a walk in the countryside with J. - oddly and encouragingly child-less.
  • Partner call, project update call, ESC call, project update call.

2015-06-17 Wednesday.

  • Built code overnight, checked out, chewed mail, reviewed bugs etc. on the train. Poked away at bugs, text review etc.
  • Plane, train - dug into some basebmp evilness with mask bitmaps, a horrifying un-readble / debugging Vigra template morass.
  • Lovely to be home with the wife & family again.

2015-06-16 Tuesday.

  • To the venue, an interesting overview of cloud bits. I have a growing suspicion that I know where some of the sysadmins went; they transmogrified into Cloud Infrastructure Engineers.
  • Quested for a Euro-compatible power socket fruitlessly, back to the hotel for a positive customer call. Prepped slideware and demos: worryingly robust - can this really be the bleeding edge ? Back to UniS - gave a talk, chatted to many people afterwards.
  • Slides available as hybrid-PDF:
    LibreOffice Online slides
  • Out for dinner with Markus W - to celebrate the PDF signing crowd-funding campaign. Up late with some interesting guys.

2015-06-15 Monday.

  • Up early, off to the Swiss Open Cloud Day (tomorrow) in Bern, somewhat unfortunate acronym (OCD) - should be dead fun. Train met a friend, train met an SI.
  • My 2nd AirBnB booking for Bern dropped out - hmm, lets hope they have comfortable park-benches. Met a Linda previously with SIL in Africa on the plane; finally found a place to stay; great mobile data connectivity on the train to Bern, and extremely beautiful scenery.
  • Found a hotel somehow; mail chew; slide creation, bed.

2015-06-14 Sunday.

  • Excited present unwrapping with E. - lots of fun lego bits, got 1kg of mixed 2nd hand lego; will that reduce the resource contention just or exacerbate it ?
  • Off to NCC, sermon on the choice of the next sermon series, good to catch up with lots of people afterwards. Lamb roast lunch.
  • David over in the afternoon, good to chat - kindly donated several original female lego-people to the cause. Much slugging, tea, stories, bed early.

2015-06-13 Saturday.

  • E's seventh birthday party - helped make egg & sandwiches etc. prep. party food; morning / lunch-party - played garden games in the mixed weather, water-pistol fight towards the end.
  • Slugged, watched movie, bit of hackery, patch review, bug triage etc. bed early, watched a movie about a writer living in a castle.

2015-06-12 Friday.

  • Interesting partner meetings; dug into a particularly interesting idle handler bug with Miklos - seems writer has always had a spell-check timer that failed to stop triggering - now we execute those bits far faster, we can get a 100% busy loop in some cases.
  • Bruce & Anne over for lunch - good to catch up with them.

2015-06-11 Thursday.

  • Travel to London for a partner meeting; great to see some old friends, albeit far too briefly. Presented, demo'd, train home, unwound a silly problem with the last warning output from Noel's newest VclPtr plugin and pushed it to Jenkins for some CI goodness.

2015-06-10 Wednesday.

  • Beat back the mail flow, board call, partner call.

2015-06-09 Tuesday.

  • Mail chew, partner call; built ESC agenda, stats. etc. More partner calls.

2015-06-08 Monday.

  • 1:1 calls variously, lunch, team meeting. Poked at misc. bugs and built some better ESC stats scripting.

2015-06-07 Sunday.

  • Up lateish; off to a Church bash on Studlands Park, played Violin in the band; free hot-dogs, free cakes, free (giant) bouncy castle; various games etc. at the center of the local community: fun, and lovely weather for it.
  • Home, slugging in the garden, reading & relaxing; good, Yes Minister with older babes in the evening.

2015-06-06 Saturday.

  • Up earlyish, off to Wicksteed Park - lots of excitement, particularly from E. - her first go on the roller-coaster. Enjoyed a number of rides, and a go-cart race between H. J and myself driving the rest of the family.
  • Alan & Claire arrived; caught up with them - played on various bits - swinging pirate ships, dodgems etc. much fun had by all; a fine lunch on the green, talked over life enjoyed the company.
  • More rides; bid 'bye eventually; fish & chips in the car on the way home, managed to survive a lot of standing up somehow today. Put babes to bed. Found that Jenkins had passed the patch I'd been testing during the day, pushed it. Watched of 'The Quartet' with J.

2015-06-05 Friday.

  • Mail chew; finally a bit of hackery; ran some timings on 'make check' and filed another Easy Hack to improve dbgutil performance.
  • Encourging to see Cor publish a 60 page LibreOffice vs. OpenOffice comparison - which rather beats my original (by now obsolete) from 2012, it'd be great to keep that up-to-date though it's quite a job.

2015-06-04 Thursday.

  • Up late; mail chew; sync. with Niall caught up with Mike. ESC call, posted minutes etc. Contract review.

2015-06-03 Wednesday.

  • Up early; mail chew; contract bits; customer call; sync. with Niall, more contract bits.

2015-06-02 Tuesday.

  • Up early, mail chew; elevating one leg all the time is a pain. Worked through admin tasks, tried to get to hacking. Customer call.

2015-06-01 Monday.

  • Back to work; mail chew; 1:1's with the team variously. Lunch, team call, chat with Kendy. Team call. Out to the Doctor's who said the previous doctor gave me a "practically homeopathic" antibiotic perscription, and put me on the "domestos" version instead.
  • J. meeting in the evening, worked late with the leg up.

2015-05-31 Sunday.

  • Leg still worse, missed church & slept much of the morning; read stories to babes, chatted to M&D etc. most of the day.

2015-05-30 Saturday.

  • Up lateish; set off to Ickworth House with the parents; an unexpected wool festival going on there too. Celulitius much worse in leg & spreading; off to get antibiotics with J. Lay with it elevated a lot.

2015-05-29 Friday.

  • Mail chew, closed a few bugs; interview; parents over in the evening.

2015-05-28 Thursday.

  • Up early; mail chew; call with Niall, admin; CRM bits, filed holiday; signed CP up for OIN. Chat with lfrb.

2015-05-27 Wednesday.

  • Day off - up late, into Cambridge; wandered around the Fitzwilliam Museum a bit with the babes, interesting place. Lunch in the courtyard outside in the sun. Back to the office for a partner call, then a customer call; picked by by a homebound family; TDF board call. Worked late.

2015-05-26 Tuesday.

  • Mail chew; pushed a few patches; Linux Format Column; customer call, chat with Niall, team call; more hackery.

2015-05-25 Monday.

  • Up lateish; off to Bruce & Anne's for a Warren End party. Good to meet up with Sue, Clive & boys; as well as lots of neighbours. Hacked in the car on the way back.

2015-05-24 Sunday.

  • Up late; read news, watched fascinating SpaceX GPU compute talk. J. and babes home in the afternoon; nasty cellulitis infected bite in leg - hmm. Out to Histon Baptist church for an afternoon service; pizza; bed early.

2015-05-23 Saturday.

  • Up early; into Cambridge together by train, encouraged to have Andrzej and Ben show up - and see Andrzej's nice calc work; and get Ben's GSOC bits started. Many left; out for curry with Bjoern, Thorsten, Ben & Andrzej. Back home late; hacked on some bits until even later.

2015-05-22 Friday.

  • Up early; train together to Cambridge - more hack-fest goodness much of the day; fixed a number of VclPtr issues.
  • Out punting in the evening with the lads; back for a swift drink & train home. Talked with team, bed late.

2015-05-21 Thursday.

  • Up early; breakfast & music practise with babes; breakfast with the guys; E-mail, drove into Cambridge - thanks to Doree - hackfest has food, rooms setup. Sam out to buy hubs, people started to arrive; encouraging to see such a great collection of hackers working away.
  • Out to the Cambridge Beer Festival in the evening, thanks to them for sponsoring the conference attendee's presence; fun - talked until late. Drove the team home - 7x seats full of adults - unusual.

2015-05-20 Wednesday.

  • Up early; into Cambridge - Andras & Kendy showed up, Quarterly review, lunch while working; misc. paperwork left & right. Back home for a roast meal, up late.

2015-05-19 Tuesday.

  • Mail chew; slideware production & stats generation. Built ESC stats & agenda. Sam over in the evening, followed by Niall - good to catch up with them.

2015-05-18 Monday.

  • Mail chew, document review, 1:1 calls variously. Lunch, team meeting, mail - dug at a SplitWindow regression; team meeting.
  • Fit new Dishwasher catch. Re-assembled the bath, having angle-ground offending nails off the back of the panelling.

2015-05-17 Sunday.

2015-05-16 Saturday.

2015-05-15 Friday.

2015-05-14 Thursday

2015-05-13 Wednesday

2015-05-12 Tuesday

2015-05-11 Monday

2015-05-10 Sunday

2015-05-09 Saturday

2015-05-08 Friday

2015-05-07 Thursday

2015-05-06 Wednesday

2015-05-05 Tuesday

2015-05-04 Monday

2015-05-03 Sunday

2015-05-02 Saturday

2015-05-01 Friday

2015-04-30 Thursday

2015-04-29 Wednesday

2015-04-28 Tuesday

2015-04-27 Monday

2015-04-26 Sunday

2015-04-25 Saturday

2015-04-24 Friday

2015-04-23 Thursday

2015-04-22 Wednesday

2015-04-21 Tuesday

2015-04-20 Monday

2015-04-19 Sunday

2015-04-18 Saturday

2015-04-15 Wednesday

2015-04-14 Tuesday

2015-04-13 Monday

2015-04-12 Sunday

2015-04-11 Saturday

2015-04-10 Friday

2015-04-09 Thursday

2015-04-08 Wednesday

2015-04-07 Tuesday

2015-04-06 Monday

2015-04-05 Sunday

2015-04-04 Saturday

2015-04-03 Friday

2015-04-02 Thursday

2015-04-01 Wednesday

2015-03-31 Tuesday

2015-03-30 Monday

2015-03-29 Sunday

2015-03-28 Saturday

2015-03-27 Friday

2015-03-26 Thursday

2015-03-25 Wednesday

LibreOffice On-Line & IceWarp

Today we announced a collaboration between IceWarp and Collabora to start the creation of LibreOffice On-Line, a scalable, cloud-hostable, full featured version of LibreOffice. My hope is that this has a huge and positive impact for the Free Software community, the business ecosystem, personal privacy, and more. Indeed, this is really one of the last big missing pieces that needs solving (alongside the Android version which is well underway). But wait - this post is supposed to be technical; lets get back to the code.

A prototype - with promise

At the beginning of the LibreOffice project, I created (for our first Paris Conference) a prototype of LibreOffice On-Line using Alex Laarson's (awesome) GTK+ Broadway - you can still see videos of that around the place. Great as the Broadway approach is (it provides essentially a simple Virtual Desktop model into your browser), the prototype taught us several important things which we plan to get right in LibreOffice On-Line:

Having said all this, Broadway was a great basis to prove the feasibility of the concept - and we re-use the underlying concepts; in particular the use of web sockets to provide the low-latency interactions we need. Broadway also worked surprisingly well from eg. a nearby Amazon cloud datacentre. Similarly having full-fidelity rendering - is a very attractive proposition, independent of the fonts, or setup of the client.

An improved approach

Caching document views

One of the key realisations behind LibreOffice On-Line is that much of document editing is not the modification itself; a rather large proportion of time is spent reading, reviewing, and browsing documents. Thus by exposing the workings of document rendering to pixels squares (tiles) via LibreOfficeKit we can cache large chunks of the document content both on the server, and in the client's browser. As the users read though a document, or re-visit it, there is no need to communicate at all with the server, or even (after an initial rendering run) to have a LibreOfficeKit instance around there either.

Thus in this mode, the ability of the browser's Javascript to understand things about the document itself allows us to move much more of the pan/zoom reading goodness into your client. That means after an inital (pre)-fetch that responsiveness can be determined more by your local hardware and it's ability to pre-cache than remote server capacity. Interestingly, this same tiled-rendering approach is used by Fennec (Firefox for Android) and LibreOffice for Android to get smooth mobile-device scrolling and rendering, so LibreOfficeKit is already well adapted for this use-case.

Browser showing hidden tile cache ready to be revealed when panning
Editing live documents

In recent times, The Document Foundation has funded, via the generosity of TDF's donors a chunk of infrastructure work to make it possible to use LibreOfficeKit to create custom document editors. There are several notable pieces of this work that intersect with this; I provide some links to the equivalent work being done for Android from Miklos Vajna:

Cursors & selection

Clearly blinking a cursor is something we can do trivially in the javascript client, rather than on the server; there are however several other interactions that benefit from browser acceleration. Text selection is a big piece of this - re-rendering text on the server simply in order to draw transparent selection rectangles over it makes very little sense - so instead we provide a list of rectangles to render in the browser. Similarly, drawing selection handles and interacting with images is something that can be handled pleasantly in the browser as well.

Keyboard / touch input

Clearly it is necessary to intercept browser keystrokes, gestures and so on, transport these over the websocket and emit them into the LibreOfficeKit core.

Tile invalidation / re-rendering

Clearly when the document changes, it is necessary to re-render and provide new tile data to the client; naturally there is an existing API for this that was put in place right at the start of the Android editing work.

Command invocation

Another piece that is required, is transporting UNO commands, and state (such as 'make it bold', or 'delete it') from the client javascript through into the LibreOfficeKit core. This is a matter again of proxying the required functionality via Javascript. The plan is to make it easy to create custom, bespoke UIs with a bit of CSS / Javascript magic wrapped around and interacting with the remote LibreOfficeKit core.

Serializing selections

Clearly as & when we decide that a user has wandered off, we can save their intermediate document, serialize the cursor location & selection - free up the resources for some other editing process. As/when they return we can then restore that with some small document load delay, as we transparently back their cached view with a live editable LibreOfficeKit instance.

What does that look like roughly ?

Of course, lots of pieces are still moving and subject to change; however here is a perhaps helpful drawing. Naturally integrating with existing storage, orchestration, and security frameworks will be important over time, contributions welcome for your pet framework:

Initial architecture sketch

The case for simple collaboration

A final, rather important part of LibreOffice On-Line; which I've left to last is that of collaborative editing.

The problem of generic, asynchronous, multi-instance / multi-device collaborative document editing is essentially horrendous. Solving even the easy problems (ie. re-ordering non-conflicting edits) is non-trivial for any large set of potentially intersecting operations. However, for this case, there are two very significant simplifying factors.

First there is a single, central instance of LibreOfficeKit rendering and providing document tiles to all clients. This significantly reduces the need to a re-order asynchronous change operation stream, it is also the case that editing conflicts should be seen as they are created.

Secondly, there is a controlled, and reasonably tractable set of extremely high-level operations based on abstract document co-ordinates - initially text selection, editing, deletion, object & shape movement, sizing, etc. which can be incrementally grown over time to extend to the core set of editing functionality.

These two simplifications, combined with managing and opportunistically strobing between users' cursor & selection contexts should allow us to provide the core of the document editing functionality.

Show me the code

The code is available as of now in gerrit's online repository. Clearly it is the Alpha not the Omega; the beginning, and not even the end of the beginning - which is a great time to get involved

Conclusion

LibreOffice On-Line is just beginning, there is a lot that remains to be done, and we appreciate help with that as we execute over the next year for IceWarp. A few words about IceWarp - having spent a rather significant amount of time pitching this work to people, and having listened to many requests for it - it is fantastic to be working with a company that can marry that great strategic sense with the resources and execution to actually start something potentially market-changing here; go IceWarp !

2015-03-24 Tuesday

2015-03-23 Monday

2015-03-22 Sunday

2015-03-21 Saturday

2015-03-20 Friday

2015-03-19 Thursday

2015-03-18 Wednesday

2015-03-17 Tuesday

2015-03-16 Monday

2015-03-15 Sunday

2015-03-14 Saturday

2015-03-13 Friday

2015-03-12 Thursday

2015-03-11 Wednesday

2015-03-10 Tuesday

2015-03-09 Monday

2015-03-08 Sunday

2015-03-07 Saturday

2015-03-06 Friday

2015-03-05 Thursday

2015-03-04 Wednesday

2015-03-03 Tuesday

2015-03-02 Monday

2015-03-01 Sunday

2015-02-28 Saturday

2015-02-27 Friday

2015-02-26 Thursday

2015-02-25 Wednesday

2015-02-24 Tuesday

2015-02-23 Monday

2015-02-22 Sunday

2015-02-21 Saturday

2015-02-20 Friday

2015-02-19 Thursday

2015-02-18 Wednesday

2015-02-17 Tuesday

2015-02-16 Monday

2015-02-15 Sunday

2015-02-14 Saturday

2015-02-13 Friday

2015-02-12 Thursday

2015-02-11 Wednesday

2015-02-10 Tuesday

2015-02-09 Monday

2015-02-08 Sunday

2015-02-07 Saturday

2015-02-06 Friday

2015-02-05 Thursday

2015-02-04 Wednesday

2015-02-03 Tuesday

2015-02-02 Monday

2015-02-01 Sunday

2015-01-31 Saturday

2015-01-30 Friday

2015-01-29 Thursday

LibreOffice under the hood: progress to 4.4.0

Today we release LibreOffice 4.4.0, packed with a load of new features for people to enjoy - you can read and enjoy all the great news about the user visible features from so many great hackers, but there are, as always, many contributors whose work is primarily behind the scenes in places that are not so easy to see. That work is, of course, still vitally important to the project. It can be hard to extract those from the over eleven thousand commits since LibreOffice 4.3 was branched, so let me expand:

Complete User Interface Dialog / Layout

The UI migration to a much improved, Glade compatible XML representation of VCL dialogs, complete with automatic layout is now almost complete (after thinking we'd done them all - Caolan discovered a lot of docking windows that need further work but these are now also migrated, all but two). Also a lot of work was put into cleaning up and tweaking the look / arrangement of dialogs. Many thanks to Caolán McNamara (Red Hat) - for his incredible work & leadership here, and to Adolfo Jayme Barrientos, Palenik Mihály (GSoC 2014), Olivier Hallot (EDX), Szymon Kłos (GSoc 2014), Rachit Gupta (GSoC 2014), Tor Lillqvist (Collabora), Jan Holesovsky (Collabora), Maxim Monastirsky, Efe Gürkan YALAMAN, Yousuf Philips and many others. Thanks also to our translators who hopefully will have much less string churn to suffer now. As a side-note the resource-compiler in rsc/ has gone on a nice diet.

Graph of progress in UI layout conversion

Initial OpenGL rendering backend

The switch to move VCL to use OpenGL for rendering is one of those things that ideally should be entirely under-the-hood, but ends up having small but important visual impact. All the work here was done by Collabora engineers, with a beefy re-factor and the initial OpenGLContext management by Markus Mohrhard, much of the rendering implemented by Louis-Francis Ratté-Boulianne with anti-aliasing, and image scaling work from Lubos Lunak, various Windows fixes and porting work from Jan Holesovsky and some bits from Chris Sherlock. During the work we also implemented a half-way decent and increasingly complete VCL demo application exercising rendering. A rational for the work with some pictures is available.

By moving to a pure OpenGL rendering model, we can accelerate those operations that badly need to taking advantage of the power and parallelism of the huge APU die-area given over to modern GPUs. Being able to interact much more directly with the underlying graphics hardware helps us to both render our image previews at high quality, and not to sacrifice scroll / zoom performance: having our cake and eating it too. We've also used some of that power to not only significantly accelerate our image rendering, but also improve its quality too from before:

Before: image down-scaling
to After (NB. if your browser scales it too you're likely to loose the sense; zoom to 1:1 and checkout eg. the top of the circular window, or other high-frequency areas.)
After: faster, better, GL image down-scaling

There is a fair bit more work to get OpenGL into a suitable state including several odd Windows / lifecycle bugs; it is necessary to export SAL_FORCEGL=1 to override the black-listing, but we hope to nail these in the 4.4.x cycle. Several ongoing and intersecting features such as the true Idle handler work from Munich's Jennifer Liebel and Tobias Madl as well as more future / pending work in-progress from Munich's Michael Jaumann (working on OpenGL canvas) and Stefan Weiberg (on OpenGL Transitions) are due in 4.5, both mentored by Thorsten Behrens (SUSE).

Mobile Viewer / LibreOfficeKit

The recently announced Android Viewer (Beta) has a number of invisible pieces included there. Particularly the improvements to LibreOfficeKit: an easy way to re-use the rendering and file-format goodness from LibreOffice from Andrzej Hunt and Kohei Yoshida (Collabora) to get Impress and Calc rendering to tiles at least to a Beta level. You can read more about the just started editing work done for TDF there too. LibreOfficeKit has also become more powerful at extracting key document meta-data from yet more of the host of file formats that LibreOffice supports - important for indexing un-structured data.

Build / platform improvements

30% faster Windows builds

With the new build system functionally completed, we've looked at the most significant problem with it: rather slow build times on Windows. An investigation and some benchmarking revealed that the usage of Cygwin make was the main cause of the slowness, and hence Michael Stahl (Red Hat) made it possible to build LO 4.4 with a Win32 native build of GNU make, cutting from-scratch build time by almost a third over stock Cygwin make, and speeding up incremental rebuilds even more.

Win64 porting action

Another major improvement is from David Ostrovsky (CIB), which is to do some significant work towards completing the native Win64 port. This we expect will ship in LibreOffice 4.5, but should significantly help eg. Java users and those with very large spreadsheets. See the Windows 64bit wiki page for more detail, thanks also to Mark Williams for some tricky UNO bridge fixing work, and to Tor Lillqvist (Collabora) who laid a lot of the initial ground-work here.

Code quality work

There has been a lot of work on code quality and improving the maintainability and cleanliness of the code. Another 59 or so commits to fix cppcheck errors are thanks to Thomas Arnhold, Julien Nabet and Simon Danner, along with the daily commits to build without any compile warnings -Werror -Wall -Wextra on many platforms with thanks primarily to Tor Lillqvist (Collabora), Caolán McNamara (Red Hat), and Thomas Arnhold.

Awesome Coverity

We have been chewing through the huge amount of analysis from the Coverity Scan, well - in particular Caolán McNamara (Red Hat) has done an awesome job here; his blog on that is typically modest.

We now have a defect density that bumps along close to 0.00, though as Coverity introduces new checks, and new code gets committed that goes up and down a little; currently 0.02 so - 2 static checking warnings per 100,000 lines. That compares extremely well with the average Open Source project which has 65 warnings per 100,000 lines.

Grokking commits with coverity in them we have 1530 fixes since LibreOffice 4.3 with the top three contributors after Caolan (1378 commits) being: Norbert Thiebaud, David Tardon (Red Hat), Miklos Vajna (Collabora).

Increasing use of asserts

In the 3.5 release we switched away from custom macros to use normal 'assert' calls to sanity check various invariants; we're doing more sanity checking left and right these days:

Graph of number of run-time assertions
Import and now export testing

Markus Mohrhard (Collabora)'s great import/export crash testing has been further expanded to cover 76,000+ problem/bug documents up from 55k last release, with a selection of odd images now also included. Another major win here was the provision by TDF (thanks to our donors) of a beefy new 64 core box to run the load/save/validate tests on. This, combined with some re-working and better parallelism of the python scripts driving that, has speeded up our test runs from five days to under one - allowing rapid diagnosis of new regressions in a much smaller range. We've also been able to do some Addresss Sanitizer runs of the document set which has resulted in a number of fixes, thanks too to Caolán McNamara (Red Hat) for some great work there.

Clang plugins / checkers

We have continued to add to our clang compiler plugins; a quick git grep for 'Registration' in compilerplugins shows that we've gone from 27 to 38 plugins in the last six months. These check all manner of nasty gotchas that people can fall into in our code. Some of these plugins are used manually but many are run by a tinderbox and some users to catch badness quickly. Thanks to: Stephan Bergmann (Red Hat) and Noel Grandin (Peralex) for their hard work on these checkers this cycle.

The plugins do all sorts of things, for example Bjoern Michaelsen (Canonical) wrote a plugin that detects deeply-nested conditionals such as these monsters. These are hard to read and a severe pain to debug through. Some of the worst offenders in sw/ have been rewritten and the plugin can easily be applied elsewhere in the codebase.

Unit testing

We also built and executed more unit tests with LibreOffice 4.3 to avoid regressions as we change the code. Grepping for the relevant TEST and ASSERT macros we continue to grow the number of unit tests:

Graph of number of unit tests and assertions
Our ideal is that every bug that is fixed gets a unit test to stop it ever recurring. With around 1000 commits, and over seventy committers to the unit tests in 4.4 it is hard to list everyone involved here, apologies for that; what follows is a sorted list of those with over 10x commits to the qa/ directories: Miklos Vajna (Collabora), Caolán McNamara (Red Hat), Kohei Yoshida (Collabora), Michael Stahl (Red Hat), Stephan Bergmann (Red Hat), Zolnai Tamás (Collabora), David Tardon (Red Hat), Noel Grandin (Peralex), Matúš Kukan (Collabora), Luboš Luňák (Collabora), Markus Mohrhard (Collabora), Tor Lillqvist (Collabora), Thomas Arnhold, Andrzej Hunt (Collabora), Eike Rathke (Red Hat), Jan Holesovsky (Collabora)

QA / bugzilla

Over the last six months the QA team has grown in size and effectiveness, doing some amazing work to bring our un-triaged bug count right down from one thousand (which we thought was good) to just over three hundred bugs. It's particularly knotty triaging some of those last bugs - with rather deeply technical, or super-hard-to-reproduce combinations lurking at the bottom: some excellent work there. It is rather hard to extract credits for confirming bugs, but the respective hero list overlaps with the non-developer / top closers listed below.

One metric we watch in the ESC call is who is in the top ten in the freedesktop Weekly bug summary. Here is a list of the people who have appeared more than five times in the weekly list of top bug closers in order of frequency of appearance: Caolán McNamara (Red Hat), Adolfo Jayme, tommy27, Julien Nabet, Jean-Baptiste Faure, Jay Philips, Urmas, Maxim Monastirsky, Beluga, raal, Michael Stahl (Red Hat), Joel Madero, ign_christian, Cor Nouws, V Stuart Foote, Eike Rathke (Red Hat), Robinson Tryon (TDF), Miklos Vajna (Collabora), Matthew Francis, foss, Sophie (TDF), Samuel Mehrbrodt, Markus Mohrhard (Collabora). And thanks to the many others that helped to close so many bugs for this release.

Bjoern Michaelsen (Canonical) also wrote up a new year QA update which is well worth reading.

Another win that should help us tweak our bugzilla to make it more user friendly and better structured is the migration from FreeDesktop infrastructure to TDF, with thanks to FreeDesktop for taking our large bugzilla load for all these years. This was completed recently - so now we file bugs at http://bugs.documentfoundation.org/. Thanks to Robinson 'colonelqubit' Tryon (TDF), and Tollef Fog Heen as well as our sysadmin team for that work. As is perhaps obvious, Robinson is working for TDF (funded by our generous donors) half-time to help improve our QA situation.

Code cleanup

Code that is dirty should be cleaned up - so we did a lot of that.

Ongoing German Comment redux

We continued to make progress, but sadly only a small amount of it on translating our last lingering German comments across the codebase into good, crisp technical English. This is a great way to get involved in LibreOffice development. Many thanks to: Philipp Weissenbacher, Christian M. Heller, Jennifer Liebel (Munich), Chris Sherlock (Collabora), Michael Jaumann (Munich), Luc Castermans, Jeroen Nijhof, Florian Reisinger and a number of others with just one commit. Further reductions in the number of false positives from bin/find-german-comments suggest that there are only ten top-level modules left containing German, nine of them worth translating: i18npool, include, reportdesign, sc, scaddins, sfx2, stoc, svx, sw

Graph of remaining lines of German comment to translate

One particularly encouraging contributor to our German Comment translation efforts was Lennart Poettering who it seems has an amusing plan afoot.

Upgrading to (some) C++11 subset

As time advances, C++ improves, with the upgrade of Visual Studio we've been able to move to a subset of C++11 (as supported by VS 2012) as a new compiler base-line. We also removed several optimization disabling workarounds for bugs in old GCC versions that don't do C++11 anyway, and hence both GCC and MSVC can now build all of LO with optimization. Thanks to Stephan Bergmann (Red Hat) for researching and driving this work.

OOXML Tokenizer cleanup

This cleanup builds on work by Miklos Vajna (Collabora) in the last release. A big chunk of our OOXML tokenizer was generated code, which is reasonable but it was generated using XSLT (which is trending below cobol). This was re-written from 4200 lines of XLST into 1300 lines of python - to produce the same output with a large increase in hack-ability. Then some optimization was done by Jan Holesovsky (Collabora for CloudOn), to reduce inefficiency in the generated output saving 2.2Mb from the 8Mb (stripped) writerfilter DSO. Great to see this sort of code cleanup, source size shrink and binary shrink at the same time. You can read more about it in Miklos' blog.

std:: containers

A systematic set of improvements to our usage of the std:: containers has been going on through the code. Things like avoiding inheritance from std::vector, changing std::deque to std::vector and starting to use the newer C++ constructs for iteration like for (auto& it : aTheContainer) { ... }. There are many people to credit here, thanks to Stephan Bergmann (Red Hat), Takeshi Abe, Tor Lillqvist (Collabora), Caolán McNamara (Red Hat), Michaël Lefèvre, and many others.

Performance improvements

Performance is one of those very hard to see things, that is nevertheless viscerally felt: "why am I still waiting ?". There are a number of rather encouraging performance improvements by different people in LibreOffice 4.4 that are worth noticing.

Autocorrect performance

For reasons that elude me, some people like to have huge auto-correct lists. These are stored as zipped XML. Daniel Sikeler (Munich) put some lovely improvements into the loading of these. In particular he discovered that we were re-parsing our BlockList.xml a large number of times, fixing this made a big difference. Combining that with switching to use the threaded & improved FastParser - yielded a further win. The auto-correct list is loaded after the 1st key-press, so getting this from 4.3 seconds down to 1.5 seconds (for huge correction lists) is a big win.

Image management

While profiling saving in various file formats, it was discovered that we frequently swap in (ie. re-load, and de-compress) images - this of course takes significant CPU time, particularly since we then immediately continue to preserve the (original) data in the file. In some cases this was taking a large proportion of save time for large image-filled presentations eg. Thanks to Tamaz Zolnai (Collabora) for cleaning up and fixing this, as well as hunting perennial image loss issues.

Fast Serializer

As a general rule any class named 'Fast' in the inherited OpenOffice code is a horrible mis-nomer. Many thanks to Matus Kukan (Collabora) for fixing this. We discovered that 25% of save time of large XLSX sheets was consumed in the Fast Serializer, which did a staggering 9.9 million system-calls, each writing some tiny fragment of an XML attribute eg. separate writes for opening elements, element names, attribute names namespaces etc. Matus reduced this to 76k calls to do the same thing, a 99% decrease. Quite apart from the system-call overhead we reduced cachegrind CPU pcycles for 'SaveXML' from over 12bn to under 3bn for a simple sample.

Bundle libjpeg-turbo

It has been known for many years that JPEG-turbo provides superior de-compression performance - "In the most general terms, libjpeg-turbo is 2.1 - 5.3x as fast as libjpeg v6b and 2.0 - 5.8x as fast as libjpeg v8d.". Naturally Linux vendors use the system packaged libjpeg, but when we distribute on Windows - we now bundle a 2x speed-up in the form of libjpeg-turbo - thanks to Matúš Kukan (Collabora) with some cleanups from Stephan Bergmann (Red Hat). Volunteers to make jpeg-turbo integrate nicely on Mac appreciated.

Mail merge performance

Mail-merge works by building a huge document containing the result of all the mails to be printed / merged into a single file. The wisdom of this is highly debatable, but nevertheless thanks to Lubos Lunak & Miklos Vajna (both Collabora for Munich) who put some significant effort in to very substantially accelerate large document merge, in some cases by several orders of magnitude. Sadly OpenOffice.org took a major regression here in version 3.3, and that is now comprehensively fixed. This turns a 2000 record mail-merge from a matter of hours down to a few minutes.

Calc Performance

There were a number of rather pleasant performance wins in this release of LibreOffice, which cumulatively have rather a helpful effect.

Range dependency re-work

For previous LibreOffice releases Kohei Yoshida (Collabora) spent a big block of time unifying runs of similar formulae into FormulaGroups - that fill down a large span of a column - since this is a common case for large data sets. This allowed a large memory reduction, and lots of great data sharing. However dependency management was de-coupled from this and was still performed per-cell. That is particularly expensive if you consider a range reference that is common for the whole formula group: resulting in lots of setup, and tear-down cost: essentially to notify the entire formula group. In 4.4 calc adds a listener type that is tailored for these formulae groups - potentially turning tens of thousands of complex data structure entries into a single entry. This saves a large chunk of memory, and a lot of CPU time walking lists, it also saves a ton of time when broadcasting the changes. There is plenty more work to be done to extend this, and ideally in future we should use the same approach for single-cell references as well. Thanks too to Eike Rathke (Red Hat) and Markus Mohrhard (Collabora) for some associated fixes.

Script type optimizations

For various reasons, detecting the script-type of a cell is an expensive operation; is it some asian text, complex text or simple - which affects the font, sizing & various metrics. Kohei Yoshida (Collabora) discovered that in several common operations - copying/pasting large chunks of data - that this work was being needlessly re-done and removed this cost. Similarly, for simple data types with standard formatting on eg. a large span of doubles, it was possible to significantly simplify the calculation of script types.

Chart deferred re-rendering

Another area that (still) causes some grief is that whenever a data range changes which a chart depends on, the entire chart is re-generated. That involves tearing down a lot of drawing shapes and re-creating them, which in the case of text is particularly expensive. Kohei Yoshida (Collabora) implemented a great optimization to defer this work until the chart is visible. This should have a pleasant effect on editing time for large data sets which are charted on many other sheets, and also for macros operating on many charts.

Getting involved

I hope you get the idea that more developers continue to find a home at LibreOffice and work together to complete some rather significant work both under the hood, and also on the surface. If you want to get involved there are plenty of great people to meet and work alongside. As you can see individuals make a huge impact to the diversity of LibreOffice (the colour legends on the right should be read left to right, top to bottom, which maps to top down in the chart):

Graph showing individual code committers per month

And also in terms of diversity of code commits, we love to see the unaffiliated volunteers contribution by volume, though clearly the volume and balance changes with the season, release cycle, and volunteers vacation / business plans:

Graph of number of commits per month by affiliation

Naturally we maintain a list of small, bite-sized tasks which you can use to get involved at our Easy Hacks page, with simple build / setup instructions. It is extremely easy to build LibreOffice, each easy-hack should have code pointers and be a nicely self contained task that is easy to solve. In addition some of them are really nice-to-have features or performance improvements. Please do consider getting stuck in with something.

Another thing that really helps is running pre-release builds and reporting bugs just grab and install a pre-release and you're ready to contribute alongside the rest of the development team.

Conclusion

LibreOffice 4.4 is the next in a series of releases that incrementally improve not only the features, but also the foundation of the Free Software office suite. It is of course not perfect yet, this is just the first in a long series of monthly 4.4.x releases which will bring a stream of bug fixes and quality improvements over the next months as we start working in parallel on LibreOffice 4.5.

I hope you enjoy LibreOffice 4.4.0, thanks for reading, don't forget to checkout the user visible feature page and thank you for supporting LibreOffice.

Raw data for many of the above graphs is available.

A great French translation of this is kindly made available at linuxfr.

2015-01-28 Wednesday

2015-01-27 Tuesday

2015-01-26 Monday

2015-01-25 Sunday

2015-01-24 Saturday

2015-01-23 Friday

2015-01-22: Thursday

2015-01-21: Wednesday

LibreOffice Android View (Beta)

Today Collabora released (cf. Press Release) a Beta snapshot of the viewer work we've been doing for and with SMOOSE for some months. This provides a version for testing (it's really not finished yet), of LibreOffice running on Android you can install it from Google Play.

What does it look like ?

The writer functionality is probably the best piece of the component, allowing pleasant screen-shottage and rather good viewing.

ODT - writer document on Android

Impress presentations are supported here is an old one around cairo rendering support, you can see the swipe-in-from-the-left page switcher:
ODP - Impress document on Android

We also have some (very early) calc tiled rendering; there are some text scaling issues that we're working on there:
Calc font test

What did that involve ?

A huge amount of work. Much of the initial work was done by Tor Lillqvist, while at SUSE (now Collabora), creating a cross-compilation framework which we continue to use for iOS, Android and was originally setup to do cross-compiles to Windows - so that we could have a predictable toolchain, and a reliable/repeatable free-software build environment. Tor also did some amazing bootstrapping work to overcome several debilitating limitations of C++ on Android, get to get the initial startup, and packaging into a good state.

Matus Kukan (Collabora) also did a chunk of work for CloudOn to help link the entirety of LibreOffice into a single shared library image - helping startup performance & improving Android support. This significantly helps to work around the built-in Android linker's low limit of shared libraries it can handle before failing. It also helps to export nearly no symbols, and to let ld's garbage-collection throw away as much object code as possible: though there is still way more to loose.

Other work done for CloudOn by Jan Holesovsky (Collabora) and Tor Lillqvist was to implement tiled rendering inside Writer - making it possible to render arbitrary portions of documents at arbitrary zoom factors. We are deeply indebted to them - you can of course enjoy the same code inside CloudOn's iOS app.

Ian Billet worked during the Google Summer of Code 2012 to make a document browser, which has subsequently been taken over and much improved by Jacobo Pérez of Igalia creating the document browser we have today. It is of course somewhat regrettable that Android has no standard built-in file browser making this necessary. Of course the viewer also hooks into the OS to allow opening downloads, attachments etc. in the normal way.

Since FOSDEM 2014, Smoose and Collabora in the person mostly of Tomaz worked to re-target the excellent Fennec tiled-rendering code from Mozilla's android browser to target instead of the Mozilla core - the LibreOffice core. This involved great chunks of re-factoring, and hard work both in the Android and C++ core. We re-use and added tiled rendering to the LibreOfficeKit API - an API to allow simple re-use of LibreOffice for file format conversion and document rendering via a rather trivial C and C++ API. LibreOfficeKit under Linux has the advantage of requiring no linkage to the code - being an (almost) purely abstract API.

Andrzej Hunt worked through Google Summer of Code 2014 to improve and extend tiled rendering to Calc and Impress and the LibreOfficeKit tiled-rendering API. You can read about that on his blog. One side-effect of that, recently much improved by Miklos Vajna (Collabora) is the gtktiledviewer test app which deliberately uses a non-VCL widget-set, and LibreOfficeKit to render a series of tiles on a stock Linux desktop. This significantly to helps accelerate debugging. Kohei Yoshida (Collabora) was also involved in reviewing, re-working and merging the calc tiled-rendering code to master.

Many thanks to all involved, particularly our client SMOOSE without whose support, Collabora would not have been able to prioritize this work.

Getting involved

The code for the viewer is almost exactly what is in the git / master branch in the android/experimental/LOAndroid3 directory. It should build out of the box with a simple make, of course after you have followed the compilation instructions and Android how-to. Needless to say - we've left a lot of juicy, easy-to-fix bugs for you to gloriously fix - give it a try.

Finally if you want to hear, and see more, and get involved with what we're up to, a great way to do that is by coming to Kendy's LibreOffice on Android talk at FOSDEM this year, or Jacobo's document manager talk.

Can you use LibreOffice in some way ?

Collabora provide bespoke consulting solutions to help people use LibreOffice in the most effective & optimized way. Sometimes that means using LibreOffice as part of your product, sometimes it means optimizing, improving or extending to meet your use-case. We also sell long-term Enterprise support for LibreOffice to make it easy and risk-free for you to deploy LibreOffice in your organisation on Windows, Linux, Mac and (soon) Android too - checkout LibreOffice-from-Collabora for more details.

2015-01-20: Tuesday

2015-01-19: Monday

2015-01-18: Sunday

2015-01-17: Saturday

2015-01-16: Friday

2015-01-15: Thursday

2015-01-14: Wednesday

2015-01-13: Tuesday

2015-01-12: Monday

2015-01-11: Sunday

2015-01-10: Saturday

2015-01-09: Friday

2015-01-08: Thursday

2015-01-07: Wednesday

2015-01-06: Tuesday

2015-01-05: Monday

2015-01-04: Sunday

2015-01-03: Saturday

2015-01-02: Friday

2015-01-01: Thursday


My content in this blog and associated images / data under images/ and data/ directories are (usually) created by me and (unless obviously labelled otherwise) are licensed under the public domain, and/or if that doesn't float your boat a CC0 license. I encourage linking back (of course) to help people decide for themselves, in context, in the battle for ideas, and I love fixes / improvements / corrections by private mail.

In case it's not painfully obvious: the reflections reflected here are my own; mine, all mine ! and don't reflect the views of Collabora, SUSE, Novell, The Document Foundation, Spaghetti Hurlers (International), or anyone else. It's also important to realise that I'm not in on the Swedish Conspiracy. Occasionally people ask for formal photos for conferences or fun.

Michael Meeks (michael.meeks@collabora.com)

pyblosxom