Stuff Michael Meeks is doing

This is my (in)activity log. You might like to visit Collabora Productivity a subsidiary of Collabora focusing on LibreOffice support and services for whom I work. Also if you have the time to read this sort of stuff you could enlighten yourself by going to Unraveling Wittgenstein's net or if you are feeling objectionable perhaps here. Failing that, there are all manner of interesting things to read on the LibreOffice Planet news feed.

Older items: 2015: ( J F M ), 2014: ( J F M A M J J A S O N D ), 2013, 2012, 2011, 2010, 2009, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999, legacy html


LibreOffice On-Line & IceWarp

Today we announced a collaboration between IceWarp and Collabora to start the creation of LibreOffice On-Line, a scalable, cloud-hostable, full featured version of LibreOffice. My hope is that this has a huge and positive impact for the Free Software community, the business ecosystem, personal privacy, and more. Indeed, this is really one of the last big missing pieces that needs solving (alongside the Android version which is well underway). But wait - this post is supposed to be technical; lets get back to the code.

A prototype - with promise

At the beginning of the LibreOffice project, I created (for our first Paris Conference) a prototype of LibreOffice On-Line using Alex Laarson's (awesome) GTK+ Broadway - you can still see videos of that around the place. Great as the Broadway approach is (it provides essentially a simple Virtual Desktop model into your browser), the prototype taught us several important things which we plan to get right in LibreOffice On-Line:

Having said all this, Broadway was a great basis to prove the feasibility of the concept - and we re-use the underlying concepts; in particular the use of web sockets to provide the low-latency interactions we need. Broadway also worked surprisingly well from eg. a nearby Amazon cloud datacentre. Similarly having full-fidelity rendering - is a very attractive proposition, independent of the fonts, or setup of the client.

An improved approach

Caching document views

One of the key realisations behind LibreOffice On-Line is that much of document editing is not the modification itself; a rather large proportion of time is spent reading, reviewing, and browsing documents. Thus by exposing the workings of document rendering to pixels squares (tiles) via LibreOfficeKit we can cache large chunks of the document content both on the server, and in the client's browser. As the users read though a document, or re-visit it, there is no need to communicate at all with the server, or even (after an initial rendering run) to have a LibreOfficeKit instance around there either.

Thus in this mode, the ability of the browser's Javascript to understand things about the document itself allows us to move much more of the pan/zoom reading goodness into your client. That means after an inital (pre)-fetch that responsiveness can be determined more by your local hardware and it's ability to pre-cache than remote server capacity. Interestingly, this same tiled-rendering approach is used by Fennec (Firefox for Android) and LibreOffice for Android to get smooth mobile-device scrolling and rendering, so LibreOfficeKit is already well adapted for this use-case.

Browser showing hidden tile cache ready to be revealed when panning
Editing live documents

In recent times, The Document Foundation has funded, via the generosity of TDF's donors a chunk of infrastructure work to make it possible to use LibreOfficeKit to create custom document editors. There are several notable pieces of this work that intersect with this; I provide some links to the equivalent work being done for Android from Miklos Vajna:

Cursors & selection

Clearly blinking a cursor is something we can do trivially in the javascript client, rather than on the server; there are however several other interactions that benefit from browser acceleration. Text selection is a big piece of this - re-rendering text on the server simply in order to draw transparent selection rectangles over it makes very little sense - so instead we provide a list of rectangles to render in the browser. Similarly, drawing selection handles and interacting with images is something that can be handled pleasantly in the browser as well.

Keyboard / touch input

Clearly it is necessary to intercept browser keystrokes, gestures and so on, transport these over the websocket and emit them into the LibreOfficeKit core.

Tile invalidation / re-rendering

Clearly when the document changes, it is necessary to re-render and provide new tile data to the client; naturally there is an existing API for this that was put in place right at the start of the Android editing work.

Command invocation

Another piece that is required, is transporting UNO commands, and state (such as 'make it bold', or 'delete it') from the client javascript through into the LibreOfficeKit core. This is a matter again of proxying the required functionality via Javascript. The plan is to make it easy to create custom, bespoke UIs with a bit of CSS / Javascript magic wrapped around and interacting with the remote LibreOfficeKit core.

Serializing selections

Clearly as & when we decide that a user has wandered off, we can save their intermediate document, serialize the cursor location & selection - free up the resources for some other editing process. As/when they return we can then restore that with some small document load delay, as we transparently back their cached view with a live editable LibreOfficeKit instance.

What does that look like roughly ?

Of course, lots of pieces are still moving and subject to change; however here is a perhaps helpful drawing. Naturally integrating with existing storage, orchestration, and security frameworks will be important over time, contributions welcome for your pet framework:

Initial architecture sketch

The case for simple collaboration

A final, rather important part of LibreOffice On-Line; which I've left to last is that of collaborative editing.

The problem of generic, asynchronous, multi-instance / multi-device collaborative document editing is essentially horrendous. Solving even the easy problems (ie. re-ordering non-conflicting edits) is non-trivial for any large set of potentially intersecting operations. However, for this case, there are two very significant simplifying factors.

First there is a single, central instance of LibreOfficeKit rendering and providing document tiles to all clients. This significantly reduces the need to a re-order asynchronous change operation stream, it is also the case that editing conflicts should be seen as they are created.

Secondly, there is a controlled, and reasonably tractable set of extremely high-level operations based on abstract document co-ordinates - initially text selection, editing, deletion, object & shape movement, sizing, etc. which can be incrementally grown over time to extend to the core set of editing functionality.

These two simplifications, combined with managing and opportunistically strobing between users' cursor & selection contexts should allow us to provide the core of the document editing functionality.

Show me the code

The code is available as of now in gerrit's online repository. Clearly it is the Alpha not the Omega; the beginning, and not even the end of the beginning - which is a great time to get involved

Conclusion

LibreOffice On-Line is just beginning, there is a lot that remains to be done, and we appreciate help with that as we execute over the next year for IceWarp. A few words about IceWarp - having spent a rather significant amount of time pitching this work to people, and having listened to many requests for it - it is fantastic to be working with a company that can marry that great strategic sense with the resources and execution to actually start something potentially market-changing here; go IceWarp !


My content in this blog and associated images / data under images/ and data/ directories are (usually) created by me and (unless obviously labelled otherwise) are licensed under the public domain, and/or if that doesn't float your boat a CC0 license. I encourage linking back (of course) to help people decide for themselves, in context, in the battle for ideas, and I love fixes / improvements / corrections by private mail.

In case it's not painfully obvious: the reflections reflected here are my own; mine, all mine ! and don't reflect the views of Collabora, SUSE, Novell, The Document Foundation, Spaghetti Hurlers (International), or anyone else. It's also important to realise that I'm not in on the Swedish Conspiracy. Occasionally people ask for formal photos for conferences or fun.

Michael Meeks (michael.meeks@collabora.com)

Made with PyBlosxom