Say Hello to Offline First

Prescript: Join the discussion on offlinefirst.org and help collecting information on the Offline First Github Organisation. And see the two follow up articles Offline First and the Circle Of Web and Offline First and the Circle of Web Part II: Breaking the Circle.

(Russian translation by @kouprianov)

The members of Team Hoodie are currently travelling around quite a bit, organising, participating in and attending conferences and workshops in various countries, and through all this we’ve not only made lots of personal experiences with limited network connectivity, but also met many people from all over the world, all with their own related stories and problems.

Frequently not having any data connection in even the wealthiest and most developed cities of the world has led us to conclude that no, the mobile connectivity/bandwidth issue isn’t just going to solve itself on a global level anywhere in the near future.

Tube

We’ve generally been rather optimistic about the whole problem, often thinking that coverage and bandwidth are sure to increase continually, but our recent experiences have made this attitude appear increasingly naïve. Capacity problems, varying coverage and zones with zero reception, unreliable connections (both wifi and telco) and problems incurred through travelling will likely persist for quite a while.

Geography is non-negotiable, as is most architecture. Travellers are doubly plagued: not only by their mode of transport often prohibiting any sort of connection (many subways and high-speed trains come to mind, as do most airlines), but also by the extortionate roaming charges imposed by the carriers, making a spy-sized collection of SIM cards a neccessity for anyone without a massive personal telco budget. And this is speaking from a solidly western-europe-centric perspective.

Gregor has spent the last six weeks travelling through Africa on the AfricaHackTrip, and we’ve had the opportunity to speak to people from all over the world at Mozfest in London. Here and in many other instances we’ve encountered a solid interest in and a passionate desire for better offline capability in both web and native applications.

We can’t keep building apps with the desktop mindset of permanent, fast connectivity, where a temporary disconnection or slow service is regarded as a problem and communicated as an error.

Mobile networks simply don’t provide that same level of quality and consistency, and it’s high time we start adapting to that reality. In the same way it makes sense to build interfaces mobile-first, it’s time to think about designing and building our applications offline-first. With an offline mode that isn’t merely a contingency for an error scenario, but a fundamentally more flexible and fault-tolerant way of application design.

With Hoodie, we’ve created an architecture that allows you to build offline apps with relative ease. There are a few issues, but we’re quite confident about the technical aspects of our syncing engine and how it handles being offline. But now we’ve stumbled on a plethora of new, hitherto unknown and unresearched use cases and UX scenarios, all of which are only now easily within the reach of most developers.

Hiking up Needle Point

This is quite literally a new frontier, largely unexplored and full of interesting problems and unimagined edge cases. In most other realms of app design we are spoilt with UX/UI patterns we can readily employ, but offline-first is terra incognita, there is practically nothing to go on in terms of patterns and metaphors.

Offline Becomes Reality

All of a sudden, offline becomes more than error handling. Offline becomes a non-threatening reality in which, yes, some aspects of your app will neccessarily stop working, but many others don’t, because they no longer have to. And this raises many fundamental questions:

Does the app even need to inform the human of the current connection state? Is this information even relevant to the human? If so, how can this best be done?

Can I give the users more trust in the app, leaving them safe in the knowledge that it won’t forget the last state it was in, and that it won’t lose any data, regardless of what the connectivity situation is?

How can I communicate to the human that creating data within the app is still possible offline, and that it will be posted/dispatched/properly dealt with in the future?

How can my interface convey changes that occur in the users’ current view when they reconnect and the server pushes new and changed data? What to do with deleted items, things that can’t be organised in lists, objects that aren’t in themselves immutable (like emails)?

How can I make the inevitable resolution of conflicts by the human as painless and intuitive as possible?
Can the app make any preemptive decisions on the part of the human and pre-load any low-overhead data they might need later?

What metrics (possibly behavioural) are there to base these decisions on?
How can I word all of these scenarios in a way that doesn’t worry the users, that embraces offline as an unproblematic reality?

There’s more, of course, but we have to start somewhere. Our Mozfest session on Offline First Application Design has made this abundantly clear to us. There’s a whole new paradigm rolling our way, and we’re going to roll with it. We want to help users of Hoodie in building robust, always-usable applications, not just on a technical level, but also with patterns, metaphors and methods for higher-level application, user experience and interface design. This will be our research project, and it’s going to live at offlinefirst.org.

If you are interested in joining the conversation, you can start by leaving comments on this post. We are looking forward about what you have to share! :)

sailing boat

53 Responses to “Say Hello to Offline First”

  1. Dan Lash

    I think the idea of offline first could be very interesting; would like to hear more.

  2. Luis Abreu

    Well, this is an interesting problem. Native apps might provide a lot of patterns aimed at solving this issue since they are very frequently offline first.

    Solution will change a lot based on the network quality and application class and data in question, as I think the post also states.

    I’d be happy to chat about this over IRC or personally (Jan’s coming to Brighton this weekend right?).

    There’s also the question of preemptive network activity, examples such as Instagram’s photo uploading which starts as soon as the photo is edited and while the person fills in optional details. (Upload state is then displayed in the timeline itself. Actions such as liking a photo are also immediate and update when the network request completes, reflecting the final result).

    There’s a bit of a crossover between application-level design decisions and API but I’m guessing hoodie wants to enable offline-first and provide convenience when appropriate and possible.

    Promises and a reactive architecture helped me in the past, code architecture, culture is crux to an offline first app. Complexity/organization is usually higher, wonder how that interacts with the apparent aim of hoodie to be easy to use.

  3. Arnaud Rinquin

    You are absolutely right insisting on UI/UX and service design being less known part of the problem.

    I am trying to build an offline + collaborative editor. This brings a lot of technical problems but we know that we will find solutions. Designing the service is something else because we don’t really know the problems yet.

  4. Rob Fielding

    A use case that really illustrates the problem: Presume that you are playing correspondence chess/go where the opponent takes nearly 24hrs to respond, when he has a few seconds of internet connectivity. Both sides can take a little bit of time computing the next bits of the game tree they are willing to commit to (ie: opening strategy, obvious move/response). When they do make contact, the trees are merged together, and some number of moves of progress are made. As an example, the two sides anticipate the same chess opening and find themselves about 10 moves into the game after only their first exchange. The game pauses for a response to the first unanticipated move. At the end of the game, the game record includes not only the realized line of play, but what both sides foresaw.

    So now don’t think of games, but think of network protocols. If the branching factor for the protocol is low, then a minimal number of exchanges need to be made to make progress – maybe even 1. This idea could be useful in a car’s navigation system, to load up working sets of data from the network based on where it thinks you are going, and what kind of detours you might take.

    Protocols can overcome large latencies, and basically function normally in the face of very sparse connectivity in this manner. Phone web browsers really need to work more like this as another example.

  5. Bobby

    I would like to know more about this approach. After mobile first approach, I think offline first must become a new best practice for building both website and Web app.

  6. Jan

    @Luis yeah, I’ll be in Brighton from tomorrow, we should definitely hang out :)
    And yes, all the things you mention are exactly the kind of thing that we want to talk about, collect, categorise and make accessible for learning :)

    @Rob great example, thanks!

    @Jake awesome, thank you!

  7. Steven

    This is pretty revolutionary. My brain is going in overdrive with all the possibilities this brings!

  8. martijn

    Look forward to following this. You may find my offline web app interesting as well: https://enketo.org. It was specifically developed around offline web technologies for use in developing countries to bridge the gap.

  9. Jay Woods

    I’ve lived the “offline first” lifestyle for 50 years and never realized that was what I was doing. Your work is going to help me a lot.

  10. Tim Wright

    Overall, I really like the concept. I’m wondering about device storage limitations though. If it were to catch on, having a 16GB device would almost be unreasonable with all the offline assets that would need to be stored to keep an app (semi) fully functioning offline. I know the data weight isn’t a huge deal, but larger information sets, like images could quickly fill up space.

    It’s obviously not a huge issue right now, but in the future 10, 20, 30 apps on a device – all with full offline capabilities – could end up being a problem. I’m really looking forward to seeing how the concept evolves (any being able to tweet from a Subway tunnel)!

  11. Gregor

    @Tim interesting point, never thought about it. I’d say: It’s a problem when it’s a problem ;) For now, most of the browser limit the available storage size per domain, so it won’t happen that 10 or even 100 apps with lots of offline data could fill up your device’s storage.

    For example, I did minutes.io around 3 years ago. It is offline capabale. And even users with hundreds of meetings don’t use more than 2mb of space. As long as you stay with text only, you get very far. And my guess is that for offline usage, not having access to images is more often than not acceptable and understandable to the user

  12. Jay Woods

    Tim Wright is introducing one of those interesting edge cases. It is not only a loss of connectivity to the web but also loss of connectivity to large storage. Both the human schlepping all the equipment around and the apps need to be tuned to what is practical. As an example, the app needs to be aware of how much local storage is available, what the priorities are for holding the data close, and what to do moving data locally when the wifi becomes available. How much is the human to be involved in this or is there a way of organizing the problem so it “just works” in the same sense as Google worked out how to prioritize search results without involving catalogers. The least recently used plus frequency counting method of balancing storage usage is an example of how to do it. However, I haven’t seen an operating system use this approach for file storage in 20 years. I understand that Oracle will prioritize database replication syncing as an option.

  13. Bruce LeSourd

    Back in 2008, I led a project to build a mobile traveling salesman sample case/order system for Amway distributors. (For those who dislike Amway’s MLM strategy, just think of this as one of the very first modern mobile sales force tools.) At that time, Amway’s core demographic in the U.S. (400K “Independent Business Owners”) were concentrated in areas with very poor mobile cellular data service, so we assumed the app would have to work offline (this also had the benefit of making it work on your basic airplane of the time, or in the NY subways, where users had a captive audience ;-)). It had a full offline catalog, and could take multiple orders offline, including securely taking credit card data, which was encrypted using a public key and sent to Amway’s e-commerce servers once connectivity was restored.

    Because we built that app from the ground up to work offline, it handled offline and transitions to offline very well. Since then, nearly every app I’ve worked on has had problems with intermittent connectivity, which is still surprisingly common as you point out. The worst type of intermittent connectivity is when a user is moving between WiFi networks, because the traditional WiFi network stack isn’t built to handle this very well (I believe this is starting to be addressed).

    The worst case scenario I’ve seen is a mobile app in a WiFi environment with intermittent connectivity doing financial transactions, where it’s quite possible for a mobile device to send a transaction, have it processed successfully, but never receive a response due to a network connectivity failure. It’s remarkable how many otherwise competent programmers fail to handle that situation correctly.

  14. buzz

    My use case is broadcasting locally hosted web content directly to audiences, without the Internet. These are not captive portals but self-contained Wi-Fi publishing systems. They have been deployed at conferences and arts venues where convenient Internet access is simply not provided, affordable or satisfactory.

  15. Tim King

    Bruce your are so right – otherwise competent programmers are too used to permanent connections and the case where you do the transaction but don’t get a reply is rare but happens in about 1% of the time. Here at 5app we have been researching the state of the UK network and provide a service that provides 2-way reliable communications. Check out http://www.5app.co.uk and my blog there.

  16. Neville Franks

    The biggest issue that I have is the complete lack of a half-way reasonable database in the Browser, that can store a decent amount of data and works cross-browser and cross-device. IndexedDB and it’s unfriendly api, and simplistic key-value store doesn’t cut it and even if it did, it isn’t cross browser. In my ideal world a NoSQL DB like MongoDB but cut-down, is what I’d like to see.

  17. Adel

    We (at Lonely Planet) built an offline app for our authors so they could keep writing in-browser reviews for fabulous places to visit even when they weren’t connected. The learning curve was pretty steep and like Neville says we found IndexedDB to be untrustworthy and in the end had to get very creative to ensure the authors’ work was reliably saved. Even though we have a captive pool of users it can still be challenging to explain the difference between working offline and online to them, it isn’t a mental model most users have.

  18. Neville Franks

    Hi Alex, yes I’ve been following PouchDB off and on for a while and as interesting as it is CouchDB itself is not a good fit for my app, whereas MongoDB is perfect. I can’t recall what PouchDb is using for storage i the Browser. It and CouchDB’s future is also of concern

  19. Kushal

    We are into traveling domain and working extensively on providing solutions in the same. Having a offline app that can capture my itinerary, my expenses on field and later synchronizing the same with the office headquarters/branches when I am in the office so that the entire data is visible on my desktop to prepare a report of the tour and vice versa. This technology could be of great use. Keen to hear a lot on the same.

  20. Jan

    @neville there is no need to worry about CouchDB’s future :)

    @Adel & @Neville — Hoodie currently uses localStorage because it is the most reliable lowest common denominator for in browser storage. But it too has limits. And you are right, getting a decent in-browser database is crucial for this. PouchDB is a nice abstraction over IndexedDB, but there are others. PouchDB just gives you CouchDB’s offline semantics in the browser, which is exactly what you would want here :)

  21. sherif

    I’m trying to imagine something liike: html5 offline storage, The MEAN Stack: MongoDB, ExpressJS, AngularJS and Node.js …

  22. Robert Reiz

    I lived/worked in San Francisco / Silicon Valley, Berlin and other big cities. And specially in the big cities connectivity issues are the rule, not the exception. Not sure why that is the case, but that’s my experience. Even simple news apps assume that I’m always online. They try to reload the current view every time I switch on/off my phone. That’s totally annoying!
    Yes! Offline first is a great approach! I will spread the word about it and keep it in mind for my next app.

  23. Lena

    It’s weird, especially in cities – I live in Berlin and often got no stable internet connection here. And it’s so annoying!

    Thanks a lot for your kind comment (and for spreading the word)! <3

  24. G

    Great to have more minds thinking about this problem at a large scale. The development of usability patterns is a must to improve the interaction and introduction to technologies. Look forward to contributing!

Leave a reply