The previous State of Sync posts were fairly technical, at least when compared to other posts that have appeared here in the past. This time, however, I will concentrate on user-level aspects – and I’ll give a timeframe for our next steps.

After the previous post, our tech support team received quite a few inquiries about what exactly our forthcoming cloud sync solution would entail for end users. To help answer these questions – at least those we could answer at that time – we published this FAQ page. Today I'd like to answer some more of your questions – extending and updating that with some new information.

How it will work

As the name already implies, our sync solution is provided as a service that lives in the cloud. This means that the creation of an online account will be a user’s first necessary step in the configuration of Things for sync.

Of course, Things on the Mac and iOS devices will also need to be updated, in order to enable communication with the cloud server. This software upgrade will come at no charge, and will also sport a revamped database layer that is not only faster, but specifically optimized for cloud sync usage.

Things will sync frequently. While there will be a way to initiate sync manually there should hardly be any reason for doing so. Every change you make is transmitted to the cloud almost instantly. No matter when you quit the app, your data is safe with the server already. Whenever you open Things, switch applications, or wake your computer from sleep, Things will check back with the server to see if there are any updates to pull.

Things will always connect to the server using an encrypted connection. Not only during log-in, as many web sites do, but for every connection. This means that your to-dos will never be sent in the clear.

Service costs

Without large scale tests, it is not realistic to estimate how much resources our users will consume in the cloud. In particular, the frequency of interactions between users’ databases and the central service. Therefore we will be doing extensive scalability testing before we make an announcement regarding possible costs for the service.

Bonjour sync

Some users were asking about the future of our existing Bonjour-based WiFi sync. As a matter of fact, our Bonjour sync – in its current form – is incompatible with the database layer improvements I mentioned above. As a consequence, there is no way for us to keep the old sync approach with the improved database layer. What we are planning to do instead, is to rebuild WiFi sync using the same core components that also power our cloud sync technology. We’ll achieve this by putting a little sync server into Things Mac. This is something that might not ship with the initial release of cloud sync, but shortly thereafter.

Things Mac cloud sync beta

I’m sure the question on the mind of many readers is: when will I be able to play with Things cloud sync first-hand? As it happens, the integration of cloud sync with the Mac version is a little further ahead of its counterparts on iOS. We have therefore decided to start with a beta of Mac-to-Mac sync first.

Initially, the cloud sync beta will be invite only. Everybody can subscribe, but we will activate accounts on a first come, first served basis as we gradually scale up the beta. We will be starting with a small number of beta testers at first – the idea is to iron out any kinks on the server early on – we will then slowly add more people in order to test our system under increasing load. iOS devices will join the fun once we are confident with everything server-side.

When will this happen? We are expecting to start sending out invitation emails on May 2.

But there is one more thing…

The subscription page is already live. If you want to become a test pilot for Mac-to-Mac sync, please sign up here.

We will publish our next sync-related blog post once the beta is ready.

The major new feature of our just-released versions of Things for iOS - Things for iPhone 1.7 and Things for iPad 1.4 - is the ability to create and edit repeating tasks directly on iOS devices. They can also be synced to the Mac version, which is enabled by the new update of Things for Mac.

Designing an interface for repeating tasks is a challenging task for any platform, let alone for small iOS devices. In this blog post I want to give you an overview of our design goals and the solution we came up with.

Those of you who have been following the development of Things for Mac since it was in beta might remember the challenges we have faced in coming up with a good UI for repeating tasks for the Mac. We wanted to have a powerful and flexible implementation that was still easy to use. After an extensive design journey with many iterations, we were very happy and proud of the outcome. Two years later, and after lots of customer feedback, we see that we did most things right. As always, there is room for improvement – but that’s another chapter.

When we started designing repeating tasks for iOS, we asked ourselves if it was possible to bring the same expressive power to iOS, while still keeping the UI clean and easy to use. Given the constraints of a small device like the iPhone, we knew that this was not going to be easy.

These were our goals when designing the repeating task UI:

1. Expressiveness

   Repeating tasks come in many different varieties and complexities. If you look at a list of typical repeating tasks people have, you will soon realize that there is no single prototype recurrence pattern they can all be mapped to. Examples are: "pay rent at the end of each month", "make backups every 2 weeks", "for the next 5 months, create a performance report at the beginning of the month", "perform health exercises every Monday and Thursday", "prepare for Macworld Expo every January 1st", "prepare club meeting every second Sunday of the month". A repeating tasks implementation that only has the expressive power to specify a subset of the tasks above arbitrarily constrains the user in the tasks he can manage efficiently.

2. Discoverability

   The UI should make it obvious to the user what kind of recurrence patterns can be specified. Expressiveness is of little use when users need to read extensive documentation of how to properly enter certain recurrence patterns.

3. Ease of use

   This goal is obvious, but what exactly does "easy to use" mean in the context of repeating tasks? For regular task entry, one very important aspect of "easy to use" is that entering a new task requires very few taps. As something that is done over and over each day, any extra taps quickly turn into an annoyance for the user. This is why our quick entry is designed to keep the number of taps to a minimum. The same does not equally apply to repeating tasks. Typically, repeating tasks are entered much less frequently. Some are even entered once when you start using the app and remain there for years ("pay rent" is one such example). So rather than speed of entry, it is much more important that the UI has a great clarity and conciseness when it comes to specifying the different recurrence patterns.

With these goals as our guides, we evaluated existing implementations, sketched out different designs, discussed the merits and disadvantages of each, and finally arrived at the version we've now shipped. We believe it is the best and most elegant repeating tasks implementation currently available on iOS. Let me walk you through it.

Just like on the Mac version, repeating tasks can be created from within the Scheduled list. For that, we have a new button in the toolbar.

Note how there is one specific place in the UI to go to when you want to create a new repeating task. The existing UI for adding regular tasks is left untouched, so adding this feature didn't complicate other parts of the application. This is very much in line with the observation that repeating tasks are not entered frequently.

When you tap the button, a new modal view comes up that is specifically designed for repeating tasks.

At the very top, we show an English sentence that describes the current recurrence rule. Below, you can enter the title, tags, and notes of the repeating task, and also specify whether it should create instances with due dates.

Note that we pre-populate the recurrence rule based on the current day of the week. In most cases, the rule you want to specify probably differs from the one we show by default. But then why do we show a default rule in the first place?

We believe that it communicates two things very nicely. First, we can present a view that already contains all the elements of a repeating task: the recurrence rule, together with the title, notes, tags, and due date. So immediately, you know what you are expected to fill out before you can save the task. Secondly, if you don't like the recurrence rule, you will immediately know that you need to tap the description text in order to change it. And what's really great about this, even before you see the next UI view with the different controls that let you specify a different rule, you already know what the end result of those settings will be: a new English sentence that accurately describes the rule you had in mind.

This is the view that appears when you tap the recurrence rule description:

When looking at the general layout of this view, you will realize that – when read from top to bottom – this is basically the same English sentence you saw before, except it is now divided into different controls that let you specify the different parts of the recurrence rule to suit your needs.

Let's now change this rule to "Every 2 months on the first and third Sunday". First, "Every week" is obviously wrong, so we tap on that. A new view slides in, letting us specify the "Every 2 months" part.

After we're done with that, we go back to the screen before, and tap on the "On" section. In the view that comes up, we select the 1st Sunday, tap on "Add More", and also select the 3rd Sunday.

Note that we can additionally specify when the recurrence should begin, and when it should end (e.g. after 5 recurrences, or on a specific date), but we won't do that for now. Tapping "Save" brings us back to the original screen where the English sentence now properly describes our new recurrence rule.

After we enter the additional repeating task properties and tap “Save”, the new task shows up inside the Scheduled list.

If you later revisit the repeating task, you will conveniently see upcoming recurrences:

This wraps up the walkthrough of the repeating tasks feature. I hope you got a good first impression, but you should definitely try them out on your device to get a complete picture. There are additional details not covered in this example:

• If you specify weekly recurrences, we have an optimized picker that lets you easily select multiple weekdays.
• You can also create tasks that repeat after the completion of the previous one.
• You can specify the number of times a repetition should occur.
• You can pause and resume repeating tasks (very useful, if you're going on vacation).
• If you use Areas, you can assign new repeating tasks directly when you create them.

We hope you enjoy using repeating tasks on iOS. Feedback is, as always, greatly welcome.

My last post received quite a bit of criticism. Some people even said I shouldn’t have written it at all. It has been claimed that the sync problem is solved already. Our experience has shown that this couldn’t be farther from the truth – in particular for Cocoa developers. Dave Peck observed this too, and made some wise suggestions. Here is a quote from his article:

If you're a die-hard Cocoa developer, the thought of building a scalable and reliable web back-end might just be daunting -- daunting enough to cut it out entirely. I've noticed a strong resistance in the Cocoa community to building such things.

The Mac community is a great ecosystem and we’ve benefitted a lot from it. Things could not exist without it. This is why creating a cloud sync solution has come to mean more to us than just an improvement of our existing WiFi-based sync. In fact, one of the motivations behind this series of posts, is to start sharing our insights with the community of Cocoa developers.

The first thing you need to do when you want to solve a problem is to get a good understanding of the nature of the problem. The previous post attempted to explain the sync problem in the least technical terms possible – to make it accessible to a non-technical audience as well. Eventually, however, we will be making more technical documents available – directly addressing fellow developers.

Today, I want to talk about the technologies we’ve tried.

Bonjour

Back in the day when Things for the Mac was still in beta we added WiFi sync using Apple’s Bonjour technology. WiFi sync was Apple’s recommended way to realize sync. There are advantages indeed. For example, there is no need to sign up for a web service. It doesn’t even require a pre-existing local network, as any Mac can create its own spontaneous network. The user’s data never leaves the local network.

However, there are notable disadvantages as well: you actually have to remember to sync, and you have to initiate it manually. Relying on the local network, of course, also eliminates the possibility of syncing your work Mac with your Mac at home.

Obviously, we had to provide a solution that would utilize the cloud instead of the local network. The most obvious choice is perhaps MobileMe.

MobileMe

There are quite a number of people using Apple’s MobileMe to sync, e.g., calendar or address book information. For such users, it is only reasonable to wish that Things would sync in the same way, using their existing MobileMe subscription. Unfortunately, the sync technology Apple uses for their own apps is not available to third party developers on iOS devices.

It is available on the Mac though, and we hoped it would eventually find its way onto iOS; this hasn’t happened yet. What some competitors are in fact referring to, when they talk about MobileMe sync, is the use of remote storage through the iDisk.

Dropbox, WebDAV et. al.

Some users have suggested using Dropbox for sync. Like many similar offerings – often based on the so-called WebDAV protocol – Dropbox allows users to store files remotely on their servers. In this sense, it is similar to Apple’s iDisk (assuming iDisk sync is turned on) – what both services do is provide a folder on your computer that is mirrored on a remote server.

Neither Dropbox nor iDisk – or any similar service for that matter – were conceived as a sync solution for apps like Things. They were designed for sharing files – like photos or PDFs. Remote file storage products do not offer merge facilities that come even close to what would be needed for Things.

What is possible, however – and others are in fact doing this – is to use a hack. Without going into detail, it basically means breaking up the database file into a large number of smaller files. Merging and conflict resolution can now be handled using these smaller fragments, which eases some of the pain.

This approach works to some extent, but it is slow and error prone to begin with, and advanced options like push or anything involving user-to-user data exchange is impossible. Developers choosing this approach for apps with complex data models similar to Things paint themselves into a corner right from the start.

Database in the Cloud

Instead of using remote file storage, it is a much better idea to use a custom-designed web service. This is an approach taken by most web applications, and a whole industry has arisen developing, deploying, and maintaining applications that are written in this style. Everyone going this route has ample expertise and help available to them.

In taking advantage of this, we began working together with a great group of web development people. It was a great experience, and one we wouldn’t want to have missed – but eventually we abandoned this approach. Here is why:

If you are thinking in terms of a web application, you are basically considering placing the user’s entire database in the cloud, together with enough logic to safely manipulate that data. This makes the database in the cloud the authoritative version which can be used to determine how the data on every device should look. This sounds like a great thing to have, but it requires that all merging and conflict resolution be done on the server - and this turns out to be really slow.

Merging potentially requires accessing the database, and hence the hard-drive, very often. Hard-drive operations, however, are the most expensive in terms of performance – in particular when many databases are hosted on one server. Of course, there is a solution to this problem, it is called sharding, or in less technical terms: throwing huge amounts of hardware at the problem. But our users have made it very clear that they consider cloud sync essential and don’t want it to become an expensive service.

Doing it Right the Wrong Way

Teams of programmers collaboratively write code by making use of so called source code management systems. In 2009 we switched our source code management system from Subversion to Git. Working with Git provided us with a lot of inspiration. Git uses a decentralized approach where contributors can work with their own local repositories. This setup is very similar to the syncing problem where data needs to travel between local databases.

We were so intrigued that we decided to develop a sync solution based on Git’s core ideas. Since these were general ideas anyway, we decided to create a solution that isn’t tied to the specific properties or needs of Things. Instead we wanted to create a general framework that could be integrated with any application no matter what the specific data model or sync policies of this application were.

But we didn’t stop there. If you create something new, knowing that the technology will be needed on multiple platforms, it is worth thinking about a cross platform strategy. We ended up with detailed plans to create a JavaScript-based cross-platform data model framework with Git-inspired sync built in. This strategy required substantial portions of all versions of Things to be rewritten. It was clearly the most ambitious project we ever took on. Dissatisfied with our previous attempts, we didn’t want to settle with anything short of perfection.

However, the complexities were huge; technical problems – such as the integration of JavaScript and Cocoa on iOS – piled up, due dates passed. We slowly began to realize that we wanted too much at once.

Conclusion

We have tried many things, from underpowered technologies to over-engineered solutions. The approach we finally settled on is one that strikes the right balance, and in our next article we'll be sharing more information about what that means.

Some people said we shouldn’t have pursued cloud sync with this level of ambition. But then, that wouldn’t be us. It is not how we developed Things. We know that people are coming to Cultured Code because we take this approach. They like companies that care, companies that try - and that is what we will keep doing.

This blog post is about the cloud sync solution we are working on. Finally, some might say. It has been about two years since we first thought about the option to create a cloud sync solution for Things. A lot has happened since then, and we learned many lessons. It was a long and winding road for us, but most of it happened behind closed doors. In the words of one user, our progress has been glacial – and as I sat down to write this post, it was this metaphor that came to mind. Why? Yes, a glacier is slow, but this object – naturally impressive and beautiful – carves its own path, and despite any obstacles, must arrive.

I am not sure whether “impressive” is the most appropriate term for the cloud sync technology we are working on, but it undoubtedly constitutes a most significant feature in all our products – existing or forthcoming. And to us, our cloud sync architecture is certainly beautiful. But why has it taken this long? The quickest way to answer this question is to quote Ken Arnold:

To stop worrying about it will require worrying about it a lot at first.

And that is exactly what we did. With what will be a forthcoming series of blog posts, I will not only look back to give an idea of what it’s like when a software company is worrying – but I will also share some details about the technology that will underpin our solution and what it will mean for our users.

Before getting into those details, it will be useful to explain just what kind of problem a sync solution is trying to solve. Today’s blog post will therefore concentrate on that, but before closing I’ll also offer an overview of where we stand right now.

Perfect Networks

If we lived in a world that had perfect networks, the sync problem would not exist. A perfect network is one that never fails and has unrestricted speeds. Access to remote servers would be instant. With a perfect network, it wouldn’t be necessary to store data on our devices. All our computers and mobile devices could simply connect to a remote server to store and retrieve data. This way all of our devices would have the same data available to them all the time.

But, of course, the network is not perfect. In order to provide a great user experience, it’s still necessary to store data locally on the device. For this reason, all versions of Things come with their own database. Now, in order to make sure the same data is present on each device – no matter where it’s added or changed – the Things databases need to be synced.

Looking for a Common State

What exactly does this mean? Let’s look at an example. A user might have modified her Things database on the iPhone while not being connected to the internet. Back at home she might make changes on the Mac before the iPhone had a chance to send its changes. Now we have a situation where the iPhone and Mac databases have diverged, no longer sharing a common state.

In order to remedy this situation, data has to travel both ways between the devices and undergo a process to make sure that the resulting databases ultimately reach a common state again. This process is called merging. In the example above, assume that the user reordered to-dos in a project on the iPhone and deleted or created to-dos in the very same project on the Mac. During the merge process the devices need to agree on a final state for the project, distribute all necessary data, and make sure the resulting changes are applied correctly.

Conflicts

The merge process gets more complicated in the presence of conflicts. Imagine a user checked off a to-do on one device but deleted it on another. In order to resolve the conflict, a decision has to be made. The deletion can either be ignored or applied; the latter effectively resulting in the completion of the to-do being ignored.

It gets even worse with objects needing to maintain or change their relationships with other objects – like to-dos, projects, areas, and tags; to-dos can be contained in projects; to-dos and projects can be contained in areas; all three of them may have tags. Changing these complex relationships on multiple devices at the same time creates ample opportunity for conflicts. It is the responsibility of the merge algorithm to resolve these conflicts and to make sure all resulting changes are distributed consistently among all devices.

Simply Make It Work?

Providing a sync solution is a complex problem. As might be apparent from what I outlined above, sync bugs have the potential to seriously mess up a user’s data. It is therefore very important to have a solid foundation. But this still does not describe the entire problem. The most difficult part is not to simply make it work, but to create a fast implementation that can scale to millions of users without diminishing the user experience.

Why is a fast sync process so important? When you launch Things, you want your database to be up to date and contain all your to-dos, no matter where you entered them. Since Things cannot update itself in the background due to iOS restrictions, this means that syncing has to take place when the app is launched. A similar situation arises when you enter new todos on your device and quit Things immediately. Add to this the possibility that a network connection might drop at any time. Every conceivable troublesome scenario points to one thing: fast sync is necessary for a good user experience.

Finally, we must consider scalability. Creating a solution for a few thousand users is one thing – creating a solution for millions of users is a different beast entirely. We have all experienced what happens when a web service is accessed by more people than it was designed for; at first, the service becomes slow, then it fails entirely. It has been our primary goal to create an architecture where scalability was not an afterthought, but rather built-in from the beginning.

A True Cloud Sync Solution

This is the first of a series of articles about our cloud sync solution. In the following installments I will talk about the various approaches and technologies we tried while working toward our goal, and why we did not continue with them. I might also touch on a few popular approaches that others have taken, and show why the trade-offs involved were not acceptable to us. I will talk about the lessons we have learned and the final solution which, at last, satisfies all the requirements we feel a true cloud sync solution demands.

Before closing this article, I would like to offer a cursory glance of where we’re at right now: We have created and deployed both server and client-side sync components. Both components are completely general and can be used for any application. They have been successfully tested using a special demo program. We are now in the process of integrating this technology into Things.

The final release of cloud sync as part of Things is still off by a few months. But we plan to publish more details about what we are doing (and have been doing) every few weeks.

Let me end this post by expressing our sincere gratitude for your patience. Driven by ambitions that were almost too high, it has taken us much longer than we expected. On a path lined with unanticipated obstacles and letdowns, it felt at times as if we would never get there – but we kept believing that we would be able to create a fine solution; a foundation for many cool things to come.

It is now in reach.