The major new feature of our just-released versions of Things for iOS - Things for iPhone 1.7 and Things for iPad 1.4 - is the ability to create and edit repeating tasks directly on iOS devices. They can also be synced to the Mac version, which is enabled by the new update of Things for Mac.

Designing an interface for repeating tasks is a challenging task for any platform, let alone for small iOS devices. In this blog post I want to give you an overview of our design goals and the solution we came up with.

Those of you who have been following the development of Things for Mac since it was in beta might remember the challenges we have faced in coming up with a good UI for repeating tasks for the Mac. We wanted to have a powerful and flexible implementation that was still easy to use. After an extensive design journey with many iterations, we were very happy and proud of the outcome. Two years later, and after lots of customer feedback, we see that we did most things right. As always, there is room for improvement – but that’s another chapter.

When we started designing repeating tasks for iOS, we asked ourselves if it was possible to bring the same expressive power to iOS, while still keeping the UI clean and easy to use. Given the constraints of a small device like the iPhone, we knew that this was not going to be easy.

These were our goals when designing the repeating task UI:

1. Expressiveness

   Repeating tasks come in many different varieties and complexities. If you look at a list of typical repeating tasks people have, you will soon realize that there is no single prototype recurrence pattern they can all be mapped to. Examples are: "pay rent at the end of each month", "make backups every 2 weeks", "for the next 5 months, create a performance report at the beginning of the month", "perform health exercises every Monday and Thursday", "prepare for Macworld Expo every January 1st", "prepare club meeting every second Sunday of the month". A repeating tasks implementation that only has the expressive power to specify a subset of the tasks above arbitrarily constrains the user in the tasks he can manage efficiently.

2. Discoverability

   The UI should make it obvious to the user what kind of recurrence patterns can be specified. Expressiveness is of little use when users need to read extensive documentation of how to properly enter certain recurrence patterns.

3. Ease of use

   This goal is obvious, but what exactly does "easy to use" mean in the context of repeating tasks? For regular task entry, one very important aspect of "easy to use" is that entering a new task requires very few taps. As something that is done over and over each day, any extra taps quickly turn into an annoyance for the user. This is why our quick entry is designed to keep the number of taps to a minimum. The same does not equally apply to repeating tasks. Typically, repeating tasks are entered much less frequently. Some are even entered once when you start using the app and remain there for years ("pay rent" is one such example). So rather than speed of entry, it is much more important that the UI has a great clarity and conciseness when it comes to specifying the different recurrence patterns.

With these goals as our guides, we evaluated existing implementations, sketched out different designs, discussed the merits and disadvantages of each, and finally arrived at the version we've now shipped. We believe it is the best and most elegant repeating tasks implementation currently available on iOS. Let me walk you through it.

Just like on the Mac version, repeating tasks can be created from within the Scheduled list. For that, we have a new button in the toolbar.

Note how there is one specific place in the UI to go to when you want to create a new repeating task. The existing UI for adding regular tasks is left untouched, so adding this feature didn't complicate other parts of the application. This is very much in line with the observation that repeating tasks are not entered frequently.

When you tap the button, a new modal view comes up that is specifically designed for repeating tasks.

At the very top, we show an English sentence that describes the current recurrence rule. Below, you can enter the title, tags, and notes of the repeating task, and also specify whether it should create instances with due dates.

Note that we pre-populate the recurrence rule based on the current day of the week. In most cases, the rule you want to specify probably differs from the one we show by default. But then why do we show a default rule in the first place?

We believe that it communicates two things very nicely. First, we can present a view that already contains all the elements of a repeating task: the recurrence rule, together with the title, notes, tags, and due date. So immediately, you know what you are expected to fill out before you can save the task. Secondly, if you don't like the recurrence rule, you will immediately know that you need to tap the description text in order to change it. And what's really great about this, even before you see the next UI view with the different controls that let you specify a different rule, you already know what the end result of those settings will be: a new English sentence that accurately describes the rule you had in mind.

This is the view that appears when you tap the recurrence rule description:

When looking at the general layout of this view, you will realize that – when read from top to bottom – this is basically the same English sentence you saw before, except it is now divided into different controls that let you specify the different parts of the recurrence rule to suit your needs.

Let's now change this rule to "Every 2 months on the first and third Sunday". First, "Every week" is obviously wrong, so we tap on that. A new view slides in, letting us specify the "Every 2 months" part.

After we're done with that, we go back to the screen before, and tap on the "On" section. In the view that comes up, we select the 1st Sunday, tap on "Add More", and also select the 3rd Sunday.

Note that we can additionally specify when the recurrence should begin, and when it should end (e.g. after 5 recurrences, or on a specific date), but we won't do that for now. Tapping "Save" brings us back to the original screen where the English sentence now properly describes our new recurrence rule.

After we enter the additional repeating task properties and tap “Save”, the new task shows up inside the Scheduled list.

If you later revisit the repeating task, you will conveniently see upcoming recurrences:

This wraps up the walkthrough of the repeating tasks feature. I hope you got a good first impression, but you should definitely try them out on your device to get a complete picture. There are additional details not covered in this example:

• If you specify weekly recurrences, we have an optimized picker that lets you easily select multiple weekdays.
• You can also create tasks that repeat after the completion of the previous one.
• You can specify the number of times a repetition should occur.
• You can pause and resume repeating tasks (very useful, if you're going on vacation).
• If you use Areas, you can assign new repeating tasks directly when you create them.

We hope you enjoy using repeating tasks on iOS. Feedback is, as always, greatly welcome.

My last post received quite a bit of criticism. Some people even said I shouldn’t have written it at all. It has been claimed that the sync problem is solved already. Our experience has shown that this couldn’t be farther from the truth – in particular for Cocoa developers. Dave Peck observed this too, and made some wise suggestions. Here is a quote from his article:

If you're a die-hard Cocoa developer, the thought of building a scalable and reliable web back-end might just be daunting -- daunting enough to cut it out entirely. I've noticed a strong resistance in the Cocoa community to building such things.

The Mac community is a great ecosystem and we’ve benefitted a lot from it. Things could not exist without it. This is why creating a cloud sync solution has come to mean more to us than just an improvement of our existing WiFi-based sync. In fact, one of the motivations behind this series of posts, is to start sharing our insights with the community of Cocoa developers.

The first thing you need to do when you want to solve a problem is to get a good understanding of the nature of the problem. The previous post attempted to explain the sync problem in the least technical terms possible – to make it accessible to a non-technical audience as well. Eventually, however, we will be making more technical documents available – directly addressing fellow developers.

Today, I want to talk about the technologies we’ve tried.

Bonjour

Back in the day when Things for the Mac was still in beta we added WiFi sync using Apple’s Bonjour technology. WiFi sync was Apple’s recommended way to realize sync. There are advantages indeed. For example, there is no need to sign up for a web service. It doesn’t even require a pre-existing local network, as any Mac can create its own spontaneous network. The user’s data never leaves the local network.

However, there are notable disadvantages as well: you actually have to remember to sync, and you have to initiate it manually. Relying on the local network, of course, also eliminates the possibility of syncing your work Mac with your Mac at home.

Obviously, we had to provide a solution that would utilize the cloud instead of the local network. The most obvious choice is perhaps MobileMe.

MobileMe

There are quite a number of people using Apple’s MobileMe to sync, e.g., calendar or address book information. For such users, it is only reasonable to wish that Things would sync in the same way, using their existing MobileMe subscription. Unfortunately, the sync technology Apple uses for their own apps is not available to third party developers on iOS devices.

It is available on the Mac though, and we hoped it would eventually find its way onto iOS; this hasn’t happened yet. What some competitors are in fact referring to, when they talk about MobileMe sync, is the use of remote storage through the iDisk.

Dropbox, WebDAV et. al.

Some users have suggested using Dropbox for sync. Like many similar offerings – often based on the so-called WebDAV protocol – Dropbox allows users to store files remotely on their servers. In this sense, it is similar to Apple’s iDisk (assuming iDisk sync is turned on) – what both services do is provide a folder on your computer that is mirrored on a remote server.

Neither Dropbox nor iDisk – or any similar service for that matter – were conceived as a sync solution for apps like Things. They were designed for sharing files – like photos or PDFs. Remote file storage products do not offer merge facilities that come even close to what would be needed for Things.

What is possible, however – and others are in fact doing this – is to use a hack. Without going into detail, it basically means breaking up the database file into a large number of smaller files. Merging and conflict resolution can now be handled using these smaller fragments, which eases some of the pain.

This approach works to some extent, but it is slow and error prone to begin with, and advanced options like push or anything involving user-to-user data exchange is impossible. Developers choosing this approach for apps with complex data models similar to Things paint themselves into a corner right from the start.

Database in the Cloud

Instead of using remote file storage, it is a much better idea to use a custom-designed web service. This is an approach taken by most web applications, and a whole industry has arisen developing, deploying, and maintaining applications that are written in this style. Everyone going this route has ample expertise and help available to them.

In taking advantage of this, we began working together with a great group of web development people. It was a great experience, and one we wouldn’t want to have missed – but eventually we abandoned this approach. Here is why:

If you are thinking in terms of a web application, you are basically considering placing the user’s entire database in the cloud, together with enough logic to safely manipulate that data. This makes the database in the cloud the authoritative version which can be used to determine how the data on every device should look. This sounds like a great thing to have, but it requires that all merging and conflict resolution be done on the server - and this turns out to be really slow.

Merging potentially requires accessing the database, and hence the hard-drive, very often. Hard-drive operations, however, are the most expensive in terms of performance – in particular when many databases are hosted on one server. Of course, there is a solution to this problem, it is called sharding, or in less technical terms: throwing huge amounts of hardware at the problem. But our users have made it very clear that they consider cloud sync essential and don’t want it to become an expensive service.

Doing it Right the Wrong Way

Teams of programmers collaboratively write code by making use of so called source code management systems. In 2009 we switched our source code management system from Subversion to Git. Working with Git provided us with a lot of inspiration. Git uses a decentralized approach where contributors can work with their own local repositories. This setup is very similar to the syncing problem where data needs to travel between local databases.

We were so intrigued that we decided to develop a sync solution based on Git’s core ideas. Since these were general ideas anyway, we decided to create a solution that isn’t tied to the specific properties or needs of Things. Instead we wanted to create a general framework that could be integrated with any application no matter what the specific data model or sync policies of this application were.

But we didn’t stop there. If you create something new, knowing that the technology will be needed on multiple platforms, it is worth thinking about a cross platform strategy. We ended up with detailed plans to create a JavaScript-based cross-platform data model framework with Git-inspired sync built in. This strategy required substantial portions of all versions of Things to be rewritten. It was clearly the most ambitious project we ever took on. Dissatisfied with our previous attempts, we didn’t want to settle with anything short of perfection.

However, the complexities were huge; technical problems – such as the integration of JavaScript and Cocoa on iOS – piled up, due dates passed. We slowly began to realize that we wanted too much at once.

Conclusion

We have tried many things, from underpowered technologies to over-engineered solutions. The approach we finally settled on is one that strikes the right balance, and in our next article we'll be sharing more information about what that means.

Some people said we shouldn’t have pursued cloud sync with this level of ambition. But then, that wouldn’t be us. It is not how we developed Things. We know that people are coming to Cultured Code because we take this approach. They like companies that care, companies that try - and that is what we will keep doing.

This blog post is about the cloud sync solution we are working on. Finally, some might say. It has been about two years since we first thought about the option to create a cloud sync solution for Things. A lot has happened since then, and we learned many lessons. It was a long and winding road for us, but most of it happened behind closed doors. In the words of one user, our progress has been glacial – and as I sat down to write this post, it was this metaphor that came to mind. Why? Yes, a glacier is slow, but this object – naturally impressive and beautiful – carves its own path, and despite any obstacles, must arrive.

I am not sure whether “impressive” is the most appropriate term for the cloud sync technology we are working on, but it undoubtedly constitutes a most significant feature in all our products – existing or forthcoming. And to us, our cloud sync architecture is certainly beautiful. But why has it taken this long? The quickest way to answer this question is to quote Ken Arnold:

To stop worrying about it will require worrying about it a lot at first.

And that is exactly what we did. With what will be a forthcoming series of blog posts, I will not only look back to give an idea of what it’s like when a software company is worrying – but I will also share some details about the technology that will underpin our solution and what it will mean for our users.

Before getting into those details, it will be useful to explain just what kind of problem a sync solution is trying to solve. Today’s blog post will therefore concentrate on that, but before closing I’ll also offer an overview of where we stand right now.

Perfect Networks

If we lived in a world that had perfect networks, the sync problem would not exist. A perfect network is one that never fails and has unrestricted speeds. Access to remote servers would be instant. With a perfect network, it wouldn’t be necessary to store data on our devices. All our computers and mobile devices could simply connect to a remote server to store and retrieve data. This way all of our devices would have the same data available to them all the time.

But, of course, the network is not perfect. In order to provide a great user experience, it’s still necessary to store data locally on the device. For this reason, all versions of Things come with their own database. Now, in order to make sure the same data is present on each device – no matter where it’s added or changed – the Things databases need to be synced.

Looking for a Common State

What exactly does this mean? Let’s look at an example. A user might have modified her Things database on the iPhone while not being connected to the internet. Back at home she might make changes on the Mac before the iPhone had a chance to send its changes. Now we have a situation where the iPhone and Mac databases have diverged, no longer sharing a common state.

In order to remedy this situation, data has to travel both ways between the devices and undergo a process to make sure that the resulting databases ultimately reach a common state again. This process is called merging. In the example above, assume that the user reordered to-dos in a project on the iPhone and deleted or created to-dos in the very same project on the Mac. During the merge process the devices need to agree on a final state for the project, distribute all necessary data, and make sure the resulting changes are applied correctly.

Conflicts

The merge process gets more complicated in the presence of conflicts. Imagine a user checked off a to-do on one device but deleted it on another. In order to resolve the conflict, a decision has to be made. The deletion can either be ignored or applied; the latter effectively resulting in the completion of the to-do being ignored.

It gets even worse with objects needing to maintain or change their relationships with other objects – like to-dos, projects, areas, and tags; to-dos can be contained in projects; to-dos and projects can be contained in areas; all three of them may have tags. Changing these complex relationships on multiple devices at the same time creates ample opportunity for conflicts. It is the responsibility of the merge algorithm to resolve these conflicts and to make sure all resulting changes are distributed consistently among all devices.

Simply Make It Work?

Providing a sync solution is a complex problem. As might be apparent from what I outlined above, sync bugs have the potential to seriously mess up a user’s data. It is therefore very important to have a solid foundation. But this still does not describe the entire problem. The most difficult part is not to simply make it work, but to create a fast implementation that can scale to millions of users without diminishing the user experience.

Why is a fast sync process so important? When you launch Things, you want your database to be up to date and contain all your to-dos, no matter where you entered them. Since Things cannot update itself in the background due to iOS restrictions, this means that syncing has to take place when the app is launched. A similar situation arises when you enter new todos on your device and quit Things immediately. Add to this the possibility that a network connection might drop at any time. Every conceivable troublesome scenario points to one thing: fast sync is necessary for a good user experience.

Finally, we must consider scalability. Creating a solution for a few thousand users is one thing – creating a solution for millions of users is a different beast entirely. We have all experienced what happens when a web service is accessed by more people than it was designed for; at first, the service becomes slow, then it fails entirely. It has been our primary goal to create an architecture where scalability was not an afterthought, but rather built-in from the beginning.

A True Cloud Sync Solution

This is the first of a series of articles about our cloud sync solution. In the following installments I will talk about the various approaches and technologies we tried while working toward our goal, and why we did not continue with them. I might also touch on a few popular approaches that others have taken, and show why the trade-offs involved were not acceptable to us. I will talk about the lessons we have learned and the final solution which, at last, satisfies all the requirements we feel a true cloud sync solution demands.

Before closing this article, I would like to offer a cursory glance of where we’re at right now: We have created and deployed both server and client-side sync components. Both components are completely general and can be used for any application. They have been successfully tested using a special demo program. We are now in the process of integrating this technology into Things.

The final release of cloud sync as part of Things is still off by a few months. But we plan to publish more details about what we are doing (and have been doing) every few weeks.

Let me end this post by expressing our sincere gratitude for your patience. Driven by ambitions that were almost too high, it has taken us much longer than we expected. On a path lined with unanticipated obstacles and letdowns, it felt at times as if we would never get there – but we kept believing that we would be able to create a fine solution; a foundation for many cool things to come.

It is now in reach.

If it is not just a different “form factor”, if it is not just old software with the “touch experience” added, then what is it? What makes the iPad so special we just had to develop Things for iPad?

For sure, the introduction of the iPad today marks the arrival of a new platform. And there is hardly anything more exciting for a software developer than the introduction of a new platform. But this isn’t just any platform.

If you want to understand what makes the iPad special, you cannot look at what it has, but what it doesn’t have. The iPad is so thin and light, it becomes the display, and the display becomes the application. No input devices. The device vanishes and turns into the application you are using. The technology is transparent.

Steve Jobs said about the iPod that “it is all about the music”. With the iPad, Apple has done the same for personal computing as it has done before with the iPod: it made technology go away. But if the device is gone, and the operating system is gone, what is left?

The iPad is an empty canvas that invites us to imagine what is possible. It inspires our imagination and it makes us want to create, because never before were we able to create software that was so close to the user.

Developing Things for iPad in such a short amount of time was exciting, it was hard work, and it was a whole lot of fun. You can see how much fun we had by watching the Things for iPad introduction video. Please enjoy it.