EuAndreh's website

Local-First Software: article review

2020-11-14T00:00:00+00:00

This article is derived from a presentation given at a Papers We Love meetup on the same subject.

This is a review of the article "Local-First Software: You Own Your Data, in spite of the Cloud", by M. Kleppmann, A. Wiggins, P. Van Hardenberg and M. F. McGranaghan.

Offline-first, local-first

The "local-first" term they use isn’t new, and I have used it myself in the past to refer to this types of application, where the data lives primarily on the client, and there are conflict resolution algorithms that reconcile data created on different instances.

Sometimes I see confusion with this idea and "client-side", "offline-friendly", "syncable", etc. I have myself used this terms, also.

There exists, however, already the "offline-first" term, which conveys almost all of that meaning. In my view, "local-first" doesn’t extend "offline-first" in any aspect, rather it gives a well-defined meaning to it instead. I could say that "local-first" is just "offline-first", but with 7 well-defined ideals instead of community best practices.

It is a step forward, and given the number of times I’ve seen the paper shared around I think there’s a chance people will prefer saying "local-first" in lieu of "offline-first" from now on.

Software licenses

On a footnote of the 7th ideal ("You Retain Ultimate Ownership and Control"), the authors say:

In our opinion, maintaining control and ownership of data does not mean that the software must necessarily be open source. (…) as long as it does not artificially restrict what users can do with their files.

They give examples of artificial restrictions, like this artificial restriction I’ve come up with:

#!/bin/sh

TODAY=$(date +%s)
LICENSE_EXPIRATION=$(date -d 2020-11-15 +%s)

if [ $TODAY -ge $LICENSE_EXPIRATION ]; then
  echo 'License expired!'
  exit 1
fi

echo $((2 + 2))

plaintext

Now when using this very useful program:

# today
$ ./useful-adder.sh
4
# tomorrow
$ ./useful-adder.sh
License expired!

plaintext

This is obviously an intentional restriction, and it goes against the 5th ideal ("The Long Now"). This software would only be useful as long as the embedded license expiration allowed. Sure you could change the clock on the computer, but there are many other ways that this type of intentional restriction is in conflict with that ideal.

However, what about unintentional restrictions? What if a software had an equal or similar restriction, and stopped working after days pass? Or what if the programmer added a constant to make the development simpler, and this led to unintentionally restricting the user?

# today
$ useful-program
# ...useful output...

# tomorrow, with more data
$ useful-program
ERROR: Panic! Stack overflow!

plaintext

Just as easily as I can come up with ways to intentionally restrict users, I can do the same for unintentionally restrictions. A program can stop working for a variety of reasons.

If it stops working due do, say, data growth, what are the options? Reverting to an earlier backup, and making it read-only? That isn’t really a "Long Now", but rather a "Long Now as long as the software keeps working as expected".

The point is: if the software isn’t free, "The Long Now" isn’t achievable without a lot of wishful thinking. Maybe the authors were trying to be more friendly towards business who don’t like free software, but in doing so they’ve proposed a contradiction by reconciling "The Long Now" with proprietary software.

It isn’t the same as saying that any free software achieves that ideal, either. The license can still be free, but the source code can become unavailable due to cloud rot. Or maybe the build is undocumented, or the build tools had specific configuration that one has to guess. A piece of free software can still fail to achieve "The Long Now". Being free doesn’t guarantee it, just makes it possible.

A colleague has challenged my view, arguing that the software doesn’t really need to be free, as long as there is an specification of the file format. This way if the software stops working, the format can still be processed by other programs. But this doesn’t apply in practice: if you have a document that you write to, and software stops working, you still want to write to the document. An external tool that navigates the content and shows it to you won’t allow you to keep writing, and when it does that tool is now starting to re-implement the software.

An open specification could serve as a blueprint to other implementations, making the data format more friendly to reverse-engineering. But the re-implementation still has to exist, at which point the original software failed to achieve "The Long Now".

It is less bad, but still not quite there yet.

Denial of existing solutions

When describing "Existing Data Storage and Sharing Models", on a footnotefootnote:devil[ This is the second aspect that I’m picking on the article from a footnote. I guess the devil really is on the details. ] the authors say:

In principle it is possible to collaborate without a repository service, e.g. by sending patch files by email, but the majority of Git users rely on GitHub.

The authors go to a great length to talk about usability of cloud apps, and even point to research they’ve done on it, but they’ve missed learning more from local-first solutions that already exist.

Say the automerge CRDT proves to be even more useful than what everybody imagined. Say someone builds a local-first repository service using it. How will it change anything of the Git/GitHub model? What is different about it that prevents people in the future writing a paper saying:

In principle it is possible to collaborate without a repository service, e.g. by using automerge and platform X, but the majority of Git users rely on GitHub.

How is this any better?

If it is already possible to have a local-first development workflow, why don’t people use it? Is it just fashion, or there’s a fundamental problem with it? If so, what is it, and how to avoid it?

If sending patches by emails is perfectly possible but out of fashion, why even talk about Git/GitHub? Isn’t this a problem that people are putting themselves in? How can CRDTs possibly prevent people from doing that?

My impression is that the authors envision a better future, where development is fully decentralized unlike today, and somehow CRDTs will make that happen. If more people think this way, "CRDT" is next in line to the buzzword list that solves everything, like "containers", "blockchain" or "machine learning".

Rather than picturing an imaginary service that could be described like "GitHub+CRDTs" and people would adopt it, I’d rather better understand why people don’t do it already, since Git is built to work like that.

Ditching of web applications

The authors put web application in a worse position for building local-first application, claiming that:

(…) the architecture of web apps remains fundamentally server-centric. Offline support is an afterthought in most web apps, and the result is accordingly fragile.

Well, I disagree.

The problem isn’t inherit to the web platform, but instead how people use it.

I have myself built offline-first applications, leveraging IndexedDB, App Cache, etc. I wanted to build an offline-first application on the web, and so I did.

In fact, many people choose PouchDB because of that, since it is a good tool for offline-first web applications. The problem isn’t really the technology, but how much people want their application to be local-first.

Contrast it with Android Instant Apps, where applications are sent to the phone in small parts. Since this requires an internet connection to move from a part of the app bundle to another, a subset of the app isn’t local-first, despite being an app.

The point isn’t the technology, but how people are using it. Local-first web applications are perfectly possible, just like non-local-first native applications are possible.

Costs are underrated

I think the costs of "old-fashioned apps" over "cloud apps" are underrated, mainly regarding storage, and that this costs can vary a lot by application.

Say a person writes online articles for their personal website, and puts everything into Git. Since there isn’t supposed to be any collaboration, all of the relevant ideals of local-first are achieved.

Now another person creates videos instead of articles. They could try keeping everything local, but after some time the storage usage fills the entire disk. This person’s local-first setup would be much more complex, and would cost much more on maintenance, backup and storage.

Even though both have similar needs, a local-first video repository is much more demanding. So the local-first thinking here isn’t "just keep everything local", but "how much time and money am I willing to spend to keep everything local".

The convenience of "cloud apps" becomes so attractive that many don’t even have a local copy of their videos, and rely exclusively on service providers to maintain, backup and store their content.

The dial measuring "cloud apps" and "old-fashioned apps" needs to be specific to use-cases.

Real-time collaboration is optional

If I were the one making the list of ideals, I wouldn’t focus so much on real-time collaboration.

Even though seamless collaboration is desired, it being real-time depends on the network being available for that. But ideal 3 states that "The Network is Optional", so real-time collaboration is also optional.

The fundamentals of a local-first system should enable real-time collaboration when network is available, but shouldn’t focus on it.

On many places when discussing applications being offline, it is common for me to find people saying that their application works "even on a plane, subway or elevator". That is a reflection of when said developers have to deal with networks being unavailable.

But this leaves out a big chunk of the world where internet connection is intermittent, or only works every other day or only once a week, or stops working when it rains, etc. For this audience, living without network connectivity isn’t such a discrete moment in time, but part of every day life. I like the fact that the authors acknowledge that.

When discussing "working offline", I’d rather keep this type of person in mind, then the subset of people who are offline when on the elevator will naturally be included.

On CRDTs and developer experience

When discussing developer experience, the authors bring up some questions to be answered further, like:

For an app developer, how does the use of a CRDT-based data layer compare to existing storage layers like a SQL database, a filesystem, or CoreData? Is a distributed system harder to write software for?

That is an easy one: yes.

A distributed system is harder to write software for, being a distributed system.

Adding a large layer of data structures and algorithms will make it more complex to write software for, naturally. And if trying to make this layer transparent to the programmer, so they can pretend that layer doesn’t exist is a bad idea, as RPC frameworks have tried, and failed.

See "A Note on Distributed Computing" for a critique on RPC frameworks trying to make the network invisible, which I think also applies in equivalence for making the CRDTs layer invisible.

Conclusion

I liked a lot the article, as it took the "offline-first" philosophy and ran with it.

But I think the authors’ view of adding CRDTs and things becoming local-first is a bit too magical.

This particular area is one that I have large interest on, and I wish to see more being done on the "local-first" space.

Feature flags: differences between backend, frontend and mobile

2020-10-19T00:00:00+00:00

This article is derived from a presentation on the same subject.

When discussing about feature flags, I find that their costs and benefits are often well exposed and addressed. Online articles like "Feature Toggle (aka Feature Flags)" do a great job of explaining them in detail, giving great general guidance of how to apply techniques to adopt it.

However the weight of those costs and benefits apply differently on backend, frontend or mobile, and those differences aren’t covered. In fact, many of them stop making sense, or the decision of adopting a feature flag or not may change depending on the environment.

In this article I try to make the distinction between environments and how feature flags apply to them, with some final best practices I’ve acquired when using them in production.

Why feature flags

Feature flags in general tend to be cited on the context of continuous deployment:

A: With continuous deployment, you deploy to production automatically

B: But how do I handle deployment failures, partial features, etc.?

A: With techniques like canary, monitoring and alarms, feature flags, etc.

Though adopting continuous deployment doesn’t force you to use feature flags, it creates a demand for it. The inverse is also true: using feature flags on the code points you more obviously to continuous deployment. Take the following code sample for example, that we will reference later on the article:

function processTransaction() {
        validate();
        persist();
        // TODO: add call to notifyListeners()
}

plaintext

While being developed, being tested for suitability or something similar, notifyListeners() may not be included in the code at once. So instead of keeping it on a separate, long-lived branch, a feature flag can decide when the new, partially implemented function will be called:

function processTransaction() {
        validate();
        persist();
        if (featureIsEnabled("activate-notify-listeners")) {
                notifyListeners();
        }
}

plaintext

This allows your code to include notifyListeners(), and decide when to call it at runtime. For the price of extra things around the code, you get more dynamicity.

So the fundamental question to ask yourself when considering adding a feature flag should be:

Am I willing to pay with code complexity to get dynamicity?

It is true that you can make the management of feature flags as straightforward as possible, but having no feature flags is simpler than having any. What you get in return is the ability to parameterize the behaviour of the application at runtime, without doing any code changes.

Sometimes this added complexity may tilt the balance towards not using a feature flag, and sometimes the flexibility of changing behaviour at runtime is absolutely worth the added complexity. This can vary a lot by code base, feature, but fundamentally by environment: its much cheaper to deploy a new version of a service than to release a new version of an app.

So the question of which environment is being targeted is key when reasoning about costs and benefits of feature flags.

Control over the environment

The key differentiator that makes the trade-offs apply differently is how much control you have over the environment.

When running a backend service, you usually are paying for the servers themselves, and can tweak them as you wish. This means you have full control do to code changes as you wish. Not only that, you decide when to do it, and for how long the transition will last.

On the frontend you have less control: even though you can choose to make a new version available any time you wish, you can’t forcefootnote:force[ Technically you could force a reload with JavaScript using window.location.reload(), but that not only is invasive and impolite, but also gives you the illusion that you have control over the client when you actually don’t: clients with disabled JavaScript would be immune to such tactics. ] clients to immediately switch to the new version. That means that a) clients could skip upgrades at any time and b) you always have to keep backward and forward compatibility in mind.

Even though I’m mentioning frontend directly, it applies to other environment with similar characteristics: desktop applications, command-line programs, etc.

On mobile you have even less control: app stores need to allow your app to be updated, which could bite you when least desired. Theoretically you could make you APK available on third party stores like F-Droid, or even make the APK itself available for direct download, which would give you the same characteristics of a frontend application, but that happens less often.

On iOS you can’t even do that. You have to get Apple’s blessing on every single update. Even though we already know that is a bad idea for over a decade now, there isn’t a way around it. This is where you have the least control.

In practice, the amount of control you have will change how much you value dynamicity: the less control you have, the more valuable it is. In other words, having a dynamic flag on the backend may or may not be worth it since you could always update the code immediately after, but on iOS it is basically always worth it.

Rollout

A rollout is used to roll out a new version of software.

They are usually short-lived, being relevant as long as the new code is being deployed. The most common rule is percentages.

On the backend, it is common to find it on the deployment infrastructure itself, like canary servers, blue/green deployments, a kubernetes deployment rollout, etc. You could do those manually, by having a dynamic control on the code itself, but rollbacks are cheap enough that people usually do a normal deployment and just give some extra attention to the metrics dashboard.

Any time you see a blue/green deployment, there is a rollout happening: most likely a load balancer is starting to direct traffic to the new server, until reaching 100% of the traffic. Effectively, that is a rollout.

On the frontend, you can selectively pick which user’s will be able to download the new version of a page. You could use geographical region, IP, cookie or something similar to make this decision.

CDN propagation delays and people not refreshing their web pages are also rollouts by themselves, since old and new versions of the software will coexist.

On mobile, the Play Store allows you to perform fine-grained staged rollouts, and the App Store allows you to perform limited phased releases.

Both for Android and iOS, the user plays the role of making the download.

In summary: since you control the servers on the backend, you can do rollouts at will, and those are often found automated away in base infrastructure. On the frontend and on mobile, there are ways to make new versions available, but users may not download them immediately, and many different versions of the software end up coexisting.

Feature flag

A feature flag is a flag that tells the application on runtime to turn on or off a given feature. That means that the actual production code will have more than one possible code paths to go through, and that a new version of a feature coexists with the old version. The feature flag tells which part of the code to go through.

They are usually medium-lived, being relevant as long as the new code is being developed. The most common rules are percentages, allow/deny lists, A/B groups and client version.

On the backend, those are useful for things that have a long development cycle, or that needs to done by steps. Consider loading the feature flag rules in memory when the application starts, so that you avoid querying a database or an external service for applying a feature flag rule and avoid flakiness on the result due to intermittent network failures.

Since on the frontend you don’t control when to update the client software, you’re left with applying the feature flag rule on the server, and exposing the value through an API for maximum dynamicity. This could be in the frontend code itself, and fallback to a "just refresh the page"/"just update to the latest version" strategy for less dynamic scenarios.

On mobile you can’t even rely on a "just update to the latest version" strategy, since the code for the app could be updated to a new feature and be blocked on the store. Those cases aren’t recurrent, but you should always assume the store will deny updates on critical moments so you don’t find yourself with no cards to play. That means the only control you actually have is via the backend, by parameterizing the runtime of the application using the API. In practice, you should always have a feature flag to control any relevant piece of code. There is no such thing as "too small code change for a feature flag". What you should ask yourself is:

If the code I’m writing breaks and stays broken for around a month, do I care?

If you’re doing an experimental screen, or something that will have a very small impact you might answer "no" to the above question. For everything else, the answer will be "yes": bug fixes, layout changes, refactoring, new screen, filesystem/database changes, etc.

Experiment

An experiment is a feature flag where you care about analytical value of the flag, and how it might impact user’s behaviour. A feature flag with analytics.

They are also usually medium-lived, being relevant as long as the new code is being developed. The most common rule is A/B test.

On the backend, an experiment rely on an analytical environment that will pick the A/B test groups and distributions, which means those can’t be held in memory easily. That also means that you’ll need a fallback value in case fetching the group for a given customer fails.

On the frontend and on mobile they are no different from feature flags.

Operational toggle

An operational toggle is like a system-level manual circuit breaker, where you turn on/off a feature, fail over the load to a different server, etc. They are useful switches to have during an incident.

They are usually long-lived, being relevant as long as the code is in production. The most common rule is percentages.

They can be feature flags that are promoted to operational toggles on the backend, or may be purposefully put in place preventively or after a postmortem analysis.

On the frontend and on mobile they are similar to feature flags, where the "feature" is being turned on and off, and the client interprets this value to show if the "feature" is available or unavailable.

Best practices

Prefer dynamic content

Even though feature flags give you more dynamicity, they’re still somewhat manual: you have to create one for a specific feature and change it by hand.

If you find yourself manually updating a feature flags every other day, or tweaking the percentages frequently, consider making it fully dynamic. Try using a dataset that is generated automatically, or computing the content on the fly.

Say you have a configuration screen with a list of options and sub-options, and you’re trying to find how to better structure this list. Instead of using a feature flag for switching between 3 and 5 options, make it fully dynamic. This way you’ll be able to perform other tests that you didn’t plan, and get more flexibility out of it.

Use the client version to negotiate feature flags

After effectively finishing a feature, the old code that coexisted with the new one will be deleted, and all traces of the transition will vanish from the code base. However if you just remove the feature flags from the API, all of the old versions of clients that relied on that value to show the new feature will go downgrade to the old feature.

This means that you should avoid deleting client-facing feature flags, and retire them instead: use the client version to decide when the feature is stable, and return true for every client with a version greater or equal to that. This way you can stop thinking about the feature flag, and you don’t break or downgrade clients that didn’t upgrade past the transition.

Beware of many nested feature flags

Nested flags combine exponentially.

Pick strategic entry points or transitions eligible for feature flags, and beware of their nesting.

Include feature flags in the development workflow

Add feature flags to the list of things to think about during whiteboarding, and deleting/retiring a feature flags at the end of the development.

Always rely on a feature flag on the app

Again, there is no such thing "too small for a feature flag". Too many feature flags is a good problem to have, not the opposite. Automate the process of creating a feature flag to lower its cost.