Sync paused for 2 hours on Tuesday 13 October

Before this: sync paused from 14:10 to 16:10, and no note was lost.

The notes app keeps working without a connection. Reading and writing never stopped. What stopped was the part that copies changes between a person's devices.

Take from this: the cost was a two-hour delay, and the people who felt it most use two devices.

Remember: writing never stopped, so the cost was a two-hour delay, not lost notes.

Before this: the app kept working, so the cost was a two-hour delay.

The two hours fall into three parts. The alert reached the team 15 minutes after sync stopped. Finding the cause took 55 minutes. Shipping the fix took 50.

Take from this: almost half the time went on finding the cause, so a clearer alert saves the most.

The cause hid well. The logs showed time-outs, not a lock, and each restart cleared the one clue the database held.

14:05  Deploy starts. The start-up script begins the index build.
14:10  Sync stops answering. The health check restarts the server.
14:22  Second restart. The build begins again. Restarts repeat every 12 minutes.
14:25  The sync alert reaches the team.
15:20  Cause found: every start re-runs the unfinished build.
15:25  A deploy that skips migrations at start-up begins.
15:50  That deploy finishes and the server starts clean.
16:00  Saves work again. The retry backlog drains.
16:10  Sync is normal.
16:40  The last queued change arrives.

Remember: finding the cause took 55 of the 120 minutes, so a clearer alert saves the most time.

Before this: almost half the time went on finding a cause that the logs did not show.

One change set off four more, and the last one kept the others going. Read the chain from left to right. The red step turned minutes into hours.

Migration

A change to the shape of the database, run as code.

Index

A lookup table that makes searches fast. Building one on a big table blocks saves.

Lock

A hold that makes other work wait while a change runs.

1. Index addedA deploy adds it for sharing, on the live notes table.
2. Table locksThe build blocks every save for about 9 minutes.
3. Phones retryRequests time out at 30 seconds, and 190 apps retry at once.
4. Server restartsThe health check sees no answer and restarts it.
5. Build starts overEach start re-runs the unfinished change, so the lock never clears.

Take from this: the build alone would have ended in nine minutes. The restart loop stretched it to two hours.

Remember: the restart loop, not the index, turned a nine-minute lock into two hours.

Before this: a restart loop turned a nine-minute lock into two hours.

The review produced five follow-ups. The one that stopped the outage is already done.

• Stop migrations at start-upDone on Wednesday 14 October. Someone now starts each migration by hand, once.
• Rehearse on a full-size copyDone. The test copy holds 400,000 notes, so a slow change shows up before it ships.
• Alert after 3 minutesDone. The sync alert used to wait for 10 minutes of failures.
• Build indexes in the backgroundUnder way. Ted is changing the build so saves keep working. Due Friday 23 October.
• Quiet hours for deploysNeeds a ruling from Mara: no deploys from 13:00 to 17:00 on weekdays?

Take from this: the outage cannot repeat as it happened, and the two open items shrink the next one.

- run-migrations && start-server
+ start-server
# migrations now run only by hand:  npm run migrate

Remember: three follow-ups are done, one is under way, and one waits for Mara's ruling.

Before this: three follow-ups are done, and two are still open.

Each follow-up guards a different weak point. A slow change can still happen, but it can no longer stop sync for two hours.

Take from this: four things changed since Tuesday, and a fifth, quiet hours for deploys, waits on a ruling.

Remember: each follow-up guards a different weak point, so one slip cannot take sync down for two hours again.