There Is No Draft State

Change a value in the vault, redeploy the service so it picks the change up, watch it come back with the old value anyway.

That is not a caching bug. A controller copies vault values into the cluster on its own schedule, roughly hourly, and the process reads its environment once when it starts. Redeploying immediately restarts the process against a secret that has not been updated yet, which means the fastest thing you can do is also the thing that proves nothing.

One of my own scripts used to tell people to do exactly this. Change the value, run the pipeline, done. It had been wrong the entire time and nobody noticed, because when the result looked wrong the natural conclusion was that the value was wrong rather than that it had never arrived.

It runs in the other direction too

That part is annoying. This part is worse.

If the sync happens on a schedule, then a value you changed but did not intend to activate yet still reaches the cluster within the hour. It then takes effect at the next process start, whenever that is. On our development cluster the machines are preemptible, so restarts happen on their own, at times nobody chooses.

So there is no such thing as getting a secret ready. You cannot write a new callback URL or hostname or credential and leave it waiting for the rest of the work to land. Saving the value starts the clock, and the clock does not care whether the thing it depends on exists yet.

Writing it is shipping it, on a timer you do not own.

The rule I follow now is simple. Change a secret in the same sitting as the things that depend on it, or do not change it yet.

Finding the stale layer

When a value is not behaving, the useful question is which of the four places it lives is still holding the old one. Check them in order instead of guessing.

What the vault says, and when it was last updated. What the sync object reports as its last refresh. What is actually stored in the cluster. And finally what the running process has in its environment, which is the only one that determines behaviour.

Each hop can be the stale one, and the ones nearer the start are more reassuring than they deserve to be. A correct value in the vault tells you about your intention, not about your system.

That is the same shape as a few other things I have written about here. The alert that reported recovery while everything was down. The grep that returned nothing because the command never ran. In all three cases the check answered a question confidently, and it was not the question being asked.

To apply a change immediately, force the sync and then restart the process, in that order. Both steps are needed. Doing the second without the first is where this started.