A refresh-token race broke the connection

Two agent processes tried to refresh the same token less than a second apart. The first request rotated it. The second used the old token, and the server revoked the connection. I wanted to preserve a legitimate connection when two clients refreshed at once without accepting a token the server had already replaced.

The server change rejected reuse within a minute of rotation and kept the connection valid. After that window, reuse still triggered revocation. I could check both outcomes in tests. The production logs raised a different question about whether clients could recover.

Concurrent use of a rotated token

After a refresh token has been exchanged, presenting its old value requires an explicit server policy. The server has limited information about why a second request arrived. It may be a replay or an attempt by another process sharing the same local credential.

Concurrent processes can begin with the same token and send refresh requests close together. If the second request also revokes the connection, that overlap can leave both processes without a usable grant. Further retries with stale state cannot restore a revoked grant.

The change preserved rejection of the reused token. A replay within a minute would receive no new credentials, but the connection would remain valid. Reuse after the grace window retained the revocation behaviour.

That grace window has a security trade-off. If a thief exchanges a stolen refresh token first and the legitimate client replays it within a minute, the replay no longer revokes the grant. The thief can keep using the new token chain until the grant is revoked or expires. Refusing the replay therefore does not resolve this theft scenario. The change tolerates concurrent refreshes at the cost of automatic revocation in that case.

Refusals also gained an explanation of the cause and the next action. A client needs enough information to select a recovery step, which may require a person to authorise a new connection. The server change and its tests did not establish how a released client handled that advice or updated its stored credentials.

Tracking refusals across an error-message change

I counted retries and queried production logs over different windows, so I couldn’t use the figures to compare recovery rates before and after the change.

In the logs, I could see responses using both the old expired-grant wording and a replacement disconnected message. The old text later disappeared, while refusals continued under the replacement text. A monitor searching only for the original message would have missed them.

In a subsequent sample, I still saw hundreds of disconnected responses alongside other token errors and some successful refreshes. Repeated retries could contribute several observations from one connection, so those counts could not establish the number of affected accounts. Client labels were also reported claims.

Disconnected-response counts had fallen in part of the period I reviewed, but the cause was unknown. The queries could not associate refused requests with a grant, establish whether the clients recovered or record human intervention. A recovery rate would require observations tied to comparable attempts and known starting states.

What the server tests covered

The concurrency tests checked that refusing reuse within a minute preserved the connection and that reuse after the grace window still revoked it. Those tests provide a regression check for the server's behaviour.

Production recovery requires additional evidence. Preventing a new revocation does not restore an already revoked grant, so existing broken connections may continue producing errors after deployment. A fall in retry volume could reflect recovery, reduced use or a client stopping. The counts alone cannot distinguish those possibilities.

The next action depends on the connection's state. A valid grant may still be unusable to a client that cannot update its credentials. A revoked grant may require a new authorisation. Error responses should make the required action clear without exposing credentials.

Verify recovery from the client

The server tests tell me that a near-simultaneous refresh can leave the grant valid. That helps only if the affected client can obtain and use a valid credential. The production logs I reviewed showed that refusals continued, but I couldn’t follow a refused request through to a recovered session.

I’d test a released client with a dedicated connection and a known credential state. After it gets the refusal, I’d watch whether it retries, uses updated credentials or asks for authorisation. A successful protected operation through that connection would show that it recovered. The server’s refusal count can’t tell me that on its own.