Impact:

Beginning on 3/22/24 at approx. 11:30 PM UTC, installed Secret Server SDKs may have been unable to retrieve Secret credentials.

Issue: A change to enhance security for IP Addresses introduced an issue where the IP Address was not correctly retrieved for calls from the Secret Server SDK. This blocked the SDK from authenticating in order to retrieve credentials due to internal ongoing security vetting of calls from the SDK. SDKs with a pre-existing valid token would continue to function correctly until their token expired and they needed to re-authenticate to get a new token.

Resolution:
The issue was resolved by rolling back the update. By 10:49 PM UTC on 3/23/24 the issue was resolved.

Action Items:

To address this and prevent future occurrences:

Add automation tests to cover installation of the Secret Server SDK and retrieval of credentials using it.
Add missing unit tests to cover the specific code area that was responsible for the issue.
Add additional monitoring for anomalous changes in the authentication endpoint to provide back-end visibility of SDK calls.