Centrify Cloud Service Advisory - SMS Delivery delays or failure.
Incident Report for Centrify Cloud Service
Resolved
Engineers have confirmed that full Service availability has been restored for SMS message delivery. This incident is now mitigated.
Posted 10 days ago. Feb 12, 2019 - 11:57 PST
Monitoring
We have completed the configuration changes on all the pods to now allow links to be included in the SMS Messages and restore the message delivery to Verizon.

As indicated in our previous update, all customers, irrespective of the carrier service, will notice a change to the URL in the text messages, the new URL will have “crfy” instead of “ctfy” ( example – https://b9.ctfy.co/u/ifjsdoj79) . Customers who may have whitelisted ctfy.co will now need to whitelist crfy.co.

Our preliminary results indicate the message deliveries are successful with no error 30007. We will continue to monitor the systems for another 48 - 72 hours to ensure no issues.
Posted 13 days ago. Feb 08, 2019 - 16:44 PST
Update
Update: We are implementing a config change to allow links to be included again and restore the service for message delivery to Verizon.

All customers will notice a change to the URL in the text messages, these changes will take effect to all customers irrespective of the provider the message is being sent to. The new URL will have “crfy” instead of “ctfy” ( example – https://b9.ctfy.co/u/ifjsdoj79)

We expect the process to complete in 24-48 hours and we will send out an update when we have finished rolling out the changes.
Posted 14 days ago. Feb 07, 2019 - 16:34 PST
Update
We have confirmed that the usage of hyperlinks in the SMS message is causing Verizon's spam filtering to block the delivery. The team is working to mitigate the issue by changing the short URL (Link) that we use to allow messages to be delivered to Verizon. This change will be transparent to all customers and will let you to continue to use the links in the templates for delivery to Verizon. This could take up to 24-48 hours to implement.

Apart from the changes above, we are still continuing our discussions with our providers to add some other functionality to enhance the robustness of the message delivery to Verizon.

We truly appreciate your patience and would highly recommend that you leverage different methods for MFA such as email, mobile authenticator, phone or security questions as a second factor for MFA.

Here are details of the workaround:

A potential workaround would be to edit the SMS template to only include the one-time passcode and remove the {ShortURL}.

Please refer to https://docs.centrify.com/Content/CoreServices/UsersRoles/SMSCustomize.htm for further details on how to edit the SMS template.
Posted 15 days ago. Feb 07, 2019 - 11:22 PST
Update
Our service provider and Verizon wireless continue to investigate the issue. Preliminary findings indicate the issue is related to the use of URL in the SMS message. A potential workaround would be to edit the SMS template to only include the one-time passcode and remove the {ShortURL}. Please refer to https://docs.centrify.com/Content/CoreServices/UsersRoles/SMSCustomize.htm? for further details on how to edit the SMS template.

We will continue to monitor this situation and provide more updates when available.
Posted 15 days ago. Feb 06, 2019 - 12:47 PST
Identified
Our teams have identified the issue with SMS delivery and are working directly with Verizon to address. We will post another update within the hour.

Our provider for messaging is currently experiencing issues with message filtration from Verizon - other mobile carriers such as AT&T and Sprint appear to be processing messages normally. As a result, some of our customers may be experiencing SMS delivery failure or receive messages with "error 30007" when using the Verizon network. Our teams are working directly with Verizon to provide a solution.

Customers are encouraged to leverage different methods for MFA such as email, mobile authenticator, phone, or security question as a temporary workaround.
Posted 16 days ago. Feb 06, 2019 - 10:55 PST
Update
Our provider for messaging is currently experiencing issues with message filtration from Verizon - other mobile carriers such as AT&T and Sprint appear to be processing messages normally. As a result, some of our customers may be experiencing SMS delivery failure or receive messages with "error 30007" when using the Verizon network.

Customers are encouraged to leverage different methods for MFA such as email, mobile authenticator, phone, or security question as a temporary workaround.

We will continue to monitor this situation and provide more updates when available.
Posted 16 days ago. Feb 06, 2019 - 08:56 PST
Investigating
Our provider for messaging is currently experiencing issues with message filtration from mobile carriers. As a result, some of our customers in may be experiencing SMS delivery failure or receive messages with "error 30007" . We will continue to monitor this situation and provide more updates when available.
Posted 16 days ago. Feb 06, 2019 - 07:16 PST
This incident affected: Infrastructure Services (pod1, pod2, pod4, pod7, pod8, fedpod0, cloud, pod9, pod10, apac, emea, pod13, pod14, pod15, pod17) and Identity Service (pod1, pod2, pod4, pod7, pod8, fedpod0, cloud, pod9, pod10, apac, emea, pod13, pod14, pod15, pod17, pod20, pod22, pod19, pod25, pod27, pod29, pod30).