Report Date: 2024-03-28
Incident Date: 2024-02-15
On 2024-02-15 at 14:44 UTC, the F5 Distributed Cloud support team received an initial report about a failure of the SMS alerting system. Upon receiving the report, the F5 Distributed Cloud team started investigation and isolated the issue to be impacting US region only. Subsequently over the next few days, the F5 Distributed Cloud support team received a few more customer reports about the SMS alert failures.
Detailed analysis revealed that the SMS alerts were failing at the messaging service provider’s end. It was identified that the F5 sender ID (From Number) was unregistered with the messaging service provider which was leading to SMS delivery failures. The SMS service provider had recently enforced this change under a compliance measure required for all US telecom carriers. F5 Distributed Cloud team had inadvertently missed updating the sender ID.
Due to an increased amount of time being taken by the existing messaging service provider to get the sender ID registered, the F5 Distributed Cloud team decided to move over to another provider instead.
On 2024-03-18 at 06:15 UTC, a configuration change was deployed by the F5 Distributed Cloud support team to fully resolve the SMS alerting issue. Post the fix being implemented, the teams confirmed SMS alerts functionality to be restored and fully functional.
The total duration of the service event was 31 days, 15 hours and 31 minutes.
| Start time of Service Event | 2024-02-15 14:44 UTC |
| Conclusion of Service Event | 2024-03-18 06:15 UTC |
| Event duration | 31 days, 15 hours and 31 minutes |
| Impact | Distributed Cloud customers in the US region were unable to receive SMS from the platform alerting system |
| Root cause | F5 Distributed Cloud inadvertently missed registering the sender ID (From Number) with the messaging provider, which caused the SMS delivery failures |
| DATE | TIME UTC | ACTION |
|---|---|---|
| 2024-02-15 | 14:44 | F5 Distributed Cloud team received the initial customer report informing us about the SMS alert failure. |
| 2024-02-15 | 14:57 | Customer re-confirmed that there was no progress when they tried to receive the verification code through SMS. |
| 2024-02-17 | 12:24 | After further investigation, support reached out to the engineering team for additional assistance. |
| 2024-02-23 | 17:56 | Support reached out to the customer and wanted to confirm if the issue has been resolved since the engineering team notified us that there was an error on our side which seems to be resolved. |
| 2024-02-27 | 13:23 | The customer informed us that they are still facing the issue and no progress was seen. |
| 2024-02-29 | 7:48 | F5 Distributed Cloud Team discovered that the F5 number has been unregistered by the messaging service provider. Re-registration efforts were started. |
| 2024-03-12 | 00:02 | Due to delay in registration process with existing vendor, F5 team registered with new messaging service vendor successfully |
| 2024-03-12 | 14:21 | Support reached back to the customer and informed them that the engineering team had implemented the hotfix, and the issue should now have been resolved. Thus, they requested customers to test from their end. |
| 2024-03-12 | 14:37 | Customer confirmed that they were still having the issue for country code +1, and one of their Solution Architect based in Mexico was able to receive the verification SMS making customers to think issue is happening with American numbers. |
| 2024-03-18 | 09:50 | F5 Distributed Cloud team informed through status page that the issue has been resolved and the SMS alerting issue has been restored for all customers |
Yes, the SMS alerting service is fully operational.
Application to Person (A2P) messaging is SMS/MMS traffic in which a person is receiving messages from an application rather than another individual. US telecom carriers consider any messages sent from any messaging provider to be A2P message. In this scenario, the F5 alert manager was sending the alerts to the person which required the sender ID (From Number) to be registered with A2P vendor. As the number registration was inadvertently missed, SMS delivery was falling.
F5 Distributed Cloud team switched over to another messaging provider, which is FED compliant, and had completed the registration process for adding the F5 sender ID (from Number). A platform configuration change was deployed to reflect the change, post which, the SMS alerting service became operational.
We will be taking several measure(s) to prevent this service event from reoccurring and to ensure that we are better prepared to react to and recover from similar scenarios more quickly.
F5® understands how important reliability of the Distributed Cloud Platform is for customers. F5 will ensure the recommended changes in this document are canonized into our operational Methods of Procedure (MoP) moving forward. We are grateful you have chosen to partner with F5® for critical service delivery and are committed to evolving our platform and tooling to better anticipate and mitigate disruptions to Distributed Cloud Platform services.
F5 Glossary