tag:www.f5cloudstatus.com,2005:/historyF5 Distributed Cloud Status - Incident History2024-03-19T09:00:56ZF5 Distributed Cloudtag:www.f5cloudstatus.com,2005:Incident/201748932024-04-09T09:00:00Z2024-03-18T16:24:22ZScheduled Maintenance of the F5 Infrastructure in Paris<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>9</var>, <var data-var='time'>09:00</var> - <var data-var='time'>12:00</var> UTC</strong></p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:52</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Paris (PA4)<br /><br />Maintenance activities:<br /><br />- Firewall Migration in Paris (PA4)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Paris (PA4) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/202268422024-03-28T04:00:00Z2024-03-13T03:21:47ZScheduled Maintenance of F5 Distributed Cloud Services - Backup Window<p><strong>THIS IS A SCHEDULED EVENT Mar <var data-var='date'>28</var>, <var data-var='time'>04:00</var> - <var data-var='time'>11:00</var> UTC</strong></p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>03:21</var> UTC</small><br><strong>Update</strong> - We apologize for the confusion on previous message. The correct maintenance time is:<br /><br />Schedule:<br />• Start Time: March 28, 2024 - 4:00AM UTC<br />• Estimated Completion Time: March 28, 2024 - 11:00AM UTC</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>17:09</var> UTC</small><br><strong>Scheduled</strong> - We will be performing planned maintenance of F5 Distributed Cloud services as follows:<br /><br />Schedule: [Backup Upgrade Window]<br />• Start Time: March 28, 2024 - 4:00AM UTC<br />• Estimated Completion Time: March 28, 2024 - 11:00AM UTC<br /><br />Activities:<br />• Upgrade Global Controller to the latest software version.<br />• Upgrade Regional Edge (RE) sites to the latest software version.<br /> o Software of the RE sites are upgraded in a rolling update manner, ensuring no<br /> disruption to Distributed Cloud services.<br /> o Rolling updates are performed on batches of RE sites, where each batch includes<br /> a set of geographically distributed RE sites.<br /><br />Impact:<br />• All services are expected to remain operational and available during the entire<br /> maintenance window with no noticeable impact. This is achieved by removing the RE<br /> site from the anycast network, before starting the upgrade, thereby causing the traffic<br /> to be re-routed to other available RE sites in the region. As a result, data plane services<br /> will continue without interruption except if tenants have a hard coded dependency to a<br /> specific RE site.<br />• Management plane configuration changes will continue to be recorded and actively<br /> propagated during most of the duration of the upgrade, except during the time period<br /> when Global Controller is being upgraded or during the time period when customers’<br /> config-path RE site is being upgraded.<br /><br />If you have any questions, please contact us at support@cloud.f5.com</p>tag:www.f5cloudstatus.com,2005:Incident/202267082024-03-26T04:00:00Z2024-03-13T12:42:12ZScheduled Maintenance of F5 Distributed Cloud Services<p><strong>THIS IS A SCHEDULED EVENT Mar <var data-var='date'>26</var>, <var data-var='time'>04:00</var> - <var data-var='time'>11:00</var> UTC</strong></p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>12:42</var> UTC</small><br><strong>Update</strong> - We will be adding new IP ranges to our RE after March-26 upgrade. If you have configured IP filters on your side to only authorize traffic from the F5 infrastructure, please add the following range.<br /><br />The new IP range(s) will be used starting from Mar 26, 2024: <br /><br />Asia 159.60.170.0/24, 159.60.172.0/24<br />America 159.60.174.0/24. 159.60.176.0/24<br />EU 159.60.178.0/24, 159.60.182.0/24<br /><br />Please refer to https://docs.cloud.f5.com/docs/reference/network-cloud-ref for more detail. If you have any questions regarding this change, F5 Support can be contacted as detailed here: https://docs.cloud.f5.com/docs/support/support</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>03:20</var> UTC</small><br><strong>Update</strong> - We apologize for the confusion on previous message. The correct maintenance time is:<br /><br />Schedule:<br />• Start Time: March 26, 2024 - 4:00AM UTC<br />• Estimated Completion Time: March 26, 2024 - 11:00AM UTC</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>16:51</var> UTC</small><br><strong>Scheduled</strong> - We will be performing planned maintenance of F5 Distributed Cloud services as follows:<br /><br />Schedule:<br />• Start Time: March 26, 2024 - 4:00AM UTC<br />• Estimated Completion Time: March 26, 2024 - 11:00AM UTC<br /><br />Activities:<br />• Upgrade Global Controller to the latest software version.<br />• Upgrade Regional Edge (RE) sites to the latest software version.<br /> o Software of the RE sites are upgraded in a rolling update manner, ensuring no disruption to Distributed Cloud services.<br /> o Rolling updates are performed on batches of RE sites, where each batch includes a set of geographically distributed RE sites.<br /><br />Impact:<br />• All services are expected to remain operational and available during the entire maintenance window with no noticeable impact. This is achieved by removing the RE site from the anycast network, before starting the upgrade, thereby causing the traffic to be re-routed to other available RE sites in the region. As a result, data plane services will continue without interruption except if tenants have a hard coded dependency to a specific RE site.<br />• Management plane configuration changes will continue to be recorded and actively propagated during most of the duration of the upgrade, except during the time period when Global Controller is being upgraded or during the time period when customers’ config-path RE site is being upgraded.<br /><br />If you have any questions, please contact us at support@cloud.f5.com</p>tag:www.f5cloudstatus.com,2005:Incident/201795832024-03-25T09:00:00Z2024-03-06T21:28:41ZScheduled Maintenance of the F5 Infrastructure in Sydney<p><strong>THIS IS A SCHEDULED EVENT Mar <var data-var='date'>25</var>, <var data-var='time'>09:00</var> - <var data-var='time'>11:00</var> UTC</strong></p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>21:28</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Sydney (SY5)<br /><br />Maintenance activities:<br /><br />- Network collector update in Sydney (SY5)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Sydney (SY5) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/201793182024-03-21T09:00:00Z2024-03-06T20:47:18ZScheduled Maintenance of the F5 Infrastructure in Frankfurt<p><strong>THIS IS A SCHEDULED EVENT Mar <var data-var='date'>21</var>, <var data-var='time'>09:00</var> - <var data-var='time'>11:00</var> UTC</strong></p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>20:47</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Frankfurt (FR4)<br /><br />Maintenance activities:<br /><br />- Network collector update in Frankfurt (FR4)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Frankfurt (FR4) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/201795462024-03-20T09:00:00Z2024-03-06T21:24:18ZScheduled Maintenance of the F5 Infrastructure in San Jose<p><strong>THIS IS A SCHEDULED EVENT Mar <var data-var='date'>20</var>, <var data-var='time'>09:00</var> - <var data-var='time'>11:00</var> UTC</strong></p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>21:24</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in San Jose (SV10)<br /><br />Maintenance activities:<br /><br />- Network collector update in San Jose (SV10)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for San Jose (SV10) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/201795162024-03-19T09:00:56Z2024-03-19T09:00:56ZScheduled Maintenance of the F5 Infrastructure in Amsterdam<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>21:17</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Amsterdam (AMS9)<br /><br />Maintenance activities:<br /><br />- Network collector update in Amsterdam (AMS9)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Amsterdam (AMS9) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/201795642024-03-18T11:00:56Z2024-03-18T11:00:56ZScheduled Maintenance of the F5 Infrastructure in Toronto<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>21:26</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Toronto (TR2)<br /><br />Maintenance activities:<br /><br />- Network collector update in Toronto (TR2)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Toronto (TR2) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/200884522024-03-18T11:00:56Z2024-03-18T11:00:56ZScheduled Maintenance of the F5 Infrastructure in Tokyo<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>14:48</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Tokyo (TY8)<br /><br />Maintenance activities:<br /><br />- Network collector update in Tokyo (TY8)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Tokyo (TY8) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/202183622024-03-18T09:50:53Z2024-03-18T09:50:53ZIssue with Email and SMS Alerts<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>09:50</var> UTC</small><br><strong>Resolved</strong> - The F5 Distributed Cloud team confirmed the SMS alerting issue is restored. This incident has been resolved.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>07:12</var> UTC</small><br><strong>Monitoring</strong> - The F5 Distributed Cloud team completed the activity and deployed the configuration change. The issue has been mitigated. We are currently monitoring the platform.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>05:01</var> UTC</small><br><strong>Update</strong> - The F5 Distributed Cloud Team would have to deploy a config change and restart a alert service at the following time. It is expected to see errors related to the alert before configuration been fully loaded. We apologize for the confusion caused this activity.<br /><br />Time:<br />Start: 05:00AM Mar 18 2024 UTC<br />Expected end time: 09:00AM Mar 18 2024 UTC</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>10:33</var> UTC</small><br><strong>Update</strong> - F5 Distributed Cloud support team will deploy a platform change to fully address the SMS alerting issue. The change is currently under testing and is expected to be completed and deployed in the week of March 18th. More updates will be provided soon.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>16:50</var> UTC</small><br><strong>Identified</strong> - F5 Distributed Cloud Team is actively addressing the SMS alerting issue. Traffic processing and core services are operating normally, however, customers cannot receive SMS notifications on a real-time basis in the US region. F5 Distributed Cloud team continues working on priority to restore services. More updates will be provided soon.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>06:32</var> UTC</small><br><strong>Monitoring</strong> - Distributed Cloud console issue has been resolved. We are continuously monitoring the system.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>03:02</var> UTC</small><br><strong>Investigating</strong> - The F5 Distributed Cloud Team has identified an issue with Distributed Cloud alert console where some of our customers are getting 500 server error while accessing the browser. We are actively addressing and resolving the issue.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>01:23</var> UTC</small><br><strong>Update</strong> - Email alerts has been restored. We are continuously monitoring the system</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>00:02</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and deployed. Presently we are monitoring the system</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>22:54</var> UTC</small><br><strong>Update</strong> - We have identified the issue and working the fix</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>19:52</var> UTC</small><br><strong>Identified</strong> - The F5 Distributed Cloud Team has identified an issue with email and SMS alerts that is currently restricting the capability for some of our customers to receive their configured notifications in real-time. We are actively addressing and resolving the issue</p>tag:www.f5cloudstatus.com,2005:Incident/201748742024-03-14T12:00:56Z2024-03-14T12:00:56ZScheduled Maintenance of the F5 Infrastructure in London<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:49</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in London (LD6)<br /><br />Maintenance activities:<br /><br />- Firewall Migration in London (LD6)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for London (LD6) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/200884222024-03-13T11:00:56Z2024-03-13T11:00:56ZScheduled Maintenance of the F5 Infrastructure in Singapore<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>14:45</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Singapore (SG3)<br /><br />Maintenance activities:<br /><br />- Network collector update in Singapore (SG3)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Singapore (SG3) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/200897432024-03-12T12:00:56Z2024-03-12T12:00:56ZScheduled Maintenance of the F5 Infrastructure in Singapore<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>17:34</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in Singapore (SG3)<br /><br />Maintenance activities:<br /><br />- Firewall Migration in Singapore (SG3)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for Singapore (SG3) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/201976992024-03-08T12:30:00Z2024-03-18T21:11:00ZLog Loss and Downtime Between 12:40-13:45 UTC<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>12:30</var> UTC</small><br><strong>Resolved</strong> - March 8, 2024 at 12:40, an incident occurred resulting in the loss of approximately 30% of logs between the period of 12:40-13:45 UTC. The issue primarily stemmed from network switch failure in one of our datacenters which caused downtime of both logging sites. Additionally, log delivery experienced delays, and UI queries were rendered inaccessible during this time as both regions became unreachable. This incident had a cascading effect on other logging-dependent components such as GLR and DOS mitigation.</p>tag:www.f5cloudstatus.com,2005:Incident/201820572024-03-07T05:59:43Z2024-03-07T05:59:43ZChanges to Distributed Apps are not reflecting<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>05:59</var> UTC</small><br><strong>Resolved</strong> - Issue has been resolved</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>05:52</var> UTC</small><br><strong>Monitoring</strong> - A fix has been applied to remediate the issue. We are presently monitoring the platform.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>05:28</var> UTC</small><br><strong>Identified</strong> - We have identified an issue on the Distributed Apps on F5XC Platform. Where any changes to new or existing Distributed App or Distributed Apps are not getting reflected with 503 error code</p>tag:www.f5cloudstatus.com,2005:Incident/200438782024-02-22T07:51:27Z2024-02-22T07:51:27ZConnectivity issue to pa2-par RE<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>07:51</var> UTC</small><br><strong>Resolved</strong> - We have confirmed logging system went back to normal, this incident has been resolved.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>05:57</var> UTC</small><br><strong>Update</strong> - We are still monitoring PA2 for traffic path. Logging Infrastructure is syncing and some delays can be observed on console.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>04:02</var> UTC</small><br><strong>Monitoring</strong> - We have implemented the fix to PA2 side and verified all working fine. PA2 has been put back to traffic path. Logging infra are syncing and customer may see delay for log showing up on the console.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>03:22</var> UTC</small><br><strong>Identified</strong> - We have removed PA2 site from the customer LoadBalancer traffic path. Currently the issue has been identified and fix been implemented. We will keep monitoring the status. Customer may still see logging not showing up.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>03:09</var> UTC</small><br><strong>Update</strong> - We are investigating the issue from network, at this moment logs may not correctly showing up on the console.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>02:42</var> UTC</small><br><strong>Investigating</strong> - We have detected an issue to one of our PA2 RE located in Pari, we are working on this issue.</p>tag:www.f5cloudstatus.com,2005:Incident/200138782024-02-18T18:55:06Z2024-02-22T21:43:33ZChange to LB or existing configuration are not reflecting<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>18:55</var> UTC</small><br><strong>Resolved</strong> - A fix has been implemented. Issue has been mitigated. We will continue to monitor the platform</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>18:47</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and mitigations are getting prepared for it.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>17:01</var> UTC</small><br><strong>Update</strong> - We are continue to investigate the issue and will update soon</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Investigating</strong> - F5XC Team noticed an issue with our platform where any change made through the console are not reflecting. This issue only effects if there are changes done to an existing configuration. Team is working on identifying the issue. <br />All existing configuration are working as expected.</p>tag:www.f5cloudstatus.com,2005:Incident/198278362024-01-29T00:42:04Z2024-01-29T00:42:04ZMaintenance on logging infra<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>00:42</var> UTC</small><br><strong>Completed</strong> - We're happy to announce that the maintenance on our logging infrastructure has been completed successfully, all logging components are operational and system are processing logs more efficiently. <br /><br />Thank you for your understanding!</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>08:20</var> UTC</small><br><strong>Verifying</strong> - We have finished all planned operation and will keep monitoring our logging queue.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>10:18</var> UTC</small><br><strong>Scheduled</strong> - We will be performing maintenance on our logging infrastructure starting at 17:00 UTC on January 25th to handle the growing logging volume.<br /><br />During this time, customer may experience a delay in access logs showing up on the console depending on the amount of log process.<br /><br />We apologize for the short notice. This upgrade paves the way for faster log processing and rock-solid stability, all to keep your experience top-notch.<br /><br />Thanks for your understanding!</p>tag:www.f5cloudstatus.com,2005:Incident/195075422024-01-24T10:44:42Z2024-01-24T10:44:42ZScheduled Maintenance of the F5 Infrastructure in New York<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:44</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>15:55</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming scheduled maintenance window for the following F5 services:<br /><br />- F5 Distributed Cloud Mesh in New York (NY2)<br /><br />Maintenance activities:<br /><br />- Firewall Migration in New York (NY2)<br /><br />Projected impact:<br />- Customers who have deliberately constructed hard-coded dependencies for New York (NY2) Regional Edge site may experience a service interruption during this window.<br />- During this window, Customer edge (CE) and applications will not experience any downtime and will continue to be served out of the other Network PoPs. <br /><br />If you have any questions feel free to contact us at support@cloud.f5.com.</p>tag:www.f5cloudstatus.com,2005:Incident/198088992024-01-24T07:43:53Z2024-01-24T07:43:53ZCustomer Traffic Impact on Regional Edges<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>07:43</var> UTC</small><br><strong>Resolved</strong> - We have deployed hotfix into all region and verified system are running stable.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>06:19</var> UTC</small><br><strong>Monitoring</strong> - We have deployed the hotfix on all the Regional edges</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>03:27</var> UTC</small><br><strong>Update</strong> - We are continuing to deploy the fix on all Regional edges</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>00:28</var> UTC</small><br><strong>Update</strong> - We are deploying the fix in production.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>22:31</var> UTC</small><br><strong>Update</strong> - We are testing the fix in our lower environments</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>21:13</var> UTC</small><br><strong>Update</strong> - We have identified issue and we are working on the fix for it.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>18:20</var> UTC</small><br><strong>Identified</strong> - F5 Distributed Cloud team has identified an issue which is causing a traffic impact on Regional edges. We are working to fix the issue.</p>tag:www.f5cloudstatus.com,2005:Incident/198052682024-01-23T13:45:37Z2024-01-23T13:45:37ZCustomer Traffic Impact on Regional Edges<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>13:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>12:06</var> UTC</small><br><strong>Monitoring</strong> - We observed an issue on a service running in regional edge between 11:10 to 11:30 UTC, which have impacted multiple customers. We have coordinated the issue and currently system are running stable. We will keep monitor the status and update.</p>tag:www.f5cloudstatus.com,2005:Incident/197584912024-01-19T20:28:30Z2024-01-19T20:28:30ZCustomer Traffic Impact on Regional Edges<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>20:28</var> UTC</small><br><strong>Resolved</strong> - Identified issue has been hotfixed and mitigate.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>11:14</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>11:14</var> UTC</small><br><strong>Update</strong> - Hotfix deployment currently is in progress.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>07:23</var> UTC</small><br><strong>Update</strong> - We are planning to start production hotfix to coordinate this issue, starting from 08:20 UTC.<br /><br />Maintenance activities: <br /> - Rolling updates across individual Regional Edge sites, non-disruptive to Distributed Cloud service delivery. <br /> - Rolling updates are done in Regional Edges batches which geographically distributed.<br /> <br />Projected impact:<br /> - Before any Regional edge site start upgrade, traffic will be forwarded to other available site. All services are projected to remain available during the entire maintenance window with no noticeable impact<br /> - Customers who have deliberately constructed hard-coded dependencies from specific F5 Regional Edge sites may experience service interruption</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>01:21</var> UTC</small><br><strong>Update</strong> - F5 has identified the cause of the traffic disruptions and is actively working to release a hotfix to revert recent changes. The change is currently under review and validation before being released. Manual fixes have been applied to currently known impacted customers. If you are currently experiencing any impact please contact the Distributed Cloud Support Team.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>23:29</var> UTC</small><br><strong>Identified</strong> - F5 has identified the cause of the traffic disruptions and is actively working to release a hotfix to revert recent changes.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>23:01</var> UTC</small><br><strong>Investigating</strong> - F5 is currently aware of impact to traffic for a subset of customers and Regional Edges. We are actively investigating this and will provide additional information.</p>tag:www.f5cloudstatus.com,2005:Incident/196202262024-01-18T20:00:15Z2024-01-18T20:00:15ZScheduled Maintenance of F5 Services<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>13:58</var> UTC</small><br><strong>Update</strong> - We would still need some time to finish the upgrade today. New ETA is 20:00 UTC.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>10:14</var> UTC</small><br><strong>Update</strong> - During our upgrade we observed slowness on our upgrade process, thus we would need to extend the upgrade window. Current ETA is 15:00 UTC today. We apologize for the delay and will update once we have progress.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>10:02</var> UTC</small><br><strong>Update</strong> - We will use this window (Jan-18 2024 06:00 AM UTC) to perform our main upgrade. Please check the updated contents below.<br /><br />Maintenance activities: <br /> - Rolling updates across individual Regional Edge sites, non-disruptive to Distributed <br /> Cloud service delivery. <br /> - Rolling updates are done in Regional Edges batches which geographically distributed.<br /> - Global Controller will be upgraded.<br /> <br />Projected impact:<br /> - Before any Regional edge site start upgrade, traffic will be forwarded to other available site. All services are projected to remain available during the entire maintenance window with no noticeable impact<br /> - Customers who have deliberately constructed hard-coded dependencies from specific <br /> F5 Regional Edge sites may experience service interruption<br /> - Config propagation will be halted during the maintenance window. Any changes in <br /> configs will be recorded and will only be replicated post maintenance activity</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>11:16</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming maintenance window for the following F5 services: <br /><br />Schedule:<br /> - Start Time: Jan 18, 2024 - 6:00AM UTC<br /> - Complete Time: Jan 18, 2024 - 11:00AM UTC<br /> <br />Services to upgraded:<br /> - F5 Distributed Cloud Additional Services<br /> <br />Maintenance activities: <br /> - Rolling non-disruptive updates to Distributed Cloud service delivery. <br /> <br />Projected impact: <br /> - All services are projected to remain available during the entire maintenance window with <br /> no noticeable impact.<br /> - This maintenance window will be used to update components which may not have <br /> impact to consumers<br /><br />Important Notice:<br /> - This window may be used as backup upgrade window if we had to postpone the original <br /> upgrade window.<br /> - We will sent out a notification confirming the same.<br /> <br />If you have any questions feel free to contact us at support@cloud.f5.com</p>tag:www.f5cloudstatus.com,2005:Incident/200023012024-01-18T14:00:00Z2024-02-16T21:04:40ZTraffic Disruption impacting Load Balancer services.<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Resolved</strong> - F5 identified users who were experiencing issues when redirecting HTTP connections to HTTPS.</p>tag:www.f5cloudstatus.com,2005:Incident/196202042024-01-16T11:01:06Z2024-01-16T11:01:06ZScheduled Maintenance of F5 Services - Non Disruptive<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>11:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Update</strong> - We have to reschedule the production maintenance to the backup window (Jan-18, 2024 at 06:00 AM UTC) due to an technical issue which has delayed pre-release testing. <br /><br />We apologize for any inconvenience this may cause, but we believe that it is important to take the time to resolve the issue and ensure the quality of our service before proceeding with the maintenance.<br /><br />This window (Jan-16 2024 06:00 AM UTC) will be kept to upgrade multiple non disruptive systems. All services are projected to remain available during the entire maintenance window with no noticeable impact.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>11:13</var> UTC</small><br><strong>Scheduled</strong> - This is a courtesy notice of an upcoming maintenance window for the following F5 services: <br /><br />Schedule:<br /> - Start Time: Jan 16, 2024 - 6:00AM UTC<br /> - Complete Time: Jan 16, 2024 - 11:00AM UTC<br /> <br />Services:<br /> - F5 Distributed Cloud Mesh <br /> - F5 Distributed Cloud Console <br /><br />Maintenance activities: <br /> - Rolling updates across individual Regional Edge sites, non-disruptive to Distributed <br /> Cloud service delivery. <br /> - Rolling updates are done in Regional Edges batches which geographically distributed.<br /> - Global Controller will be upgraded.<br /> <br />Projected impact:<br /> - Before any Regional edge site start upgrade, traffic will be forwarded to other available site. All services are projected to remain available during the entire maintenance window with no noticeable impact<br /> - Customers who have deliberately constructed hard-coded dependencies from specific <br /> F5 Regional Edge sites may experience service interruption<br /> - Config propagation will be halted during the maintenance window. Any changes in <br /> configs will be recorded and will only be replicated post maintenance activity<br /><br />If you have any questions feel free to contact us at support@cloud.f5.com</p>