1

So it appears that RingCentral is down. While I try to reach their support I’m greeted with a busy signal and their support chat is down as well. Their status message says they’ll update again in 4 hours but figured while I wait for an update I’d check with you spicy folks and see if you’ve heard anything.

8 Spice ups
2

Same results here in the NorthEast. Ring Phone system is completely down. Tried calling support and they go right to a busy signal. I even tried to submit a support request and was greeted with a message to call the non working support number. No status other than they are investigating the issue. Still waiting…

4 Spice ups
3

Yeah, just looked on Downdetector and something is definitely going on:

https://downdetector.com/status/ringcentral/

screenshot-downdetector_com-2025_01_22-13_32_47
screenshot-downdetector_com-2025_01_22-13_32_47777×479 17.1 KB

I do see they are calling out Calling, RingCX, & Contact Center in the Americas on their status page:

Update (from me, that is):

2 Spice ups
4

We’ve got a few people in the PI and they have no issues. It looks like the problems are just for The Americas from their status page. On a plus, SMS is still good…for now

2 Spice ups
5

This seems larger than just Ring Central. MS just put out an alert about Teams phone issues and we got another notice from a vendor that their phone system is out (though I guess they could be using Ring Central)

2 Spice ups
6

We are having RingCentral issues here, too. Verizon cell service (phone calls only) seems to possibly be affected. Haven’t heard about other carriers from our users though.

2 Spice ups
7

Yup. It’s been down for several hours now. I see some comments on Downdetector indicating that it may be resolving for some users. I have a few of our users comment in the last 20 minutes that they were able to make a call, so hopefully they are close to getting this fixed.

I don’t think that Verizon is affected like RingCentral. I have some guys on Verizon and have not heard of an issue from cell to cell. They cannot call in because we are on RingCentral at the office.

1 Spice up
8

We were up for a second but then went back down. I was able to reach support and their phone number now has a recording saying they’re having technical difficulties.

2 Spice ups
9

I hadn’t noticed that till you mentioned it now. We don’t use the calling in Teams much…

That’s an understatement. I tried to reach them around 1:30 EST and their phone didn’t ring at all and the virtual chat bot took almost 10 minutes to respond - and all it did was link the Service Status page.

1 Spice up
10

While it looks like it’s getting better (or maybe less people reporting it): https://downdetector.com/status/ringcentral/

screenshot-downdetector_com-2025_01_22-15_12_17
screenshot-downdetector_com-2025_01_22-15_12_17770×277 5.99 KB

I do see that RC has a new status update: https://status.ringcentral.com/

January 22, 2025
14:38:08 CSTRingCentral Operations teams are continuing to perform troubleshooting for this issue and will provide the next update in 1 hour, or when a significant change in status occurs. If you are experiencing this issue, please reach out to customer care.

2 Spice ups
11

There is a bridge integration for RingCentral for teams, so hard to tell if it is mostly confused users regarding what part didn’t work…

But…weirdly…DD is reporting failures with MS store…scratching head

2 Spice ups
12

It looks like RingCentral was able to implement a fix last night. Our users are reporting it working this morning:

image
image1920×917 83.4 KB
5 Spice ups
13

I’m not sure what caused the service issue, but I can share my opinion (right?)

Our RC setup is pretty recent (less than a year), but we have a userbase of ~750 users.

I think it was an interchange issue, i.e. the portion of the service where calls cross from one phone provider to another. All of our internal calls went through during the outage, with no issues, regardless of how the call was made. External calls (inbound and outbound) has a 25 - 33% success rate.

Verizon wireless and AT&T wireless was a no-go. Our Verizon landline was a no-go. We were able to speak with our service partner, as they are also on RC, with no issues.

We followed the service status instructions, and opened a service ticket. It wasn’t acknowledged, but we got our RC rep to confirm that they received it.

What was most frustrating, is that we run a number of critical call center for crisis services, and we did not have the ability to forward our calls because of the outage.

I’m waiting for the full report from RC, but I’m going to be looking at a service that can act as an intermediate, between (some of) our numbers, and RC. We had a conversation over a year ago with an outfit named CallTower, which offered a MS Teams centric solution; we might have been able to redirect our calls from RC to MS Teams (which is conveniently billed for this purpose), and minimize the impact of this RC outage. Most of our critical department/services are already on Teams, so this would have been a relatively easy workaround for them.

Fingers crossed.

5 Spice ups
14

That’s exactly the same conversation we had today as well. Looking for a relatively cheap solution but considering just getting Teams Voice licenses for about a half dozen users and a toll free call queue and just publish an emergency number if this issue happens again.

3 Spice ups
15

We have been dealing with this as well with hundreds of our clients on RingCentral today…

Happy to discuss options via CallTower and other teams solutions as well as various other ways to avoid these major outages.

There are also a few very good consumption based Teams solutions which can scale up in an emergency with very low overhead cost wise when not being used

Feel free to reach out and we can get our agnostic engineering team involved to discuss and put some options on the table.

Pm me here or request help on our website…www.commquotes.com/spiceworks

6 Spice ups
16

Kinda interesting how it coincided with taking TiKTok down :sunglasses:

1 Spice up
17

I would say that is a good a guess as any.
Zoom said this at the same time:
“We are investigating a subset of customers that use a specific carrier that may experience inbound call failures for US region. We are engaged with the carrier and will provide regular updates.”

They later updated and said it didn’t affect them, but there is a nice hint in there.
(of course, the “carrier” may have just been ringcentral…)

Security by obscurity is deeply ingrained in telco’s and is a hard habit to break…sigh

18

The official apology: https://www.ringcentral.com/us/en/blog/message-from-CEO-regarding-intermittent-service-interruption/

4 Spice ups
21

Well, I was mostly right.
Final RFO for INC-43042 Inbound and Outbound Call Drops and Failures for RingCentral EX, RingCX, and Contact Center Customers.pdf (197.1 KB)

(RFO attached)

Here’s the timeline:
Incident & Remediation Timeline
Incident Start: January 22, 2025, at 10:13 AM PST
Incident Resolution: January 22, 2025, at 4:58 PM PST
Total Duration: 6 hours and 45 minutes
02:54 AM PST - Maintenance
• During a planned maintenance window at 02:54 AM PST, RingCentral Operations
implemented a call routing optimization change that involved decommissioning a legacy
call route that was incorrectly classified as unused.
10:13 AM PST - Incident Began
• Over seven hours after the change was implemented, at 10:13 AM PST, the incident
manifested when a specific customer phone number utilizing the decommissioned call
route triggered a call loop condition, which caused significant network congestion.
10:13 AM to 11:45 AM PST - Initial Response and Mitigation Attempts
• RingCentral Operations executed several mitigation steps to address the primary call loop:
o Blocked the phone number utilizing the decommissioned route
o Implemented congestion reduction measures
o Completed a rollback of the original maintenance change
11:45 AM to 1:47 PM PST - Escalated Response and Investigation
• RingCentral Operations started its second phase of investigation and troubleshooting,
including the following measures:
o Initiated hardware reboots for edge telecom components
o Engaged third-party telecom component vendors for specialized support
• At 1:47 PM PST, RingCentral detected a bug-induced secondary call loop, which first
originated at 10:18 AM PST
4 |
RingCentral RFO Report
[43042: 22 January 2025]
1:47 PM to 4:58 PM PST - Service Restoration Phase
• After identifying the secondary call loop, RingCentral Operations undertook the following
service restoration measures:
o Achieved partial service restoration through reboots of the application servers
involved in the secondary loop
o Deployed an operational workaround to remediate the secondary loop
o Rebooted downstream and upstream network components to fully restore service
o Confirmed that system availability was fully restored
4:58 PM PST and after - Post-Incident Monitoring
• Following the resolution, extensive monitoring and testing continued to ensure the
stability of the fix and verify that no residual effects remained

3 Spice ups
22

Can you say…OOPS

Irony, decades ago AT&T did almost the same thing when long distance switches had a
protocol glitch that made all the switches try to failover to other switches endlessly…

Did you turn it off and on again? lol

Good to have followup, it is amazing how easily things can go to H*ll in a handbasket.

3 Spice ups