Postmortem -
Read details
Oct 29, 19:55 CDT
Resolved -
This incident has been resolved.
Oct 29, 19:47 CDT
Update -
Update - 5:20 pm CST, 29 October 2025
Status: Monitoring
We continue to closely monitor platform performance. Services remain accessible with only minor intermittent issues as Azure completes their infrastructure recovery.
Microsoft's expected full resolution remains 23:20 UTC. We will provide another update within 2 hours or if conditions change.
Thank you for your continued patience.
Oct 29, 17:26 CDT
Monitoring -
Here's a status update reflecting your current state:
Update - 3:00 pm CST, 29 October 2025
Status: Monitoring - Partial Recovery
Our platform is now responding following Azure's configuration deployment. However, we are continuing to monitor closely as the full recovery process is still underway.
Current State:
✓ Core services are accessible
⚠️ Intermittent issues may still occur as Azure completes node recovery
🔄 Performance stabilization in progress
What to Expect:
You should be able to access our platform, but may experience occasional slowness, timeouts, or temporary errors as Azure continues routing traffic through recovering infrastructure. Service quality will continue improving as Microsoft completes its mitigation efforts.
Azure's Timeline:
Microsoft expects full resolution by 23:20 UTC (approximately 4 hours from now). We will continue monitoring throughout this period and will confirm full stability once Azure completes their recovery.
Next Update:
We will provide another update within 1 hours, or immediately if we observe any significant changes in service stability.
Thank you for your patience as we work through this recovery period.
Oct 29, 15:06 CDT
Identified -
Update - 2:28 CST, 29 October 2025
Status: Observing Recovery
Good news - Microsoft Azure has successfully deployed their stable configuration and recovery is now underway.
Current Progress:
✓ Last known good configuration deployed successfully
✓ Azure Portal access restored (direct routing active)
🔄 Node recovery and traffic rerouting in progress
🔄 Service improvements ongoing as healthy nodes come online
What You're Seeing:
We are still experiencing stability issues. These should gradually resolve as Azure continues recovering infrastructure nodes. We will keep you updated.
Expected Full Resolution:
Microsoft anticipates complete mitigation by 23:20 UTC (approximately 4 hours from now) as they finish recovering all affected nodes.
Known Limitations:
Configuration changes remain temporarily blocked on Azure infrastructure
Some Azure Portal extensions (e.g., Marketplace) may still experience intermittent loading issues
Next Update:
We will provide another update within 1 hour or sooner if significant progress is made.
We appreciate your patience as recovery continues. Service quality should continue improving steadily over the next several hours.
Oct 29, 14:29 CDT
Update -
Here's the updated status message:
Update - 1:18 CST, 29 October 2025
Microsoft Azure has initiated deployment of their last known good configuration and expects initial signs of recovery within approximately 30 minutes.
Recovery Timeline:
Phase 1 (In Progress): Deploying stable configuration - Expected completion ~17:50 UTC
Phase 2 (Next): Node recovery and traffic rerouting to healthy infrastructure
Current Impact:
Our services remain affected during this recovery process. You may continue to experience access issues or degraded performance until Azure completes both recovery phases.
Restrictions:
Microsoft has temporarily blocked all customer configuration changes to its infrastructure while it completes the rollback. This restriction will remain in place until full mitigation is achieved.
What to Expect:
Service stability should begin improving within the next 30-45 minutes as Azure's deployment completes. We will monitor closely and provide another update once we confirm meaningful recovery.
Thank you for your continued patience.
Oct 29, 13:19 CDT
Update -
Update - 12:33 CST, 29 October 2025
Microsoft Azure has identified the root cause as an inadvertent configuration change to Azure Front Door (AFD) services starting at approximately 16:00 UTC.
Current Status:
Microsoft is actively rolling back to the last known good configuration
Azure Portal access has been restored via failover routing
Our services remain impacted while the AFD rollback is in progress
What This Means:
You may still experience intermittent issues or degraded performance as Microsoft completes its infrastructure rollback. We are closely monitoring the situation and will restore full service as soon as Azure's remediation is complete.
Next Update:
Microsoft has committed to providing updates within 30 minutes. We will post our next update shortly after receiving additional information from Azure.
We appreciate your patience as this incident is resolved.
Oct 29, 12:34 CDT
Update -
We are continuing to investigate this issue.
Oct 29, 11:34 CDT
Investigating -
Service Disruption - Azure Infrastructure Issue
Status: Investigating
Last Updated: 11:18 AM, 29 October 2025
We are currently experiencing service disruptions due to an ongoing issue with Microsoft Azure Portal infrastructure. This is affecting our ability to deliver normal service levels.
What's Happening:
Microsoft Azure is experiencing portal access issues affecting multiple services, including LoanNEX. Our platform relies on Azure infrastructure, and we are directly affected by this outage.
Impact:
You may experience difficulties accessing our platform or encounter degraded performance during this time.
What We're Doing:
Actively monitoring the situation with Microsoft Azure
Our technical team is on standby to restore full service immediately once Azure resolves their infrastructure issues
We will provide updates as more information becomes available
We understand this disruption is frustrating and sincerely apologize for the inconvenience. This issue is beyond our direct control, but we remain committed to restoring service as quickly as possible.
For real-time updates on the Azure issue, you can visit: Azure Status Page
https://azure.status.microsoft/en-us/status
We will continue to post updates here as the situation develops.
Oct 29, 11:34 CDT
