Summary of impact
Last week customers reported a variety of Azure Bot Service issues such as:
- Internal Server Error dialog on bot load
- Inability to connect via FTP
- Inability to connect via Kudu
- Inability to setup continuous deployment
- Files disappeared from the bot editor
Only customers who created a bot with the Azure Bot Service between April 3rd and April 12th, 2017 were affected. Customer experiencing these issues were partially or completely unable to use their bots after creation.
Engineers identified as the root cause a change made on April 3rd to the default ARM template that provisions all Azure Bot Services. This change introduced an extra Azure Storage account during provisioning, which, coupled with a misconfiguration, caused the bot to look for the file system in the wrong Storage Account. Two factors contributed to delaying the resolution of the problem:
- Other changes were deployed around the same time, causing engineers to investigate in the wrong direction
- The ARM change was done by another team, and not all people were notified of the upcoming change
Once the problem was identified, engineers applied a patch on April 12th at approximately 7PM PST to mitigate the issue. With this, the issue did not affect newly created bots anymore. Then on April 13th at around 12PM, the faulty ARM template change was reverted.
All customers impacted are encouraged to create a new Azure Service Bot.
We’ve updated our deployment steps to make sure sign-off is obtained from both the ARM team and Bot Framework teams prior to deployments of ARM Templates. This mechanism will prevent issues like this from happening again.
The Bot Framework Team