Fixing an Azure web app zip deployment issue

The problem

Some time ago I created an Azure linux web app. I used an Azure DevOps build pipeline to build the solution and deploy to it. Because we where deploying into subfolders, I used the Azure CLI to deploy the artifacts using a zipfile deployment, about which I blogged here. The builds and deployments ran fine, until they started failing recently with the following error:

ERROR: Zip deployment failed. 
    {'id': '<some guid>', 'status': 3, 'status_text': '', 'author_email': 'N/A', 'author': 'N/A', 'deployer': 'OneDeploy', 'message': 'OneDeploy', 'progress': '', 'received_time': '2022-03-17T10:44:24.2199057Z', 'start_time': '2022-03-17T10:44:25.6031069Z', 'end_time': '2022-03-17T10:44:31.9924311Z', 'last_success_end_time': None, 'complete': True, 'active': False, 'is_temp': False, 'is_readonly': True, 'url': 'https://<some-app-name>.scm.azurewebsites.net/api/deployments/latest', 'log_url': 'https://<some-app-name>.scm.azurewebsites.net/api/deployments/latest/log', 'site_name': '<some-app-name>'}. 
    Please run the command az webapp log deployment show -n <some-app-name> -g <some-resource-group>
##[error]Script failed with exit code: 1

The error is not very descriptive, like it often is. So what’s going on here and how can we solve it?

The deployment logs

The trick is to find the deployment-logs of the web app, you can use the Azure CLI to execute the mentioned command az webapp log deployment show as mentioned in the error. If you haven’t installed the Azure CLI: the Azure portal also has a web based cloud shell you can use. The command output shows the logs of the latest deployment, but this was not very descriptive either. A details-url is attached though, which shows the logs in json format.

It’s easier though to just open the deployment logs on the Kudu console, using the Azure Portal > Your App Service > Advanced Tools menu option. Or deeplink to it, using: https://<some-app-name>.scm.azurewebsites.net/deploymentlogs/. This page gives a basic but nice overview of all the deployments.

In my case, when I opened the logfiles of the failing deployments, it showed me an extra section I hadn’t seen in the logs of successful deployments: (I removed some lines for readability)

2022-03-17T10:36:21  Running oryx build...
2022-03-17T10:36:21    Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform php --platform-version 7.4 -i /tmp/8da0801fa1dd3c7 --log-file /tmp/build-debug.log 
2022-03-17T10:36:24    Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
2022-03-17T10:36:24    You can report issues at https://github.com/Microsoft/Oryx/issues
2022-03-17T10:36:24    
2022-03-17T10:36:24    Detecting platforms...    
2022-03-17T10:36:24    Could not detect any platform in the source directory.
2022-03-17T10:36:27    Error: Couldn't detect a version for the platform 'php' in the repo.
2022-03-17T10:36:27  
2022-03-17T10:36:28  Deployment Failed.

The culprit: Oryx

Apparently, Oryx is causing the problem here. Oryx has been around for some time and is a build system created by Microsoft. It’s integrated into Kudu, the source control manager (SCM) that’s deployed with every app service and that is reachable through the https://<some-app-name>.scm.azurewebsites.net endpoint. The idea is that you can arrange continuous deployment by connecting a web app with a git repository. Oryx will build your changes for you, as soon as you push them to the repository. In this scenario you don’t need to create a CI/CD pipeline. It’s an easy way to deploy your web apps and it’s really powerful: Oryx can build dotnet apps, python apps, php, ruby, nodejs apps and even hugo sites.

If you have enabled this type of deployment for your site, the files of your git repository are copied to the /home/site/repository folder after every change in the main branch. Oryx will start building after that and output the files to wwwroot. In my case, I’m building the code myself through a DevOps pipeline and I’m using a zip deployment so the code is pushed to a different folder: /tmp/zipdeploy/extracted. So why is Oryx running a build all of a sudden? I’m not sure here, but I think the clue is in the following sentence from the docs:

The idea is probably that Microsoft wants Oryx to be invoked at all times, with the purpose of building and starting the application. And if it doesn’t need to be built, it still needs to be started up. As you can see in the loglines above, Oryx is starting a PHP build. This is because my web app target framework is configured as PHP. I’m not actually serving php, I just use this framework to be able to serve HTML and other static content. Oryx is trying to detect the php version of my code, it doesn’t find it, and hence it breaks.

The solution

…using appsettings

In my case, I’m pushing static content that Oryx does not have a build image for, so I just want to disable the Oryx build altogether. Luckily Oryx is configurable through appsettings. Read more here. and here.

I tried the following appsettings:

• SCM_DO_BUILD_DURING_DEPLOYMENT=false - This appsetting should disable the build entirely. But it did not have any effect for me, which is strange, because it is the documented way to go about this.
• DISABLE_PHP_BUILD=true - I also tried this one to manipulate the Oryx build, but Oryx failed on this as well, because now there is nothing it can do.
• ENABLE_ORYX_BUILD=false - This is what eventually worked for me. According to the documentation, this forces the App Service to use the pre-oryx Kudu build system.

Eventually I chose to use the ENABLE_ORYX_BUILD setting as it was easiest to configure. But there are other routes available.

…using CLI

There’s also another route to take. The Azure CLI has a second web app deployment command which works almost the same as the one I used. It has fewer options though. It works as follows:

az webapp deployment source config-zip --src 'path/to/zip' --resource-group 'some-resource-group' --name 'some-app-name'

This command uses a different mechanism and Kudu does not trigger an Oryx build here.

…using Startup command

The Oryx documentation has one specific sentence that springs out in this context. It describes the COMMAND appsetting as ‘provide an alternate build-and-run script. Bypasses automatic build completely.’ It also refers to the startup command option that app services have. So what I did was create an empty bash file called startup_script.sh. I added some dummy bash:

#!/bin/bash
echo "Bypassing oryx"

Next, I uploaded this using FTP to the site folder. (The folder above wwwroot). I initially configured the Startup Command option in the General settings tab of my app service with the value /home/site/startup_script.sh and ran my deployment script. Unfortunately, this still produced an error. Oryx seemed to ignore the startup script. I then created an appsetting called COMMAND and added the same path of the startup file there. Now, the deployment ran fine. My deployment logs contained the following lines:

2022-03-22T09:27:51  Running custom deployment command...
2022-03-22T09:27:52  Not setting execute permissions for /home/site/startup_script.sh
2022-03-22T09:27:52  Running deployment command...
2022-03-22T09:27:52    Command: /home/site/startup_script.sh
2022-03-22T09:27:52    Bypassing Oryx

Parting words

I’m still not sure whether we’re not also dealing with a bug here. I’d expect the SCM_DO_BUILD_DURING_DEPLOYMENT appsetting to be enough. And I’d also expect the azure webapp deploy command to bypass build. However it is, I learned some new things while searching for the solution. And I hope you did too. And thanks for reading! If you have any additions or feedback about the content of this blog, please contact me. I’m no expert on Oryx and the bowels of Kudu, so I’d like to be corrected if necessary.

Sources

• Oryx & App Services
• Oryx Configuration
• App Service Zip deployment
• Azure CLI webapp deployment command