How to implement a quick CI/CD for Azure Data Factory using Azure DevOps (2024)

Highlight

If you need to set up CI/CD for your Data Factory, here is how to do it in less than 30 minutes!

Summary

Many of us are using Data Factory almost daily for our Data Engieering jobs. But not many of us are doing it by leveraging proper DevOps continous delivery standards. In this small tutorial I’m sharing templates that you can add to your data factory in just 30 minutes and have nice automated deployments to your environments.

Setup guide

Step 1 - Azure Data Factory

1. Create folder /devops/ and then, in that folder create following files
   1. Create file package.json

      { "scripts":{ "build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index" }, "dependencies":{ "@microsoft/azure-data-factory-utilities":"^0.1.5" }} 

   2. Create file adf-build-job.yml

      parameters:- name: subscriptionId  type: string - name: resourceGroupName  type: string- name: dataFactoryName  type: string- name: repoRootFolder type: string default: /- name: packageJsonFolder type: string default: /- name: artifactName type: string default: data-factoryjobs:- job: BUILD displayName: 'Build ARM Template' variables: workingDirectory: $(Build.Repository.LocalPath)${{ parameters.repoRootFolder }} packageJsonFolder: $(Build.Repository.LocalPath)${{ parameters.packageJsonFolder }} dataFactoryResourceId: /subscriptions/${{ parameters.subscriptionId }}/resourceGroups/${{ parameters.resourceGroupName }}/providers/Microsoft.DataFactory/factories/${{ parameters.dataFactoryName }} artifactTempDirectory: data-factory-arm steps: - task: NodeTool@0 inputs: versionSpec: '14.x' displayName: 'Install Node.js' - task: Npm@1 inputs: command: 'install' workingDir: $(packageJsonFolder) verbose: true displayName: 'Install npm package' - task: Npm@1 inputs: command: 'custom' workingDir: $(packageJsonFolder) customCommand: 'run build validate $(workingDirectory) $(dataFactoryResourceId)' displayName: 'Validate' - task: Npm@1 inputs: command: 'custom' workingDir: $(packageJsonFolder) customCommand: 'run build export $(workingDirectory) $(dataFactoryResourceId) "$(artifactTempDirectory)"' displayName: 'Validate and Generate ARM template' - script: 'mv -v $(packageJsonFolder)$(artifactTempDirectory)/${{ parameters.dataFactoryName }}_GlobalParameters.json $(packageJsonFolder)$(artifactTempDirectory)/GlobalParameters.json' displayName: 'Rename Global Parameter File post' - task: PublishPipelineArtifact@1 inputs: targetPath: '$(packageJsonFolder)$(artifactTempDirectory)' artifact: ${{ parameters.artifactName }}

2. Create new pipeline under path /devops/adf-azure-pipelines.yml, paste in the sample code

   trigger:- mainpool: vmImage: ubuntu-lateststages:- stage: BUILD jobs: - template: <path_to_adf-build-job.yml_file> parameters: subscriptionId: <subscription_id> resourceGroupName: <resource_group_name> dataFactoryName: <data_factory_name> repoRootFolder: <absolute_path_to_datafactory_folder, ex. /data-factory/> packageJsonFolder: <absolute_path_to_package_json, ex. /devops/)

   See Also

   Azure Pipelines Agents - Azure PipelinesDeploy to Linux VMs - Azure PipelinesRedeploy Windows virtual machines in Azure - Virtual MachinesDeployment best practices - Azure App Service
   1. Replace temp variables with your env values
3. Run and test pipeline
4. In /devops/ folder create file adf-deploy-job.yml

   parameters:- name: environmentName  type: string - name: serviceConnectionName  type: string - name: subscriptionId  type: string - name: resourceGroupName  type: string - name: dataFactoryName  type: string - name: location  type: string  default: westeurope- name: artifactName type: string default: data-factory- name: overrideParameters type: string default: jobs:- deployment: ${{ parameters.environmentName }} displayName: Deployment to ${{ parameters.environmentName }} variables: - name: artifactsDirectory value: $(System.ArtifactsDirectory)/data-factory environment: '${{ parameters.environmentName }}' strategy: runOnce: deploy: steps: - script: echo Deploying to ${{ parameters.environmentName }} displayName: 'Script - Display Environment Stage Name' - task: DownloadPipelineArtifact@2 displayName: 'Download ADF ARM Template' inputs: source: current artifact: ${{ parameters.artifactName }} downloadPath: $(artifactsDirectory) - script: 'ls $(artifactsDirectory)' displayName: 'List Artifact contents' - task: AzurePowerShell@5 displayName: 'Stop Triggers' inputs: azureSubscription: '${{ parameters.serviceConnectionName }}' ScriptPath: '$(artifactsDirectory)/PrePostDeploymentScript.ps1' ScriptArguments: "-armTemplate $(artifactsDirectory)/ARMTemplateForFactory.json \ -ResourceGroupName ${{ parameters.resourceGroupName }} \ -DataFactoryName ${{ parameters.dataFactoryName }} \ -predeployment $true \ -deleteDeployment $false" azurePowerShellVersion: LatestVersion - task: AzureResourceManagerTemplateDeployment@3 displayName: 'ARM Template deployment' inputs: azureResourceManagerConnection: '${{ parameters.serviceConnectionName }}' subscriptionId: '${{ parameters.subscriptionId }}' resourceGroupName: '${{ parameters.resourceGroupName }}' location: ${{ parameters.location }} csmFile: '$(artifactsDirectory)/ARMTemplateForFactory.json' csmParametersFile: '$(artifactsDirectory)/ARMTemplateParametersForFactory.json' overrideParameters: > -factoryName ${{ parameters.dataFactoryName }} ${{ parameters.overrideParameters }}  - task: AzurePowerShell@5 displayName: 'Deploy Global Parameters' inputs: azureSubscription: '${{ parameters.serviceConnectionName }}' ScriptPath: '$(artifactsDirectory)/GlobalParametersUpdateScript.ps1' ScriptArguments: "-globalParametersFilePath $(artifactsDirectory)/GlobalParameters.json \ -resourceGroupName ${{ parameters.resourceGroupName }} \ -dataFactoryName ${{ parameters.dataFactoryName }}" azurePowerShellVersion: LatestVersion - task: AzurePowerShell@5 displayName: 'Start Triggers' inputs: azureSubscription: '${{ parameters.serviceConnectionName }}' ScriptPath: '$(artifactsDirectory)/PrePostDeploymentScript.ps1' ScriptArguments: "-armTemplate $(artifactsDirectory)/ARMTemplateForFactory.json \ -ResourceGroupName ${{ parameters.resourceGroupName }} \ -DataFactoryName ${{ parameters.dataFactoryName }} \ -predeployment $false \ -deleteDeployment $false" azurePowerShellVersion: LatestVersion

5. Navigate to Project Settings » Service Connections and create new connection to Azure using Service Principal and grant at least Data Factory Contributor role to all data factories that you will be deploying to
   1. In Azure Portal navigate to Azure Active Directory and create new App Registration
   2. For ADF only piplines grant Data Factory Contibutor role on Azure Data Factory resource, or for full CI/CD in Azure grant Contributor role to an entire resource group
   3. Copy the details of this service principal and subscription to Azure DevOps
6. Create Environment (Pipelines » Environment)
7. Create Variable Group (Pipelines » Library » Variable Groups)
   1. Add variables that needs to be overriden in ADF template (factoryName is overriden by default)
8. Edit your adf-azure-pipelines.yml pipeline/file and add this stage

   - stage: <stage_name> variables: - group: <variable_group_name> jobs: - template: <path_to_adf-deploy-job.yml_file> parameters: environmentName: <environment_name> subscriptionId: <subscription_id> resourceGroupName: <resource_group_name> dataFactoryName: <data_factory_name> location: <data_factory_region> serviceConnectionName: <service_connection_name> overrideParameters: >  -<param_1> <val_1>

   1. Replace temp variables with your env values
9. Run and test
10. Add any extra deployment stages you need by repeating steps 5, 6, 7, 8, 9

Hope this helps! Good luck!