The pipelines bring to us the opportunity to create automatic tasks to execute development operations, like deploys, migrations, tests, etc. Focused in the deploy routine, some applications need other operations to ensure the stability between the application and other resources, like the database.

During the development process, the developers usually customize and improve the application database in the development environment scope, and it can be made easily by using frameworks that contains routines to create source files that have the responsability to keep the history of changes and are used to update the database for determinated version. These files are known as migrations. 

In this post, we will see how to apply migrations for the databases that are located in different environments (staging, production, etc.), using the Code-First entity concept, and the Entity Framework Core.

Migrations

To generate database migrations, we can execute the “Add-Migraton <name>” command from the Entity Framework. This command will generate some source files that will contain the code to apply the migration to the database. To execute these migration files, you just need to execute the “Update-Database” command. But, Microsoft recommends to create a SQL file individually, and suggest to execute this file direct to the database, without need to execute an intermediary software (C# + EF) to access the database. To do this, we will have to follow the steps below:

• 1. Create a Pipeline in Azure Devops;
• 2. Install dotnet-ef-tool;
• 3. Generate SQL script;
• 4. Execute SQL script to the database;
• 5. Script File as a Pipeline Artifact.

NOTE: We don’t need to worry about which migration need to be applyed to the database. The dotnet-ef-tool command will generate a script cantaining the needed logic. Check the third step for more details.

1. Create a Pipeline in Azure Devops

We will create a common deployment pipeline in Azure Devops for ASP.NET Core backend applications. Check the end of the post to see the full result of the pipeline.

2. Install dotnet-ef-tool

To execute tool commands of Entity Framework, we have to install a package named dotnet-ef-tool. This needs to be available into the pipeline’s context to be used for the SQL script generation. To install this tool by a pipeline task, we can use the task below:

- task: DotNetCoreCLI@2
  displayName: 'Install EF tool'
  inputs:
    command: 'custom'
    custom: 'tool'
    arguments: 'install --global dotnet-ef'

3. Generate SQL script

To generate the SQL script containing the logic to apply migrations, we will use the dotnet-ef-tool. By default, the command does not generate the logic to check which migration should be applied, so we have to enable this behavior using the parameter: –idempotent. By using this behavior, we don’t need to worry about the migrations that will be applyed to the database since the generated script already has it. Below, you can see the task that could be used to generate the script file:

- task: DotNetCoreCLI@2
  displayName: 'Generate SQL Script'
  inputs:
    command: 'custom'
    custom: 'ef'
    arguments: 'migrations script --idempotent --project $(Build.SourcesDirectory)\Data.csproj --output $(System.ArtifactsDirectory)/script.sql'

NOTE: We don’t need to stablish any database connection for the script generation, since we are following the Code First concept for entity creation, so, the command will just look at the entities source code of the project to generate the SQL file. If we don’t use the –idempotent argument, the script will generate the SQL without the conditional logic to determinate which migration should be applied, causing an error if the database already exists.

4. Execute SQL script to the database

Now, with the SQL script available, we just need to execute this file to the database. In this post, we will use a common Azure Database as an example, so for this case, we can use the following task to accomplish this requirement:

- task: SqlAzureDacpacDeployment@1
  displayName: 'Update Database'
  inputs:
    azureSubscription: '<Service Connection Identifier>'
    AuthenticationType: 'connectionString' # You can use other method to authenticate to the database if you want
    ConnectionString: '<Database Connection String>'
    deployType: 'SqlTask'
    SqlFile: '$(System.ArtifactsDirectory)/script.sql'

5. Script File as a Pipeline Artifact

Well done. The last task will only be used to make the file available to be accessed by the pipeline runner user. To do this, you can use the task below to add the generated script.sql file to the file bundle (artifacts). To check the artifacts, just access the artifacts option from the pipeline details after the job’s execution.

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts'
  inputs:
    PathtoPublish: '$(System.ArtifactsDirectory)/script.sql'
    ArtifactName: '$(artifactName)'
    publishLocation: 'Container'

Conclusion

In this post, we could learn an easy way to apply the database migrations to the remote environment by using automation (pipelines) to accomplish it. This method is simple, and does not have any database validation before script execution. So, if you have an active and sensitive environment, be sure that the needed validations are considered before the database update, to ensure that the database will not be update wrongly.

Pipeline Result

trigger:
- none

pool:
  vmImage: windows-latest

variables:
  solution: '**/*.sln'
  buildPlatform: 'AnyCPU'
  buildConfiguration: 'Release'
  artifactName: 'artifacts'
  environment: 'Development'

- task: NuGetToolInstaller@1
  displayName: 'NuGet Installation'

- task: NuGetCommand@2
  displayName: 'NuGet Restore'
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  displayName: 'Build Project'
  inputs:
    solution: 'backend/WebAPI'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: PublishSymbols@2
  displayName: 'Publish Symbols'
  inputs:
    SearchPattern: '**/bin/**/*.pdb'
    SymbolServerType: 'FileShare'
    CompressSymbols: false

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts'
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'artifacts'
    publishLocation: 'Container'

- task: DownloadBuildArtifacts@0
  displayName: 'Build Artifacts'
  inputs:
    buildType: 'current'
    downloadType: 'single'
    artifactName: '$(artifactName)'
    downloadPath: '$(System.ArtifactsDirectory)'

- task: DotNetCoreCLI@2
  displayName: 'Install EF tool'
  inputs:
    command: 'custom'
    custom: 'tool'
    arguments: 'install --global dotnet-ef'

- task: DotNetCoreCLI@2
  displayName: 'Generate SQL Script'
  inputs:
    command: 'custom'
    custom: 'ef'
    arguments: 'migrations script --idempotent --project $(Build.SourcesDirectory)\Data.csproj --output $(System.ArtifactsDirectory)/script.sql'

- task: SqlAzureDacpacDeployment@1
  displayName: 'Update Database'
  inputs:
    azureSubscription: '<Service Connection Identifier>'
    AuthenticationType: 'connectionString'
    ConnectionString: '<Connection String>'
    deployType: 'SqlTask'
    SqlFile: '$(System.ArtifactsDirectory)/script.sql'

- task: AzureRmWebAppDeployment@4
  displayName: 'Deploy Application'
  inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: '<Service Connection Identifier>'
    appType: 'webApp'
    WebAppName: 'armazemlegalapi'
    deployToSlotOrASE: true
    ResourceGroupName: '<Resource Group Identifier>'
    SlotName: 'production'
    packageForLinux: '$(System.ArtifactsDirectory)/Project.zip'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts'
  inputs:
    PathtoPublish: '$(System.ArtifactsDirectory)/script.sql'
    ArtifactName: '$(artifactName)'
    publishLocation: 'Container'

References

• Microsoft Managing Schemas Article

The post Entity Framework – Update Database using Azure Pipelines appeared first on Blog IT.

From the get-go there were some challenges we needed to overcome in order to create an usable system, that was user-friendly, fast, reliable, and where no conflitcts would come up when data was being updated from multiple users.

The solution was a new Umbraco Package we called Search Manager (since the database tables ultimately feed an Azure Search index) that uses Telerik’s Kendo UI Grid, WebAPI and Entity Framework. The following presents the process it took to reach the end result.

Step 1

We created a simple Umbraco Package that implemented a Kendo UI Grid which got its data from a WebAPI method that returned all rows from a table using Petapoco.

This allowed us to have the complete table data in memory using a Kendo UI Grid. It worked great but had major problems:

• A table with around 40k rows was actually a 40 MBs json download in order to populate the grid
• Users could be working with outdated data, since it was loaded in memory and changes could have been made to the source

Step 2

We started using the server paging feature of the Kendo Grid. This dropped the download size greatly, but we still had problems:

• Even though we are working with only a subset of results client-side, we were still getting the complete table data server-side, using petapoco.

We needed an easy way to translate the client-side queries, such as filters, paging, order, etc to SQL queries.

Step 3

We changed the type of queries being generated by the Grid to OData. Using the OData WebAPI nuget package we are able to map the odata query  to a C# object and then directly apply the query to any IQueryable data.

Petapoco does not provide any IQueryable data support, so we changed our ORM to Entity Framework, which does, and voilá!

Finally

With everything in place, we are able to have a filtered subset of results in a Grid, that gets exactly what it needs to display from a dynamically generated SQL query.

• It is fast because we only get exactly what we need:

An OData query generated by the Kendo UI Grid looks like this:

http://localhost:14231/umbraco/backoffice/SearchManager/PricesApi/Read?%24inlinecount=allpages&%24top=20&%24skip=40&%24orderby=ProviderName+desc&%24filter=(NatureProcCode+eq+%27exames%27+and+AvgAmtClaimed+gt+10)

Which is translated to the follwing SQL query:

exec sp_executesql N'SELECT 
    [Project1].[Approved] AS [Approved], 
    [Project1].[DiffWeightedRate] AS [DiffWeightedRate], 
    [Project1].[SubmitPrvId] AS [SubmitPrvId], 
    [Project1].[PracticeSeq] AS [PracticeSeq], 
    [Project1].[NatureProcCode] AS [NatureProcCode], 
    [Project1].[GroupProcCode] AS [GroupProcCode], 
    [Project1].[Network] AS [Network], 
    [Project1].[AmtClaimedWeighted] AS [AmtClaimedWeighted], 
    [Project1].[ManualPrice] AS [ManualPrice], 
    [Project1].[IsImported] AS [IsImported], 
    [Project1].[DiffCalculatedRate] AS [DiffCalculatedRate], 
    [Project1].[AmtContract] AS [AmtContract], 
    [Project1].[AvgAmtClaimed] AS [AvgAmtClaimed], 
    [Project1].[ProviderName] AS [ProviderName], 
    [Project1].[ClinicName] AS [ClinicName]
    FROM ( SELECT 
        [Extent1].[Approved] AS [Approved], 
        [Extent1].[DiffWeightedRate] AS [DiffWeightedRate], 
        [Extent1].[SubmitPrvId] AS [SubmitPrvId], 
        [Extent1].[PracticeSeq] AS [PracticeSeq], 
        [Extent1].[NatureProcCode] AS [NatureProcCode], 
        [Extent1].[GroupProcCode] AS [GroupProcCode], 
        [Extent1].[Network] AS [Network], 
        [Extent1].[AmtClaimedWeighted] AS [AmtClaimedWeighted], 
        [Extent1].[ManualPrice] AS [ManualPrice], 
        [Extent1].[IsImported] AS [IsImported], 
        [Extent1].[DiffCalculatedRate] AS [DiffCalculatedRate], 
        [Extent1].[AmtContract] AS [AmtContract], 
        [Extent1].[AvgAmtClaimed] AS [AvgAmtClaimed], 
        [Extent1].[ProviderName] AS [ProviderName], 
        [Extent1].[ClinicName] AS [ClinicName]
        FROM (SELECT 
    [PriceGrid].[Approved] AS [Approved], 
    [PriceGrid].[DiffWeightedRate] AS [DiffWeightedRate], 
    [PriceGrid].[SubmitPrvId] AS [SubmitPrvId], 
    [PriceGrid].[PracticeSeq] AS [PracticeSeq], 
    [PriceGrid].[NatureProcCode] AS [NatureProcCode], 
    [PriceGrid].[GroupProcCode] AS [GroupProcCode], 
    [PriceGrid].[Network] AS [Network], 
    [PriceGrid].[AmtClaimedWeighted] AS [AmtClaimedWeighted], 
    [PriceGrid].[ManualPrice] AS [ManualPrice], 
    [PriceGrid].[IsImported] AS [IsImported], 
    [PriceGrid].[DiffCalculatedRate] AS [DiffCalculatedRate], 
    [PriceGrid].[AmtContract] AS [AmtContract], 
    [PriceGrid].[AvgAmtClaimed] AS [AvgAmtClaimed], 
    [PriceGrid].[ProviderName] AS [ProviderName], 
    [PriceGrid].[ClinicName] AS [ClinicName]
    FROM [dbo].[PriceGrid] AS [PriceGrid]) AS [Extent1]
        WHERE ([Extent1].[NatureProcCode] = @p__linq__0) AND ([Extent1].[AvgAmtClaimed] > @p__linq__1)
    )  AS [Project1]
    ORDER BY [Project1].[ProviderName] DESC, [Project1].[AmtClaimedWeighted] ASC, [Project1].[AmtContract] ASC, [Project1].[Approved] ASC, [Project1].[AvgAmtClaimed] ASC, [Project1].[ClinicName] ASC, [Project1].[DiffCalculatedRate] ASC, [Project1].[DiffWeightedRate] ASC, [Project1].[GroupProcCode] ASC, [Project1].[IsImported] ASC, [Project1].[ManualPrice] ASC, [Project1].[NatureProcCode] ASC, [Project1].[Network] ASC, [Project1].[PracticeSeq] ASC, [Project1].[SubmitPrvId] ASC
    OFFSET @p__linq__2 ROWS FETCH NEXT @p__linq__3 ROWS ONLY  option (Recompile)',N'@p__linq__0 varchar(8000),@p__linq__1 decimal(2,0),@p__linq__2 int,@p__linq__3 int',@p__linq__0='exames',@p__linq__1=10,@p__linq__2=40,@p__linq__3=20

• It’s user-friendly and reliable because users can filter the data as if they were using an excel file on steroids
• It avoids conflicts because we only get a subset of the results each time and each change is commited in a single transaction

Code samples:

grid = $("#grid").kendoGrid({
                dataSource: new kendo.data.DataSource({
                    transport: transport,
                    schema: schema,
                    serverFiltering: true,
                    serverPaging: true,
                    serverSorting: true,
                    type: "odata",
                    pageSize: 20
                }),
                dataBound: onDataBound,
                pageable: true,
                filterable: true,
                sortable: true,
                scrollable: false,
                navigatable: true,
                resizable: true,
                batch: true,
                columnMenu: true,
                editable: true,
                toolbar: [
					{ template: '<a class="k-button k-button-icontext k-grid-save-batch" href="\\#""><span class="k-icon k-i-update"></span>Guardar Alterações</a>' },
					{ name: "cancel", text: "Cancelar" },
                ],
                columns: [...]
            });

[HttpGet]
        public IHttpActionResult Read(ODataQueryOptions<PriceGrid> opts)
        {
            Mapper.CreateMap<PriceGrid, PriceGridBO>();

            using (var context = new EntitiesContext())
            {
                using (var qh = new HintScope(context, QueryHintType.Recompile))
                {
                    List<PriceGrid> results = opts
                        .ApplyTo(context.PriceGrids)
                        .Cast<PriceGrid>()
                        .ToList();

                    List<PriceGridBO> boEntities = results.Select(x => Mapper.Map<PriceGrid, PriceGridBO>(x)).ToList();

                    PriceResponse response = new PriceResponse()
                    {
                        Entities = boEntities,
                        Total = Request.ODataProperties().TotalCount.HasValue ? Request.ODataProperties().TotalCount.Value : 0
                    };

                    return Ok(response);
                }
            }
        }

The post KendoUI Grid, OData, WebAPI and Entity Framework appeared first on Blog IT.