Performance Archives - Blog IT

Query performance for JSON objects inside SQL Server using JSON_VALUE function

Gonçalo Melo — Thu, 20 Dec 2018 09:39:12 +0000

Following up on this article about querying JSON Data I would like to talk about how to improve searches on JSON data inside SQL Server. Starting SQL Server 2016 Microsoft deployed a set of functions that allow us to work with JSON data in a structured way inside SQL Server.

I will introduce a small usage sample for the JSON_VALUE function in combination with indexes to improve information retrieval from a table containing one JSON object. For our testes, we have a UserDetailTest table that has more than 500k rows with 2 columns: an UserId and a nvarchar(max) to hold a small JSON details string like the following:

SELECT *, LEN(DetailsJSON) AS [Len(DetailsJSON)] FROM UserDetailTest

I will activate time statistics and clean SQL Server cache between each query to have some consistency across the execution times using the following SQL statements:

CHECKPOINT
DBCC DROPCLEANBUFFERS
SET STATISTICS TIME ON

JSON_VALUE Function intro

The JSON_VALUE function extracts a value from a JSON string. This functions receives 2 arguments, the first being an expression for the JSON value and the second a path for the value we want to obtain. A simple sample with an inline JSON object would be the following:

SELECT JSON_VALUE('{"PostalCode":"376-3765","PhoneNumber":"351003765718"}', '$.PhoneNumber')

The return is a scalar value (nvarchar(4000)) for the PhoneNumber element.

Query scenarios

Starting with one of the really worst case scenario (yes, this can happen…):

More than 17 seconds really seems like a worst scenario! Next, we will use the JSON_VALUE function to get the PhoneNumber from the JSON string and use it in our where clause:

This still takes more than 3 seconds. A good improvement, but yet a high cost if we need to search information in this way.

Index creation

Let’s add a new virtual column to table that displays the result from the JSON_VALUE function – this will allow us to create an index and simplify the SELECT queries.

ALTER TABLE UserDetailTest 
	ADD vPhoneNumber AS 
	CAST(JSON_VALUE(DetailsJSON, '$.PhoneNumber') AS NVARCHAR(255));

The cast truncates the output from the JSON_VALUE to ensure that it does not exceed the maximum lenght for the index key. Now, if we search using the column, the result is still more or less the same 3 seconds has before:

Let’s create an index and perform the search again:

CREATE INDEX idx_vPhoneNumber ON [UserDetailTest] (vPhoneNumber);

Consequently the improvement is huge, 13 miliseconds is more interesting! But what’s the cost?

The cost

When we create an index, we’re basically trading space for time – more occupied space versus faster operations. Therefore let’s see what the increase is using sp_spaceused function before and after index creation:

	rows	reserved	data	index_size	unused
Initial	557801	1120904 KB	1115584 KB	5224 KB	96 KB
After Index	557801	1148752 KB	1115584 KB	32992 KB	176 KB

Probably, an interesting comparison would be if the PhoneNumber column was an explicit column in that table containing the value. Let’s do that!

ALTER TABLE UserDetailTest  
	ADD PhoneNumberCopy NVARCHAR(255)

UPDATE UserDetailTest
	SET PhoneNumberCopy = vPhoneNumber

The sp_spaceused remained the same – SQL Server internal black magic regarding space allocation (for another time!). But performance wise, for this approach in my machine this query still took more than 2 seconds to complete:

Slightly better than the query with the JSON function but the advantage is that in this scenario SQL can better optimize the searches. The following runs of the same query without clearing the cache returns results much faster – each took around 220 miliseconds to complete:

Summary Results

Just to sum up all the results for the different types of searches comparing the first execution after cache clean up and the following runs. I did several iterations for each step just to make sure the results were consistent although the goal was just to have a baseline. Certanly, the use of this will depend on each case.

WHERE clausule type	First run	Following runs
WHERE DetailsJson LIKE '%PhoneNumber":"351003765718"%'	17754 ms	around the same
WHERE JSON_VALUE(DetailsJson, '$.PhoneNumber') = '351003765718'	3375 ms	1958 ms
WHERE vPhoneNumber = '351003765718' vPhoneNumber is not indexed	3339 ms	1332 ms
WHERE vPhoneNumber = '351003765718' vPhoneNumber is indexed	13 ms	0 ms
WHERE PhoneNumberCopy = '351003765718' PhoneNumberCopy is of type nvarchar(255) and explicitly contains all values	2309 ms	222 ms

The post Query performance for JSON objects inside SQL Server using JSON_VALUE function appeared first on Blog IT.

Straight A’s in WebPagetest with Umbraco

André Santos — Tue, 27 Nov 2018 21:40:06 +0000

Before launching a new website, there’s a checklist I go through, to make sure that everything is ready. One of the items in my checklist is to test the website against WebPagetest.

WebPagetest is a tool that was originally developed by AOL for use internally and was open-sourced in 2008 under a BSD license. The online version at www.webpagetest.org is run for the benefit of the performance community with several companies and individuals providing the testing infrastructure around the globe.

This tool tests any website against 6 major performance affecting factors, and provides a myriad of graphs and logs that make abundantly clear what might be slowing down your site.

In this post I’ll provide ways to make your site get straight A’s in WebPagetest.

Setup

To start this off, let’s setup our environment. We’ll just need the following:

Visual Studio 2017
Microsoft Azure account

In Visual Studio, let’s create a new empty ASP.NET Web Application project. Then, we’ll need the latest and greatest Umbraco NuGet package (I used version 7.12.4). Once it finishes installing, just launch the website and install Umbraco with all defaults. This will bootstrap Umbraco with the Starter Website, which we’ll use as our “guinea pig” for WebPagetest.

Next: publish it! We can use the publish wizard to automatically create our new WebApp and SQL Database in Azure. Before installing Umbraco in Azure, we’ll need to change the Web.config so that the install wizard is run again (I use Filezilla to change it in Azure):

Clear the Umbraco version number:

Clear the Umbraco connection string:

With these changes in place, we’re good to go. This time, we’ll not use the defaults in the Umbraco install wizard, since we’ll want to use the SQL Database we’ve just created in Azure.

First test

For our tests, we’ll use the people page of the Starter Website. This is the score I got with a standard (S0) database and a basic WebApp:

Since this is a very small site, only used for demonstration purposes, half of the metrics are already cleared! However, this is not usually the case for bigger websites. For this reason, I’ll still present some solutions to improve the grade for these metrics.

First Byte Time

These test measures the time it takes for the first byte to reach the client’s browser after the initial http request.

There are two main factors that influence this result:

Server power
The webpage complexity (integrations with external services, complex logic involved, etc)

How to get an A

The easiest way to mitigate this problem is by caching. You can see how to do output caching in Umbraco by reading this old post of mine: https://blogit.create.pt////andresantos/2016/06/30/umbraco-and-donut-output-cache/.

Keep-alive Enabled

Keep alive is a method to allow the same tcp connection for HTTP conversation instead of opening a new one with each new request.

How to get an A

This setting is active by default in IIS, so, it’s also active by default in Azure WebApps, so it’s easy to get an A!

Compress Transfer

Gzip compresses your webpages, style sheets and javascripts, before sending them over to the browser. This drastically reduces transfer time since the files are much smaller.

How to get an A

Just add this setting to your Web.config file:

Compress Images

Image compression is minimizing the size in bytes of a graphics file without degrading the quality of the image to an unacceptable level.

How to get an A

In Umbraco, to get an A in this grade, you need to do two things:

Crop every image and use srcsets where you can
Use the PostProcessor plugin for Image Processor

Cropping an image is easy in Umbraco:

In order to use the PostProcessor plugin, you just need to install it via nuget: https://www.nuget.org/packages/ImageProcessor.Web.PostProcessor/1.3.1.25.

Cache Static Content

Static content is content that changes rarely. For this reason it can be cached in the user’s browser to avoid downloading the same file over and over again.

How to get an A

Just set the time it takes for the content to expire in the user’s browser and add extra mime types if you want:

Effective use of CDN

A content delivery network (CDN) refers to a geographically distributed group of servers which work together to provide fast delivery of Internet content. A CDN allows for the quick transfer of assets needed for loading Internet content including HTML pages, javascript files, stylesheets, images, and videos.

How to get an A

In Umbraco, you can achieve this last grade by doing two things:

Use an Azure Blob Storage for media storage by installing this nuget package: https://github.com/JimBobSquarePants/UmbracoFileSystemProviders.Azure.
Create an Azure CDN for serving these blobs through a content delivery network.

After creating an Azure CDN service and waiting about an hour for it to be available, the following presents my configuration for the media assets to be provided by it:

Conclusion

So, if you followed these tips correctly, you’ll be able to run WebPagetest and get the same result as I did:

You can find the complete report here: https://www.webpagetest.org/result/181127_2A_bea6941dcd20d38ab54c29409fca9363/.

The post Straight A’s in WebPagetest with Umbraco appeared first on Blog IT.

Optimizing a VM for SharePoint 2010

André Vala — Sat, 06 Mar 2010 11:38:39 +0000

Creating an optimized SharePoint development environment was not an easy task with Sharepoint 2007, but it’s even harder with the new SharePoint 2010. The new version has a lot more features but, because of that, it’s also much more resource demanding, and that will make you sweat when building a new virtual machine to perform well on a laptop.

Following Raul’s post on SharePoint 2010 Tips for Performance, and because there seems to be a lot of people in the community facing the same issues I have, I’ll be describing in this article some recommendations and best practices that I’ve been collecting here and there.

Some of these recommendations apply not only to development virtual machines, but also to non virtualized production servers.

Creating the Virtual Machine

When creating the virtual machine, you should take the following hardware requirements into account:

Allow your virtual machine to use all CPU cores available in the host
Allow your virtual machine, at least, 4GB of RAM
Place your virtual machine in the fastest hard drive of the computer
Use e-Sata external drives because USB 2.0 and FireWire are too slow
Use 7200 RPM drives or, ideally, SSD drives

To avoid waste of resources, configure your VM and turn off everything you don’t need:

CD Drive
USB Support
Printer Support

Additionally, access the BIOS for the VM and disable the serial and parallel ports.

Optimizing the Operating System

The operating system to use is Windows Server 2008 R2 64-bits. The standard edition is enough for a development environment, but be sure to check out the other versions in case you need some feature that is only present in the enterprise version.

In a development environment, there are several optimizations that can help you free some resources:

In the network configuration options, uninstall the QoS Packet Scheduler and the Link Layer Topology Discovery components, and uncheck IPv6.
Turn off the Windows Firewall.
In the Computer properties > Advanced > Performance Options > Visual Effects, select the option Adjust for best performance.
In the Control Panel > Hardware > Sound, don’t turn on the Windows Audio Service and select No Sounds as the Sound Scheme.
Turn off IE ESC for both administrators and users

The pagefile is also a possible optimization target:

If possible, place it in a disk different from the OS disk
Define the pagefile initial dimension to be the amount of RAM of the machine, and the final dimension to be the triple of the amount of RAM of the machine (if you have enough disk space)

Whenever possible, it is a good idea to defragment the disks before shutting down the machine (this also applies to the host machine).

Optimizing SQL Server

SQL Server is where SharePoint stores all the data and, for that reason, is one of the main causes of the high resource consumption. You should use SQL Server 2008 64-bits SP1 with CU2 (or higher) or, ideally, SQL Server 2008 R2 64-bits. The important thing here is that it’s Enterprise or Developer edition.

Storage

One of the main optimization points for SQL Server is storage, that is, disk performance. To assure the best possible performance, you should create additional virtual disks so that you can separate in different disks:

SQL Server’s TempDB
Database data files (.mdf e .ldf)
Database log files (.ldf)

These disks should have the following characteristics:

Single file
Pre-allocated size
NTFS Formatted using 64K blocks

Ideally, each of these virtual disks should be stored in a distinct physical disk, but when you intend to use the virtual machine in a laptop, it’s not easy to carry 3 or more external hard drives in your backpack.

Anyway, even if you don’t have enough physical hard drives, the virtual disks should be stored in physical disks according to their speed, using the following order:

TempDB disk (fastest)
Log file disk
Data file disk
System disk (slowest)

Data Files

By default, SQL Server specifies small sizes for the data files and uses small auto-growth sizes to save disk space. However, the auto-growth operation as well as the fact that they grow in small blocks, causes performance degradation as well as fragmentation of the files in the disk.

To increase the performance you should:

Use more than one data file in the primary file group for the TempDB, search DBs and content DBs of SharPoint.
Always use, at most, one data file for each CPU core of the system. Ideally, you should use one data file for each 2 CPU cores.
If possible, store the data files in different physical disks (very hard for a development environment).
Pre-allocate all data files, including the data files for TempDB.
The data files for each DB should be equally sized.
Specify a size for the data files that is enough to avoid auto-growth (for the TempDB, the total size should be 10% of the total size of the largest DB of the system).
Specify data file growth size to be 256MB or 512MB

Lock Pages in Memory

Another important configuration, is to authorize SQL Server to use the Lock Pages in Memory feature to keep Windows from unloading the memory pages that might still be needed, leading to a performance degradation (more information in: http://support.microsoft.com/kb/918483).

Optimizing SharePoint

Use SharePoint’s complete installation, instead of the stand alone installation, so that it doesn’t install a new SQL Server instance or automatically configures the Farm.

After the installation, configure SharePoint turning on only the features that you absolutely need. Each unnecessary service you turn on will mean less memory and CPU time available for the services that really matter.

The post Optimizing a VM for SharePoint 2010 appeared first on Blog IT.