How do we know collaboration is happening?

My reflection results from one recent team experience, in a software development context. We worked collaboratively to implement a new feature and after a work session two things about the final result called my attention:

1. It was not possible to identify the “owner” of the idea. Everyone said something that contributed to the work produced. It truly belonged to everyone.

2. The result we found was better than something I was aware of being able to find.

It is something I read, heard, and even talked about often. But for me it was good to be able to experience and learn in practice. There are few things that replace practice.

This is basic, it is. I knew that. But for me it was a special moment to be able to feel this energy. At the same time, I look at this and think “how come we don’t always do this?”.

How collaboration happens?

Some simple and practical actions to improve collaborative work.

Accept all contributions

The brain is very quick to identify reasons for not doing anything. Perhaps it is a hidden survival mechanism. As a result, idle is safer than boiling. This is a common place, on the other hand it is differentiating. The process of building on each other’s ideas is one of the critical factors to generate energy that will keep the work session active.

Feed the ideas that come up

Initially denying an idea because we see a point we disagree with inhibits collaboration. In other words, the team must be comfortable sharing. A flow of new ideas when each team member builds and contributes on top of each other’s ideas is fantastic. At the end it is not even possible to identify whose final idea is. There is only “us” in the result.

Desire to discover something new

When we restrict ourselves only to what we know, what we can discover will be limited. Dismantle pre-conceived ideas, especially our own. Being open minded to different points of view allows more contributions. It is easy and safe to say no. Take a chance and say why not? Or even better how can we achieve this?

Try It Out

Offer someone the opportunity to share their ideas. These are small but extremely important things, and they are also actionable. They are within everyone’s reach.

The post Collaborative work, why and how? appeared first on Blog IT.

NoSQL is like a new definition of something that is a database but is different than the usual relational model. Likewise, making a parallel by going back 30 years ago. Back then probably no one knew what a relational database was also. Let’s use this and start with a small historical and motivational point of view for NoSQL. PS: I was not a developer back then.

This article will be a first part of a series of articles. The goal will be to unite some knowledge about this topic.

Starting in the 1980’s

Relational databases emerged bringing ACID properties – Atomicity, Consistency, Isolation and Persistency. Properties taken for granted nowadays. Also, they also brought the SQL language. SQL is common enough across different systems for one to use. Although there are different flavors, can almost be considered a standard.

Relation databases also allowed for a simple and very common integration mechanism between 2 or more systems. Data can be easily shared by reading or writing data in a table on a shared relational database. This is today still a very common integration pattern.

In the 1990’s

Object database’s started to appear with more strength. They had been around for some time. Most important, they implement a different database paradigm model. This new paradigm tried to solve the impedance mismatch problem. The impedance mismatch was caused by relational databases. It emerges from the need to map objects used in memory in our application model to tables in a relational database. This page https://en.wikipedia.org/wiki/Object-relational_impedance_mismatch is a good reference for this issue.

Saving an application object into these database model should be a direct operation. Conceptually no mapping would be necessary when an object database type was used. Albeit, these gains, object databases were not able to substitute relational databases.

In the 2000’s

As internet availability grows 2 things started to have a big impact:

• Generation of enormous amount of data that had to be stored and processed;
• Accesses from anywhere in the globe became more and more frequent. Therefore latency was an issue to consider. And even speed of light limitation can contribute significantly to this. Data can now be stored far from where we are. For example, a distance of 10.000 km will add about 100 ms on each round trip. Therefore, data needs to spread around the globe to provide a quick access.

In this environment, relational databases do not always solve customer needs, either because of flexibility, price or performance issues:

• Relational databases enforces a ridged set of rules in the relational model. These will impact the flexibility of the application development;
• Price because of the need for more powerful machines and also the software licenses that go with this;
• Performance, because relational databases do not naturally meet the increasing workload. Huge CPU, memory, storage and throughput is needed to be perform in this environment. And for relational model, vertical scaling can only get you so far.

Scaling Vertically versus Horizontally

Relational databases usual scaling method is a vertical one. As a result whenever the workload incresses we will use a bigger machine with more hardware resources.

On the other hand, the big internet companies adopted an horizontal scaling paradigm. This is a cluster type environment with lots and lots of machines. In other words, this means that more machines will be used to handle bigger workloads.

Relational databases do not thrive very well in this paradigm. They hardly take any benefit of using more machines. Hence, NoSQL like databases arises to take advantage of the horizontal scale paradigm. Also companies like Google and Amazon started researching in this area. As a result, Google created BigTable and Amazon DynamoDB.

Relational databases dominated the market, why are NoSQL databases being used now?

I think this is important. Dominance and usage factors are usually a combination of several aspects. What has been changing:

• Developers hide databases behind integration layers. This makes it simpler to use. And in addition, easier to replace one database or persistence method per other;
• Cloud growth in some cases means we can try different approaches with less effort regarding infrastructure;
• Handling big quantities of data introduced new needs. NoSQL provides a good development flexibility. And in addition can also take advantage of cluster solutions.

Finally, what is NoSQL?

Common NoSQL databases characteristics

NoSQL doesn’t have a clear direct meaning. The term alludes to something like “Not Only SQL”. A more exact explanation should be “Non Relational database”. This would reinforce the new model paradigm and not the SQL language itself. The fact that some NoSQL databases actually supports some form of SQL language adds even more to the confusion.

But, the name is catchy and so common that will probably stay. The point is: defining NoSQL is hard. Therefore, we will do the next best thing. Introduce the dominant traits of these database systems:

• Non relational;
• Usually (not all) cluster-friendly;
• Most of them are open source;
• No-schema / schema-less;
• Have a big internet drive (cluster and bigdata friendly).

By these common traits, almost any non relational database can be a NoSQL database! Also, there are probably published studies in all these areas dating back 30 or 40 year ago. Out of curiosity, the name NoSQL itself is referred to being born by the end of 2009. The term emerged as a twitter hashtag for a meetup were people talked about these subjects. Even though some of these traits were not new at the time.

Next steps

NoSQL has a lot to explore. It will provide a more direct paradigm that will make simpler specific needs like this ones to query Json objects inside Sql Server and performance considerations.

My plan for the next article(s) will be to:

• Present some of the NoSQL data models;
• Talk about the no schema or the schema less feature;
• Discuss the aggregate concept. And how this relates to the CAP theorem.

Further reads

There is a lot of information online. For me, I like Martin Fowler approach. Some of these information’s are from his content. You can check his website here: https://martinfowler.com/nosql.html. He also has a book: https://martinfowler.com/books/nosql.html.

The post NoSQL First Act – a historical introduction appeared first on Blog IT.

I will introduce a small usage sample for the JSON_VALUE function in combination with indexes to improve information retrieval from a table containing one JSON object. For our testes, we have a UserDetailTest table that has more than 500k rows with 2 columns: an UserId and a nvarchar(max) to hold a small JSON details string like the following:

SELECT *, LEN(DetailsJSON) AS [Len(DetailsJSON)] FROM UserDetailTest

I will activate time statistics and clean SQL Server cache between each query to have some consistency across the execution times using the following SQL statements:

CHECKPOINT
DBCC DROPCLEANBUFFERS
SET STATISTICS TIME ON

JSON_VALUE Function intro

The JSON_VALUE function extracts a value from a JSON string. This functions receives 2 arguments, the first being an expression for the JSON value and the second a path for the value we want to obtain. A simple sample with an inline JSON object would be the following:

SELECT JSON_VALUE('{"PostalCode":"376-3765","PhoneNumber":"351003765718"}', '$.PhoneNumber')

The return is a scalar value (nvarchar(4000)) for the PhoneNumber element.

Query scenarios

Starting with one of the really worst case scenario (yes, this can happen…):

More than 17 seconds really seems like a worst scenario! Next, we will use the JSON_VALUE function to get the PhoneNumber from the JSON string and use it in our where clause:

This still takes more than 3 seconds. A good improvement, but yet a high cost if we need to search information in this way.

Index creation

Let’s add a new virtual column to table that displays the result from the JSON_VALUE function – this will allow us to create an index and simplify the SELECT queries.

ALTER TABLE UserDetailTest 
	ADD vPhoneNumber AS 
	CAST(JSON_VALUE(DetailsJSON, '$.PhoneNumber') AS NVARCHAR(255));

The cast truncates the output from the JSON_VALUE to ensure that it does not exceed the maximum lenght for the index key. Now, if we search using the column, the result is still more or less the same 3 seconds has before:

Let’s create an index and perform the search again:

CREATE INDEX idx_vPhoneNumber ON [UserDetailTest] (vPhoneNumber);

Consequently the improvement is huge, 13 miliseconds is more interesting! But what’s the cost?

The cost

When we create an index, we’re basically trading space for time – more occupied space versus faster operations. Therefore let’s see what the increase is using sp_spaceused function before and after index creation:

Probably, an interesting comparison would be if the PhoneNumber column was an explicit column in that table containing the value. Let’s do that!

ALTER TABLE UserDetailTest  
	ADD PhoneNumberCopy NVARCHAR(255)

UPDATE UserDetailTest
	SET PhoneNumberCopy = vPhoneNumber

The sp_spaceused remained the same – SQL Server internal black magic regarding space allocation (for another time!). But performance wise, for this approach in my machine this query still took more than 2 seconds to complete:

Slightly better than the query with the JSON function but the advantage is that in this scenario SQL can better optimize the searches. The following runs of the same query without clearing the cache returns results much faster – each took around 220 miliseconds to complete:

Summary Results

Just to sum up all the results for the different types of searches comparing the first execution after cache clean up and the following runs. I did several iterations for each step just to make sure the results were consistent although the goal was just to have a baseline. Certanly, the use of this will depend on each case.

WHERE DetailsJson LIKE '%PhoneNumber":"351003765718"%'

WHERE JSON_VALUE(DetailsJson, '$.PhoneNumber') = '351003765718'

WHERE vPhoneNumber = '351003765718'

WHERE vPhoneNumber = '351003765718'

WHERE PhoneNumberCopy  = '351003765718'

The post Query performance for JSON objects inside SQL Server using JSON_VALUE function appeared first on Blog IT.