The pipelines bring to us the opportunity to create automatic tasks to execute development operations, like deploys, migrations, tests, etc. Focused in the deploy routine, some applications need other operations to ensure the stability between the application and other resources, like the database.

During the development process, the developers usually customize and improve the application database in the development environment scope, and it can be made easily by using frameworks that contains routines to create source files that have the responsability to keep the history of changes and are used to update the database for determinated version. These files are known as migrations. 

In this post, we will see how to apply migrations for the databases that are located in different environments (staging, production, etc.), using the Code-First entity concept, and the Entity Framework Core.

Migrations

To generate database migrations, we can execute the “Add-Migraton <name>” command from the Entity Framework. This command will generate some source files that will contain the code to apply the migration to the database. To execute these migration files, you just need to execute the “Update-Database” command. But, Microsoft recommends to create a SQL file individually, and suggest to execute this file direct to the database, without need to execute an intermediary software (C# + EF) to access the database. To do this, we will have to follow the steps below:

• 1. Create a Pipeline in Azure Devops;
• 2. Install dotnet-ef-tool;
• 3. Generate SQL script;
• 4. Execute SQL script to the database;
• 5. Script File as a Pipeline Artifact.

NOTE: We don’t need to worry about which migration need to be applyed to the database. The dotnet-ef-tool command will generate a script cantaining the needed logic. Check the third step for more details.

1. Create a Pipeline in Azure Devops

We will create a common deployment pipeline in Azure Devops for ASP.NET Core backend applications. Check the end of the post to see the full result of the pipeline.

2. Install dotnet-ef-tool

To execute tool commands of Entity Framework, we have to install a package named dotnet-ef-tool. This needs to be available into the pipeline’s context to be used for the SQL script generation. To install this tool by a pipeline task, we can use the task below:

- task: DotNetCoreCLI@2
  displayName: 'Install EF tool'
  inputs:
    command: 'custom'
    custom: 'tool'
    arguments: 'install --global dotnet-ef'

3. Generate SQL script

To generate the SQL script containing the logic to apply migrations, we will use the dotnet-ef-tool. By default, the command does not generate the logic to check which migration should be applied, so we have to enable this behavior using the parameter: –idempotent. By using this behavior, we don’t need to worry about the migrations that will be applyed to the database since the generated script already has it. Below, you can see the task that could be used to generate the script file:

- task: DotNetCoreCLI@2
  displayName: 'Generate SQL Script'
  inputs:
    command: 'custom'
    custom: 'ef'
    arguments: 'migrations script --idempotent --project $(Build.SourcesDirectory)\Data.csproj --output $(System.ArtifactsDirectory)/script.sql'

NOTE: We don’t need to stablish any database connection for the script generation, since we are following the Code First concept for entity creation, so, the command will just look at the entities source code of the project to generate the SQL file. If we don’t use the –idempotent argument, the script will generate the SQL without the conditional logic to determinate which migration should be applied, causing an error if the database already exists.

4. Execute SQL script to the database

Now, with the SQL script available, we just need to execute this file to the database. In this post, we will use a common Azure Database as an example, so for this case, we can use the following task to accomplish this requirement:

- task: SqlAzureDacpacDeployment@1
  displayName: 'Update Database'
  inputs:
    azureSubscription: '<Service Connection Identifier>'
    AuthenticationType: 'connectionString' # You can use other method to authenticate to the database if you want
    ConnectionString: '<Database Connection String>'
    deployType: 'SqlTask'
    SqlFile: '$(System.ArtifactsDirectory)/script.sql'

5. Script File as a Pipeline Artifact

Well done. The last task will only be used to make the file available to be accessed by the pipeline runner user. To do this, you can use the task below to add the generated script.sql file to the file bundle (artifacts). To check the artifacts, just access the artifacts option from the pipeline details after the job’s execution.

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts'
  inputs:
    PathtoPublish: '$(System.ArtifactsDirectory)/script.sql'
    ArtifactName: '$(artifactName)'
    publishLocation: 'Container'

Conclusion

In this post, we could learn an easy way to apply the database migrations to the remote environment by using automation (pipelines) to accomplish it. This method is simple, and does not have any database validation before script execution. So, if you have an active and sensitive environment, be sure that the needed validations are considered before the database update, to ensure that the database will not be update wrongly.

Pipeline Result

trigger:
- none

pool:
  vmImage: windows-latest

variables:
  solution: '**/*.sln'
  buildPlatform: 'AnyCPU'
  buildConfiguration: 'Release'
  artifactName: 'artifacts'
  environment: 'Development'

- task: NuGetToolInstaller@1
  displayName: 'NuGet Installation'

- task: NuGetCommand@2
  displayName: 'NuGet Restore'
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  displayName: 'Build Project'
  inputs:
    solution: 'backend/WebAPI'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: PublishSymbols@2
  displayName: 'Publish Symbols'
  inputs:
    SearchPattern: '**/bin/**/*.pdb'
    SymbolServerType: 'FileShare'
    CompressSymbols: false

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts'
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'artifacts'
    publishLocation: 'Container'

- task: DownloadBuildArtifacts@0
  displayName: 'Build Artifacts'
  inputs:
    buildType: 'current'
    downloadType: 'single'
    artifactName: '$(artifactName)'
    downloadPath: '$(System.ArtifactsDirectory)'

- task: DotNetCoreCLI@2
  displayName: 'Install EF tool'
  inputs:
    command: 'custom'
    custom: 'tool'
    arguments: 'install --global dotnet-ef'

- task: DotNetCoreCLI@2
  displayName: 'Generate SQL Script'
  inputs:
    command: 'custom'
    custom: 'ef'
    arguments: 'migrations script --idempotent --project $(Build.SourcesDirectory)\Data.csproj --output $(System.ArtifactsDirectory)/script.sql'

- task: SqlAzureDacpacDeployment@1
  displayName: 'Update Database'
  inputs:
    azureSubscription: '<Service Connection Identifier>'
    AuthenticationType: 'connectionString'
    ConnectionString: '<Connection String>'
    deployType: 'SqlTask'
    SqlFile: '$(System.ArtifactsDirectory)/script.sql'

- task: AzureRmWebAppDeployment@4
  displayName: 'Deploy Application'
  inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: '<Service Connection Identifier>'
    appType: 'webApp'
    WebAppName: 'armazemlegalapi'
    deployToSlotOrASE: true
    ResourceGroupName: '<Resource Group Identifier>'
    SlotName: 'production'
    packageForLinux: '$(System.ArtifactsDirectory)/Project.zip'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifacts'
  inputs:
    PathtoPublish: '$(System.ArtifactsDirectory)/script.sql'
    ArtifactName: '$(artifactName)'
    publishLocation: 'Container'

References

• Microsoft Managing Schemas Article

The post Entity Framework – Update Database using Azure Pipelines appeared first on Blog IT.

Introduction

At the moment, there are quite a few tools and products that have adopted CloudEvents or AsyncAPI. Knative Eventing is a tool that helps developers in a serverless context, Azure Event Grid natively supports CloudEvents. More recently, Jenkins added integration with CloudEvents with a new plugin. It allows users to configure Jenkins as a source and a sink for CloudEvents. There is also interesting integrations between Kubernetes and Azure Event Grid that are compliant with the CloudEvents v1.0 spec. Checkout the GitHub repository or this blog post and learn more about it.

Postman has joined forces with AsyncAPI along with organizations such as Salesforce, Slack and Solace. Postman in particular is publishing public collections related to AsyncAPI. For example a list of companies adopting AsyncAPI, with links to those resources (GitHub repositories, websites, etc).

I hope these projects have got you excited to learn more about these specs. Let’s dive into some of their details!

CloudEvents

The CloudEvents specification is under the CNCF Serverless working group since 2018. The spec’s purpose is describing event data in a common way. This is useful in many scenarios, for example, routing events to the appropriate subscribers depending on the type of the event. Since applications can use a lot of different transports to send and receive events, the CloudEvents spec is protocol-agnostic so it defines protocol bindings in order for the metadata to be correctly mapped for HTTP, AMQP, Kafka, etc.

There are many use cases in using the CloudEvents spec, but perhaps the main one would be interoperability. Imagine applications across clouds, being able to communicate in an event-driven architecture. Where there are producers of all sources, and consumers using all kinds of protocols (e.g. HTTP, AMQP, WebSockets). We can have middleware that connects these applications, adds E2E tracing and more with the use of CloudEvents. Of course we could connect the same applications without a common format, but it requires mapping between event formats (cloud providers use different schemas). Middleware would also need to parse the event data to get specific information.

Another use case is SaaS (Software-as-a-service) that publishes events that clients are interested in to integrate with their own systems. For example, hooking into the checkout flow in a Shopify storefront to add extra checks. By leveraging CloudEvents these events can be consistent, opening the door for numerous integrations between 3rd party software.

Extensions

There are a few extensions worth mentioning, one of them is for distributed tracing. However, it seems there is some discussion around removing this extension from the spec (check this PR on GitHub). There are open issues on some SDKs to support it, and others have already made changes to remove it. The future isn’t clear, but I’d argue it’s interesting to follow this closely for any updates, since tracing events is very important in an event-driven architecture.

The Partioning extension is another interesting extension, it defines a field to be handled by message brokers that can separate load via a partition key. This is used for example in the Kafka protocol binding that requires implementations to map the partitionKey attribute to the key of the Kafka message. In Kafka the concept of a partition is well known, so this maps out really well.

Relation with Serverless computing

Serverless computing has increased in popularity and use in the industry, especially for it’s cost model. But many FaaS (Functions-as-a-service) providers have their own function interface. Which means developers can’t write a function in JavaScript, and deploy them in two cloud providers without making changes. This specification improves portability between FaaS platforms, so that developers receive an event in the same format and can reuse libraries for handling the event.

AsyncAPI

I’ll start with a description of what AsyncAPI is: a specification that describes and documents event-driven APIs in a machine-readable format. It’s protocol-agnostic like CloudEvents, so it can be used for APIs that work over many protocols, including MQTT, WebSockets, and Kafka.

The following is AsyncAPI’s vision stated on their website:

I find this vision to be very interesting, mainly because of the part: Any kind of APIs. At first, you might wonder if this means the AsyncAPI spec will define rules and more concepts for other types of APIs like GraphQL or OpenAPI. But this is not at all the case, the goal is to integrate with existing tools and specs! This is valuable for developers because usually enterprise architectures consist of a mix of technologies, each for it’s appropriate use case. Developers nowadays don’t just interact with RESTful APIs in a request/response model. There are different demands and considerations for the ever increasing devices users can use, the software we build needs to match these demands and still maintain manageable. Where we can evolve and create new applications that leverage the numerous APIs that exist internally or from a 3rd party.

Concepts

The spec version 2.1.0 defines a few concepts apart from the common Producer, Consumer and Message. A Channel can be seen as a topic/exchange or queue, an application can send messages to a channel and consumers can subscribe to it to receive them. The Operation object indicates if it’s a publish or a subscribe operation and how an application can send or receive messages. A Binding (or “protocol binding”) is a mechanism to define protocol-specific information or query parameters for the channel bindings. For example, for the AMQP protocol we can specify the channel is an exclusive queue like this:

"bindings": {
    "amqp": {
      "is": "queue",
      "queue": {
        "exclusive": true
      }
    }
}

Each protocol has it’s own JSON schema and we can have bindings for Messages, Servers, Channels, Operations and others.

Components

You can define components to re-use in multiple AsyncAPI documents and we can reference other AsncAPI documents. Let’s say you have two publishers who publish the same message, but with different values in one of the message properties. We’d have two AsyncAPI documents specifying the publishers, referencing a 3rd document specifying the common message with it’s properties. The $ref field is a string that can be the path to the other file, or a URL for an external file where the schema we want is defined. This reference object uses the same rules and format of JSON Reference, which opens the door for many possibilites (check the docs to know more).

When we start to have a lot of apps that depend on each other’s schemas, we can take a look at some solutions to scale out AsyncAPI documents. Perhaps we use Confluent’s Schema Registry for our JSON schemas and setup a catalog of events in our organization. This empowers new developers seeking for ways to integrate with existing systems and event producers. We can also just store these components in a GitHub repository, and reference them in our AsyncAPI documents.

Tooling

There is already quite a few tools and the tooling ecosystem is increasing! I’ve recently seen a repository that enables the creation of Postman collections from an AsyncAPI spec. I’ve also seen architecture documents being generated from multiple AsyncAPI specs too, having a tool that can understand relations between applications and then output a diagram is pretty cool.

Generators for AsyncAPI

One piece of tooling that is often used are generators that produce documentation and code. For example gRPC tools have this capability using the protocol buffer compiler. AsyncAPI generators can take an AsyncAPI document and generate client/server code or documentation in HTML and markdown. Currently, it depends on the template we use to generate server-side code, for example the Node.js WebSocket template generates both server and client code. This can be improved and extended overtime, especially because of the way the generator is designed, enabling extensibility so we can have templates for many other languages that support more protocols, etc. For example, there is only a NATS generator for .NET Core… but perhaps in the future there could be more protocols supported for .NET Core and examples built for Azure 😃.

Conclusion

There is a lot of exciting stuff happening in the event-driven architectures world 😄. We have only touched the surface in this post. In the CloudEvents space there are new specs being designed and worked on: Discovery; Subscription and Schema Registry APIs. Since AsyncAPI defines a document that you can use to describe your API, it’d be interesting to see how these correlate to each other, and using them together.

I encourage you to join these communities and contribute to their open source projects, CloudEvents and AsyncAPI, both specs are very community-driven. Collaboration between everyone is the way forward, with Hacktoberfest and AsyncAPI’s Hackathon coming up, searching good first issues is a great way to start and to contribute 👍!

Let us know in the comments if you’re using these specs and what are your thoughts on them. The next blog post will be about a practical example for .NET Core, Azure and AMQP messaging using CloudEvents and AsyncAPI, so stay tuned!

Additional Links

Here are some links to talks, docs and blog posts that you may find useful if you’re interested to know more about CloudEvents and AsyncAPI:

• The Serverless and Event-Driven Future – Austen Collins, Serverless (Intermediate Skill Level)
• OpenFaaS supports CloudEvents
• The Future of API Specifications talk by Fran Méndez
• AsyncAPI 2.0: Enabling the Event-Driven World

The post Getting Started with CloudEvents and AsyncAPI appeared first on Blog IT.

In this article we will talk about Event-Driven Architectures. We choose to use the Azure Cloud Infrastructure.
Service Bus provides reliable, secure asynchronous messaging at scale. This article is written by the engineering team at CreateIT and it is intended to show you a case study in one of our projects for a client.

We’ll take a deeper dive into the Service Bus technology, architecture, and design choices. The post will cover both conceptual material as well as implementation details. Most importantly, we will discuss design and implementation of some of the features that provide secure and reliable messaging at scale, while minimizing operational cost.

Service Bus Entities

When we are working with Azure Service Bus, we can choose two Entities: Topics or Queues. You can have multiple Topics or Queues per Service Bus Namespace, but firstly you need to differ one from another. If you want a FIFO queue and only have one Consumer, then Queues are the way to go. If you need multiple Consumers, then the Topic is the better option. In this specific case we will create a Subscription per Consumer (Topics are only available from the Standard Pricing Tier).

Event-driven architectures

Benefits with event-driven architectures

What are the benefits of using a queue in the middle of these systems?

• We can decide to load balance the input from Customer Services. Let’s say there are a lot of updates being made to a customer, meaning a lot of events being published. We scale the number of consumers and user the competing pattern
• We can throttle the input. If on black Friday there are a ton of events and in case our Audit Log system is down. We simply store these events on the queue and consume them when the service is back online again. Of course we’d need to implement some logic for this behavior, but adding this “middleware” buys us options.

In our use case, we wanted to move to an implementation where the Web API doesn’t get affected by any changes on these external systems. But in order to change the implementation, we must first figure out what are the challenges associated with this change.

Challenges with event-driven architectures

Message/Event order

Azure Service Bus has a feature called sessions. A session provides a context to send and retrieve messages that will preserve ordered delivery. However, in our use case we chose not to use it.

Message Lock Duration

When we are using Queues, every message has a lock Duration. During this time the consumer needs to process it. But if this consumer needs to contact multiple external systems this time may rise and our messages could stop in the Dead-Letter. So the best practice is to change it accordingly to your needs. We recommend you to set this time extremely high in the beginning, and then do some tests to calculate the average.

After that add 30% more of its value, in case of some lengthy  requests (in this case if we have outliers, they might stop in the Queue). If you are using a Topic, you will have a Lock Duration per Subscription. So make sure to adjust this time accordingly to its functions.

Implementation

In the Figure 1 you can see the initial architecture for the Customer Management system. It was responsible to make the requests to the other systems.

With the new implementation, a message broker was introduced and we used the event-carried state transfer pattern, meaning our events had all the information the consumer needed in order to do their job. We took in consideration the event notifications pattern, where the consumer would have to make a request to the API that originated the event, in order to get more information. But this brings new problems to the table. What if when the consumer code runs, the information for that customer ID changed? What if the event was CUSTOMER_CREATED but in the meanwhile the customer was deleted?

Retries with Polly

In a distributed system, many things can go wrong. The network can fail or have additional latency, systems may be temporarily down, etc. We use the Azure.ServiceBus.Messaging NuGet package so we are able to check if the exception is a transient fault or not (more information on these docs), then use Polly to setup retry logic and fallbacks. There are other options to implement retry policies, for example we took in consideration the Retry guidance for Azure Services documentation from Microsoft. Since we use the latest Azure SDK,  the appropriate class would be ServiceBusRetryPolicy.
We configured Polly to retry to publish a message three times (this configuration is on appsettings.json), with exponential times between each attempt.
If after the third retry we can’t publish the message we need to save it, because it has crucial information. So to solve this issue we created a Fallback Gateway, to write these messages to a Container inside an Azure Storage Account.

Filters for message routing

This section only applies for Topics Entities on the Azure Service Bus.
We can add Filters on our Subscriptions to help us with routing each message to its specific Consumer. We considered two filter types:

• SQL Filter
• Correlation Filter

Using the Correlation Filter you can configure Custom Properties and create Filters for your needs. You just need to make sure the producer of the messages, includes the header, that you are currently using to filter, on the message.

With SQL Filters you can create conditional expression to evaluate the current message. Just make sure that all the system properties are prefixed with sys. in the expression. Either way, both filters work just choose one that suits you the most!

Dead-Letter Queue

In case the consumer application can’t process the message after the Max Delivery Count attempts, instead of returning it to the queue it will be sent automatically to the Dead-Letter queue. If you are using the Topic, each subscriber has its own Dead-Letter queue. You can also, configure different Max Delivery Counts Values for each Subscriber.

All messages that are published to the Service Bus have a TTL (Time-To-Live). After this time ends the message will be transferred automatically to the Dead-Letter. So make sure you adjust this time accordingly to your needs.

With this we are able to save messages that weren’t processed by the consumer application, but we should always strive to have an empty dead-letter queue.

Conclusion

Our first steps into an Event-Driven Architecture was a truly success!
We were able to expand our previous solution to be compatible with multiple external systems and instead of having the API sending a HTTPS request for each one we had this Application sending one message to a Topic in the Service Bus.
One of our goals was to have load balance in the Publisher Application. We went from a 1->3 dependency to a 1-> 1, as you can see in the Figure 2.

Which is great and keeps the system scalable and future proof. Our solution became more decoupled in order to keep the Application agnostic to these changes.
If you have a similar situation with an Event-Driven Architecture then we totally recommend you to check more about this technology and it’s features.

We would like to share a link to Microsoft Azure Service Bus GitHub. Most of the implementations of either the publisher or the subscriber were inspired by this documentation, so make sure you check it out!
If you have any questions, please write them down below.

Additional Links

Service Bus Exceptions
Service Bus Basic Steps
Tutorial with DotNet

The post Case Study: Azure Service Bus and Event-Driven Architectures appeared first on Blog IT.

Introduction

In AWS the main ways that we have to send an email are:

• Using Amazon Simple Email Service (SES)
• Using Amazon Simple Notification Service (SNS)

For this example we are going to use Amazon Simple Notification Service.

You can check the main differences between SES and SNS here

The components

Now i’m going to explain the components that we are going to use.

Amazon Simple Queue Service (SQS)

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

More Info: SQS

Amazon Simple Notification Service (SNS)

Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Amazon SNS provides topics for high-throughput, push-based, many-to-many messaging. Using Amazon SNS topics, your publisher systems can fan out messages to a large number of subscriber endpoints for parallel processing, including Amazon SQS queues, AWS Lambda functions, and HTTP/S webhooks. Additionally, SNS can be used to fan out notifications to end users using mobile push, SMS, and email.

More Info: SNS

AWS Lambda

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running.

With Lambda, you can run code for virtually any type of application or backend service – all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

More Info: Lambda

Objective

We are going to build a system using a SQS queue where we are going to send the body of the email. The lambda that long pools the queue will be triggered when the message is sent to the queue and will publish the message to the SNS topic so we can send an email with the message body.

More Info: Long Pooling

Lets Go

Create a SQS Queue where we are going to send the email body message. This queue will be a Standard Queue. This example will not have a deadletter queue configurated, but is always good to have one.

We need to create a SNS Topic. An Amazon SNS topic is a logical access point which acts as a communication channel.

Create a subscription to that topic. The subscription resource subscribes an endpoint to an Amazon Simple Notification Service (Amazon SNS) topic. For a subscription to be created, the owner of the endpoint must confirm the subscription. The protocol of the subscription will be Email. The endpoint will be the email address that will receive the email. For this example i used a temporary mail so that i didn’t fill my personal inbox with emails.

More Info: Temp-Mail

After you configure the subscription you will receive an email with a link to confirm the subscription.

When you click on the link you will be redirected to the subscription confirmation page

Create the lambda that will use long pooling to check if there are new messages in the queue and publish the messages to the SNS Topic.

In this example i used Python to develop the lambda and called it “EmailLambda”. Note: you need to set the role of the lambda so that the lambda has permissions to long pool the sqs queue and publish a message to the SNS topic.

More Info: SNS IAM Roles SQS IAM Roles

Here is the lambda code that i used.

#!/usr/bin/python3
import json
import boto3
import os

def lambda_handler(event, context):
    batch_processes=[]
    for record in event['Records']:
        send_request(record["body"])
		

def send_request(body):
    # Create an SNS client
    sns = boto3.client('sns')

    # Publish a simple message to the specified SNS topic
    response = sns.publish(
        TopicArn=os.environ['email_topic'],    
        Message=body,    
    )

    # Print out the response
    print(response)
 

Set the lambda enviroment variable email_topic to the ARN of the SNS Topic we created earlier.

Add the lambda trigger and set it with the sqs queue that will receive the body of the email.

Lets Test it !!!

Now the only thing that we need to do is send a new message to the SQS queue with the email body that we want.

Now we will receive an email from no-reply@sns.amazonaws.com with the body of the queue message.

And there you have it. An email notification in AWS using SQS, SNS and Lambda.

Thanks for reading this post and have an…

The post Send Email In AWS Using SQS, SNS and Lambda Function appeared first on Blog IT.

NoSQL is like a new definition of something that is a database but is different than the usual relational model. Likewise, making a parallel by going back 30 years ago. Back then probably no one knew what a relational database was also. Let’s use this and start with a small historical and motivational point of view for NoSQL. PS: I was not a developer back then.

This article will be a first part of a series of articles. The goal will be to unite some knowledge about this topic.

Starting in the 1980’s

Relational databases emerged bringing ACID properties – Atomicity, Consistency, Isolation and Persistency. Properties taken for granted nowadays. Also, they also brought the SQL language. SQL is common enough across different systems for one to use. Although there are different flavors, can almost be considered a standard.

Relation databases also allowed for a simple and very common integration mechanism between 2 or more systems. Data can be easily shared by reading or writing data in a table on a shared relational database. This is today still a very common integration pattern.

In the 1990’s

Object database’s started to appear with more strength. They had been around for some time. Most important, they implement a different database paradigm model. This new paradigm tried to solve the impedance mismatch problem. The impedance mismatch was caused by relational databases. It emerges from the need to map objects used in memory in our application model to tables in a relational database. This page https://en.wikipedia.org/wiki/Object-relational_impedance_mismatch is a good reference for this issue.

Saving an application object into these database model should be a direct operation. Conceptually no mapping would be necessary when an object database type was used. Albeit, these gains, object databases were not able to substitute relational databases.

In the 2000’s

As internet availability grows 2 things started to have a big impact:

• Generation of enormous amount of data that had to be stored and processed;
• Accesses from anywhere in the globe became more and more frequent. Therefore latency was an issue to consider. And even speed of light limitation can contribute significantly to this. Data can now be stored far from where we are. For example, a distance of 10.000 km will add about 100 ms on each round trip. Therefore, data needs to spread around the globe to provide a quick access.

In this environment, relational databases do not always solve customer needs, either because of flexibility, price or performance issues:

• Relational databases enforces a ridged set of rules in the relational model. These will impact the flexibility of the application development;
• Price because of the need for more powerful machines and also the software licenses that go with this;
• Performance, because relational databases do not naturally meet the increasing workload. Huge CPU, memory, storage and throughput is needed to be perform in this environment. And for relational model, vertical scaling can only get you so far.

Scaling Vertically versus Horizontally

Relational databases usual scaling method is a vertical one. As a result whenever the workload incresses we will use a bigger machine with more hardware resources.

On the other hand, the big internet companies adopted an horizontal scaling paradigm. This is a cluster type environment with lots and lots of machines. In other words, this means that more machines will be used to handle bigger workloads.

Relational databases do not thrive very well in this paradigm. They hardly take any benefit of using more machines. Hence, NoSQL like databases arises to take advantage of the horizontal scale paradigm. Also companies like Google and Amazon started researching in this area. As a result, Google created BigTable and Amazon DynamoDB.

Relational databases dominated the market, why are NoSQL databases being used now?

I think this is important. Dominance and usage factors are usually a combination of several aspects. What has been changing:

• Developers hide databases behind integration layers. This makes it simpler to use. And in addition, easier to replace one database or persistence method per other;
• Cloud growth in some cases means we can try different approaches with less effort regarding infrastructure;
• Handling big quantities of data introduced new needs. NoSQL provides a good development flexibility. And in addition can also take advantage of cluster solutions.

Finally, what is NoSQL?

Common NoSQL databases characteristics

NoSQL doesn’t have a clear direct meaning. The term alludes to something like “Not Only SQL”. A more exact explanation should be “Non Relational database”. This would reinforce the new model paradigm and not the SQL language itself. The fact that some NoSQL databases actually supports some form of SQL language adds even more to the confusion.

But, the name is catchy and so common that will probably stay. The point is: defining NoSQL is hard. Therefore, we will do the next best thing. Introduce the dominant traits of these database systems:

• Non relational;
• Usually (not all) cluster-friendly;
• Most of them are open source;
• No-schema / schema-less;
• Have a big internet drive (cluster and bigdata friendly).

By these common traits, almost any non relational database can be a NoSQL database! Also, there are probably published studies in all these areas dating back 30 or 40 year ago. Out of curiosity, the name NoSQL itself is referred to being born by the end of 2009. The term emerged as a twitter hashtag for a meetup were people talked about these subjects. Even though some of these traits were not new at the time.

Next steps

NoSQL has a lot to explore. It will provide a more direct paradigm that will make simpler specific needs like this ones to query Json objects inside Sql Server and performance considerations.

My plan for the next article(s) will be to:

• Present some of the NoSQL data models;
• Talk about the no schema or the schema less feature;
• Discuss the aggregate concept. And how this relates to the CAP theorem.

Further reads

There is a lot of information online. For me, I like Martin Fowler approach. Some of these information’s are from his content. You can check his website here: https://martinfowler.com/nosql.html. He also has a book: https://martinfowler.com/books/nosql.html.

The post NoSQL First Act – a historical introduction appeared first on Blog IT.

Today i’m going to talk about SharePoint Scenarios and what are the main difference between this three types of scenarios and what scenario should be the option for your environment…

So, the three types of architecture scenario that you can have on your SharePoint environment are the following:

What should you choose?! Well in fact there are so many question that you need answered before take an decision!?

It isn’t easy take a decision, let’s analyse what are the main benefits each architecture:

SharePoint On-Premises:

• Private Cloud
• Dedicated environment
• Internally hosted
• Internally managed
• Internal designed

• Migration Flow:

SharePoint Online:

• Public Cloud
• Partially or fully dedicated
• Externally hosted
• Externally or Internally managed
• Internally managed
• Minimal customization

Cloud benefits according to cloud users?

• Increase efficiency (55 %)
• Improved employee mobility (49 %)
• Increased ability to Innovate (32 %)
• Freed current IT staff for other projects (31 %)
• Reduce IT operation costs (25 %)
• Enabled us to offer new products/ services (24 %)

• Migration Flow:

SharePoint Hybrid Solution:

What is Hybrid SharePoint?

“Productivity services in SharePoint Online (Office 365) which are securely integrated with on-premises SharePoint Server 2016 to provide unified functionality and access to data.”

Why considering a Hybrid SharePoint Solution?

• Large existing investments (customized SP deployments w/lots of data and settings,custom solutions, LOB systems, etc…)
• Functionalities I can’t do in the Cloud that i can do on premises;
• Keep Sensitive Data on Premise
• Collaboration with External Partners
• Capacity Flexibility
• Intranet – Extranet
• Geo Location

Same Benefits of Hybrid SharePoint Solution?

SharePoint Business to Business Collaboration: Extranet for Partners with Office 365

“on-premises extranet site involves complex configuration to establish security measures and governance, including granting access inside the corporate firewall, and expensive initial and on-going cost….SharePoint Online, partners connect directly to a members-only site in Office 365, without access to the corporate on-premises environment or any other Office 365 site. Office 365 Extranet sites can be accessed anywhere…”

SharePoint hybrid sites and search

“A hybrid environment can help your company get started in the cloud, taking a first step to explore the cloud functionality at own your pace. It also enables enterprise users to be connected from almost anywhere to the resources and content they need… SharePoint hybrid features, you can consolidate search results between SharePoint Server and Office 365, consolidate user profiles in Office 365, and offload your users’ personal storage to the cloud…”

• Migration/Interactions Flow:

Thanks

Fábio Carvalho
SharePoint Consultant
|create|it|

The post SharePoint Scenarios – Online, On-Premises or Hybrid!? appeared first on Blog IT.

One issue that did come up is the definition of “Large”: this was a large project for Portugal’s standards, plus it was complex and took a long time until release. But for local Swedish standards, it wasn’t that “large” . Even so, the contents are general enough to be interesting, or so I was told. Also it was interesting to learn about some cultural differences between Portugal and Sweden – and those were much less than would be expected.

Great session, loved doing it. Here’s the slidedeck, for those interested.

The post ITARC15 Architecting a Large Software Project – Lessons Learned appeared first on Blog IT.

The slides include architecture aspects, technical aspects, as well as negotiation and functional hints. It’s not meant to be an absolute best-practices architecture guide, it’s only the result of this very specific project.

You can check/download the deck here.

Special thanks to NetPonto for the invitation, and to the audience for the participation and great feeedback in this LONG session :).

The post «Architecting a Large Software Project – Lessons Learned» @ Netponto 50th Meeting – Lisboa 22/Nov appeared first on Blog IT.

The session was mostly demo-based, and included the “traditional” WebMatrix/WordPress demo, but with the added value of demoing a content migration from another blog engine. After this I did a continuous integration, test and automated deployment demo. Didn’t have the time to show custom deployment scripts, but it was the star of the show, in my opinion. It’s easy to get it working and covers a large percentage of Webdev use cases. The following demo was simply showing how to associate a custom domain to a web site, and what is possible in terms of SSL (both kinds, SNI&IP).

The next demo was related to a request and the need to clarify that you can host wcf web services in an Azure WebSite, and for this demo I also showed , especially helpful for diagnostics and operations.

I ended the session talking about architecture, scaling, service tiers and pricing, and a Q&A. There were several questions, perhaps the most relevant being about Azure Web Sites vs Azure Web Roles. The stronger scaling possibilities, and running setup scripts, are strong arguments for Web Roles, the simplicity and more focused web offering, for Web Sites. This would be my choice, in general.

Another interesting event, with good session feedback.

The post Microsoft Developer TechRefresh (Lisboa 2013.09.24)–“Power your Website with Windows Azure” appeared first on Blog IT.

Also interesting and related to something I have in hands at the moment is Nuno Godinho’s post “Lessons Learned: Taking the best out of Windows Azure Virtual Machines”. Interesting feedback from using IaaS. SharePoint Server has been implemented everywhere in the last few years. Azure IaaS is especially interesting for these scenarios.

The post AzureConf2013 Today! appeared first on Blog IT.