The first step here is to ensure that your consumer code is idempotent, but that’s something that you should always ensure – just imagine that you have to re-consume a couple of messages, in that case lacking idempotency may result in your system getting strange data and bugs.

Setting up Kafka on your machine

(Feel free to skip this section if you have everything set up.)

Kafka releases aren’t included in most package managers, except for Homebrew in Mac systems. They should be available, however, in the official download page. For Unix systems (Mac, Linux, WSL, *BSD), simply get one of the binary downloads and extract it wherever you prefer. Then, make sure to add the bin folder to your $PATH, so that you can use these scripts in any folder. In my case, I put the extracted archive in $HOME/.local/share/kafka, so I edit my .zshrc (or .bashrc in a plainer Linux/WSL system) and add:

export PATH=$HOME/.local/share/kafka/bin:$PATH

Now we can use the official kafka scripts in any directory. That is, unless we have some form of authentication. We can add additional configs to a file somewhere, and include that file in our commands. Let’s put a file called dev.cfg in $HOME/.config/kafka/ (the exact directory doesn’t matter, but .config is the appropriate directory for configs), and write the following on it:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="me" \
    password="noneofyourbusiness";

This is just an example, your broker’s security config may vary. After doing this, just add to any of the Kafka commands you use:

--command-config $HOME/.config/kafka/dev.cfg

Checking consumer offsets

Before migrating topics, it’s best if we make sure that the old topic consumer is totally up-to-date. We can use this command to check the consumption lag:

kafka-consumer-groups.sh \
	--bootstrap-server='<your broker endpoints, separated by commas>' \
	--describe \
	--group='<your consumer group>'

This should yield something like the output below:

TOPIC      PARTITION   NEWEST_OFFSET   OLDEST_OFFSET     CONSUMER_OFFSET     LEAD         LAG
your-topic 0           30000000        0                 30000000            30000000     0
your-topic 1           23900000        0                 23900000            23900000     0

CLIENT_HOST      CLIENT_ID         TOPIC          ASSIGNED_PARTITIONS
/128.0.0.0       your-consumer     your-topic     0
/128.0.0.1     	 your-consumer     your-topic     1

From this we can see that there is zero lag, since the newest offset is equal to the consumer offset on both partitions. We can also see that our consumers are still running (they’re the clients with ID “your-consumer”).

Performing the Kafka consumer migration

Before switching topics or brokers, we should first create a consumer group for the new configuration. This will allow us to change the offsets before beginning the message consumption proper. The easiest way to do this is to use kcat, a simple command-line tool that allows us to consume and produce Kafka messages. To create a consumer group, you must consume messages, so it’s a matter of setting up kcat (check the install guide on the kcat repo) and consuming a few messages from the new topic:

kcat \
	-C \
	-b '<your brokers>' \
	-t new-topic \
	-G new-consumer-group

# If you have authentication, just add: 

	-X security.protocol=SASL_SSL \
	-X sasl.mechanisms=SCRAM-SHA-512 \
	-X sasl.username="me" \
	-X sasl.password="noneofyourbusiness"

Just run this for a second, check that messages come out, and then run the describe command for the new group:

kafka-consumer-groups.sh \
	--bootstrap-server='new-broker' \
	--describe \
	--group='new-group'


TOPIC     PARTITION   NEWEST_OFFSET   OLDEST_OFFSET     CONSUMER_OFFSET     LEAD         LAG
new-topic 0           12387000        0                 12                  12           12386988
new-topic 1           12360000        0                 38                  38           12359962

CLIENT_HOST      CLIENT_ID         TOPIC          ASSIGNED_PARTITIONS

We have huge lag here, but that’s fine, as we’re going to reset the offsets anyway. Stop the workers consuming from the older topic by whatever means you can and note down the time at which they stopped. We will now reset the consumer group offset for this new group. Just a note: in this case, we don’t have any consumers connected to this group, but if we did we’d also have to turn them off. You cannot reset a consumer group offset if there are clients connected to it.

To reset the offsets, just run the following command (here we’ll assume that we stopped the previous consumers at 10:00 of March 15th, 2023, and subtract ten minutes just to be safe):

kafka-consumer-groups.sh \
	--bootstrap-server='new-brokers' \
	--group='new-group' \
	--topic='new-topic' \
	--reset-offsets \
	--to-datetime 2023-03-15T09:50:00.000 \
	--dry-run

Running this dry run will not make any changes, It will only show you the new offsets for the consumer group, e.g.:

TOPIC     PARTITION   NEWEST_OFFSET   OLDEST_OFFSET     CONSUMER_OFFSET     LEAD         LAG
new-topic 0           12387000        0                 12386998            12386988     2
new-topic 1           12360000        0                 12369951            12369951     49

CLIENT_HOST      CLIENT_ID         TOPIC          ASSIGNED_PARTITIONS

If nothing looks strange to you, go ahead, substitute --dry-run with --execute and run the command. Afterwards, deploy the new consumer configuration, turn on the worker, and let it run.

The post Performing broker migrations on your Kafka consumers appeared first on Blog IT.

Introduction

In AWS the main ways that we have to send an email are:

• Using Amazon Simple Email Service (SES)
• Using Amazon Simple Notification Service (SNS)

For this example we are going to use Amazon Simple Notification Service.

You can check the main differences between SES and SNS here

The components

Now i’m going to explain the components that we are going to use.

Amazon Simple Queue Service (SQS)

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

More Info: SQS

Amazon Simple Notification Service (SNS)

Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Amazon SNS provides topics for high-throughput, push-based, many-to-many messaging. Using Amazon SNS topics, your publisher systems can fan out messages to a large number of subscriber endpoints for parallel processing, including Amazon SQS queues, AWS Lambda functions, and HTTP/S webhooks. Additionally, SNS can be used to fan out notifications to end users using mobile push, SMS, and email.

More Info: SNS

AWS Lambda

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running.

With Lambda, you can run code for virtually any type of application or backend service – all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

More Info: Lambda

Objective

We are going to build a system using a SQS queue where we are going to send the body of the email. The lambda that long pools the queue will be triggered when the message is sent to the queue and will publish the message to the SNS topic so we can send an email with the message body.

More Info: Long Pooling

Lets Go

Create a SQS Queue where we are going to send the email body message. This queue will be a Standard Queue. This example will not have a deadletter queue configurated, but is always good to have one.

We need to create a SNS Topic. An Amazon SNS topic is a logical access point which acts as a communication channel.

Create a subscription to that topic. The subscription resource subscribes an endpoint to an Amazon Simple Notification Service (Amazon SNS) topic. For a subscription to be created, the owner of the endpoint must confirm the subscription. The protocol of the subscription will be Email. The endpoint will be the email address that will receive the email. For this example i used a temporary mail so that i didn’t fill my personal inbox with emails.

More Info: Temp-Mail

After you configure the subscription you will receive an email with a link to confirm the subscription.

When you click on the link you will be redirected to the subscription confirmation page

Create the lambda that will use long pooling to check if there are new messages in the queue and publish the messages to the SNS Topic.

In this example i used Python to develop the lambda and called it “EmailLambda”. Note: you need to set the role of the lambda so that the lambda has permissions to long pool the sqs queue and publish a message to the SNS topic.

More Info: SNS IAM Roles SQS IAM Roles

Here is the lambda code that i used.

#!/usr/bin/python3
import json
import boto3
import os

def lambda_handler(event, context):
    batch_processes=[]
    for record in event['Records']:
        send_request(record["body"])
		

def send_request(body):
    # Create an SNS client
    sns = boto3.client('sns')

    # Publish a simple message to the specified SNS topic
    response = sns.publish(
        TopicArn=os.environ['email_topic'],    
        Message=body,    
    )

    # Print out the response
    print(response)
 

Set the lambda enviroment variable email_topic to the ARN of the SNS Topic we created earlier.

Add the lambda trigger and set it with the sqs queue that will receive the body of the email.

Lets Test it !!!

Now the only thing that we need to do is send a new message to the SQS queue with the email body that we want.

Now we will receive an email from no-reply@sns.amazonaws.com with the body of the queue message.

And there you have it. An email notification in AWS using SQS, SNS and Lambda.

Thanks for reading this post and have an…

The post Send Email In AWS Using SQS, SNS and Lambda Function appeared first on Blog IT.