The first step here is to ensure that your consumer code is idempotent, but that’s something that you should always ensure – just imagine that you have to re-consume a couple of messages, in that case lacking idempotency may result in your system getting strange data and bugs.

Setting up Kafka on your machine

(Feel free to skip this section if you have everything set up.)

Kafka releases aren’t included in most package managers, except for Homebrew in Mac systems. They should be available, however, in the official download page. For Unix systems (Mac, Linux, WSL, *BSD), simply get one of the binary downloads and extract it wherever you prefer. Then, make sure to add the bin folder to your $PATH, so that you can use these scripts in any folder. In my case, I put the extracted archive in $HOME/.local/share/kafka, so I edit my .zshrc (or .bashrc in a plainer Linux/WSL system) and add:

export PATH=$HOME/.local/share/kafka/bin:$PATH

Now we can use the official kafka scripts in any directory. That is, unless we have some form of authentication. We can add additional configs to a file somewhere, and include that file in our commands. Let’s put a file called dev.cfg in $HOME/.config/kafka/ (the exact directory doesn’t matter, but .config is the appropriate directory for configs), and write the following on it:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="me" \
    password="noneofyourbusiness";

This is just an example, your broker’s security config may vary. After doing this, just add to any of the Kafka commands you use:

--command-config $HOME/.config/kafka/dev.cfg

Checking consumer offsets

Before migrating topics, it’s best if we make sure that the old topic consumer is totally up-to-date. We can use this command to check the consumption lag:

kafka-consumer-groups.sh \
	--bootstrap-server='<your broker endpoints, separated by commas>' \
	--describe \
	--group='<your consumer group>'

This should yield something like the output below:

TOPIC      PARTITION   NEWEST_OFFSET   OLDEST_OFFSET     CONSUMER_OFFSET     LEAD         LAG
your-topic 0           30000000        0                 30000000            30000000     0
your-topic 1           23900000        0                 23900000            23900000     0

CLIENT_HOST      CLIENT_ID         TOPIC          ASSIGNED_PARTITIONS
/128.0.0.0       your-consumer     your-topic     0
/128.0.0.1     	 your-consumer     your-topic     1

From this we can see that there is zero lag, since the newest offset is equal to the consumer offset on both partitions. We can also see that our consumers are still running (they’re the clients with ID “your-consumer”).

Performing the Kafka consumer migration

Before switching topics or brokers, we should first create a consumer group for the new configuration. This will allow us to change the offsets before beginning the message consumption proper. The easiest way to do this is to use kcat, a simple command-line tool that allows us to consume and produce Kafka messages. To create a consumer group, you must consume messages, so it’s a matter of setting up kcat (check the install guide on the kcat repo) and consuming a few messages from the new topic:

kcat \
	-C \
	-b '<your brokers>' \
	-t new-topic \
	-G new-consumer-group

# If you have authentication, just add: 

	-X security.protocol=SASL_SSL \
	-X sasl.mechanisms=SCRAM-SHA-512 \
	-X sasl.username="me" \
	-X sasl.password="noneofyourbusiness"

Just run this for a second, check that messages come out, and then run the describe command for the new group:

kafka-consumer-groups.sh \
	--bootstrap-server='new-broker' \
	--describe \
	--group='new-group'


TOPIC     PARTITION   NEWEST_OFFSET   OLDEST_OFFSET     CONSUMER_OFFSET     LEAD         LAG
new-topic 0           12387000        0                 12                  12           12386988
new-topic 1           12360000        0                 38                  38           12359962

CLIENT_HOST      CLIENT_ID         TOPIC          ASSIGNED_PARTITIONS

We have huge lag here, but that’s fine, as we’re going to reset the offsets anyway. Stop the workers consuming from the older topic by whatever means you can and note down the time at which they stopped. We will now reset the consumer group offset for this new group. Just a note: in this case, we don’t have any consumers connected to this group, but if we did we’d also have to turn them off. You cannot reset a consumer group offset if there are clients connected to it.

To reset the offsets, just run the following command (here we’ll assume that we stopped the previous consumers at 10:00 of March 15th, 2023, and subtract ten minutes just to be safe):

kafka-consumer-groups.sh \
	--bootstrap-server='new-brokers' \
	--group='new-group' \
	--topic='new-topic' \
	--reset-offsets \
	--to-datetime 2023-03-15T09:50:00.000 \
	--dry-run

Running this dry run will not make any changes, It will only show you the new offsets for the consumer group, e.g.:

TOPIC     PARTITION   NEWEST_OFFSET   OLDEST_OFFSET     CONSUMER_OFFSET     LEAD         LAG
new-topic 0           12387000        0                 12386998            12386988     2
new-topic 1           12360000        0                 12369951            12369951     49

CLIENT_HOST      CLIENT_ID         TOPIC          ASSIGNED_PARTITIONS

If nothing looks strange to you, go ahead, substitute --dry-run with --execute and run the command. Afterwards, deploy the new consumer configuration, turn on the worker, and let it run.

The post Performing broker migrations on your Kafka consumers appeared first on Blog IT.

If you can listen to talk/news radio programs or make phone calls, and at the same time do something else, such as driving, house chores, exercising or routine shopping, then, you can gradually build up to get a full extra day (or more) in your week out of this ability. Podcasts are an easy picking towards taking advantage of your parallelizable time.

Basics: What are Podcasts and how do you get them?

Podcasts are (mostly) free downloadable audio shows in a myriad of languages which you can subscribe and listen to. If there is a topic you are interested in, there most likely is one or more podcasts shows on that topic.

A common way (as of this writing) to keep up with podcasts is to have a podcatcher app on your smartphone, which keeps track of subscribed show feeds, which in turn contain the individual episodes. The podcatcher app will playback the episodes as well, by either streaming them, or downloading to the phone’s memory first. Each episode corresponds to an audio file (usually in MP3 format) and its size can vary from a few megabytes to several hundreds.

For iOS, you can easily start with the Apple Podcasts app. For Android, I recommend the Podcast Addict app. Some other non-podcast focused apps, such as Spotify, also have podcast listening functionality. Try different podcatchers until you have found one that suits you. After you have installed your podcatcher app:
1. Browse through the directory of shows or use the search feature.
2. Subscribe to shows of your interest.
3. Start listening!

Level Up: Accelerating content and adjusting topics to your daily energy focus shifts

Once you have gotten the hang of using your chosen podcatcher app and in which contexts of your day you can effectively parallelize listening to your catalog of shows, there are a few boosts you can get without much effort.

For starters, most podcatcher apps nowadays can time shift (accelerate) content, keeping the pitch correct. For instance, if a podcast episode at the regular 1.0x speed takes 20 minutes to listen to, accelerating it to 2.0x will reduce the time to listen to it in half. This way you can consume two shows in same amount of time you’d only consume one. Remain on guard not to go greedy on this feature, as some podcast hosts and guests might speak in an unfamiliar accent or the content may be too complex to gulp down at such high speeds. Adjust accordingly.

Another simple adjustment, which can take several iterations to master, is choosing when and where to listen to each show. The purpose is to effectively retain and enhance the usefulness of the content you are listening to by aligning moods and contexts. This will vary significantly amongst individuals, considering their subscriptions catalog, how their days look and what is the desired outcome of the overall daily/weekly listening experience.

To give an example, I usually start my workdays off with inspirational content, changing to work-related content an hour or so before starting work. Then, after work, I switch to leisure/hobby content, making sure to savor those at 1.0x. When I’m engaged in more cognitively demanding tasks, I switch to listening to music in the background (sometimes in podcast form too).

Each podcatcher app has different ways to shift the kinds of content you are listening to. Either by adjusting a manual playlist, using categories/channels or some other feature which can go by another name. Explore your podcatcher app to find out how.

Beyond Podcasts: Audiobooks, online classes, conferences, video-to-audio, etc.

There are other sources of parallelizable content which you can consume that are not available in podcast form. Usually they require other non-podcatcher apps to play them back, may require purchasing (most Audiobooks), manual download, transcoding (for the more technically inclined) and other means to get access to them. My recommendation is to remain on the lookout and integrate them into your routine as you see fit.

Make the most out of your 8^th day in the week, and most importantly, be sure to enjoy it!

The post The 8+ Day Week with Podcasts – Hidden Parallelizable Time appeared first on Blog IT.

According to FAST Search’s TechNet documentation, you can use the Set-FASTSearchMetadataManagedProperty cmdlet to configure the main settings of a managed property. Among other things, you can enable stemming for a specific managed property. Here’s how it’s done:

$title = Get-FASTSearchMetadataManagedProperty –Name title
Set-FASTSearchMetadataManagedProperty –ManagedProperty $title –StemmingEnabled $true

The problem is that this doesn’t work. The StemmingEnabled argument is actually ignored by the cmdlet so, we need to use a different approach. This is the right way:

$title = Get-FASTSearchMetadataManagedProperty –Name title
$title.StemmingEnabled = 1
$title.update()

Although Microsoft is yet to update TechNet’s documentation, it has published a support article in its knowledge base (KB2468430) which describes this very solution.

The post FAST Search Tip #1: Enabling Stemming in PowerShell appeared first on Blog IT.

<Field

    ID="{538c71e4-8650-4ce7-b021-920effa66346}"

    Type="Choice"

    Name="Publishing_x0020_Status"

    StaticName="Publishing_x0020_Status"

    DisplayName="Publishing Status"

    Required="FALSE" 

    Format="Dropdown"

    FillInChoice="FALSE"

    Group="Custom Columns">

    <Default>Not Published</Default>

    <CHOICES>

       <CHOICE>

          Not Published

       </CHOICE>

       <CHOICE>

          Pending

       </CHOICE>

       <CHOICE>

          Published

       </CHOICE>

       <CHOICE>

          Error

       </CHOICE>

    </CHOICES>

</Field>

Although the XML is well formed and all the attributes and values are correct, this field definition has a problem. If you create a site column using this definition, here is what happens:

• If you check the site column configuration in SharePoint, everything will look fine
• If you add this site column to a list (either through a content type, or directly) and you edit an item using the default list forms, everything will work as expected.
• If you try to edit the list items using a DataSheet View, you won’t be able to select any value from this choice field and SharePoint will always throw a validation error.

The problem here are the invisible characters (spaces, newlines and tabs) between the <CHOICE> and </CHOICE> tags and their inner values. Apparently, when using list forms these characters are trimmed from the valid choices, but when using the datasheet view they are not, causing a strange behavior when editing an item in that view.

The correct definition for this field would be:

<Field

    ID="{538c71e4-8650-4ce7-b021-920effa66346}"

    Type="Choice"

    Name="Publishing_x0020_Status"

    StaticName="Publishing_x0020_Status"

    DisplayName="Publishing Status"

    Required="FALSE" 

    Format="Dropdown"

    FillInChoice="FALSE"

    Group="Custom Columns">

    <Default>Not Published</Default>

    <CHOICES>

       <CHOICE>Not Published</CHOICE>

       <CHOICE>Pending</CHOICE>

       <CHOICE>Published</CHOICE>

       <CHOICE>Error</CHOICE>

    </CHOICES>

</Field>

The post SharePoint Tip #27: Choice Columns and DataSheet Views appeared first on Blog IT.

<?xml version="1.0" encoding="utf-8" ?>
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
  <ContentType
    ID="0x0100C5647A362F236548B218C15302286758"
    Name="MyCT"
    Description="Simple Custom Content Type" 
    Inherits="TRUE"
    Overwrite="TRUE"
    Version="0">
    <Folder TargetName=”_cts/MyCT” />
    <FieldRefs>
      <FieldRef ID="{fa564e0f-0c70-4ab9-b863-0177e6ddd247}" Name="Title" />

      <!-- This is a comment –->   
      <FieldRef ID="{b402db15-ee44-4ec4-89e3-23e10a8fc64c}" Name="My_x0200_Field" />
      <FieldRef ID="{538c71e4-8650-4ce7-b021-920effa66346}" Name="Status" />
    </FieldRefs>
  </ContentType>
</Elements>

The Content Type feature element above will cause the Content Type to be created with only the Title field, which is inherited from the parent Content Type. All the fields referenced in this Content Type definition will be ignored.

The post SharePoint Tip #26: Comments in Content Type Definition appeared first on Blog IT.

Needless to say, this can save you a lot of time, especially if you already have a large base of users with a few populated properties on Active Directory. This is, however, a one way process, that is, only from the Directory to SharePoint, never the other way around.

One question you might be asking is: what if I update a User Profile on SharePoint, putting new data on a mapped property and then a Profile Import occurs?

The answer to this is not straightforward, because it depends on your answer to one or two questions.

Question 1: In Active Directory (or LDAP server), does the mapped property have a value?

If so, then the value in SharePoint’s User Profile property will get overwritten by the one in the directory’s user account property. No need to check Question 2.

If not, then check Question 2.

Question 2: Did you update the User Profile using SharePoint’s UI, or using the API?

If you used the SharePoint’s UI then the User Profile property will retain the value, since the mapped property in the directory service has no value to replace it.

If you used the API, the Profile Import will clear the value of the mapped property on SharePoint’s User Profile. I cannot explain why this difference exists, but I have seen it happen.

Conclusion: If you are updating User Profile properties by using SharePoint’ API, make sure you remove their mappings (to directory services’ properties) to prevent you from losing data when the Profile Import executes.

The post SharePoint Tips #25: Beware of User Profile Import appeared first on Blog IT.

Most navigation options on a SharePoint web site are manipulated through the SPWeb object’s property bag. Set the following properties to configure a web site’s navigation settings:

• __IncludeSubSitesInNavigation: Set this property to True to show the subsites of the current site on the global and current navigation menus. Set it to False to hide the subsites.
• __IncludePagesInNavigation: Set this property to True to show the pages on the global and current navigation menus. Set it to False to hide the pages.
• __InheritCurrentNavigation: Set this property to True to display the same navigations items as the parent site in the current navigation menu. Set it to False to display only items from the current site and subsites.
• __NavigationShowSiblings: Set this property to True to display the navigation items below the current site and the current site’s siblings, in the current navigation menu.
• __NavigationOrderingMethod: Set this property to one of the following values:
  • 0: Sort the navigation items automatically
  • 1: Sort the subsites manually but the pages automatically
  • 2: Sort the navigation items manually
• __NavigationAutomaticSortingMethod: When using automatic ordering, set this property to one of the following values:
  • 0: Sort by Title
  • 1: Sort by Created Date
  • 2: Sort by Last Modified Date
• __NavigationSortAscending: When using automatic ordering, set this property to True if the automatic sorting of the navigation items should be done in ascending order. Set it to False to use descending order.

Note: there are two underscore characters (_) in each property name.

Example
In the example below, I’m setting the navigation options to:

• Show only the current site’s navigation items in the global navigation
• Show the current site’s sub sites in the global navigation
• Don’t show the pages in the global navigation
• Show only the current site’s navigation items in the current navigation
• Don’t show the current site’s siblings in the current navigation
• Sort the navigation items manually

// break inheritance of global navigation
web.Navigation.UseShared = false;

// show subwebs but don't show pages in global navigation
web.AllProperties["__IncludeSubSitesInNavigation"] = "True";
web.AllProperties["__IncludePagesInNavigation"] = "False";

// show only current site's navigation items in current navigation
web.AllProperties["__InheritCurrentNavigation"] = "False";
web.AllProperties["__NavigationOrderingMethod"] = "2";
web.AllProperties["__NavigationShowSiblings"] = "False";
web.Update();

The post SharePoint Tip #24: Changing a Web Site’s Navigation Settings appeared first on Blog IT.

• A Profile Page that shows the user profile data, according to the user viewing it and the privacy options of each profile field. You can find that page (Person.aspx) in the root web site of the My Site host site collection (e.g http://mysites/Person.aspx?accountname=domain\username).
• A Personal Site which is a site collection where you can show personal and public information and documents.

The Personal Site is automatically provisioned for you when you try to access your My Site (using the My Site link or clicking your username). However, you might want to create it programmatically for one or more users, before they even try to access it. See below an example of how to do it.

// open a site collection
using (SPSite site = new SPSite("http://myportal"))
{
    // get the server context
    ServerContext context = ServerContext.GetContext(site);

    // create the profile manager object
    UserProfileManager profileManager = new UserProfileManager(context);

    // get the user profile
    UserProfile profile = profileManager.GetUserProfile("domain\\username");

    // create the user's personal site
    profile.CreatePersonalSite();
}

The really important line of code here is the last one. There rest was already explained in one of my previous posts (WSS Tip #20: Get User Profile).

The post SharePoint Tip #23: Creating a Personal Site appeared first on Blog IT.

// open a site collection
using (SPSite site = new SPSite("http://myportal"))
{
    // get the server context
    ServerContext context = ServerContext.GetContext(site);

    // create the profile manager object
    UserProfileManager upm = new UserProfileManager(context);

    // create the user profile
    UserProfile profile = upm.CreateUserProfile("domain\\username");
}

Just like getting a User Profile from the User Profile Store, to create one you need a UserProfileManager object. After that, you just need to call the CreateUserProfile method and pass it the user account name (including the domain).

Attention: User Profiles are a MOSS only feature. You cannot use them with WSS.

The post SharePoint Tip #22: Create a User Profile appeared first on Blog IT.

But what if you want to start a profile import programmatically, without having to wait for the next scheduled import? Well, that can be done using the following code snippet.

// open a site collection
using (SPSite site = new SPSite("http://myportal"))
{
    // get the server context
    ServerContext context = ServerContext.GetContext(site);

    // create the profile configuration manager object
    UserProfileConfigManager upcm = new UserProfileConfigManager(context);

    // check if there is already an import in progress
    if (!upcm.IsImportInProgress())
    {
        // if not, start a new one
        // pass "true" as a parameter, if the import is incremental
        // pass "false" as a parameter, if you want a full import
        upcm.StartImport(true);
    }
}

The profile import process is managed using the UserProfileConfigManager object, which can be created using a ServerContext object. Before starting an import, using the StartImport method, you should always check if there is already an import in progress (using the IsImportInProgress method).

The StartImport method accepts a single boolean parameter which specifies if you want an incremental import (true) or a full import (false).

If necessary, you can build a while cycle to wait for the import to be over, testing the IsImportInProgress return value, until it returns false.

Attention: User Profiles are a MOSS only feature. You cannot use them with WSS.

The post SharePoint Tip #21: Importing User Profiles appeared first on Blog IT.