NATS Weekly #1

Week of November 15 - 21, 2021

🗞 Announcements, writings, and projects

A short list of  announcements, blog posts, projects updates and other news.

⚡️ Client library releases

A handful of client library releases by the NATS core team.

⚡️ Core maintainer Wally Quevedo tagged nats.py v2.0.0a2 release.

📝 Building Microservices with NATS

A NATS blog guest article by Chanaka Fernando describes the complexity of a system of microservices using service mesh with point-to-point messaging and compares it with one using NATS as the messaging hub. Many of the complexities and infrastructure needs just go away.

⚡️ NATS Server v2.6.5 released

A new version of the NATS Server has been released and it comes with a demo of some new capabilities.

👉 Pro-tip: pluggable/async consumers at runtime

One of the advantages of a publish-subscribe by default is the ability to plug in read-only consumers at runtime..

📖 Book release: Platforms with NATS

A new book has been released! This is by the author of the blog post linked above 👆.

💡 Recently asked questions

Questions sourced from Slack, Twitter, or individuals. Responses and examples are in my own words, unless otherwise noted.

Can I have multiple subscribers and have each message go to only one subscriber?

This is referred to as a queue group and applies to both a NATS subject and a stream. Here is a very terse example in Go (without error handling) supporting both examples.

nc, _ := nats.Connect("localhost:4222")

// Subscribe using a queue group. This allows one or more subscriptions
// to be added (or removed) dynamically.
nc.QueueSubscribe("weather.us.>", "weather-us", func(msg *nats.Msg) {
  // handle US weather alerts
})

Or with Jetstream:

js, _ := nc.Jetstream()

// Create a single stream that receives all weather events regardless
// of locale.
js.AddStream(&nats.StreamConfig{
  Name: "weather",
  Subjects: []string{
    "weather.>",
  },
})

// Subscribe using a queue group which implicitly creates a consumer
// to the stream with subject filter for US-based messages.
js.QueueSubscribe("weather.us.>", "weather-us", func(msg *nats.Msg) {
  // handle US weather alerts
})

What is the difference between using a stream versus not? Persistence. When a message is published and received on a stream, it is stored and then the message is forwarded to active consumers.

In practice, a pure NATS subject provides at-most-once delivery while a stream provides at-least-once delivery to a consumer. This opens up a whole host of options for consumers. A consumer (or a group of them) can start consuming a stream anytime and have the option of replaying messages earlier than they started started consuming. This is dictated by one of the delivery options such as nats.DeliverAll, nats.DeliverLast and nats.StartSequence (among a couple other options).

Can I implement the saga pattern using NATS?

Yes! The saga pattern can be modeled using orchestration or choreography. NATS supports orchestration via its request-reply support. Of course it supports orchestration since its natively an asynchronous message broker.

Depending on the complexity of the saga or how long running they are, a NATS stream should likely be used for increased resiliency and persistence of messages (especially for compensating actions).

Is there a document with all the known headers anywhere?

To my knowledge these are not yet documented centrally, but they do exist in the source code as a starting point. You may be able to search the docs by the header name to discover additional context.

A few key ones include:

• Nats-Msg-Id - Add this to a message to NATS server can de-duplicate if the client re-sends the same message due to some network issue.
• Nats-Expected-Stream - When a client publishes a message on a subject and this subject is expected to map to a stream, this header can be set to assert this expectation. Using the example above, PUB weather.us.pa {...} with the Nats-Expected-Stream: weather will assert the subject is bound to the stream.
• Nats-Expected-Last-Sequence - Ensure the message being published is only accepted by the stream if the sequence known by the publishes matches that of the server. This provides optimistic concurrency control at the stream level.
• Nats-Expected-Last-Subject-Sequence - Same as the previous header, but more granular to the subject level. Given a message from weather.us.pa and weather.us.nj, each would have their own last sequence number. This approach is preferred over the previous header to reduce contention if total ordering of a stream is not needed.
• Nats-Expected-Last-Msg-Id - The same semantics as Nats-Expected-Last-Sequence except it asserts it matches the (user-defined) Nats-Msg-Id of the last message written to the stream rather than the server-generated sequence number.

I have a use case for thousands or millions of streams? Can NATS support this?

When developers ask this question, they are often talking about some logical idea of a stream, such as an isolated series of messages in their domain.

NATS has both the concept of a stream, which can be defined as a durable sequence of messages as well as a subject which indicates the subject of a message that is sent.

A NATS stream can have one or more concrete or wildcard subjects bound to a stream. This is shown in the first question above. A stream named weather has a single subject pattern weather.> bound to it. The > wildcard means any subject under the weather. namespace will be send to the stream, such as weather.us.pa or weather.us.pa.18069 or something even more granular, theoretically for all countries, states, locales, etc.

This means that the size of the subject space to a single stream is (theoretically) unbounded. Each concrete subject can be treated as a logical, horizontal partition of the stream (most literature on partitioning is in regards to relational database tables, but the concept still applies).

There are a handful of benefits to this approach such as:

• optimistic concurrency control on a per-subject level (see the Nats-Last-Expected-Subject-Sequence header above)
• subject-level authorization controls for clients, independent of the stream
• optimized subject-level (single partition) filtering of messages for consumers

To answer the original question, yes NATS can handle millions of application/actor streams using subjects that are multiplexed onto a handful of streams. A rule of thumb is to use a stream per bounded context (not per service) which usually result in 10s of streams or 100s in larger systems.