Skip to content

Azure Service Bus Adapter

Overview

Windows Azure Service Bus is a Platform as a Service (PaaS) capability provided by Microsoft that provides a highly robust messaging fabric hosted by Microsoft Windows Azure.

Azure Service Bus extends on-premise messaging fabrics such as Neuron ESB by providing pub-sub messaging capable of traversing firewalls, a taxonomy for projecting entities and very simple orchestration capabilities via rules and actions.

As shown in Figure 1, Azure Service Bus bridges on-premise messaging capabilities enabling the ability to develop hybrid cloud applications that integrate with external services and service providers that are located behind the firewall allowing a new, modern breed of compositions to transcend traditional network, security and business boundaries.

Bridging ESBs in Hybrid Clouds
Figure 1: Bridging ESBs in Hybrid Clouds – Azure Service Bus extends on-premise messaging fabrics such as Neuron ESB enabling a next generation of hybrid cloud applications that transcend traditional network, security and business boundaries.

There are two services supported by Azure Service Bus:

  • Azure Service Bus Relay: Serves as a push-based relay between two (or more) endpoints. A client and service (or services) establish an outbound, bi-directional socket connection over either TCP or HTTP on the relay and thus, messages from the client tunnel their way through the relay to the service. In this way, both the client and service are really peers on the same messaging fabric.
  • Azure Service Bus Brokered Messaging: Provides a push-based durable message broker that supports queues, topics and subscriptions. A party wishing to send messages to Azure Service Bus establishes a TCP or HTTP connection to a queue or topic and pushes messages to the entity. A party wishing to receive messages from Azure Service Bus establishes a TCP or HTTP connection and pulls messages from a queue or subscription.

Neuron ESB 3.0 supports both Azure Service Bus services and this topic focuses on support of Azure Service Bus Brokered Messaging via the Neuron Azure Service Bus Adapter.

For more information on support for Azure Service Bus Relay support, please see “Azure Service Bus Integration” in the “Service Endpoints” topic in the Neuron ESB 3.0 product documentation.

About the Neuron Azure Service Bus Adapter

The Neuron Azure Service Bus Adapter provides full support for the latest capabilities provided by the Windows Azure SDK version 1.7.

Once the Neuron Azure Service Bus adapter is registered and an Adapter Endpoint is created, all configuration is managed through the property grid of the Adapter located on the properties tab of the Adapter Endpoint’s Details Pane:

Neuron Azure Service Bus Adapter – Property Grid
Figure 2: Neuron Azure Service Bus Adapter – Property Grid – All configurations for an adapter is managed through the property grid. Properties are divided into 3 sections, General, Publish Mode Properties, and Subscribe Mode Properties.

Please note that in order to connect to an Azure Service Bus entity with the Neuron Azure Service Bus adapter, you need to sign up for an Azure account and create an Azure Service Bus namespace with the required entities and ACS configuration. For more information, visit http://azure.com

Features

The Neuron Azure Service Bus adapter supports the following Azure Service Bus Brokered Messaging features:

  • Send to Azure Service Bus Queue
  • Send to Azure Service Bus Topic
  • Receive from Azure Service Bus Queue
  • Receive from Azure Service Bus Subscription

In addition, the Neuron Azure Service Bus adapter simplifies the development experience by providing additional capabilities typical in production scenarios without the need to write custom code including:

  • Smart Polling
  • Eventual Consistency
  • Transient Error Detection and Retry

The Neuron Azure Service Bus adapter is installed as part of the core Neuron ESB installation. The adapter is packaged into a single assembly located within the \Adapters folder under the root of the default Neuron ESB installation directory:

  • Neuron.Esb.Adapters.AzureServiceBusAdapter.dll

In addition, the following assembly is required and automatically installed in the root of the folder created for the service instance name:

  • Microsoft.ServiceBus.dll (Azure SDK version 1.7)

To use the adapter, it must first be registered within the Neuron ESB Explorer Adapter Registration Window. Within the Adapter Registration Window, the adapter will appear with the name “Azure Service Bus Adapter”. Once registered, a new Adapter Endpoint can be created and configured with an instance name of your choice:

Neuron ESB Explorer Adapter Registration Window - Property Grid
Figure 3: Neuron ESB Explorer Adapter Registration Window – Property Grid – Before configuring the adapter instance for Publish or Subscribe mode, the adapter must first be registered.

Supported Modes

Once the initial registration is complete, the Neuron Azure Service Bus adapter can be configured in one of 2 modes: Publish and Subscribe.

Publish

Publish mode allows Neuron ESB to monitor an Azure Service Bus Queue or Subscription by regularly polling, de-queuing all the messages, and publishing those messages to a Neuron ESB Topic. Messages are read synchronously via a one-way message exchange pattern.

Receiving Messages from Azure Service Bus
Figure 4: Receiving Messages from Azure Service Bus – When in Publish mode, the adapter supports receiving messages from an Azure Service Bus entity and publishing the messages on Neuron ESB.

Configuration

Configuring the Publish mode of the Neuron Azure Service Bus adapter requires that minimally, the following properties are set:

General Properties
  • Azure Service Bus Namespace Name – A registered namespace on Azure Service Bus. For example ‘neudesic’ would be the namespace for: sb://neudesic.servicebus.windows.net  (for information on how to provision, configure and manage Azure Service Bus namespaces, please see the Azure Service Bus topic on http://azure.com).
  • Azure ACS Issuer Name – The account/claim name for authenticating to the Windows Azure Access Control Service (ACS – For information on how to provision,  configure and manage Azure Access Control namespaces, please see the Azure Access Control topic on http://azure.com).
  • Azure ACS Key – The shared key used in conjunction with Azure ACS Issuer Name.
  • Azure Entity Type – Queue or Subscription
  • Azure Channel Type – Default, if outbound TCP port 9354 is open or HTTP to force communication over HTTP port 80/443 (In Default mode, the Neuron Azure Service Bus Adapter will try to connect via TCP. If outbound TCP port 9354 is not open, choose HTTP).
  • Retry Count – The number of Service Bus operations retries to attempt in the event of a transient error (for more information on this setting, see the “Understanding Transient Error Detection and Retry” topic).
  • Minimum Back-Off – The minimum number of seconds to wait before automatically retrying a Service Bus operation in the event that a transient error is encountered (for more information on this setting, see the “Understanding Transient Error Detection and Retry” topic).
  • Maximum Back-Off – The maximum number of seconds to wait before automatically retrying a Service Bus operation in the event that a transient error is encountered (for more information on this setting, see the “Understanding Transient Error Detection and Retry” topic).
Publish Properties
  • Azure Queue Name- The name of the queue that you want to receive messages from (this option appears when you choose “Queue” as the Azure Entity Type in General Properties).
  • Azure Topic Name – The name of the topic that the subscription you want to receive messages from is associated with (this option appears when you choose “Topic” as the Azure Entity Type in General Properties).
  • Azure Subscription Name – The name of the subscription you want to receive messages from (this option appears when you choose “Topic” as the Azure Entity Type in General Properties).
  • Delete After Receive – False by default. If set to True, deletes the message from the queue or topic after it is received regardless of whether it is published to Neuron successfully (for more information on this setting, see the “Understanding Eventual Consistency” topic).
  • Wait Duration – Duration (in seconds) to wait for a message on the queue or subscription to arrive before completing the poll request (for more information on this setting, see the “Understanding Smart Polling” topic).
  • Neuron Publish Topic – The Neuron topic that messages will be published to. Required for Publish mode.
  • Error Reporting – Determines how all errors are reported in the Windows Event Log and Neuron Logs. Either as Errors, Warnings or Information.
  • Error on Polling – Register failed message and exception with Neuron Audit database. Please note that a valid SQL Server database must be configured and enabled.
  • Audit Message on Failure – Determines if polling of data source continues on error and if consecutive errors are reported.

The following shows the General configuration for an instance of the Neuron Azure Service Bus adapter called “Azure – Receive” in Publish mode:

Publish Mode General Configuration
Figure 5: Publish Mode General Configuration – When in Publish mode, the adapter supports receiving messages from an Azure Service Bus entity and publishing the messages on Neuron ESB.

The following shows the Properties configuration for a fully configured instance of the Neuron Azure Service Bus adapter in Publish mode:

Publish Mode Properties Configuration
Figure 6: Publish Mode Properties Configuration – When in Publish mode, the adapter supports receiving messages from an Azure Service Bus entity and publishing the messages on Neuron ESB.

Subscribe

Subscribe mode allows Neuron ESB to write messages that are published to Neuron ESB to an Azure Service Bus queue or topic. In this manner, Neuron ESB supports the ability to bridge an Azure Service Bus entity, allowing for on-premise parties to seamlessly communicate with Azure Service Bus.  Once Neuron ESB receives a message, it sends the message to an Azure Service Bus Queue or Topic.

Sending Messages to Azure Service Bus
Figure 7: Sending Messages to Azure Service Bus – When in Subscribe mode, the adapter supports sending messages published on Neuron ESB to an Azure Service Bus entity.

Configuration

In addition to the General Properties covered under the Publish mode documentation, configuring the Subscribe mode of the Neuron Azure Service Bus adapter requires that minimally, the following properties are set:

Subscribe Properties
  • Adapter Send Mode – Choose Asynchronous for maximum throughput or Synchronous for maximum reliability (for more information on this setting, see the “Choosing Synchronous vs. Asynchronous” topic).
  • Adapter Queue Name – The name of the queue you want to send messages to (this option appears when you choose “Queue” as the Azure Entity Type in General Properties).
  • Adapter Topic Name – The name of the topic you want to send messages to (this option appears when you choose “Topic” as the Azure Entity Type in General Properties).

The following shows the General configuration for an instance of the Neuron Azure Service Bus adapter called “Azure – Send” in Subscribe mode:

Subscribe Mode General Configuration
Figure 8: Subscribe Mode General Configuration – When in Subscribe mode, the adapter supports sending messages from Neuron ESB to an Azure Service Bus entity.

The following shows the Properties configuration for a fully configured instance of the Neuron Azure Service Bus adapter in Subscribe mode:

Subscribe Mode General Configuration
Figure 9: Subscribe Mode General Configuration – When in Subscribe mode, the adapter supports sending messages from Neuron ESB to an Azure Service Bus entity.

Understanding Transient Error Detection and Retry

When working with services in general and multi-tenant PaaS services in particular, it is important to understand that in order to scale to virtually hundreds of thousands of users/applications, most services like Azure Service Bus, SQL Azure, etc. implement a throttling mechanism to ensure that the service remains available.

This is particularly important when you have a process or application that is sending or receiving a high volume of messages because in these cases, there is a high likelihood that Azure Service Bus will throttle one or several requests. When this happens, a fault/HTTP error code is returned and it is important for your application to be able to detect this fault and attempt to remediate accordingly.

Unfortunately, throttle faults are not the only errors that can occur. As with any service, security, connection and other unforeseen errors (exceptions) can and will occur, so the challenge becomes not only being able to identify the type of fault, but in addition, know what steps should be attempted to remediate.

Per the guidance provided by the Azure Customer Advisory Team, the Neuron Azure Service Bus adapter uses an exponential back-off based on the values provided for the Retry Count, Minimum Back-Off and Maximum Back-Off properties within the Properties tab for both Publish and Subscribe mode.

Given a value of 3 retries, two seconds and ten seconds respectively, the adapter will automatically determine a value between two and ten and back off exponentially one time for each retry configured:

Exponential Back-Off  Configuration
Figure 10: Exponential Back-Off  Configuration – The adapter will automatically detect transient exceptions/faults and retry by implementing an exponential back-off algorithm given a retry count, initial and max back-off configuration.

Taking this example, as shown in Figure 11, if the adapter chose an initial back-off of two seconds, in the event of a transient fault being detected (i.e. throttle, timeout, etc.) the adapter would wait two seconds before trying the operation again (i.e. sending or receiving a message) and exponentially increment the starting value until either the transient error disappears or the retry count is exceeded:

Figure 11: Exponential Back-Off Example – Illustrates the automatic detection and retry capability of the adapter given a retry count, initial and max back-off.

In the event that the retry count is exceeded, the Neuron Azure Service Bus adapter will automatically persist a copy of the message in the audit database to ensure that no messages are lost (provided a SQL Server database has been configured).

Understanding Smart Polling

When configuring the Neuron Azure Service Bus Adapter in Publish mode, the adapter can take advantage of a Neuron ESB feature known as Smart Polling. With Smart Polling, the adapter will connect to an Azure Service Bus queue or subscription and check for messages. If one or message is available, all messages will be immediately delivered (see “Understanding Eventual Consistency” for more information on supported read behaviors). However, if no messages are available, the adapter will open a connection to the Azure Service Bus entity and wait for a specified timeout before attempting to initiate another poll request (essentially resulting in a long-polling behavior). In this manner, Azure Service Bus quotas are honored while ensuring that the adapter issues a receive request only when the configured timeout occurs.

Understanding Eventual Consistency

When working with Azure Service Bus, it is important to note that the model for achieving consistency is different than traditional distributed transaction models. For example, when working with modern relational databases or spanning multiple services that are composed into a logical unit of work (using WS-Atomic Transactions for example), it is a common expectation that work will either be performed completely or not at all. These types of transactions have the characteristics of being atomic, consistent, independent and durable (ACID). However, to achieve this level of consistency, a resource manager is required to coordinate the work being carried out by each service/database that participates in a logical transaction.

Unfortunately, given the virtually unlimited scale of the web and cloud computing, it is impossible to deploy enough resource managers to account for the hundreds of thousands if not millions of resources required to achieve this level of consistency. Even if this were possible, the implications on achieving the scale and performance demanded by modern cloud-scale applications would be physically impossible.

Of course, consistency is still as important for applications that participate in logical transactions across or consume cloud services. An alternative approach is to leverage an eventually consistent, or basically available, soft state, eventually consistent (BASE) approach to transactions.

Ensuring Eventual Consistency in Publish Mode

Azure Service Bus supports this model for scenarios that require consistency and the Neuron Azure Serviced Bus adapter makes taking advantage of this capability simply a matter of setting the “Delete After Receive” property (available in the Publish Mode Settings) to False, which is the default.

When set to False, when receiving a message, the adapter will ensure that the message is not discarded from the Azure Service Bus entity until the message has been successfully published to Neuron ESB. In the event that an error occurs when attempting to publish a message, the message will be restored on the Azure Service Bus entity ensuring that it remains available for a subsequent attempt to receive the message (Please note that lock durations configured on the entity will affect the behavior of this feature. For more information, please refer to the Azure Service Bus documentation on MSDN: http://msdn.microsoft.com/en-us/library/ee732537.aspx).

Choosing Synchronous versus Asynchronous Receive

When the Neuron Azure Service Bus adapter is configured in Subscribe mode, you can choose to send messages to an Azure Service Bus queue or topic in either synchronous or asynchronous mode by setting the Adapter Send Mode property to either “Asynchronous” or “Synchronous” in the Subscribe Mode Property group.

If reliability is a top priority such that the possibility of message loss cannot be tolerated, it is recommended that you choose Synchronous. In this mode, the adapter will transmit messages to an Azure Service Bus queue or topic at rate of about 4 or 5 per second. While it is possible to increase this throughput by adding additional adapters in subscribe mode, as a general rule, use this mode when choosing reliability at the expense of performance/throughput.

To contrast, if performance/low-latency/throughput is a top priority, configuring the adapter to send asynchronously will result in significantly higher throughput (by several orders of magnitude). While the send performance in this mode is much higher, in the event of a catastrophic failure (server crash, out of memory exception) it is possible for messages that have left the Neuron ESB process but have not yet been transmitted to the Azure Service Bus (i.e. are in memory) the possibility for message loss is much higher than when in synchronous mode because of the significantly higher density of messages being transmitted.

Other Scenarios

Temporal Decoupling

One of the benefits of any queue-based messaging pattern is that the publisher/producer is decoupled from the subscribers/consumers. As a result, parties interested in a given message can be added and removed without any knowledge of the publisher/producer.

By persisting the message until an interested party receives the message, the sending party is further decoupled from the receiving party because the receiving party need not be available at the time the message was written to persistence store. Azure Service Bus supports temporal decoupling with both queues and topics because they are durable entities.

As a result, a party that writes new order messages to an Azure Service Bus queue can do so uninhibitedly as shown below:

Temporal Decoupling

Figure 12:Temporal Decoupling– One of the benefits of pull-based brokered messaging is that the consumer of a message need not be available or ready to begin processing messages when the producer of a message sends a message to the queue or topic.

When you configure an instance of the Neuron Azure Service Bus adapter in Publish mode, you can disable the adapter by unchecking the “Enabled” box. Any new messages written to the Azure Service Bus queue or subscription will persist until the adapter is enabled once again.

Competing Consumers

Another messaging pattern that allows you to take advantage of the benefits of pull-based pub-sub model from a performance and scalability perspective is to adjust the number of consumers supported by the resources available to you and keep adding consumers until throughput requirements are met.

Competing Consumers

Figure 13:Competing Consumers– One of the benefits of pull-based brokered messaging is that the additional consumers can be added as needed to achieve higher throughput and performance.

To take advantage of this pattern with the Neuron Azure Service Bus adapter and Azure Service Bus, simply add additional instances of the Publishing adapter as needed:

Competing Consumers

Figure 14: Competing Consumers – Adding additional consumers with Neuron Azure Service Bus is simply a matter of adding additional instances of the Publishing adapter.

Property Table

Property NameRequiredDescription
General These properties are used for all modes of the adapter
Azure Service Bus Namespace NameYesA registered namespace on Azure Service Bus. For example ‘neudesic’ would be the namespace for: sb://neudesic.servicebus.windows.net
Azure ACS Issuer NameYesThe account/claim name for authenticating to the Windows Azure Access Control Service (ACS)
Azure ACS KeyYesThe shared key used in conjunction with Azure ACS Issuer Name.
Azure Entity TypeYesDefault Queue. Queue or Topic
Azure Channel TypeYesDefault is Default. Default, if outbound TCP port 9354 is open or HTTP to force communication over HTTP port 80/443
Retry CountYesDefault 5. The number of Service Bus operations retries to attempt in the event of a transient error (for more information on this setting, see the “Understanding Transient Error Detection and Retry” topic).
Minimum Back OffYesDefault 3. The minimum number of seconds to wait before automatically retrying a Service Bus operation in the event that a transient error is encountered (for more information on this setting, see the “Understanding Transient Error Detection and Retry” topic).
Maximum Back OffYesDefault 3. The maximum number of seconds to wait before automatically retrying a Service Bus operation in the event that a transient error is encountered (for more information on this setting, see the “Understanding Transient Error Detection and Retry” topic).
Publish Mode These properties are only used when the adapter is in either Request/Response or Publish mode.
Azure Queue NameYesThe name of the queue that you want to receive messages from (this option appears when you choose “Queue” as the Azure Entity Type in General Properties).
Azure Topic NameYesThe name of the topic that the subscription you want to receive messages from is associated with (this option appears when you choose “Topic” as the Azure Entity Type in General Properties).
Azure Subscription NameYesThe name of the subscription you want to receive messages from (this option appears when you choose “Topic” as the Azure Entity Type in General Properties).
Delete After ReceiveYesDefault False. If set to True, deletes the message from the queue or topic after it is received regardless of whether it is published to Neuron successfully (for more information on this setting, see the “Understanding Eventual
Wait DurationYesDefault 5. Duration (in seconds) to wait for a message on the queue or subscription to arrive before completing the poll request (for more information on this setting, see the “Understanding Smart Polling” topic).
Neuron Publish TopicYesThe Neuron topic that messages will be published to. Required for Publish mode.
Error ReportingYesDefault Error. Determines how all errors are reported in the Windows Event Log and Neuron Logs. Either as Errors, Warnings or Information.
Error on PollingYesDefault Stop Polling On Error. Register failed message and exception with Neuron Audit database. Please note that a valid SQL Server database must be configured and enabled.
Audit Message on FailureYesDefault False. Determines if polling of data source continues on error and if consecutive errors are reported.
Subscribe Mode These properties are only used when the adapter is in either Solicit/Response or Request/Response mode.
Adapter Send ModeYesDefault Asynchronous. Choose Asynchronous for maximum throughput or Synchronous for maximum reliability (for more information on this setting, see the “Choosing Synchronous vs. Asynchronous” topic).
Adapter Queue NameYesThe name of the queue you want to send messages to (this option appears when you choose “Queue” as the Azure Entity Type in General Properties).
Adapter Topic NameYesThe name of the topic you want to send messages to (this option appears when you choose “Topic” as the Azure Entity Type in General Properties).

Message Format

Azure Service Bus uses a proprietary message envelope called a Brokered Message as the unit of communication between all messaging entities including queues, topics and subscriptions.

Publish Mode

In Publish mode, the Neuron Azure Service Bus Adapter will automatically map the body of the incoming Brokered Message to the Body property of the Neuron ESBMessage serializing the payload based on the detected encoding type as follows:

BrokeredMessage.ContentTypeESBMessage.Header.BodyType
text/plaintext/plain
text/xmltext/xml
application/msbin-1application/msbin-1
binary/bytesbinary/bytes
Othertext/xml

Note per the table above that unless otherwise specified, the Neuron Azure Service Bus adapter will assume that the incoming message payload is text/xml.

In addition, any properties stored in the Property property bag of the BrokeredMessage will be automatically mapped to the ESBMessage property bag provided the “Include Metadata” option is checked on the General tab in the Adapter Endpoints configuration.  An exception to this rule is that the adapter will always map the BrokeredMessage.LockToken to the ESBMessage property bag with the same name regardless of whether “Include Metadata” is checked.

Subscribe Mode

In Subscribe mode, the Neuron Azure Service Bus Adapter will automatically create a new Brokered Message for each transmission and map the body of an outgoing ESBMessage to the new message body as follows:

ESBMessage.Header.BodyTypeBrokeredMessage.ContentType
text/plaintext/plain
text/xmltext/xml
application/msbin-1application/msbin-1
binary/bytesbinary/bytes
Othertext/xml

In addition, any properties stored in the Property property bag of the ESBMessage will be automatically mapped to the BrokeredMessage property bag provided the “Include Metadata” option is checked on the General tab in the Adapter Endpoints configuration. 

Brokered Message Limitations

Note that the total payload size for Azure Service Bus messages is 256KB. The Neuron Azure Service Bus adapter will throw a runtime exception if a message greater than or equal to 256KB is sent and will save the message to the failed audit table.

Was this article helpful?
Dislike 0
Previous: AWS SQS Adapter
Next: CertainSafe Adapter