Neuron ESB 3.6 Released!
The Neuron ESB 3.6 release represents a significant next step toward the evolution of the Neuron Platform. Much of the infrastructure has been refactored to facilitate our near-term vision of cloud hosting, as well as make independently hosted and deployable services, adapter and workflow endpoints a reality for our customers today. More organizations are seeking the agility to quickly develop, test, deploy and update business capabilities encapsulated using a Microservices architecture.
Neuron ESB 3.6 has been built on the principal of SOA and Microservices, hence it is the natural choice to build Microservice based APIs. In previous versions of Neuron ESB, organizations could easily create REST APIs or SOAP based services. However, in this release, they can now control the deployment aspect of it as well. Using Neuron ESB 3.6, organizations can now choose to run a service, (or group of services), as a fully independent service that can be scaled in or out based on consumption using platform services like Docker.
If you’re using Neuron ESB 3.5 today, please download the Neuron ESB 3.6 setup to update your existing installation. Current customers can obtain either the Patch or the full installation from the Neuron ESB Support website. If you are new to Neuron ESB, you can download the full installation from the Neuron ESB website.
Neuron ESB 3.6 provides dozens of new features, enhancements, and connectors, some of which include:
- Endpoint level Hosting
- Self-registration of Adapters/Connectors
- Messaging free Service and Adapter Endpoints
- Environment sensitive enabling/disabling of Endpoints
- ServiceNow Adapter
- New OAuth Providers
- Custom OAuth Providers
- JSON Schema Validation
- New Monitoring REST APIs
- 100+ Product Enhancements
All the changes included in the Neuron ESB 3.6 release can be found in the Neuron ESB Change Log which gets installed with Neuron ESB. In this blog post, I thought I would elaborate on some of the new enhancements and features we’ve added to the update.
Endpoint Level Hosting
Previous Hosting Model
In previous versions of Neuron ESB, the Neuron ESB Runtime Windows Service (esbservice.exe) was used to host all Adapter and Service Endpoints, as well as all Messaging Publishing Services and all internal Services. These services were isolated in their own .NET AppDomains within the Neuron ESB Runtime service. However, a fault tolerant hosting environment was provided for the Neuron ESB Workflow Engine. The hosting was embodied by Neuron ESB Availability Groups. Availability Groups were used to load balance the execution of Workflow Instances across multiple servers in dedicated/isolated host processes.
These Availability Groups were designed with built in clustering to achieve high availability and fault tolerance without the need to deploy Microsoft Windows Failover Cluster Services. Servers within a Neuron ESB Deployment Group could be assigned either Primary or Failover roles which allowed failed workflows to automatically rollover onto available servers and start where they left off, providing both resiliency and reliability for mission critical functions.
Users could create any number of Availability Groups which in turn would be assigned to any number of Workflow Endpoints. Availability Groups were isolated into their own process space and executed as child processes (hosts) of the Neuron ESB Runtime service. If a fatal error occurred that caused an Availability Group host to fail, the Neuron ESB Runtime service could restart the Availability Group on the same server, or on a different server assigned the Failover role.
In Neuron ESB 3.6, we’ve refactored and extended Availability Groups to allow them to host Adapter and Service Endpoints. As a result, Availability Groups have been renamed to Endpoint Hosts and have taken on a new and expanded role within Neuron ESB.
Endpoint Hosts can be used to:
- Host a single endpoint
- Facilitate Microservice deployment
- Host multiple endpoints of the same type (i.e. Adapter, Service or Workflow)
- Host multiple endpoint of different types (i.e. Adapter, Service or Workflow)
- Host endpoints under a unique service account (different from that of the Neuron ESB Runtime service)
- Isolate endpoints for system wide fault tolerance
- Reduce memory utilization
- Scale processing power
In previous versions of Neuron ESB, every Adapter and Service endpoint competed for the CPU processing of a single executable (i.e. esbservice.exe). Other competitors for those processing cycles and threads included Neuron ESB’s own internal services, which would sometimes take precedent. In some cases, either an internal service or a specific endpoint could consume the majority of processing cycles, leaving other endpoints waiting for available threads to be returned to the pool before being allowed to service requests. Additionally, even though all endpoints and internal services were encapsulated in their own .NET AppDomain, it did not preclude the possibility that one misbehaving Business Process, Adapter Endpoint or, custom code writing by a customer could crash the Neuron ESB Runtime process (i.e. esbsevice.exe).
Endpoint Hosts relieves these issues as each runs as an isolated process (i.e. NeuronEndpointHost.exe), with its own allocation of resources and threads. Using Endpoint Hosts, endpoints can be isolated into discreet, lightweight processes and deployed across multiple machines.
In Neuron ESB 3.6, Endpoint Hosts are more lightweight than their immediate predecessor, Availability Groups. Their executable has renamed and the communication mechanism and been changed to use an HTTP based API (default port 51004 can be changed in appSettings). Additionally, endpoints using this hosting model are no longer encapsulated in .NET AppDomains.
NOTE: By default, all new endpoints will run in the Neuron ESB Runtime service unless they are specifically assigned to an Endpoint Host using the “Endpoint Host” property located on their respective UI configuration screens. However, if migrating a solution saved in a previous version of Neuron ESB, those endpoints will automatically be assigned to run in the new Endpoint Host environment.
Neuron ESB Endpoint Health has been refactored to accommodate real time activity monitoring for the Endpoint Host environment as well as the different endpoints that they host. The new user interface provides users the ability to group and sort, and now lists all machines in the selected deployment group.
The Neuron ESB Endpoint Health interface has a scalable horizontal divider, separating Neuron ESB Messaging entities such as Topics, Adapter and Service Endpoints that are hosted in the Neuron ESB Runtime service from those hosted in the Endpoint Host environment. A context menu is exposed at the entity level that allows users to restart any selected endpoint or Endpoint Host.
Neuron ESB has always provided options for managing runtime security. In previous versions of Neuron ESB, the Neuron ESB Runtime could be set to run under the context of a specific Windows local user or domain account, providing it had the necessary permissions and was granted the “Log on as a Service” right. Any service or endpoint hosted by the Neuron ESB Runtime would essentially inherit that security context during execution. If an Availability Group was created, it would inherit those same credentials. Unfortunately, this meant that there was no way to run a given Workflow Endpoint under a set of credentials other those set at the Neuron ESB Runtime level. Only Adapter Endpoints had a “Run As” option which allowed users to specify an alternate set of credentials for an Adapter Endpoint to use at runtime.
NOTE: Requirements for changing the Neuron ESB Runtime service account can be found in the “Neuron ESB Runtime – Running under User Account” section of the Readme.html document installed in the default location of the Neuron ESB installation folder.
We modified this capability in Neuron ESB 3.6, introducing a Run As option at the Endpoint Host level. Unlike the service account used for the Neuron ESB Runtime service, these credentials do not require the “Log on as a Service” right. However, all other caveats that apply to the Neuron ESB runtime service account apply to the Endpoint Host credentials as well. Using Neuron ESB 3.6, users can create the appropriate Windows Credential by navigating to Security->Authentication->Credentials within the Neuron ESB Explorer. Once created, this credential will appear as an account that can be assigned to an Endpoint Host via its Run As property.
The Run As capability for Endpoint Hosts functions much like the Run As for Adapter Endpoints. Everything executing in the Endpoint Host process space will execute under the context of the Run As credential. That means any resource that the endpoint, its underlying Party or Business Processes access will be under the context of the assigned credential. Essentially this provides the ability to change the security context at the endpoint level or for a group of endpoints, regardless of their type (i.e. Adapter, Service or Workflow).
While developing the new hosting model for Neuron ESB 3.6, we took the opportunity to reevaluate the memory utilization footprint of both the Neuron ESB Runtime and Endpoint Host. Historically, the Neuron ESB Runtime made liberal use of memory to enhance execution time. It also provided the configuration and document repositories as in memory objects that could be retrieved and manipulated at runtime, by developers as well as internal Neuron ESB services. But there was a price, especially due to our usage of .NET AppDomains. It wasn’t unusual where large solutions with 100’s of Business Processes, 100s of Service Endpoints and 100s of Adapter Endpoints could easily consume 10+ Gigabytes of Working Set memory on startup.
In Neuron ESB 3.6, we completely changed how we manage and share configuration objects internally as well as shared resources. Users will find that solutions which required 10+ gigabytes of memory on startup, will consume anywhere from 2X to 5X less memory.
We tested this with a large solution, containing almost 200 Adapter Endpoints. We chose Adapter Endpoints as they tend to consume more memory than Service or Workflow Endpoints due to the proprietary assemblies that must be loaded with each one. Additionally, the solution contained a couple of hundred large Business Processes, 50 Topics, several Service Endpoints and other Neuron specific entities. When this solution was loaded and started in Neuron ESB 3.5.4, it consumed over 11 gigabytes of Working Set memory and took several minutes to startup.
We migrated the solution to Neuron ESB 3.6. We assigned all the Service and Adapter Endpoints to a single Endpoint Host. The Neuron ESB Runtime was relegated to Messaging Publishing Services (i.e. Topics) and our internal services. A noted observation was the startup time. The entire solution took less than 1 minute to startup, whereas under Neuron ESB 3.5, several minutes was required.
Once the solution was completely started in Neuron ESB 3.6, we noticed a significant reduction in the Working Set memory. By default, an Endpoint Host startup memory footprint is just under 200 MB of memory (no endpoints assigned). The total combined Working Set memory of the Neuron ESB Runtime and Endpoint Host was under 2.3 gigabytes. This represents about a 5X improvement in memory utilization.
Adapters are a key piece of capability in the world of an integration broker. They serve as the bridge to and from the bus between applications, databases, transports and protocols. The completeness of “what” you ship as well as how easy it is for others to build their own adapters is critical in accelerating the development of any solution. Not only is Neuron ESB one of the easiest adapter frameworks to learn in the industry, it also has pre-built adapters for most commodity transports, queuing technologies, databases and on-premise/cloud applications. Staying true to our roots, several changes have been introduced in this release to make connectivity easier, more intuitive as well as performant.
In previous versions of Neuron ESB, users had to go through multiple steps before they could use an adapter. The first step was to manually register the adapter followed by creating an Adapter Endpoint that could be used to interact with their system of choice. Starting with version 3.6 of Neuron ESB, all .NET based adapters, either shipped with or custom developed, are dynamically registered. There is no longer a need to register individual adapters before creating Adapter Endpoints. As part of this effort, we took the opportunity to rename the adapters shipped with Neuron ESB to more closely align them with their respective technologies. When migrating from previous versions, users must open their existing solution using the 3.6 version of the Neuron ESB Explorer. The Neuron ESB Explorer will automatically update the adapter names used in any existing Adapter Endpoints. Any existing registered adapters will be removed from the solution, as they are no longer needed. For reference, names of the Adapters prior to the 3.6 release, as well as their corresponding updated names are listed in the table below.
|Previous Names||Version 3.6|
|Active Directory Adapter||Microsoft Active Directory|
|AWS SQS Adapter||Amazon Simple Queue Service|
|ActiveMQ Adapter||Apache ActiveMQ|
|Apple Push Notification Service Adapter||Apple Push Notification Service|
|Client Socket Adapter||Client TCP Socket|
|WebSphere MQ Adapter||IBM MQ|
|Azure Service Bus Adapter||Microsoft Azure Service Bus|
|Dynamics AX Adapter||Microsoft Dynamics AX|
|Dynamics CRM 4.0 Adapter||Microsoft Dynamics CRM 4.0|
|Dynamics CRM Web API Adapter||Microsoft Dynamics CRM Web API|
|Dynamics CRM XRM Adapter||Microsoft Dynamics CRM XRM|
|Dynamics GP eConnect Adapter||Microsoft Dynamics GP|
|Microsoft Exchange Adapter||Microsoft Exchange|
|MSMQ Adapter||Microsoft MSMQ|
|SharePoint Subscription Adapter||Microsoft SharePoint|
|SQL Service Broker Adapter||Microsoft SQL Service Broker|
|Server Socket Adapter||Server TCP Socket|
Historically, Neuron ESB used its powerful messaging sub system to abstract and decouple service and adapter endpoints from one another. In essence, endpoints would adopt either a publisher or subscriber role, where neither would have to know about the other and, where modifying one would not affect the other. This model remains incredibly beneficial for those systems requiring a loosely coupled design. New subscribing or publishing endpoints could be added without impacting the processing of the other endpoints. However, those designs usually employ one way, multicast patterns. There are other scenarios (request/response patterns for example) though where inserting messaging as a decoupling agent makes little sense. In those scenarios it imposes unnecessary overhead, configuration and restrictions. For example, it’s unusual in request/response patterns that there would ever be more than one subscribing endpoint at a time fulfilling a request.
In Neuron ESB 3.5, we introduced the capability for users to call Adapter and Service Endpoints directly (without the messaging sub system) using either the Business Process engine or the Workflow engine. However, messaging elements would still need to be created, even though those elements were not used at runtime. For example, to create an Adapter Endpoint that could be used in the Business Process or Workflow Designer users were still required to first create a Topic, Publisher and Subscriber, and then associate those elements with the Adapter Endpoint using the Neuron ESB Explorer user interface.
In this release we’ve made it a lot easier for developers to create Adapter Endpoints that they intend to execute directly within our Business Process or Workflow engines. Starting with Neuron ESB 3.6, users can choose to simply disable the Messaging option located on the General tab of an Adapter Endpoint for all subscribe mode type of Adapter Endpoints. This allows the developer to create an Adapter Endpoint without needing to first create a Topic or Subscriber. At runtime, the Adapter Endpoint can be called directly within the Business Process or Workflow Engine but will not be a “live” subscriber to messages published by other systems. Some immediate benefits this provides are:
- Reduced Complexity
- No need to create a Topic or Subscriber before configuring an Adapter Endpoint.
- Reduced Runtime Utilization
- The Adapter Endpoint does not need to be loaded in memory with its own resources.
- The Topic Publishing Service is never created.
- Faster Startup Time
- Since there is no subscribing Adapter Endpoint, the master runtime does not attempt to start it as a manageable endpoint.
- The Topic is also not created and loaded by the master runtime.
The Neuron ESB 3.6 release introduces a new hosting model that allows Adapter Endpoints (and all endpoints) to be hosted either independently or with other endpoints in a configured Endpoint Host. This is in contrast to previous releases where, although each Adapter Endpoint was isolated in their own .NET App Domain, they were hosted within the main Neuron ESB runtime process (i.e. esbservice.exe) with all other internal services, Messaging Publishing services and Service Endpoints.
Using this new model, one or more Adapter Endpoints can be assigned to a dedicated Endpoint Host to provide resiliency, failover, and scale as well as process isolation. The latter becomes important in the case of a misbehaving endpoint or, Business Processes associated with that endpoint, which have the potential to crash a process/host. If isolated to an Endpoint Host, the misbehaving endpoint or Business Processes can’t affect any other running endpoints or the core Neuron ESB runtime process.
In the Neuron ESB 3.6 release, users can configure an Adapter Endpoint to run in either the existing Neuron ESB runtime host (as in previous versions) or the new Neuron Endpoint Host. The default behavior is for new Adapter Endpoints to run in the existing Neuron ESB runtime host. Users can change the hosting model by selecting a configured Neuron Endpoint Host in the Endpoint Host property dropdown box located on the General tab of the Adapter Endpoint.
To make deployments easier and more configurable at the environment level, all Adapter Endpoints now support the use of Environment Variables to disable/enable an Adapter Endpoint. The property exposed via the Bindings Expressions dialog is “Enable Adapter Endpoint”.
Neuron ESB 3.6 includes a new ServiceNow Adapter that provides access to an instance of ServiceNow. Using this adapter, organizations can easily integrate their business applications such as SAP, NetSuite or Marketo with ServiceNow. This adapter operates in both subscribe as well as solicit/response (or query) mode. The adapter utilizes the ServiceNow OAuth Provider to manage access to ServiceNow instances. Organizations can use the adapter to query for data, create, update, retrieve and delete entities and execute ServiceNow Functions and Actions. The adapter can be used to access the entire ServiceNow API. When developing applications that utilize this adapter, developers will be able to use the Adapter Metadata Generation Wizard to generate JSON samples that describe the request and/or response documents that are exchanged with ServiceNow. The Wizard can also generate Swagger documents, describing the APIs, which can be hosted by Neuron ESB Service Endpoints. Developers can also use the new Http Utility process step in our process designer to very easily define the calls to ServiceNow.
More information about the ServiceNow adapter can be found in the article posted here: https://www.neuronesb.com/resources/?post
Several other important modifications have been made to Neuron ESB Adapters that help simplify the configuration and deployment of secure solutions. The entire list of changes can be found in the change log section of this document.
Services and OAuth
Neuron ESB includes a Service Broker that enables organizations to deploy Neuron ESB as a Service Gateway, providing mediation, security, hosting and a number of other services. Service Connectors are essentially registrations within Neuron ESB that point to existing services (i.e. REST or SOAP APIs) hosted within an organization, partner or cloud domain. Client Connectors on the other hand are REST or SOAP APIs created using the Neuron ESB toolset and hosted by Neuron ESB. These are live services that can accept external requests, initiate business processing logic and/or broker to external services or applications.
In this release, Service Endpoints have been redesigned to use the same Messaging option as introduced to Adapter Endpoints. Users can choose to simply disable the Messaging option located on the respective tab of the Client Connector (i.e. publisher role) and Service Connector (i.e. subscriber role) of the Service Endpoint. Unlike Adapter Endpoints though, this is not limited to subscribe mode Service Endpoints (i.e. Service Connectors). Messaging can be disabled on Client Connectors as well.
Just as with Adapter Endpoints, subscribe side Service Endpoints (i.e. Service Connectors) can be executed directly within a Business Process or Workflow.
Because of their similarity with subscribe mode Adapter Endpoints, in Neuron ESB 3.6 we made it just as easy for developers to create Service Connectors that they intend to execute directly within our Business Process or Workflow. Users can now specifically choose to disable the Messaging option located on the Service Connector tab of the Service Endpoint. This allows the developer to create a Service Connector without needing to first create a Topic or Subscriber. At runtime, the Service Endpoint can be called directly within the Business Process or Workflow but will not be a “live” subscriber to messages published by other systems. The immediate benefits to this approach are identical to Adapter Endpoints:
- Reduced Complexity
- No need to create a Topic or Subscriber before configuring a Service Connector.
- Reduced Runtime Utilization
- The Service Connector does not need to be loaded in memory with its own resources. The Topic Publishing Service is never created.
- Faster Startup Time
- Since there is no subscribing Service Connector, the master runtime does not attempt to start it as a manageable endpoint.
- The Topic is also not created and loaded by the master runtime.
In this release, although Messaging can also be disabled for Client Connectors, it is for a different purpose than that of Service Connectors. Rather than disabling live hosting of the Service Endpoint, disabling Messaging allows the user to configure a Client Connector to call and execute Business Processes directly, without the need to configure the Client Connector with a Publisher and Topic.
In previous releases, users could always expose a Business Process as a web service (REST or SOAP) indirectly. The steps were simple, but like everything with multiple parts, complexity introduces itself. For example:
- Create a Topic
- Create a Publisher
- Create a Business Process
- Configure the Publisher to call the Business Process
- Create a Client Connector
- Configure the Client Connector to send the inbound message to the Publisher, which executes the Business Process and forwards to the Topic
Using Neuron ESB 3.6, we can reduce this to a 3 step task and eliminate unnecessary dependencies i.e.
- Create a Business Process
- Create a Client Connector
- Configure the Client Connector to call the Business Process
Selecting the Business Process option of the Processing Mode section of the Client Connector effectively disables the Messaging option and allows users to select one or more Business Processes to execute directly from the Service Endpoint UI.
Selecting the “Edit Processes” toolbar button, displays the same Processes dialog used to select Business Processes for Publishers and Subscribers.
Once a Client Connector is configured to execute Business Processes directly through the new Processing Mode option, developers will no longer need to create a Publisher or Topic as part of the configuration process. Additionally, when creating Business Processes, users can forego the placement of the Cancel Process Step as the last step in the Business Process when dealing with request/reply patterns. Client Connectors directly executing Business Processes incur other benefits as well including
- Reduced Complexity
- No need to create a Topic or Publisher before configuring a Client Connector.
- No mapping of Business Process to Publisher and Publisher to Client Connector
- No need to use a Cancel Process Step in Business Processes to return the reply message to the calling client or prevent a message from being forwarded to a Topic.
- Reduced Runtime Utilization
- The Topic Publishing Service is never created.
- Less processing and memory consumed by the Client Connector
- Faster Startup Time
- The Topic is not created and loaded by the master runtime.
- Better Performance
- All requests are no longer being brokered by a Publisher object
Client Connectors configured for direct execution of Business Processes should be the preferred mode of operation for users. This will generally provide simpler and more efficient execution of messages and Business Processes. Even though in this configuration there is no Publisher or Topic associated with an endpoint, messages received by the Client Connector can still be published to topics using the Publish Process Step within the Business Processes it’s executing. All Topic transports are supported. The only use case that is not supported is using MSMQ based topics when the Publish Step is configured with the Request Semantic.
Just as with Adapter Endpoints, in previous releases each Service Connector and Client Connector were isolated in their own .NET App Domain. They were hosted within the main Neuron ESB runtime process (i.e. esbservice.exe) with all other internal services, Messaging Publishing services and Adapter Endpoints.
In this release, Service Endpoints have been redesigned to use the new Neuron Endpoint Host model that Adapter Endpoints can now use. In Neuron ESB 3.6, all endpoints (Workflow, Service and Adapter) share the same hosting model. That means that both Client Connectors and Service Connectors can be configured to run using either the existing Neuron ESB runtime host (esbservice.exe) or the newer Endpoint Host for process isolation.
In the Neuron ESB 3.6 release, users can configure a Service Endpoint to run in either the existing Neuron ESB runtime host (as in previous versions) or the new Neuron Endpoint Host. The default behavior is for new Service Endpoints to run in the existing Neuron ESB runtime host. Users can change the hosting model by selecting a configured Neuron Endpoint Host in the Endpoint Host property dropdown box located on the General tab of the Service Endpoint.
The Neuron ESB 3.5 CU4 release introduced OAuth 2.0 support for authorizing web service invocations using Service Connectors (i.e. HTTP/REST API service endpoints) and supported Adapters. Using OAuth, Neuron ESB can obtain access tokens that can be passed to REST services to authorize protected resource invocations. In CU4, OAuth bearer tokens based on RFC 6750 are supported.
In the 3.6 release, we’ve extended this support, adding three new providers (ADP, ServiceNow and Thinktecture Identity Server) as well as the ability for users to create their own custom OAuth provider that can be re-used within the Neuron ESB Explorer. Below are the OAuth providers currently configurable within the Neuron ESB Explorer:
Custom OAuth Providers
Using Neuron ESB 3.6 users can develop their own OAuth providers to use with Neuron ESB endpoints. four different Grant Types are supported and several sample custom OAuth providers are now included within the Neuron ESB Sample Browser which can be modified and used as templates.
OAuth 2.0 Grant Types
When developing custom OAuth providers, the first step is to determine which grant type(s) must be supported. There are five grants for obtaining an access token:
- Authorization Code Grant
- Implicit Grant
- Client Credentials Grant
- Resource Owner Credentials Grant
- Refresh Token Grant
Each of these except for the Implicit Grant type can be used by Neuron ESB. The Implicit Grant type is intended to be used by user-agent-based clients, like single page web apps that are unable to keep a client secret. Below are descriptions of the other four grants and how Neuron ESB utilizes them.
Developing Custom OAuth Providers
To develop a custom OAuth provider, either two or three classes must be implemented, depending on which OAuth grant type required:
- A Provider class which inherits from Neuron.Esb.OAuth.OAuthProvider
- An OAuth Client class, which inherits from either Nemiro.OAuth.OAuth2Client (OAuth 2.0) or Nemiro.OAuth.OAuthClient (OAuth 1.0)
- An optional Login class, which inherits from Nemiro.OAuth.LoginForms.Login. This is only needed for the Authorization Code grant type.
These are the steps for creating an OAuth provider:
- Determine which grant type is needed to use with the OAuth provider.
- Determine provider-specific requirements for the grant type that must be used
- Build the OAuth Provider solution using Microsoft Visual Studio
Compile and Deploy
Once the custom OAuth provider solution is compiled, the .NET assembly must be copied/deployed to the following folder:
<Neuron Install Location>\<Instance>\OAuthProviders
To configure the custom OAuth provider, close and reopen the Neuron ESB Explorer and navigate to Security->Authentication->OAuth Providers. Click on the “New” button and select the custom OAuth provider from the Provider drop-down list.
More information about creating custom OAuth providers can be found in the article posted here: https://www.neuronesb.com/resources/?post
To make deployments easier and more configurable at the environment level, all Service Endpoint, Client Connectors and Service Connectors now support the use of Environment Variables to disable/enable their respective function. The properties exposed via the Bindings Expressions dialog are:
- Enable Service Endpoint
- Enable Service Connector
- Enable Client Connector
Binding Expressions dialog – Used to map Environment Variables to Service Endpoint properties. This is displayed by selecting the Bindings toolbar button on the Service Endpoint.
Migration from 3.5
This release introduces significant changes in Neuron ESB’s runtime environment and infrastructure. In consideration of this, users must upgrade elements of any existing solutions before attempting to run them using Neuron ESB 3.6. Other changes may be necessary if users have modified or introduced custom settings in Neuron ESB’s native .config files, specifically in the AppSettings section. Overall, the migration should be straight forward for most users. One of the first things a user must do is update any existing solutions by opening them directly in the new version of the Neuron ESB Explorer.
When migrating from previous versions, users must open their existing solution using the 3.6 version of the Neuron ESB Explorer. The Neuron ESB Explorer will automatically update their solution by doing the following:
- Create a new Endpoint Host named “Neuron ESB Default Host”
- Assign new Endpoint Host to all existing Deployment Groups.
- Assign all existing Adapter Endpoints to new Endpoint Host.
- Assign all existing Service Endpoints to new Endpoint Host.
- Update the internal Adapter Name property (not the Adapter Endpoint Name) of all existing Adapter Endpoints so that they properly map to all previously registered Adapters
- Remove all previously registered Adapters from the Adapter Registration section of the Neuron ESB Explorer
- Any Adapter Endpoints whose property names match either “Password”, “Connectionstring”, “key” or “passphrase” will have their respective values encrypted.
Once the updates are made, the user will be prompted with the Review Changes dialog similar to the one shown below, but specific to their solution.
The changes should be accepted and the solution should be saved for the changes to take affect and be committed.
Every Neuron ESB solution requires a Microsoft SQL Server database to be associated with each Deployment Group. These databases can be created and configured by opening the solution within the Neuron ESB Explorer and navigating to Deployment->Manage->Databases.
In this release, the data type for the Sequence ID associated with the Audit tables has been changed from an int to a long. This requires users to update any existing databases used in Neuron ESB solutions. This can be done by simply clicking the “Test/Create” located on the database user interface within the Neuron ESB Explorer. Users can optionally run the SQL update scripts using the PowerShell script located in the installation folder of Neuron ESB. More information on updating the database can be found under the “Upgrading an Existing Database” section within in the Readme.html file located at the root of the Neuron ESB installation folder.
If there are a large amount of records in the Audit tables, this update could take a significant amount of time. It’s strongly recommended that users first backup and purge the Audit Tables. This can be done either manually or by using the PurgeArchiveNeuron Microsoft SQL Agent Job. More information on updating the database can be found under the “Archive and Purge Neuron Database Job” section within in the Readme.html file located at the root of the Neuron ESB installation folder.
In Neuron ESB, many of the monitoring and server management functions were exposed through REST API based interfaces. These interfaces can be used to extend and build custom monitoring and management solutions for Neuron ESB Deployments. The Neuron ESB REST interfaces are hosted by the Neuron ESB Discovery service (DiscoveryService.exe) which is installed with the Neuron ESB Server Runtime. The default port (51002) for the REST interfaces is configurable through the Neuron ESB Discovery service’s app.config. The default installation location of the Neuron ESB Discovery service is: “C:\Program Files (x86)\Neudesic\Neuron ESB v3”.
In previous versions, the Swagger documentation for the REST API could be browsed at the default URL: http://localhost:51002/help/ui/index.html. In Neuron ESB 3.6 the default URL has been changed to: http://localhost:51002/help/index.
The REST interfaces are self-documenting, providing detailed instructions on how to call each specific function and are separated into several categories. In Neuron ESB 3.6 a new category, “Configuration”, was added to support its integration with the future platforms. The following categories of REST APIs are listed below:
- Endpoint Health
Neuron ESB is essentially a .NET application and as such uses its associated .config file to maintain runtime configuration settings. In previous versions, these settings were maintained in the appSettings section within the following two files
These files are installed in the following default location: C:\Program Files\Neudesic\Neuron ESB v3\<Instance>.
Many of the settings are maintained and modified by using the Configure Server dialog launched from the Neuron ESB Explorer, while others can be set manually by editing the respective .config files.
In the Neuron ESB 3.6 release, we’ve consolidated the appSetting entries into a dedicated .config file named “appSettings.config” located in the same default location as all the other .config files. The following .config files now contain a reference to this new file.
Additionally, several new key entries were added to facilitate integration with our future platforms. If users have added their own custom key/values to the existing Neuron ESB .config files, these will need to be migrated to the new appSettings.config file.
The table below lists all the appSetting key/values used in Neuron ESB 3.6
|esbInstanceName||Name of Neuron ESB runtime Instance||This is the name of the Neuron ESB runtime instance set by the installer and visible within the Service Control Manager|
|esbEnvironment||Full path of solution folder||X||Set by the Configure Server dialog. Full path of solution folder|
|esbZone||Enterprise||X||Set by the Configure Server dialog. Should always be Enterprise.|
|esbDeploymentGroup||Active Deployment Group||X||Set by the Configure Server dialog. Active Deployment Group that the runtime should load on startup|
|overrideDotNetThreadPoolSize||False||X||Set by the Configure Server dialog. Performance tab. Whether or not to override .NET thread pool for Neuron ESB runtime.|
|dotNetMaxWorkerThreads||500||X||Set by the Configure Server dialog. Performance tab. Used only if Override is enabled.|
|dotNetMaxIOThreads||1000||X||Set by the Configure Server dialog. Performance tab. Used only if Override is enabled.|
|dotNetMinWorkerThreads||50||X||Set by the Configure Server dialog. Performance tab. Used only if Override is enabled.|
|dotNetMinIOThreads||100||X||Set by the Configure Server dialog. Performance tab. Used only if Override is enabled.|
|MaximumLogFileSize||104857600||X||Set by the Configure Server dialog. Logging tab. Default is 100 MB|
|MinimumDiskSpaceThreshold||10485760||X||Set by the Configure Server dialog. Logging tab. Default is 10 MB|
|LogFileSchedule||Daily||X||Set by the Configure Server dialog. Logging tab. Default is Daily. Can be set to Weekly or None. Determines when a new Log file will be created for an endpoint.|
|StartupTimeout||00:10:00||Maximum amount of time the Neuron ESB runtime is allowed to take to start up all internal services and endpoints. Timespan value|
|ConfigurationReloadDelay||00:01:00||The time between the detection and processing of Configuration change events. Timespan value|
|WSDiscoveryEnabled||true||Enables WS-Discovery on all Client Connectors, Neuron ESB Runtime as well as hosted Parties. When enabled, WS-Discovery can accept/respond to probe request, returning information and status of these endpoints.|
|WSDiscoveryBroadcast||true||Enables the Neuron ESB Runtime to broadcast WS-Discovery Hello/Goodbye messages. WSDiscoveryEnabled must also be set to true for broadcasting to be enabled.|
|WSDiscoveryAnnouncementInterval||00:00:10||Determines interval for WS-Discovery broadcasts. Timespan value.|
|WSDiscoveryApplicationScope||default||WS-Discovery scope used in announcements. Recommended that is should be the value of the esbInstanceName key.|
|DiscoverySwaggerPort||51003||Default Port used to host Swagger documents when Swagger is configured for a REST based Client Connector|
|DefaultHttpConnectionLimit||100||Used to set the value for the ServicePointManager.DefaultConnectionLimit property. Used for all HTTP service calls.|
|ClusterNetworkName||Empty String||This is set by the Neuron ESB runtime at startup. If Neuron ESB is installed on a Windows Failover Cluster as resource, this value will be the network name associated with the cluster resource. Otherwise it will be the machine name|
|MaxHeartbeatFailures||2||Max number of heartbeat failures against the Neuron ESB runtime master control service allowed before a Client Connector, Service Connector or Adapter Endpoint will shut down.|
|esbUseMachineEnvironmentVars||false||If set to true, allows users to use Windows Operating System Environment Variables within the Neuron ESB Explorer’s Binding Expressions dialog.|
|XsltFilesLocation||X||Empty String||Used by the Transform-XSLT Process step and the Transform Message/XML Workflow Activities. Defines a global location to retrieve XSLT files, rather than using the Neuron ESB repository location. This is a Global override.|
|XsdFilesLocation||X||Empty String||Used by the Validate-Schema Process step and Validate XML Workflow Activity. Defines a global location to retrieve XSD files, rather than using the Neuron ESB repository location. This is a Global override.|
|LogFolderCleanupSchedule||X||10||Number of days of log files that Neuron should not delete/remove at startup. Any log file older than the value will be deleted. Must be greater than 1.|
|SavedHistoryCleanupSchedule||X||0||Number of history files to retain in the solution’s History folder. Executes whenever a save is committed on configuration. A value of Zero means unlimited.|
|TempDirectory||X||Empty String||If a dedicated TEMP directory is not provided, the default TEMP directory of the service account will be used, consistent with 3.5 behavior.|
|DisableGlobalExceptionHandler||X||true||Enables/Disables the Neuron ESB Global Exception handler. When troubleshooting an issue, this should be set to false.|
|NeuronEndpointHostPort||X||51004||Communication port used by the Neuron Endpoint Host REST API. Commands between the Neuron ESB runtime and Endpoint Host are brokered across this port.|
|NeuronHybridFeaturesEnabled||X||false||X||Set by the Configure Server dialog. Server tab. This value represents the “Enable Neuron Event Processor” property. Can only be enabled if the Neuron ESB Instance has been registered within the Neuron Management Suite Portal.|
|NeuronHybridManagementSuiteHost||X||localhost||Location of the Neuron Management Suite Portal. Set by the portal during registration process|
|NeuronEventProcessorMachine||X||localhost||Location of the Neuron Event Processor. Set by the portal during registration process|
|NeuronEventProcessorPort||X||51005||Port configured for the Neuron Event Processor API. Set by the portal during registration process|
|NeuronServiceRateFlushInterval||X||5||Configuration specific to the Neuron Event Processor. Set by the portal during registration process|
|NeuronServiceRateCacheSize||X||10000||Configuration specific to the Neuron Event Processor. Set by the portal during registration process|
|NeuronEventConsumerLevel||X||1||Configuration specific to the Neuron Event Processor. Set by the portal during registration process|
|NeuronEventTimeToLive||X||1440||Configuration specific to the Neuron Event Processor. Set by the portal during registration process|
|NeuronEventPoolSizeInit||X||0||Configuration specific to the Neuron Event Processor. Set by the portal during registration process|
|NeuronEventPoolSizeMax||X||200||Configuration specific to the Neuron Event Processor. Set by the portal during registration process|
Neuron ESB is entirely built on the Microsoft .NET Framework. This makes it very easy for .NET developers to learn and extend the product using common C# skills. With the release of Neuron ESB 3.6 we took the opportunity to upgrade our support for version 4.7.1 of the Microsoft .NET Framework. This allows developers to take advantage of the new features included in the framework when building any extensibility as part of their Neuron ESB solution such as adapters, workflow activities, process steps and OAuth providers. This support is also provided through our own C# editors within the Business Process and Workflow designers.
Additionally, we upgraded our support for a number of other dependent libraries, including the following:
- Log4Net from version 2.0.3 to 2.0.8
- RabbitMQ Client version 4.1.3 to 5.0.1
- Newtonsoft Json .NET version 8.0.2 to 11.0.1
Lastly, Neuron ESB 3.6 has been upgraded to support Erlang version 20.2 and Rabbit MQ version 3.7.1. Installation of RabbitMQ Server is optional and is not required to use Neuron ESB 3.6. If RabbitMQ Server is not installed, the Neuron ESB 3.6 installer will provide users the option of automatically downloading and installing Erlang and RabbitMQ Server. If a developer wants to use RabbitMQ based Topics, but the computer does not have Internet access, the following software packages can be downloaded and installed either before or after installing Neuron ESB 3.6.
One of the great things about Neuron ESB is its ability to auto detect changes and, based on those changes, re-configure itself. Part of that process involves dynamically compiling .NET assemblies from the changes detected. For instance, if a Business Process is modified, Neuron ESB will auto compile that Business Process and its respective C# code (if any) into a .NET assembly before it can be executed. This process also happens after the Neuron ESB runtime is initially started. Before the first execution of any Business Process, it is dynamically compiled. All other consecutive executions then run from the compiled assembly. Dynamic compilation only happens before the first execution of a Business Process or modified Business Process.
For this to happen, heavy use is made of the Windows TEMP directory of the service account assigned to the Neuron ESB runtime service (esbservice.exe) within the Service Control Manager. For instance, given a service account named “NeuronSvc”, the Windows TEMP directory location would be: “C:\Users\NeuronSvc\AppData\Local\Temp”. Due to our heavy use of this directory, Neuron ESB attempts to delete all the temporary files it generates during the startup of the Neuron ESB runtime service. Because Windows directories can only hold up to 65535 files, it’s important that the TEMP directory location be maintained and regularly cleared out.
However, outside processes (other than the Neuron ESB runtime) can negatively affect a running Neuron ESB solution by either placing locks on the temporary files Neuron ESB generates or, deleting those files. In other cases third party processes may also be using the same TEMP directory that Neuron ESB is configured to use (i.e. where a dedicated account is not used), generating files that exceed the maximum limit of the directory. Lastly, anti-virus programs are notorious for scanning the TEMP directories of user accounts as they are commonly exploited by viruses. Unfortunately, many tend to lock files as they scan.
To avoid these issues, Neuron ESB 3.6 can be configured to use any Windows directory as a TEMP directory. The location can be specified in a new appSettings key (i.e. TempDirectory) located within the appSettings.config file, the default location of which is: C:\Program Files\Neudesic\Neuron ESB v3\<Instance>. If a dedicated TEMP directory is not configured in the appSettings section, the default TEMP directory of the service account will be used as in previous versions of Neuron ESB. In all cases, the service account must be given full rights to the TEMP directory.
Using a dedicated TEMP directory can assist to eliminate the ability for third party processes to generate temporary files in the location that Neuron ESB uses. Also, Security protocols will usually allow the exclusion of an application specific folder from Anti-Virus scanning.
Neuron ESB provides the ability to use Environment Variables for configuration of any adapter or service endpoint, database connection string or process step. By using the Bindings Expression editor, users can bind Environment Variables directly to the properties exposed by those entities. Many developers access the values of Environment Variables directly in Code Process steps within processes or workflow. An Environment Variable is defined at the Deployment Group level and can contain a unique value for each Deployment Group. On startup, the Neuron ESB service runtime replaces the Environment Variable with the value associated with the Deployment Group the service is configured for. To learn more about Environment Variables please review the Using Environment Variables section within the Neuron ESB Business Process documentation.
By default, all Environment Variable values are stored encrypted within the solution. However, those same values remain human readable in the Neuron ESB Explorer. In this release, we’ve introduced the ability to “mask” all the values of an Environment Variable. A check box, “Mask Values” has been added to the General Tab of the Environment Variable user interface and is disabled by default. If enabled, all the values (current and future) for that specific Environment Variable will no longer be visible. Each character will be masked by an asterisk (“*”). Once enabled, it cannot be disabled.
Windows OS level Environment Variables
One last thing was modified regarding the use of Windows OS level Environment Variables. In Neuron ESB 3.5, support was added for using Windows Operating System Environment Variables within the Neuron ESB Explorer’s Binding Expressions dialog. Previously, users could create Neuron ESB based Environment Variables to set most any property of most any Neuron ESB Entity, like Service Connectors, Client Connectors and Adapter Endpoints. The specific values of the Environment Variables are linked to Deployment Groups within the Neuron ESB Explorer. Hence the values used in one Deployment Group could be different than those used in another. The Neuron ESB runtime resolves these values dynamically at runtime on startup.
This new feature allowed users to enter the name of environment variables created at the machine’s Operating System level within the Binding Expressions dialog, rather than use Neuron ESB’s built in Environment Variables. There are a few requirements to use this:
- Environment Variables had to be set at the System level. This could be done using the Windows Operating System’s UI tool, or using the Windows command line tool, “SETX” followed by the “/M” switch
- Operating System level Environment Variables that are used within Neuron ESB must be preceded with “Neuron_”. For example, users could create an Environment Variable named “Neuron_myUrl” using the following command line:
3. Within the Neuron ESB Explorer’s Binding Expression dialog, the Environment Variable must be enclosed in the Neuron ESB bracket notation as shown below:
4. Lastly, the following flag must be added the AppSettings group within the esbservice.exe.config file: <add key=”esbUseMachineEnvironmentVars” value=”true”/>
However, because these were at the system level, this required that the Environment Variables had to be created AFTER the process is started. This was problematic when hosting Neuron ESB within Docker containers. In Neuron ESB 3.6, this has been changed to retrieve values stored at the Machine level.
MOD – Azure Service Bus Adapter – The Asynchronous Subscribe option has been removed
MOD – Azure Service Bus Adapter – The following dependent libraries have been updated:
WindowsAzure.ServiceBus from version 220.127.116.11 to 4.1.5
Microsoft.WindowsAzure.ConfigurationManager from version 2.0.3 to 323
MOD – Dropbox Adapter – The following dependent libraries have been updated:
Dropbox.Api from version 3.0.5 to 3.5.0
MOD – RabbitMQ Adapter – The following dependent libraries have been updated:
RabbitMQ.Client from version 4.1.3 to 5.0.1
MOD – FTP/FTPS Adapter – The following dependent libraries have been updated:
Xceed libraries updated from version 5.8 to 6.5
MOD – SFTP Adapter – The following dependent libraries have been updated:
Xceed libraries updated from version 5.8 to 6.5
NEW – Proxy Server Support – This has been added as a configurable set of properties to the following adapters:
- Microsoft Dynamics AX
NEW – Microsoft Dynamics AX – The Transfer Mode property has been added allowing the user to choose between buffered or streamed.
NEW – NetSuite – Support for 2018_1 SOAP Version of NetSuite API has been added
NEW – ODBC Adapter – The proper message samples and schemas can now be obtained using the Adapter Metadata Generation Wizard.
NEW – SalesForce Adapter – Support for versions 39 through 41 of the SOAP API has been added
MOD – SAP Adapter – The following dependent libraries have been updated:
ERPConnect45 from version 18.104.22.168 to 22.214.171.124
NEW – SharePoint Adapter – Site Collections and Lists can now be dynamically set at runtime using the following Neuron ESB Message Context Properties:
NEW – Encrypted Storage of Sensitive Properties – The following property values will always be stored encrypted:
MOD – Compression Process Step – The following dependent libraries have been updated:
Xceed libraries updated from version 5.8 to 6.5
NEW – Validate – JSON Process Step – This is a new Business Process Step that uses the Newtonsoft JSON Schema library for validation of JSON formatted messages. JSON schemas can be entered directly into the Process Step, or retrieved from the new JSON Schema repository (similar to the Validate–Schema Process Step). Like the Validate–Schema Process Step, it has a Valid and Invalid branch. This uses version 3.0.3 of the Newtonsoft.Json.Schema library. Additionally, like the Validate-Schema Process Step, the JSON Schema or the name of the Schema to retrieve from the JSON Schema repository can be dynamically configured at runtime by setting the following custom Neuron ESB Message properties:
FIX – Business Process Test Client – When selecting the Load File option and an XML file is selected with a BOM character, the BOM would be loaded as well. This is now being removed.
FIX – Business Process Test Client – This only occurred when testing with a JSON body and the bodytype property is changed to something other than ‘text/xml’. After the user opened the Test Client after the first test run, the text editor where the JSON message should be would be empty.
FIX – HTTP Client Utility – If a URL was not resolvable or could not be parsed, we would throw a null exception
MOD – JSON Process Step – Has been moved to the JSON folder within the Business Process Step Toolbox
MOD – Flat File Parser, Excel to Xml – These have been moved to the XML folder within the Business
MOD – Audit Process Step – If pools size equals 1, we reuse an instance of the audit proxy object, rather than retrieve from pool.
MOD – Service Endpoint Process Step – If pools size equals 1, we reuse an instance of the service proxy, rather than retrieve from pool.
NEW – Validate-Schema Process Step – A new property, “Referenced Schema Location” has been added that allow users to configure the Process Step to retrieve schemas from an external file share. If a schema contains any import or include directives, this is the location they will be searched for, if only a filename or a relative path is provided as the SchemaLocation in the schema. If this is left blank, the schema location will be resolved based on the Neuron ESB Instance that is hosting this solution as in previous solutions. This property can be overridden globally by setting the appSetting key, “XsdFilesLocation”, to the location.
NEW – Transform XSLT Process Step – A new property, “Referenced Schema Location” has been added that allow users to configure the Process Step to retrieve XSLTs from an external file share. If this xslt contains any import directives, this is the location they will be searched for when only a filename or a relative path is provided as the SchemaLocation in the xslt. If this is left blank, the xslt location will be resolved based on the Neuron ESB Instance that is hosting this solution as in previous solutions. This property can be overridden globally by setting the appSetting key, “XsltFilesLocation”, to the location.
MOD – Audit Workflow Activity – When used within the Catch Block of a Try/Catch and the Action property is set to Failure, the Audit step will automatically get the Exception object in the catch block rather than requiring it to be manually entered in the Exception property.
NEW – Validate XML Workflow Activity – A new property, “Referenced Schema Location” has been added that allow users to configure the Workflow Activity to retrieve schemas from an external file share. If a schema contains any import or include directives, this is the location they will be searched for, if only a filename or a relative path is provided as the SchemaLocation in the schema. If this is left blank, the schema location will be resolved based on the Neuron ESB Instance that is hosting this solution as in previous solutions. This property can be overridden globally by setting the appSetting key, “XsdFilesLocation”, to the location.
NEW – Transform Message/XML Workflow Activities – A new property, “Referenced Schema Location” has been added that allow users to configure the Workflow Activity to retrieve XSLTs from an external file share. If this xslt contains any import directives, this is the location they will be searched for when only a filename or a relative path is provided as the SchemaLocation in the xslt. If this is left blank, the xslt location will be resolved based on the Neuron ESB Instance that is hosting this solution as in previous solutions. This property can be overridden globally by setting the appSetting key, “XsltFilesLocation”, to the location.
MOD – Import program – This was modified to work with Endpoint Hosts
FIX – Decryption Error on Load – If a decryption error occurred during a user attempt to load a solution in Neuron ESB explorer, a second attempt to load the same solution would cause the Neuron ESB Explorer to hang.
MOD – Neuron ESB Explorer making unnecessary network calls – The Neuron ESB Explorer was calling the Discovery service to retrieve status as well as the entire solution configuration twice a second. This is no longer a service call and it is no longer retrieving the solution configuration.
MOD – Certificates – When selecting and saving Certificates, users can no longer search by “find subject by name”. This could not guarantee uniqueness when the runtime must use the criteria to retrieve the correct criteria. Users should use “find by thumbprint”.
MOD – Bindings Expression Dialog – All binding properties are now displayed alpha sorted.
FIX – JSON Document Repository – If a document was modified or changed, the Apply button would not become enabled.
Neuron ESB Runtime
MOD – Environment Variables – Previously, when the Windows OS Environment Variables were enabled (by setting the esbUseMachineEnvironmentVars appSettings flag to true), Neuron would use user level Environment Variables. This has been changed to Machine level.
MOD – Global Exception Handling – This is now turned off by default using the DisableGlobalExceptionHandler appSettings flag. Users should turn this on when troubleshooting issues to ensure all exceptions are recorded.