Monday, August 20, 2018

WSO2 ESB Clustering


Why do we need a clustered Application/Environment?

Deploying the artifacts in a single server may completely unusable if that server goes down or unavailable.The huge risk is there due to the probability of single point of failure.
When designing a deployment architecture for a software system, two things need to be concerned as a high importance.
  •  Availability
  •  Scalability
What is availability?

How well the system provides there functionality in the given set of time period without any service failures.



What is scalability?

It is a property of what amount of work the system or application can be handled and easily be expanded on demand.
There are two different aspects of scalability.



Vertical scalability

Scaling up by adding more processing power to the existing systems to cater more requests.




Horizontal scalability

Expanding the processing ability of the system by adding new nodes parallelly which have identical functionality and distribute the incoming traffic among those nodes via a load balancer.



Note: Clustering will addressed the above concepts of availability and scalability.

Challenges in adding multiple nodes to the clustered system


  • Keeping the identical behavior in all the nodes(Replicate the changes among the nodes).
  • Managing the better communication among the servers.
Active-Active & Active-Passive Clusters
  • Active-Active - All the nodes of the cluster are in active mode. In that case the load will balance among all the nodes.
  • Active-Passive - Some of the nodes are act as backup nodes.If Active nodes are down, the passive nodes are activate and handle the load.

Multicast vs Unicast(WKA) Clusters

A cluster can be created in the Runtime Manager from servers that are already registered with the Runtime Manager console. While configuring a cluster, you can specify either unicast or multicast options for identifying a node within the cluster:
  • Unicast: A unicast cluster requires that you configure the IP addresses of which nodes to associate together. No special network configuration is necessary other than to indicate which are the server IP addresses that make up the cluster.
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22Node1%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23fff2cc%3BstrokeColor%3D%23d6b656%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22160%22%20y%3D%22180%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%223%22%20value%3D%22%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23f8cecc%3BstrokeColor%3D%23b85450%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22334%22%20y%3D%22187%22%20width%3D%2246%22%20height%3D%2246%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%224%22%20value%3D%22Multicast%20Channel%22%20style%3D%22text%3Bhtml%3D1%3Bresizable%3D0%3Bpoints%3D%5B%5D%3Bautosize%3D1%3Balign%3Dleft%3BverticalAlign%3Dtop%3BspacingTop%3D-4%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22296%22%20y%3D%22245%22%20width%3D%22110%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%225%22%20value%3D%22Node4%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23d5e8d4%3BstrokeColor%3D%2382b366%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22470%22%20y%3D%22250%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%226%22%20value%3D%22Node3%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23d5e8d4%3BstrokeColor%3D%2382b366%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22470%22%20y%3D%22180%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%227%22%20value%3D%22Node2%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23d5e8d4%3BstrokeColor%3D%2382b366%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22470%22%20y%3D%22110%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%228%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BexitX%3D1%3BexitY%3D0.5%3B%22%20edge%3D%221%22%20source%3D%222%22%20target%3D%223%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22240%22%20y%3D%22130%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22290%22%20y%3D%2280%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%229%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22380%22%20y%3D%22210%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22470.5%22%20y%3D%22139.5%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2210%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22380%22%20y%3D%22210%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22474%22%20y%3D%22210%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2211%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22380%22%20y%3D%22210%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22468%22%20y%3D%22279.5%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E



%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22Node1%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23fff2cc%3BstrokeColor%3D%23d6b656%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22160%22%20y%3D%22180%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%223%22%20value%3D%22%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23f8cecc%3BstrokeColor%3D%23b85450%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22334%22%20y%3D%22187%22%20width%3D%2246%22%20height%3D%2246%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%224%22%20value%3D%22Multicast%20Channel%22%20style%3D%22text%3Bhtml%3D1%3Bresizable%3D0%3Bpoints%3D%5B%5D%3Bautosize%3D1%3Balign%3Dleft%3BverticalAlign%3Dtop%3BspacingTop%3D-4%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22296%22%20y%3D%22245%22%20width%3D%22110%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%225%22%20value%3D%22Node4%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23d5e8d4%3BstrokeColor%3D%2382b366%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22470%22%20y%3D%22250%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%226%22%20value%3D%22Node3%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23d5e8d4%3BstrokeColor%3D%2382b366%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22470%22%20y%3D%22180%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%227%22%20value%3D%22Node2%22%20style%3D%22ellipse%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3Baspect%3Dfixed%3BfillColor%3D%23d5e8d4%3BstrokeColor%3D%2382b366%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22470%22%20y%3D%22110%22%20width%3D%2260%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%228%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BexitX%3D1%3BexitY%3D0.5%3B%22%20edge%3D%221%22%20source%3D%222%22%20target%3D%223%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22240%22%20y%3D%22130%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22290%22%20y%3D%2280%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%229%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22380%22%20y%3D%22210%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22470.5%22%20y%3D%22139.5%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2210%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22380%22%20y%3D%22210%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22474%22%20y%3D%22210%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2211%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22380%22%20y%3D%22210%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22468%22%20y%3D%22279.5%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E

  • Multicast: A multicast cluster groups servers that automatically detect each other. Servers that are part of a multicast cluster should be on the same network segment.One advantage of using multicast is that a server does not need to be running to be configured as a node in a cluster.


WSO2 ESB clustering concepts


1. Cluster Nodes
  • Manger Node- Manager Node is responsible to do the management operations of the cluster.Generally the manager node is not handle the work load.
  • Worker Node- Worker node is handling the work load in the clustered environment.It is not allowed to provide any management operations. If multiple worker nodes are there, the work load will be balance among them via a load balancer.

Note: In the ESB clustered environment, we are not allowed to access the management console of the worker nodes and only allow in manager node.

2. Database Setup

When database setup in wso2 esb clustered environment, we need to make our attention on following databases.
  • User Management Db- Keep the information of ESB users
  • Registry Db - There are three major registries maintained by WSO2 ESB.

Local Registry -Stores configuration and run time data that is local to the server.This partition is not shared with multiple servers and can be browsed under /_system/local in the registry browser.

Configuration Registry -Stores product-specific configurations. This partition can be shared across multiple instances of the same product, such as sharing ESB configuration across a ESB cluster, and can be browsed under /_system/config in the registry browser.

Governance Registry -Stores configurations and data that are shared across the whole platform. This space typically includes services, service descriptions, endpoints, and data sources, and it can be browsed under /_system/governance in the registry browser.


In clustered environment, the following database structure should be maintained.
  • Separate databases for local registry in each node.
  • Shared database for user management.
  • Shared database for config and governance registry.

3. Deployment Synchronization

Deployment Synchronizer provides capability to synchronize deployment artifacts across the nodes of a product cluster. The following scenarios will be covered by Dep Sync in ESB cluster.
  • Maintaining an up-to-date backup of the artifact repository.
  • Sharing a single artifact repository among multiple servers in a cluster.
  • Enforcing artifact updates to be deployed on a server at run time.

 There are two different ways of deployment synchronization.

  • SVN based Deployment Synchronizer- Use check in and check out functionality of SVN repository.




  • rsync for deployment Synchronization- Use rsync tool for copying files.



Deployment diagram




Create SVN Repository

For the purpose of deployment synchronization, here we are using SVN based synchronizer and we can locally setup it as below.

1.Create SVN repository.

 svnadmin create <repostitory name>;  

2.Add userName and password properties to the <repository name>/conf/passwd file as below.

 ### This file is an example password file for svnserve.  
 ### Its format is similar to that of svnserve.conf. As shown in the  
 ### example below it contains one section labelled [users].  
 ### The name and password for each user follow, one account per line.  
 [users]  
 randika = randika123  

Set the passwd file as password-db in <repository name>/conf/svnserver.conf file as below.

 ### users have read and write access to the repository.  
 anon-access = write  
 # auth-access = write  
 ### The password-db option controls the location of the password  
 ### database file. Unless you specify a path starting with a /,  
 ### the file's location is relative to the directory containing  
 ### this configuration file.  
 ### If SASL is enabled (see below), this file will NOT be used.  
 ### Uncomment the line below to use the default password file.  
 password-db = passwd  


Setting up Databases

Need to create two shared databases for registry(conf and gov) and user management.

 create user db  
 mysql> create database wso2_user_db;  
 mysql> use wso2_user_db;  
 mysql> source /home/engineer/Documents/mitrai/cluster/manager/dbscripts/mysql.sql  
 create registry db for shared config and governance  
 mysql> create database wso2_reg_db;  
 mysql> use wso2_reg_db;  
 mysql> source /home/engineer/Documents/mitrai/cluster/manager/dbscripts/mysql.sql;  

note: Here we are using mysql database and need to copy the jdbc driver .jar to the repository/components/lib directory.


Configuring wso2 ESB Cluster(manger with 2 workers)

In each node of the cluster, we need to change the configurations in below files.
  • axis2.xml- Enable the clustering, membership scheme and cluster node configurations.
  • carbon.xml- Deployment synchronization related configurations.
  • registry.xml- Registry mounting configurations.
  • user-mgt.xml- Change User management database configurations.
  • master-datasources.xml- Configurations for shared databases(Registry and User-management )

Configurations of Manager node

axis2.xml


 <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"  
         enable="true">  
     <parameter name="clusteringPattern">WorkerManager</parameter>  
     <parameter name="AvoidInitiation">true</parameter>  
      <parameter name="membershipScheme">wka</parameter>  
      <parameter name="domain">wso2.carbon.domain</parameter>  
      <!-- The host name or IP address of this member -->  
     <parameter name="localMemberHost">172.20.0.1</parameter>  
     <!--  
     The TCP port used by this member. This is the port through which other nodes will  
     contact this member  
      -->  
     <parameter name="localMemberPort">4000</parameter>  
     <!--  
     Properties specific to this member  
     -->  
     <parameter name="properties">  
       <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>  
       <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>  
         <!-- Manger Setup with Port Mapping-->  
       <property name="subDomain" value="mgt"/>  
        <!-- Worker Setup-->  
        <!--property name="subDomain" value="worker"/-->  
     </parameter>  
     <!--  
       The list of static or well-known members. These entries will only be valid if the  
       "membershipScheme" above is set to "wka"  
     -->  
     <members>  
       <member>  
         <hostName>172.20.0.1</hostName>  
         <port>4100</port>  
       </member>  
        <member>  
         <hostName>172.20.0.1</hostName>  
         <port>4200</port>  
       </member>  
     </members>  
     <!--  
     Enable the groupManagement entry if you need to run this node as a cluster manager.  
     Multiple application domains with different GroupManagementAgent implementations  
     can be defined in this section.  
     -->  
     <groupManagement enable="false">  
       <applicationDomain name="wso2.esb.domain"  
                 description="ESB group"  
                 agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"  
                 subDomain="worker"  
                 port="2222"/>  
     </groupManagement>  
   </clustering>  

carbon.xml

     <Offset>1</Offset>  

 <DeploymentSynchronizer>  
     <Enabled>true</Enabled>  
     <AutoCommit>true</AutoCommit>  
     <AutoCheckout>true</AutoCheckout>  
     <RepositoryType>svn</RepositoryType>  
     <SvnUrl>file:///home/engineer/Documents/mitrai/repo/</SvnUrl>  
     <SvnUser>randika</SvnUser>  
     <SvnPassword>randika123</SvnPassword>  
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>  
   </DeploymentSynchronizer>  


master-datasources.xml

 <datasource>  
       <name>WSO2_REG_DB</name>  
                <description>The datasource used for shared config and governance registry</description>  
                <jndiConfig>  
                     <name>jdbc/WSO2SharedDB</name>  
                </jndiConfig>  
                <definition type="RDBMS">  
                <configuration>  
                     <url>jdbc:mysql://localhost:3306/wso2_reg_db</url>  
                     <username>root</username>  
                     <password>root</password>  
                     <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
                     <maxActive>50</maxActive>  
                     <maxWait>60000</maxWait>  
                     <testOnBorrow>true</testOnBorrow>  
                     <validationQuery>SELECT 1</validationQuery>  
                     <validationInterval>30000</validationInterval>  
                </configuration>  
                </definition>  
     </datasource>  
     <datasource>  
       <name>WSO2_USER_DB</name>  
                <description>The datasource used for shared user db</description>  
                <jndiConfig>  
                     <name>jdbc/WSO2UserDB</name>  
                </jndiConfig>  
                <definition type="RDBMS">  
                <configuration>  
                     <url>jdbc:mysql://localhost:3306/wso2_user_db</url>  
                     <username>root</username>  
                     <password>root</password>  
                     <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
                     <maxActive>50</maxActive>  
                     <maxWait>60000</maxWait>  
                     <testOnBorrow>true</testOnBorrow>  
                     <validationQuery>SELECT 1</validationQuery>  
                     <validationInterval>30000</validationInterval>  
                </configuration>  
                </definition>  
     </datasource>  


registry.xml


 <dbConfig name="sharedregistry">  
 <dataSource>jdbc/WSO2SharedDB</dataSource>  
 </dbConfig>  
 <remoteInstance url="https://localhost:9444/registry">  
 <id>instanceid</id>  
 <dbConfig>sharedregistry</dbConfig>  
 <readOnly>false</readOnly>  
 <enableCache>true</enableCache>  
 <registryRoot>/</registryRoot>  
 </remoteInstance>  
 <mount path="/_system/config" overwrite="true">  
 <instanceId>instanceid</instanceId>  
 <targetPath>/_system/esbnodes</targetPath>  
 </mount>  
 <mount path="/_system/governance" overwrite="true">  
 <instanceId>instanceid</instanceId>  
 <targetPath>/_system/governance</targetPath>  
 </mount>  

user-mgt.xml

       <Property name="dataSource">jdbc/WSO2UserDB</Property>  


Configurations of Worker node1

axis2.xml


 <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"  
         enable="true">  
     <parameter name="clusteringPattern">WorkerManager</parameter>  
     <parameter name="AvoidInitiation">true</parameter>  
      <parameter name="membershipScheme">wka</parameter>  
      <parameter name="domain">wso2.carbon.domain</parameter>  
      <!-- The host name or IP address of this member -->  
     <parameter name="localMemberHost">172.20.0.1</parameter>  
     <!--  
     The TCP port used by this member. This is the port through which other nodes will  
     contact this member  
      -->  
     <parameter name="localMemberPort">4100</parameter>  
     <!--  
     Properties specific to this member  
     -->  
     <parameter name="properties">  
       <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>  
       <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>  
         <!-- Manger Setup with Port Mapping-->  
       <!--<property name="subDomain" value="mgt"/>--> 
        <!-- Worker Setup-->  
        <property name="subDomain" value="worker"/>  
     </parameter>  
     <!--  
       The list of static or well-known members. These entries will only be valid if the  
       "membershipScheme" above is set to "wka"  
     -->  
     <members>  
       <member>  
         <hostName>172.20.0.1</hostName>  
         <port>4000</port>  
       </member>  
        <member>  
         <hostName>172.20.0.1</hostName>  
         <port>4200</port>  
       </member>  
     </members>  
     <!--  
     Enable the groupManagement entry if you need to run this node as a cluster manager.  
     Multiple application domains with different GroupManagementAgent implementations  
     can be defined in this section.  
     -->  
     <groupManagement enable="false">  
       <applicationDomain name="wso2.esb.domain"  
                 description="ESB group"  
                 agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"  
                 subDomain="worker"  
                 port="2222"/>  
     </groupManagement>  
   </clustering>  

carbon.xml

     <Offset>2</Offset>  

 <DeploymentSynchronizer>  
     <Enabled>true</Enabled>  
     <AutoCommit>false</AutoCommit>  
     <AutoCheckout>true</AutoCheckout>  
     <RepositoryType>svn</RepositoryType>  
     <SvnUrl>file:///home/engineer/Documents/mitrai/repo/</SvnUrl>  
     <SvnUser>randika</SvnUser>  
     <SvnPassword>randika123</SvnPassword>  
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>  
   </DeploymentSynchronizer>  


master-datasources.xml

 <datasource>  
       <name>WSO2_REG_DB</name>  
                <description>The datasource used for shared config and governance registry</description>  
                <jndiConfig>  
                     <name>jdbc/WSO2SharedDB</name>  
                </jndiConfig>  
                <definition type="RDBMS">  
                <configuration>  
                     <url>jdbc:mysql://localhost:3306/wso2_reg_db</url>  
                     <username>root</username>  
                     <password>root</password>  
                     <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
                     <maxActive>50</maxActive>  
                     <maxWait>60000</maxWait>  
                     <testOnBorrow>true</testOnBorrow>  
                     <validationQuery>SELECT 1</validationQuery>  
                     <validationInterval>30000</validationInterval>  
                </configuration>  
                </definition>  
     </datasource>  
     <datasource>  
       <name>WSO2_USER_DB</name>  
                <description>The datasource used for shared user db</description>  
                <jndiConfig>  
                     <name>jdbc/WSO2UserDB</name>  
                </jndiConfig>  
                <definition type="RDBMS">  
                <configuration>  
                     <url>jdbc:mysql://localhost:3306/wso2_user_db</url>  
                     <username>root</username>  
                     <password>root</password>  
                     <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
                     <maxActive>50</maxActive>  
                     <maxWait>60000</maxWait>  
                     <testOnBorrow>true</testOnBorrow>  
                     <validationQuery>SELECT 1</validationQuery>  
                     <validationInterval>30000</validationInterval>  
                </configuration>  
                </definition>  
     </datasource>  


registry.xml


 <dbConfig name="sharedregistry">  
 <dataSource>jdbc/WSO2SharedDB</dataSource>  
 </dbConfig>  
 <remoteInstance url="https://localhost:9444/registry">  
 <id>instanceid</id>  
 <dbConfig>sharedregistry</dbConfig>  
 <readOnly>false</readOnly>  
 <enableCache>true</enableCache>  
 <registryRoot>/</registryRoot>  
 </remoteInstance>  
 <mount path="/_system/config" overwrite="true">  
 <instanceId>instanceid</instanceId>  
 <targetPath>/_system/esbnodes</targetPath>  
 </mount>  
 <mount path="/_system/governance" overwrite="true">  
 <instanceId>instanceid</instanceId>  
 <targetPath>/_system/governance</targetPath>  
 </mount>  

user-mgt.xml

       <Property name="dataSource">jdbc/WSO2UserDB</Property>  



Configurations of Worker node2

axis2.xml


 <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"  
         enable="true">  
     <parameter name="clusteringPattern">WorkerManager</parameter>  
     <parameter name="AvoidInitiation">true</parameter>  
      <parameter name="membershipScheme">wka</parameter>  
      <parameter name="domain">wso2.carbon.domain</parameter>  
      <!-- The host name or IP address of this member -->  
     <parameter name="localMemberHost">172.20.0.1</parameter>  
     <!--  
     The TCP port used by this member. This is the port through which other nodes will  
     contact this member  
      -->  
     <parameter name="localMemberPort">4200</parameter>  
     <!--  
     Properties specific to this member  
     -->  
     <parameter name="properties">  
       <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>  
       <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>  
         <!-- Manger Setup with Port Mapping-->  
       <!--<property name="subDomain" value="mgt"/>--> 
        <!-- Worker Setup-->  
        <property name="subDomain" value="worker"/>  
     </parameter>  
     <!--  
       The list of static or well-known members. These entries will only be valid if the  
       "membershipScheme" above is set to "wka"  
     -->  
     <members>  
       <member>  
         <hostName>172.20.0.1</hostName>  
         <port>4100</port>  
       </member>  
        <member>  
         <hostName>172.20.0.1</hostName>  
         <port>4200</port>  
       </member>  
     </members>  
     <!--  
     Enable the groupManagement entry if you need to run this node as a cluster manager.  
     Multiple application domains with different GroupManagementAgent implementations  
     can be defined in this section.  
     -->  
     <groupManagement enable="false">  
       <applicationDomain name="wso2.esb.domain"  
                 description="ESB group"  
                 agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"  
                 subDomain="worker"  
                 port="2222"/>  
     </groupManagement>  
   </clustering>  

carbon.xml

     <Offset>3</Offset>  

 <DeploymentSynchronizer>  
     <Enabled>true</Enabled>  
     <AutoCommit>false</AutoCommit>  
     <AutoCheckout>true</AutoCheckout>  
     <RepositoryType>svn</RepositoryType>  
     <SvnUrl>file:///home/engineer/Documents/mitrai/repo/</SvnUrl>  
     <SvnUser>randika</SvnUser>  
     <SvnPassword>randika123</SvnPassword>  
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>  
   </DeploymentSynchronizer>  


master-datasources.xml

 <datasource>  
       <name>WSO2_REG_DB</name>  
                <description>The datasource used for shared config and governance registry</description>  
                <jndiConfig>  
                     <name>jdbc/WSO2SharedDB</name>  
                </jndiConfig>  
                <definition type="RDBMS">  
                <configuration>  
                     <url>jdbc:mysql://localhost:3306/wso2_reg_db</url>  
                     <username>root</username>  
                     <password>root</password>  
                     <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
                     <maxActive>50</maxActive>  
                     <maxWait>60000</maxWait>  
                     <testOnBorrow>true</testOnBorrow>  
                     <validationQuery>SELECT 1</validationQuery>  
                     <validationInterval>30000</validationInterval>  
                </configuration>  
                </definition>  
     </datasource>  
     <datasource>  
       <name>WSO2_USER_DB</name>  
                <description>The datasource used for shared user db</description>  
                <jndiConfig>  
                     <name>jdbc/WSO2UserDB</name>  
                </jndiConfig>  
                <definition type="RDBMS">  
                <configuration>  
                     <url>jdbc:mysql://localhost:3306/wso2_user_db</url>  
                     <username>root</username>  
                     <password>root</password>  
                     <driverClassName>com.mysql.jdbc.Driver</driverClassName>  
                     <maxActive>50</maxActive>  
                     <maxWait>60000</maxWait>  
                     <testOnBorrow>true</testOnBorrow>  
                     <validationQuery>SELECT 1</validationQuery>  
                     <validationInterval>30000</validationInterval>  
                </configuration>  
                </definition>  
     </datasource>  


registry.xml


 <dbConfig name="sharedregistry">  
 <dataSource>jdbc/WSO2SharedDB</dataSource>  
 </dbConfig>  
 <remoteInstance url="https://localhost:9444/registry">  
 <id>instanceid</id>  
 <dbConfig>sharedregistry</dbConfig>  
 <readOnly>false</readOnly>  
 <enableCache>true</enableCache>  
 <registryRoot>/</registryRoot>  
 </remoteInstance>  
 <mount path="/_system/config" overwrite="true">  
 <instanceId>instanceid</instanceId>  
 <targetPath>/_system/esbnodes</targetPath>  
 </mount>  
 <mount path="/_system/governance" overwrite="true">  
 <instanceId>instanceid</instanceId>  
 <targetPath>/_system/governance</targetPath>  
 </mount>  

user-mgt.xml

       <Property name="dataSource">jdbc/WSO2UserDB</Property>  


Setting up nginx

Install nginx
 $sudo apt-get install nginx  

Configurations in /etc/nginx/nginx.conf file
 upstream test_cluster{  
      server worker1.com:8282;  
      server worker1.com:8283;  
 }  
 server {  
      listen 80;  
      server_name mitrai.test.com;  
      location / {  
           proxy_pass http://test_cluster;  
      }  
 }  

Check the status of nginx.conf file
 engineer@engineer-VirtualBox:/etc/nginx$ sudo nginx -t  
 nginx: the configuration file /etc/nginx/nginx.conf syntax is ok  

Start nginx server

 sudo /etc/init.d/nginx start  


Starting Server Nodes

Start the Manager node first and after worker nodes.

  • Manager node:
     ./Wso2server.sh  
    
  • Worker Nodes:
     ./Wso2server.sh -DworkerNode=true