SOA BPEL Issue

Issue:

In Clustered SOA environment, An external client synchronously invokes a BPEL process which in turn invokes another asynchronous BPEL process.

Some time synchronous response is returned to the client and sometime No response is sent to the client, in which the client eventually times out. Although the async process is getting succeeded in it's operation.

Root Cause: This is happening because of internal BPEL OC4J load balancing feature. when a sync process comes to Node A (as an OHS thread), and calls the Asyn process, callback returns to load balancer but routed to Node B BPEL. and Sync process on Node A BPEL (orginal OHS thread) keep on waiting and time outs.

Solution:

 This is a limitation in SOA 10g but there is a workaround by doing the callback to the actual node where ohs thread is still waiting.

There are few steps:


- declare variable

variable name="replyTo" messageType="ns1:WSAReplyToHeader"/

where ns1 is the namespace for Async partnerlink. It should not matter which one, as the type is the same.

- assigned values to the variable. Make sure the Address contains partnerlink and role of the requester

- use the variable when doing the invoke

invoke name="Invoke_NoFeeProvider"
partnerLink="WizzitAccountInfoAsyncABCSNoFeeProvider"
portType="ns4:WizzitAccountInfoAsyncABCSNoFeeProvider"
operation="initiate"
inputVariable="Invoke_NoFeeProvider_initiate_InputVariable"
bpelx:inputHeaderVariable="replyTo"

- modify bpel.xml to contain

property name="optSoapShortcut"
false
property

for the partnerlink of async process.


- modify mod_oc4j.conf to enable local affinity when apache calls oc4j.


Oc4jSet StatusUri /oc4j-status
Oc4jSelectMethod roundrobin:local

All About Oracle Weblogic Server

#Automated script to check Admin Server(s) and Managed Server(s) Status/Availability.

You will need to scripts for this. first is base script second is driver script. please find both below.

1. Create a file through VI editor "AllServerStatus.py" and paste below code.

#AllServerStatus.py

username = 'weblogic'
password = 'welcome1'
URL='t3://testap02.dubaiworld.ae:7001'

connect(username,password,URL)
domainConfig()
serverList=cmo.getServers();
domainRuntime()
cd('/ServerLifeCycleRuntimes/')

for server in serverList:
     name=server.getName()
     cd(name)
     serverState=cmo.getState()
     if serverState=='SHUTDOWN':
        print '**** Shutdown Servers ****'
        print 'Server *****'+ name +'***** State *****'+serverState
        break
     print 'Server *****'+ name +'***** State *****'+serverState
     cd('..') 

2. Save it.

Remember the path where you have created/copied this file as you need to supply the same in below driver script.

  

3. Create another file through VI "weblogicstatus.sh" with below contents.

#weblogicstatus.sh

#to set the wls env

sh $BEA_HOME/wlserver_10.3/server/bin/setWLSEnv.sh > /dev/null

#to check the status of running servers., modify output by adjusting tail swith, for me only last 4 lines was required.

java -cp $BEA_HOME/wlserver_10.3/server/lib/weblogic.jar weblogic.WLST $BEA_HOME/user_projects/domains/SOAOSBDevDomain/bin/AllServerStatus.py|tail -4

Make sure the above path given is correct, you can give direct path in case of issue.

4. Save it. 

5. Give execute permission on both file. 

chmod 755 <filename>

6. Script is ready, you can execute like below.

 sh weblogicstatus.sh

#How to get Weblogic Version Information

#Vesrion Info
sh $BEA_HOME/wlserver_10.3/server/bin/setWLSEnv.sh
java weblogic.version

#exact WLS version can be checked using the following commands:
cat $MW_HOME/wlserver_10.3/.product.properties | grep WLS_PRODUCT_VERSION



# Use SmartUpdate to check the Product Version:
$MW_HOME/utils/bsu/bsu.sh -view -status=applied -prod_dir=$MW_HOME/wlserver_10.3 | grep ProductVersion

# For Fusion Middleware products version, use the command Oracle Home/Opatch/opatch lsinventory and note the output:
# For the MDS version, look in the file RCU Home/rcu/log/logdir./mds.log:
# To get the schema version from the database, login to the database as sysdba and enter the command as shown
select comp_id, comp_name, version from schema_version_registry;

creating a WebLogic administrator user that has only application deployments admin rights.




1.    Open WebLogic Console.
2.    In the Domain Structure window click on Security Realms.
3.    On the right Content Pane click on security realm for which you are creating a user (for example, myrealm).
 Follow below steps.

4.    Click Users and Groups.
5.    The Users and Groups page displays all the users currently defined in the WebLogic Authentication provider's database.
6.    Click the New button link to display the Create a New User page.



7.    Enter the name of the user in the Name field. (User names are case sensitive.)
8.    Optionally, enter a description of the user (such as their full name) in the Description field.
9.    Enter a password for the user in the Password/Confirm Password fields.
10.    Click OK to save your changes.



Adding Users to Groups:
1.    In the Domain Structure window click on Security Realms.
2.    On the right Content Pane click on security realm for which you are creating a user (for example, myrealm).
3.    Click Users and Groups.
4.    Click the name of the user that we just created.



5.    Click on the Groups tab.
6.    All the groups available in the WebLogic Authentication provider's database appear in the Parent Groups box. Use the check-box to select Deployers group and click the right arrow to move it to the Chosen box.



7.    Click Save to save your changes.

Now if you log out and login as the user we just created you will see that most of the actions are disabled. This new user can manage deployments (Install/Update/Delete/Start/Stop) but nothing else. This is how you can create admin users with specific grants.

All About Oracle ESB

Oracle ESB is technically an 'enterprise service bus' designed and implemented in an Oracle Fusion Architecture's SOA environment; to simplify the interaction and communication between existing Oracle products, third-party applications, or any combination of these.

 

ESB Queues

Use these procedures with great care as corrupting the queues could cause the ESB to be irreparable and or loss of transactions!

select count(*) as CONTROL from aq$esb_control; (this queue is used to communicate between the design time and the run time for exampel to disable an adapter)

select count(*) as ERROR_COUNT from aq$esb_error; (this queue is used to signal errors)

select count(*) as ERROR_R from aq$esb_error_retry; (this queue is used to schedule retrial errors for retry attempts)

select count(*) as DEF from aq$esb_java_deferred; (this queue is used to provide guaranteed delivery for Asynchronous services as in yesterdays issue)

select count(*) as MON from aq$esb_monitor; (this queue is used to send audit/logging information from the run time to the design time which then writes it to the DB)

NOTES

It is important to understand that the deferred queue comes into play ONLY when an ESB service is defined as Asynchronous. For synchronous requests there is no queuing and failures are returned to the client. I believe the KT sessions cover which of our services are Asynchronous in depth.

We cannot check the transaction in process very easily as normally it would not be there long enough. Occasionally  there are exceptions where the transactions become dead-locked on the Application. This means the transactions stay in an open state indefinitely and it is possible to find the current ones by looking at the list of transactions in ESB console. Find the oldest that are in "gray" state and then look at the tracking field values for them..

ESB Control Topic

Due to a known issue with ESBRT forced shutdown ESB_CONTROL topic subscribers are not removed always cleanly. This leaves some zombie subscribers to the mutli-consumer ESB_CONTROL topic there by persisting messages for non-existent consumers. In such scenarios following procedure can be followed to clean-up ESB_CONTROL topic.

At any point there should be at most three subscribers to ESB_CONTROL topic. One for DT and one for each RT in the cluster. More than three subscribers means we are running into zombie subscriber scenario. Use the following SQL to verify number of subscribers;

SELECT COUNT(*) FROM AQ$ESB_CONTROL_S

If the above select returns more than 3 we can follow the below procedure to clean-up ESB_CONTROL topic;

1. Shutdown RT and DT instances on all nodes of ESB cluster

2. Login to ESB repository database as ORAESB user

3. Make sure ESB_CLEAR_TOPIC procedure exists. If not compile the below procedure to create one

create or replace procedure esb_clear_topic(pQueName varchar2, status varchar2) as

pOptions dbms_aqadm.aq$_purge_options_t;

begin

pOptions.block := true;

dbms_aqadm.purge_queue_table(

queue_table => pQueName,

purge_condition => 'MSG_STATE = ''' || status || '''',

purge_options => pOptions);

end;

4. Run the following script to remove zombie subscribers

declare cursor c1 is

select * from aq$esb_control_s

subscriber sys.aq$_agent;

begin

for rec in c1 loop

subscriber := sys.aq$_agent(rec.name, NULL, NULL);

DBMS_AQADM.remove_SUBSCRIBER(

queue_name => 'ESB_CONTROL',

subscriber => subscriber);

end loop;

esb_clear_topic('ESB_CONTROL', 'EXPIRED');

end; /

5. Restart DT and RT instances on all nodes in the cluster

ESB Monitor/Java Deferred/Error/Error Retry

Occasionally you may find messages in these topics in states other than 'READY' status like 'EXPIRED' or 'UNDELIVERABLE'. Use the following SQL to verify that;

SELECT MSG_STATE, COUNT(*) FROM AQ$ESB_MONITOR GROUP BY MSG_STATE

SELECT MSG_STATE, COUNT(*) FROM AQ$ESB_JAVA_DEFERRED GROUP BY MSG_STATE

SELECT MSG_STATE, COUNT(*) FROM AQ$ESB_ERROR GROUP BY MSG_STATE

SELECT MSG_STATE, COUNT(*) FROM AQ$ESB_ERROR_RETRY GROUP BY MSG_STATE

If there are messages in unwanted statuses use the following script to clean them (or enter into SQLDeveloper and do Run as script);

BEGIN

ESB_CLEAR('ESB_CONTROL', 'EXPIRED');

END;

This will do all;

BEGIN

ESB_CLEAR('ESB_JAVA_DEFERRED', 'EXPIRED');
ESB_CLEAR('ESB_JAVA_DEFERRED', 'PROCESSED');
ESB_CLEAR('ESB_JAVA_DEFERRED', 'READY');
ESB_CLEAR('ESB_CONTROL', 'EXPIRED');
ESB_CLEAR('ESB_CONTROL', 'PROCESSED');
ESB_CLEAR('ESB_CONTROL', 'READY');
ESB_CLEAR('ESB_MONITOR', 'EXPIRED');
ESB_CLEAR('ESB_MONITOR', 'PROCESSED');
ESB_CLEAR('ESB_MONITOR', 'READY');
ESB_CLEAR('ESB_ERROR', 'EXPIRED');
ESB_CLEAR('ESB_ERROR', 'PROCESSED');
ESB_CLEAR('ESB_ERROR', 'READY');
ESB_CLEAR('ESB_ERROR_RETRY', 'EXPIRED');
ESB_CLEAR('ESB_ERROR_RETRY', 'PROCESSED');
ESB_CLEAR('ESB_ERROR_RETRY', 'READY');

END;

Refresh ESB Topics

Over a period of time, ESB Topics may get corrupted for unknown reasons. After using all other options below is the procedure to recreate the ESB Topic underlying AQ queues. Note that this procedure is only applicable when ESB Topics are enabled for AQ persistence.

1. ready SOUP UI for test.

3. Stop the environment

opmnctl stopall

4. Access the environment database schema used for ESB (either oraesb or esb).

5. Make sure Environment stopped successfully and JAVA DEFERRED queue has no message in ready state.

1. opmnctl status
 

2. select consumer_name, msg_state, count(*) from aq$esb_java_deferred group by consumer_name, msg_state order by consumer_name, msg_state;

3. Carry out the following query:
select * from gv$active_services;

This should show a list of services as follows:

esb should appear at least twice with different values in the INST_ID column
DB1 should appear once with INST_ID = 1
DB2 should appear once with INST_ID = 2


If this is not the case please contact the DBA team to ensure the DTP services have been created correctly.

4. Access the script {RT_HOME}/integration/esb/sql/oracle/create_esb_topics.sql

Locate the following code and change to latest version :

dbms_aqadm.create_queue_table(Queue_table => qtablename,
Queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE',
multiple_consumers => true,
primary_instance => 1,
secondary_instance=> 2,
compatible => '10.2');
dbms_aqadm.create_queue (Queue_name => qname,
Queue_table => qtablename);
dbms_aqadm.start_queue(qname);

5. From SQL prompt Run the modified script against the ESB schema for the environment.

6. When script completes run the below query against .

select queue_table, primary_instance, secondary_instance, compatible from all_queue_tables
where owner = '';

Confirm the values for primary_instance(1),secondary_instance(2) and compatible(10.0.0).


7. Bring up the environment, Run the regression test, it should went fine.

8. Regression test does not exercise AQ, run the provider part of Load Test and confirm Queues are working fine.

This procedure is destructive with respect to in-flight messages within ESB layer. So proper analysis is required before executing this option!


It is important to understand that the deferred queue comes into play ONLY when an ESB service is defined as Asynchronous. For synchronous requests there is no queuing and failures are returned to the client. I believe the KT sessions cover which of our services are Asynchronous in depth.

We cannot check the transaction in process very easily as normally it would not be there long enough. Occainsonnally there are exceptions where the transactions become dead-locked on the Application. This means the transactions stay in an open state indefinitely and it is possible to find the current ones by looking at the list of transactions in ESB console. Find the oldest that are in "gray" state and then look at the tracking field values for them. This is not normally possible and cannot be proceduralised.
Removing Messages From ESB Deferred Queue

Follow these steps to identify large message payload in ESB deferred queue and remove them if required.

1. Run this query to get the consumers, Message State and Count

select consumer_name, msg_state, count(*)
from aq$esb_java_deferred
group by consumer_name, msg_state;

2. Run this query to get list of messages by enqueue time for a specific consumer in a specific state

select a.queue, a.consumer_name, a.msg_state, a.enq_timestamp, a.retry_count, a.msg_id
from aq$esb_java_deferred a
where consumer_name = 'consumer from previous query' and msg_state = 'state from previous query'
order by enq_timestamp;

3. Run this query to get the size of a specific message in the queue

select a.queue, a.consumer_name, a.msg_state, a.enq_timestamp,
length(a.user_data.text_lob) msg_size, a.retry_count, a.msg_id
from aq$esb_java_deferred a
where msg_id = 'message id from previous query';

4. To remove the message in the queue for a specific subscriber first determine the message ID of the message in the queue from above mentioned queries and then call the function as in the following example:

DECLARE
po_t dbms_aqadm.aq$_purge_options_t;
BEGIN
po_t.block := false;
dbms_aqadm.purge_queue_table('ESB_JAVA_DEFERRED', 'msg_id=' ' message id from previous query ' ' ', po_t);
END;

Note:
  This action should NEVER be taken without full analysis of the issue and that this is the right course

Purging of WSM/ESB schemas
Applies to: Oracle Web Services Manager - Version: 10.1.3.1 to 10.1.3.4.0 Information in this document applies to any of platform.

The following options can be performed to shrink the database without altering the operation of WSM.
1. Message logs 2. Log objects. 3. Policy versions

we can truncate tables messagelogs and log_objects directly, there is zero impact.
we can purge the policy version from the WSM layer to shrink the database. for safe side you can take backup of owsm schema and oc4j_soa under oracle_home/j2ee/.

ESB DVM Update Procedure
Login to the specific ESB console.
Example: For UAT: https://uat02-test.com:7778/esb

Step 1. Enter oc4j username/password
Step 2. Click on Maps...
You will get all the domain value maps.
Step 3. Select required value on from center side panel.
Example: selected CountryCode.
Wait for a moment… You will get the all the CountryCode domain values on the right hand panel. Where you can search for specific code. like CN for China.
Step 4. For exporting the map click on Export
Click "OK" You will be asked to save the XML file, save it.
Step 5. Make the required corrections or get the updated file that Need be uploaded. Now you need to upload it back to ESB console. Be carful while uploading the file, file structure should not be modified.
Step 6. Click on Import 

Locate the XML file. Choose the "Import Options"
Step 7. Click on "OK"
Wait for a moment…
Click "Save".
Note that a restart of ESB is NOT required.
Crosscheck
Perform Step 3 Now you will be able to see the modified values or uploaded file’s contents on the right hand panel.
 

WSM/ESB Tables for purging.

TABLES:
ORAESB.ESB_TRACKING_FIELD_VALUE
ORAESB.ESB_ACTIVITY
ORAESB.ESB_FAULTED_INSTANCEORA

ORAWSM.MEASUREMENT_ALARM_STORE
ORAWSM.MESSAGELOGS
ORAWSM.LOG_OBJECTS

#JVM heap utilization check (works on solaris)
$JAVA_HOME/bin/jmap -heap $(/usr/ucb/ps -axwww | grep java |grep oc4j_wsm |awk '{print $1}')
 oc4j soa
$JAVA_HOME/bin/jmap -heap $(/usr/ucb/ps -axwww | grep java |grep oc4j_soa |awk '{print $1}')