Two Startup Parameters on Server process

You might have noticed two heap settings when you grep for a java processes on a linux machine. Below is an example:

This happens when StartScriptEnabled=true is enabled in nodemanager.properties file. The classpath from startWebLogic script and jvm arguments from the script setDomian.env will be used first and the values you specify on the Server Startup tab (In the Admin Console) will get appended to the end. 
If two heap settings bothers or confuses you when you grep for the process on a linux machine, then this what you need to do to get rid of it.

1. Change the value of StartScriptEnabled=true to "StartScriptEnabled=false" in the nodemanager.properties file

2. Restart Nodemanager and the Managed server for the changes to take effect.

Script to fire GC using WLST

Linux:

#/usr/bin/env python

export CLASSPATH=$CLASSPATH:/opt/java1001/bin::/bea/track10200/wlserver_10.0/server/lib/weblogic.jar

export PATH=$PATH:/opt/java1001/bin::/bea/track10200/wlserver_10.0/server/lib/weblogic.jar::/usr/bin

/bea/track10200/wlserver_10.0/server/bin/setWLSEnv.sh

/bea/track10200/user_projects/domains/track10200/bin/setDomainEnv.sh

while true

do

java weblogic.WLST /bea/track10200/scripts/python/gc.py ls -al | tee -a /bea/track10200/scripts/logs/heapmonitor.out

tempfile=heapmonitor.out

if [ -f $tempfile ]; then rm $tempfile;fi

sleep 5m

sed -i -e '/Welcome*/d' -e '/Initializing*/d' -e '/Type help()*/d' -e '/^ *$/d' /bea/track10200/scripts/logs/heapmonitor.out

done

Python:

import os

import time

localtime = time.asctime(time.localtime(time.time()))

if __name__ == "__main__":   

    from wlstModule import *#@UnusedWildImport 

print 'starting the script ....'

redirect('/dev/null','false')

connect('username','password','adminurl')

domainRuntime()

cd('ServerRuntimes')

servers=domainRuntimeService.getServerRuntimes();

print '================='+ localtime + '========================'

print 'Garbage collections performed will be logged below.'

for server in servers:

    try:

        cd("/ServerRuntimes/"+server.getName()+"/JVMRuntime/"+server.getName())

        if int(get('HeapFreePercent'))< 20:

            cmo.runGC()

            print "*> Finished garbage collect on.." +server.getName()

            freejvm = long(get('HeapFreeCurrent'))/(1024*1024)

            totaljvm = long(get('HeapSizeCurrent'))/(1024*1024)

            usedjvm = (totaljvm - freejvm)           

            print server.getName()+ '\tTotal:' + str(totaljvm) + 'MB' '\tFree:' +str(freejvm) + 'MB' '\tUsed:' +str(usedjvm)+'MB'                            

    except WLSTException,e:

        print e

print '=================================================================='

disconnect()  

exit() 

Running Commands on remote machine without a passoword

Linux and Windows
OpenSSH to OpenSSH (ssh2)

Check that ssh functions at all: SSH can replace telnet even without keys. Suppose you are connecting to the remote computer remote.com. as user “testacct”. Run

$ ssh testacct@remote.com

and ssh will ask if you want to keep connecting, type “yes”, and then it should ask for your password and open a shell in testacct’s home directory on remote.com, just like telnet. If this fails, there is a problem somewhere. Make sure ssh is installed on your end, and also make sure that remote.com is accepting ssh connections. If it’s not, you’re wasting your time.

Once ssh is functioning we will set up the keys so that it will no longer be necessary to send passwords. If you are curious about the theory of this then read up on “public key cryptography”.

Create your keys: You need to create private and public ssh keys and put them in the proper place with the proper permissions. In your home directory create a folder .ssh ($ mkdir .ssh), if there is none. Note that Windows may make it difficult for you to create a file starting with “.” if you try to do it with their tools; e.g. Windows Explorer. Next, create the keys with the command

$ ssh-keygen -t dsa

The ssh-keygen program will ask for a passphrase, just hit the “Enter” key unless for some reason you know you want a passphrase. This creates the keys id_dsa and id_dsa.pub and puts them in .ssh/. The private key id_dsa must be readable only by you; change its permissions with

$ chmod 600 .ssh/id_dsa

Put the public key on the remote computer: In this section we are assuming the remote computer is also running OpenSSH. Somehow, you must get the .ssh/id_dsa.pub key onto the remote computer, whether by email, ftp, carrying it over on a floppy (sneakernet), etc.; the cool way to do it is to use scp, which was installed along with ssh. Suppose the remote computer is named remote.com, and your account there is “testacct”. To copy the file to remote, run

$ scp .ssh/id_dsa.pub testacct@remote.com:

Don’t forget the trailing colon. You will be asked for testacct’s password on remote before the copying commences. The file will be copied to testacct’s home directory on remote.

Install the public key on the remote computer: (We assume the remote computer is running OpenSSH on Linux or UNIX!) Once id_dsa.pub is on the remote computer, login into the remote computer (you can use ssh to login with your password as described above). From your home directory (where you should see your newly arrived id_dsa.pub) create a .ssh folder if none exists. Then append your id_dsa.pub to a file in .ssh with

$ cat id_dsa.pub >> .ssh/authorized_keys

This will create the file authorized_keys if none exists. The id_dsa.pub key may be removed from the remote computer’s home directory, if you like. The .ssh folder on the remote computer must have the correct permissions, you may set them with

$ chmod 700 .ssh

Checking the password-less connection: Now the command

$ ssh testacct@remote.com

should give you a password-less connection to remote.com. Likewise, scp should be password-free.

By the way, all the commands you do by first logging into the remote computer can be done remotely, one at a time, using ssh. For example, you can run run “$ ssh testacct@remote.com ls” and get a listing of your home directory files on the remote computer. See the documentation for details.

WLDF: DomainRuntime MBean Server Missing

While configuring Watch and Notification or collecting Metrics for checking (Server state -For Example) whether it's running or not,  DomainRuntimeServer  is required as ServerRuntime dies along with the managed Server. Thus we won't be able to send an email alert as soon as a Managed Server dies using WLDF. 

Sometimes you will not be able see DomainRuntime in MBean server List when setting up WLDF.

Here is how you enable the DomainRuntimeMBean Server:

1. Create the Watch with ServerRuntime and configure the Watch with all the settings you need.

2. Once the settings are applied, edit the query inside the watch and change "ServerRuntime" to DomainRuntime.

3. Restart the whole Domain. You should see DomainRuntime in the MBean Server List. 

Example:

(${ServerRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != 'RUNNING')

Should be changed to

(${DomainRuntime//[weblogic.management.runtime.ServerLifeCycleRuntimeMBean]//State} != 'RUNNING') 

Why don't we see DomainRuntime Initially ?? 

WebLogic Server does not initialize the Domain Runtime MBean Server until a client requests a connection to it

Huge Pages & Memory Utilization

At times you may find that Unix machines are showing 100% memory utilized, however you may not be running anything on the machine .  This could happen for the following reasons:

1.       Zombie/defunct processes consuming memory

Command to find them if there any: ps aux | awk '{ print $8 " " $2 }' | grep -w Z

2.       Java processes consuming all the memory for several reasons (Eg : memory leaks, heap spaces consuming more memory than the physical RAM etc .. )

3.       Huge pages configured on the machine.

The 1^st & 2^nd points are common issues that you may want to look into as soon as an issue occurs. The 3^rd point is what we will be explaining here:

Look at a screen shot below of a linux machine that has nothing (No programs) running on it.

A description about Huge Pages: A large page is essentially a block of contiguous physical-memory addresses that are reserved for a process. These blocks of memory is blocked only for programs (user processes) running on the server.

 Here is how you can view information about huge pages:

If your applications are memory intensive then using huge pages is recommended. In a 64 bit Linux OS heaps as large as 80% of the available physical memory can be configured using huge pages.

References:

For Linux: http://www.mjmwired.net/kernel/Documentation/vm/hugetlbpage.txt

For Solaris: http://www.solarisinternals.com/wiki/index.php/Multiple_Page_Size_Support