Remote Java Monitoring Tools

Recently I have been digging into the operational status of the IBM Informix JSON Listener in high-load, multi-tenant production environments. While I do most of my development work on a Mac using the Oracle (Sun) JVM, the systems I have been analyzing are all using the IBM JVM. This gave me the opportunity to learn a new tool, the IBM Health Center, that I wasn’t so sure about at first, but ultimately I found it provided enlightening lock utilization data. I will be creating a follow-up post on the analysis performed using the Health Center.

Network Connectivity

When connecting to a JVM running on the same machine, folders such as /tmp/hsperfdata_username (where ‘username’ is your user name) are used. Most tools also support remote monitoring and management using JMX (MX Beans). Rather than engineer a new transport layer for efficient messaging between remote applications and the JVM, the existing RMI transport mechanism is used. As a result, two ports will need to be opened in the firewall for the remote application to connect to the JVM. The initial JMX port will only be used for a short time to establish the port number and protocol (typically JRMP or IIOP) of the RMI port. In the example below, if ports 1098 and 1099 are not both opened, applications such as JConsole and Visual VM will not be able to connect.

To enable JMX and specify the ports that JMX and RMI are listening on, define the following system properties when starting a Java application from the command-line:

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=1098 
-Dcom.sun.management.jmxremote.rmi.port=1099 
-Dcom.sun.management.jmxremote.local.only=false 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false

Note: I have intentionally added line-breaks after each property in the above example to improve readability. These line breaks should not be present in use.

If you are using an IBM JVM, you should also add the following flag:

-Djavax.management.builder.initial=

Note: There is a single space after the equals sign. While it may seem odd, just trust me on that one.

System PropertyDescription
com.sun.management.jmxremoteEnsures that JMX is running. By default it is enabled on Sun JVMs. I am not sure about the default behavior of an IBM JVM, but it shouldn't hurt to include it.
com.sun.management.jmxremote.portThe port number of the socket JMX listens on for connections
com.sun.management.jmxremote.rmi.portThe port number of the socket RMI listens on for connections
com.sun.management.jmxremote.authenticateA boolean indicating if authentication should be enabled (true) or not (false).
com.sun.management.jmxremote.sslA boolean indicating if SSL should be enabled (true) or not (false).
com.sun.management.jmxremote.local.onlyA boolean indicating if the JMX connection info should be added to the RMI registry in a way that is only accessible to the local machine (true).
javax.management.builder.initialA flag used with the IBM JVM when enabling JMX.

If the JSON listener is run on port 27017 with JMX and RMI on ports 1098 and 1099, respectively, netstat -nlp shows the following information:

netstat Java JMX and RMI ports

And on the client system, after connecting, the only port in use is 1099.

JConsole Open Ports

JConsole

JConsole is a simple but useful graphical analysis tool. With JConsole you can monitor:

  • heap, non-heap, and pool memory usage
  • the number of threads, the current stack of a thread, basic thread deadlock detection
  • the number of loaded classes
  • CPU usage

Connecting to JConsole is as simple as adding a JMX listen thread to your application as described in the previous section.

Below are some example screenshots:

JConsole Overview Tab

JConsole Overview Tab

JConsole Memory Tab

JConsole Memory Tab

JConsole Threads Tab

JConsole Threads Tab

JConsole Classes Tab

JConsole Classes Tab

JConsole Summary Tab

JConsole Summary Tab

Oracle (Sun) VisualVM

When I encounter a performance issue, VisualVM is my go-to tool. VisualVM does everything JConsole does, and much more when jstatd is available in addition to JMX, with a much cleaner interface. The jstatd program is included with the Sun JRE/JDK but can be used with an IBM JVM. To use jstatd, you will want to create a permissive security file in your home directory. Create the file $HOME/jstatd.all.policy with the following contents:

grant codebase "file:${java.home}/../lib/tools.jar" {
  permission java.security.AllPermission;
};

To run jstatd:

jstatd -J-Djava.security.policy=${HOME}/jstatd.all.policy

Below are some example screenshots.

Java VisualVM Monitor Tab

Java VisualVM Monitor Tab

Java VisualVM Threads Tab

Java VisualVM Threads Tab

Java VisualVM Sampler Tab

Java VisualVM Sampler Tab

Oracle Java Mission Control

 

IBM Health Center

The IBM JVM equivalent of VisualVM is the IBM Health Center. To use the health center, you must be using an IBM JVM with the Health Center Agent installed. IBM JDKs include the necessary HCA jar. If you are using an IBM JRE, you can install the HCA jar as an add-on. When running a Java program, add the following flag:

-Xhealthcenter

The health center can be installed into an existing Eclipse using the following p2 update site:

http://public.dhe.ibm.com/software/websphere/runtimes/tools/healthcenter/

Although the Eclipse feature can be installed in an Eclipse running on any JVM that supports Eclipse, it will only be able to connect if an IBM JVM is used both to run Eclipse and the Java process to be monitored.

IBM Health Center Garbage Collection View

IBM Health Center Garbage Collection View

IBM Health Center Locking View

IBM Health Center Locking View

IBM Health Center Profiling View

IBM Health Center Profiling View

IBM Health Center Classes View

IBM Health Center Classes View

IBM Heap Analyzer

If you experience an OOM Exception when using an IBM JVM and have a heap dump available, the IBM Heap Analyzer can truly be a blessing. The heap analyzer is available from the FTP site:
ftp://public.dhe.ibm.com/software/websphere/appserv/support/tools/HeapAnalyzer/
After downloading the latest haVVV.jar (where VVV is the version), the jar can be run with a simple java -jar haVVV.jar from the command line. When using the heap analyzer, you will typically need to configure a similar maximum heap size (-Xmx) to that being used by the jvm when the heap dump was created. Although the analyzer can only be used to analyze heaps from an IBM JVM, the tool can be used with a Sun or IBM JVM. Below is a screen shot of the heap analyzer in action.

IBM Heap Analyzer

IBM Heap Analyzer

IBM SmartCloud Meetings on SuSE Linux

Hoorah! After many years, there is finally a reasonable solution available to IBM’s Linux users to deliver presentations over the Internet to both internal and external audiences.

Background

Previously IBM had two online presentation solutions and neither was ideal.

The first software offering, IBM SmartCloud Meetings, also known as Lotus Live, is the most popular presentation software within IBM. This offering is the direct result of IBM’s 2007 acquisition of WebDialogs (http://www-03.ibm.com/press/us/en/pressrelease/22210.wss). This platform supports delivering presentations to internal and external audiences with viewers using Windows, Mac OS X, and Linux. Unfortunately, the ability to present was exclusive to Windows. Support for presenting from Mac OS X was recently added. For a Linux user that needs to present using WebDialogs, your only choice is to use a Windows virtual machine.

The second software offering, IBM SameTime Meetings, is a plug-in running within stand-alone IBM SameTime or Lotus Notes with integrated SameTime. This solution is also known as SameTime Instant Meetings because while chatting with a colleague in SameTime a presentation can be initiated with a single button click. This convenience also results in the first deficiency–all meeting participants must have an IBM SameTime account and have access to the IBM intranet. A key strength is the plug-in’s support for Windows, Mac OS X, and Linux–so all IBM users can take advantage of this solution. The most notable disadvantage is that presentations cannot be made to external participants (non-IBMers).

IBM Solution

IBM’s solution was to grow the capabilities of the SameTime Meetings infrastructure by adding support for participants without an IBM IIP (username/password) that are not on the IBM intranet and by eliminating the requirement to install any IBM software (Notes/SameTime) on the presenter or attendee systems. The new product’s name is still IBM SmartCloud Meetings, but internal documentation has chosen to add the suffix ‘(New)’ to distinguish it from documentation relevant to the old solution. The new SmartCloud Meetings bears very little (if any) resemblance to the prior product of the same name. The prior product was clearly thinly veiled cover placed over the existing WebDialogs platform. The new solution has the same layout as the SameTime plug-in, but instead of being implemented in Java/Eclipse it is implemented using Dojo with a Java applet running on the presenter’s system to grab the screen/application. The URL for the new service is: https://stmeetings.na.collabserv.com

New Problem

While Linux is supported by IBM SmartCloud Meetings (New), the current product has limited support for anything other than RHEL6 / IBM OpenClient when presenting. Here is a brief overview of the steps that occur when a user attempts to share that I have been able to infer from my debugging.

  1. After clicking on the button to begin sharing your screen, a first Java applet, vmverifier.jar, is downloaded and run to determine if your machine is compatible with the application. From looking at the code in this application, the checks are rather minimal and mostly relate to blacklisting older virtual machines.
  2. If your machine passes the verification test, a second applet, appshareapplet.jar, is downloaded and run. This applet creates the folder ~/SameTimeMeetings/NativeLibraries and downloads two files to this location. One of the files is an executable and for me had the name “asnat852_20130827.2051″. The second file is a library that has the same name as the executable with the prefix “lib”. In my case, the name was “libasnat852_….”. Here are my thoughts on the name’s constituents. The “as” at the start stands for “application sharing”. I am not sure about the “nat” part, but perhaps it really does have to do with going through a NAT firewall. (Somewhat odd to put that in a name. Why not throw in TCP or IPv4 in there for good measure while at it? :-) ) The “852” almost certainly refers to the version of SameTime Meetings this code is from (8.5.2). And the “20130827.2051” is a timestamp, likely created when the build started. The format is YYYYMMDD.HHMM.
  3. For the current system to work on Linux, a client-side application is used to grab the screen or an application’s video buffer and send it upstream. This native application has over 50 library dependencies. If any of these are missing, the application won’t work and won’t present any useful error messages to the user.

My Solution

For me, the first step in getting to the heart of this problem was to enable the Java console and crank up the log level to 5. To enable the Java console, run the ‘jcontrol’ application, go to the ‘Advanced’ tab, expand ‘Java console’, and select ‘Show console’. After restarting Firefox the Java console now appeared up when I tried to start sharing. This got me to the point where I could see what was going on under-the-hood and to have an exception dialog appear!

Seeing this exception with its clear reference to the file libasnat… requiring a library prompted me to go and run ldd against the library. Not surprisingly, among the 50+ libraries that were listed, the following two entries appeared:

./libasnat852_20130827.2051.so: /usr/lib64/libjpeg.so.62: version `LIBJPEG_6.2' not found (required by ./libasnat852_20130827.2051.so)
./libasnat852_20130827.2051.so: /usr/lib64/libpng12.so.0: version `PNG12_0'
not found (required by ./libasnat852_20130827.2051.so)

My first instinct was to simply make symbolic links from the current versions of these libraries installed on my systems to the versions that were being requested. Unfortunately, when I looked at the contents of the /usr/lib64 folder I found that symbolic links for libjpeg.so.62 and libpng12.so.0 to the current versions were already present. Clearly the problem wasn’t simply a missing older library that could be solved by making a symbolic link to the current version. The references to the missing declaration of ‘LIBJPEG_6.2′ and ‘PNG12_o’ called out by ldd were no joke. I went to an older system running SuSE 11.3 and looked at the png and jpeg libraries present there. The first good news was that the system used the same version of these libraries that I needed to run ‘libasnat…’. The next step was to see what type of dependencies the jpeg and png libraries had. Mercifully, neither had any dependencies that would prevent me from simply copying these older libraries into /usr/lib64 on my newer system and getting things to work.

Note: All thoughts on this blog are my own and should in no way be construed to represent IBM. I am just a programmer and Linux user trying to get things done.

IBM Informix Mongo Listener Connection Analysis

The IBM Informix JSON Listener is a high-performance abstraction layer and translation engine that enables Mongo clients to interact with Informix. The out-of-the-box configuration is optimized for use by application servers with the following characteristics:

  1. The application server is accessing a relatively small number of databases (< 100).
  2. When the application server starts, hundreds or thousands of connections are opened to the listener.
  3. Connections to the listener are infrequently closed. Once established, a connection to the listener generally stays open for a long time (>60 seconds).
  4. The application server uses a connection pooling mechanism to make efficient use of the available connections to service the requests of tens or hundreds of thousands of users.
  5. If there are no sessions using a particular Mongo db, the associated Informix database connection pool is unlikely to be needed in the near future.

I recently encountered an application in BlueMix with the following characteristics:

  1. The application is accessing a relatively large number of databases (>1000).
  2. The application establishes connections “on-demand”.
  3. Connections to the listener are frequently closed. Connections are generally open a relatively short time (5 to 10 seconds).
  4. The application does not use any connection pooling mechanism.
  5. Even if there are no sessions using a particular Mongo db at the moment, it is likely to be referenced in the near future.

In every way this application is the anti-thesis of the scenario the default configuration addresses. For those interested, the framework running the application is Node.js and this interaction pattern is common when using Node.js. If the listener is run with the default configuration, such an application is likely to experience significant performance issues.

Continue reading ‘IBM Informix Mongo Listener Connection Analysis’ »

Uploading DNG to WordPress

Apparently WordPress tries to get smart with uploaded images that it believes it understands the format of and renames them to have the “correct” extension. After modifying WP to understand that DNG was an allowed file upload type, I found that upon upload it had renamed my .dng file to a .tif file. Needless to say, I was a bit miffed. Briefly, DNG, which stands for Digital NeGative, is an open, unencumbered format for storing the raw data from a camera’s sensor. A DNG file contains a small, embedded, TIFF formatted preview image and another stream of data that contains the raw sensor data.

Short Term Solution

I added DNG as a supported type using the info found here: http://chrismeller.com/2007/07/modifying-allowed-upload-types-in-wordpress

After digging around I found the function wp_check_filetype_and_ext to be the culprit. To temporarily “fix” the issue, I simply commented out the line mentioning that a MIME type of “image/tiff” should have the file extension ‘tif’. This got me what I needed in a hurry.

Long Term Solution

Now I am debating what to do long term. Should I file a defect? I can’t imagine the WP developers would be too upset about simply taking out the rename of TIFF files. Given the gigantic size of a TIFF, it isn’t likely that someone will upload one. The major reasons why a user might be uploading a TIFF file are:

  1. The need for 16-bit precision for each color channel not provided by many of the more common formats. However, most folks don’t own a monitor that is able to take advantage of this extra precision.
  2. For those that are using 16-bits of precision, there is a high likelihood that they are not using the sRGB color space as they wanted a wider gamut as well. Few browsers have good support for color profiles in images. Even though my browser supports and has color profiles enabled, it only supports ICC v2 profiles. To see what your browser supports, go here: http://www.color.org/version4html.xalter

Ultimately, I doubt there is much of a compelling reason to be uploading a TIFF for viewing in a web browser. Smart users should be downloading TIFF and DNG files to their local machine so they can open it with an application that supports color management.

Enabling Custom Firewall Rules on SuSE 11.x

Having just had to dig around for this to block a hacker on my game server, I thought I should point out what must be done to write your own iptables rules on SuSE. Continue reading ‘Enabling Custom Firewall Rules on SuSE 11.x’ »

Monitor Comparison

To decide what I would be using for my second monitor, I compared my NEC PA301W to my 27″ Apple LED Cinema Display (abbreviated hereafter as ALCD). For fun, I also profiled my Apple Macbook Pro 15″, early 2010 model (hereafter abbreviated as MBP). The results were both surprising and as expected. Continue reading ‘Monitor Comparison’ »

PDO::Informix on SuSE 11.x

After helping out an intern developing a demonstration of Informix TimeSeries using PHP/PDO this summer, I decided that I would like to be able to run the demo and other programs, such as OAT, from my existing openSuSE’s Apache installation. The steps below got me to this end goal and hopefully will help others do similarly. Continue reading ‘PDO::Informix on SuSE 11.x’ »

Dependencies of org.eclipse.core.runtime

I was recently asked to come up with the list of required plug-ins to run the core of Eclipse and their size. Eclipse’s plug-in dependencies view is perfect for this task.

Eclipse Plug-in Dependencies View for org.eclipse.core.runtime

To select a plug-in to focus on, click on the magnifying glass icon and a dialog to input the plug-in name will appear.

After scanning through the list, I decided it might be interesting to put together a diagram of the dependency hierarchy.

Dependencies of org.eclipse.core.runtime

Benchmarking getCurrentTime in milliseconds

I recently came upon the need to retrieve the current system time represented as milliseconds from the epoch in a shell script. In essence, I wanted the shell equivalent of Java’s System.currentTimeMillis() : long call. After doing a few searches I came up with a slew of reasonable Bash shell ways of computing this value. Just to throw another variable into the mix, I wrote an extremely short C program to output the same information.
Continue reading ‘Benchmarking getCurrentTime in milliseconds’ »

Nokia CEO Speaks Honestly About Company’s Status

I was pleasantly surprised to see the willingness of Nokia’s new CEO to take a realistic look at the company current state and to plan a strategy to get them back into contention.

http://www.engadget.com/2011/02/08/nokia-ceo-stephen-elop-rallies-troops-in-brutally-honest-burnin/

Although I find the analogy of deciding whether to jump into frozen waters from a burning oil platform to be a bit extreme, Nokia certainly is in no position to dwadle.

I was also very pleased to see that Elop understands the criticality of ecosystems. This is a lesson that many large and previously successful companies still need to learn if they want to continue their success.

I am eagerly awaiting the opportunity to pick up a Meego equipped N9. On the other hand, if the N9 ships with Windows 7 my current E90 Communicator will be the last Nokia product I buy.

Best wishes Nokia.