Monday, April 19, 2010

Connecting to JMX on Tomcat 6 through a firewall

One of the flaws (in my opinion, and shared by others) of the design of JMX/RMI is that the server listens on a port for connections, and when one is established it negotiates a new secondary port to open on the server side and expects the client to connect to that. Well, OK, except that it will pick an available port at random, and if your target machine is behind a firewall, well, you're out of luck because you don't know which port to open up!

With the release of Tomcat 6.0.24, a new Listener (the JmxRemoteLifecycleListener) is available that lets you connect to JMX running on your Tomcat server using jconsole. Using this Listener you can specify the secondary port number instead of it being picked at random. This way, you can open two known ports on your firewall and jconsole will happily connect and read data from Tomcat's JVM over JMX.

Setting it up is pretty easy. First, copy catalina-jmx-remote.jar from the extras folder of the binary distribution into Tomcat's lib folder.

Update your server.xml to include the Listener:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002"/>

Replace the ports with whichever ones you wish. Make sure to open up those ports on your firewall. Be sure to properly configure JMX using an authentication and SSL. Or if you're just setting this up for testing, you can go with the totally insecure and unsafe configuration and add the following JVM arguments to your Tomcat startup script (typically CATALINA_OPTS or JAVA_OPTS):

-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

Now you can start Tomcat. On your client machine, start jconsole and drop in the following URL for your remote process:

service:jmx:rmi://your.public.dns:10002/jndi/rmi://your.public.dns:10001/jmxrmi

Obviously you need to replace your.public.dns with the DNS address of your Tomcat machine, and if you chose different ports, change those as well. With some luck, you'll connect and be getting data!

If you're on EC2 or a similar network where you have an internal DNS name that's different from your external/public DNS name, one more step is required. Additionally set the following property to the server's external/public DNS name

-Djava.rmi.server.hostname=your.public.dns

And with that bit of magic you should be off and collecting data!

Posted by at 10:51 PM
Labels:

12 comments:

  1. Thanks Gabe,

    This certainly is simpler than configuring JMX using custom "javaagent's" etc. A couple of notes for getting this working using SSH tunneling, you have to:

    1. Obviously tunnel both ports
    2. Set useLocalPorts="true" in the JmxRemoteLifecycleListener definition, otherwise the client will try connect directly to the server IP instead of using the tunnel (i.e. localhost).
    3. Have the catalina-jmx-remote.jar available on the client classpath as well, otherwise you'll get a "java.lang.ClassNotFoundException: org.apache.catalina.mbeans.JmxRemoteLifecycleListener$RmiClientLocalhostSocketFactory (no security manager: RMI class loader disabled)"

    ReplyDelete

  2. I had to place a file called "jmxremote.access" in the $CATALINA_HOME directory to make this work. The file contained the following 2 lines:

    monitorRole readonly
    controlRole readwrite

    After that, I opened the ports 10001 and 10002 to my static IP via the security group settings and I was able to use jconsole for the first time on EC2!

    Thanks Gabe.

    ReplyDelete

  3. Hi all I am trying to use ssl to connect remote machine with jconsole using rmi but not able to connect in this way without ssl it working.

    ReplyDelete

  4. You should also *avoid* to set
    -Dcom.sun.management.jmxremote.port=X
    for this to work.

    This setting should just overwrite or repeat what was set in rmiRegistryPortPlatform.

    However it also erases the fixed port setting from rmiServerPortPlatform and makes the server use some random port again.

    ReplyDelete

  5. thanks so much for this tip, definitely the most efficient and painless way to get it done.

    ReplyDelete

  6. you can have both ports set to one number:
    rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10001"

    then tunnel the connection:
    ssh -L 10001:localhost:10001 remote-host

    and then from jvisualvm just add new jmx connection:
    localhost:10001

    ReplyDelete

  7. This note was extremely helpful. Much appreciated

    ReplyDelete

  8. Thanks :) been banging my head against EC2 all day, all i needed was the rmi hostname.. finally!

    ReplyDelete

  9. If you start Tomcat with a shell script, you can get the public DNS name using the following code:

    export PUBLIC_DNS=`dig +short -x ${PUBLIC_IP} | sed s/.$//`

    Thanks for a very helpful blog post!

    ReplyDelete

  10. Whoops! I left out how to get the PUBLIC_IP. :-)

    export PUBLIC_IP=`curl http://169.254.169.254/latest/meta-data/public-ipv4 2> /dev/null`

    ReplyDelete

  11. I wrote a similar blog but targeting the use of SSH tunneling. Take a look at: http://www.liferay.com/pt/web/thiago.moreira/blog/-/blogs/how-to-monitor-liferay-tomcat-remotely-through-firewalls-using-visualvm

    ReplyDelete

  12. How to bind it to specific interface because by default its listening on all interface.

    ReplyDelete

Subscribe to: Post Comments (Atom)