Monday, April 19, 2010

Connecting to JMX on Tomcat 6 through a firewall

One of the flaws (in my opinion, and shared by others) of the design of JMX/RMI is that the server listens on a port for connections, and when one is established it negotiates a new secondary port to open on the server side and expects the client to connect to that. Well, OK, except that it will pick an available port at random, and if your target machine is behind a firewall, well, you're out of luck because you don't know which port to open up!

With the release of Tomcat 6.0.24, a new Listener (the JmxRemoteLifecycleListener) is available that lets you connect to JMX running on your Tomcat server using jconsole. Using this Listener you can specify the secondary port number instead of it being picked at random. This way, you can open two known ports on your firewall and jconsole will happily connect and read data from Tomcat's JVM over JMX.

Setting it up is pretty easy. First, copy catalina-jmx-remote.jar from the extras folder of the binary distribution into Tomcat's lib folder.

Update your server.xml to include the Listener:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002"/>

Replace the ports with whichever ones you wish. Make sure to open up those ports on your firewall. Be sure to properly configure JMX using an authentication and SSL. Or if you're just setting this up for testing, you can go with the totally insecure and unsafe configuration and add the following JVM arguments to your Tomcat startup script (typically CATALINA_OPTS or JAVA_OPTS):

-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

Now you can start Tomcat. On your client machine, start jconsole and drop in the following URL for your remote process:

service:jmx:rmi://your.public.dns:10002/jndi/rmi://your.public.dns:10001/jmxrmi

Obviously you need to replace your.public.dns with the DNS address of your Tomcat machine, and if you chose different ports, change those as well. With some luck, you'll connect and be getting data!

If you're on EC2 or a similar network where you have an internal DNS name that's different from your external/public DNS name, one more step is required. Additionally set the following property to the server's external/public DNS name

-Djava.rmi.server.hostname=your.public.dns

And with that bit of magic you should be off and collecting data!

18 comments:

  1. Thanks Gabe,

    This certainly is simpler than configuring JMX using custom "javaagent's" etc. A couple of notes for getting this working using SSH tunneling, you have to:

    1. Obviously tunnel both ports
    2. Set useLocalPorts="true" in the JmxRemoteLifecycleListener definition, otherwise the client will try connect directly to the server IP instead of using the tunnel (i.e. localhost).
    3. Have the catalina-jmx-remote.jar available on the client classpath as well, otherwise you'll get a "java.lang.ClassNotFoundException: org.apache.catalina.mbeans.JmxRemoteLifecycleListener$RmiClientLocalhostSocketFactory (no security manager: RMI class loader disabled)"

    ReplyDelete
  2. I had to place a file called "jmxremote.access" in the $CATALINA_HOME directory to make this work. The file contained the following 2 lines:

    monitorRole readonly
    controlRole readwrite

    After that, I opened the ports 10001 and 10002 to my static IP via the security group settings and I was able to use jconsole for the first time on EC2!

    Thanks Gabe.

    ReplyDelete
  3. Hi all I am trying to use ssl to connect remote machine with jconsole using rmi but not able to connect in this way without ssl it working.

    ReplyDelete
  4. You should also *avoid* to set
    -Dcom.sun.management.jmxremote.port=X
    for this to work.

    This setting should just overwrite or repeat what was set in rmiRegistryPortPlatform.

    However it also erases the fixed port setting from rmiServerPortPlatform and makes the server use some random port again.

    ReplyDelete
  5. thanks so much for this tip, definitely the most efficient and painless way to get it done.

    ReplyDelete
  6. you can have both ports set to one number:
    rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10001"

    then tunnel the connection:
    ssh -L 10001:localhost:10001 remote-host

    and then from jvisualvm just add new jmx connection:
    localhost:10001

    ReplyDelete
  7. This note was extremely helpful. Much appreciated

    ReplyDelete
  8. Thanks :) been banging my head against EC2 all day, all i needed was the rmi hostname.. finally!

    ReplyDelete
  9. If you start Tomcat with a shell script, you can get the public DNS name using the following code:

    export PUBLIC_DNS=`dig +short -x ${PUBLIC_IP} | sed s/.$//`

    Thanks for a very helpful blog post!

    ReplyDelete
  10. Whoops! I left out how to get the PUBLIC_IP. :-)

    export PUBLIC_IP=`curl http://169.254.169.254/latest/meta-data/public-ipv4 2> /dev/null`

    ReplyDelete
  11. I wrote a similar blog but targeting the use of SSH tunneling. Take a look at: http://www.liferay.com/pt/web/thiago.moreira/blog/-/blogs/how-to-monitor-liferay-tomcat-remotely-through-firewalls-using-visualvm

    ReplyDelete
  12. How to bind it to specific interface because by default its listening on all interface.

    ReplyDelete
  13. I am trying to connect my tomcat running on ec2 instance (aws cloud), I opened ports 10002 and 10001 ,I am connecting with the url

    service:jmx:rmi://your.public.dns:10002/jndi/rmi://your.public.dns:10001/jmxrmi

    I replaced your.public.dns with the public dns which I can check for an ec2 instance in aws console something like (ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com)
    If I use the public dns name of the instance will it not work ?
    as David P. Nesbitt suggested
    export PUBLIC_DNS=`dig +short -x ${PUBLIC_IP} | sed s/.$//`

    should I do this ?
    I am not alinux guy so donto understand dig +short -x...etc

    Please advice me.

    ReplyDelete
  14. Make sure you are also starting Tomcat with the JAVA_OPTS including

    -Djava.rmi.server.hostname=your.public.dns

    ReplyDelete
  15. I did this to my setenv.sh

    export JAVA_OPTS="-Xms1024m -Xmx1024m -XX:MaxPermSize=256m"
    JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.ssl=false"
    JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.authenticate=false"

    PUBLIC_DNS=$(curl http://169.254.169.254/latest/meta-data/public-hostname)
    #echo PUBLIC_DNS ${PUBLIC_DNS}
    JAVA_OPTS="${JAVA_OPTS} -Djava.rmi.server.hostname=${PUBLIC_DNS}"
    echo "JAVA_OPTS=${JAVA_OPTS}"

    this works , I did not understand why you need like

    export PUBLIC_DNS=`dig +short -x ${PUBLIC_IP} | sed s/.$//`

    ReplyDelete
  16. Glad it works for you!

    The dig command is just a way to get your public DNS name from your public IP. (dig is a tool that can do DNS and reverse DNS lookups). You can just set it manually.

    ReplyDelete
  17. My TOMCAT server is behind a firewall. I can use jconsole from any computer within the intranet and connect to the TOMCAT server just fine.
    The problem is connecting from outside the network.
    I've done the following:
    1-Forward ports 9001 and 9002 on the firewall to the TOMCAT server

    2-Added the following line to my TOMCAT server.xml file:




    3-Added the following lines to my TOMCAT startup script:

    -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Djava.rmi.server.hostname=my.public.ip \
    -Dcom.sun.management.jmxremote.local.only=false \

    I can telnet the ports 9001 9002 from the remote computer and it connects with no problem. I also noticed jconsole is connecting to the remote server but it just simply can't get any data from tomcat...
    I really don't now what I'm missing... Any help would be greatly appreciated.

    ReplyDelete
  18. Try it without the jmxremote.local.only line, I don't think this works with tunneling.

    ReplyDelete