Tech

Using the Shovel plugin with RabbitMQ for vCloud Director

Jim Freeman

Jim Freeman
Senior Cloud Architect

OnApp uses RabbitMQ with vCloud Director for bi-directional synchronisation. Unfortunately, with vCD you can only add one AMQP server, so the use of the Shovel plugin is helpful if you already have an AMQP server set up in vCloud Director for a third party.

A standard OnApp install will ask you to put OnApp’s own RabbitMQ server in the AMQP section of vCloud Director. Using Shovel will allow you to keep an existing AMQP server in vCD and “push” the specified exchange to OnApp’s RabbitMQ.

This guide covers the installation and configuration of a “new” RabbitMQ instance, as well as just the configuration of the Shovel plugin for those who already have RabbitMQ set up and configured.

The Installing/Configuring RabbitMQ section is based around having a fresh CentOS 7 machine to start with.

Installing RabbitMQ

First of all we need to install Erlang. Prior to this, make sure to update the packages on your CentOS machine to the latest:

yum -y update

Once the update has completed, install the epel repository to access additional packages and download the erlang RPM from their website:

yum -y install epel-release
wget https://packages.erlang-solutions.com/erlang-solutions-1.0-1.noarch.rpm

Now, install the RPM that you just downloaded and install erlang:

rpm -Uvh erlang-solutions-1.0-1.noarch.rpm
yum -y install erlang

Now that erlang has been installed, we can download RabbitMQ and install it:

wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.5/rabbitmq-server-3.6.5-1.noarch.rpm
rpm --import https://www.rabbitmq.com/rabbitmq-release-signing-key.asc
yum -y install rabbitmq-server-3.6.5-1.noarch.rpm

Configuring RabbitMQ

With RabbitMQ now installed, we have to run a few configuration items so we have a user to log in with and ensure the appropriate plugins are enabled. From the command line run the following to ensure that the management plugins and the Shovel plugin are enabled.

rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_shovel_management

Now we need to create a user so we can access the RabbitMQ Web UI. From the command line, run the following:

rabbitmqctl add_user admin password
rabbitmqctl set_user_tags admin administrator
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"

The correct plugins have now been enabled and also a user created. Restart the RabbitMQ service and then log in to the RabbitMQ Web UI by going to http://<server_ip>:15672

service rabbitmq-server restart

 

rabbitmq

Click on the ‘Exchanges’ tab of the UI and find the exchange that you wish to push to OnApp. An example scenario here is that you have an existing exchange that is used for 3rd party software that you wish to use for OnApp with vCloud Director as well.

If the exchange doesn’t already exist, create it now and note the exchange name.

Configuring Shovel

Go to the Admin section of the UI and go to ‘Shovel Management’ sidebar item. We are going to configure our shovel from our RabbitMQ instance here, to OnApp’s pre-installed RabbitMQ.

add-new-shovel

Fill in the details as per below:

  • Name: The name of the shovel so you can distinguish it from any others you may have!
  • Source URI: This will be the URI of the RabbitMQ instance that you are currently logged into. You can use the URI examples that are below in the UI, but for the majority of cases you can leave this as is, simply saying “amqp://” – since this tells it to connect to the local server as the default user.Also here, be sure to change from the default “queue” dropdown to “Exchange” and enter the name of the exchange that you either created, or found in the previous step.
  • Destination URI: This is a little bit more difficult – this will be the related to the OnApp RabbitMQ instance and we will need to pass the credentials through here as well. It should look something like this:
amqp://{rabbitmq_user}:{rabbitmq_password}@{OnApp_IP_Address}:5672

Or a real world example would look something like:

amqp://rbtvcd:gfk45ksgrv@172.16.200.4:5672

You can retrieve the RabbitMQ username from OnApp by running the following at the command line:

cat /onapp/interface/config/on_app.yml |grep rabbit

Also, if the password has certain special characters in (eg, the @ sign) then it will need to be encoded to ensure that it is passed correctly. In the example of the @ sign, you should use ‘%40’.

Again, change from “queue” to “exchange” and the default would be ‘vcloud’ (unless you changed it during the setup of your OnApp vCD.)

The remaining items can mostly be left as default, unless you specifically want to change them. Click “Add Shovel”, wait a few minutes and click the “Shovel Status” sidebar item to confirm it has connected OK.

 

shovel-status

If the state is “running”, then everything is working OK. You can confirm on your OnApp server by running the following command, ensuring that you can see the vcloud session events every couple of minutes:

tail -f /onapp/interface/log/production_vcloud.log

You can find more information about RabbitMQ and the use of the Shovel plugin on their website: https://www.rabbitmq.com/shovel.html

Feel free to post a comment below if you run into any difficulties. If you’re an existing OnApp customer you can also email the VMware Architecture team for further assistance: vmwareteam@onapp.com.

Thanks!