Home > openstack > Intercept RabbitMQ messages in Openstack nova

Intercept RabbitMQ messages in Openstack nova

The purpose of this tutorial, is to show how we can intercept messages passing between nova components. For example, when we issue a ‘nova boot’ command using python-novaclient to create a new VM, the ‘nova-Api’ service first receive the REST call to create the VM. Then ‘nova-API’ then, issue a rpc call to ‘nova-compute’ to do the actual stuff where nova-compute makes additional RPC call to nova-network and so on. The actual RPC method name (to called on the remote host) , and arguments for the rpc call is wrapped in a message. This message is passed to a message queue using RabbitMQ by the rpc caller and rpc callee ( or message-consumer) on the other side , receives the message from the Rabbit Queue. As mentioned before for the ‘nova boot’ command, ‘nova-API’ is service that calls rpc method and ‘nova-compute’ is the service which is responsible to rpc method as called by ‘nova-api’. For example, when I issue the command,

nova boot --image cirros-0.3.1-x86_64-uec --flavor m1.tiny BBBBBBBBB,

the following message is passed from nova-api to nova-compute. (where BBBBBBBBB is the name (yes, this is a stupid name :p) of the to-be-created VM)

{'_context_roles': [u'anotherrole', u'Member', 'admin'], '_context_request_id': 'req-c68d1b0f-3b7b-4ad4-acc5-74133d5c2a65', '_context_quota_class': None, 'system_metadata': {u'image_kernel_id': '5f9e9c04-2421-4dce-97cc-730d18e09344', 'instance_type_memory_mb': 512L, 'instance_type_swap': 0L, 'instance_type_vcpu_weight': None, 'instance_type_root_gb': 0L, 'instance_type_id': 2L, u'image_ramdisk_id': 'f21bb2a3-0891-484e-8370-ad765f05535d', 'instance_type_name': u'm1.tiny', 'instance_type_ephemeral_gb': 0L, 'instance_type_rxtx_factor': 1.0, 'instance_type_flavorid': u'1', 'instance_type_vcpus': 1L, 'image_base_image_ref': u'dcab5323-e848-4189-b4d7-3f6ee2b9554a'}, 'user_id': u'4e964336db4746e08df7097380d80452', 'display_description': u'BBBBBBBBB', 'key_data': None, 'power_state': 0, 'progress': 0, 'project_id': u'1a19325fce8d4bc593025f373f157099', 'config_drive': '', 'ephemeral_gb': 0L, 'access_ip_v6': None, 'access_ip_v4': None, 'kernel_id': u'5f9e9c04-2421-4dce-97cc-730d18e09344', 'key_name': None, 'display_name': u'BBBBBBBBB', (modified and truncated )

Now, to clarify the purpose of this note, is to show how to intercept all of these kind of messages that is passed through the rabbit queue for every nova services.

In order to do this, you have to have a good understanding of RabbitMQ message passing concepts (queue, exchange, consumers etc) and how openstack works around these concepts. Here I like to note some of the facts that I came across for the interception of rabbitmq messages for openstack.

1. when a message publisher publish a message , it publish it to an exchange. One / more message queues are bound to an exchange using routing keys/ topics. In openstack nova, every rpc.cast message (and some part of the rpc.call message) is published to a single topic exchange. The name of the topic exchange is ‘nova’.

To clarify more, when I see all the rabbitmq topic exchanges in my devstack machine, there are only four topic exchanges.

stack@a30539c9-e721-4bc2-b97f-01dbed6254cb:/opt/stack$ sudo rabbitmqctl list_exchanges | grep topic
amq.topic topic
amq.rabbitmq.trace topic
amq.rabbitmq.log topic
nova topic
openstack topic

I don’t know what the topic exchange named ‘openstack’ is for. But I found that using ‘nova’ exchange, I can get most of the message passing around nova service.

2. Once, I know all topic publisher in openstack nova uses a single exchange named ‘nova’, life is very easy. All that I need to do is to create a new message queue and bind it to the ‘nova’ exchange. One important fact here, once we have a queue and bind it to a topic exchange with appropriate routing key (which is “#”), the queue can get all the messages of the exchange. So, in other word, the queue will get all topic messages that are received by every other queues. In my case, I used a queue ‘simple_queue’ and bind it to the ‘nova’ topic exchange with the routing key “#” which is a wildcard to capture all messages of a topic exchange associated with any topic.

for example, as you can see, my ‘simple_queue’ is associated with a consumer ( using the consumer tag ‘ctag1.0’), which is topic consumer with routing key “#”.

stack@9734efd5-6fcd-4127-92b4-6715a66fda9d:~/rabbitmq$ sudo rabbitmqctl list_consumers | grep simple_queue
simple_queue ctag1.0 false

Enough background. lets see the code that intercept the rabbitmq messages.

1 #!/usr/bin/env python
3 ''' purpose : Intercepting messages that pass around nova-services directly from rabbitmq
4 author: Prosunjit Biswas
5 Date: Nov 8, 2013
7 '''
9 import pika
10 import sys
12 global messageno
13 messageno = 0
15 parameters = pika.URLParameters('amqp://guest:admin@localhost:5672/%2F')
16 connection = pika.BlockingConnection(parameters)
18 exchange_name="nova"
19 queue_name = "simple_queue"
20 binding_key = "#"
22 channel = connection.channel()
23 channel.exchange_declare(exchange = exchange_name,
24 type='topic')
26 result = channel.queue_declare(exclusive=True)
27 channel.queue_bind(exchange=exchange_name,
28 queue=queue_name,
29 routing_key=binding_key)
31 print ' [*] Waiting for logs. To exit press CTRL+C'
33 def callback(ch, method, properties, body):
34 global messageno
35 messageno = messageno + 1
36 print "\n\n"
37 print ("----------------{}th message -----------------\n".format(messageno))
38 print " [x] %r:%r" % (method.routing_key, body,)


> This code used the python ‘pika’ library which has a nice and working introduction at [1].

> The first thing to do, is to connect, the openstack nova rabbitmq server which has a user named ‘guest’ and password ‘admin’. This is shown in line 15.

> In line 18-20, I specify the queue, the exchange, and the routing key which is the most critical part of this task.

> The rest of the code, does what it should do to bind the queue to the given exchange.

Bang! Now you can intercept all rpc messages (to be true most of the messages, and all topic messages) that are passed around different nova services using the rabbitmq.

Some notes from the references that helped me a lot:
1. A Topic Consumer connects to the same topic-based exchange either via a shared queue or via a unique exclusive queue [1].
2. A message broker node will have only one topic-based exchange for every topic in Nova.[1] which says that openstack nova has only one topic exchange which is named ‘nova’
3. There are many instances of direct exchange throughout the life-cycle of a message broker node[1]. In fact, direct exchanges have a name which is the msg_id of the message it is carrying.

[1] http://docs.openstack.org/developer/nova/devref/rpc.html
[2] http://www.rabbitmq.com/tutorials/amqp-concepts.html
[3] http://www.rabbitmq.com/getstarted.html
[4] https://lists.launchpad.net/openstack/msg11400.html

Categories: openstack Tags: , ,
  1. sabstien samuel
    May 27, 2015 at 8:10 am

    Hello, You did a nice job there, but can you explain how you made a “simple_queue”, if i need to make a queue what steps should i follow.
    And i ran the code you provided but it exits at the line
    print ‘ [*] Waiting for logs. To exit press CTRL+C’

  2. seeni
    December 30, 2015 at 12:23 am

    Great Work, Thank You.

  3. Ryan
    January 5, 2016 at 10:43 am

    Hey I have a question and I hope you can answer. Now, how are you not stealing messages? Say the producer hits the exchange. The exchange routs default round robin. So it will push to a queue based on a topic. Well, if you queue is in the list. Than that message goes to your queue. Unless you process that message just like openstack would. Is that process not then lost? If I want to find out when an instance is created, I want to take that info and go do DNS stuff for example. That’s outside of Openstack. But I don’t want to steal the create message from openstack. Wouldn’t your example do just that considering the exchange isn’t fanout but topic? Any help would be awesome. Thanks

    • john jones
      January 27, 2016 at 2:31 pm

      I am interested in doing a similar thing.. Have you found an answer to your question?

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: