[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Slow instance launch times due to RabbitMQ

On 8/6/19 5:10 PM, Ben Nemec wrote:
> Another thing to check if you're having seemingly inexplicable messaging
> issues is that there isn't a notification queue filling up somewhere. If
> notifications are enabled somewhere but nothing is consuming them the
> size of the queue will eventually grind rabbit to a halt.
> I used to check queue sizes through the rabbit web ui, so I have to
> admit I'm not sure how to do it through the cli.

On the cli.
Purging Rabbit notification queues:

rabbitmqctl purge_queue versioned_notifications.info
rabbitmqctl purge_queue notifications.info

Getting the total number of messages in Rabbit:

NUM_MESSAGE=$(curl -k -uuser:pass
2>/dev/null | jq '.["queue_totals"]["messages"]')

The same way, you can get a json output of all queues using this URL:

and playing with jq, you can do many things like:
jq '.[] | select(.name == "versioned_notifications.info") | .messages'
jq '.[] | select(.name == "notifications.info") | .messages'
jq '.[] | select(.name == "versioned_notifications.error") | .messages'
jq '.[] | select(.name == "notifications.error") | .messages'

If sum add the output of all of the above 4 queues, you get the total
number of notification messages. What I did is outputing to graphite
like this:

echo "`hostname`.rabbitmq.notifications ${NUM_TOTAL_NOTIF} `date +%s`" \
	| nc -w 2 graphite-node-hostname 2003

for the amount of notif + the other types of messages.

Doing this every minute makes it possible to graph the number of
messages in Grafana, which gives me a nice overview of what's going on
with notifications and the rest.

I hope this will help someone,

Thomas Goirand (zigo)