Key Lime Box' Simple MQ

A simple Message Queue with persistent subscribers.

Key Lime Box' Simple MQ

Why another Message Queue - because we needed something simple that does message publish/subscribe the way we wanted it. We looked at all the established message queue systems. They are good, but they are also very complex, handle many messaging needs, etc.

Most publish/subscribe model we found available were based on connected subscribers. We wanted persistent messages to subscribers whether connected or not.

Basic Architecture

We have a web server providing the REST API. This is a Spring Boot (Spring 4) based web application.

The configuration of queues and persistence of messages and configuration is through MongoDB.


When a message is published to a queue, Simple MQ looks at the queue's subscriber and creates a queued message for each subscriber and persist them. If there are no subscriber to the queue, no messages are queued.


In this initial implementation, the persistence to MongoDB is done in a very simple way. There is a collection named messages in which the QueuedMessages are stored.

Each QueuedMessage contains the ID of the queue they are for as well as the subscriber they are for.

Idea of Improvement

An idea on how this could be improved would be to dynamically create collections for each queue/subscriber pair. The advantages of doing it this way are:

There are also some downside to this approach:




Viewing Existing Publishers

GET /api/publishers

Returns the list of existing publishers.

Registering a New Publisher

POST /api/publishers/new

Before publishing messages, the applications need to be registered as publisher. This is a one-time action which is normally done during installation, not at runtime.


Listing the Subscribers

GET /api/subscribers

Returns the list of all registered subscribers.

Registering a New Subscriber

POST /api/subscribers/new

Before a subscriber can register with queues, it needs to be registered as a subscriber so it gets its own ID.

This is done by posting the following parameters to the above REST service:

This is normally done only once when an application is deployed either manually or by the configuration management system. It is not something that generally gets put into the code of the subscriber application.


Listing the Queues

GET /api/queues

This API call returns a list of the queues which exist.

Creating a Queue

POST /api/queues/new

Applications that are going to be publishing messages must first create a queue. This is a one-time action which is normally done during the deployment of the application (i.e. manually or through a configuration management system, not necessarily in the application's code).

To create a queue, post the following parameters:

Subscribing to a Queue

POST /api/queues/{queueId}/subscribe

To subscribe to a queue, post the following parameters to this REST service:

Publishing a Message

POST /api/queues/{queueId}/publish

A message can be published by posting a payload object (JSON) to this REST service with the following parameters:

A simple String payload can also be sent using a request parameter called payload, which is more convenient for payloads which consist of an ID for example.

Note: if there are no subscriber to a queue, the messages published will not be recorded.

Retrieving the Next Message

GET /api/queues/{queueId}/next



A REST service is available for use with monitoring systems. At the moment, the Nagios format is supported. A queue can be monitored by itself, or, a Subscriber ID can be supplied to see the state of the queue as regards to that given subscriber.

Monitoring a complete queue is not in itself very useful as it does not tell you which subscriber stopped processing messages, but there may be some use-cases where this is needed (for example, if you are going to have only 1 subscriber to a queue, then it is one less parameter to pass).

The monitoring API is as follow:

GET /status/queues/{queueId}/?minutes={numberOfMinuteToWarn}&subscriber={subscriberId}&format=nagios

Queue Details

The API for the queues also has a status service returning detalis in JSON format as follow:

GET /api/queues/{queueId}/status


This can be convenient for manual inspection or for monitoring systems which can deal with JSON format.