Celery Worker/ RabbitMQ Broker: Step By Step Introduction

I found some tutorials about using Celery, but in general it is used with Django (sometimes Flask), but in this tutorial, I will explain how to simply run Celery (with RabbitMQ) without neither Django nor Flask and I will try to be as simple as possible.

Celery is a task queue/job queue, works with synchronous and asynchronous modes. It is based on distributed message passing and focuses on real-time operations, with a good support to scheduling. With Celery, tasks can execute asynchronously (in the background) or synchronously (wait until ready).

RabbitMQ is a message broker that accepts and forwards messages. As mentioned in the official website :

 

You can think about it as a post office: when you send mail to the post box you’re pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman.

We will interface both tools to allow our application to use Celery to schedule a task, send it to RabbitMQ and the latter will send the message to the server (where Celery is also installed).

Installation

Well, on the client we will install Celery and on the server both Celery and RabbitMQ. (For testing our client and our server will be on the same host 127.0.0.1)

Install pip and Virtualenv then activate it.

Install Celery and RabbitMQ:

Creating a task and running it

Create a file called tasks.py where we are going to declare a task:

and the client.py where we are going to declare the client configuration to be launched:

Notice that celeryconfig.py will contain the celery configuration:

Start the Celery worker in a different terminal windows:

Selection_694

Notice in the terminal output that we can see that multiply is considered as a recognized task by Celery.

Go back to you initial terminal window and type:

The latter command will order the client to send a message using Celery to RabbitMQ server, but since there is no info about task execution, nothing will happen.

Modify client.py so that it will be similar to the following:

now re-execute:

and you will see on the screen (Celery terminal) that the server is receiving the messages (4 messages with the result 4):

Scheduling a task

We will use the same code for now, a little modification.

Client.py

celeryconfig.py

In the latter file, I added CELERYBEAT_SCHEDULE to make the task run every minute.

tasks.py will remain the same:

The server side terminal( running Celery) should be working, otherwise start it:

and on the client side run:

Notice that Celery will receive a message every minute:

I create a github repos for this code, you can find it here.

That’s it!

Cheers.

You may also like...