Configuring celery application

Celery application should be configured and instantiated before starting workers and executing tasks. Creating celery application requires very less configuration to get started.

Celery application should be configured and instantiated before starting workers and executing tasks. Creating celery application requires very less configuration to get started.

Before we start creating celery instance, we need broker. Broker is a very important component in the celery architecture which controls the task flow. RabbitMQ and Redis are stable message brokers supported by celery. We need one of them and celery should be installed in your virtual environment or any python environment to get started.

Lets look at a basic configuration to get the feel of the celery. I will be using redis as a message broker in my posts. Please bear with me about this.

  • Download and install Redis as per the instructions from the official documentation and run the redis server.
  • Install celery and redis via pip (pip install -U “celery[redis]”) as bundle.

Creating celery application

  • Import the celery module
  • Create app as celery_demo with broker and backend url

To get started with Celery you only need broker url. You can even ignore main parameter while initiating celery. But it will create functions as __main__ module. For better clarity, lets give a name to call it.

# celery_app.py
from celery import Celery
 
app = Celery(main="celery_demo",
             broker="redis://localhost:6379/0",
             backend="redis://localhost:6379/1")

Run worker from module

Import the created app in the __init__.py in the package. when you add the below import in init file, celery application instance which we created will loaded into module namespace. We will see why that is needed in the next section.

from .celery_app import app

You are all set to start the worker now.

celery -A celery_demo worker

celery_demo is the app name we gave while creating the celery instance. You can give the app name (celery_demo) straight to start the worker.

When you are trying to run with module approch, you should be out of the package and try to run. For example If your module is in /blog/celery/celery_app.py, starting worker from /blog.

Run worker from script

This is more straightforward way then previous one. You can start the worker or load the celery app by giving the script which has the celery instance.

celery -A celery_app worker

Once you started the worker you will be seeing similiar message as below.

I more or less lean towards module approach which we discussed in the first section. It is a personlized style. You can pick the style you want to follow.

Feel free to leave your comments, if you want to explain more or clarify any doubts.

Happy coding!!

Related posts
Celery Architecture

Celery Architecture

Durai Pandian Mar 28, 2020

This post provides insights about how celery executes task using worker with execution pools and how broker handles mess...
Continue reading...

Comments
We'll never share your email with anyone else.