Managing media and static files in Django using S3

Django lets you serve media files from itself in the development. But when it comes to deployment/ Production, Django recommends to use different domain to serve user uploaded contents to mitigate some security risk. You can check the highlighted link for more details.

We will see how to serve media and static files from aws s3 for production.

Versions

We need boto3 package which is AWS SDK in python to interact with AWS Services. Django-storages package provides multiple storage backend for Django. 

Python - 3.7
Boto3 - 1.13.3
django-storages - 1.9.1

Creating AWS IAM user

You need user to work with S3. You can create one with AWS IAM (Identity and Access management). You can follow through below steps if you unfamiliar with creating a user.

  1. Go to IAM services of your AWS and Click users

  2. Add the user name which you want to keep and select programmatic access for the user. It is better to keep the separate access for API and
  3. Create new policy for S3 upload by clicking "create policy". If you are okay, you can still use "AdministratorAccess" (skip the next step)
  4. Select "S3" service and all actions as highlighted below and create policy.


  5. Assign the above created policy to the user. You can skip the tags sections and create new user.

  6. Download the .csv shown below and keep it safe. You need Access key ID and Secret key to upload file.

S3 bucket configuration

We will create S3 bucket with public access and configure it with ACL for read-only for public.

Creating AWS S3 bucket

We will create aws S3 bucket like below.

  1. Navigate to S3 service in AWS console or simply click here and click "create bucket".

  2. Provide unique bucket name, select the region which you are closer to reduce network latency of the transfer. 

    We will keep the access open to public by unchecking "Block all public access". We restrict access through bucket policy.

  3. Navigate to the bucket and create a folder which is called prefix. Creating a "prefix/folder" will keep your bucket clean.

Adding Bucket policy

There are two kinda of policy we need to add. One is for our admin user and another one is for public access. We need user ARN and aws S3 bucket ARN. ARN (Amazon Resource Name) is unique identifier for your resource like aws S3 bucket, IAM User.

Django-storages suggests minimal permission for AWS user. We will use that permission with our users and aws S3 bucket.

  1. Get your user's ARN from IAM. Click here it will get you to AWS users. Click on "s3_cli" or the user you need. You can copy the below highlighted value.
  2. You need ARN for S3 bucket as well.

  3. You can copy below policy JSON to your Bucket policy editor. Replace YOUR_USER_ARN, YOUR_S3_ARN with yours and save. This will public access to read the files.

    1. Sid - AWSUSERPerm1 will give minimal access to create object for AWS user.
    2. Sid - Public1589101959685 will give only read access to the object under media to public.

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "AWSUSERPerm1",
    "Effect": "Allow",
    "Principal": {
    "AWS": "YOUR_USER_ARN"
    },
    "Action": [
    "s3:PutObject",
    "s3:GetObjectAcl",
    "s3:GetObject",
    "s3:ListBucket",
    "s3:DeleteObject",
    "s3:PutObjectAcl"
    ],
    "Resource": [
    "YOUR_S3_ARN/*",
    "YOUR_S3_ARN"
    ]
    },
    {
    "Sid": "Public1589101959685",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": [
    "YOUR_S3_ARN/media/*",
    "YOUR_S3_ARN/media"
    ]
    }
    ]
    }
  4. Now we are ready to configure our S3 bucket in out Django application.

Installing dependencies

You can use pip to install boto3 and django-storages.

(blog)   thuruthuru git:(development pip install boto3 django-storages

Configuring for media files

You can configure django-storages in two ways. Traditional way is to keep the configuration in settings.py. Another one is using custom backend. Custom backend will be flexible when you need to use multiple storages.

Settings.py

We need to add set of configurations in settings.py to push user uploaded contents to S3.

  1. You need to add storage app to INSTALLED_APPS.
  2. Update DEFAULT_FILE_STORAGE backend with django-storages backend
  3. Update AWS access key with your AWS keys
  4. AWS_S3_FILE_OVERWRITE - by default, files will be overwritten in S3 by storage.
  5. AWS_STORAGE_BUCKET_NAME - name of your S3 bucket.
  6. AWS_LOCATION - prefix/ folder created for the media file location in S3.
  7. AWS_DEFAULT_ACL - by default, all the objects pushed to S3 will have public read access, You can disable this as we configured in S3. You can turn this off by setting it to None.
  8. AWS_QUERYSTRING_AUTH - by default, as we set ACL to public read in the policy, we can set this to False.

Now you are set to upload files. You don't need to change anything in your models. It will work as it is. You don't need to change anything in your models, forms or views.

# add storages app to INSTALLED_APPS
INSTALLED_APPS += ['storages']

# changing default storage for media files
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

# AWS access keys
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']

# Storage overwrites file by default. You can remove
# if you want to overwrite the file in S3.
AWS_S3_FILE_OVERWRITE = False

# your app bucket to upload files
AWS_STORAGE_BUCKET_NAME = "your-app-static"

# remove query parameter authentication resource url
AWS_QUERYSTRING_AUTH = False

# prefix path in the S3 bucket. ex: media/prod or media
AWS_LOCATION = "media"

# by default, object uploaded through storage will have public access.
# We will disable this behaviour and use the permissions configured in S3
AWS_DEFAULT_ACL = None

Custom backend

If you decided to use S3 for static files as well and keep it to different location, above configuration will save your static files under media prefix in S3 just like below. You can define your custom backend and provide to DEFAULT_FILE_STORAGE

  1. Create new MediaStorage backend by extending S3Boto3Storage and provide the given values. You can figure this out inspecting S3Boto3Storage for class variables. 
    If you have different user for media and static files, you might be having different access key, secret key. That needs to be configured in backend separately.
    # storage_backends.py
    from storages.backends.s3boto3 import S3Boto3Storage


    class MediaStorage(S3Boto3Storage):
    bucket_name = 'your-app-static'
    location = "media"
    file_overwrite = False     querystring_auth = False
    default_acl = None

  2. You have to map the above storage to DEFAULT_FILE_STORAGE in settings.py. Let's assume that you have django2 project.


    Map your custom storage as below in the settings.py.
    # changing default storage for media files
    DEFAULT_FILE_STORAGE = 'django2.storage_backends.MediaStorage'

    # AWS access keys
    AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
    AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']

Configuring for static files

If you want to serve static files for production from aws S3, you can follow below steps. Process of serving and collecting static files in S3 is same as doing it in local. So you don't need to do any extra step if you are migrating from local filesystem to S3.

  1. You might have already set it up STATICFILES_DIRS in settings.py, if have your custom css and js.
  2. Create folder for static files. 

  3. Add permission in the bucket policy resource as how we added for media files.
    "YOUR_S3_ARN/staticfiles/*",
    "YOUR_S3_ARN/staticfiles"
  4. Create custom storage for static files.  We will keep both backend class in the storage backends in the same file.
    # storage_backends.py
    from storages.backends.s3boto3 import S3Boto3Storage


    class StaticStorage(S3Boto3Storage):
    bucket_name = 'your-app-static'
    location = "staticfiles"     querystring_auth = False
    default_acl = None

  5. Set STATICFILES_STORAGE storage to django-storages.
    # DJANGO STORAGE CONFIGURATION for static files
    STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "static"),
    ]
    STATICFILES_STORAGE = 'django2.storage_backends.StaticStorage'

  6. Run collectstatic now. Static files will be collected to staticfiles in S3. Now you are set to use your application with S3.

    (django2)   django2 python manage.py collectstatic --no-input



Happy reading. Please comment if you are stuck or you have any questions.

Related posts
AWS S3 configuration for backup

AWS S3 configuration for backup

Durai Pandian May 07, 2020

Amazon Simple Storage Service (AWS S3) configuration for backup with expiration, maintaining versions and configuration ...
Continue reading...
Django simple deployment with gunicorn and whitenoise

Django simple deployment with gunicorn and whitenoise

Durai Pandian May 03, 2020

Django simple deployment using gunicorn and whitenoise serving static files with STATIC_URL, STATIC_ROOT, collectstatic....
Continue reading...
Deploy python django app in heroku app

Deploy python django app in heroku app

Durai Pandian Mar 28, 2020

Deploying Django application in free Heroku using heroku-cli, git repository and setting environment variables to Heroku...
Continue reading...

Comments
We'll never share your email with anyone else.