Django

Introduction

Named after Django Reinhardt. It is a Python web framework. Packages to support Django can be found at http://awesom-django.com It includes

• ORM (Object mapping to DB)
• URL Mapping
• Templates
• Forms
• Admin
• Package

Some of the design principles can be found at http://goo.gl/PRrEMe. Django using an Model Template View approach which is like MVC Where the template is the view and the view is a controller.

Installation And Create am App

We can create a python environment and install with. Using and environment keeps the packages local and not available at a user level or global level

python3 -m venv django-env
. django-env/bin/activate
pip install django

We can set the environment in VS Code with
Within Python we can now create our project

django-admin startproject myapp

Key Files created are

• manage.py (Management)
• settings.py (Settings)
• urls.py (Routing)
• wsgi.py (Packaging)

We can run the development server with

python3 manage.py runserver

Creating a View

We can create view by creating a view and add a route.

from django.http import HttpResponse

def welcome(request):
    return HttpResponse("Hello World")

And adding the route to urls.py

from .views import welcome
from django.contrib import admin
from django.urls import path

urlpatterns = [
    path('admin/', admin.site.urls),
    path('welcome/', welcome),
]

Not you can use regex for urls with re_path e.g.

    re_path(r'welcome/', welcome),

Creating Apps

Within in the directory we can divide the app into apps. So go into the directory containing manage.py

python3 manage.py startapp gameplay

Within the settings file we need to add the new app, in this case gameplay.

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
...
    'gameplay'
]

Models

Introduction

We Create models using the django models class. You can see we can create foreign keys and cascade deletes which are all documented.

from django.db import models
from django.contrib.auth.models import User

class Game(models.Model):
    first_player = models.ForeignKey(
        User,
        related_name="games_first_player", 
        on_delete=models.CASCADE,)

    second_player = models.ForeignKey(
        User,
        related_name="games_second_player",
        on_delete=models.CASCADE,)

    start_time = models.DateTimeField(auto_now_add=True)
    last_active = models.DateTimeField(auto_now=True)
    status = models.CharField(max_length=1, default='F')


class Move(models.Model):
    x = models.IntegerField()
    y = models.IntegerField()
    comment = models.CharField(max_length=300, blank=True)
    by_first_player = models.BooleanField()

    game = models.ForeignKey(Game, on_delete=models.CASCADE)

Field Types

Options

• Make Field Nullable (Default is non Null)

Model.IntegerField(null = True)

• Allow Empty Values in Forms (Not db-related)

Model.CharField(blank = True)

• Default Value

Model.IntegerField(default = 'F')

• Typed-specific options

Model.DateTimeField(auto_now = 'True')

Here is a list of some of the fields provided.

Field Name	Description
AutoField	It An IntegerField that automatically increments.
BigAutoField	It is a 64-bit integer, much like an AutoField except that it is guaranteed to fit numbers from 1 to 9223372036854775807.
BigIntegerField	It is a 64-bit integer, much like an IntegerField except that it is guaranteed to fit numbers from -9223372036854775808 to 9223372036854775807.
BinaryField	A field to store raw binary data.
BooleanField	A true/false field. The default form widget for this field is a CheckboxInput.
CharField	It is a date, represented in Python by a datetime.date instance.
DateField	A date, represented in Python by a datetime.date instance. It is used for date and time, represented in Python by a datetime.datetime instance.
DecimalField	It is a fixed-precision decimal number, represented in Python by a Decimal instance.
DurationField	A field for storing periods of time.
EmailField	It is a CharField that checks that the value is a valid email address.
FileField	It is a file-upload field.
FloatField	It is a floating-point number represented in Python by a float instance.
ImageField	It inherits all attributes and methods from FileField, but also validates that the uploaded object is a valid image.
IntegerField	It is an integer field. Values from -2147483648 to 2147483647 are safe in all |databases supported by Django.
GenericIPAddressField	An IPv4 or IPv6 address, in string format (e.g. 192.0.2.30 or 2a02:42fe::4).
NullBooleanField	Like a BooleanField, but allows NULL as one of the options.
PositiveIntegerField	Like an IntegerField, but must be either positive or zero (0).
PositiveSmallIntegerField	Like a PositiveIntegerField, but only allows values under a certain (database-dependent) point.
SlugField	Slug is a newspaper term. A slug is a short label for something, containing only letters, numbers, underscores or hyphens. They’re generally used in URLs.
SmallIntegerField	It is like an IntegerField, but only allows values under a certain (database-dependent) point.
TextField	A large text field. The default form widget for this field is a Textarea.
TimeField	A time, represented in Python by a datetime.time instance.
URLField	A CharField for a URL, validated by URLValidator.
UUIDField	A field for storing universally unique identifiers. Uses Python’s UUID class. When used on PostgreSQL, this stores in a uuid datatype, otherwise in a char(32).

Model API

There are lots of operation you can do on Models. Here are some common ones

# Must return 1
Game.objects.get(pk=5)
# Returns all rows
Game.objects.all()
# Returns matching objects
Game.objects.filter(status = 'A')

Migrations

Django can generate scripts to recreate the database. We can manage this with

# Make
python manage.py makemigrations
# Show
python manage.py showmigrations
# Run
python manage.py migrate

Admin Site

This allow you to

• Register Models
• UI
• Create Super User

Create Super User

We can create the superuser account in the root manage.py with

python3 manage.py createsuperuser

Register Models

We can register the Models in admin.py with

from .models import Game, Move
from django.contrib import admin

# Register your models here.
admin.site.register(Game)
admin.site.register(Move)

User Interface

Amending the admin site is well documented. I not sure they screens are great and wonder how manual entry is a good thing but I guess setting up test data would be easier. Here is an example. Previously is was
By amending as follows

from .models import Game, Move
from django.contrib import admin

# Register your models here.

# admin.site.register(Game)
@admin.register(Game)
class GameAdmin(admin.ModelAdmin):
    list_display = ('id', 'first_player', 'second_player', 'status')

admin.site.register(Move)

We get

Templates

So templates provide the HTML for the pages. I suspect that React or Angular will replace this going forward but here goes.

Setup a new App

Create a new app

python3 manage.py startapp player

Create a view

from django.shortcuts import render
from django.http import HttpResponse

# Create your views here.
def player(request):
    return HttpResponse("Player Page")

Add to Settings

# Application definition
INSTALLED_APPS = [
    'django.contrib.admin',
...
    'gameplay',
    'player',
]

Create a route

In the base route

...
urlpatterns = [
    path('admin/', admin.site.urls),
    path('welcome/', welcome),
    path('player/', include('player.urls')),
]

Then create the apps urls.py

from django.urls import path

from .views import player

urlpatterns = [
    path('', player),
]

Create the Template

It is best practice to create the template in a directory templates/<app> because of naming clashing

Installing Rabbit MQ

Installing Erlang

Here is how to do it on 20.04

echo "deb https://packages.erlang-solutions.com/ubuntu focal contrib" | sudo tee /etc/apt/sources.list.d/rabbitmq.list
sudo apt update
sudo apt install erlang

If we get and error with the keys then, if you are sure, you can add the keys.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys XXXXX

Install RabbitMQ Repository

sudo apt install apt-transport-https -y
wget -O- https://dl.bintray.com/rabbitmq/Keys/rabbitmq-release-signing-key.asc | sudo apt-key add -
wget -O- https://www.rabbitmq.com/rabbitmq-release-signing-key.asc | sudo apt-key add -
echo "deb https://dl.bintray.com/rabbitmq-erlang/debian focal erlang-22.x" | sudo tee /etc/apt/sources.list.d/rabbitmq.list
sudo apt update
sudo apt install rabbitmq-server
systemctl status  rabbitmq-server.service 
systemctl is-enabled rabbitmq-server.service

Install RabbitMQ Management

sudo rabbitmq-plugins enable rabbitmq_management
# ss -tunelp | grep 15672
tcp   LISTEN  0       128                    0.0.0.0:15672  
sudo ufw allow proto tcp from any to any port 5672,15672

Now create a user

rabbitmqctl add_user admin StrongPassword
rabbitmqctl set_user_tags admin administrator

We can see the interface at

http://server:15672

All of this was taken from the website https://computingforgeeks.com/how-to-install-latest-rabbitmq-server-on-ubuntu-linux/

Creating A Container With Composer

• Create a requirements.txt document with what is required for the container
• Create a Docker File definition
• Create a composer file

Requirements

This will differ from app to app below are examples for Django and Flask

Docker File

Again this will differ from app to app Here is a docker file to support Dango on for 8000

FROM python:3.9
ENV  PYTHONNUMBUFFERED 1
WORKDIR /app
COPY requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt
COPY . /app 

CMD python manage.py runserver 0.0.0.0:8000

Composer File

Here is the composer file for Dango.

version: '3.8'
services: 
  django:
    container_name: djangoapp
    build: 
      context: .
      dockerfile: Dockerfile
    command: 'python manage.py runserver 0.0.0.0:8000'
    ports: 
      - 8000:8000
    volumes: 
      - .:/app
    depends_on:
      - db
    networks:
      django_demo:

  queue:
    build: 
      context: .
      dockerfile: Dockerfile
    container_name: djangoqueue
    command: 'python consumer.py'
    depends_on:
      - db
    networks:
      django_demo:

  db: 
    image: mysql:5.7.32
    container_name: djangodb
    restart: always
    environment: 
      MYSQL_DATABASE: admin
      MYSQL_USER: root
      MYSQL_PASSWORD: root
      MYSQL_ROOT_PASSWORD: root
    volumes: 
      - .dbdata:/var/lib/mysql
    ports: 
      - 33066:3306
    networks:
      django_demo:

networks:
  django_demo:
    external: true

Networking with Containers

General

In order to connect on the network container to container I ended up setting up my own network.

docker network create --attachable django_demo

Once we have the network we can assign the containers to the network by setting the network name and the network in this case django_demo

version: '3.8'
services: 
  django:
    container_name: djangoapp
    build: 
      context: .
      dockerfile: Dockerfile
    command: 'python manage.py runserver 0.0.0.0:8000'
    ports: 
      - 8000:8000
    volumes: 
      - .:/app
    depends_on:
      - db
    networks:
      django_demo:
...

networks:
  django_demo:
    external: true

We need to specify the container_name because the default is to use the service name an, underscore and a number which in this case would have been django_1. This creates an invalid host as the underscore is not allowed.

Commands

We can see the network with

docker network inspect django_demo

We can delete network with

docker network prune

And we can see the containers with

docker ps

Django

To ensure no errors we need to specify which hosts django allows. This is done in the settings.py using

...
ALLOWED_HOSTS = ['127.0.0.1','djangoapp']
...

We need to also need to account for the hostname when using the container_name in the docker-compose yml

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'dont',
        'USER': 'copy',
        'PASSWORD': 'this',
        'HOST': 'djangodb',
        'PORT': '3306'
    }
}

Flask

Database

In this case I created a flask app so the database is specified in the main.py in the SQLALCHEMY_DATABASE_URI. Here we use the hostname for URI.

app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql://root:root@flaskdb/main'

Container to Container

To call the django service we do this by using the hostname of the django service. In our case.

...
  req = requests.get('http://djangoapp:8000/api/user') 
  return jsonify(req.content.decode('utf-8'))

Django

On the container you need to define the app

Setup

Requirements

Here are the specific requirements for the Django cotainer.

Django==3.1.5
djangorestframework==3.12.2
mysqlclient==2.0.3
django-mysql==3.10.0
django-cors-headers==3.6.0
pika==1.1.0

Django Docker File Specific

This command runs the django app at start up

CMD python manage.py runserver 0.0.0.0:8000

Django Docker Composer Specific

This just needs a service and a database defined. The port needs to be available. In the default above this was 8000

Create and Setup App

python manage.py startapp products

Django App Settings

We need to modify the following sections of settings.py the settings file for django

INSTALLED_APPS = [
...
    'django.contrib.staticfiles',
    'rest_framework',
    'corsheaders',
    'products'
]

MIDDLEWARE = [
...
    'django.contrib.sessions.middleware.SessionMiddleware',
    'corsheaders.middleware.CorsMiddleware',
...
]
...

# Database
# https://docs.djangoproject.com/en/3.1/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'admin',
        'USER': 'root',
        'PASSWORD': 'root',
        'HOST': 'db',
        'PORT': '3306'

    }
}
...
CORS_ALLOW_ALL_ORIGINS = True

Define the Website

Create Model Classes

Edit the model classes under the Product app

from django.db import models

# Create your models here.
class Product(models.Model):
    title = models.CharField(max_length=200)
    image = models.CharField(max_length=200)
    likes = models.PositiveIntegerField(default=0)

class User(models.Model):
    pass

Run the Migrations

This will create the tables. Log back into the container and type

python manage.py makemigrations
python manage.py migrate

Make Serializers

We make serializer for the product table with

from django.db import models
from rest_framework import serializers
from .models import Product

class ProductSerializer(serializers.ModelSerializer):
    class Meta:
        model = Product
        fields = '__all__'

Make Url (Routing)

In the main routing add the api route with an include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('products.urls'))
]

In the app we can now add routes for the REST end points. Which are just maps to the views.

from django.urls import path

from .views import ProductViewSet

urlpatterns = [
    path('products', ProductViewSet.as_view({
        'get' : 'list',
        'post': 'create'
    })),
    path( 'products/<int:pk>', ProductViewSet.as_view({
        'get' : 'retreive',
        'put': 'update',
        'delete': 'destroy'
    }))
]

Make Views (Repository or Data Layer)

This is like the repository layer and provides the CRUD functions to support the routing.

from django.shortcuts import render

# Create your views here.
from rest_framework import viewsets, status
from rest_framework.response import Response
from rest_framework.serializers import Serializer

from .models import Product
from .serializer import ProductSerializer

class ProductViewSet(viewsets.ViewSet):

    def list(self, request): # /api/products/<str:id>
        products = Product.objects.all()
        serializer = ProductSerializer(products, many=True)
        return Response(serializer.data)

    def create(self, request): # /api/products
        serializer = ProductSerializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        serializer.save()
        return Response(serializer.data, status=status.HTTP_201_CREATED)

    def retreive(self, request, pk=None): # /api/products/<str:id>
        product = Product.objects.get(id=pk)
        serializer =ProductSerializer(product)
        return Response(serializer.data)

    def update(self, request, pk=None): # /api/products/<str:id>
        product = Product.objects.get(id=pk)
        serializer =ProductSerializer(instance=product, data=request.data)
        serializer.is_valid(raise_exception=True)
        serializer.save()
        return Response(serializer.data, status=status.HTTP_202_ACCEPTED)

    def destroy(self, request, pk=None): # /api/products/<str:id>
        product = Product.objects.get(id=pk)
        product.delete()
        return Response(status=status.HTTP_204_NO_CONTENT)

class UserAPIView(APIView):
    def get(self, _):
        users = User.objects.all()
        user = random.choice(users)
        return Response({
            'id': user.id
        })

Flask

Setup

Requirement

Flask==1.1.2
Flask-SQLAlchemy==2.4.4
SQLAlchemy==1.3.20
Flask-Migrate==2.5.3
Flask-Script==2.0.6
Flask-Cors==3.0.9
requests==2.25.0
mysqlclient==2.0.3
pika==1.1.0

Flask Docker File Specific

For Flask this is the start command in the container

CMD python main.py

Flask Docker Composer Specific

In the composer file we need to make sure the port Flask uses is unique e.g.

services: 
  backend:
    build: 
      context: .
      dockerfile: Dockerfile
    ports: 
      - 8001:5000
    volumes: 
      - .:/app
    depends_on:
      - db

Create App

Create an empty app

from flask import Flask
app = Flask(__name__)

@app.route('/')
def index():
    return 'Hello'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

Define App

Create the Model Classes

Here we define the model

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import UniqueConstraint

from flask_cors import CORS

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql://root:root@db/main'
CORS(app)

db = SQLAlchemy(app)

class Product(db.Model):
    id = db.Column(db.Integer, primary_key=True, autoincrement=False)
    title = db.Column(db.String(200))
    image = db.Column(db.String(200))


class ProductUser(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    user_id = db.Column(db.Integer)
    product_id = db.Column(db.Integer)

    UniqueConstraint('user_id', 'product_id', name='user_product_unique')

@app.route('/')
def index():
    return 'Hello'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

Run the Migrations

We create the tables by creating a script e.g. manager.py

from main import app, db
from flask_migrate import Migrate, MigrateCommand
from flask_script import Manager

migrate = Migrate(app, db)

manager = Manager(app)
manager.add_command('db', MigrateCommand)

if __name__ == '__main__':
    manager.run()

And run the script using the script

python manager.py db init 
python manager.py db migrate
python manager.py db upgrade

Rabbit MQ

Example Consumer

import pika

params = pika.URLParameters('amqp://test123:PASS@heaven.com:5672/%2f')

connection = pika.BlockingConnection(params)

channel = connection.channel()

channel.queue_declare(queue='admin')

def callback(ch, method, properties, body):
    print('Received in admin')
    print(body)

channel.basic_consume(queue='admin', on_message_callback=callback, auto_ack=True)

print('Started consuming')

channel.start_consuming()

channel.close()

Example Producer

And here is a producer

import pika

params = pika.URLParameters('amqp://test:test@192.168.1.70:5672/%2f')

connection = pika.BlockingConnection(params)

channel = connection.channel() 

def publish():
    pass
    channel.basic_publish(exchange='',routing_key='admin',body='hello')

Adding Queue to Container

Simply add another service to the Django container.

  queue:
    build: 
      context: .
      dockerfile: Dockerfile
    command: 'python consumer.py'
    depends_on:
      - db

Application Events

Publishing Event

Next we can change the events to publish. First we change the publish function in the producer to add the method and the body to message using json.

def publish(method, body):
    print('Sending from django to flask')
    properties = pika.BasicProperties(method)
    channel.basic_publish(
        exchange='',
        routing_key='main',
        body=json.dumps(body), 
        properties=properties)

We now need to go to the methods to produce these events.

    def create(self, request): # /api/products
...
        publish('product_created',serializer.data)
        return Response(serializer.data, status=status.HTTP_201_CREATED)
...
    def update(self, request, pk=None): # /api/products/<str:id>
...
        publish('product_updated',serializer.data)
        return Response(serializer.data, status=status.HTTP_202_ACCEPTED)

    def destroy(self, request, pk=None): # /api/products/<str:id>
...
        publish('product_deleted', pk)
        return Response(status=status.HTTP_204_NO_CONTENT)

Consuming Events

In the flask app we now need to act on the messages being sent. There is one action for each of the CRUD operations.

import pika, json

from main import Product, db

params = pika.URLParameters('amqp://test:test@192.168.1.70:5672/%2f')

connection = pika.BlockingConnection(params)

channel = connection.channel()

channel.queue_declare(queue='main')

def callback(ch, method, properties, body):
    print('Received in admin')
    data=json.loads(body)
    print(body)

    if properties.content_type == 'product_created':
        product = Product(id=data['id'], title=data['title'], image=data['image'])
        db.session.add(product)
        db.session.commit()
        print('product created')

    elif properties.content_type == 'product_updated':
        product = Product.query.get(data['id'])
        product.title = data['title']
        product.image = data['image']
        db.session.commit()
        print('product updated')

    elif properties.content_type == 'product_deleted':
        product = Product.query.get(data)
        db.session.delete(product)
        db.session.commit()
        print('product deleted')

channel.basic_consume(queue='main', on_message_callback=callback, auto_ack=True)

print('Started consuming')

channel.start_consuming()

channel.close()

Implementing the Likes

Introduction

Within the app we are going to

• allow flask to serialize json for tables (Flask)
• post a like for a product id in flask
• call the django app for a random user,
• notify the django app of the like for the random user
• update Django with likes

Allow Flask to Serialize Json for Tables (Flask)

We need to add the dataclass decorator and add the types to the class and set the end points

from flask import Flask, jsonify
from dataclasses import dataclass
...
@dataclass
class Product(db.Model):
    id: int
    title: str
    image: str

    id = db.Column(db.Integer, primary_key=True, autoincrement=False)
    title = db.Column(db.String(200))
    image = db.Column(db.String(200))
....
@app.route('/api/products')
def index():
    return jsonify(Product.query.all())

Post a Like for a Product in Flask

We will receive the post, get a random use from the django app. We send a request to the django app to get a random user. We then write a record in the Product User in Flask app and publish the result to Rabbit for the Django app to receive.

@app.route('/api/products/<int:id>/like', methods=['POST'])
def like(id):

    httpclient_logging_patch()
    try:
        req = requests.get('http://djangoapp:8000/api/user') 
        json = req.json()

        app.logger.info("ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ")
        app.logger.info(json)

        productUser = ProductUser(user_id=json['id'], product_id=id)
        db.session.add(productUser)
        db.session.commit()
    
        publish('product liked', id)
    except:
        abort(400, 'You already liked this product')
    
    return jsonify({
        'message' : 'success'
    })

Notify the Django App of the Like for Random User

In the like function we call publish. This sends a message to rabbit queue for the Django app to consume

import pika, json

params = pika.URLParameters('amqp://test:test@192.168.1.70:5672/%2f')

connection = pika.BlockingConnection(params)

channel = connection.channel() 

def publish(method, body):
    print('Sending from django to django')
    properties = pika.BasicProperties(method)
    channel.basic_publish(
        exchange='',
        routing_key='admin',
        body=json.dumps(body), 
        properties=properties)

Update Django likes

Once we receive the message from Rabbit MQ in Django we increase the likes

# Require because consumer runs outside of the app
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "admin.settings")

import django
django.setup()
# End of Require because consumer runs outside of the app

def callback(ch, method, properties, body):
    id = json.loads(body)
    product = Product.objects.get(id=id)
    product.likes = product.likes + 1
    product.save()
    print('products were increased')

channel.basic_consume(queue='admin', on_message_callback=callback, auto_ack=True)

channel.start_consuming()
print('Started consuming')

channel.close()

Useful Commands

Login to Container

Login to the container with

docker-compose exec backend bash

Build Container

Login to the container with

docker-compose up --build