A Guide To Django Performance Tips For Optimization & Testing

In web-application development, performance optimization is an endless yet quintessential task. This not only helps in increasing the number of users but also helps in increasing the usability of the applications. Today, we will discuss Django Performance Tips.

Django is based on the Python Web Framework. Django is an easy to go solution for developing, releasing, and maintaining a program. It is used by several big houses such as Google, Instagram, Youtube, NASA, etc.

Django is based on MVT (Model-View-Template) architecture. Django’s data can be handled by “model”, whereas “view” helps to see how the website or web application would look to the user, and the template helps in designing of the website by showing us the wide range of layouts which are present in Django.

Top-notch Python Web Development Company suggests that Django, along with python, provides you with high-performance web application but you can further optimize the Django Performance based web application by implementing these tips:

Caching: A Powerful Technique

Typically, if you are going to the same value repetitively, it might be a good idea to save it so that every time a client requests, we do not have to recompute it again and again.

This caching of data is a powerful technique that can provide you huge benefits while serving a high number of requests.

High Performance Django has a strong caching framework in which you can save the dynamic content so that you don’t have to compute the data again and again. It gives you granular options to either cache the whole page or some selected fragments or some specific output.

You can use the cached-property decorator on a function so that next time the function is called, you will get the saved value instead of recomputing the function again.

Laziness: Avoids Computation

Laziness and Caching go hand-in-hand. As Caching prevents the computation of the values, again and again, Laziness helps you by executing a certain function only when the value is required.

Laziness avoids computation in the first place and it is done only when it is required. This helps since the value that was computed earlier might not be even used in the future and we will save this computation.

The operations in High Performance Django are lazy. For instance, QuerySets are lazy. QuerySets will not fetch data when you create, pass, or combine them. Only when the values in QuerySets are being used, those values will be fetched from the database.

One more way you can optimize the performance is by forcing delayed executions of fetching data from the database. Certain operations force the QuerySets to prematurely fetch the data. You should try to avoid such operations.

You can also add keep_lazy () decorator to the function. This decorator forces Django to evaluate the function only when the values from those are required. Typically you should use this decorator for functions that are computation extensive.

Database Optimization

One of the main factors slowing the performance of the web applications is the requests done to the database. So, the database operations and queries have to be molded for a function to work optimally.

With SQL, you need to take care of certain aspect of database queries such as:

• Database triggers,
• stored views,
• functions,
• and stored procedures.

The database operation can be optimized if you fetch all the data required in one query. QuerySets are lazy and fetch only required data which may cause multiple fetch operations on the database as the function gets executed.

You can see from the below code snippet that when we get the authors list again from the database, it fetches the data from the cache.

>> entry = Entry.objects.get (id=1)

>> entry.authors.all()  #query performed

>> entry.authors.all() #query performed again

Another way to optimize the database is to get only the required fields. Typically, Django gets all the fields from the table irrespective of the size of the table.

But, if you require only a few fields and it brings all, then the effort of bringing the other irrelevant fields is wasted. This can be done by using defer ( ) and only( ).

Defer command tells what all fields should not be fetched whereas only () command specifies the field which should be fetched.

HTTP Performance

Django provides middleware such as ConditionalGetMiddleware and GzipMiddleware which you can use to enhance the performance of the requests.

The ConditionalGetMiddleware enhances the performance by conditionally sending GET responses. It adds Etag and Last-Modifier header to the requests.

Next time it gets a request, if the header value is If-None-Match or If-Modified-Since, it will not process the GET request and send HttpResponseNotModified response.

The GzipMiddleware zips all the responses from the browsers that understand gzip compression. You should add this middleware once the response is completely formed and all headers are added.

Yet, GzipMiddleware is considered a security threat currently by the security experts. So, make sure you use it with caution.

Sessions: order to optimize performance

Cached sessions can be stored in order to optimize performance. The cached Session will help to bring down the number of times a system tries to reach the database. The cached Session will save the information in itself.

Django supports the session cached by using SessionMiddleware. You can add the middleware by adding “django.contrib.sessions.middleware.SessionMiddleware” in your default settings.py.

High Performance by default stores your session in the database, but you can change it to store the data either on the file system or in the cache.

You can set the SESSION_ENGINE field to “django.contrib.sessions.backends.cache” in order to save the sessions in the cache.

You can also save the sessions in cookies. The session saved in the cookies is encrypted using the SECRET_KEY and you should make sure attackers should not get access to SECRET_KEY otherwise they may read the session data and execute code remotely.

Code Optimization

It is important that you optimize your domain logic code so that it does not take a lot of time to execute.

There are three prominent ways to lift the efficiency of the code:

• Upgrading/Replacing the third-party packages
• Simplifying the code
• Restructuring the code

To upgrade or replace the third-party packages you can give a quick glance at the code and find the third-party packages that come with a lot of baggage for any simple function.

Then you can remove or replace them as per requirement. Otherwise, you can also use a Python profiler to do this task.

Another way is to simplify the code: In Django, the rest framework is really multifaceted and has a lot of innovative features. But for simpler purposes, you can write your own serializer and save your time.

Also, you can use python profiler for not only third-party packages but also your own code. You can restructure the part of the code which is eating a lot of CPU and memory in order to optimize the Django Performance.

Below is a very simple example where we are profiling regex call

import cProfile

import re

cProfile.run('re.compile("foo|bar")')

Once you run the above application, you get the output as below. It shows how many functions calls were executed and how many of them are primitive.

197 function calls (192 primitive calls) in 0.002 seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

     1    0.000    0.000    0.001    0.001 <string>:1(<module>)

     1    0.000    0.000    0.001    0.001 re.py:212(compile)

     1    0.000    0.000    0.001    0.001 re.py:268(_compile)

     1    0.000    0.000    0.000    0.000 sre_compile.py:172(_compile_charset)

     1    0.000    0.000    0.000    0.000 sre_compile.py:201(_optimize_charset)

     4    0.000    0.000    0.000    0.000 sre_compile.py:25(_identityfunction)

   3/1    0.000    0.000    0.000    0.000 sre_compile.py:33(_compile)

Cached Template Loading

The Cache Template loader of Django is another weapon that can be used for optimization. You can enable Cache Template Loader and it will save all of your time and bandwidth to bring the same templates now and then.

For instance, some templates are frequently used and if you enable Cache Template Loader, those templates will be saved.

You can always get these templates directly from the memory, saving you a round trip to the file system. This can help in significantly reducing the response time and making the application performance better. So, Django Performance Testing is important. 

Specific Functions

Since QuerySets lazily loads the data, Django added two functions – select_related and prefetch_related() which can be used to execute the query and get better results.

select_related()

This function will return additional related objects based on foreign keys when the query is executed. It is a single more complex query, which if executed, gives foreign objects. If we had used traditional select objects, we might have to query the database again to get those objects.

prefetch_related()

This function is also similar to select_related() and prefetches the foreign objects. select_related() is limited to single-value relationships but prefetch_related() performs a separate lookup for each relationship and fetches more data.

Upgradation Using Newer Version

Upgradation is obvious in development. It is always thought that the newer the better and efficient it is.

Yet, one should not always run behind the newer versions. When you try to get the latest version of Python or Django make sure you have your system optimized.

Also, you should not keep your hopes high from the latest versions as sometimes it can be the other way around. So, being more cautious while updating the newer version is the key.

Make Use Of line_profiler

You can use line_profiler if you don’t know which line of code is eating up the time. You can install the line_profiler package using pip.

Once you suspect a function that is causing the delay, you need to add IPython Debugger just before the function is called. Once the function is hit, you need to load the line profiler extension in the debugger and execute the function.

The line profiler tells which lines take how much time to execute. Thus you can fix those lines and have better performance.

For example, imagine a function that returns a name of books from the database.

def get_books_by_library_id():

    libraries = Library.objects.all()

    result = {}

    for library in libraries:

        books_in_library = library.books.all()

        result[library.id] = list(books_in_library)

    return result

This slow downs the process. So to find the root cause of this problem, you can make use of IPython debugger as shown below:

from django.http import HttpResponse

from test_app.helpers import get_books_by_library_id

def books_by_library_id_view(request):

    from IPython import embed; embed()

    books_by_library_id = get_books_by_library_id()

    ...

    return HttpResponse(response)

When you use this function, you will be able to know the exact time that CPU spends in solving this code. This is as shown below:

Turn On SQL Logging

Another way to find out the cause behind the delay of execution is by enabling SQL Logging.

By adding these lines of code in settings.py, you can print all the SQL queries executing in your code. This also helps in Django Performance Testing.

This may sound like too much but it will give you meaningful insight about which all queries are getting executed and how much time each query is taking.

It can also help you in finding if the query is repeated.

# settings.py

LOGGING = {

    'version': 1,

    'filters': {

        'require_debug_true': {

            '()': 'django.utils.log.RequireDebugTrue',

        }

     },

     'handlers': {

        'console': {

            'level': 'DEBUG',

            'filters': ['require_debug_true'],

            'class': 'logging.StreamHandler',

         }

    },

    'loggers': {

        'django.db.backends': {

            'level': 'DEBUG',

            'handlers': ['console'],

        }

    }

}

One of the best ways to do this would be to put a debugger at the starting and end of the code, which you think is taking a lot of time. Then, slowly move further and find the queries which are slow and repeated.

Avoid Queries in Loops

There are a few instances where we execute a query to get some data and then again execute the query to get the related data.

Imagine you want to fetch all countries of the world, so you execute a query for it.  Later you realize you also need their capitals, so you would have to query again.

So in order to solve this problem of executing queries in the loop, Django introduced functions like select_related() and prefetch_related() in the QuerySets API.

It can be used to fetch all the related data in one trip to the database. As we can see from the below code snippet, it fetches the related also.

def get_books_by_library_id():

    libraries = Library.objects.all()

    result = {}

    for library in libraries:

        books_in_library = library.books.all()

        result[library.id] = list(books_in_library)

    return result

This creates a performance problem. Instead of this, use the following coding snippet:

def get_books_by_library_id_one_query():

    books = Book.objects.all()

    result = defaultdict(list)    

    for book in books:

        result[book.library_id].append(book)

    return result

This  will generate only 1 SQL query and it will reflect in the Django Performance, without a pinch of a doubt.!

Use Memcached Server

Django provides a very powerful and efficient cache solution called Memcached.  It was initially developed for livejournal.com and later it was open-sourced.

Few prominent websites like Facebook and Wikipedia use Memcached to reduce database operations and fetching data from the cache.

You have to set your backend to use Memcached by setting BACKEND to ‘django.core.cache.backends.memcached.MemcachedCache’.

Memcached runs as a daemon and it is allocated specific memory on the RAM, so all the operations performed by Memcached are directly interacting with memory and they never fetch data from database or filesystem,

Another major advantage of Memcached is it can run on multiple servers and treats all the cache on these servers as a single cache. As a result of this, duplicate values will not be saved.

However, you cannot completely rely on Memcached for data storage. This feature can only be used for short-term data storage.

Read also: Top 20 Python Based CMS That Every Developer Should Know

Make Use of Elasticsearch

Django has a very efficient paginator. It works well but as the length of pages increases, you might have to fetch a lot of data from the database and it slows down the applications.

An efficient way is to use Elasticsearch where you store your data in indexes and once the index is fetched, you can fetch the data in the database for that particular index. This helps us still have high performance when we have many pages.

Reduce the Number of Queries

One of the most overlooked factors when considering performance is the query execution speed and the way queries are written. Typically, when we develop applications, the amount of data present in the database is less and we are not able to see the performance hit.

But when your application is used by multiple users, the database can grow in size and the query execution can cause slowness. Also since we use Django ORM, which is a wrapper, we do not get a picture of how the query is being executed.

Django offers a tool – Debug Toolbar which has a SQL panel in which you can see the execution of queries and the time taken by each of them.

Harikrishna Kundariya

CEO, eSparkBiz

Harikrishna Kundariya, a marketer, developer, IoT, chatbot and blockchain savvy, designer, co-founder, Director of eSparkBiz @Software Development Company where you can Hire Software Developers. His 14+ experience enables him to provide digital solutions to new start-ups based on Web app development.