So, you’re a Python developer or you just use Python to make your life easier by writing utility scripts to automate repetitive tasks and boost your productivity.
Such Python scripts are usually local to your machine, run from the command line and require some Python skills to use including for instance setting up a virtual environment and knowing the basics of using the CLI.
You may have wondered what it involves to convert a script like this into a web app to make it available for other users such as your family members and co-workers.
In this article, I’ll walk through the general process I followed to “convert” a Python utility script into a full-blown web app by building a backend API using Django and Django Rest Framework and a single-page-app frontend using React.js.
But first, here is a bit of a background…
My family often needed me to merge (combine) PDF files for them. When they apply for online services they’re often required to upload all documents as a single PDF file.
As they only use Windows, they couldn’t find a reliable tool of merging PDFs. The apps they found worked only sometimes, they also didn’t want to upload sensitive docs to random PDF merging websites.
Being the family tech guy, they would ask me for help. They would send me the files to merge and I’d send them back a single PDF file; merged and ready to upload.
I used pdftk for a while. It’s available only on Linux (as far as I know) and the command looks like this:
pdftk 1.pdf 2.pdf 3.pdf cat output out.pdfI then upgraded to a tiny Python script using PyPDF. Something like this:
from pathlib import Path
from pypdf import PdfWriter
merger = PdfWriter()
path = Path.home() / "Documents" / "uni-application"
with merger as PdfWriter:
for file in path.glob('*.pdf'):
merger.append(file)
merger.write("merged.pdf")I had to set the path manually every time and rename the PDFs so that they are merged in the desired order.
At some point, I felt I was merging PDFs for the family too often that I needed a way to expose this script to them somehow. I thought I should just build a web interface for it, host it on my local home server and let them merge as many files as they want without needing my help.
There are many ways to package and distribute a Python script including making a stand-alone executable. However, unless you build an intuitive GUI for it, average users may not be able to benefit from it.
From all GUI options out there, I chose to make it a web app mainly because web apps have a common look and an expected functionality, users are used to interacting with them almost everyday and they run (hopefully) everywhere.
So, as a side little project over months, I built a basic REST-ish API for the backend that allows you to create a merge request, add PDF files to it, merge the files and get back a single PDF file to download and use.
I also built a simple frontend for it using React.js.
For deployment, I used docker to run the app on my home server and expose it on the LAN with a custom “.home” local domain. Now, my family use it almost everyday to merge PDFs on their own and everyone is happy.
I’ll go over the general process I followed to build this small web app that serves only one purpose: merging PDF files.
You can swap PDF merging for any functionality like: converting audio/video files, resizing images, extracting text from PDFs or images, etc. The process is pretty much the same.
This is not a step-by-step guide. I’ll just explain the overall process. Tutorials on any technology mentioned here can be easily found online. Also, I won’t clutter the article with code examples as you can refer to the source code in the Github repo.
Backend API and Auth
I chose to go with Django and Django Rest Framework for building the backend as I’m more familiar with these two than other Python options like Flask and FastAPI. In addition, being a battery-included framework, Django comes with a lot of nice features I needed for this specific app, the most important of which are authentication and ORM.
Why the API? Why not just full-stack Django?
I could have just built a typical Django app that works as the backend and frontend at the same time. But I knew for sure I’m going to build a REST API alongside it at some point down the road, so why not just start by an API?
Moreover, the API-based backend is frontend-agnostic. It doesn’t assume a specific client-side technology. I can swap React for Vue for example. I can also build more clients to consume the API like a mobile or desktop app.
Building the API:
I started by thinking what the original Python script does and what the user usually wants to get back using it. In this case the user wants a way to:
- upload at least 2 PDF files,
- merge the PDF files on the server,
- and download the single merged PDF file.
Models
To satisfy these requirements, I started a new Django project with a “merger” app, installed Django Rest Framework, added the models to save everything in the DB. Meaning, when a user creates a merge request, it’s saved to the DB to track various things about it like how many files it has and whether it’s been merged or not. Later, we can utilize this data and build upon it if we decide for example to put each user on a plan/package with a limited number of merges/files per merge etc.
I used the name “order” for the merge request effectively treating it as a transaction. My idea is to make things easier later if the app turns into a paid service.
Serializers
Then I created DRF serializers which are responsible for converting data from Python to JSON format and back (APIs expect and return JSON, not Python data types like dicts).
They say: don’t trust user uploads. So, in serializers also, I validate the uploaded PDF file to make sure they’re indeed a PDF file by verifying that the content type is ‘pdf’ and checking MIME type by reading the first 2084 bits of the file.
This is not a bullet-proof validation system. It may keep away casual malicious users but it won’t probably protect against determined attackers. Nevertheless, it prevents common user errors like uploading the wrong file format.
The validation also checks the file size to make sure it doesn’t exceed the limit set in settings.py, otherwise users can intentionally or naively upload huge files and block server response while uploading.
If the file fails to validate, the API will return a message containing the reason for rejection.
Although you can leave some of this validation for the frontend, you should always expect that some advanced users may go around frontend validation by trying to send requests directly to the API or even build their own frontend. After all, the API is open and can be consumed from any client. This off course doesn’t mean that they can bypass auth or permissions but by having validation in the backend, if they use a custom-built client, they can’t bypass the backend validation.
Views & Endpoints
I then switched to building the views which are essentially the endpoints the frontend will call to interact with the backend.
Based on the requirements of this specific app, following are the API endpoints with a short description of what each does:
GET: /orders/
when a GET request is sent to this endpoint, it returns back a list all orders that belong to the currently authenticated user.
POST: /orders/
A POST request with a payload containing the merge name like: {“name”: “my merge”} creates a new order (merge request).
GET: /orders/{uuid}/
returns details of an order identified by its id
DELETE /orders/{uuid}/
deletes an order
POST /orders/{uuid}/add_files
A POST request with a payload containing a file add files to an order after validating file type and max files limit. It also checks to see if the order has already been merged in which case no files can be added.
GET /orders/{uuid}/files/
lists files of an order
GET /orders/{uuid}/merge/
merges order identified by its id.
This is where the core feature of the app lives. This view/endpoint does some initial checks to verify that the order has at least 2 files to merge and has not been previously merged then it hands work over to a utility function that preforms the actual merging of all PDF files associated with the order:
import os
import uuid
from pypdf import PdfWriter
from pypdf.errors import PdfReadError, PyPdfError
from django.conf import settings
from .exceptions import MergeException
def merge_pdf_files(pdf_files):
merger = PdfWriter()
for f in pdf_files:
try:
merger.append(f.file)
except FileNotFoundError:
raise MergeException
merged_path = os.path.join(settings.MEDIA_ROOT, f"merged_pdfs/merged_pdf_{uuid.uuid4()}.pdf")
try:
merger.write(merged_path)
except FileNotFoundError:
raise MergeException
except PdfReadError:
raise MergeException
except PyPdfError:
raise MergeException
merger.close()
if os.path.isfile(merged_path):
return merged_path
return NoneNothing fancy here, the function takes a list of PDF files, merges them using PyPDF and returns the path for the merged PDF.
On a successful merge, the view takes the path and sets it as the value for download_url property of the order instance for the next endpoint to use. It also marks the order and all of its files as merged. This can be used to cleanup all merged orders and their associated files to save server space.
GET /orders/{uuid}/download/
download the merged PDF of an order after verifying that it has been merged and ready for download. The API allows a max number of downloads of each order and max time on server. This prevents users from keeping merged files on the server forever and sharing the links, turning the app basically into a free file sharing service.
DELETE /files/{uuid}/
delete a file identified by its id
I then wired the urls to the views connecting each endpoint to the corresponding view.
Auth and Permissions:
To insure privacy (not necessarily security), and to allow the app to be used as a paid service later, I decided to require an account for every user to use the app. Thanks to Django’s built-in features, this can be done fairly easily with the auth module. However, I didn’t want to use the traditional session-based authentication flow. I just prefer the token-based auth as it’s more suitable for API-based apps.
I chose to use JWT (JSON Web Token). Basically it’s a way to allow the user to exchange their credentials for a set of tokens: an access token and a refresh token.
The access token will be attached in the header of every request (prefixed with “Bearer “) that requires authentication. Otherwise the request will fail. This token has shorter life span (usually hours or even minutes) and when it expires, the user can send the refresh token to get a new access token and continue using the app.
There are a number of packages that can add JWT auth flow to Django. Most of them use simple-jwt which you can use directly but it requires you to write more code yourself to implement the minimum register/login/out flow.
I went with Djoser which is a REST implementation of Django’s auth system.
It allowed me to use a custom user model to have extra fields in the user table and most importantly, utilize email/password for registration and login instead of Django’s default username/password although I had to tweak the models in the example project to work as intended.
Djoser also gave me the following endpoints for free:
- GET /auth/users/ : lists all users if you’re an admin. Only returns your user info if you aren’t admin.
- POST /auth/users/ : a POST request with a payload containing: name, email and password will register a new user.
- POST /auth/jwt/create: a POST request with a payload containing: email and password will return access and refresh tokens to use for authenticating subsequent requests.
- POST /auth/refresh : a POST request with a payload containing: the refresh token will return a new access token.
as well as some other useful endpoints for changing and resetting passwords.
On top of Django’s auth, DRF uses permissions to decide what API resources a user can access. In short, in the API, I check if the user is logged in first. Then I have only two permissions:
- check if the user is the owner of the order before allowing them to access it, upload files to it, merge it or download it.
- check if the user is the owner of the order that a file is associated with before allowing them to view the file details or delete it.
Failure to meet required permissions causes the API to raise a permission error and return a descriptive message.
During development of the backend, I used the browser extension “ModHeader” to include the access token in all requests I made through the browser (for testing via DRF built-in web UI).
Frontend
I chose React.js for building the frontend as a single-page-app (SPA) because it’s relatively easy to use for small apps and has a huge community. I won’t go into much details of how the frontend is built (this is a Python blog after all) but I will touch on the main points.
Auth in the frontend
First, here is a brief description of the auth workflow in the frontend:
- when a user first visits the app, they’re asked to login or signup to continue.
- the user can signup by filling out their name, email, password. This form data will be sent via an API “POST” request to the backend endpoint /auth/users and register the user.
- for existing users, email and password are sent in a “POST” request to /auth/jwt/create which will return a pair of tokens: refresh and access. These tokens are saved in browser cookies. The access token will be sent in the header of all subsequent request to authenticate the user. When it expires, the frontend will request a new token on behalf of the user by sending the refresh token to /auth/refresh. If both expire, the user will be redirected to login again to obtain a new set of tokens.
- when a user navigates to /logout, all tokens are cleared from browser cookies and the user is redirected to /login route.
Routes:
I built the necessary React component and routes for the app to roughly match the backend endpoints I discussed earlier. These are the routes the app has for authenticated users:
- / => home screen to choose to create a new merge or list previous merges.
- /orders => list all merges for the currently logged in user with links to edit or delete any order.
- /create => create new merge.
- /order/{id} =>details of a merge by id.
- /logout => logs the user out.
Merging workflow:
The workflow for merging a PDF file from the frontend is as follows:
- when the user creates a new merge, they’re redirected to the merge detail route showing the merge they’ve just created with three buttons: add PDFs, merge, download.
- merge detail route allows the user to upload PDF files to the merge. If the files are less than 2, only “Add PDF files” is enabled. When the user adds 2 files the “merge” button is activated. When the files reach the max number of files allowed in a single merge (5 currently) the “Add PDFs” button is disabled and the upload form is hidden.
- When the user is done adding PDFs, they can click “merge” which will merge the files on the server and activate only the download button.
- clicking the download button opens a new browser tab with the merged PDF.
UI & CSS:
For UI, the app uses daisyUI, a free component library that makes it easier to use TailwindCSS. The latter is super popular in the frontend world as a utility-first CSS framework.
Deployment:
I’ve not deployed the app to a real production server yet as the home server environment is very forgiving and you can skip some steps that you wouldn’t skip in a production deployment.
For now, I just have a basic Dockerfile and a docker-compose file to spin up the backend API (a regular Django project) and have it ready to accept calls from the frontend.
Likewise, a set of docker files is used to spin up the frontend. After building it using “npm run build”, the docker file copies the deployable app from the “dist” folder to the Nginx document root folder inside the docker container and just runs it as any other website hosted on a web server.
This setup is probably enough for development and hosting locally. When it comes time to publish the on the web, “real” deployment considerations must be taken.
It’s worth noting that I have a separate repo for backend and frontend to keep both ends decoupled from each other. The backend API can be consumed from any frontend be it a web, mobile or desktop app.
Further improvements:
The app in its current state works and does the intended job. It’s far from perfect and can use some improvements. I’ll include these in the README of the repos.
Source code:
- Backend: https://github.com/ahmedlemine/pdf-merger
- Frontend: https://github.com/ahmedlemine/pdfmerger-frontend
Conclusion
In this article I walked through the general process of how I built a web app for a Python script to make it available for use by average end users.
Python scripts are great starting points for apps. They’re also a source of inspiration for app ideas. If you have a script that performs a common daily-life task, consider building an app for it. The process will teach you a ton on the lifecycle of app development. It forces you to think of and account for new aspects you don’t usually consider when writing a stand-alone script.
As you build the app though, always remember to:
- keep it simple. Don’t over complicate things.
- ship fast. Aim at building an MVP (Minimum Viable Product) with the necessary functionality. Don’t wait until you’ve built every feature you think the app should have. Instead, ship it and then iterate on it and add features slowly as they’re needed.
- not to feel intimidated by other mature projects out there. They’ve been built most likely over a long period of time and have been iterated over tens or even thousands of times before they reached their mature state they are in today.
I hope you found this article helpful and I look forward to seeing you in a future one.