Virtual environments are vital if you’re developing Python apps or just writing some Python scripts. They allow you to isolate different app requirements to prevent conflicts and keep your global OS environment clean.
This is super important for many reasons. The most obvious one is requirements isolation. Let’s say that you’re working on two different projects:
- Project #1 requires Python 3.6 and the “requests” package version 2.1.0.
- Project #2 requires Python 3.10 and requests 2.31.0.
Although you can install both Python 3.6 and Python 3.10 globally on your OS, you may have to manually set the default version which may end up breaking some packages in your OS that expect the other version.
As for other packages, “requests” in this example, installing multiple versions of the same Python package globally is not possible (as far as I know).
The solution is to create a virtual environment, or “venv”, for each project which will isolate it from the other environment and, most importantly isolate it from your OS global env. Each venv has its own folder that contains all the packages and libraries the project needs to run correctly including a copy of Python itself. Then you can activate “venv1” when you’re working on Project #1 or when you finally deploy it or run it as a service or a process. When it comes time to work on Project #2, you just deactivate “venv1” and activate “venv2”.
Better yet, you can have two separate terminal or IDE windows open at the same time, one for each project with its own “venv” activated.
Another great advantage of venvs is the ability to replicate the project’s environment on almost any computer regardless of its OS or currently installed packages. This eliminates the popular “well, it runs on my laptop” excuse when your code doesn’t run on other team members’ machines. Everybody can recreate the exact project’s environment on their machine using venvs and the requirements list conventionally saved in the “requirements.txt” file.
Now, the most common place that you’ll see developers put their virtualenvs is within the same directory where the project they’re developing is saved. This has some great benefits but it also has major drawbacks. I will briefly mention both the pros and cons of this approach then I will suggest a better place for your ‘venv’ folder, the one I actually do and prefer.
Benefits of creating the “venv” inside the project folder:
Easier to create:
It’s easier when you create a new project to “cd” into it and create the “venv” right inside it.
Easier to activate:
It’s also easier for you to activate the venv by just running “source venv/bin/activate” from within the project folder. No need to remember where this project’s venv lives on your disk and the absolute or relative path to it.
One bundle:
When you create the venv inside the project folder, it’s always included and bundled with the project. If you copy the project, move it or send it to a team member, the venv is right there with it. No need to recreate it or “pip install” the requirements again (not sure about this last one).
IDE optimization:
Most IDE’s will discover your venv folder if it’s directly in your project’s folder. Some IDE’s will activate the venv automatically for you, while others will at least use it as the default venv without requiring you to manually set it for each project.
Drawbacks of creating the “venv” inside the project folder:
Committing venv to Git is gross:
When you have your venv inside your project’s folder, you risk committing it to Git or your version control system either by accident or unknowing that it is a bad practice.
Virtual environments can contain thousands of files and their size can be in gigabytes. Committing them to Git can overload and clutter your source code repo with unnecessary files and cause confusion for anyone trying to clone and run the source code on their machine.
I can hear you saying that the venv folder should be added to the .gitignore file and thus never committed to Git. This is true but remember that the .gitignore file is not created automatically when you initialize a Git repo. You must manually create it and manually edit it to exclude the venv folder, or use a common template for Python projects. You can easily forget to do this step and won’t notice until you see the venv folder on Github. Simply adding it then to .gitignore file won’t remove it completely from Git history which makes it look unprofessional.
Making the project difficult to backup and sync:
Venvs can get huge both in size and number of files. If you’re using any system to backup your laptop or sync your home/user folder to a remote backup location such as a NAS or Nextcloud/Dropbox/Google Drive, you’ll make your life miserable by backing up your venv folders.
I know, you can set some rules to ignore venvs but believe me, I’ve tried to do so but I always forget to add newly created projects to the list of ignored files.
I often don’t notice until it’s too late. This usually comes after a backup job takes too long to complete then I discover after some digging that a venv folder is being backed up to my NAS or even worse, being uploaded to my pCloud folder.
The Solution: A better place for your virtualenvs
Virtual environments are disposable in nature. They contain packages downloaded from the internet and should not be backed up, sync’ed, copied or moved around. They also are re-producible. Using the “requirements.txt” file that’s ideally included in every Python project, you can reinstall all the content of the venv folder anywhere you have an internet connection.
With that in mind, you should not worry about loosing the venv folder or sharing the project with that folder missing. If the person you share it with has any basic Python knowledge they should be able to replicate its venv easily.
This suggests that you should create the venv in a separate folder outside the project’s folder. The question is “Where?”
Keep all your Virtualenvs in a disposable, central folder outside the project’s folder:
As the heading suggests, the ideal place for venvs is a central folder in a disposable location on your disk outside your project’s folder. This can be your “Downloads” , “var/tmp/” or even “/tmp” folder. Basically any place that any backup or sync software would NOT backup/sync by default and would automatically ignore. Most if not all backup/sync apps and systems will ignore your “Download” folder and “/var/tmp” folder. For me, it’s my Downloads folder where I keep a sub folder of all my venvs.
This guards against human error when you forget to ignore the venv folder manually. You don’t even have to think about it.
So, the venv location must be:
- outside.
- disposable.
- central.
Here’s a breakdown of why:
Why outside the project’s folder?
So that you avoid drawbacks I mentioned above:
- Automatically ignored from Git:
Because they live outside the Git repo, they will never get a chance to be committed or even added to the tracked files. - Clutter-free source code:
You won’t commit them to Git. They won’t clutter and overload your source code with unnecessary files. As you probably know, you repo should only contain the code you write or incorporate in your project in addition to non-reproducible assets, and NOT source code for packages and libraries you download over the internet, i.e. venvs. - Automatically excluded from Backups and Sync:
When you backup or sync a project, you usually wouldn’t include any outside folders in the process.
Why a disposable folder?
Simply put, venvs can be reproduced anytime. No need to back them up as you would your personal documents and valuable source code and assets.
So, having them in a disposable location, like the Downloads folder is ideal. You don’t have to worry about excluding them from backups or sync. They’re excluded by default.
Why central location for all venvs?
There are several advantages you will gain by having all your venvs in one central folder on your disk:
Easily exclude them from backup and sync:
There are many consequences for backing up venv folders or sync’ing them to the cloud.
Most backup and sync software have a way of listing the files/folders you want to exclude. This varies between those systems. Some have an option in the GUI where you specify the list, some require you to include a hidden file with a specific name (like .nobackup in the case of Borg) in every folder you want to exclude. And some simply require that you type out “–exclude {folder name}” in the command that triggers the backup like it’s the case with rsync.
Whatever the case may be, you must do it manually for every folder. However, by having all your venvs under one, and only one, folder, you only have to do this exclusion thing once for the parent folder and be done with it. Every child folder will inherit this and be excluded automatically.
Easily remember the path to your venvs:
If you just apply my first suggestion and create your venv outside your project’s folder, each of your projects will have a different location and you’ll find it hard to remember where each project’s venv is.
By having all your venvs under one folder like “~/Downloads/venvs/” for example, it’s very easy to link to your venv from anywhere on your system.
An Example Setup:
Finally, an ideal setup I suggest and actually have been doing for years now is as follows:
- The location I use for my venvs is:
~/Downloads/virtual-envs/ . That is: my home directory > Downloads > virtual-envs. - When I start a new project, I immediately create a virtualenv for it under my central venvs folder (~/Downloads/virtual-envs/) using this command:
Python3 -m venv ~/Downloads/virtual-envs/project1- Then I activate the venv. For a quick completion of the path, I hit “Alt +.” (that’s the Alt key and the “dot” key) to add the last argument I gave to the most recent command, which is in this case “~/Downloads/virtual-envs/project1”. So, instead of typing the path again, the activation command becomes:
source + “Alt” + “.” + /bin/activate
I even make it quicker by hitting “Tab” after /b and /a in the path so that it auto-completes “bin” and “activate” words respectively for me.
That’s it. You now have a better setup for your venvs. You don’t have to worry about adding them to the .gitignore file or excluding them from backup and sync software anymore.