Dependencies and isolated environments¶

Exercise dependencies-and-isolated-environments-annex.pdf · PDF

Your program works perfectly on your laptop. You send it to a friend. On her laptop it crashes on the first line. Same code, same file. "But it works on my machine!" Every developer in the world has said that sentence. It is always true, and it never helps.

The program did not change. The machine did. Today you find out what your program silently needs from the machine, why two projects on the same computer can fight over it, and how professionals make sure the program that runs today also runs on a colleague's laptop, and on yours in six months.

Eleven lines that broke the web¶

22 March 2016. All over the world, programmers start their day and their projects refuse to build. Nobody has touched the code. The error says a package called left-pad cannot be found.

left-pad was eleven lines of JavaScript that add spaces to the left of a text, written by Azer Koçulu. He had a dispute with a company over the name of another of his packages, lost it, and in protest deleted everything he had published on npm, the big public store of JavaScript packages. Thousands of projects used left-pad without knowing it, among them Babel, a tool a large part of the web was built with. They never asked for it directly: something they used, used something that used it.

Whose fault was it? There is no clean answer. And that is the point.

A dependency is code written by somebody else that your program needs in order to run.

Almost no program is written from zero, and dependencies have dependencies of their own. One small missing piece deep in the chain is enough to stop everything above it.

flowchart TB
    APP["<b>Your project</b><br>the dish you cook"] --> B["<b>Babel</b><br>a tool you asked for"]
    B --> X["<b>Another package</b><br>that Babel uses"]
    X --> LP["<b>left-pad</b><br>11 lines nobody knew about"]
    LP -.->|"deleted on<br>22 March 2016"| GONE["Build fails"]

The tomato sauce problem¶

Back to the kitchen. Your recipe says "add the house tomato sauce". The sauce is another cook's recipe: a dependency. Two problems appear in every real kitchen:

• The sauce recipe changes: version 2 has garlic. Your dish was tested with version 1.
• Two dishes need different versions of the same sauce, and the shelf only has room for one jar with that label.

Debian says no¶

On deb-lab, install the tools and try to install a package for the whole system:

$ sudo apt install python3-venv python3-pip
$ python3 --version
Python 3.13.5
$ pip install cowsay
error: externally-managed-environment
...

Read the whole message before searching the web. The cure is inside: it tells you to create a virtual environment using python3 -m venv.

Why does Debian refuse? Parts of the system itself are written in Python, and they rely on the versions that apt installed. A pip install for the whole system could replace one of them and break the system's own tools. The message even names the door it leaves open, --break-system-packages. The name is honest.

A shelf per project: the virtual environment¶

A virtual environment is a folder with its own copy of the Python commands and its own set of libraries, for one project.

In the kitchen, it is a private shelf per dish: each project keeps its own jar of sauce, in the version it needs.

mkdir ~/greeter
cd ~/greeter
python3 -m venv .venv            # create the environment in a hidden folder .venv
source .venv/bin/activate        # step into it

The prompt now starts with (.venv). Look at what changed:

(.venv) $ which python
/home/student/greeter/.venv/bin/python
(.venv) $ echo $PATH | tr ':' '\n' | head -1
/home/student/greeter/.venv/bin

That is all activate does: it puts one directory in front of PATH. The shell searches PATH in order, so python and pip now come from your project's folder.

Now install and run:

(.venv) $ pip install cowsay
Successfully installed cowsay-6.1
(.venv) $ pip list

pip list shows two packages, cowsay and pip. Write app.py with two lines:

import cowsay
cowsay.cow("Tramuntana Apps")

python app.py draws a cow that says Tramuntana Apps. Then leave the environment and try again:

(.venv) $ deactivate
$ python3 app.py
ModuleNotFoundError: No module named 'cowsay'

Same file, same machine, other environment. The library lives on the project's shelf, not on the system's.

"It works on my machine": write the list¶

The cure is old and simple: write the shopping list down, with exact versions, and keep it next to the code.

(.venv) $ pip freeze > requirements.txt
(.venv) $ cat requirements.txt
cowsay==6.1

==6.1 is the important part: not "cowsay", but this cowsay. On another machine, one command rebuilds the same shelf:

pip install -r requirements.txt

You can rehearse it on deb-lab: a new folder plays Pau's laptop.

deactivate
mkdir ~/pau-laptop
cd ~/pau-laptop
cp ~/greeter/app.py ~/greeter/requirements.txt .
python3 -m venv .venv
source .venv/bin/activate
python app.py                     # fails: the new environment is empty
pip install -r requirements.txt
python app.py                     # the cow appears

You copied only two files. The environment itself (.venv, about 12 MB here) is never sent: it holds paths of your own machine, and it can be thrown away and rebuilt from the list in seconds.

When dependencies disagree¶

Sometimes two dependencies of the same project disagree with each other. Then the installer stops and tells you:

(.venv) $ pip install "requests==2.25.0" "urllib3==2.2.0"
...
The conflict is caused by:
    The user requested urllib3==2.2.0
    requests 2.25.0 depends on urllib3<1.27 and >=1.21.1
...
ERROR: ResolutionImpossible

Read it from the bottom up: requests 2.25.0 demands a urllib3 older than 1.27, and you asked for 2.2.0. Both cannot be on the same shelf, so nothing is installed. That is good: a half-installed project would be worse.

Three sizes of isolation¶

A virtual environment isolates the libraries of one language. Sometimes that is not enough: the program also needs a certain Python version, a database, a system library. There are bigger boxes.

flowchart TB
    subgraph VM ["Virtual machine: its own kernel"]
        subgraph CT ["Container: shares the host's kernel"]
            subgraph VE ["Virtual environment: one project's libraries"]
                LIB["cowsay==6.1"]
            end
        end
    end

In March 2013, at the PyCon conference in California, Solomon Hykes showed a tool called Docker in a five-minute talk. The idea: do not send only the program; send it packed together with everything it needs. You will build containers later in the course. Virtual machines you have used since Unit 2: deb-lab is one.

Sizing laptops for three people¶

Tramuntana Apps has three laptops and three people. Each laptop goes to exactly one person. The method: add up the official requirements of what each person runs at the same time (the numbers come from What a development machine needs).

Only one assignment works for all three. Marta on B would start, since 16 GB is her minimum, but then Biel would have to live on A, where 12 GB do not fit. And if B had to serve Marta, what would you buy first? RAM.

Nobody asked which laptop is the best. The question was: which one is enough, for this person, for this work. That is what evaluating the needs of a system means.

Now do the whole session yourself, as Marta and then as Pau, on the Exercise sheet (button at the top of the page).

Key takeaways¶

• A dependency is code by somebody else that your program needs; dependencies have their own dependencies, and one missing piece (like left-pad in 2016) can stop everything above it.
• Debian refuses a system-wide pip install (externally-managed-environment) because system tools are written in Python. Never sudo pip install.
• A virtual environment (python3 -m venv .venv) is a folder plus a change in PATH: activate puts .venv/bin first. One environment per project.
• pip freeze > requirements.txt writes the list with exact versions; pip install -r requirements.txt rebuilds it anywhere. Send the code and the list, never .venv.
• Three sizes of isolation: virtual environment (libraries), container (all the software, shares the kernel), virtual machine (its own kernel).
• To size a workstation, add up what each person runs at the same time and pick what is enough, not the best.

Quick check¶

That closes Unit 5. Next: your network gets its first server. Unit 6 builds the Tramuntana office server, starting with Clients, servers and shared resources.