So I’m no expert, but I have been a hobbyist C and Rust dev for a while now, and I’ve installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn’t work in python, it seems more obtuse than any other language’s ecosystem. Why is it like this?
Python is the only programming language that has forced me to question what the difference is between an egg and a wheel.
No, it’s not just you, Python’s tooling is a mess. It’s not necessarily anyone’s fault but there are a ton of options and a lot of very similarly naked things that accomplish different (but sometimes similar) tasks. As someone who considers themselves between beginner and intermediate proficiency in Python this is my biggest hurdle right now.
Python developer here. Venv is good, venv is life. Every single project I create starts with
python3 -m venv venv
source venv/bin/activate
pip3 install {everything I need}
pip3 freeze > requirements.txt
Now write code!
Don’t forget to update your requirements.txt using pip3 freeze again anytime you add a new library with pip.
If you installed a lot of packages before starting to develop with virtual environments, some libraries will be in your OS python install and won’t be reflected in pip freeze and won’t get into your venv. This is the root of all evil. First of all, don’t do that. Second, you can force libraries to install into your venv despite them also being in your system by installing like so:
pip3 install --ignore-installed mypackage
If you don’t change between Linux and windows most libraries will just work between systems, but if you have problems on another system, just recreate the whole venv structure
rm -rf venv (…make a new venv, activate it) pip3 install -r requirements.txt
Once you get the hang of this you can make Python behave without a lot of hassle.
This is a case where a strength can also be a weakness.
Okay, now give me those steps but what to do if I clone an already existing repo please
The git repo should ignore the venv folder, so when you clone you then create a new one and activate it with those steps.
Then when you are installing requirements with pip, the repo you cloned will likely have a requirements.txt file in it, so you ‘pip install -r requirements.txt’
This is the way
Yes it’s terrible. The only hope on the horizon is
uv
. It’s significantly better than all the other tooling (Poetry, pip, pipenv, etc.) so I think it has a good chance of reducing the options to just Pip oruv
at least.But I fully expect the Python Devs to ignore it, and maybe even make life deliberately difficult for it like they did for static analysers. They have some strange priorities sometimes.
I like the idea of
uv
, but I hate the name. Libuv is already a very popular C library, and used in everything from NodeJS to Julia to Python (through the popularuvloop
module). Every time I see someone mentionuv
I get confused and think they’re talking about uvloop until I remember the Astral project, and then reconfirm to myself how much I disapprove of their name choice.
The venv stuff is pretty annoying, I agree.
As a baby Python Dev, I’m glad it’s not just me.
I’ve been full time writing python professionally since 2015. You get used to it. It starts to just make sense and feel normal. Then when you move to a different language environment you wonder why their tooling doesn’t use a virtualenv.
No it’s not. E.g. nobody who starts a new project uses setup.py anymore
OP seems to be trying to install older projects, rather than creating a new project.
Just out of curiosity, I haven’t seen anyone recommend miniconda… Why so, is there something wrong I’m not aware of?
I’m no expert, but I totally feel you, python packages, dependencies and version matching is a real nightmare. Even with
venv
I had a hard time to make everything work flawlessly, especially on MacOS.However, with miniconda everything was way easier to configure and worked as expected.
Isn’t conda specifically for mathy things?
I’ve started using poetry and the experience has improved.
The reason you do stuff in a venv is to isolate that environment from other python projects on your system, so one Python project doesn’t break another. I use Docker for similar reasons for a lot of non-Python projects.
A lot of Python projects involve specific versions of libraries, because things break. I’ve had similar issues with non-Python projects. I’m not sure I’d say Python is particularly worse about it.
There are tools in place that can make the sharing of Python projects incredibly easy and portable and consistent, but I only ever see the best maintained projects using them unfortunately.
Python is hacky, because it hacks. There’s a bunch of ways you can do anything. You can run it on numerous platforms, or even on web assembly. It’s not maintained centrally. Each “app” you find is just somebodies hack project they’re sharing with you for fun.
Python is the new Perl
On that note, I’m hesitant between writing my scripts in perl or python right now. Bash prevent sharing with Windows peoples… I just want to provide easy wrappers tools that are usually aroud 10 lines of shell, but testers ain’t on linux so they cannot use them.
I don’t know perl, but each time I interract with pyton’s projects I have a different venv/poetry/… to setup. Forget adout it the next time and nothing is kept easy to reuse.
Perl isn’t really any better. There aren’t easy tools that do the same thing as venv. They exist, but they are not easy. Plus there are a much larger amount of cpan modules that have c in them than python.
After using python, I’m of the opinion that perl was much cleaner.
Yes. Its line noise was of a much higher quality. 😉
Yep, they are not portable, every app should come bundled with its own interpreter. As to why, I think historically it didn’t target production grade application development.
Python’s packaging is not great. Pip and venvs help but, it’s lightyears behind anything you’re used to. My go-to is using a venv for everything.
It’s something of a “14 competing standards” situation, but uv seems to be the nerd favourite these days.
I still do the python3 -m venv venv && source venv/bin/activate
How can uv help me be a better person?
If you’re happy with your solution, that’s great!
uv combines a bunch of tools into one simple, incredibly fast interface, and keeps a lock file up to date with what’s installed in the project right now. Makes docker and collaboration easier. Its main benefit for me is that it minimizes context switching/cognitive load
Ultimately, I encourage you to use what makes sense to you tho :)
And pip install -r requirements.txt
Fuck it, I just use sudo and live with the consequences.
the software equivalent of leaving the dirt on your vegetables to harden your immune system
Oh no
This! Haven’t used that one personally, but seeing how good ruff is I bet it’s darn amazing, next best thing that I used has been PDM and Poetry, because Python’s first party tooling has always been lackluster, no cohesive way to define a project and actually work it until relatively recently
I moved all our projects (and devs) from poetry to uv. Reasons were poetry’s non standard pyproject.toml syntax and speed, plus some weird quirks, e. g. if poetry asks for input and is not run with the verbose flag, devs often don’t notice and believe it is stuck (even though it’s in the default project README).
Personally, I update uv on my local machine as soon as a new release is available so I can track any breaking changes. Couple of months in, I can say there were some hiccups in the beginning, but currently, it’s smooth sailing, and the speed gain really affects productivity as well, mostly due to being able to not break away from a mental “flow” state while staring at updates, becoming suspicious something might be wrong. Don’t get me wrong, apart from the custom syntax (poetry partially predates the pyproject PEP), poetry worked great for us for years, but uv feels nicer.
Recently, “uv build” was introduced, which simplified things. I wish there was an command to update the lock file while also updating the dependency specs in the project file. I ran some command today and by accident discovered that custom dependency groups (apart from e. g. “dev”) have made it to uv, too.
“uv pip” does some things differently, in particular when resolving packages (it’s possible to switch to pip’s behavior now), but I do agree with the decisions, in particular the changes to prevent “dependency confusion” attacks.
As for the original question: Python really has a bit of a history of project management and build tools, I do feel however that the community and maintainers are finally getting somewhere.
cargo is a bit of an “unfair” comparison since its development happened much more aligned with Rust and its whole ecosystem and not as an afterthought by third party developers, but I agree: cargo is definitely a great benchmark how project and dependency management plus building should look like, along with rustup, it really makes the developer experience quite pleasant.
The need for virtual environments exists so that different projects can use different versions of dependencies and those dependencies can be installed in a project specific location vs a global, system location. Since Python is interpreted, these dependencies need to stick around for the lifetime of the program so they can be imported at runtime. poetry managed those in a separate folder in e. g. the user’s cache directory, whereas uv for example stores the virtual environment in the project folder, which I strongly prefer.
cargo will download the matching dependencies (along with doing some caching) and link the correct version to the project, so a conceptual virtual environment doesn’t need to exist for Rust. By default, rust links everything apart from the C runtime statically, so the dependencies are no longer neesed after the build - except you probably want to rebuild the project later, so there is some caching.
Finally, I’d also recommend to go and try setting up a project using astral’s uv. It handles sane pyproject.toml files, will create/initialize new projects from a template, manages virtual environments and has CLI to build e. g. wheels or source distribution (you will need to specify which build backend to use. I use hatchling), but thats just a decision you make and express as one line in the project file. Note: hatchling is the build backend, hatch is pypa’s project management, pretty much an alternative to poetry or uv.
uv will also install complete Python distributions (e. g. Python 3.12) if you need a different interpreter version for compatibility reasons
If you use workspaces in cargo, uv also does those.
uv init, uv add, uv lock --upgrade, uv sync, uv build and how uv handles tools you might want to install and run should really go a long way and probably provide an experience somewhat similar to cargo.
I think you responded to the wrong comment, I didn’t question the need for uv or other tools like that
everyone focuses on the tooling, not many are focusing on the reason: python is extremely dynamic. like, magic dynamic you can modify a module halfway through an import, you can replace class attributes and automatically propagate to instances, you can decompile the bytecode while it’s running.
combine this with the fact that it’s installed by default and used basically everywhere and you get an environment that needs to be carefully managed for the sake of the system.
js has this packaging system down pat, but it has the advantage that it got mainstream in a sandboxed isolated environment before it started leaking out into the system. python was in there from the beginning, and every change breaks someone’s workflow.
the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.
You re not stupid, python’s packaging & versionning is PITA. as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
A perfect summary of the history of computer code!