All Posts

Comments

Python packaging: Why we can't have nice things
Part 3: Premature Compilation

Pip 25.0 has been out for a bit over a month now; and we now also have an official blog post about the release, as well as a 25.0.1 patch for a regression.

Pip 25.0 has what I consider a very serious security vulnerability. In the Python ecosystem, it's normal and expected that third-party packages provide their own, arbitrary "setup" code for installation (for example, to run C compilers in project-specific ways, when the code uses a C extension). But Pip will run such code in many more situations than you might naively expect. I think it's obvious that running arbitrary code when you aren't expecting it and prepared for it is a much bigger problem. The user should have a chance to decide whether to trust the code, first.

I believe that warnings are more important than baiting people to read the post, so here's the PSA up front:

  1. Never use Pip to download, test, "dry-run" etc. an untrusted source distribution (sdist). It will try to build the package, potentially running arbitrary code (as building an sdist always entails). Instead, use the PyPI website directly, or the API it provides.

  2. Never use sudo to run Pip (nor run it with administrative privileges on Windows). Aside from the potential problems caused by conflicting with the system package manager, Pip will not drop privileges when it runs as root and attempts to build an sdist - which again, potentially runs arbitrary code.

  3. If you expect wheels to be available for the packages you want to install with Pip, strongly consider adding --only-binary=:all: to the Pip command to ensure that only wheels are used. If you really need to use sdists, it's wise to inspect them first, which by definition isn't possible with a fully automated installation.

  4. If you release Python packages, please try to provide wheels for them, even if - no, especially if your package includes only Python code and doesn't require explicitly "compiling" anything. An sdist is much slower to install than a wheel even in these cases, and making a wheel available allows your users to demand wheels from Pip - raising the overall baseline for trust and safety in the Python ecosystem.

Okay, I did clickbait a bit. This security issue isn't some new discovery. In fact, it has plagued Pip for its entire history.

Please enjoy my detailed analysis below.

Read more (35)…

Comments

Leaning IN to my U/X 

$ stat / | grep Birth | cut -d ' ' -f 3
2022-01-24

Today (although this won't go live until the 25th in my time zone) is three years since the day I finally kicked Windows to the curb and installed Linux on my home desktop. I'd used Linux before, but all my life I'd had a habit of just using whatever OS was provided to me, without really giving it much thought.

In late 2021, Windows Update told me - entirely unprompted - that my system didn't meet minimum specs to run Windows 11. (Perhaps it would now - I understand that the declared requirements have been relaxed, even if nothing has been optimized or stripped down - but I don't care to check. I'm also told that people still on Windows 10 are under increasing pressure to switch, even though they're still comfortably in the majority - unless something has radically changed in the last few months.)

I found this impolite - as I hadn't asked, and didn't consider anything wrong or inadequate about my computer - and a bit absurd (whatever they're offering now that requires more computing power, I'm not interested; especially not if it's anything to do with Cortana). So I took that as my queue to switch. I bought a new SSD (figuring I would need the space anyway) and attempted to set up a dual boot - GRUB never worked properly for me, but I could still use the BIOS screen to boot Windows from the old SSD.

And I never looked back.

Today, I'd like to relate a few anecdotes about that experience.

Read more (12)…

Comments

A Brief Annotation

I've been quite busy working on both the next article in my packaging series and on the overall appearance of the blog (I wasn't able to keep that confined to the weekend, apparently).

So, today, just a quick note, on the occasion of the 4th anniversary of the creation of PEP 649 "Deferred Evaluation Of Annotations Using Descriptors".

Yes, that's a mouthful, but in short: starting in Python 3.14, if you use annotations, you'll be able to defer the evaluation of the annotation code. (The feature was supposed to be added for 3.13, but didn't make it in.) That means you don't have to rely on strings for forward references in your type annotations, but you can still make full use of annotations at runtime (you'll have a proper object for the annotation itself, rather than just a string). Thank you to core Python developer Mr. Larry Hastings for putting a tremendous amount of effort into refining this proposal.

Now, I don't personally use type annotations very much - I don't use a type checker at all; I only write annotations as a form of documentation. But as it happened, when I was new to the Python Discourse forum, I came across Mr. Hastings' post, puzzling over the best way to name what will soon become the __annotate__ attribute of annotated objects.

The name __annotate__ was my suggestion.

I was not credited in the PEP, and I was ignored when I tried to point this out. So it falls to me to draw attention to my contribution.

This is all very niche stuff, of course - but I'm happy to have been able to leave this mark on the Python language itself, as opposed to just making useful things with it.

Comments

Python Packaging: Why we can't have nice things
Part 2: Stupid Pipx Tricks

Pip has a lot of problems (that I'll be discussing in future posts in this series), but the good news is that you don't have to resort to heavyweight third-party tools to improve your experience with Python packaging. Pipx (now under the Python Packaging Authority (PyPA) umbrella) is a focused wrapper around Pip that handles the major pain points without trying to take over your entire workflow.

In this post I'll talk about Pipx's major use cases, its limitations, and how to get more mileage out of it with a few simple tweaks.

Read more (10)…