Python in Emacs: Vanilla is a Powerful Flavor

Emacs typically has commands that are specific to the active major mode bound to C-c C-<letter>. What each <letter> does will depend on the buffer you're currently in and what major mode is active. In our case, that's python-mode, which has a lot of handy shortcuts already mapped out. For any of the keyboard shortcuts you can always use C-h k, or C-h f for the function names (prefixed by M-x below) to get the official documentation.

Run Python: python3 -i█

Run Python: poetry run█python3 -i

# editing.py
# --snip--
# Here we use `C-c C-t d` and follow the prompts to design a new
# function signature.
def whatever(my_string: str = hello_text, my_integer: int = 0):
    """Whatever, man"""
    return f"{hello_text}, {my_integer}"

# Next, `C-c C-t c` to make a new class
class MyGuy:
    """My guy is ALWAYS there for me"""
    pass
# --snip-- "__main__"

The nimble, builtin imenu is a way to quickly navigate between major symbol definitions in the current buffer - especially those off screen. In our editing.py we now have three functions, hello_text(), world_text(), and whatever(), and one class MyGuy. If we use C-c C-j, a minibuffer menu like this comes up:

1/5 Index item: █
*Rescan*
MyGuy.(class)
whatever.(def)
world_text.(def)
hello_text.(def)

My minibuffer displays a vertical preview of the options because I've set (fido-mode) and (vertical-fido-mode) in my init.el, both of which are included in Emcacs 28.1 or later. Then, if I partially type out a result the list will filter down to possible completions:

1/1 Index item: My█
MyGuy.(class)

imenu is very, very handy across Emacs, not just for Python, so it's worth trying in a variety of major modes.

This mode has built-in error parsing, so it's the best way to run a script for real if we want to quickly navigate any traceback messages that come up. Conversely, the M-& async shell command does not have error parsing, so it's not the right tool for launching processes we have to debug. Same goes for booting up a shell and running Python from there. Taking our script from the previous section, if we run M-x compile and give it an argument of python3 editing.py, up pops the *compilation* buffer, with the starting time, output of our program, and finish time.

-*- mode: compilation; default-directory: "~/repos/renzmann.github.io/content/posts/006_emacs_2_python/" -*-
Compilation started at Sun Aug 14 13:50:39

python3 editing.py
Hello, world!

Compilation finished at Sun Aug 14 13:50:39

Now, let's try a different script, with an error in it:

# hello_error.py
print("Not an error yet!")
fdafdsafdsafdsa
print("Shouldn't make it here...")

Now, M-x compile will error out:

-*- mode: compilation; default-directory: "~/repos/renzmann.github.io/content/posts/006_emacs_2_python/" -*-
Compilation started at Sun Aug 14 13:53:26

python3 hello_error.py
Not an error yet!
Traceback (most recent call last):
  File "/home/robb/repos/renzmann.github.io/content/posts/006_emacs_2_python/hello_error.py", line 4, in <module>
    fdafdsafdsafdsa
NameError: name 'fdafdsafdsafdsa' is not defined

Compilation exited abnormally with code 1 at Sun Aug 14 13:53:26

Emacs will parse the error message, so that after "compiling", we can use M-g M-n and M-g M-p to move between error messages, or just click the link provided by the *compilation* buffer directly.

If just parsing Python tracebacks doesn't excite you, mypy is also supported out of the box. Assuming mypy is already installed, M-x compile with mypy hello_error.py as the command results in this:

-*- mode: compilation; default-directory: "~/repos/renzmann.github.io/content/posts/006_emacs_2_python/" -*-
Compilation started at Sun Aug 14 14:02:03

.venv/bin/mypy hello_error.py
hello_error.py:4: error: Name "fdafdsafdsafdsa" is not defined
Found 1 error in 1 file (checked 1 source file)

Compilation exited abnormally with code 1 at Sun Aug 14 14:02:04

The hello_error.py:4: error: ... message will be a functional link, just as before. mypy is much more suitable for general error-checking though, so as scripts (and bugs) grow, the M-x compile command can keep up:

# errors.py
import typing

import requests
import aaaaaaa

foo
print(typing.fdafdsafdsafdsafdsafdsafdsa)


def whatever(x: str) -> str:
    """Here's a docstring!"""
    return x + 1

M-x compile RET mypy errors.py

-*- mode: compilation; default-directory: "~/repos/renzmann.github.io/content/posts/006_emacs_2_python/" -*-
Compilation started at Sun Aug 14 14:06:55

.venv/bin/mypy errors.py
errors.py:6: error: Cannot find implementation or library stub for module named "aaaaaaa"
errors.py:6: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
errors.py:8: error: Name "foo" is not defined
errors.py:9: error: Module has no attribute "fdafdsafdsafdsafdsafdsafdsa"
errors.py:14: error: Unsupported operand types for + ("str" and "int")
Found 4 errors in 1 file (checked 1 source file)

Compilation exited abnormally with code 1 at Sun Aug 14 14:06:55

Now, we can use M-g M-n and M-g M-p to quickly navigate between the errors in our code, even after navigating away from the original errors.py buffer - Emacs will remember what's going on in the *compilation* buffer so we can hop all around the code base while addressing errors one at a time.

python-mode centers heavily around the use of an active, running Python session for some of its features, as we'll see in the next section. Its documentation recommends regular use of C-c C-c, which sends the entire buffer to the active inferior Python process. That means actually executing Python code, which may feel a bit dangerous for those of us who grew up with static analysis tools. So the first thing we need to make sure we don't accidentally kick off our whole script is ensure that the main part of our program is properly ensconced.

# editing.py
# --snip--
if __name__ == "__main__":
    print(f"{hello_text()}, {world_text()}!")

Simply populates the command to run with python -m pdb. Can be configured with the variable gud-pdb-command-name

Unlike the mypy output, the error messages from pyright aren't links, and we can't hop between messages using M-g M-n and M-g M-p like before. In order to gain this functionality, we need to add a regex that can parse pyright messages. There are two objects of interest to accomplish this:

• compilation-error-regexp-alist
• compilation-error-regexp-alist-alist

Here's the formal description from C-h v compilation-error-regexp-alist:

Alist that specifies how to match errors in compiler output.
On GNU and Unix, any string is a valid filename, so these
matchers must make some common sense assumptions, which catch
normal cases.  A shorter list will be lighter on resource usage.

Instead of an alist element, you can use a symbol, which is
looked up in ‘compilation-error-regexp-alist-alist’.

In not so many words, this says we should modify the *-alist-alist version, and simply add a symbol to the *-alist variable. Examining the current value via C-h v compliation-error-regexp-alist-alist, it's easy to see that we're after an expression a bit like this,

(add-to-list 'compilation-error-regexp-alist-alist
             '(pyright "regexp that parses pyright errors" 1 2 3))

eventually replacing the string in the middle with an actual Emacs regexp. Thankfully, Emacs has the M-x re-builder built in for doing exactly that! Since *Python check: poetry run pyright errors.py* is a buffer like any other, we can hop over to it, and run M-x re-builder to piece together a regex that extracts file name, line number, and column number from each message.

Clearly, there are some errors in the regexp so far, but as we edit the text in the *RE-Builder* buffer, the highlighting in the *compilation* buffer will update live to show us what would be captured by the regexp we've entered. After fiddling with the contents in the bottom buffer to get the highlighting correct, we've got this regular expression:

"^[[:blank:]]+\\(.+\\):\\([0-9]+\\):\\([0-9]+\\).*$"

Now we just need to add this into the compilation-error-regexp-alist-alist in our init.el:

;; init.el
(require 'compile)
(add-to-list 'compilation-error-regexp-alist-alist
             '(pyright "^[[:blank:]]+\\(.+\\):\\([0-9]+\\):\\([0-9]+\\).*$" 1 2 3))
(add-to-list 'compilation-error-regexp-alist 'pyright)

After restarting emacs with the modified alist, we get error prasing from pyright output:

Keeping a .venv folder at the top level of a project is one valid way to organize things, but (vanilla) Emacs isn't going to make it easy for us to use it that way. Instead, I'd recommend keeping all virtual environments in a central place. For me, that looks like this:

M-! python3 -m venv ~/.cache/venvs/website

This builds a virtualenv named website for python utilities that help buld my blog under the ~/.cache directory on Unix. To use this virtualenv explicitly for shell utilities, I can always run commands like this

M-! ~/.cache/venvs/website/bin/python -m pip install mypy
M-! ~/.cache/venv/website/bin/mypy errors.py

Of course, adding the prefix ~/.cache/venvs/website/bin every time is a bit cumbersome, especially for frequent commands like M-x python-check.

One quick way to reduce some typing is to add entries in a project file called .dir-locals.el. This is a special data file that Emacs will read, if it exists, and apply to all new buffers within the project. For our needs, we want to apply a couple changes to python-mode specifically to use the virtual environment instead of system python. The two easy ones are the python-check-command and python-shell-virtualenv-root:

;; .dir-locals.el
((python-mode . ((python-check-command . "%HOME%\\.cache\\venvs\\website\\Scripts\\python.exe -m mypy")
                 (python-shell-virtualenv-root . "~/.cache/venvs/website"))))

I've included a quirk of working on Microsoft Windows here - the python-check-command needs to run through your shell, which is cmd.exe by default, and hence requires Windows-style paths. The python-shell-virtualenv-root, however, is evaulated by Emacs, and can use tilde-expansion and Unix-style paths. Changing default shell commands to run through pwsh on Windows would likely alleviate this issue, but it's worth calling out for cmd.exe users.

It's also worth mentioning here that M-x add-dir-local-variable provides an easy interactive interface to editing the .dir-locals.el file.

The python-shell-virtualenv-root part only affects running Python as a shell within Emacs, it does not affect things like PATH, async commands, or M-x compile. To demonstrate this, once we've set up .dir-locals.el as above, and we either revert a Python buffer with C-x x g or open a new Python buffer in the same project, a popup like this appears:

The local variables list in c:/Users/robbe/repos/renzmann.github.io/content/posts/006_emacs_2_python/
contains values that may not be safe (*).

Do you want to apply it?  You can type
y  -- to apply the local variables list.
n  -- to ignore the local variables list.
!  -- to apply the local variables list, and permanently mark these
      values (*) as safe (in the future, they will be set automatically.)
i  -- to ignore the local variables list, and permanently mark these
      values (*) as ignored

  * python-check-command : "%HOME%\\.cache\\venvs\\website\\Scripts\\python.exe -m mypy"
  * python-shell-virtualenv-root : "~/.cache/venvs/website"

Responding with y will set the python-check-command and python-shell-virtualenv-root for just the current session, while ! will add both of these values to the custom section in either init.el or wherever you've set your custom-file. This is another reason for using a common, central spot for virtual environments, since across workstations I can use the same path relative to my $HOME directory. After confirming, and using C-c C-p, we can check which Python executable we're using in the *Python* buffer now:

Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.executable
'c:\\Users\\robbe\\.cache\\venvs\\website\\Scripts\\python.exe'

Keep in mind, the values provided in .dir-locals.el are evaluated on a per-buffer basis, so attempting to set a relative path like (python-shell-virtualenv-root . ".venv/website") will only work when executing run-python in the same directory as .dir-locals.el and .venv/.

The various compile and shell commands will not respect the virtualenv we've set via .dir-locals.el. On *nix, M-x compile RET which python3 will still bring back some variant of /usr/bin/python3, as will M-& which python or M-! which python. In a follow-up article we might explore how it is possible to take care of all this via .dir-locals.el and the special exec variable, but it's not very elegant.

pyvenv is a very lightweight package, clocking in at around 540 source lines of code, designed specifically around the challenge of ensuring the correct python virtual environment is at the front of PATH when running (async) shell commands, M-x eshell, M-x shell, M-x term, M-x python-check, M-x compile, and more. When written, it was based around virtualenv and virtualenvwrapper.sh, and some of the language it uses will reflect that. Although virtualenv has mostly fallen out of favor, the core functionality of pyvenv is still very relevant. Especially if you choose to adopt a central store of virtual environments, as above, you can set that as a WORKON_HOME variable ("workon" is terminology held over from virtualenvwrapper.sh) to a directory that all your virtual environments sit under, so that it's easy to select one with the pyvenv-workon function. When using poetry, that usually looks like this:

(if (eq system-type 'windows-nt)
    ;; Default virtualenv cache directory for poetry on Microsoft Windows
    (setenv "WORKON_HOME" "$LOCALAPPDATA/pypoetry/Cache/virtualenvs")
  ;; Default virtualenv cache directory for poetry on *nix
  (setenv "WORKON_HOME" "~/.cache/pypoetry/virtualenvs"))
(pyvenv-mode)

Setting WORKON_HOME to ~/.cache/venvs as in the previous examples is another valid option. Doing it this way also plays nice with .dir-locals.el, since pyvenv exposes a way to set a project-level venv with a single variable:

;; .dir-locals.el
((python-mode . ((pyvenv-workon . "website"))))

Also of use for folks who frequently swap between different projects is (pyvenv-tracking-mode), which will automatically change the active python virtual environment when you navigate to a different buffer.

And, of course, if the whole "workon" and virtualenvs grouped together under ~/.cache/venvs isn't to taste, there's always M-x pyvenv-activate, which lets you choose a virtual environement anywhere on your system. So, all-in-all, I'll probably stick with pyvenv in my configuration, because setting all the different utility PATHs without it is just such a pain.