Thursday, November 28, 2024

Python, pre-commit, and black

If you've had trouble configuring a python project with local repo copies of black and get errors you can't explain, this may help. Maybe.

I ran into some config issues somewhere between my pre-commit config and my black config. In case someone else runs into this I wanted to leave some bread crumbs.

The issue. Black keeps exiting with "exit code: 123" after it tries formatting non-python files. It then told me it couldn't format files that weren't python files: yaml, md, and toml. I'd thought I'd excluded everything I needed to, but something was still tripping me up.

BLUF (bottom line up front). Specify "types: [python]" for black to make it parse only python files.

The setup. This is a small personal python project with pandas, numpy, and xlsxwriter. Because in my day job my test environment is completely disconnected from the Internet I wanted to set this up with "repo: local" so I could test that kind of config. There's just not a ton of info I've found online on using local pre-commit hooks, which is funny in its own way. I'm not saying that there's /no/ documentation, since https://pre-commit.com/#repository-local-hooks exists and Anthony Sottile of pre-commit fame has commented on many stackoverflow.com questions. I just had trouble finding "the answer."

I'm on a System76 laptop running Pop!_OS 22.04 (hopefully they'll finish the COSMIC desktop soon!), so it defaults to python 3.10-vintage software.

The errors. My pre-commit config parsed fine, as did the pyproject.toml file for black.

(.venv) andrew@bun-bun:~/src/myproj$ pre-commit validate-config
(.venv) andrew@bun-bun:~/src/myproj$ pre-commit

black....................................................................Failed

- hook id: black
- exit code: 123

error: cannot format .pre-commit-config.yaml: Cannot parse for target version Python 3.10: 1:6: repos:

error: cannot format README.md: Cannot parse for target version Python 3.10: 4:15: andrew@bun-bun: [text from this line]

error: cannot format pyproject.toml: Cannot parse for target version Python 3.10: 20:8:   [text from this other line]

Oh no! 💥 💔 💥
3 files failed to reformat.

check-toml...............................................................Passed

Well, at least I got check-toml configured correctly!

The configs. I've not tried this setup before and I'm still unclear about what's supposed to be in pyproject.toml and what's supposed to be in .pre-commit-config.yaml so I'd not copy and paste this in toto. I started with .pre-commit-config.yaml:

repos:

-   repo: local
    hooks:

    -   id: black
        name: black
        entry: black
        language: python
        types: [python]
        args: [--line-length=95]
        exclude: .yaml$

    -   id: check-toml
        name: check-toml
        language: system
        entry: check-toml
        files: toml

The toml config seemed to work, at least after I did a "pip install pre-commit-hooks" at some point. But if you leave out the "types" line, black tries to do all files, which you don't want. My "exclude:" line also didn't work, but I see as I type this up that I had "exclude: .yaml^" instead of "exclude: .yaml$", so that's a bit of a mess. Whoops!

I tried to exclude various text files in my pyproject.toml file, too, but it was the "types: [python]" line that really did the trick. Here's my [tool.black] section. Some I've obviously grabbed from somewhere else (who needs to exclude both git and mercurial directories?) but I was willing to try. The data I'm munging (various excel spreadsheets) lives under "data" so I wanted to exclude all of that. (The data is also not checked into git.)

[tool.black]
line-length = 95
target-version = ['py310']
include = '\.pyi?$'
exclude = '''
    /(
        \.git
      | \.hg
      | \.mypy_cache
      | \.tox
      | \.venv
      | _build
      | buck-out
      | build
      | dist
      | \.toml
      | \.md
      | \.yaml
      | data
    )/
    '''

Final trick. If you're trying to figure out what labels the files have, use "identify-cli". Some file checks are "AND", some are "OR."

This'll give you some idea of what popular values are.

(.venv) andrew@bun-bun:~/src/myproj$ identify-cli .pre-commit-config.yaml
["file", "non-executable", "text", "yaml"]
(.venv) andrew@bun-bun:~/src/myproj$ identify-cli library.py
["file", "non-executable", "python", "text"]
(.venv) andrew@bun-bun:~/src/myproj$ identify-cli script.py
["executable", "file", "python", "text"]