Extraction of Salt core modules

Scripted example

A tool named saltext-migrate was created based on the manual example below. It removes many obstacles in the extraction process. Let’s use the same example module (stalekey).

1. Install saltext-migrate and git-filter-repo

pipx install git-filter-repo
pipx install git+https://github.com/salt-extensions/salt-extension-migrate

pip install git-filter-repo git+https://github.com/salt-extensions/salt-extension-migrate

If you want to install using pip, consider creating a virtual environment beforehand.

2. Run the tool

Important

Run the tool inside a dedicated directory serving as the working directory for all your Salt extension migrations. This avoids accidental data loss and speeds up repeated migrations.

mkdir migrated-saltexts && cd migrated-saltexts
saltext-migrate stalekey

The tool will:

  1. Ensure Salt is cloned and the history analysis is available

  2. Filter for paths containing stalekey and ask for approval

  3. Filter the history into a separate branch, renaming paths as needed

  4. Auto-cleanup the history, as far as possible non-interactively

  5. Run copier with sane defaults and remove the project starter boilerplate

  6. Create a virtual environment for your project

  7. Apply rewrites (with fixes and improvements versus salt-rewrite)

  8. Install and run pre-commit

  9. Provide an overview of issues to fix and next steps

A manual module extraction example

Below are some rough steps to extract an existing set of modules into an extension while preserving the Git history. Let’s use the stalekey engine as an example.

1. Install the Git history filtering tool

pipx install git-filter-repo

pip install git-filter-repo

If you want to install using pip, consider creating a virtual environment beforehand.

2. Clone the Salt repo and analyze its history

mkdir workdir && cd workdir
git clone https://github.com/saltstack/salt --single-branch
cd salt
git filter-repo --analyze
tree .git/filter-repo/analysis/
grep stalekey .git/filter-repo/analysis/path-{all,deleted}-sizes.txt | \
    awk '{print $NF}' | sort | uniq | \
    grep -vE '^(.github|doc/ref|debian/|doc/locale|salt/([^/]+/)?__init__.py|tests/(pytests/)?(unit|functional|integration)/conftest.py)'

The main objective of this step is to find all relevant files (modules, utils, automated tests, fixtures, documentation). For the stalekey engine, they are:

  • salt/engines/stalekey.py - the engine itself

  • tests/unit/engines/test_stalekey.py - old style unit tests (historic path, no longer exists in HEAD)

  • tests/pytests/unit/engines/test_stalekey.py - new-style unit tests using pytest

3. Filter the history into a separate branch

git checkout -b filter-source
git filter-repo \
    --path salt/engines/stalekey.py \
    --path-rename salt/engines/stalekey.py:src/saltext/stalekey/engines/stalekey.py \
    --path tests/pytests/unit/engines/test_stalekey.py \
    --path-rename tests/pytests/unit/engines/test_stalekey.py:tests/unit/engines/test_stalekey.py \
    --path tests/unit/engines/test_stalekey.py \
    --refs refs/heads/filter-source --force

The --path-rename option moves the files into the directory structure used by Salt extensions.

4. Clean up the history

git log --name-only
git rebase -i --empty=drop --root --committer-date-is-author-date

The purpose of this step is to drop commits that don’t touch the extracted files plus the last commit that removes them. Merge commits are deleted automatically during the rebase.

While reviewing the Git log, please note the major contributors (in order to add them as code authors later).

5. Populate the extension repo

Answer the Copier questions, choosing the engine module type only, and specify yourself as the author:

cd ..
mkdir saltext-stalekey && cd saltext-stalekey
git init --initial-branch=main
copier copy --trust https://github.com/salt-extensions/salt-extension-copier ./

Remove unwanted boilerplate files:

rm -f tests/**/test_*.py src/**/*_mod.py

Merge the history:

git remote add repo-source ../salt
git fetch repo-source
git merge repo-source/filter-source
git remote rm repo-source
git tag | xargs git tag -d

6. Create a virtualenv and activate it

To create the virtualenv, it is recommended to use the same Python version (MAJOR.MINOR) as the one listed here.

python3.10 -m venv venv --prompt saltext-stalekey
source venv/bin/activate

Please ensure you’re inside your virtual environment from here on.

7. Clean up and test

Run the automatic fixups:

pip install git+https://github.com/saltstack/salt-rewrite
SALTEXT_NAME=stalekey salt-rewrite -F fix_saltext .

Important

You may need to re-rewrite some imports, as salt-rewrite assumes the project is named saltext.saltext_stalekey rather than saltext.stalekey.

pip install -e ".[dev,tests,docs]"
pre-commit install --install-hooks
pre-commit run -a  # ensure it is happy
git status
git add .
git commit -m 'Add extension layout'

Add the main authors to pyproject.toml:

vi pyproject.toml
git add pyproject.toml
git commit -m 'Add authors'

Try running the test suite and building the docs locally until both pass, then commit and push it to run the full test suite on GitHub.

Basic fixes (automated)

Unit test module imports

Unit tests import the modules directly. After migration, these imports need to be adjusted, otherwise the tests will run against the modules found in Salt, but still pass (or fail once they are removed in a future release). Example:

from salt.modules import vault

from saltext.vault.modules import vault

Unit test tests.support imports

Many unit tests in the Salt code base use an indirect import for unittest.mock.

from tests.support.mock import MagicMock, Mock, patch

from unittest.mock import MagicMock, Mock, patch

Migrated tests in tests/pytest

The generated Salt extension project does not account for a tests/pytests subdirectory. Its contents need to be moved to the top-level tests directory.

Issues needing manual fixing

__utils__ into Salt extension utils

Some Salt core modules access their utilities via the __utils__ dunder instead of direct imports, which ensures that the called utility function has access to Salt’s global dunders.

Accessing a Salt extension’s utils this way does not work. If this is the case for your extracted set of modules, you need to adjust the utils to not rely on the dunders, e.g. by passing in the required references:

salt/modules/foo.py
def get(entity):
    return __utils__["foo.query"](entity)
salt/utils/foo.py
def query(entity):
    base_url = __opts__.get("foo_base_url", "https://foo.bar")
    profile = __salt__["config.option"]("foo_profile")
    return __utils__["http.query"](base_url, data=profile)
saltext/foo/modules/foo.py
from saltext.foo.utils import foo


def get(entity):
    base_url = __opts__.get("foo_base_url", "https://foo.bar")
    return foo.query(base_url, entity, __salt__["config.option"])
saltext/foo/utils/foo.py
import salt.utils.http


def query(base_url, entity, config_option):
    profile = config_option("foo_profile")
    return salt.utils.http.query(base_url, data=profile)

__utils__ from Salt extension utils

Some modules in salt.utils still expect to be accessed via __utils__. While this works for modules loaded through the Salt loader (e.g., those using any loaders), it fails if your Salt extension’s utils are calling these modules directly.

Here are some options to address this:

  • Remove the dependency on the core module or call it from the modules calling the utils directly

  • Migrate the dependency into your Salt extension repository and modify it locally as described here

  • Submit a PR to Salt core with the necessary changes to eliminate the code duplication in the long term.

__utils__ from other Salt extension modules

If any other Saltext module relies on a Salt core utility that requires being called via __utils__, it will still work. However, you should consider creating a PR to remove this dependency, as __utils__ is scheduled for deprecation.

Pre-pytest tests

Salt core contains both Pytest-based and legacy tests, but Salt extension projects only support pytest. To keep legacy tests running, you may need to convert them. If you prefer to skip this task for now, you can:

  • Exclude the corresponding files from pylint

  • Skip the legacy tests entirely

# pylint: disable-all
import pytest

pytest.skip(reason="Old non-pytest tests", allow_module_level=True)

Considerations

Library dependencies

Some modules have library dependencies. Since Salt core cannot include every possible dependency, these modules often include a safeguard to handle missing libraries or library alternatives. They typically use the following pattern:

HAS_LIBS = False

try:
    import foo
    HAS_LIBS = True
except ImportError:
    pass

__virtualname__ = "foobar"


def __virtual__():
    if HAS_LIBS:
        return __virtualname__
    return False, "Missing 'foo' library"

If all the dependencies are hard dependencies, declare them in the dependencies section of your Saltext’s pyproject.toml and remove the conditional import logic:

import foo

__virtualname__ = "foobar"


def __virtual__():
    return __virtualname__

For modules that can work with multiple interchangeable libraries, declare at least one of them in the optional-dependencies section for tests in your pyproject.toml to ensure the tests can run.

Dedicated docs

Salt core modules often include inline documentation. Consider extracting the general parts of this inline documentation into separate topics within the docs/topics directory.