Our Pythons currently patches distutils to install packages to
/usr/lib(64)/pythonX.Y/site-packages when the $RPM_BUILD_ROOT environment
variable is set (and to /usr/local/lib(64)/pythonX.Y/site-packages otherwise).
With the deprecation of distutils [1] we want to change the patch to create
and use a different sysconfig install scheme [2].
However, we have realized that macros defined as %(%{__python3} ...) don't
"see" the environment variables set by rpmbuild because they are expanded earlier
and hence e.g. %{python3_sitelib} evaluates to
/usr/local/lib/python3.X/site-packages -- which is not desired.
To be able to reliably detect an RPM build environment by checking
the presence of the $RPM_BUILD_ROOT environment variable,
we manually set it in the macro definitions.
Since %{buildroot} in not fully populated
(e.g. it can expand literally to
/home/anna/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64),
we don't use it here.
The variable simply needs to present in the environment.
See also the analysis of the build failures when this is not done [3].
[1] https://www.python.org/dev/peps/pep-0632/
[2] https://bugs.python.org/issue43976
[3] https://src.fedoraproject.org/rpms/python3.10/pull-request/63#comment-79042
- For depth, use sys.getrecursionlimit()
- the default from bytecompile2
- we'll get a RecursionError before it's exceeded, anyway
- Set real_libdir locally
This allows us to maintain our own BuildRoot Policy scripts in an easier way.
This change needs to be coordinated with the removal of the files from redhat-rpm-config.
redhat-rpm-config requires python-srpm-macros, so no change is expected for the packagers.
With $PATH and $PYTHONPATH set to the %buildroot,
the macro tries to import the given Python 3 module(s).
Useful as a smoke test in %check when ruining tests is not feasible.
Accepts spaces or commas as separators.
Package python-six:
%check
%py3_check_import six
Executing(%check): ...
...
+ PATH=...
+ PYTHONPATH=...
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -c 'import six'
+ RPM_EC=0
++ jobs -p
+ exit 0
%py3_check_import six seven
...
+ /usr/bin/python3 -c 'import six, seven'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'seven'
error: Bad exit status from ... (%check)
...
%py3_check_import five, six, seven
+ /usr/bin/python3 -c 'import five, six, seven'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'five'
error: Bad exit status from ... (%check)
Package python-packaging:
%py3_check_import packaging, packaging.markers packaging.requirements, packaging.tags
Executing(%check): ...
...
+ PATH=...
+ PYTHONPATH=...
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -c 'import packaging, packaging.markers, packaging.requirements, packaging.tags'
+ RPM_EC=0
++ jobs -p
+ exit 0
%py3_check_import packaging, packaging.markers packaging.notachance, packaging.tags
...
+ /usr/bin/python3 -c 'import packaging, packaging.markers, packaging.notachance, packaging.tags'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'packaging.notachance'
error: Bad exit status from ... (%check)
The %{? in the comment made the entire block of macros not work.
Since nobody actually used those on Fedora, because they did not exist,
we can safely remove them. No need to document this in the %changelog.
The %{?load:%{SOURCE102}} construct no longer works in RPM 4.17+
Currently, we:
1. Load %{SOURCE102} if it exists.
This should always be the case when actually building the RPM or SRPM package.
2. Else, load macros.python-srpm if it exists.
This is the case when something parses the spec from dist-git without setting
%_sourcedir to the current working directory. E.g. rpmdev-bumpspec does this.
3. Else, don't load anything, get %{__default_python3_version} from the environment.
This is the case when something parses the spec in isolation.
Getting the version from sources is impossible, because the sources are missing.
So we get the installed version instead. Note that this will blow up on Fedora < 33,
but it already did before.
Distutils which were used to define the macros are deprecated in Python3.10:
https://www.python.org/dev/peps/pep-0632/.
Sysconfig isn't and it works across our Pythons, making it better choice for the task.
This allows e.g.:
%global extras cli,ghostwriter,pytz,dateutil,lark,numpy,pandas,pytest,redis,zoneinfo,django
%{pyproject_extras_subpkg -n python3-hypothesis %{extras}}
...
%pyproject_buildrequires -x %{extras}
(Note that %pyproject_extras_subpkg is a tiny wrapper around %python_extras_subpkg.)
Avoid using the full path and instead rely on PATH being correctly set
up to find the executable.
This fixes byte compilation for python2.7 when doing flatpak module
builds where python2.7 can be in either /usr/bin or /app/bin, depending
on how it's compiled.
Since Fedora 33, the package version always == %{__default_python3_version}.
When we update Python to 3.X+1, we update the version and the macro.
When the macro is not updated confusing things happen.
See for example https://bugzilla.redhat.com/show_bug.cgi?id=1931421#c4
We could assert the versions match in %check instead,
but this way a temporary pull request for a new Python version, such as
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/50https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/84
can be kept rebased via the git forge UI.
Note that the value of %{__default_python3_version} is loaded from sources in
dist git, otherwise it would be defined by the previous build of this package.
As a result, the spec file is no longer parsable on it's own, but IMHO that's OK.
https://fedoraproject.org/wiki/Changes/Disable_Python_2_Dist_RPM_Generators_and_Freeze_Python_2_Macros
This is to be shipped together with an upgrade of python2.7:
The python2.7 RPM package will contain the removed macros instead.
The release is intentionally over-bumped to allow some changes of
python-rpm-macros in lower versions of Fedora without the need to bump
the version-release of python2-rpm-macros obsoleted by python2.7.
This only works for package and extra names less than 79 characters long.
I don't expect many actual packages to exceed this limit.
78 characters should be enough for everybody.
Why 78? 79 is the line-lenght limit from rpmlint.
And we need to put punctuation in there.
Versioned pathfixX.Y.py is available in main as well as in
alternative Pythons so this change enables to build
an alternative Python stack without a dependency on the main
python3-devel.
By default, %{py3_dist foo} generates python3dist(foo).
This change makes it respect %python3_pkgversion so when
it is redefined as X.Y, %{py3_dist foo} generates pythonX.Y(foo).
A new script brp-fix-pyc-reproducibility creates an opt-in way of how to fix
problems with the reproducibility of byte-compiled Python files. The script
uses marshalparser [0] which currently doesn't provide solutions for all issues
but can fix at least problems with reference flags. For more info see
this Bugzilla [1].
If you want to use this new feature, you need to define
`%py_reproducible_pyc_path` to specify a path you want to fix `.pyc`
files in (recursively) and build-require /usr/bin/marshalparser.
if you forget to build-require the parser. The error message is:
```
+ /usr/lib/rpm/redhat/brp-python-bytecompile '' 1 0
Bytecompiling .py files below /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64/usr/lib/python3.9 using /usr/bin/python3.9
+ /usr/lib/rpm/redhat/brp-fix-pyc-reproducibility /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64
ERROR: If %py_reproducible_pyc_path is defined, you have to also BuildRequire: /usr/bin/marshalparser !
error: Bad exit status from /var/tmp/rpm-tmp.UUJr4v (%install)
```
A build fails if the parser is not able to parse any of the `.pyc` files.
And finally, if a build is properly configured it produces fixed `.pyc` files.
Currently, `.pyc` files in the tldr package contain a lot of unused reference flags:
```
$ dnf install -y tldr
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
… long output …
190 - Flag_ref(byte=9610, type='TYPE_SHORT_ASCII_INTERNED', content=b'init', usages=0)
191 - Flag_ref(byte=9633, type='TYPE_SHORT_ASCII_INTERNED', content=b'source', usages=0)
192 - Flag_ref(byte=9651, type='TYPE_SHORT_ASCII_INTERNED', content=b'argv', usages=0)
193 - Flag_ref(byte=9657, type='TYPE_SHORT_ASCII_INTERNED', content=b'print_help', usages=0)
194 - Flag_ref(byte=9669, type='TYPE_SHORT_ASCII_INTERNED', content=b'stderr', usages=0)
195 - Flag_ref(byte=9682, type='TYPE_SHORT_ASCII_INTERNED', content=b'parse_args', usages=0)
196 - Flag_ref(byte=9737, type='TYPE_SHORT_ASCII_INTERNED', content=b'encode', usages=0)
197 - Flag_ref(byte=9782, type='TYPE_SHORT_ASCII_INTERNED', content=b'parser', usages=0)
198 - Flag_ref(byte=9790, type='TYPE_SHORT_ASCII_INTERNED', content=b'options', usages=0)
199 - Flag_ref(byte=9799, type='TYPE_SHORT_ASCII_INTERNED', content=b'rest', usages=0)
200 - Flag_ref(byte=9821, type='TYPE_SHORT_ASCII_INTERNED', content=b'result', usages=0)
202 - Flag_ref(byte=10022, type='TYPE_SHORT_ASCII_INTERNED', content=b'__main__', usages=0)
203 - Flag_ref(byte=10102, type='TYPE_SHORT_ASCII_INTERNED', content=b'argparse', usages=0)
204 - Flag_ref(byte=10433, type='TYPE_SHORT_ASCII_INTERNED', content=b'__name__', usages=0)
205 - Flag_ref(byte=10463, type='TYPE_SHORT_ASCII_INTERNED', content=b'<module>', usages=0)
```
This new feature fixes them:
```
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
<empty output>
```
[0] https://github.com/fedora-python/marshalparser
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1686078
--no-index saves a ~1 minute timeout of pip possibly checking if there is
a newer pip version (in mock with disabled network).
--no-warn-script-location removes a bogus warning that says:
WARNING: The scripts easy_install and easy_install-3.9 are installed in
'/builddir/build/BUILDROOT/python-setuptools-49.2.0-1.fc33.x86_64/usr/bin'
which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning,
use --no-warn-script-location.
See https://fedoraproject.org/wiki/Changes/PythonMacroError
While doing it, make %python macros more consistent with %python3 macros,
mostly wrt whitespace but also to use python -m pip over plain pip etc.
One significant change is the removal of sleeps from python macros,
this could affect packages that use python macros to build for Python 2
while also using python3 macros to build for Python 3.
In reality, I consider that unlikely. The sleep in python2 macros stays.
The --strip-file-prefix option was already removed from %pyX_install_wheel
but we forgot to remove it from %py_install_wheel.
Lumir Balhar
Miro Hrončok
Petr Viktorin
Fedora Release Engineering
Ben Burton
Karolina Surma
Kalev Lember