pytorch libs poetry installation on different operating systems
Poetry python library allows, in an easy manner, to track package dependencies within a project. It creates a snapshot of the environment in a form of a lockfile, containing packages names with installed versions, to ensure ease of repeatable installations. That is super helpful for maintaining same setups on development and production environments or for ensuring developers to have the same working space.
However I experienced a problem with poetry when trying to capture a package which has different versions for different operating systems. That happened to be the case with 4 pytorch related libraries:
- torch_scatter
- torch_sparse
- torch_cluster
- torch_spline_conv
HOW TO
To install poetry within you project, follow steps from the official documentation. I was using Python 3.9 and poetry 1.4.0.
In a nutshell, poetry provides you with a pyproject.toml file, where you specify your package requirements. It is used to orchestrate your project and its dependencies. Based on this file, the poetry.lock file is being created, which captures information about the environment.
Generally what you do is set desired packages versions in this pyproject.toml file, following the pattern. To simplify the example I stick to one of the mentioned pytorch package, but the problem and solution can easily be transferred for the 3 remaining ones.
|
1 2 3 |
... torch-cluster = {version = "^1.6.0"} ... |
The caret (^) in front of 1.6.0 is one of the versioning constraints that poetry provides. It means that poetry can install any version that matches the leftmost non-zero digit of the version string. In this case using 1.6.1 or 1.7 is allowed, but version 2.0 wouldn’t be allowed.
With such a setup, after you update lockfile, using the command:
|
1 |
poetry lock --no-update |
you get following entry for the torch-cluster package:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
... [[package]] name = "torch-cluster" version = "1.6.0+pt112cpu" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = true python-versions ">=3.9" files = [ {file = "torch_cluster-1.6.0+pt112cpu-cp310-cp310-linux_x86_64.whl", hash = "sha256:699787ecf02305874082c8a7d490cdb97d171ca47885698fe6b0f256b2c6ad76"}, {file = "torch_cluster-1.6.0+pt112cpu-cp310-cp310-win_amd64.whl", hash = "sha256:6945fae39b701dc15ffe7e104d9577fa9df2100039b290118648ad09efa4e63a"}, {file = "torch_cluster-1.6.0+pt112cpu-cp39-cp39-linux_x86_64.whl", hash = "sha256:dc942ec73f5dcfac55ce7452ce4769aa77e8299b55926e11eeff409e127b35e6"}, {file = "torch_cluster-1.6.0+pt112cpu-cp39-cp39-win_amd64.whl", hash = "sha256:c3a87bb37776663ac921ab36c0719e64e576a9cd41972a3b6a0737ef8eeae7b2"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "legacy" url = "https://data.pyg.org/whl/torch-1.12.0+cpu.html" reference = "torch-scatter" ... |
As you can see, automatically version 1.6.0+pt112cpu was picked up and logged. If you inspect closer, you may notice that only installations for Windows and Linux operating systems were found. That leads to an installation error when trying to recreate the environment on MacOS:
|
1 2 3 |
Error: Installing torch-cluster (1.6.0+pt112cpu): Failed RuntimeError Unable to find installation candidates for torch-cluster (1.6.0+pt112cpu) |
After further inspection you may find that it is because particular MacOS package exists noted as version 1.6.0, so it is different than the one that was automatically picked up (1.6.0+pt112cpu). You may check it out here. As I wanted my installation to also cover MacOS, I needed some other way of setting it all up.
What if we set the package version strictly to 1.6.0?
|
1 2 3 |
... torch-cluster = {version = "1.6.0"} ... |
Actually the situation will remain exactly the same… Still version 1.6.0+pt112cpu gets picked up.
You may try to explicitly hardcode the version in lockfile (which should not be done, but for the sake of the experiment):
|
1 2 3 4 5 |
... [[package]] name = "torch-cluster" version = "1.6.0" ... |
That will result in only MacOS related version being recorded!
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
... [[package]] name = "torch-cluster" version = "1.6.0" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = true python-versions = ">=3.9" files = [ {file = "torch_cluster-1.6.0-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:3ba2d62c9da4c6ff90c620ff89c3b4a3cf3c1baa82fcc2b41acfb609c7bd7d6d"}, {file = "torch_cluster-1.6.0-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:ff277890764daabc296fa677eead3198796869c97de212d2684475e7170510a8"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "legacy" url = "https://data.pyg.org/whl/torch-1.12.0+cpu.html" reference = "torch-scatter" ... |
So how can we solve that? Well, you may try to install this library manually following the description.
Alternatively, in a more automated way, you may leverage the option to set up version specification based on the operating system in the pyproject.toml file, using sys_platform parameter:
- sys_platform == win32, for Windows
- sys_platform == linux, for Linux
- sys_platform == darwin, for MacOS
Additionally, you may relate it to python version with python_version option.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
... torch-cluster = [ {url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0-cp310-cp310-macosx_10_15_x86_64.whl", markers = "sys_platform == 'darwin' and python_version == '3.10'"}, {url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0-cp39-cp39-macosx_10_15_x86_64.whl", markers = "sys_platform == 'darwin' and python_version == '3.9'"}, {url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp310-cp310-win_amd64.whl", markers = "sys_platform == 'win32' and python_version == '3.10'"}, {url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp39-cp39-win_amd64.whl", markers = "sys_platform == 'win32' and python_version == '3.9'"}, {url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp310-cp310-linux_x86_64.whl", markers = "sys_platform == 'linux' and python_version == '3.10'"}, {url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp39-cp39-linux_x86_64.whl", markers = "sys_platform == 'linux' and python_version == '3.9'"} ] ... |
Now, after update, we can observe 6 entries in poetry.lock file, each for the specific configuration:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 |
... [[package]] name = "torch-cluster" version = "1.6.0" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = false python-versions = ">=3.9" files = [ {file = "torch_cluster-1.6.0-cp310-cp310m-macosx_10_15_x86_64.whl", hash = "sha256:0f740b5490a4955308a7b6853793b1131b0376c2ebc53e66b890162a6b108b7b"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "url" url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0-cp310-cp310m-macosx_10_15_x86_64.whl" [[package]] name = "torch-cluster" version = "1.6.0" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = false python-versions = ">=3.9" files = [ {file = "torch_cluster-1.6.0-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:ff277890764daabc296fa677eead3198796869c97de212d2684475e7170510a8"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "url" url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0-cp39-cp39-macosx_10_15_x86_64.whl" [[package]] name = "torch-cluster" version = "1.6.0+pt112cpu" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = false python-versions = ">=3.9" files = [ {file = "torch_cluster-1.6.0+pt112cpu-cp310-cp310m-linux_x86_64.whl", hash = "sha256:11d22b528516957aa2b32bf4569ff67373b0a2ace7378b3200d168c532322912"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "url" url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp310-cp310m-linux_x86_64.whl" [[package]] name = "torch-cluster" version = "1.6.0+pt112cpu" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = false python-versions = ">=3.9" files = [ {file = "torch_cluster-1.6.0+pt112cpu-cp310-cp310m-win_amd64.whl", hash = "sha256:9af7770d12436f56a402064b533e0d7331c5e025dea95d48ee6327bc6eddf147"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "url" url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp310-cp310m-win_amd64.whl" [[package]] name = "torch-cluster" version = "1.6.0+pt112cpu" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = false python-versions = ">=3.9" files = [ {file = "torch_cluster-1.6.0+pt112cpu-cp39-cp39-linux_x86_64.whl", hash = "sha256:dc942ec73f5dcfac55ce7452ce4769aa77e8299b55926e11eeff409e127b35e6"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "url" url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp39-cp39-linux_x86_64.whl" [[package]] name = "torch-cluster" version = "1.6.0+pt112cpu" description = "PyTorch Extension Library of Optimized Graph Cluster Algorithms" category = "main" optional = false python-versions = ">=3.9" files = [ {file = "torch_cluster-1.6.0+pt112cpu-cp39-cp39-win_amd64.whl", hash = "sha256:c3a87bb37776663ac921ab36c0719e64e576a9cd41972a3b6a0737ef8eeae7b2"}, ] [package.dependencies] scipy = "*" [package.extras] test = ["pytest", "pytest-cov"] [package.source] type = "url" url = "https://data.pyg.org/whl/torch-1.12.0%2Bcpu/torch_cluster-1.6.0+pt112cpu-cp39-cp39m-win_amd64.whl" ... |
That solves the issue, however requires some more entries to be specified.
The same situation applies to 3 additional pytorch packages mentioned at the beginning:
- torch_scatter
- torch_sparse
- torch_spline_conv
They can be specified in the pyproject.toml file in the exact same manner, using sys_platform parameter.