Packages from remote unavailable

[fish-face]

I'm not sure if this is an issue with the documentation, my method, or a bug. This started as an issue seen on our local pulp repository running a slightly older version; I've now tested it on the latest docker images and am still having no joy.

I have a package, package with a dependency on dependency, available in PyPI and want to be able to install package like this:

$ pip install --index-url https://pulp/pulp/content/foo/simple/ package

What happens is an error something like this:

...
  Looking in indexes: https://pulp/pulp/content/foo/simple/
  ERROR: Could not find a version that satisfies the requirement dependency>=1.0.0 (from versions: none)
  ERROR: No matching distribution found for dependency>=1.0.0
...
ERROR: Could not find a version that satisfies the requirement package (from versions: 1.0.0)
ERROR: No matching distribution found for package

Note that pip finds (and downloads) package successfully - it's only the dependencies (which are not in the repository foo) which cause an issue.
The documentation here is not especially clear. My understanding is that it should be sufficient to perform the following:

$ pulp python remote create --name pypi-excludes --url https://pypi.org/ --excludes '["package"]'
$ pulp python repository sync --name foo --remote pypi-excludes

Once the task completes, I notice that no new repository version is created. Nevertheless, something happened: if I inspect the docker volumes, they now have information from the index. I see that the the remote has been assigned to the repository (I also tried to do this by hand, but the docs say the sync does it):

$ pulp python repository show --name foo                                                                                                                                   
{
  "pulp_href": "/pulp/api/v3/repositories/python/python/6f6d94b0-9906-4b34-ba90-2973ccb767d9/",
  "pulp_created": "2021-05-27T11:27:29.503126Z",
  "versions_href": "/pulp/api/v3/repositories/python/python/6f6d94b0-9906-4b34-ba90-2973ccb767d9/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/python/python/6f6d94b0-9906-4b34-ba90-2973ccb767d9/versions/2/",
  "name": "foo",
  "description": null,
  "remote": "/pulp/api/v3/remotes/python/python/051db320-ad11-4e1d-ac84-8d9002f13720/"
}

and here is the repository version:

$ pulp python repository version show --repository foo
{
  "pulp_href": "/pulp/api/v3/repositories/python/python/6f6d94b0-9906-4b34-ba90-2973ccb767d9/versions/2/",
  "pulp_created": "2021-05-27T17:43:42.798016Z",
  "number": 2,
  "base_version": null,
  "content_summary": {
    "added": {
      "python.python": {
        "count": 1,
        "href": "/pulp/api/v3/content/python/packages/?repository_version_added=/pulp/api/v3/repositories/python/python/6f6d94b0-9906-4b34-ba90-2973ccb767d9/versions/2/"
      }
    },
    "removed": {},
    "present": {
      "python.python": {
        "count": 2,
        "href": "/pulp/api/v3/content/python/packages/?repository_version=/pulp/api/v3/repositories/python/python/6f6d94b0-9906-4b34-ba90-2973ccb767d9/versions/2/"
      }
    }
  }
}

(There are 2 things present because I added a second version of the package to see if one had to have the remote already assigned to the repository when adding a new content unit for it to work)

I shall skip the response for the remote since it is long, but it has excludes: ["package"] and policy: "on_demand" as I expected.

Maybe there is some issue with the way I have created the remote, or something else?

By the way, there is a definite documentation bug regarding the policy - the description is inconsistent on possible values. There is a common issue throughout the auto-generated API documentation where only the types are included. Often it is not clear without already being fully familiar with the system and API what object a pulp_href refers to, since all objects have hrefs. In this specific case, the meaning of the different policies is not explained beyond the names of the enum, but there are many others.

[gerrod3]

I'm not sure if this is an issue with the documentation, my method, or a bug. This started as an issue seen on our local pulp repository running a slightly older version; I've now tested it on the latest docker images and am still having no joy.

If you can run pulp status and report the version of pulpcore and pulp_python you are running that would be helpful.

I have a package, package with a dependency on dependency, available in PyPI and want to be able to install package like this:

I just want to clarify that these are the actual package names and not pseudo-example names correct? Is this is the dependency package (https://pypi.org/project/dependency/) that your package depends on? This dependency package only goes up to version 0.0.3 on PyPI.

...
  Looking in indexes: https://pulp/pulp/content/foo/simple/
  ERROR: Could not find a version that satisfies the requirement dependency>=1.0.0 (from versions: none)
  ERROR: No matching distribution found for dependency>=1.0.0
...
ERROR: Could not find a version that satisfies the requirement package (from versions: 1.0.0)
ERROR: No matching distribution found for package

Assuming that you synced dependency from PyPI successfully, you would still get this error as PyPI doesn't have version >=1.0.0.

The documentation here is not especially clear. My understanding is that it should be sufficient to perform the following:

$ pulp python remote create --name pypi-excludes --url https://pypi.org/ --excludes '["package"]'
$ pulp python repository sync --name foo --remote pypi-excludes

If you are only using pulp_python to host your custom package, then I would suggest using the --includes filter to only sync the extra dependencies needed for package. PyPI is quite large and even with on_demand syncing it will still take over a day to sync all of it depending on your internet speeds and hard-drive speeds.

Once the task completes, I notice that no new repository version is created. Nevertheless, something happened: if I inspect the docker volumes, they now have information from the index. I see that the the remote has been assigned to the repository (I also tried to do this by hand, but the docs say the sync does it):

If there is no new repository version after the sync task completes, then that means the sync task failed. Use pulp task list --name-contains sync to view all the sync tasks and check their completion status. If the tasked failed there will usually be an error message with a traceback. This is the reason why pip can't find dependency since it wasn't properly synced.

Also, sync doesn't assign a remote to the repository. It's a bit confusing but sync tries to use the remote passed in through the command, if no remote is specified, then it tries to use the remote attached to the repository and errors if there is none.

By the way, there is a definite documentation bug regarding the policy - the description is inconsistent on possible values. There is a common issue throughout the auto-generated API documentation where only the types are included. Often it is not clear without already being fully familiar with the system and API what object a pulp_href refers to, since all objects have hrefs. In this specific case, the meaning of the different policies is not explained beyond the names of the enum, but there are many others.

You are correct about the docs being wrong, if you could file issues for documentation errors or confusing parts that you find, that would be helpful. A little bit on syncing policy: immediate syncs the whole package, on_demand syncs just the package metadata and will download and store the full package when needed, and streamed is like on_demand but it doesn't save the full package to the database. By default pulp_python uses on_demand syncing because all of PyPI is over 10 terabytes.

@fish-face Hope this helps.

[fish-face]

OK, the task status command reveals that, on my test instance, the task has failed with a traceback inside the Django DB backend with an error from postgres,

could not resize shared memory segment "/PostgreSQL.952192526" to 16777216 bytes: No space left on device
CONTEXT:  parallel worker

I assume that is something to do with the large size of PyPI, though 16.7 MB for a memory resource seems small - maybe you know off the top of your head if this could be, say, something tuned very small on the postgres server in the docker image, or something to that effect.
In contrast, the tasks on the production pulp server actually have not finished even though they were started a month ago and are running in the same cluster as my test VM which hosts the docker container - I'm not sure if we have some rules on there which throttle it or something like that. In any case that seems to be the culprit. (The production instance is not running the latest pulp core by the way, so I can't test with pulp-cli)

For more information, we didn't want to specify the exact packages in case the package we wish to host a custom version of incorporates a new package and then stops being ugpradeable without manual intervention, but I guess that is not really possible... Ideally there'd be some way to specify a remote as being "includes every (transitive) dependency" or "pass unhandled requests on to here". The reason this is an issue is that pip itself doesn't support any kind of different handling of different indexes, so you are forced to use your local index to do this for you.

As for the confusion here, perhaps you could advise now: in my mind there is a UX bug here because the failure of the task is not notified through the CLI, even though it waits for the task - I can file a bug against pulp-cli unless there's some reason not to.

The above probably makes the following redundant, but just in case it is not, I include it to not waste time:

$ pulp status
{
  "versions": [
    {
      "component": "core",
      "version": "3.12.2"
    },
    {
      "component": "python",
      "version": "3.2.0"
    },
...
}

package and dependency are in fact pseudonames. The key thing which I didn't mention explicitly though is that package does exist on PyPI (hence the exclusion on the remote).

[gerrod3]

I assume that is something to do with the large size of PyPI, though 16.7 MB for a memory resource seems small - maybe you know off the top of your head if this could be, say, something tuned very small on the postgres server in the docker image, or something to that effect.

As far as I know that error means that the directory holding the postgres database ran out of space, not sure what its default value is for the docker image if there is one (I'm pretty sure you pass in which directory you want to use to store the database).

In contrast, the tasks on the production pulp server actually have not finished even though they were started a month ago and are running in the same cluster as my test VM which hosts the docker container - I'm not sure if we have some rules on there which throttle it or something like that. In any case that seems to be the culprit. (The production instance is not running the latest pulp core by the way, so I can't test with pulp-cli)

This is interesting. What version of pulpcore is it running? The pulp-cli should work regardless of versions, just some commands won't work if they don't meet the required versions, e.g. none of the pulp_python commands will work if pulp_python version is less than 3.2.0.

I know syncing all of PyPI takes a while, but not finishing after a month is kind of worrisome. If you can view the task, there should be a progress report with the number of package metadatas downloaded, artifacts downloaded, and content associated. For syncing all of PyPI the number of metadatas downloaded should be around 300,000, and the number of artifacts and contents associated should be around 4.2 million.

For more information, we didn't want to specify the exact packages in case the package we wish to host a custom version of incorporates a new package and then stops being ugpradeable without manual intervention, but I guess that is not really possible... Ideally there'd be some way to specify a remote as being "includes every (transitive) dependency" or "pass unhandled requests on to here". The reason this is an issue is that pip itself doesn't support any kind of different handling of different indexes, so you are forced to use your local index to do this for you.

This is a pretty common issue with pip. I think specifying the exact packages could still be more useful then syncing all of PyPI because you can specify to not sync newer versions of a package using version specifiers on your includes and excludes statements. However, having Pulp use multiple indexes as a fallback is a useful feature and something we could add without much change to pulp_python. I'll file an issue to look into implementing this as a feature.

As for the confusion here, perhaps you could advise now: in my mind there is a UX bug here because the failure of the task is not notified through the CLI, even though it waits for the task - I can file a bug against pulp-cli unless there's some reason not to.

No need to file a bug since it is already being fixed: pulp/pulp-cli#258. Currently the CLI waits 2 mins before it stops waiting on a task, but some tasks will inevitably run longer then 2 mins, so it was confusing why the CLI stopped waiting before the task completed. Next release of the CLI will allow you to wait indefinitely for a task to complete or properly warn the user if the task is taking longer than the timeout.

[fish-face]

As far as I know that error means that the directory holding the postgres database ran out of space, not sure what its default value is for the docker image if there is one (I'm pretty sure you pass in which directory you want to use to store the database).

Turns out it was a docker default limit on shared memory. Increasing that limit results in the task being canceled with no error message in the response to showing the task, but also not having been canceled by me. Do you know what other scenarios could have caused a task cancellation? I looked through the logs from the container - they show uninterrupted HTTP requests (I was watching the task status after the CLI stopped doing so). I wondered if perhaps a worker process had been killed, but nothing suggests this - in particular there is no message from the OOM killer.

This is interesting. What version of pulpcore is it running? The pulp-cli should work regardless of versions, just some commands won't work if they don't meet the required versions, e.g. none of the pulp_python commands will work if pulp_python version is less than 3.2.0.

I misremembered the error message; it is indeed the pulp_python version that is older.

I know syncing all of PyPI takes a while, but not finishing after a month is kind of worrisome. If you can view the task, there should be a progress report with the number of package metadatas downloaded, artifacts downloaded, and content associated. For syncing all of PyPI the number of metadatas downloaded should be around 300,000, and the number of artifacts and contents associated should be around 4.2 million.

Another oversight on my part, these tasks are also in the canceled state (at the time I was looking for errors, I suppose!) One made it to about 1,100,000 content units and 90,000 projects, the other only to 820,000 contents and 62,000 projects. Both got further than the task on my test instance, which may reflect the larger VM that the production instance is running on (I don't know the exact details; I just know I didn't make a big VM. It ran out of memory the first time I tried this, but seems not to now)

This is a pretty common issue with pip. I think specifying the exact packages could still be more useful then syncing all of PyPI because you can specify to not sync newer versions of a package using version specifiers on your includes and excludes statements. However, having Pulp use multiple indexes as a fallback is a useful feature and something we could add without much change to pulp_python. I'll file an issue to look into implementing this as a feature.

Are you suggesting modifying the filter on the remote when building our package? That could work actually, though it could be a little fiddly to get working initially...

No need to file a bug since it is already being fixed: pulp/pulp-cli#258. Currently the CLI waits 2 mins before it stops waiting on a task, but some tasks will inevitably run longer then 2 mins, so it was confusing why the CLI stopped waiting before the task completed. Next release of the CLI will allow you to wait indefinitely for a task to complete or properly warn the user if the task is taking longer than the timeout.

Ah, I see - my quick search didn't find anything - fair enough :)

[gerrod3]

Turns out it was a docker default limit on shared memory. Increasing that limit results in the task being canceled with no error message in the response to showing the task, but also not having been canceled by me. Do you know what other scenarios could have caused a task cancellation? I looked through the logs from the container - they show uninterrupted HTTP requests (I was watching the task status after the CLI stopped doing so). I wondered if perhaps a worker process had been killed, but nothing suggests this - in particular there is no message from the OOM killer.

It's hard to tell what is happening. You can find a list of all the known tasking issues here and maybe one of them is similar to what you are experiencing. For upcoming pulpcore 3.14 we have redesigned the tasking system to solve all these problems.

Another oversight on my part, these tasks are also in the canceled state (at the time I was looking for errors, I suppose!) One made it to about 1,100,000 content units and 90,000 projects, the other only to 820,000 contents and 62,000 projects. Both got further than the task on my test instance, which may reflect the larger VM that the production instance is running on (I don't know the exact details; I just know I didn't make a big VM. It ran out of memory the first time I tried this, but seems not to now)

Looking at https://pypi.org/stats/ you need at least 10 TB of storage for the artifacts. Also, the postgres database will need a hefty amount of storage to contain all the metadata for each package. I think all of PyPI metadata is around 30 GB.

Are you suggesting modifying the filter on the remote when building our package? That could work actually, though it could be a little fiddly to get working initially...

Yes. If all of the requirements for your package is in a requirements.txt then you can use the latest version of the CLI to specify the remote's includes and excludes with the file. e.g. pulp file remote create --name foo --includes @requirements.txt. The one issue with this is that we currently don't support recursive dependency syncing, so you would still need to list all dependencies of each dependency.