gh-139899: Introduce MetaPathFinder.discover and PathEntryFinder.discover

[FFY00]

• Issue: Introduce MetaPathFinder.discover and PathEntryFinder.discover #139899

---

📚 Documentation preview 📚: https://cpython-previews--139900.org.readthedocs.build/

[FFY00]

Still needs tests, but I'll wait to see the feedback on the issue.

[FFY00]

I've updated the method to take a module spec, instead of a module object, as in some cases the parent might have failed to import.

In a follow-up, I will add a protocol for finders implementing .discover(), as @brettcannon suggested.

[gpshead]

I think this can be None when parent is a non-package module? make a nicer error message or should this situation use sys.path?

[FFY00]

How can it be a non-package module if it has a child? It could be a namespace package, but that's still a package, and parent.submodule_search_locations should be an iterable objects when the spec is fully initialized, which it should always be at this point.

Unless I am missing something? Do we support package-like module extensions?

[ncoghlan]

I think @gpshead is referring to the error case where parent.submodule_search_locations is None because the caller supplied a "parent" spec from a non-package module. Instead of the default "'NoneType' object is not iterable" message from the failed iteration attempt, we should raise something more specific.

[gpshead]

use a with statement to explicitly close the iterator - https://docs.python.org/3/library/os.html#os.scandir.close

also consider handling exceptions similar to what _fill_cache does, which is also be needed around the is_dir and is_file bits within the loop below - https://docs.python.org/3/library/os.html#os.scandir.close

document exception handling semantics of discover. I doubt callers ever expect to need be prepared for those?

[gpshead]

should this be cached or not? we should document the cache interaction behavior either way regardless.

[FFY00]

I don't think so? Because it could change at runtime, no?

[FFY00]

Getting back to this, I don't think we should touch anything in the cache if we keep using os.scandir. Alternatively, we can rewrite _find_children to use the cache / os.listdir.

[gpshead]

entry.name could exist with multiple loader suffixes on the filesystem. dedupe to avoid redundant specs?

[gpshead]

would this be the first time we have a public API yielding things from importlib, do we actually want to do that or should this return a list?

callers consuming the results might be writing code that makes changes that could impact future results... yielding could get messy.

what are the intended use cases for the API? if it's something we expect callers to short circuit and stop iterating on after the first match maybe yield makes sense, but then we should probably just have an explicit direct discover_first API for that instead.

[FFY00]

This is something I considered.

The main use-case is finding a similar-named module to show as a hint on ModuleNotFoundError (eg. "Did you meant numpy?", when trying to import numby), so I think it would make sense to make this new API a generator, or at least some kind of lazy container.

For cases such as you describe, the user could just consume the full generator into a list to avoid any issue. Still leaving opportunity for the code that could leverage the benefit of this being a lazy API — scaning directories with a lot of files can take a while, not to mention the other exotic finders out there that may operate over the network or something like that.

I am not fundamentally opposed to make this method return a list, but I can't see the value in the trade-of if we document it properly.

callers consuming the results might be writing code that makes changes that could impact future results... yielding could get messy.

While this is technically possible, I would find it extremely uncommon. And I think it should be reasonable to assume that people who are knowledgeable enough to do that, would probably be aware of the downsides of making changes to the import machinery, while consuming the API

but then we should probably just have an explicit direct discover_first API for that instead

And what would that look like? Would it take a predicate function and return the first entry that matches?

---

So, would have a warning in the documentation regarding your concern be a good enough compromise?

[ncoghlan]

True, for the "nearest matching module name" error message case, you just want to keep the closest match, so even though all the names need to be checked, you only need to keep a reference to one of them (the closest so far).

Since the potential number of results may be absurdly high in some situations, and the iterable form provides more options for consumers to handle that appropriately, I do think it makes sense to make it an iterable. The warning in the docs should explain why it's an iterable, since consumers that unconditionally convert the result to a list can run into problems regardless.

As far as prior examples of iterable import APIs goes, importlib.resources.contents was the first example I found in a quick look (covering a very similar situation, just for non-module resources rather than submodules)

[gpshead]

path can contain duplicate entries. should we dedupe?

[FFY00]

Yeah, sure 👍

[ncoghlan]

This is definitely looking promising, but I do agree with a couple of @gpshead's requested tidy ups.

[ncoghlan]

I think @gpshead is referring to the error case where parent.submodule_search_locations is None because the caller supplied a "parent" spec from a non-package module. Instead of the default "'NoneType' object is not iterable" message from the failed iteration attempt, we should raise something more specific.