async_generator: Async generators and related tools for older Pythons¶

API documentation¶

Async generators¶

In Python 3.6+, you can write a native async generator like this:

async def load_json_lines(stream_reader):
    async for line in stream_reader:
        yield json.loads(line)

Here’s the same thing written with this library, which works on Python 3.5+:

from async_generator import async_generator, yield

@async_generator
async def load_json_lines(stream_reader):
    async for line in stream_reader:
        await yield_(json.loads(line))

Basically:

decorate your function with @async_generator
replace yield with await yield_()
replace yield X with await yield_(X)

That’s it!

Yield from¶

Native async generators don’t support yield from:

# Doesn't work!
async def wrap_load_json_lines(stream_reader):
    # This is a SyntaxError
    yield from load_json_lines(stream_reader)

But we do:

from async_generator import async_generator, yield_from_

# This works!
@async_generator
async def wrap_load_json_lines(stream_reader):
    await yield_from_(load_json_lines(stream_reader))

You can only use yield_from_ inside an @async_generator function, BUT the thing you PASS to yield_from_ can be any kind of async iterator, including native async generators.

Our yield_from_ fully supports the classic yield from semantics, including forwarding asend and athrow calls into the delegated async generator, and returning values:

from async_generator import async_generator, yield_, yield_from_

@async_generator
async def agen1():
    await yield_(1)
    await yield_(2)
    return "great!"

@async_generator
async def agen2():
    value = await yield_from_(agen1())
    assert value == "great!"

Introspection¶

For introspection purposes, we also export the following functions:

isasyncgen(agen_obj)¶: Returns true if passed either an async generator object created by this library, or a native Python 3.6+ async generator object. Analogous to inspect.isasyncgen() in 3.6+.

isasyncgenfunction(agen_func)¶: Returns true if passed either an async generator function created by this library, or a native Python 3.6+ async generator function. Analogous to inspect.isasyncgenfunction() in 3.6+.

Example:

>>> isasyncgenfunction(load_json_lines)
True
>>> gen_object = load_json_lines(asyncio_stream_reader)
>>> isasyncgen(gen_object)
True

In addition, this library’s async generator objects are registered with the collections.abc.AsyncGenerator abstract base class (if available):

>>> isinstance(gen_object, collections.abc.AsyncGenerator)
True

Semantics¶

This library generally tries hard to match the semantics of Python 3.6’s native async generators in every detail (PEP 525), with additional support for yield from and for returning non-None values from an async generator (under the theory that these may well be added to native async generators one day).

Garbage collection hooks¶

This library fully supports the native async generator finalization semantics, including the per-thread firstiter and finalizer hooks. You can use async_generator.set_asyncgen_hooks() exactly like you would use sys.set_asyncgen_hooks() with native generators. On Python 3.6+, the former is an alias for the latter, so libraries that use the native mechanism should work seamlessly with @async_generator functions. On Python 3.5, where there is no sys.set_asyncgen_hooks(), most libraries probably won’t know about async_generator.set_asyncgen_hooks(), so you’ll need to exercise more care with explicit cleanup, or install appropriate hooks yourself.

While finishing cleanup of an async generator is better than dropping it on the floor at the first await, it’s still not a perfect solution; in addition to the unpredictability of GC timing, the finalizer hook has no practical way to determine the context in which the generator was being iterated, so an exception thrown from the generator during aclose() must either crash the program or get discarded. It’s much better to close your generators explicitly when you’re done with them, perhaps using the aclosing context manager. See this discussion and PEP 533 for more details.

Context managers¶

As discussed above, you should always explicitly call aclose on async generators. To make this more convenient, this library also includes an aclosing async context manager. It acts just like the closing context manager included in the stdlib contextlib module, but does await obj.aclose() instead of obj.close(). Use it like this:

from async_generator import aclosing

async with aclosing(load_json_lines(asyncio_stream_reader)) as agen:
    async for json_obj in agen:
        ...

Or if you want to write your own async context managers, we’ve got you covered:

@asynccontextmanager¶: This is a backport of contextlib.asynccontextmanager(), which wasn’t added to the standard library until Python 3.7.

You can use @asynccontextmanager with either native async generators, or the ones from this package. If you use it with the ones from this package, remember that @asynccontextmanager goes on top of @async_generator:

# Correct!
@asynccontextmanager
@async_generator
async def my_async_context_manager():
    ...

# This won't work :-(
@async_generator
@asynccontextmanager
async def my_async_context_manager():
    ...

Release history¶

Async_Generator 1.10 (2018-07-31)¶

Features¶

Add support for PEP 525-style finalization hooks via set_asyncgen_hooks() and get_asyncgen_hooks() functions. On Python 3.6+, these are aliases for the versions in sys; on Python 3.5, they’re work-alike implementations. And, @async_generator generators now call these hooks at the appropriate times. (#15)

Fixes¶

Package now properly includes license files. (#11)

1.9 (2018-01-19)¶

Add asynccontextmanager()
When a partially-exhausted async_generator is garbage collected, the warning printed now includes the generator’s name to help you track it down.
Move under the auspices of the Trio project. This includes a license change from MIT → dual MIT+Apache2, and various changes to internal organization to match Trio project standard.

1.8 (2017-06-17)¶

Implement PEP 479: if a StopAsyncIteration leaks out of an async generator body, wrap it into a RuntimeError.
If an async generator was instantiated but never iterated, then we used to issue a spurious “RuntimeWarning: coroutine ‘…’ was never awaited” warning. This is now fixed.
Add PyPy3 to our test matrix.
100% test coverage.

1.7 (2017-05-13)¶

Fix a subtle bug where if you wrapped an async generator using functools.wraps, then isasyncgenfunction would return True for the wrapper. This isn’t how inspect.isasyncgenfunction works, and it broke sphinxcontrib_trio.

1.6 (2017-02-17)¶

Add support for async generator introspection attributes ag_running, ag_code, ag_frame.
Attempting to re-enter a running async_generator now raises ValueError, just like for native async generators.
100% test coverage.

1.5 (2017-01-15)¶

Remove (temporarily?) the hacks that let yield_ and yield_from_ work with native async generators. It turns out that due to obscure linking issues this was causing the library to be entirely broken on Python 3.6 on Windows (but not Linux or MacOS). It’s probably fixable, but needs some fiddling with ctypes to get the refcounting right, and I couldn’t figure it out in the time I had available to spend.

So in this version, everything that worked before still works with @async_generator-style generators, but uniformly, on all platforms, yield_ and yield_from_ now do not work inside native-style async generators.
Now running CI testing on Windows as well as Linux.
100% test coverage.

1.4 (2016-12-05)¶

Allow await yield_() as an shorthand for await yield_(None) (thanks to Alex Grönholm for the suggestion+patch).
Small cleanups to setup.py and test infrastructure.
100% test coverage (now including branch coverage!)

1.3 (2016-11-24)¶

Added isasyncgen and isasyncgenfunction.
On 3.6+, register our async generators with collections.abc.AsyncGenerator.
100% test coverage.

1.2 (2016-11-14)¶

Rewrote yield from support; now has much more accurate handling of edge cases.
yield_from_ now works inside CPython 3.6’s native async generators.
Added aclosing context manager; it’s pretty trivial, but if we’re going to recommend it be used everywhere then it seems polite to include it.
100% test coverage.

1.1 (2016-11-06)¶

Support for asend/athrow/aclose
Support for yield from
Add a __del__ method that complains about improperly cleaned up async generators.
Adapt to the change in Python 3.5.2 where __aiter__ should now be a regular method instead of an async method.
Adapt to Python 3.5.2’s pickiness about iterating over already-exhausted coroutines.
100% test coverage.

1.0 (2016-07-03)¶

Fixes a very nasty and hard-to-hit bug where await yield_(...) calls could escape out to the top-level coroutine runner and get lost, if the last trap out to the coroutine runner before the await yield_(...) caused an exception to be injected.
Infinitesimally more efficient due to re-using internal ANextIter objects instead of recreating them on each call to __anext__.
100% test coverage.

0.0.1 (2016-05-31)¶

Initial release.