CompletionStage callbacks can prefer dispatcher over common ForkJoinPool

[leviramsey]

The Scala standard library's conversion from Scala future to Java CompletionStage defines the "non-Async" versions to effectively call the Async version with the Java common ForkJoinPool specified. This subtly conflicts with the common understanding of the difference between the non-Async version, and the Asyncversion, as evidenced by various AI answers:

Claude

thenCompose(fn): The function fn runs in the same thread that completed the previous stage (or the calling thread if the stage is already complete).
thenComposeAsync(fn): The function fn is always submitted to an executor, guaranteeing it runs asynchronously on a separate thread.

Copilot

thenCompose: Uses the thread of the previous stage; no new thread pool
thenComposeAsync: If you don’t pass an Executor, it uses the ForkJoinPool.commonPool(). If you do pass an Executor, it uses that.

Gemini

thenCompose (without "Async") aims to run the dependent task on the same thread that completed the previous CompletableFuture.
thenComposeAsync guarantees that the dependent task will run on a different thread (either the default ForkJoinPool.commonPool() Executor or a specific Executor you provide).

The parasitic behavior of the non-Async version is not desirable for the futures returned by Akka (consider that the future returned from asking a remote actor would run callbacks on the remoting dispatcher!), but there's not a great reason to use the common FJP in an Akka application: the default dispatcher provides an FJP which is well-suited to short CPU-bound tasks (including with better observability) and running both the common FJP and the default dispatcher may result in far more threads running such tasks than is desirable.

This adapts the Scala standard library's conversion to one where the executor to prefer is specified at the time of conversion. CompletionStage-returning calls where it's practical to shift onto the default dispatcher (those where there's an actor system readily available) are then adapted to use this:

• asks of EntityRefs in typed cluster sharding
• coordination lease operations
• events-by-slice firehose methods which return completion stages

[johanandren]

While I generally like this idea of improvement of the Java side, I have some doubt whether we should introduce it in a patch release. I think there is a quite hight risk that Java users doing blocking logic in .thenApply / .thenCompose etc. on these CompletionStage that would now starve the dispatcher and, worst case, break currently working systems.

[patriknw]

The Scala standard library's conversion from Scala future to Java CompletionStage defines the "non-Async" versions to effectively call the Async version with the Java common ForkJoinPool

Does this mean that whenever we run on the Akka FJP and then do asJava conversion from std lib it will switch over to Java's common FJP? This PR would change that for 3 specific cases, where the EntityRef.ask is most important. Wouldn't there be a gazillion more places where we convert with asJava? Would we replace more, including in downstream Akka libraries?

[johanandren]

Yeah, I think if we do it, it should be in a bigger release than patch, and we should do it across all libraries.

[johanandren]

Another, perhaps better, option than to implement in Akka would be to look into doing something with controllable/implicit ec in scala.jdk.FutureConverters.FutureOps. That would have the drawback that we couldn't use it in Akka until we strictly depend on a Scala version with that in though.

[leviramsey]

I think there is a quite hight risk that Java users doing blocking logic in .thenApply / .thenCompose etc. on these CompletionStage that would now starve the dispatcher and, worst case, break currently working systems.

There's definitely that risk, though it's mitigated by the common advice on when to be parasitic, which is common enough lore for the AIs to be aware of it, e.g. Claude summarizes its answer for "In Java, when should I use thenApply versus thenApplyAsync" to:

Fast, non-blocking transform → thenApply
Slow, blocking, or thread-sensitive work → thenApplyAsync
Production pipelines with latency requirements → pass an explicit Executor to thenApplyAsync so you're not at the mercy of the common pool

And many of the blocking cases will actually be things like .get/.join which on an FJP like the default dispatcher will cause a worker to be added to the pool.

Wouldn't there be a gazillion more places where we convert with asJava? Would we replace more, including in downstream Akka libraries?

There are plenty of such places, though the others in core are ones where we don't have easy access to the system dispatcher (e.g. classic ask pattern) without changing the API for callers, so it would be a choice between common FJP and the Scala global EC. (Kind of related: akka-http's Java FutureDirectives move at least some future callbacks through the Scala global EC... so it's plausible that a Java app which does an ask, attaches some transformations based on the ask and uses an onSuccess directive will jump from [system/remoting] dispatcher (completing the ask) to common pool (transformations) to Scala global EC (future directives in HTTP) to system dispatcher (HTTP streaming engine): https://github.com/akka/akka-http/blob/5a75805ed7058b15209f1763d1b5c14ce59d9e27/akka-http/src/main/scala/akka/http/javadsl/server/directives/FutureDirectives.scala#L18)

Another, perhaps better, option than to implement in Akka would be to look into doing something with controllable/implicit ec in scala.jdk.FutureConverters.FutureOps. That would have the drawback that we couldn't use it in Akka until we strictly depend on a Scala version with that in though.

My understanding is that there's a very high bar for API-level changes in scala-lib (though @lrytz might correct me if wrong).

[johanandren]

I think we landed on, in conversation elsewhere, to make this opt in for now, and the default once we release the next minor.

[johanandren]

I don't think this should be public API but Akka-internal.