I think with how currently the assignment of tasks to slots works there is no way of ensuring that the source tasks are evenly spread to the TaskManagers (TaskExecutors). The rescale() API is from a time where scheduling worked a bit different in Flink, I'm afraid.
I'm cc'ing Till, who might know more about scheduling.
Seems you are the committer of rescale api, any help about this question?
I see the rescale api allow us to somehow redistribute element locally, but is it possible to make the upstream operator distributed evenly on task managers?
For example I have 10 task managers each with 10 slots. The application reads data from Kafka topic with 20 partitions, then rescale it to full parallelism. To me it seems that the 20 slots needed to read from Kafka won’t distributed evenly on 10 task managers, which means further rescale still needs to shuffle data over network.
This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.