Thanks for the reply.
I don’t use batch, so I assume what I am seeing is streaming related. I thought rocksdb writes to a different dir though ( as defined by checkpoint.data.uri)?
From: Till Rohrmann [mailto:trohrmann@xxxxxxxxxx]
spilling using the io directories is only relevant for Flink's batch processing. This happens, for example if you enable blocking data exchange where the produced data cannot be kept in memory. Moreover, it is used by many of Flink's out-of-core data structures to enable exactly this feature (e.g. users are the MutableHashTable, the MergeIterator to combine sorted ata which has been spilled or the SorterMerger to actually spill data).
In streaming Flink uses the RocksDB state backend to spill very large state gracefully to disk. Thus, you would need to configure RocksDB in order to control the spilling behaviour.
On Mon, Oct 8, 2018 at 2:18 PM Kostas Kloudas <k.kloudas@xxxxxxxxxxxxxxxxx> wrote:
Visit our website at http://www.ubs.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mails are not encrypted and cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. UBS Limited is a company limited by shares incorporated in the United Kingdom registered in England and Wales with number 2035362. Registered Office: 5 Broadgate, London EC2M 2QS UBS Limited is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. UBS AG is a public company incorporated with limited liability in Switzerland domiciled in the Canton of Basel-City and the Canton of Zurich respectively registered at the Commercial Registry offices in those Cantons with new Identification No: CHE-101.329.561 as from 18 December 2013 (and prior to 18 December 2013 with Identification No: CH-270.3.004.646-4) and having respective head offices at Aeschenvorstadt 1, 4051 Basel and Bahnhofstrasse 45, 8001 Zurich, Switzerland and is authorised and regulated by the Financial Market Supervisory Authority in Switzerland. Registered in the United Kingdom as a foreign company with No: FC021146 and having a UK Establishment registered at Companies House, Cardiff, with No: BR 004507. The principal office of UK Establishment: 5 Broadgate, London EC2M 2QS. In the United Kingdom, UBS AG is authorised by the Prudential Regulation Authority and subject to regulation by the Financial Conduct Authority and limited regulation by the Prudential Regulation Authority. Details about the extent of our regulation by the Prudential Regulation Authority are available from us on request. UBS Business Solutions AG is a public company incorporated with limited liability in Switzerland domiciled in the Canton of Zurich registered at the Commercial Registry office with Identification No: CHE-262.289.477 and having its head office at Bahnhofstrasse 45, 8001 Zurich, Switzerland. Registered in the United Kingdom as a foreign company with No: FC034139 and having a UK Establishment registered at Companies House, Cardiff, with No: BR019277. The principal office of UK Establishment: 5 Broadgate London EC2M 2QS. UBS reserves the right to retain all messages. Messages are protected and accessed only in legally justified cases.