Do you have any plans regarding how you'd implement resilience and the equivalent of Spark's concept of 'lineage', where you keep a history of how a given RDD was computed, and then you can recompute it if it gets lost?
I haven't looked into Spark in depth, but I believe that 'lineage' relies heavily on Scala's delayed evaluation and the underlying Java RMI facilities. Doing something similar in C++ may require a lot more effort and a significantly different set of tradeoffs regarding the performance model.
I'm not that directly involved in Thrill, so I can't really speak with authority. There aren't any concrete plans on fault tolerance but it would certainly be an interesting topic to work on, partially because the existing solutions seem quite inefficient.
I haven't looked into Spark in depth, but I believe that 'lineage' relies heavily on Scala's delayed evaluation and the underlying Java RMI facilities. Doing something similar in C++ may require a lot more effort and a significantly different set of tradeoffs regarding the performance model.