RayOrch: Programming and Executing Lineage-Controlled Multi-Grain Dataflows for Foundation-Model Data Preparation
RayOrch keeps parent-child lineage intact while still batching GPU work across parents.
The paper targets data-prep pipelines where one document or video can expand into many ordered child tasks. RayOrch lets programs declare those variable expansions and matching gathers, then validates and tracks membership, ordinals, parent links, and terminal states at runtime. Its queues batch ready children across parents, while gathers rebuild results from declared lineage instead of completion order. The authors report up to 15.14x scaling speedup on NVIDIA H20 GPUs and lower end-to-end time than Ray Data and Daft in the tested pipelines. HF Daily Papers' note
The paper targets data-prep pipelines where one document or video can expand into many ordered child tasks. RayOrch lets programs declare those variable expansions and matching gathers, then validates and tracks membership, ordinals, parent links, and terminal states at runtime. Its queues batch ready children across parents, while gathers rebuild results from declared lineage instead of completion order. The authors report up to 15.14x scaling speedup on NVIDIA H20 GPUs and lower end-to-end time than Ray Data and Daft in the tested pipelines. HF Daily Papers' note
score 5