OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Replacing a small slice of long-context training data with dependency-rich code contexts improved retrieval, state tracking, and coding tasks.
OctoLong builds million-token code contexts by recursively pulling references through an AST parser, language server, and package manager. The authors trained open long-context models from 600M to 14B parameters on a mix that included about 6.2B tokens from those contexts. In their ablations, using OctoLong for 12% of the context-extension corpus produced gains across long-range retrieval, repository-level code understanding, and agentic tasks. Source: ArXiv · AI/CL/LG's note.
OctoLong builds million-token code contexts by recursively pulling references through an AST parser, language server, and package manager. The authors trained open long-context models from 600M to 14B parameters on a mix that included about 6.2B tokens from those contexts. In their ablations, using OctoLong for 12% of the context-extension corpus produced gains across long-range retrieval, repository-level code understanding, and agentic tasks. Source: ArXiv · AI/CL/LG's note.
score 6