Cross-modal Steering Vectors Exist Across Both Image and Text Generation
A vector that makes text more formal can also steer generated images toward formal clothing and design.
The preprint tests unified multimodal models that share parameters for text and image generation. Vectors learned to steer concepts such as emotion and object size transferred in both directions without additional training. Because the researchers intervened in generation, the tests go beyond measuring correlations between representations. Strongly transferable vectors for the same concept tended to point in similar directions, despite a gap between text and image representations.
HF Daily Papers' note
The preprint tests unified multimodal models that share parameters for text and image generation. Vectors learned to steer concepts such as emotion and object size transferred in both directions without additional training. Because the researchers intervened in generation, the tests go beyond measuring correlations between representations. Strongly transferable vectors for the same concept tended to point in similar directions, despite a gap between text and image representations.
HF Daily Papers' note
score 5