Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities

Jobanputra, Mayank; Veitsman, Yana; Sarrof, Yash; Bakalova, Aleksandra; Demberg, Vera; Pavlick, Ellie; Hahn, Michael

by Mayank Jobanputra, Yana Veitsman, Yash Sarrof, Aleksandra Bakalova, Vera Demberg, Ellie Pavlick, Michael Hahn

Reference:

Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural AbilitiesMayank Jobanputra, Yana Veitsman, Yash Sarrof, Aleksandra Bakalova, Vera Demberg, Ellie Pavlick, Michael HahnAdvances in Neural Information Processing Systems (NeurIPS), 2025.

Bibtex Entry:

@inproceedings{born2025,
    title={Born a Transformer -- Always a Transformer?  On the Effect of Pretraining on Architectural Abilities},
    author={Mayank Jobanputra and Yana Veitsman and Yash Sarrof and Aleksandra Bakalova and Vera Demberg and Ellie Pavlick and Michael Hahn},
    booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
    year={2025},
	url={https://arxiv.org/abs/2505.21785},
	month={December},
    eprint={2505.21785},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}