Synthetic semantic supervision improves small transformer code embeddings

Small transformer encoders trained via contrastive pretraining on synthetically generated natural-language descriptions outperform pretraining baselines on five of eight tasks. This approach achieves performance parity with larger models across C, C++, and Java retrieval and classification benchmarks without requiring human-written docstrings.