Transition between Instability and Seeded Self-Modulation of a Relativistic Particle Bunch in Plasma