I've been including several new instruments in my classes for young children lately - recorders, harmonicas, and ukeleles. Music is Elementary is a great source for these and many other instruments. It is located at 5228 Mayfield Road in Lyndhurst, phone number 800-888-7502. Be sure to say hi to Sadie the dog while you're there!
For recorders, I recommend a one-piece model (less to lose or fiddle with) such as the Aulos
A103N Soprano Recorder, Baroque ($7.95 each). With children under 5, I like to put a little bit of first aid tape on the top front hole and back hole. This means even little ones can play the note "B" rather than squeaking. If you hold your child on your lap, you can finger the recorder while they provide the air, playing simple tunes they'll recognize.
For harmonicas, a student model works well, such as the Hohner Student Model Harmonica for $5.95. Some children might try humming into it, but you can show them to breathe in and out to make the sound. This is great for exploration in the early years, growing to discovering of higher and lower notes, single notes and chords as they get older. Playing along with a blues recording is a fun activity with these.
Ukeleles are a great early instrument; I've been using the Lanikai Soprano Ukulele ($59.00) with children as young as 18 months. There are good teaching videos on Youtube, such as "Madeline's Ukelele Class" (Madeline Pots). Also, here is an iPhone/iPad app for tuning the ukelele: Tuner+
