This is something that is not to be taken lightly and it is certainly not the time to look for something cheap. Get a horse or pony that is safe, quiet, well-disciplined and will build your child's confidence. The last thing you want is for your child to get hurt and then be afraid of riding again.
The first horse or pony does not have to be perfect, but must be safe. You may come across one that is not that attractive or has a slight unsoundness. A gelding in most cases will be better than a mare because you won't have to deal with all the hormonal issues that mares can have.
An older horse or pony that has been ridden by children and proven to be safe with them would be ideal. When looking at the horse or pony make sure you see a child riding them and watch how the horse or pony behaves.
It is best to find a horse or pony to match the size of your child. Children can become discouraged and loose interest easily if they have to struggle to saddle and bridle the horse because it's to tall. A horse that is tall can be intimidating to a child. Children do fall off and with a tall horse it is further to the ground.
Yes I know children grow fast and will eventually outgrow the pony, but a good well-mannered pony will always be wanted by the next beginner. If you decide on a pony get one that has been trained by an adult because children do not have the experience to train them.
The worst thing you can do is buy a young horse or pony for your child to grow up with. Take the time do your research and don't buy the first horse or pony you come across. Ask lots of questions.
Before you buy a horse or pony, you might want to consider riding lessons for your child to see how serious they are about having a horse. Too many times parents purchase a horse for their child only to find out they've lost interest after a few months. Horses are time-consuming and take lots of care.
Take a trainer or riding instructor with you when looking no matter what your experience level is. Be prepared as it is not easy to find a beginner's mount and it might take several months to find the one that is a good match for your child. Make sure you put your child in riding lessons no matter how experienced of a rider you are, it will be the best thing you ever did.