These are all very good picture books for children that are about Christmas. I will add to this list as I find more.

Babushka by Sandra Ann Horn
The Little Drummer Boy by Ezra Jack Keats
A song in Bethlehem by Marni McGee
Why Christmas trees aren’t perfect by Richard H. Schneider
Eloise at Christmastime by Kay Thompson

Specifically featuring the Magi –

The Stone: a Persian Legend of the Magi by Dianne Hofmeyr
The Fourth King: The story of the other wise man by Ted Sieger
The Fourth Wise Man (retold by Susan Summers, based on the story by Henry Van Dyke)
Small Camel Follows the Star by Rachel W.N. Brown