The effect of the mesmer is to make the target do exactly what you want. It's generally used to force people to give up information.
In the third book, Artemis Fowl, Juliet and Butler trick the fairies into believing that they are mesmerized. However, they were acting, as all of them were wearing mirrored contact lenses.
To a human, the voice of the caster of the mesmer sounds like an entire choir singing in harmony. The sound is almost irresistible to a human. After the mesmer has been cast the human's eyes will look bloodshot and red, and therefore it is easy to tell if someone has had the mesmer used on them.
The eye contact needed for the mesmer can be obtained through a digital screen. When the mesmer was used to order a crowd to kill Butler and Juliet Butler the eye contact was made through a large screen.
By fairy law, a fairy can't use mesmer on another fairy. This law has been violated by Opal Koboi and Lt. Cudgeon. Although Leon Abbot (N'Zall) used mesmer, demons are not bound by fairy law, so he did not violate it.
In the Atlantis Complex it is shown that if mesmerized before you have grasped a language of any kind you will still obey the command given with quite exceptional grace and agility. After effects of early mesmerization are learning the language of Gnommish before other languages and limited magical skills. It is possible that Artemis has had the same part of his brain that allows magic to function activated as Artemis unlocks at the end of Artemis Fowl and The Lost colony.
In the Artemis Fowl film release, though not specifically identified as such, Holly Short attempts to use the mesmer on Juliet Butler. Trapped in a cage at Fowl Manor, she tells Juliet "I bet you have beautiful eyes. I wish I could see them," speaking in a distorted and layered voice and her own eyes going yellow. Juliet, however, is protected, and Artemis Fowl appears to tell Holly as much.