You have to tap the character to whom the ball is being passed, at the moment just before the ball hits them. The one who "served" the ball must hit it first, after it comes back down. Tap your character at the right moment, and he or she will spike up the ball volleyball-style to another character. The ball goes to each of your party members once, and you must tap every one in the same manner. If you miss the timing of any tap, the spell will fail. If you get through all six characters without missing a tap, the spell is executed.
This is a lot of trouble and MP for a mediocre spell, although it does give your party a defense buff IIRC. It's kind of a nice novelty, but once you tire of it your MP is best spent elsewhere.
Brawlyball can be learned by talking to a student at Ambergris Prep. The student says that brawlyball was invented by another student from the school.