Generally any drop will affect the integrity of the helmet. If you drop it directly on the crown, write it off and get a new one. An Arai rep told me that this is the most vulnerable spot on the helmet.
Depending on where else and how high up it dropped from, you can keep wearing it. As a rule I'd say two or three drops and it's time to get a new helmet even if there isn't visible damage. And like other people said, you can always send it in to the manufacturer for them to check out. I know both Arai and Shoei will assess your helmet for free - all you have to do is pay for shipping.