I know I'm naughty, but I always listen carefully when this word is uttered, because I assume it has been used as if it were singular again. Usually it is.
This one's for Gordon:
Wrongly used: That's no criteria
Rightly used: There are several criteria that determine if a piece of music is worth listening to.
It derives from a Greek word, whose singular is [in English transliteration] criterion. The plural in Greek, transliterated into English is our friend criteria.