So, I did say I hated ldap right? After fighting and fighting with I finally gave in and tried the JDBCRealm with Glassfish's security configuration.
So much nicer. There is a brilliant post here that explains how to do it. The only comment I would have is that you can sent the encryption to none instead of MD5 or something else.
So, yeah I pretty much wasted a week trying to do things the "right" way. I'm totally digging the JDBC Realm though. It works just the way you think it should.