Bear with me and try not to shout at the screen “we all know that” – this blog is about the 10 hrs I lost yesterday troubleshooting a problem I distinctly remembering seeing before and realising, once I solved it, that last time it had also taken me hours and ended up being the same issue. In my defence the last time I had this problem it was with Quickr so that’s a throwback and even if this blog isn’t news to you, it will hopefully be there for me in another 5 years…
I was using Domino as a LDAP source for Connections. I don’t manage the Domino side of things for this customer so I had just asked them to add a secondary directory (in this case for External users) to Directory Assistance on their LDAP servers. I wanted the DA document set to be LDAP only rather than LDAP & Notes / Internet Authentication**. They did that and I tried to login from Connections to discover that I could login as a user in names.nsf but not as a user in the secondary directory. Time to look at the configuration. Here’s what I did
1. Confirmed the DA document looked OK. It actually wasn’t set to trust for credentials so I enabled that.
2. Tried “sh xdir” to verify the directory was listed. It was, as Directory #4 out of 6. Tried sh xdir reload to refresh Directory Assistance and then tried restarting the server
No luck but at least I knew DA was configured correctly
3. Turned on LDAPDebug=3 so I could see the debug information. At this point I could see the failing accounts (any in the secondary directory) were coming up with “authentication failure using internet password” in Domino and in the SystemOut.log of the WAS server that hosts the homepage application I saw references to PasswordFailedCheckException behind CWWIM4529E and SECJ0369E errors. Password failed? That made no sense at all. One thing that was an issue was that the server I was working on was being probed every few seconds by a remote machine for availability on LDAP so with debug turned on the screen was filling up with thousands of lines of content making it difficult to see and track my own issues. In retrospect if I’d asked for that to be disabled it would have saved me hours.
4. I then took a step back and installed Softerra’s LDAP Browser so I could test things outside of Connections. I could bind using any credential in names.nsf but when trying to bind using a credential in the secondary directory I got “invalid credentials” and LDAP wouldn’t bind.
5. I then cut and paste a person document from the secondary directory to names.nsf to verify if the issue was the directory itself or the format of the person documents. I knew those documents worked fine on another server where they were in the names.nsf. Turns out that if I moved them to names.nsf they worked fine. I could bind with Softerra and I could login with Connections.
6. I go back and check the ACLs of both names.nsf and the secondary directory. I may even have bumped up default to something stupidly high *cough*Editor*cough* for 30 seconds to rule that out.
7. I paste the person document back into names.nsf again and bind with Softerra. This time I try and search for a name I know is in both the names.nsf and secondary directory (not the same name, just the same lastname). Interestingly I get access denied / unauthorised – it finds the two entries but doesn’t let me see the content of them. The fact that it found the entries meant that it could search LDAP but it can’t display them? Surely that’s ACL issues. So back I go to check the -default- rights on both directories and even test effective access for the specific account i’m using. Nothing.
Then I see it. As I try searching and searching and trying to catch errors on the server logs amongst the mass of LDAP debug information.. I see
searching directory names.nsf for sn=davis
searching directory directories\custnames.nsf for sn=davis
search directory directories\morenames.nsf for sn=davis unauthorised, skipping
search directory directories\externalnames.nsf for sn=davis
search directory directories\suppliers.nsf for sn=davis
Right there – in the middle. A directory I don’t care about, that has two dummy documents in it but happens to be part of Directory Assistance. I go look at yes – -Default- is set to No Access. I change that to “Reader” and ta-da! suddenly I can both bind and login. Then I remember I had this exact problem before at another customer with many directories that I didn’t set up or configure and it took me forever to find because I simply don’t touch what I’m not meant to be managing. In this case a directory that’s nothing to do with me and isn’t being used by my application on a server I don’t manage.
So what happened? It appears that Domino LDAP will search multiple directories but once it comes across one it can’t access with those bind credentials it doesn’t skip over it.. it stops. The “skipping” isn’t strictly true. So when the credentials were in directories one or two they worked. in directories four or five they failed because it stopped at directory three.
My lessons are
1. Remove as much extraneous activity as you can or you won’t be able to debug quickly enough
2. Always check everything (or in my case ask permission to check everything) even if it looks unrelated and especially if you didn’t set it up yourself 🙂
You’re welcome Gab of the future….
**Added on this morning. Using LDAP only for authentication doesn’t work because a Directory Assistance document set to LDAP only doesn’t actually work for anything but LDAP searching. Not for authentication at all. Foolish me for trying to be logical. Here’s what the pop up help says – and they’re right. I tested it :-)]