A Statement From IBM On El Capitan and iOS9 Support

IBM have today released a statement explaining why some applications will be unable to connect to Domino servers from iOS9 and El Capitan devices due to Apple removing support for Elliptic curve technology (no – me either) and enhanced transport security.  This doesn’t affect only IBM but it’s something you need to be aware of.  There will be an interim fix for Domino 9.0.1 FP4 and also a new FP5 to resolve these issues (eta end Sept) but there will be no fix for Domino 8.5.x servers.

The full statement and explanation is here but the key summary is

Additionally, IBM is working on an Interim Fix for 9.0.1 Fix Pack 4 (and the upcoming 9.0.1 Fix Pack 5) that will implement Elliptic Curve cipher support for TLS 1.2 and TLS 1.0 that remedies this issue and implements Elliptic Curve support for the following protocols: HTTP/HTTPS, LDAP/LDAPS, SMTP, IMAP, and POP3. Currently, the ETA for the Interim Fix posting is end of September 2015.

Elliptic Curve support will not be available for Domino 8.5.x releases since the specification requires updated cryptographic libraries that are available only in Domino 9.0 and above.

Domino 9.0.1 FP4 Crashes With HTTP On Linux and AIX

I discovered this on a customer site this weekend.  Their servers are running SLES Linux 64bit and already had Domino 9.0.1 FP2 on them.  I upgraded to  FP4 but only one of the clustered mail servers runs iNotes – that server kept crashing as soon as someone tried to access their mail.  The other server was stable and if I disabled HTTP the crashing server stayed up.

Turns out the IBM installer for FP4 on Linux and AIX is setting the ownership of the dojo folder incorrectly which causes the crash.  The dojo folder is under <notesdatadirectory>/domino/js and the ownership was set to invalid names.  From the js directory (which just has the dojo folder in it) I ran

chown notes:notes * -R

which told Linux to change the ownership of the dojo folder and everything beneath to the account / group used to run Domino.

There is a technote dated 28th August that i’ll post here but the fix on the technote is incorrect.  On their fix they say the permissions are wrong and need changing to 755 using chmod but that’s not true, they are already 755 in my installs but the actual ownership is wrong.  Maybe they’ll fix the technote but the background is here http://www-01.ibm.com/support/docview.wss?uid=swg21964549

Domino LDAP And A Failure To Authenticate

Bear with me and try not to shout at the screen “we all know that” – this blog is about the 10 hrs I lost yesterday troubleshooting a problem I distinctly remembering seeing before and realising, once I solved it, that last time it had also taken me hours and ended up being the same issue.  In my defence the last time I had this problem it was with Quickr so that’s a throwback and even if this blog isn’t news to you, it will hopefully be there for me in another 5 years…

I was using Domino as a LDAP source for Connections.  I don’t manage the Domino side of things for this customer so I had just asked them to add a secondary directory (in this case for External users) to Directory Assistance on their LDAP servers. I wanted the DA document set to be LDAP only rather than LDAP & Notes / Internet Authentication**. They did that and I tried to login from Connections to discover that I could login as a user in names.nsf but not as a user in the secondary directory. Time to look at the configuration.  Here’s what I did

1. Confirmed the DA document looked OK.  It actually wasn’t set to trust for credentials so I enabled that.
No luck.

2. Tried “sh xdir” to verify the directory was listed. It was, as Directory #4 out of 6.  Tried sh xdir reload to refresh Directory Assistance and then tried restarting the server
No luck but at least I knew DA was configured correctly

3. Turned on LDAPDebug=3 so I could see the debug information. At this point I could see the failing accounts (any in the secondary directory) were coming up with “authentication failure using internet password” in Domino and in the SystemOut.log of the WAS server that hosts the homepage application I saw references to PasswordFailedCheckException behind CWWIM4529E and SECJ0369E errors. Password failed? That made no sense at all.   One thing that was an issue was that the server I was working on was being probed every few seconds by a remote machine for availability on LDAP so with debug turned on the screen was filling up with thousands of lines of content making it difficult to see and track my own issues.  In retrospect if I’d asked for that to be disabled it would have saved me hours.

4. I then took a step back and installed Softerra’s LDAP Browser so I could test things outside of Connections.  I could bind using any credential in names.nsf but when trying to bind using a credential in the secondary directory I got “invalid credentials” and LDAP wouldn’t bind.

5. I then cut and paste a person document from the secondary directory to names.nsf to verify if the issue was the directory itself or the format of the person documents. I knew those documents worked fine on another server where they were in the names.nsf.  Turns out that if I moved them to names.nsf they worked fine.  I could bind with Softerra and I could login with Connections.


6. I go back and check the ACLs of both names.nsf and the secondary directory.  I may even have bumped up default to something stupidly high *cough*Editor*cough* for 30 seconds to rule that out.
No luck

7. I paste the person document back into names.nsf again and bind with Softerra. This time I try and search for a name I know is in both the names.nsf and secondary directory (not the same name, just the same lastname).  Interestingly I get access denied / unauthorised – it finds the two entries but doesn’t let me see the content of them.  The fact that it found the entries meant that it could search LDAP but it can’t display them?  Surely that’s ACL issues.  So back I go to check the -default- rights on both directories and even test effective access for the specific account i’m using.  Nothing.

Then I see it.  As I try searching and searching and trying to catch errors on the server logs amongst the mass of LDAP debug information.. I see
searching directory names.nsf for sn=davis
searching directory directories\custnames.nsf for sn=davis
search directory directories\morenames.nsf for sn=davis unauthorised, skipping
search directory directories\externalnames.nsf for sn=davis
search directory directories\suppliers.nsf for sn=davis

Right there – in the middle. A directory I don’t care about, that has two dummy documents in it but happens to be part of Directory Assistance.  I go look at yes – -Default- is set to No Access. I change that to “Reader” and ta-da! suddenly I can both bind and login.  Then I remember I had this exact problem before at another customer with many directories that I didn’t set up or configure and it took me forever to find because I simply don’t touch what I’m not meant to be managing. In this case a directory that’s nothing to do with me and isn’t being used by my application on a server I don’t manage.

So what happened? It appears that Domino LDAP will search multiple directories but once it comes across one it can’t access with those bind credentials it doesn’t skip over it.. it stops.  The “skipping” isn’t strictly true.  So when the credentials were in directories one or two they worked. in directories four or five they failed because it stopped at directory three.

My lessons are
1. Remove as much extraneous activity as you can or you won’t be able to debug quickly enough
2. Always check everything (or in my case ask permission to check everything) even if it looks unrelated and especially if you didn’t set it up yourself 🙂

You’re welcome Gab of the future….

**Added on this morning.  Using LDAP only for authentication doesn’t work because a Directory Assistance document set to LDAP only doesn’t actually work for anything but LDAP searching. Not for authentication at all.  Foolish me for trying to be logical.  Here’s what the pop up help says – and they’re right. I tested it :-)]


Configuring Domino for SPNEGO

I was recently asked to put together a live demonstration for a customer on how they could use SPNEGO to access their web based Domino applications. To do that I opted to build a lab environment consisting of a Windows domain controller, a Windows based Domino server and a Windows 7 workstation so I could have a clean environment to present with.

Here is a modified copy of that presentation which will hopefully help you configure SPNEGO in your own environment.

Thank you to Sean Cull at Focul for clearing this presentation so I could share it.

If enough people are interested I would consider doing a webcast of this with a demo against my lab environment.

Creating SHA-2 4096 SSL Certificates for Domino

I’ve been doing a lot of work recently re-creating SSL certificates for customers who have SHA-1 or who want stronger certificates, mostly because so many sites are now failing validation in standard browsers because of SHA-1.  IBM have published several pieces of documentation on how to do this but I wanted to share my bullet list on the quickest and simplest way I have found .  It’s not hard there are just lots of new steps.

Firstly you need to know that SHA-2 support only really started with 9.0.1 FP3.  That means the Domino Admin client you are going to use to do this work must be at that level (yes there are hotfixes for FP2 but go with the latest Fix Pack on your client).

You also need to know that NO Domino 8.5x server will be able to use the keyfile you create, it simply doesn’t have the cryptographic understanding to decode SHA-2.

Finally if you use the CA process to generate internet certificates you will need to upgrade the server running that process to 9.0.1 FP3 too.

Oh and you’re also not going to be using the Domino Server Certificate database to do this at all.

  1. Download 9.0.1 and FP3 for Domino Administrator and upgrade your client. Fix pack 3 is in Fix Central and 9.0.1 is on the IBM download site (CIQ91EN)
  2. Download the latest “lite” version of OpenSSL from here and install it on your Windows machine where you have Domino Administrator running.  I installed it in c:\openssl for example
  3. Download the kyrtool from here and copy the executable to your Notes program directory
  4. Set the environment variable for OpenSSL by typing in a command prompt
    Set OpenSSL_Conf=c:\openssl\bin\openssl.cfg (or whatever your path is)
  5. Now we create our keypair using OpenSSL.  From C:\OpenSSL\bin directory type
    “openssl genrsa -out server.key 4096”
    obviously you can use any name if you don’t want to use server.key and you don’t have to create a 4096 strength keypair.  When finished you should have a file in that directory called server.key
  6. Now you have your keypair you need to create a CSR to send to the certificate authority
    openssl req -new -sha256 -key server.key -out server.csr
    the server.key name must match what you created in step 5 so if you used a different name there you need to use that name here. Similarly your server.csr filename can be anything you like.  When you enter this command be prepared to answer all the questions about the certificate you want generated including the common name etc.  The CSR this generates will be uploaded to your CA (Verisign, GoDaddy, Thawte, whoever) and your SSL certificate created based on the answers you give to those questions.
  7. Now we need to create a keyring file ready to add to certificate into when the CA sends it back.  Go to your Notes program directory and run the kyrtool
    kyrtool  create -k c:\notes\data\keyring.kyr -p <passwordyouwanttouse>
    again the keyring.kyr file can be called anything you like.  Once run you should have both a keyring.kyr and keyring.sth files in your data directory
  8. By now your CA should have sent you your certificate as well as some trusted and intermediate root certificates for their issuer.  We are going to create a single text file that contains the server.key we generated in step 5, the SSL certificate the CA just sent us (usually a .crt or .pem file) and any intermediate or root certificates the CA needs us to use.  Doing this is very simple.  Go to your c:\openssl\bin directory (the one where your server.key file was created) and enter
    type server.key server.crt intermediate.crt root.crt >server.txt
    note all the filenames will be specific to whatever you were sent by your CA.  If you were sent a single bundle then you would use server.key bundlename.crt for instance
  9. To verify your server.txt is created successfully your can validate it using the kyrtool.  Go back to your c:\notes program directory and type
    kyrtool verify <path to server.txt>
  10. Now we import our server.txt will all the certificates into our newly created keyring file we created in step 7
    kyrtool import all -k c:\notes\data\keyring.kyr -i c:\openssl\bin\server.txt
    again your filenames and paths may vary depending on what you chose

And that’s it.  You now have a keyring (kyr) file and stashed password file (sth) you can copy to you Domino 9.x servers and use.  If you want to validate the keys are correctly in the file then you can again use the kyrtool

kyrtool show keys -k c:\notes\data\keyring.kyr  AND

kyrtool show certs -k c:\notes\data\keyring.kyr

IBM’s documentation on the process and the supported platforms is here and here

Domino LDAP Insufficient Access

Here’s where you all get to laugh and point at me for not knowing this sooner.  I was setting up Domino for LDAP access on a server with multiple directories in DA.  Everything was working fine until I wanted to write values from another source into the Domino LDAP.  Insufficient access.  OK so let’s check

  1. Account being use to authenticate has Editor access to the ACL in all directories in Directory Assistance
  2. Global Configuration document in Domino is set to allow LDAP write activity
  3. Global Configuration document in Domino is set to allow write activity that doesn’t conform to the schema
  4. I can login to the web interface of Domino using the LDAP credentials and successfully edit the person document I’m trying to change through LDAP

So what was my problem?  Apparently with LDAP write activity the Global Configuration document enabling LDAP to do writes has to appear in every directory used by Directory Assistance !  I finally got there by trial and error but that makes no sense at all, especially because the secondary directory doesn’t even need to use the pubnames.ntf template.  The Global Configuration document by definition controls LDAP activity for the entire domain which surely includes any secondary directories that are set up.  But that’s what it was.

I created a Global Configuration document in my secondary directory and set it to allow LDAP and write activity and my “Insufficient Access” went away.

Ooh look – wordpress has a poll facility , let’s try it.