Engage – Was It Really Over A Week Ago?

It’s 2am so apologies in advance for any rambling in this post but I’ve been wanting to write about the Engage conference in Antwerp ever since I got back last Thursday (and if I leave it much longer I might as well write about next  year’s conference).

This year Engage was held in Antwerp which is only a 3.5hr drive for me so we met everyone else there who came by train.  Top tip – don’t try and drive in Antwerp, the one way systems will get you every time.  Yet another beautiful city and conference location by Theo and the Engage team.  The Elizabeth conference center was spacious and since there were 400 of us and the Engage team had made sure to provide lots of seating / meeting areas, it felt right.  One thing I really enjoy at conferences is the opportunity to meet people (OK I hate approaching people to talk but I like being part of a conversation) and I had the opportunity for some great conversations with sponsors and attendees. I managed to bore people to death about my latest obsession (docker).  IBM also sent a lot of speakers this year with Scott Souder and Barry Rosen updating us on Domino and Verse futures and both Jason Roy Gary and Maureen Leland there to sprinkle some (Connections) pink around.  There was a lot of open discussion about technology now and what we were each learning and working with along with a fair amount of enthusiasm for what we’re each working with, so thanks to everyone for that.

This year the agenda expanded to including emerging technologies and one of my sessions was in that track – on IoT in the Enterprise, GDPR and data.  I try to aim my presentations at the audience I’m talking to and when it comes to IoT the IT audience naturally has a lot more concerns then line of business managers.  Outside of IT IoT is purely about opportunity but since IT need to take care of the rest my presentation was more technical with a security checklist for deploying IoT devices.  All the opportunity for businesses will inevitably involve a lot of work from IT in the areas of data retention, data analysis, security and process redesign.  Some really interesting technologies are evolving and IoT is very fast moving as evolutionary technologies are so now is the time to start planning how your business can take advantage of the incoming swarm of data and tools.

My second session was on configuring a Domino  / Cloud Hybrid solution with step by step instructions for setting up your first environment.  That presentation is on my slideshare and also shared below.  The key thing to understand about hybrid cloud is that as a Domino administrator you still manage all your users, groups, policies and your on premises and hybrid servers, in fact the only things you don’t manage are the cloud servers themselves.  Getting started with a hybrid cloud deployment is a good way to understand what the potential might be for migrating or consolidating some of your mail services.

As always the Engage team put on an amazing event, lots to sessions to learn from, lots of people to meet and a lot of fun.  I was very pleased to see Richard Moy who runs the US based MWLUG event there for the first time and I’m looking forward to attending his event in the US in August.   Finally my crowning achievement of the week was when no-one on my table could identify either a Miley Cyrus or Justin Bieber song at the closing dinner and none of us considered cheating by using Shazam (I’m looking at YOU Steph Heit and Amanda Bauman :-)).  Theo promises us Engage will be back in May 2018 at a new location.   See you there.

So What About Domino @ IBM Connect? Review Post #2

Domino was very visible at Connect this year, not only in both of the opening sessions but in about 40% of the sessions overall.   The ones I picked to attend were talking about strategy and futures so that’s what I wanted to talk about here.

Verse on premises which shipped at the end of Dec 2016 is a very nice browser mail client right now which is easy to install on your Domino server (and you should) but it’s missing an updated calendar interface,  so I was pleased to hear the commitment to deliver that and other functionality to bring on premises in line with Verse in the cloud.  If you don’t have Verse installed on premises now on your Domino servers you need to be looking at it as your path forward.

Feature packs continue to be the strategic path with updates coming via FP installers but with template updates slipstreamed in optionally and separately downloadable through Fix Central.  I wouldn’t look for the templates to ship in step with the feature packs so you’re going to have to plan to subscribe to fix central for updates if you aren’t already.

From Barry Rosen’s strategy presentation here are a couple of snapshots showing planned feature pack features including those for FP8 which should ship soon.

Notes Feature Pack highlights screen-shot-2017-02-26-at-20-51-26

Domino Application Development feature pack highlights (FP8 shipping soon)screen-shot-2017-02-26-at-20-51-41

 

For application design the path IBM appear to be on is one we and many other Business Partners have been pursuing for some time with Domino as a back end data store and a web based UI on whatever platform you choose.  To that end the really good news is that we will finally be getting some extensions to the existing REST APIs including ones for

  • Directory
  • Mail Contacts
  • Mail File Search
  • Polling for changes in databases

In addition the application modernisation story at the conference was focused around partner solutions.  Of particular interest is Panagenda’s ApplicationInsights tool coming in a freemium model to all maintenace customers in Q2.  That version I believe will allow you to analyse your most prominent existing applications and instances to see what is being used by who and how. More information about it can be found here.

So lots of Domino sessions, lots of talk of future client and server developments, lots of confirmation of support at least to 2021.  For a nearly 30 year old product that’s not bad going.  With the investment in Verse and the introduction of cognitive features in on premises applications as well as a cognitive plugin for Notes, I’m feeling positive about where we are and the support IBM are offering.

Oh and my watch word for 2017 continues to be “Hybrid”

 

 

So THAT’S what it’s about

“A New Way To Work” – all of us in the IBM mail space have heard the phrase and seen presentations but does it actually mean anything to me – a set in her ways mail and calendar user who just wants mail to work and be fast?

In the past few months I have been using Verse On Premise in its beta form as well as the beta versions of Verse for iOS.  I had previously played around with Verse in the cloud but since my 20 year old mail file is on our Domino servers, the majority of my mail and searching activity was done via Notes and Traveler.  iNotes was never a client of choice for me.  I don’t archive mail very often (alright never) and so there’s a lot of history there.

So what have I found? I absolutely love Verse.  It’s fast and the layout is clean but as promised there are features that are so ridiculously useful** they have literally changed how I work for example

  1. Flagging for “needs action”  via a single click which then lets me set an action date for today, tomorrow, a week or other
  2. At the same time I can add a  personal note to the  message that travels with it,  and is visible when reading it in Verse.
  3. I can then remove the  message from my inbox by clicking on it.
  4. Viewing just those messages that need action which are automatically categorised by “today”, “tomorrow” etc.
  5. All of that, the action, the date, even the private notes transfer to Verse for iOS on my phone and iPad.

Add to all of this is the fact that it’s my same mail file on my same Domino servers. I could and can still open using the Notes client, using iNotes, even using Outlook if I want.  There are lots of things to enjoy about Verse and everyone works differently, but this is one group of features that live up to the hype of having tricked me without noticing into a new way of working 🙂

** no screenshots in here as it’s advanced beta still under NDA but the features have been shown publicly and hopefully will be in final product

From F to A In A Day

As I went to bed last night I set the alarm early, I have a lot to do this week especially since I’ll be at Icon UK for 2 days of it and I wanted to get started early.  So of course today was the day my work went out of the window and I lost 10 hrs debugging one of my own servers. Let’s back up…

This weekend I was prepping my presentations for Icon UK this Thursday.  One is called “Domino In The Back, Party In The Front” so I’m going to be talking about all the client options available to you using Domino as a back end.

On Sunday I had the idea of installing IMSMO (IBM Mail Services For Microsoft Outlook) on one of my lab machines.  I had a customer wanting to deploy and I wanted to try and mirror their setup, plus it meant I’d have something to demo from.  The lab server was already running 9.0.1 FP6 with a SHA2 SSL certificate delivering TLS1.2.  I hadn’t used any web services on it in a couple of weeks so I went ahead and added IF3 (required by IMSMO) and installed the application addin service.  It actually installs as a variant of Traveler (and I’ll be blogging on that later).  I completed the install and Outlook worked fine.  Unfortunately it was the only HTTPS service that worked.  Everything failed.  By failed I mean the browser – any browser – refused to connect.

So off I went to investigate why the browsers wouldn’t connect.  I started with testing via SSLLabs and that reported AN F as apparently the server was demanding SSLv3 instead of TLS 1.2   Of course just about every browser will refuse to accept a negotiation of SSLV3.  But why was the server suddenly demanding it when it had never done so before?

Well 10 hrs later I’d exhausted everything I could think of:

  • verified notes.ini had no additional unexpected settings
  • forced Disable_SSLV3=1 even though that server had been fine serving TLS 1.2 previously
  • disabled internet site documents and reproduced using web configuration
  • recreated the internet site and web rule documents
  • generated a new keyfile from my wildcard certificates
  • uninstalled IF3
  • uninstalled IMSMO including all the cleanup
  • scanned for anything that could be hijacking HTTPS
  • restarted and restarted and restarted http and clear cache upon cache upon cache
  • bothered Darren Duke for a sanity check – I believe the words “I don’t know what the hell is going on” came up
  • uninstalled Domino (around hour 8) because I couldn’t spend any more time troubleshooting

After uninstalling Domino. Reinstalling up to FP6, copying in the databases and templates and restarting.  I was back with TLS 1.2 again and suddenly SSLLabs was giving me an A+. 

Of course then I did what I should have done in the first place (saving time is never a time saver), I built a new lab server purely for IMSMO.  Installed FP6 and IF3 and the addin and everything worked perfectly including TLS1.2.

I have no idea what part of the IMSMO install , the addin or IF3, conflicted with my existing lab server configuration or what it did to force the server to only serve SSLV3 no matter how I tried to push it otherwise – but an uninstall and clean install ended up being my only fix in the time I had.  Someone somewhere knows the setting that made it do that.  I’d love to know what.

Now it’s 4.15am and I’m back where I thought I was at 11pm Sunday night.  The 4 days work I had to fit in 2 days , I have to fit in 1 day.  This week’s lesson. Never start something new when you barely have time to get the existing things completed.

See you at Icon UK

 

 

Domino 9.0.1 FP4 Crashes With HTTP On Linux and AIX

I discovered this on a customer site this weekend.  Their servers are running SLES Linux 64bit and already had Domino 9.0.1 FP2 on them.  I upgraded to  FP4 but only one of the clustered mail servers runs iNotes – that server kept crashing as soon as someone tried to access their mail.  The other server was stable and if I disabled HTTP the crashing server stayed up.

Turns out the IBM installer for FP4 on Linux and AIX is setting the ownership of the dojo folder incorrectly which causes the crash.  The dojo folder is under <notesdatadirectory>/domino/js and the ownership was set to invalid names.  From the js directory (which just has the dojo folder in it) I ran

chown notes:notes * -R

which told Linux to change the ownership of the dojo folder and everything beneath to the account / group used to run Domino.

There is a technote dated 28th August that i’ll post here but the fix on the technote is incorrect.  On their fix they say the permissions are wrong and need changing to 755 using chmod but that’s not true, they are already 755 in my installs but the actual ownership is wrong.  Maybe they’ll fix the technote but the background is here http://www-01.ibm.com/support/docview.wss?uid=swg21964549

Domino LDAP And A Failure To Authenticate

Bear with me and try not to shout at the screen “we all know that” – this blog is about the 10 hrs I lost yesterday troubleshooting a problem I distinctly remembering seeing before and realising, once I solved it, that last time it had also taken me hours and ended up being the same issue.  In my defence the last time I had this problem it was with Quickr so that’s a throwback and even if this blog isn’t news to you, it will hopefully be there for me in another 5 years…

I was using Domino as a LDAP source for Connections.  I don’t manage the Domino side of things for this customer so I had just asked them to add a secondary directory (in this case for External users) to Directory Assistance on their LDAP servers. I wanted the DA document set to be LDAP only rather than LDAP & Notes / Internet Authentication**. They did that and I tried to login from Connections to discover that I could login as a user in names.nsf but not as a user in the secondary directory. Time to look at the configuration.  Here’s what I did

1. Confirmed the DA document looked OK.  It actually wasn’t set to trust for credentials so I enabled that.
No luck.

2. Tried “sh xdir” to verify the directory was listed. It was, as Directory #4 out of 6.  Tried sh xdir reload to refresh Directory Assistance and then tried restarting the server
No luck but at least I knew DA was configured correctly

3. Turned on LDAPDebug=3 so I could see the debug information. At this point I could see the failing accounts (any in the secondary directory) were coming up with “authentication failure using internet password” in Domino and in the SystemOut.log of the WAS server that hosts the homepage application I saw references to PasswordFailedCheckException behind CWWIM4529E and SECJ0369E errors. Password failed? That made no sense at all.   One thing that was an issue was that the server I was working on was being probed every few seconds by a remote machine for availability on LDAP so with debug turned on the screen was filling up with thousands of lines of content making it difficult to see and track my own issues.  In retrospect if I’d asked for that to be disabled it would have saved me hours.

4. I then took a step back and installed Softerra’s LDAP Browser so I could test things outside of Connections.  I could bind using any credential in names.nsf but when trying to bind using a credential in the secondary directory I got “invalid credentials” and LDAP wouldn’t bind.

5. I then cut and paste a person document from the secondary directory to names.nsf to verify if the issue was the directory itself or the format of the person documents. I knew those documents worked fine on another server where they were in the names.nsf.  Turns out that if I moved them to names.nsf they worked fine.  I could bind with Softerra and I could login with Connections.

hmmm

6. I go back and check the ACLs of both names.nsf and the secondary directory.  I may even have bumped up default to something stupidly high *cough*Editor*cough* for 30 seconds to rule that out.
No luck

7. I paste the person document back into names.nsf again and bind with Softerra. This time I try and search for a name I know is in both the names.nsf and secondary directory (not the same name, just the same lastname).  Interestingly I get access denied / unauthorised – it finds the two entries but doesn’t let me see the content of them.  The fact that it found the entries meant that it could search LDAP but it can’t display them?  Surely that’s ACL issues.  So back I go to check the -default- rights on both directories and even test effective access for the specific account i’m using.  Nothing.

Then I see it.  As I try searching and searching and trying to catch errors on the server logs amongst the mass of LDAP debug information.. I see
searching directory names.nsf for sn=davis
searching directory directories\custnames.nsf for sn=davis
search directory directories\morenames.nsf for sn=davis unauthorised, skipping
search directory directories\externalnames.nsf for sn=davis
search directory directories\suppliers.nsf for sn=davis

Right there – in the middle. A directory I don’t care about, that has two dummy documents in it but happens to be part of Directory Assistance.  I go look at yes – -Default- is set to No Access. I change that to “Reader” and ta-da! suddenly I can both bind and login.  Then I remember I had this exact problem before at another customer with many directories that I didn’t set up or configure and it took me forever to find because I simply don’t touch what I’m not meant to be managing. In this case a directory that’s nothing to do with me and isn’t being used by my application on a server I don’t manage.

So what happened? It appears that Domino LDAP will search multiple directories but once it comes across one it can’t access with those bind credentials it doesn’t skip over it.. it stops.  The “skipping” isn’t strictly true.  So when the credentials were in directories one or two they worked. in directories four or five they failed because it stopped at directory three.

My lessons are
1. Remove as much extraneous activity as you can or you won’t be able to debug quickly enough
2. Always check everything (or in my case ask permission to check everything) even if it looks unrelated and especially if you didn’t set it up yourself 🙂

You’re welcome Gab of the future….

**Added on this morning.  Using LDAP only for authentication doesn’t work because a Directory Assistance document set to LDAP only doesn’t actually work for anything but LDAP searching. Not for authentication at all.  Foolish me for trying to be logical.  Here’s what the pop up help says – and they’re right. I tested it :-)]

DirectoryAssistance

THIS is how our community learns, thrives and has fun like no other

In just over two weeks’ time I’m heading to Atlanta for the MWLUG conference.  It’s my first MWLUG visit and this year’s conference is ridiculously packed with technical experts, champions, sponsors and more great content than you’re going to see in person anywhere else in the US this year.  Take a look at this schedule (you’ll see me on it).

4.45 on Thursday I have a Domino session called “What is your server trying to tell you“.  I’ve done similar sessions with this title before but I always update it to talk about the best tools and new tricks I use to manage or healthcheck Domino environments.  It’s great having a pure Domino Admin session so I hope you’ll stick around to catch mine.

11.30 on Friday morning I have a session on “Planning and Completing A Connections Upgrade” whether it’s a version upgrade in place, a side by side upgrade, a fixpack or a cumulative release I’ll talk about how to plan, what to look out for, how not to finish until you’re completely done and deciding when to upgrade and when not.  If you’re thinking of upgrading to CR3 which shipped last week this should be a valuable session.

If you haven’t registered go do that now and i’ll see you there (the weather should be balmy in August yes?) REGISTER