What Kept Me Busy In 2016 and Where Am I Going Now?

I think this post might be just under the wire for 2016 reviews so let’s talk about what I was working on and learning for the past year.  I always need to be learning, if I’m not I feel like I’m standing still and last year most of my learning moved outside of the core IBM products simply because there was little new to learn.

So what kind of projects did I work on?

  • Security reviews of Domino, Connections, HTTP environments
  • Single Sign On projects including deploying SAML using ADFS and TFIM as well as lots of Kerberos / IWA integration projects
  • Designing hybrid environments for customers moving mail to the cloud
  • Lots of TLS configurations on lots of different products
  • IBM Connections upgrades to 5.5
  • IBM Sametime deployments from sites that had 8.5.2
  • Domino consolidation, maintenance and hardware migrations
  • High Availability for Traveler, Domino HTTP and Sametime

What was I learning?  I’m always looking for interesting and challenging technologies that can make a difference to those smaller customers who need to stretch a tight budget.  It’s how I got involved with Notes originally in the early 90s – It allowed me to make big changes quickly for smaller customers.  This year that has meant staying on top of cloud and hybrid security issues and single sign on products and technologies.  Beyond that I have become really interested in data visualisation and have been working with products like Tableau and some of its cheaper competitors to see what they can offer.

Then in December I signed up for a Lynda.com subscription to ensure I have a good grounding in wider technologies and how they can work together.  Of course signing up and actually making time to learn are two different things so that takes us to 2017.

Goals for 2017

  • More data visualisation tools / learning cool things to do with Tableau
  • Building myself a Lynda training plan
  • Deploying Verse on Premise for existing Domino customers and introducing those without Connections to that integration piece
  • More work with database technologies around performance and security
  • Identify ways to deploy docker solutions with better stability and security
  • Improving my languages (I’ve been working on Italian and want to learn Spanish)
  • Working on interesting projects or ones that make a difference

As you can see my “goals” are fairly loose, I am always open to new ideas for technologies to learn (except development languages – blech).  It may be my review of 2017 will be nothing like my goals list and I won’t consider that a failure.

 

 

From F to A In A Day

As I went to bed last night I set the alarm early, I have a lot to do this week especially since I’ll be at Icon UK for 2 days of it and I wanted to get started early.  So of course today was the day my work went out of the window and I lost 10 hrs debugging one of my own servers. Let’s back up…

This weekend I was prepping my presentations for Icon UK this Thursday.  One is called “Domino In The Back, Party In The Front” so I’m going to be talking about all the client options available to you using Domino as a back end.

On Sunday I had the idea of installing IMSMO (IBM Mail Services For Microsoft Outlook) on one of my lab machines.  I had a customer wanting to deploy and I wanted to try and mirror their setup, plus it meant I’d have something to demo from.  The lab server was already running 9.0.1 FP6 with a SHA2 SSL certificate delivering TLS1.2.  I hadn’t used any web services on it in a couple of weeks so I went ahead and added IF3 (required by IMSMO) and installed the application addin service.  It actually installs as a variant of Traveler (and I’ll be blogging on that later).  I completed the install and Outlook worked fine.  Unfortunately it was the only HTTPS service that worked.  Everything failed.  By failed I mean the browser – any browser – refused to connect.

So off I went to investigate why the browsers wouldn’t connect.  I started with testing via SSLLabs and that reported AN F as apparently the server was demanding SSLv3 instead of TLS 1.2   Of course just about every browser will refuse to accept a negotiation of SSLV3.  But why was the server suddenly demanding it when it had never done so before?

Well 10 hrs later I’d exhausted everything I could think of:

  • verified notes.ini had no additional unexpected settings
  • forced Disable_SSLV3=1 even though that server had been fine serving TLS 1.2 previously
  • disabled internet site documents and reproduced using web configuration
  • recreated the internet site and web rule documents
  • generated a new keyfile from my wildcard certificates
  • uninstalled IF3
  • uninstalled IMSMO including all the cleanup
  • scanned for anything that could be hijacking HTTPS
  • restarted and restarted and restarted http and clear cache upon cache upon cache
  • bothered Darren Duke for a sanity check – I believe the words “I don’t know what the hell is going on” came up
  • uninstalled Domino (around hour 8) because I couldn’t spend any more time troubleshooting

After uninstalling Domino. Reinstalling up to FP6, copying in the databases and templates and restarting.  I was back with TLS 1.2 again and suddenly SSLLabs was giving me an A+. 

Of course then I did what I should have done in the first place (saving time is never a time saver), I built a new lab server purely for IMSMO.  Installed FP6 and IF3 and the addin and everything worked perfectly including TLS1.2.

I have no idea what part of the IMSMO install , the addin or IF3, conflicted with my existing lab server configuration or what it did to force the server to only serve SSLV3 no matter how I tried to push it otherwise – but an uninstall and clean install ended up being my only fix in the time I had.  Someone somewhere knows the setting that made it do that.  I’d love to know what.

Now it’s 4.15am and I’m back where I thought I was at 11pm Sunday night.  The 4 days work I had to fit in 2 days , I have to fit in 1 day.  This week’s lesson. Never start something new when you barely have time to get the existing things completed.

See you at Icon UK

 

 

User Denied Access To Files and Wikis

Another PMR this week on a new 5.5 side by side build. Once built everything looked OK except a couple of users in IT who received access denied errors when going to Files or Wikis, everything else worked.  Those two applications have databases with pretty much the same schema so we often see matching problems in both applications.

Checking the application security I could see that both were set to All Authenticated so there was no reason why those users couldn’t get at files.  The browser error contained

Identifier: LC6C54CE35BA4D41BF8CB2461634B9EAE6 EJPVJ9275E: Unable to add a group with the directory ID [E7F267C7-8811-D8EC-8025-7E57004A5278, 4339D1D3-2F37-ACDB-8025-7E57004A5285, C0085F47-7A84-EFD4-8025-7E57004A51FA, 4DB58BD6-77EA-80AC-8525-6B700078923E, A5456CF5-9FA0-E49B-8025-7E57004A5316, 54578802-623A-2E18-8025-7E57004A5289, 4EEFAFD1-A098-4155-8025-7F1D00522430, 0D1FD4C5-F61E-CB15-8025-7E57004A51F6, 5A3E2519-52BE-F072-8025-7E57004A527B, 04CE2967-BD15-B84D-8025-7E57004A52F1, 0D162A8C-223C-33C3-8025-7EB4002F6ADF, 6DCCEAE9-6A16-2A75-8025-7E57004A5377, 2B5D0EBA-B225-BA42-8025-7E57004A52DF, C6B296E2-5D27-0F89-8025-7E57004A532A].

If I count how many directory IDs are listed there, there are 14 which matched the number of groups that user was a member of when doing an LDAP query.  Still we weren’t using groups for any access and this exact configuration was working for the same users in 5.0.

In the SystemOut.log I could also see

CWWIM4546E  Duplicate entries were found in the external identifier ‘d68bb54dea77ac8085256b700078923e’ in repository ‘d68bb54dea77ac8085256b700078923e’.

That ID (formatted in various ways) would not resolve to any group in Files or Wikis never mind to duplicates.

Eventually David McCarthy @ IBM got me to change the wimconfig.xml file on the deployment manager which fixed the problem.  My configuration didn’t exactly match the documentation which said to change

<config:baseEntries name=”o=ORGX” nameInRepository=”o=ORGX”/>
to
<config:baseEntries name=”” nameInRepository=””/>

my configuration only had <config:baseEntries name=”” – no ORGX and no nameInRepository at all.  I believe that’s because we use  Domino for LDAP and “root” as the base entry so my federated repository looks like this – a configuration that results in no entry for nameInRepository in wimconfig.xml.

Screen Shot 2016-06-29 at 14.52.26

Once more this isn’t a problem in 5.0 but possibly due to a change in WebSphere behaviour in a newer version, I had to manually edit wimconfig.xml to add the nameInRepository=”” value.

At IBM’s request I also added the Group Membership Attribute which is used for resolving nested group memberships.  This customer uses Domino for LDAP and doesn’t really use nested groups in Connections so in 5.0 it was empty and worked fine however 5.5 may have been struggling with resolving group memberships for some individuals.  In 5.5 having it set to empty could have been contributing to the access problem.

The screenshot below is from 5.0. Screen Shot 2016-06-28 at 19.13.56this is how I changed it in 5.5 (same LDAP source, same users, same everything else)

Screen Shot 2016-06-29 at 15.06.31

Resyncing and restarting then fixed the problem and the users concerned could suddenly access Files and Wikis.

Not sure why it didn’t work for those users before the changes but it could have been something to do with one particular group and its nesting or maybe even a replication conflict which I couldn’t find.

Go figure.

 

Ephox Textbox.io – documentation error

When installing Textbox.io, one of the rich text editors for Connections 5.5 from Ephox,  you have the option post install to configure a spellchecker.  It’s actually a very nice feature that spellchecks on the fly in any rich text field within connections.

To enable it you have to install one of the ear files that comes with the Ephox installers and configure a configuration file that allows the spellchecker to run.  It’s a simple thing to do and the instructions are here however a few issues you should be aware of

  1. The documented example refers to you using server ports but if Connections is correctly configured via IHS and you have regenerated the plugin-cfg.xml then you don’t need to add the server ports for access
  2. The example refers to only one  origin URL but often we have more than one.  To add additional origin URLs you add a comma and a space. My example is
     ephox { allowed-origins { origins = [ "http://connections101.turtlehost.net", "https://devtest.turtlehost.net"], url = "https://connections101.turtlehost.net/ephox-allowed-origins/cors" } }
  3. The biggest problem is that the documentation is wrong when it says where to create the application.conf file

    On Windows: BOOT_DRIVE_LETTER:\opt\ephox\application.conf where BOOT_DRIVE_LETTER is the boot drive for your system

    it’s fairly clear it wants me to put the file on the C drive which is the boot drive for Windows but if you do that the spellchecker won’t work and the URL

    https://connections101.turtlehost.net/ephox-allowed-origins/cors will return

    {"value":["http://localhost"]} 

    which is obviously a default value when the file can’t be found.  

    You need to create the application.conf file not on the boot drive but on whatever drive you have installed WebSphere for the deployment manager which could be the D, E or even Z drive. By creating the /opt/ephox directories there and an application.conf file the spellchecker will find it and start working.

Severe TDI Issue In Connections 5.5

I have been working with a customer who is migrating to Connections 5.5 from Connections 5.0.   When I do a migration I like to do it properly and create clean data by using dbt.jar to migrate content to new databases.  I know a lot of people are happy with the backup/restore of databases idea but for me that leaves too much scope for bad data to carry over from old system to new.

Everything was going fine, the profiles data migrated and then I tried a sync all dns to sync the ldap data to the database.  Something we schedule daily at this customer.  It failed as it tried to hash the database tables.  The error in the ibmdi.log was

Error: The sort page size property – source_ldap_sort_page_size= – must be greater than 10 if it is not 0. Aborting.

That’s a value that is set in profiles_tdi.properties and it was already set to 0.  So why was it aborting?

I decided to troubleshoot just with a cutdown list of names in collect.dns and using populate_from_dn_file function.  Again it failed but with the strangest error that would find the user in LDAP, get all their values then fail to find the user in the database and fail to update.

In SyncUpdates.log I could see the following error no matter what user I chose for populate_from_dn_file.

ERROR [com.ibm.di.log.FileRollerAppender.bc9c35a0-aae5-416e-9a99-1d418c3c564c] – [callSyncDB_mod] [ProfileConnector] null
java.lang.IndexOutOfBoundsException
at java.util.Collections$EmptyList.get(Collections.java:87)

I then tried copying the collect.dns to my 5.0 production environment and running there and it worked fine, found the users as duplicates and didn’t update them which is the correct behaviour.

I compared the map_dbrepos_from_source.properties files in 5.0 with 5.5 and it all looked pretty much the same.  So I opened a PMR which was eventually escalated to development. As soon as they received it they knew what the problem was – apparently a known but not documented bug that was fixed in CR1 with files that you have to manually deploy (we were already at CR1).

Development’s report of the problem was

log4j.logger.com.ibm.lconn.profiles.internal.service=ALL           
                                                                       
in log4j.properties causes TDI populate and sync commands to crash if an
EMPLOYEE is altered        

Well the crashing was true but the value  log4j.logger.com.ibm.lconn.profiles.internal.service=ALL was # out and unused so it wasn’t related to that particular log setting in my case.

The fix was to go find the two files

lc.profiles.core.service.impl.jar
lc.profiles.core.service.api.jar

in the Connections install and copy them to your TDI\lib directory in your tdisol environment.  In my case I had created a folder called TDISOL55 and under that I had a TDI directory with all the properties, script etc files in and the lib subdirectory full of jar files.  That came from the D1 (day 1) release download of Wizards which contained updated TDIPopulation directory and was dated 18th Dec.  There was no new tdisol with CR1 but clearly there should have been.

I found the files in my Websphere Application profile directory for the profiles application server under the directory

D:\IBM\WebSphere\AppServer\profiles\AppSrv01\installedApps\conn55Cell01\Profiles.ear

I copied those two files over and it al-most worked.  I had one more problem.  The value source_ldap_sort_attribute in profiles_tdi.properties which was initially set to empty (not null but = ) had been changed at the request of L2 to source_ldap_sort_attribute=mail which matched the 5.0 properties we were using.  They asked me to change it for exact comparison and that broke the updates.  Once I took out the “mail” mapping the scripts, both populate_from_dn_file and sync_all_dns ran perfectly.

The new environment does use different LDAP servers (but the same source data) and I don’t know if attempting to tell the server to sort the LDAP results failing is a problem with that server configuration (both environments are Domino 9.0.1) or 5.5 itself. I’ll investigate that and update.

So my two fixes were

  • copy the two jar files from your CR1 installedapps directory to your TDISol directory (lib subdirectory).
  • make sure source_ldap_sort_attribute= in profiles_tdi.properties

Sametime Audio and Video For External Users – NWTL

Today I did the second in my series of Sametime presentations for IBM’s New Way To Learn (NWTL) initiative.    The session was recorded with audio and is available by joining the Community here http://bit.ly/1t7e0LE . The session slides by themselves are on slideshare and shown below.

If your Sametime environment is going to include Audio and Video you will probably want to be able to talk to people outside your own company, or at least to your own users on their mobile devices who aren’t connected via VPN. In this recorded online session as part of IBM’s New Way To Work initiative we reviewed the infrastructure behind the Audio and Video elements of Sametime and how best to extend those features beyond your firewall.