Connections Upgrade UI Weirdness

My latest PMR for a Connections 5.5 upgrade was concerning some very strange behaviour in the UI.  Seemingly unrelated but looking suspiciously like the same cause. Here’s what we discovered (and yes it was tested with multiple IDs , browsers and locations)

  1. The public files window had no scroll bar
  2. There were no upload buttons on the Files app
  3. When editing a profile picture I couldn’t remove it, only change it.  If I chose remove it just closed the profile with no changes.  The wsadmin profile command to remove the picture worked fine
  4. Editing a Community I owned just showed a blank page with a Community title but not editable fields

Weird right?

No errors anywhere in SystemOut or even in filebug or fiddler.  IBM could find no problems in the configuration or errors anywhere so today we did a screenshare to see what was going on.  After 90 minutes we had no luck, not even an error so Charlie Price joined the call as we talked over what I did to build the environment I explained it was a side by side upgrade where I

  • Built a new 5.5 server
  • Upgraded it to CR1
  • migrated the data to 5.0 CR1 databases using dbt.jar
  • upgraded those databases first to 5.5 then to CR1
  • used the migration tool to export artifacts from 5.0
  • copied the work folder the migration tool created to the new 5.5 server
  • used migration lc-import (from the updated tool issued D1 Dec 18) to import the artifacts

When I looked at the customization folder after completing the migration it was full of content that had been imported so the first thing I did when I saw this problem was empty that folder (by copying it somewhere else) and getting it to match my Connections101 5.5 CR1 server.  My suspicion was that somehow the migration lc-import had broken something.

Charlie compared with his system and noticed that both the javascript folder under Customizations and the webresources folder under provision were both completely wrong.  When we checked the 5.0 environment we saw that the contents matched that so must have been overwritten during migration lc-import.  The clue was the missing javascript and jar files for Ephox which I had installed and were now missing.  Charlie sent me a zipped up copy of his webresources directory and I overwrote mine with that and everything started working.

So.. be very careful using the migration wizard.  Take copies of your javascript and provision\webresources directories before doing an import and then make sure the files look up to date and there is nothing missing when you’re done.

 

 

 

Connections Failure To Authenticate

Last week was spent working on a PMR where a newly migrated (side by side) Connections 5.5 environment refused to let anyone access any applications.  I could login using any credential but the Homepage wouldn’t load and any application that required authentication failed including Communities.

Here are some of the errors in the logs

CLFRW0016E: Could not retrieve details for the user with login ID gabriella.davis@domainname.com due to an exception. The exception occurred when retrieving the details via the virtual member manager directly: {1} (in system out for utilcluster which contains homepage)

ADMN0022E: Access is denied for the expandVariable operation on AdminOperations MBean because of insufficient or empty credentials. (in ffdc)

“CustomAuthent E com.ibm.connections.httpClient.CustomAuthenticatorFactory <init> SONATA: authenticator class name is missing!  {in SystemOut for InfraCluster)

webapp E com.ibm.ws. webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]- [action]: com.ibm.tango.exception.AuthContextException: com.ibm. connections.directory.services.exception.DSException: com.ibm. connections.directory.services.exception.DSOutOfServiceException: java. lang.NullPointerException (in Systemout for InfraCluster).

 

Here (amongst others) are the things we tested / changed / reverted that didn’t fix it.  Bear in mind a working 5.0 production environment with the exact same configuration had no problems during this time.

  • LDAP was fine (we could login). For giggles we changed credentials and back again
  • We changed the login options from mail;cn;uid (which we use in this environment and works fine) to uid;mail;cn
  • We removed the mapped credentials for application security that were put there by the installer and put them back again - apparently that sometimes works
  • Set the authentication under application security for Communities and Profiles from None to Everyone just to confirm where the problem was
  • About 100 other things

Basically we managed to establish the issue was any intraservice communication but not why.  Eventually it went to L3 who isolated the error  as being something in the LotusConnections-Config.xml.

CustomAuthent E com.ibm.connections.httpClient.CustomAuthenticatorFactory <init> SONATA: authenticator class name is missing!  

That file had been migrated as an artifact via the migration tool and was the same as 5.0 but in there was the line <tns:customAuthenticator name=”DefaultAuthenticator” xmlns:tns1=”http://www.ibm.com/uiextensions-config”/>;

which they asked to be changed to <customAuthenticator name=”DefaultAuthenticator”/>

That immediately fixed the problem.

No-one is quite sure how that setting ever got into LotusConnections-Config.xml but my guess is during a CCM/Filenet installation.  The interesting thing is that it works in 5.0 but breaks 5.5. Maybe it requires you to have CCM installed to work as the 5.5 environment (mine or IBMs) didn’t have that.

Still a nice simple fix for such a painful problem and maybe somewhere for you to check when doing your own debugging.

Thanks very much to David McCarthy & the IBM L2 team for prioritising and working the problem.

Severe TDI Issue In Connections 5.5

I have been working with a customer who is migrating to Connections 5.5 from Connections 5.0.   When I do a migration I like to do it properly and create clean data by using dbt.jar to migrate content to new databases.  I know a lot of people are happy with the backup/restore of databases idea but for me that leaves too much scope for bad data to carry over from old system to new.

Everything was going fine, the profiles data migrated and then I tried a sync all dns to sync the ldap data to the database.  Something we schedule daily at this customer.  It failed as it tried to hash the database tables.  The error in the ibmdi.log was

Error: The sort page size property - source_ldap_sort_page_size= - must be greater than 10 if it is not 0. Aborting.

That’s a value that is set in profiles_tdi.properties and it was already set to 0.  So why was it aborting?

I decided to troubleshoot just with a cutdown list of names in collect.dns and using populate_from_dn_file function.  Again it failed but with the strangest error that would find the user in LDAP, get all their values then fail to find the user in the database and fail to update.

In SyncUpdates.log I could see the following error no matter what user I chose for populate_from_dn_file.

ERROR [com.ibm.di.log.FileRollerAppender.bc9c35a0-aae5-416e-9a99-1d418c3c564c] - [callSyncDB_mod] [ProfileConnector] null
java.lang.IndexOutOfBoundsException
at java.util.Collections$EmptyList.get(Collections.java:87)

I then tried copying the collect.dns to my 5.0 production environment and running there and it worked fine, found the users as duplicates and didn’t update them which is the correct behaviour.

I compared the map_dbrepos_from_source.properties files in 5.0 with 5.5 and it all looked pretty much the same.  So I opened a PMR which was eventually escalated to development. As soon as they received it they knew what the problem was - apparently a known but not documented bug that was fixed in CR1 with files that you have to manually deploy (we were already at CR1).

Development’s report of the problem was

log4j.logger.com.ibm.lconn.profiles.internal.service=ALL           
                                                                       
in log4j.properties causes TDI populate and sync commands to crash if an
EMPLOYEE is altered        

Well the crashing was true but the value  log4j.logger.com.ibm.lconn.profiles.internal.service=ALL was # out and unused so it wasn’t related to that particular log setting in my case.

The fix was to go find the two files

lc.profiles.core.service.impl.jar
lc.profiles.core.service.api.jar

in the Connections install and copy them to your TDI\lib directory in your tdisol environment.  In my case I had created a folder called TDISOL55 and under that I had a TDI directory with all the properties, script etc files in and the lib subdirectory full of jar files.  That came from the D1 (day 1) release download of Wizards which contained updated TDIPopulation directory and was dated 18th Dec.  There was no new tdisol with CR1 but clearly there should have been.

I found the files in my Websphere Application profile directory for the profiles application server under the directory

D:\IBM\WebSphere\AppServer\profiles\AppSrv01\installedApps\conn55Cell01\Profiles.ear

I copied those two files over and it al-most worked.  I had one more problem.  The value source_ldap_sort_attribute in profiles_tdi.properties which was initially set to empty (not null but = ) had been changed at the request of L2 to source_ldap_sort_attribute=mail which matched the 5.0 properties we were using.  They asked me to change it for exact comparison and that broke the updates.  Once I took out the “mail” mapping the scripts, both populate_from_dn_file and sync_all_dns ran perfectly.

The new environment does use different LDAP servers (but the same source data) and I don’t know if attempting to tell the server to sort the LDAP results failing is a problem with that server configuration (both environments are Domino 9.0.1) or 5.5 itself. I’ll investigate that and update.

So my two fixes were

  • copy the two jar files from your CR1 installedapps directory to your TDISol directory (lib subdirectory).
  • make sure source_ldap_sort_attribute= in profiles_tdi.properties

Determining Connections Versions

As I start a new Healthcheck today I thought I’d share a tip with you.  One of the first things I do when coming clean to someone’s Connections environment is try and determine what’s installed, including CRs and fixes.  Installation Manager is good at telling you what it installed but less so if you installed fixes outside of its interface.  There are other methods too like checking the version logs and reading the about.jsp, but it can be fiddly to piece together all the information.

One of the best resources I’ve found is an IBM technote from this July which shows how to identify exactly what fixes and CRs are installed.  The most comprehensive is updateSilent which produces a report on screen of every version, CR and iFix.  Here’s the table of what each utility can do.

The updateSilent utility is run from with the updaterInstaller directory under your Connections install and the command is:

sh updateSilent (bat or sh depending on your OS) -fix -installDir <ConnectionsInstallDir>

You may have to set WAS_HOME first before it will run so my commands in Linux are:

WAS_HOME=/opt/IBM/WebSphere/AppServer

export WAS_HOME

sh updateSilent.sh -fix -installDir /opt/IBM/Connections

It will then output to screen every CR and iFix that exists.

Icon UK, Upgrading Connections and Adapting To Technology

This week I was at Icon UK at IBM’s offices on South Bank and as usual it was a great event.  Lots of technical and non technical content, attendees, speakers and sponsors and it gave me the chance to talk to people I rarely get to see (some I hadn’t seen since January).  It always give me a boost to attend a community event and reconnect in person with everyone.  There were several sessions I missed and wished I could have attended so I’ll keep an eye out for the the published slides.  In the meantime I thought I’d share the slides from my two sessions.

The first one is on upgrading Connections - this is a new session and I tried to take a lot of content and present it in a more accessible way.  I’d be interested in feedback on whether you think it’s useful though

My second session was with Mark Myers and we talked about the importance of learning to change your technology.  There aren’t many slides because it was an informal talk but the point I certainly wanted to make was that as technologists or engineers we need to stop identifying ourselves with a brand or a product or a system and instead identify ourselves around our areas of expertise which can cross into lots of technologies.  Once you take a step back and do that you realise things like “Cloud” are not a threat, they are an opportunity to bring the skills you already have into play.

External User Registration Application - Some Screenshots And Details

Thank you everyone for the great feedback and interest in our external users registration app.  I had hoped that people would find it useful and I think we have a way of distributing it at no cost to anyone interested.

The app is a single Domino database which has two versions depending on whether you want users to be able to register themselves or be invited to register by your internal users.  I’ve tried to show both below

The Notes menu is very simple because it’s not intended to be used by anyone other than the occasional administrator.  Everything else is done via a web interface

The Notes client menu

The Notes client menu

First you need to set up the configuration telling the app where the Directory that will contain external user names is.  This the directory that TDI will reference when creating policies but user accounts aren’t copied into it until the registration process is entirely complete.

This setup is for the internal registration app

Setup

This is the registration page for the public registration where anyone can sign up for access. Obviously you could modify this to have further checks in place but bear in mind Connections only allows access for external users to Communities they are invited into so if I did register myself and login, I wouldn’t be able to see or do anything without a further invite.

We ask for an email address and confirm the registration back to that address asking the external user to click a link to activate - that way we ensure the email addresses are valid and monitored.  The code also checks that no one else has registered with the address already

RegistrationPublic

An external user would then receive an email with an activation link to click on

ConfirmationEmail

The registration page used for the internal invite model is slightly different but still checks the email address being registered is not already being used.

RegisterInternal

Then generates a unique registration code that can be emailed out to the external user manually (or automated if you want to add that code)

RegisterInternalConfirm

In each case the activation screen resulting from clicking on the link is the same. The password requirements can be modified by changing the code.

Activation And finally when the external user creates a valid password they get the following screen

So how do you get hold of a version of the app?  Obviously this is only part of the external user registration process which also includes LDAP and TDI configuration.  I would be very happy to quote on helping you with those pieces too but it’s not a requirement you use my consultancy to get access to the app, we are happy to make it available. I believe the setup can be completed in 2 - 3 hrs at most and again I’m happy to bill you to do that if you need me to or you can ask another Business Partner. You are welcome to take the app and support it yourself but in all cases our copyright remains in place (and is everywhere 🙂

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Please email Mike Smith (mikes @ turtlepartnership.com) or myself if you need more information or a copy of either the public registration or internal registration app. Bear with us and we’ll get it out to you as soon as we can.

Adding External Users To Connections - A Nice Simple Tool

Ever since Connections 5 gave us the ability to add external users to Communities it has been the number one requested feature from customers. The problem is that external users must exist in a LDAP source and also must have a profile in Connections that is created via TDI.

There are lots of ways to do this but few that are end user friendly and simple.  For that reason, some time ago we started to use our own XPages application that we make available to customers that automates the onboarding of external users into Connections.  The application is very simple and has two possible modes you can run in

  1. User self registration.  An external user can go to your website and create their own name / pw and email address. The system then sends an email to confirm the address is valid and asks them to click to activate.  On activation they create their own password that is checked against password requirements in the app (such as length, upper case, special character etc).  Once created the user can also self service a password reset via an email request sent to their account.
  2. User invite.  In this mode external users cannot create their own accounts but are instead invited by someone inside the company who again - goes to a webpage, creates a request and emails an activation link to their contact.  The rest of the process, activation, password checking, password reset remains the same.

It’s a single Domino database and can be set up in only a couple of hours.  Of course you still have to create the TDI sync but that’s a requirement no matter what you do.  For some time we’ve been considering how to make this tool available to the community at large since every customer we work with struggles with the same issue and we now have several good iterations of it we could share. We aren’t a product company and don’t want to sell it but we also can’t afford to commit to free support for a free download.

I’m not entirely sure of the answer. So far it’s been a non-issue since I’ve given it to customers we already do consultancy for or who ask us for consultancy. We’ve only charged for custom changes if required.  If you’d be interested in a copy of the database, seeing or testing it let me know and we’ll work something out.  It’s not open source, the code is our copyright but If you have any suggestions as to how we can make it more available without committing a lot of resource to productise it (which I would have to do for OpenNTF) I’d be very happy to hear them.

More IBM Docs Fun And Games

…a few more notes from my latest IBM Docs install.  Previous installs including in test at this customer proceeded with no problems but this one presented several challenges so I’m sharing them here in case anyone else has the same.  Firstly since there’s a Windows machine involved let’s rule out the biggest possible issues

1. Make sure Windows is activated. Microsoft does restrict behaviour and performance in non activated Windows. No I don’t have proof I just have solid evidence of that behaviour.  Activating Windows often makes the pain go away

2. Make sure you disable the Windows local firewall.  Even if you can only do so during the install.  The server is going to have to talk to - and be talked to - the deployment manager at least and with Windows firewall enabled your install will fail

3. Make sure every server can ping every other server, even itself. And using an IPV4 not IPV6 routable address

4. Disable UAC.  PLEASE.  In Windows 2012 that’s a registry hack where you set EnableLUA to 0 under “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\policies\system”

So now we’re ready to install.  There are two options - Installation Manager and using the manual scripts.  Obviously Installation Manager is easier, if you’re installing all components at the same time and if it works.  Here are the standard components I’d usually install for full IBM Docs in a Connections environment with no CCM.

My problem was that in this instance the installer failed during the Docs Proxy server install.  I could see in the logs (found under the IBM Docs Conversion install directory - in my case D:\IBM\ConnectionsDocs\Conversion\logs) that Conversion, Docs and Viewer all installed and deployed with no problems.  However since I chose six components, when it failed on one it rolled back the entire thing.

The error was “Target with name docsserver.domain.com was not found“.  Why would it say that when the script is running on doscserver.domain.com and it can certainly find itself?  The answer is in how the installer works.  It has local python scripts that are actually called by the Job Manager in your Deployment Manager so the error (which exists only on the docs server) is basically saying “the Deployment Manager cannot run the python script on this server”. That’s curious.  Then I realise that to run a remote script the Deployment Manager must contain a job target.  A configuration setting that tells it how to reach a remote server and gives it credentials to run the code.  I checked and although the installer had created a job target , when I  tested there were no stored credentials.  My guess is this was from an earlier attempt when UAC wasn’t fully disabled and the job target was created incompletely.  I re-created it to make sure it worked ok (it tests on save).

So back to square 1 (or snapshot 1).  I removed the half created clusters for Docs, Conversion and Viewer, I removed the Docs Proxy cluster, but I left the job target in place and relaunched the install.  This time my plan was to install in stages taking snapshots between each one.  This was a VERY bad idea.  Docs and Conversion installed and tested perfectly.  However when I went to Installation Manager and chose “Modify” to add the Viewer component it failed.  It took 8hrs to fail, during which time I monitored the logs carefully and this is what it did.

  • To modify an existing IBM Docs install and add a new component the install first UNINSTALLS all existing components even the working ones you may have installed months before
  • It then reinstalls the components it just uninstalled and attempts to install the new component as well
  • When that failed , it uninstalled all the components again and then reinstalled the original two. Leaving me back where I started 8 hrs later

It wasn’t so much the time lost as my fear that during the whole uninstalling / reinstalling of perfectly good servers it would somehow fail and break something that worked.  So.  New plan.

I now had a working IBM Docs and Conversion server to which I needed to add Viewer and Docs Proxy.  I’m staying away from Installation Manager at this point.  I want more control and I don’t want to waste another 8hrs before I can troubleshoot.  Luckily we do have the option to manually install components instead of using installation manager. To do that I extracted the installers and modified the cfg.properties files as per the documentation.  That worked fine after an initial failure.  The instructions don’t say to pre-create the Clusters and server members before running the scripts but you must do that and use the Cluster server names as in the documentation.  If you don’t, the scripts will fail when they try and connect to the deployment manager to find the servers to install onto.  If you’re using Installation Manager you don’t need to do this as the installer does it for you.

Finally there are test URLs as you install each component of <hostname>/componentname/version.txt eg http://connect.turtlepartnership.com/docs/version.txt.  To ensure this works you must regenerate and propagate the plug-cfg.xml and restart your IHS server.  Also bear in mind the syntax must be lower case /docs/version.txt /viewer/version.txt and /conversion/viewer.txt.

So there you go.  This was probably the 5th 1.0.7 install I’ve done and the first one to hit a problem. Try it first with Installation Manager. Make sure you backup (or better yet snapshot) both Deployment Manager and your  IBM Docs server before starting and if it starts failing switch to running the manual scripts.

Have fun!