Integration Verse On Premises & Connections – Small Problem

This week I have been working on integrating Verse On Premises running on Domino 9.0.1 with IBM Connections.  Alright Connections 6 which isn’t supported right now.  The actual configuration is fairly simple and detailed on one page of IBM documentation.  It involves pasting some CORS redirection lines from the IBM document into your httpd.conf file and then replacing the server hostname with your own.

When I first did it against new servers the profiles integration worked fine but the Files (both accessing and uploading) as well as Follow/Unfollow actions failed.  That pointed fairly clearly to an issue with the OPTIONS request type and firebug confirmed that was throwing a 401 unauthorized error.  Finally I started to run out of ideas and asked Roberto Boccadaro if he could share his httpd.conf with me.  When I compared three separate environments the most glaring difference was with the SetEnvIf variable which is Step 5d on this page

SetEnvIf Origin “^https://(vop_server_hostname\.)?(domain_name)$” origin_is=$0

IBM’s instructions on that page tell us to use escape characters before each full stop so my variable setting would be

SetEnvIf Origin “^https://(mail\.)?(turtleweb\.com)$”origin_is=$0

However that was the problem.  The https:// caused the 401 error and I had to replace it with escaped forward slashes so my new URL is

SetEnvIf Origin “^https:\/\/(mail\.)?(turtleweb\.com)$”origin_is=$0

I applied that single change and everything immediately worked.

I have let the IBM documentation team know but if you have the same problem, check your SetEnvIf variable.

More Adventures In *** RHEL Configuration

I know I shouldn’t have blogged on Saturday – as soon as I think I have a problem fixed the universe rises up and slaps me roundly about the head.  So fast forward to the end, it’s Sunday night and I’m installing Connections on RHEL 7 so that’s good.  However to get there I had more hurdles which I’ll note here both for myself and for anyone else

I configured and enabled VNC and SSH for access which worked fine on the same network but not from any other network (“Connection Refused”).  The obvious first guess is that the firewall on the server hasn’t been disabled.  It’s always the first thing I do since I have perimeter firewalls between networks and I don’t like to use OS ones. So Saturday and Saturday night was an adventure in checking, double checking and checking again that I had the firewall disabled.  RHEL 7 has replaced iptables with firewalld but iptables still exists so my worry was that I had something enabled somewhere.  I didn’t think it could be my perimeter firewall since I had built the server with the same ip as an earlier server that already worked. My other guess was VNC being accidentally configured with –nolisten but that wasn’t true either.

By the time I went to bed Sunday morning I had ruled out it being the OS and was going to start fresh a few hours later.  I’d also noticed that although I could connect via VNC it was slow as hell despite having a ton of resources.

Sunday morning I decided to delete all the entries referring to that server’s ip on our Sonicwall perimeter device and recreate them.  That fixed the network access. The one thing I didn’t build from scratch was the one thing that was broken. *sigh*.

At this point I did consider switching to Windows 2016 on a new box but I already planned to use that for another server component and wanted to build with mixed OS. Also #stubborn.

So now I have VNC and SSH access but the GUI is awful. I can’t click on most of the menus and it keeps dropping out.  I’m running GNOME 3 and I can find endless posts about problems with GNOME 3 and Cent OS or Redhat so I bite the bullet and install KDE because all I want is a GUI.  KDE is as bad, slow, menus not clickable.  I make sure SELINUX is set to “Disabled” but still no luck.   I try installing NoMachine as an alternative method but that has the same problem with the GUI – slow, unresponding, menus unclickable and eventually a crash with “Oh no!, Something has gone wrong”.  Which isn’t nearly as entertaining the 100th time you see it.  Along the way I disable IPV6 entirely and found and fixed this bug

https://bugzilla.redhat.com/show_bug.cgi?id=912892

and this one

https://bugzilla.redhat.com/show_bug.cgi?id=730378

oh and this irritating setting

https://access.redhat.com/solutions/195833 “Authentication is required” prompt

Throughout Sunday I’m continually working with /etc/systemd/system/vncserver@:1.0 to modify the settings, create new instances, create new VNC users but no matter what I try it proves unworkable.

I’m using the Red Hat instructions from here which has a configurator you can use to automatically create the file vncserver@ file according to your settings.  I’m suspicious of that file because it has settings I don’t normally use like  -RANDR so eventually I edit the file and change

ExecStart=/sbin/runuser -l turtlevnc -c \”/usr/bin/vncserver %i -extension RANDR -geometry 1024×768\”
PIDFile=~turtlevnc/.vnc/%H%i.pid

To

ExecStart=/sbin/runuser -l turtlevnc -c “/usr/bin/vncserver %i -geometry 1024×768”
PIDFile=~turtlevnc/.vnc/%H%i.pid
Cleared the /tmp/X11.unix/X? directories and restart once more.  Everything including GNOME 3 works and it’s zippy zippy fast.

 

So. Note to self. Next time remove that RANDR setting and win yourself an entire day back.

 

Me vs Technology (spoiler: I win)

Yesterday Connections 6 shipped and although I was in meetings all day my goal for last night was to get everything downloaded and in place on a VM and have that VM built with a configured and hardened OS.  That was the plan.  I thought it might be fun to share my 4pm – 4am battle against technology and maybe it will help someone else.  It might also explain all the “other” work that tends to take up my time before I  ever get to the actual stuff I’m meant to be installing.

All my servers are hosted in a data centre and mostly I run ESXi boxes with multiple servers on them. I have 5 current ESXi boxes. So first things first, create a new virtual machine on a box with capacity so I can download the software.  All of this is done from a Windows VM on my Mac which connects to Turtle’s data centre

Vsphere lets me create the machine then gives me VMRC disconnected when I try and open a console.  After some checking I realise it’s the older ESXi boxes that are throwing that error for every VM and only since I upgraded to Windows 10.  If I can’t open a console on the VM I can’t do anything so I search the internet for various random advice which included

  • Disable anti virus
  • Remove Vsphere
  • Install latest Vsphere (which keeps being overwritten with an older one each time I connect to an older machine)
  • Uninstall VMware Converter (which I had forgotten was even there) – that required me booting into safe mode in my VM which only worked if I used msconfig to get it to restart in safe mode
  • Downgrade Windows
  • Create a new clean desktop VM to install Vsphere into

This is a bigger problem than just this install because I also can’t manage any of my servers on those boxes.  I rarely connect to them via the console so I don’t know how long it’s been like that but it can’t stay like that.

Several hours later.. still no luck. Vsphere lets me do everything to a virual machine except open a console.  I could use another ESXi box but I’m being stubborn at this point. I want to use this box

Then I find reference to VGC – Virtual Guest Console  https://labs.vmware.com/flings/vgc.  Created in VMWare labs in 2010 and still in “beta” it does one thing I need which is open a console.  So now I have VSphere where I can create and manage the instances and the VGC to open a console I’m ready to install and OS.

But which OS?  The host boxes have ISOs on them I already use but those are Windows 2012 R2 and RHEL 6.4.  I want either Windows 2016 or RHEL 7.1  Again I could use Windows 2012 but #stubborn.

I download Windows 2016 to my Mac and it’s over 5GB.  That’s going to take a few hours to upload to the datastore and I’m optimistically thinking I don’t have a few hours to waste.  So Plan B is that I take an existing RHEL 6.4 ISO and use that to install then upgrade it to 7.1 in place since you can now do that with Redhat if you’re moving from the latest 6.x to 7.x.  Top tip – it would have been quicker to upload Windows 2016.

I start building the new VM using RHEL 6.4 and eventually I get to the point where I can tell it to get all updates and off it goes.  It’s now 1am and it’s showing 19/1934 updates.  So.. I go to bed taking my iPad with me and leaving my laptop downstairs.  Once I’m in bed I can use Jump on the iPad to connect to my laptop which is on the same network and Terminus and the VPN on the iPad to open a putty session to the data centre.  The 6.4 updates finish and now I need to get it to 7.1  First thing I need to do is download 7.1 directly to that new VM which I can do easily because I installed a browser so I download the 3GB ISO directly to the VM which only takes 3 minutes and I’m ready to install.

Except not quite.  Redhat requires to you run their pre upgrade utility before doing an inplace upgrade.  In fact the upgrade won’t even run until you run pre-upgrade.  So I do that and as expected it fails a bunch of stuff that I don’t care about because this is a new machine and I’m not using anything yet so I’m not bothered if something stops working.  Except the upgrade still won’t run because it spots I failed the pre upgrade test.  That’s where “redhat-upgrade-tool -f” comes in.  Around 4am I left that running and got some sleep.

Incidentally this is a great document on upgrading but I think you may need a login to read it https://access.redhat.com/solutions/637583

At 7am I found it completed at RHEL 7.1 and then ran one more update to make sure everything was on the latest patches,  added the GUI and configured the firewall.

I’m NOW ready to download Connections 6

A Few Things From Connections 6 System Requirements

Connections 6 is due to be released on Friday but yesterday we had the announcement and the system requirements.  I took a look at them here and there’s a few things to be aware of if you’re an existing Connections customer

Operating Systems

  • Windows 2012 R2 and 2016 are both supported as server platforms but that’s it
    *Connections Content Manager isn’t supported on Windows 2016
  • Linux OS is SLES 12 or RHEL 7 so if you will almost certainly need to upgrade your OS if you’re already on Linux
  • Only 64bit OS are supported for servers
  • Installation Manager remains at 1.8.5 / 6 which can be upgraded in place if necessary
  • WebSphere Application Server is 8.5.5.10 which again can be upgraded in place if necessary
  • DB2 requires 11.1 minimum
  • SQL Server 2016 only
  • IBM HTTP Server 8.5.5 fixpack 10 required
  • Tivoli Directory Integrator 7.1.1 fixpack 6 is a minimum requirement and can be upgraded in place
  • Sametime integration requires Sametime 9.0.1 for chat and meetings

 

So those are the highlights that jumped out at me but the installation documentation isn’t out yet so I’ll find out more on Friday hopefully.  Start your engines…….

 

 

You Lie! Error Messages and When To Ignore Them

Building Connections this week and troubleshooting some errors reminded me to share the process I have adopted when dealing with IBM error messages – which is to treat them as hints that can set you on the right path but also send you badly down the wrong one.

Problem 1:

Installing Connections itself via Installation Manager.  One of steps during the install requires you to specify the DB2 server, the database names and credentials to connect to them.  I click validate and it fails  with error CLFRP0030E and launch error!.  That points to this technote which says I left a space after the hostname for the DB2 server.

I absolutely didn’t leave a space and didn’t copy/paste.  Just in case (and working on the assumption that it’s always me and not the product) I cleared it all and typed carefully again. I confirmed the hostname was correct and could be reached.  I also relaunched Installation Manager and started from the beginning.  No luck.

It’s  at this point I have to accept the error is referring to something else and that’s all the information I’m going to get from Installation Manager.  So now I move to asking myself “what if I saw no error but it just failed to connect”.  Well the first answer to that is to check if the connection details, hostname, credentials etc actually work at all.   Having confirmed the hostname and ports (there were no firewalls turned on or virus software), I logged into the DB2 server and checked the LCUSER account. Locked out.  I unlocked the account and the install then completed.

Problem 2

The test server in this environment is one box with everything DB2, TDI and all the applications on it.  My base WebSphere install was WAS 8.5.5 FP10 since Connections System Requirements for WebSphere 8.5.5 says FP8 and higher and I wanted to test that out. Everything installed fine right up to when I went to install Connections Surveys.  That’s when I hit a 2 day brick wall.  Installation Manager couldn’t connect to the Deployment manager despite it being on the same server.

screen-shot-2016-12-09-at-18-26-10

Well that’s odd.  Deployment manager is running.  The hostname resolves. The port is listening. I try to find out what the system requirements are for Connections Surveys but for 2 days last week and through the weekend the IBM system requirements pages for Survey were down.  I’m stubborn so I won’t let it go.  Even the Forms Experience Builder requirements for earlier versions were down.  So eventually I had to leave it and move onto the production build. The work needs completing and I was suspicious that the issue might have been installing everything on one server.

I build production across 4 servers and this time I stick with WebSphere 8.5.5 FP8 just in case.  When I get to the Surveys install it goes without a hitch.  So back to the test server I go.  Roll back Websphere to 8.5.5.0 and then forwards to FP8 (thank you Installation Manager!).  Surprise surprise Surveys installed perfectly.

So. Not an issue connecting to deployment manager or port or the server running but instead “Connections Surveys cannot install onto WebSphere 8.5.5.10 at all.

 

 

A Sametime Chat Mystery

Today I was contacted urgently by a site I did an install for back in early September.  The install went well and I left them several months ago with working components, but apparently about a week ago people stopped being able to login to the Community server. In fact not even the SSC could access it.

.. and yet no-one had changed anything at all.  I do love a good mystery so I thought it would be useful to someone (or even just future Gab) to document what I did:

  • verified if port 1533 was listening using netstat -an |find /i “1533”.
  • verified there were no running AV services that could interfere with the ports.
  • checked if the ST services were running, in fact only about 6 were.
  • tried to start some of the services that weren’t running and they failed immediately.
  • since no-one touched Sametime my next guess was a Windows update that caused a problem.
  • checked the Windows networking settings hadn’t been overwritten (they had) . Although those settings shouldn’t cause the services to fail completely it was worth resetting them.
  • I then added vp_trace_all=1 to the [Debug] settings in the sametime.ini which creates detailed log files in the \ibm\domino\trace directory.
  • having added that I could see log files being created for every service, even the ones that wouldn’t stay started. In fact those ones recreated every couple of minutes.  So the services were trying to start and failing.
  • reviewing the log files I could see on things like STPlaces there was a JVM error, but I put that aside for the time being in case it was a dependency issue.
  • in other logs such as STDirectory I could see broken networking errors and just before that I could see a comment about switching to TLS.

    A-ha! Well, that’s new.

  • checking the sametime.ini I found:
    VPS_PORT=1516
    VPS_TLS_PORT=1516

    which I changed to:
    VPS_PORT=1516
    #VPS_TLS_PORT=1516

    My guess being an incomplete TLS configuration from the SSC.  Having done that the server restarted perfectly and all services started.  The SSC could then access the server with no problem.

Of course once I had spent 4hrs doing that I then found a technote on it which I never would have found before I saw the TLS entry.  Here’s the technote .

Sometimes it’s a rollercoaster but so long as I get things working  I’m calling that a good day.  Now back to building more Connections servers.

 

Domino in the Back, Party in the Front

This is my presentation from Icon UK in September which discusses making client decisions whilst keeping Domino, the best mail server in the world, as your underlying architecture

How often do you hear that the business is discussing moving mail platforms because “our users want X” where X is nothing to do with the server and everything to do with the client UI. Domino remains the best mail server available but often user dissatisfaction drives a move and that comes from being asked to use the wrong client or from a bad deployment. If you’re using Domino you have an ever expanding range of clients to choose from browsers, iNotes, Verse, Traveler with iOS integration, Android applications, POP3 and IMAP. 

screen-shot-2016-10-24-at-09-51-18

The full presentation is here