Today I was contacted urgently by a site I did an install for back in early September. The install went well and I left them several months ago with working components, but apparently about a week ago people stopped being able to login to the Community server. In fact not even the SSC could access it.
.. and yet no-one had changed anything at all. I do love a good mystery so I thought it would be useful to someone (or even just future Gab) to document what I did:
- verified if port 1533 was listening using netstat -an |find /i “1533”.
- verified there were no running AV services that could interfere with the ports.
- checked if the ST services were running, in fact only about 6 were.
- tried to start some of the services that weren’t running and they failed immediately.
- since no-one touched Sametime my next guess was a Windows update that caused a problem.
- checked the Windows networking settings hadn’t been overwritten (they had) . Although those settings shouldn’t cause the services to fail completely it was worth resetting them.
- I then added vp_trace_all=1 to the [Debug] settings in the sametime.ini which creates detailed log files in the \ibm\domino\trace directory.
- having added that I could see log files being created for every service, even the ones that wouldn’t stay started. In fact those ones recreated every couple of minutes. So the services were trying to start and failing.
- reviewing the log files I could see on things like STPlaces there was a JVM error, but I put that aside for the time being in case it was a dependency issue.
- in other logs such as STDirectory I could see broken networking errors and just before that I could see a comment about switching to TLS.
A-ha! Well, that’s new.
- checking the sametime.ini I found:
which I changed to:
My guess being an incomplete TLS configuration from the SSC. Having done that the server restarted perfectly and all services started. The SSC could then access the server with no problem.
Of course once I had spent 4hrs doing that I then found a technote on it which I never would have found before I saw the TLS entry. Here’s the technote .
Sometimes it’s a rollercoaster but so long as I get things working I’m calling that a good day. Now back to building more Connections servers.
The first thing I do when building any product is go check the latest system requirements in case they have changed. That’s a bit of a challenge with Sametime since the system requirements are (and have been for some time) nonsense. No reference to WebSphere 8.5.5 and definitely no reference to the fixpacks or even the individual servers.
Try it yourself. Go here http://www-01.ibm.com/support/docview.wss?uid=swg27007792 and click on any of the requirements for 9.0.1 Complete, Communicate or Conference to see what I mean.
Buried in the actual help documentation is the phrase “Restriction: Most of the Sametime 9.0.1 servers that run IBM WebSphere® Application Server require version 18.104.22.168” but that doesn’t help anyone wanting detailed system requirements or who doesn’t find that page.
If anyone from the ST or documentation team sees this – please fix it.
My final New Way To Learn session today was looking at the Sametime mobile clients, Connections Chat and Sametime Meetings. I hope you find it useful and as always the full recorded session is available in the #NWTL Community.
The slides by themselves are below
In this session we looked at the architecture behind the Sametime mobile applications for chat and meetings. What do you need to deploy to support mobile users and what features are available to them on the different mobile platforms. We also looked at potential bottlenecks, security and troubleshooting for the mobile clients.
Today I did the second in my series of Sametime presentations for IBM’s New Way To Learn (NWTL) initiative. The session was recorded with audio and is available by joining the Community here http://bit.ly/1t7e0LE . The session slides by themselves are on slideshare and shown below.
If your Sametime environment is going to include Audio and Video you will probably want to be able to talk to people outside your own company, or at least to your own users on their mobile devices who aren’t connected via VPN. In this recorded online session as part of IBM’s New Way To Work initiative we reviewed the infrastructure behind the Audio and Video elements of Sametime and how best to extend those features beyond your firewall.
I have been participating in IBM’s New Way To Learn (NWTL) initiative with presentations around Sametime 9.0.1. The presentations are done online and recorded so they can be viewed later and are available with the audio recordings to anyone who joins the NWTL IBM Community. If you want to watch the presentation and see other great NWTL presentations you can join the community here http://bit.ly/1XXakab
My first presentation which was last week was on how to upgrade your Sametime 8.5.2 or 9.0 environment to Sametime 9.0.1. The slides without the audio are on Slideshare and shown below.
In this recorded online session we looked at all the options to upgrade your existing Sametime environment to Sametime 9.0.1. Whether you have only a single Community server on an early Sametime version or an entire infrastructure including audio and video on 9.0 we outlined how to plan for an upgrade and the pros and cons of doing the work side by side vs in place.
This week I will be presenting on upgrading Sametime to 9.0.1 as part of IBM’s New Way To Learn program (see here for details – requires login ). In preparation for that I wanted to take an existing environment I had and step through the upgrade of all components using the documentation. I discovered a few things I’ll share in my presentation and on this blog but one spectacular reoccuring critical full stop can’t move any further what was THAT – problem I thought best to share now.
After successfully upgrading the Community server (I know it was successful because the installer and the logs told me so 🙂 I discovered that the server couldn’t start the policy servlet. It was hard to see since all the other servlets started fine but if I watched the console as it tried to start I saw a servlet error when loading Policy and a message saying com.lotus.sametime.admin.policy.PolicyServlet could not be located. Luckily I’ve seen similar errors before in some 9.0 upgrades and on those it was the STCore.jar file which sits in the Domino program directory that was at fault. I took a backup of that STCore.jar and replaced the one in the program directory with one from a 9.0 server (bear with me, it was just to prove something) and sure enough, the server came up and launched Sametime this time finding the Policy servlet but missing the UserInfo servlet.
OK so I knew where I was. The STCore.jar that installed as part of the 9.0.1 upgrade was missing some policy files. I rename both the new 9.0.1 STCore.jar and the copy of my 9.0 STCore.jar to STCore.zip and then extracted them both so I could compare. I drilled down to the folder it claimed was mising com\lotus\sametime\admin\policy and in the screenshots below you can see my 9.0.1 version only has 4 files whereas my 9.0 version had 6 files including the missing one (PolicyServlet).
The STCore.jar as installed by the 9.0.1 upgrade
The STCore.jar from my 9.0 server
As you can see, the two missing files include the one the server was looking for. I extracted the two files and added them to my 9.0.1 folder then compressed everything again as STCore.zip and renamed to STCore.jar. I copied this new “fixed” (I hope) STCore.jar to the Domino directory and the server started with no problems. At least none I could immediately see.
I had come across this once before (an incorrect STCore.jar) on an earlier customer upgrade so it’s a recurring problem. I’m not sure what happens during the upgrade process – the file itself is dated 25th April 2016 so it’s not built during the install and isn’t broken for new installs. So two suggestions
1. Always backup STCore.jar before starting any upgrade along with sametime.ini vpuserinfo.nsf stconfig.nsf etc
2. If your server console is reporting a missing servlet during launch then verify that servlet exists in the STCore.jar
Like the sun breaking through the clouds on a gorgeous May holiday weekend, the IBM site has just published a document announcing Sametime 9.0.1 with a release date of May 3.
There’s no documentation or even system requirements out there yet but here are some delicious part numbers from the IBM download site to get your teeth into.
I’m not a big fan of installing without documentation but as soon as it appears I’ll be documenting both a clean install and an upgrade process. If you want any advice on how to upgrade your existing environment feel free to email me.