This week’s Sametime PMR was a problem with Audio / Video on a newly deployed infrastructure.  This is a long blog but hopefully you’ll find it all useful. The installs all went fine and the peer to peer calling worked, which meant the clients were able to register with the proxy registrar.  However multi user or meeting video was failing.

The first thing you need to know about ST Audio / Video is that there are several moving parts - in this instance all servers are installed on SLES11

1. Proxy Registrar / Conference Manager - in this environment both these applications are installed into one instance of STMediaServer
2. Video Manager which is a WebSphere server installed as a standalone node (outside the SSC cell) and requires SolidDB (which the Video Manager installer places and configures)
3. VMCU - the Video MCU which will handle the multi way video traffic via the Video Manager

The second thing you need to know - and it’s not well documented at all - is that the start order of those elements is vitally important. Start them in the wrong order and you won’t get any audio / video at all (if you check your Sametime client preferences you will not see any A/V components or options).  So what’s the start and stop order?

Start with Video Manager components

1. Soliddb must be started first using /opt/soliddb/soliddb-7.0/bin/solid -c /opt/soliddb/soliddb-7.0/eval/standalone*
2. Once started the Video manager can be started using the server name STMediaServer
3. Start the Video MCU by typing  :  service soft_mcu start (also “status” and “stop”) work
4. Start the PR/CM WebSphere server STMediaServer

To stop all elements do 4-3-2-1 in reverse

To stop soliddb type solsql then when prompted for login details use the name and password admin
issue the commands (with a semi colon at the end of each line)

admin command ‘force shutdown’;

exit;

*soliddb listens on port 2315 - you can verify it’s running or stopped by doing a netstat. On linux that’s
netstat -an | grep -i “2315”

(the solid.ini file in /opt/soliddb/solidb-7.0/eval/standalone will tell you which port is being used by the server)

The next thing you need to know is that even if it all installed perfectly you must go through the process of exchanging certificates between the PR/CM in the SSC cell and the Video Manager standalone server.  This is documented here and this is where my PMR occurred   The problem was once the certificates were exchanged we lost all video completely.  Even peer to peer.  I assumed it was a small problem, maybe my start order or I wasn’t letting everything have enough time to start but no.. the problem was that we were using a wildcard certificate.

IBM do support wildcards, they have to since the ST Advanced server and ST Proxy server must share a certificate.  Unfortunately we discovered that the underlying video software (which actually comes from Polycom licensed to IBM) doesn’t support a wildcard certificate so when I did the exchange, everything broke.  Once I knew that I reverted the Video servers (PR/CM and Video Manager) to the IBM installed certificate (since the clients don’t directly connect there) and everything started working.

I am waiting to hear back from L3 if using the mixed certificates (wildcard for ST Proxy, Meeting and Advanced and IBM installed for the Video and SSC) will present any problems but right now we are back in business with all ST features.

Every since Sametime 8.5.2 was released I have seen a continual problem with Sametime trusted ips that  is still there in Sametime 9.0.1.  The issue is that the trusted ips list (which tells the Community Server which server ips to accept connections from) is now entered into the Sametime System Console in WebSphere and not directly into the CommunityConnectivity document in stconfig.nsf.  This means that since 8.5.2  the trusted ips in the Community Server configuration in WebSphere are then written to the Domino document at intervals.

So what’s the problem?  Well when WebSphere writes the list of trusted ips into the Domino document, it does so as a string, not as a list.  A small thing but that means when the Community server restarts the trusted ips don’t work as what Sametime sees is a long string instead of multiple values.  To fix this I wait until WebSphere has updated and then open and save the CommunityConnectivity document which refreshes and parses the string with commas in it into a list (since the field is a multi value list field anyway Domino is smart enough to do that).

Of course I then have to restart the server. Below are the examples of what I mean, first how WebSphere writes the values and secondly how Sametime needs to see them written.

I first opened a PMR on this back in 8.5.2 days and have tried occasionally since then  to open others but never got very far (around the time I am explaining Domino multi value fields to someone in China I lose the will to live). It always occurs if I have several ips to enter, not so much if there is just one or two.  The annoying thing is remembering to check every time I make any change to the Community Server configuration (which isn’t often once it’s setup).  Anyway, this has been my built in workaround for 3 years, it’s not hard and I know one of two other people out there have seen this too so here’s my “fix”…..

Building out another Sametime environment this week and I hit a roadblock. Fortunately because I’m a control freak I always read along with the documentation when I do an install, no matter how many times I’ve done it before.  I do this because it’s always possible IBM have updated their documentation since I last saw it…..and so I found,  buried in the documentation here, on the install page of the VMCU.. under

Deploying -

Deploying Common Component -

Deploying Audio and Video -

Sametime Media Manager on Linux or Windows -

Installing the Sametime Media Manager’s VMCU component -

Installing the Sametime the Sametime Video MCU - Step 9)

I find this

“Download and install the following prerequisite RPMs if they are not already installed.

For the list of RPMs to install, see the IBM Technote, List of RPMs to install on the Sametime Video MCU“

Yes a shiny list of pre-reqs required only by the VMCU and not on the system requirements.  Unfortunately they are all fairly old RPMs and at the current site although the packages are there, they are all newer versions of the ones needed.  The tech note is very specific about that

“Important: Each RPM’s file name includes a version number in the format X.X.X.Y, where X is a mandatory level that cannot be changed, and Y is a minimum level. If your RPM has a higher level for the value in the Y position, you can use it.”

So you may have zlib installed but if you have zlib-1.2.7-0.*.x86_64.rpm but the tech note calls for zlib-1.2.3-106.*.x86_64.rpm then you’re out of luck unless you can revert back to zlib-1.2.3. something

I assume the tech note (which is only a couple of weeks’ old) is a result of support having to deal with VMCU problems and determining those exact packages are needed for the VMCU to work.  It’s not a problem so long as you know about it and make sure those packages are in place before you start.

Thanks to Jeffrey Miller @ IBM for posting a blog page with links to all the latest fixes for the Sametime components.  He has offered to keep this up to date and I strongly suggest you bookmark the page (I did) to save trying to navigate through the hundreds of individual items on fix list and work out what supersedes what.

http://www.mymiller.name/wordpress/sametime/sametime-9-0-latest-published-versions/

I’ve recently run into a problem deploying Sametime Community Server 9.0.1 at two new sites and on an existing 8.5.2 IFR1 site which I’m not 100% convinced is the same issue but as part of my troubleshooting I discovered a missing piece of  policy behaviour that I”m finding extremely useful.

Prior to Sametime 9, policies were deployed on the Community Server and used the database stpolicy.nsf.  That database no longer exists in v9 and later.  In Sametime 8.5.2, if you didn’t deploy the System Console and just had a standalone Community Server you were still using stpolicy.nsf.  As of v9 of Sametime you can no longer do that as stpolicy.nsf no longer exists.   The Community Server must be deployed with the System Console in order to manage policies from within the Console itself. Carry on reading, that’s not the missing link:-)

Here’s a screenshot of the Sametime System Console showing where you set up policies, this is stored in the STSC DB2 database.

From here the policies are pushed down to Community server (Domino) at intervals (approximately hourly) or when the server or policy service restarts so they can be applied to users on login.  This means that clients logging in are receiving policies from the Community server, they aren’t directly looking up policies from the System Console.  If there’s a breakdown in communication between the SSC and the Community server, you can’t push policy updates down to the users.

When installing the Sametime Community Server, the default policy is to allow minimal features through the embedded client, things like screen capture, file transfer and rich text editing are disabled, however I have discovered on two different sites with new 9.0.1 installs, the changes to the default policy were not feeding down to the clients.  The problem was where to track this down.  The policy was right in the System Console but if I turned on POLICY_DEBUG_LEVEL=5 (in the [Debug] section of sametime.ini) I could see that the policy settings being applied did not match those from the System Console.  I even created and deleted additional policies and saw them continue to be ignored through reboots.

So where was the missing piece - somewhere the Community Server was picking up old values but with no stpolicy.nsf there was seemingly nowhere for me to find them.  A separate earlier PMR to IBM pointed me to two new (to me) Xml files on the Community Server file system (domino program directory)

policies.server.xml

policies.user.xml

These are where the System Console policies are written and updated and where the Community server policy service accesses the settings to deploy to users.  The date / time stamp on those files was suspiciously that of the original install, so they hadn’t been updated since then.  The next thing to check is why these weren’t updating.

The first thing to do is test that the Community Server can access and read policies using your wasadmin (or whatever your administrative account it) account.  To do that launch a browser on the Community Server and go to http://sscserver.turtlehost.net:9080/stpolicy/policy/all - you should be prompted for a login, give it your wasadmin name and credentials and the policies should display as a string of values in your browser.  If that works but the policies.server.xml and policies.user.xml files still aren’t updating then the problem may be with how you are telling the Community Server to connect to the SSC.

In the Domino program directory there is a “console” subdirectory and in there is a console.properties file that tells the Community Server how to connect to the System Console.  The contents of that property file are

SSCEncodedAuthorization= [the encoded password for the wasadmin account or whatever your admin account is}
SSCSSLEnabled]=false
SSCHTTPPort=9080
SSCHostName=sscserver.turtlehost.net
SelectedDeploymentId={deployment id of the community server plan in the SSC}
SSCHTTPSPort=9443
LogLevel=FINEST

What’s missing from there is the SSCUserName which identifies the name of the user who is going to login (usually wasadmin) and SSCPassword which contains the unencrypted password for wasadmin (removed and replaced with SSCEncodedAuthorization on first use).  Both of those were required in 8.5.2 versions but don’t seem to be there in 9.0.1  It may be that they shouldn’t be needed but twice now I have seen policies not update after initial install and adding those values to the console.properties , removing the SSCEncodedAuthorization and restarting fixed the problem permanently.  If you add the SSCPassword and remove the SSCEncodedAuthorization you can tell if the connection to the SSC was successful because the properties file will then remove the SSCPassword and replace the SSCEncodedAuthorization.

So there you have it - three missing pieces to help debug policy deployment in Sametime

1. The Domino server based XML files policies.server.xml and policies.user.xml

2. The URL http://sscserver.turtlehost.net:9080/stpolicy/policy/all

3. The console.properties file in the console subdirectory under the Domino program directory

Someone on our Sametime exam team questioned this this morning and I realised it definitely needs to be publicly called out. The Sametime 9 Community Server no longer has any stpolicy.nsf database or a policies view under the old school web based admin. If you upgrade to Sametime 9 you must install the system console (and db2) to be able to manage and maintain policies going forward.

Something for your planning…

As I said when Sametime 9 shipped, I wanted to spend a few weeks working with it and trying to install it and migrate my existing sites before I blogged.  I’m coming near the end of that now and so wanted to share a few things.  This first blog is about Sametime Communicate which includes Domino , Sametime Community Server, DB2, LDAP, Sametime System Console and Sametime Proxy.  It also includes installing the Sametime Advanced server for Persistent Chat and Broadcast Tools but I want to talk about that separately.

Whether you have installed Sametime 8.5x with WebSphere components or not, Sametime 9 and its install is a very different proposition.  I’m going to start by saying that I would never attempt to upgrade an existing install of WebSphere elements.  IBM in fact say that you should do a side by side upgrade and then move the existing databases for the System Console, Meetings, Advanced and ST Proxy (possibly) over.  That basically involves building an entirely new environment and then switching DNS when you’re ready so your users point there.

It’s my nature to be risk averse and in my testing migrating the existing System Console database is a nightmare. The version of DB2 you should use for Sametime 9 is 10.1, so that means that you’d have to upgrade the database as you migrate. In addition, the schema for the Sametime 9 system console database is not the same as for Sametime 8.5x and, though you can theoretically fix that using the scripts IBM supply, I would rather start completely clean.  The only databases I would make an effort to migrate over are the Meetings and Sametime Advanced because they contain data you can’t lose.  Even so there are no good instructions in the documentation for migrating a Sametime 8.5x Meetings database on DB2 9.7  to a Sametime 9 Meetings database on DB2 10.1 - I would contact IBM support in advance and ask for a tech note with instructions because the documentation has some large gaps there.

Of course, if you don’t have Meetings or ST Advanced right now then you can go ahead and create shiny new databases for your new install.

Download: The first step is to download all the software and get it in place.  Sametime 9 uses WebSphere 8.5 which installs differently than with previous versions of Sametime.  It’s actually a much nicer and easier to manage install, but you will need to install WebSphere by itself before you can install any of the Sametime components.  Make sure you download the version of WebSphere and Installation Manager that is part of the Sametime eAssembly or verify very carefully with the system requirements that you are installing the right version.  Sametime 9 uses WebSphere 8.5 (no fix packs) with additional Sametime specific iFixes, all of which can be downloaded together.

DB2: The version of DB2 supported for Sametime 9 is now 10.1 which is very different in UI from DB2 9.7. For starters, there is no longer a Command Center with a graphical interface allowing you to see and manage databases.  You have to install a separate DB2 client if you want to access the DB2 server and look at the databases. You can install that client on any machine that can access the DB2 server.

WebSphere:  One of the main reasons an in-place upgrade can’t be done is that the underlying version of WebSphere has changed and can’t be upgraded for Sametime.   We have to install WebSphere cleanly.  When installing WebSphere 8.5 you’ll notice the download comes in three parts.  You’ll need to extract all three parts to the same directory which will then contain folders disk 1, disk2 and disk 3 and a file called repository.config in the root folder.  When you install Installation Manager you can then use it to install WebSphere and every other product (other than Domino and the Community Server). You launch Installation Manager and point to the folder where you put your extracted files, it will do the rest.  It sounds complicated but it’s actually very simple and has a huge advantage in that it’s able to search the IBM site for fixes and updates rather than download them each time.

Launch Installation Manager - Choose File - Preferences from the menu and set up your repositories as I have done below (these point to the fixes which were zip files, these didn’t need to be extracted but I wanted them listed separately so I could check them)

Community Server: When installing the Community Server, IBM have added some much needed additional steps to the documentation providing details on performance tuning Windows 2008 and 2012 networking and securing the server to protect against vulnerabilities discovered in the past few years.  None of this is new, it was all public information in technotes but it’s good to see it brought together in the documentation as part of the deployment instructions.  Don’t be tempted to skip over these steps and come back later, they will double the amount of time it takes to install a Community server (from about a day to about a day and a half) but they are important.

If you are moving from an earlier version of Sametime you will need to be using LDAP if you aren’t already and you can’t use your Sametime Community Server as its own LDAP server, that’s not supported and will  present problems.  In fact you should disable LDAP on the Domino server running Sametime completely.

Sametime Proxy Server: The Sametime Proxy server is used for mobile clients, for awareness in web based meetings, for a browser based IM client and more.  You need to install this as a WebSphere component.  It is IBM’s recommendation that each component have its own VM but I have had success in the past co-locating multiple server elements depending on number of users.  There are a few more  settings some of which were available in Sametime 8.5x but again in technotes, etc and so weren’t well known.  Once a Sametime Proxy Server is installed there are several steps to finish the install, as with the Community server, that will improve performance and security. One interesting item that everyone now will probably come across is that the Sametime Advanced server must use the same SSL certificate as the Sametime Proxy server for awareness to work, making wildcard certificates more suitable to our installs.  Previously I had avoided wildcard certs since WebSphere had issues with them in earlier releases but that appears to be resolved now.

Additional steps on completing the install of Sametime Proxy include making sure you connect to the notification servers for both Apple and Google to ensure mobile devices running iOS and Android can receive updates.  There are also settings to tell the Sametime Proxy server to not connect to the user’s home Community server allowing you to explicitly direct traffic to a dedicated Community member instead.  Instructions for that here.

Finally we usually have a WebSphere Proxy server in front of our Sametime Proxy to handle traffic over port 443.  In the Sametime 9 documentation IBM now seem happy to recommend a reverse proxy for accessing  the Sametime Proxy (I have customer doing that and using products like Netscaler) and only using a WebSphere Proxy in front of a cluster of servers.  The WebSphere Proxy is an intelligent authenticating server that will validate the user prior to directing traffic to a Sametime Proxy server.  If you have multiple Sametime Proxy servers in a cluster, the WebSphere Proxy may redirect the traffic to any of them.  Performance tuning for the WebSphere proxy has been nicely consolidated here.

This was meant to be a short blog entry, obviously I haven’t covered everything but hopefully I have given you some pointers.  More to follow…