Connections CCM Problems – Libraries not “quite” working

My 2nd PMR this week was for Connections and Content Manager.  I had already installed FileNet in the test environment for this customer and the only difference between test and production was really the number of servers with CCM having its own dedicated VM.  The install completed and I tested uploading files, editing files, clicking “like” etc and it all seemed OK so I handed it over to the customer.

Turns out there was a problem.  No library , once created, could be edited.  Not the title, not the security, nothing.  Any editing threw up an error

All the installs were correct.  The updates had applied OK.  The correct versions of FileNet were running.  We even checked the security on the ICObjectStore in FileNet’s ACCE administration interface.  Nothing looked wrong but the error message was strange

“The requested approval  action could not be performed because the library, CCM Libraries, is not enabled for document approval. The library’s repository, ICObjectStore, might not have the document approval addon installed, or the library might not be a teamspace. Contact your administrator and report this error message”

As part of the investigation trying to find out what was wrong (and whilst waiting for L3 to review) I saw this option when I right clicked on the ICObjectStore in ACCE – add on features.  AddOnFeatures

 

So , whilst we waited, the IBM support guy (can I name him here?) sent me a list of all his add ons and I compared them to all of mine and sure enough about 8 were missing.  I added those and everything started working.  Why those 8 failed to install is another matter since all the logs said everything installed fine.  Interestingly at this customer we’ve had trouble deploying applications in the past due to network timeouts between the Dmgr and other servers so I do wonder if that was it (for instance CR3 seemed to install but several of the applications were corrupted when we tried to use them and I had to install them again manually).

An interesting one and a nice easy fix.  I’ve added screenshots below of all the add ons we should have had so you can compare if you find a similar problem.

Addons Pt1 Addons Pt1

 

 

Sametime Audio and Video Problems

This week’s Sametime PMR was a problem with Audio / Video on a newly deployed infrastructure.  This is a long blog but hopefully you’ll find it all useful. The installs all went fine and the peer to peer calling worked, which meant the clients were able to register with the proxy registrar.  However multi user or meeting video was failing.

The first thing you need to know about ST Audio / Video is that there are several moving parts – in this instance all servers are installed on SLES11

  1. Proxy Registrar / Conference Manager – in this environment both these applications are installed into one instance of STMediaServer
  2. Video Manager which is a WebSphere server installed as a standalone node (outside the SSC cell) and requires SolidDB (which the Video Manager installer places and configures)
  3. VMCU – the Video MCU which will handle the multi way video traffic via the Video Manager

The second thing you need to know – and it’s not well documented at all – is that the start order of those elements is vitally important. Start them in the wrong order and you won’t get any audio / video at all (if you check your Sametime client preferences you will not see any A/V components or options).  So what’s the start and stop order?

Start with Video Manager components

  1. Soliddb must be started first using /opt/soliddb/soliddb-7.0/bin/solid -c /opt/soliddb/soliddb-7.0/eval/standalone*
  2. Once started the Video manager can be started using the server name STMediaServer
  3. Start the Video MCU by typing  :  service soft_mcu start (also “status” and “stop”) work
  4. Start the PR/CM WebSphere server STMediaServer

To stop all elements do 4-3-2-1 in reverse

To stop soliddb type solsql then when prompted for login details use the name and password admin
issue the commands (with a semi colon at the end of each line)

admin command ‘force shutdown’;

exit;

*soliddb listens on port 2315 – you can verify it’s running or stopped by doing a netstat. On linux that’s
netstat -an | grep -i “2315”

(the solid.ini file in /opt/soliddb/solidb-7.0/eval/standalone will tell you which port is being used by the server)

The next thing you need to know is that even if it all installed perfectly you must go through the process of exchanging certificates between the PR/CM in the SSC cell and the Video Manager standalone server.  This is documented here and this is where my PMR occurred   The problem was once the certificates were exchanged we lost all video completely.  Even peer to peer.  I assumed it was a small problem, maybe my start order or I wasn’t letting everything have enough time to start but no.. the problem was that we were using a wildcard certificate.

IBM do support wildcards, they have to since the ST Advanced server and ST Proxy server must share a certificate.  Unfortunately we discovered that the underlying video software (which actually comes from Polycom licensed to IBM) doesn’t support a wildcard certificate so when I did the exchange, everything broke.  Once I knew that I reverted the Video servers (PR/CM and Video Manager) to the IBM installed certificate (since the clients don’t directly connect there) and everything started working.

I am waiting to hear back from L3 if using the mixed certificates (wildcard for ST Proxy, Meeting and Advanced and IBM installed for the Video and SSC) will present any problems but right now we are back in business with all ST features.

The IBM Support Overnight Mystery

Several days this week I have worked on a different PMR (two ST bugs one CCM more on later) with people from IBM support who have been helpful, informed and as curious about the problem as I was (or faking it really really well) . We’ve had screen shares, investigated the problem and left it at the end of day the as “escalate to L3 development”.

Then each morning I wake up to an overnight email from someone new saying they are in charge of the PMR but who has seemingly never seen the problem and is asking me to do basic stuff like send in logs or apply a patch that was already checked (and updated in the PMR) at least a day earlier.

I understand the difficulties in providing 24×7 support and I’m sure there’s an alert somewhere that gives someone a kick overnight and tells them I HAVE to be followed up even if there’s no action task back from L3. Clearly there is a process for “following up” out of hours which does exactly that and only that based on the original call. I now reluctantly set those emails to ignore , or respond asking them to read the PMR history, but I worry what customers do .

Do they run around in circles doing this repeat “make work” until someone who has read the actual updates comes in ?

Oh and two out of the three PMRs are now closed. I will blog both which are interesting and apparently a googlewhack of problems (we were the first to report) later today. :-). So thank you to everyone who worked with me on them this week.