Sametime Proxy “Loading” Problem

Having recently installed a new Sametime 11 environment on Linux with Sametime Proxy I ran into a problem with the proxy client where it would continually show “loading” when starting a conversation. Awareness of users on other servers worked, type ahead worked, it would open a chat window but just show a looping circle that never ended. Even if I tried talking to myself (usually a good test).

I have been working with HCL for a few days on this and it seemed to point to an issue with Mongodb as they found a failure to connect in the logs. That was odd because Mongo is installed on the same server as the ST Community server and was binding to localhost on 127.0.0.1, it didn’t appear to have any problems but the chatlogging service couldn’t connect so the Sametime Proxy clients wouldn’t work. It’s worth knowing what I subsequently found out which is that if the chat logging service doesn’t start up correctly, the Sametime clients will continue to work but the Sametime Proxy clients won’t.

Thanks to HCL identifying that the issue was with Mongo, I got it fixed and here are the changes I made.

Wanting Mongo to bind to the fqhn of the machine not localhost - originally I edited /etc/mongo.conf to use bindIpAll to bind all ips and that should have worked but instead I ended up using bindIP to explicitly bind the mongo service to the ip that the fqhn resolved to. That worked but to access the mongo console I had to then type “mongo -host fqhn” .. if I just typed “mongo” I’d get connection refused as it would default to using 127.0.0.1.

Now Mongodb was listening on the right port and hostname I had two other places to change in Sametime so they matched. The sametime.ini MONGO_URI setting and the chatlogging.ini CL_MONGO_HOST setting.

With all of that set I restarted Sametime and the proxy worked fine.

Take a look at Template Experience

Next week the TE team will be presenting at Engage with the version we hope to soon put into beta. There has been a lot of work completed in the past few months and the developers have really come up with clever solutions for seemingly impossible problems, all using standard Domino Designer techniques and code.

I hope to see you at our session at 1.30pm on Tuesday, we’ve got a lot to show and explain. If you can’t make it then I will be updating the site over the coming weeks with more information including hopefully where to get the beta version and design whitepaper.

For a preview of what you can see next week go here

Installing Sametime 10 - The Proxy Server

The Sametime Proxy Server is needed if you have a Community server you want people to login to using either a web or mobile client, this includes having Sametime awareness for other products such as Verse or Connections. Since I had already installed a Community Server without using a System Console I now wanted to add a Sametime Proxy Server with as minimal additional software as possible.

The Sametime Proxy server uses a DB2 database for iOS push notifications. If you don’t create that database the server will work in every other way but you won’t have working push notifications to iOS devices. You can create the database on any DB2 10.x or 11.x server but if you don’t have DB2 your Sametime license includes entitlement to a limited for Sametime only use license of DB2 10.5 (at least that’s what is in the download).

To start we need to install Installation Manager. Installation Manager is a package management software that will be used to install WebSphere and the Sametime Proxy Server on top of it. The Sametime download comes with version 1.6.2 but you are going to want to upgrade that to at least 1.8.6 before installing anything. Once Installation Manager is installed we need to install WebSphere 8.5.5 fixpack 14. Don’t worry about the fixpack download for now, if you have internet access during the install that can be done automatically so you will want to start by extracting each of the three WebSphere installers to a directory on your file system. It doesn’t matter which one. I extracted mine to C:\Software\WAS855. This is just the install directory so you don’t need to keep it once the install is complete.

Now we need to add a repository under Installation Manager so it can find the installer for WebSphere. The installer file is always repository.config so I added a respository (File - Preferences) and pointed to c:\software\was855\repository.config

Once the repository is added I can choose “Install” from the main Installation Manager menu and I will be presented with the option to install WebSphere 8.5.5. As you can see it has automatically checked Version 8.5.5.0 and I have checked “Show all versions” so I can install Fixpack 14 directly during this one install step.

The installer finds 15 fixpacks and automatically selects the newest. Instead I uncheck that and choose 8.5.5.14 which is what we want for this install
We accept all the recommended fixes that are part of 8.5.5.14
The installer warns us that there is a missing required fix so we uncheck “show recommended only” and search for the missing fix “ph01621” and select that to install along with WebSphere 8.5.5.14
Choose the shared resource directory that Installation Manager will use. This can’t be changed later without uninstalling everything so make sure4 you have room
The WebSphere install directory wil lbe where the profiles and program files will be created. All WebSphere servers will be installed under this path
This is the confirmation of what we want to install
Once the WebSphere install is complete go ahead and close Installation Manager without creating a profile (choose “None”). The Sametime Proxy Server will create its own profile as a standalone node so you don’t need to manually create one here.

Now the WebSphere install is complete we can proceed to installing the Sametime Proxy Server. In my design it’s a standalone server so has no deployment manager or System Console. The instructions to do this require us to perform a silent install which means using a response file specifying the configuration that is accessed from outside the Installation Manager GUI.

The extracted installer for the Sametime Proxy Server contains a response file we can edit for our own install. My response file was in

C:\Software\SametimeProxyServer\responseFiles and was called ip.rsp.

I took a copy of that file and made my own changes - see the bottom of this blog for my ip.rsp file but one very important consideration is that in the sample ip.rsp the default password of “wasadmin” is set. If you run the install without changing that you will end up with a login of wasadmin and a password of wasadmin.

The lines in the rsp file that set this are

<data key=’user.com.ibm.lotus.sametime.proxyserver.was.adminid’ value=’wasadmin‘/>

<data key=’user.com.ibm.lotus.sametime.proxyserver.was.password’ value=’zN39fpCc9SqIryGJM7+02A==‘/>

zN39fpCc9SqIryGJM7+02A== is the encoded value of “wasadmin”, you must enter the encoded value of the password you want to set in the response file. To generate the encoded value before adding it to the response file you use the “generateEncodedPassword” utility which can be found in your Sametime Proxy Server extracted install e.g.

c:\Software\SametimeProxyServer\GenerateEncodedPassword\

From the IBM Installaton Manager install directory (C:\IBM\Installation Manager for me) I ran

C:\Software\SametimeProxyServer\GeneratedEncodedPassword\GenerateEncodedPassword.bat gabpass >pw.txt - which generates the encoded value for “gabpass” and writes it to the pw.txt text file as “rNFGfQ/sPi082+AaXFMNxA==. I write the value to a text file so I can have it recorded whilst I complete the next step.

I can then modify the value of <data key=’user.com.ibm.lotus.sametime.proxyserver.was.password’ value=’rNFGfQ/sPi082+AaXFMNxA==’/>

Important note; do not try and use special characters in your WebSphere passwords especially @ and !.

Now we are ready to run the silent install from the eclipse directory where Installation Manager is installed e.g c:\IBM\Installation Manager\eclipse using the command

IBMIMc -launcher.ini silent-install.ini -input response_file -log log_path_and_filename -acceptLicense

See output of my install running below

With the install completed successfully we can sign onto the node via https://hostname:9043/ibm/console using the credentials we set in our rsp install file and see the Sametime Proxy Server in place. The community server that the Sametime Proxy connects to was specified in my response file when I did the install.

In my environment I wanted to have the Sametime Proxy Server on port 443 securely and to install my own SSL certificate. To modify the ports for the Samtime Proxy Server click on the server name and choose “ports” under “Communication” then edit the value for “wc_defaulthost_secure”.

Once this step is complete you should be able to connect to Sametime through a mobile client or through the web using the new web client on https://hostname:port/chat

Once the server is installed any changes to the configuration such as which Community server or cluster it should connect to should be changed by directly editing the stproxyconfig.xml file.

I could do that but then I hit a problem in that anyone accessing Sametime via the Proxy Server could send but not receive messages or see any they sent. My next blog “Issues With MongoDB During Sametime Install” explains what happened and how HCL helped me identify and fix it.

For the official install instructions from HCL please see this PDF

My ip.rsp file is shown below as an example. You will need to modify it for your own install.



<?xml version="1.0" encoding="UTF-8"?>


<!--The "acceptLicense" attribute has been deprecated. Use "-acceptLicense" command line option to accept license agreements.-->

<agent-input temporary='true' clean='true'  acceptLicense='false'>


<server>

<!-- When the repository location is specified as a relative path, the path is interpreted relative to the location of this response file -->

<repository id='IBM® Sametime® Server Platform'>

<!-- 

The files which are installed in the directory specified by the value of the

user.installlocation relate to WAS installer code.

If the specified directory exists prior to running the installer it must be empty.

Therefore, if you are installing upon an existing copy of WebSphere,

you will probably need to change it from the default value.

-->

<data key='user.installlocation' value='C:/IBM/SametimeServer'/>
<!-- This attribute specifies where WebSphere Application Server is already installed. -->

<data key='user.was.installlocation'value='C:/IBM/WebSphere/AppServer'/>

<!--

This location is the path we will install the offering to i.e. the files used for creating

Sametime WAS server profiles.

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.offering.installlocation' value='C:/IBM/WebSphere/STPServerCell'/>

<!--

*** CHANGEME ***

The following two XML elements specify the host and port of the Sametime community server that

you wish to connect to with this Proxy server. You will need to alter it. The host name needs to be the FQDN of the CS
The default port to use is 1516, because this is the default port for Virtual Places 

server to server communication (1533 is the default for client to server communication_

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.sthostname' value=‘stsite.turtlepartnership.com’/>

<data key='user.com.ibm.lotus.sametime.proxyserver.stport' value='1516'/>

<!--

This attribute defines the type of WAS profile(s) you wish to create.

This attribute is case-sensitive and Valid options are:  

'Cell' - (default) create a cell consisting of a Deployment manager and a managed node

'PN' - create a stand-alone primary node  federated to an existing SSC Deployment manager

'SN' - create a secondary node federated to an existing Deployment manager

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.installtype' value='PN'/>
<!-- 

If you specified that you wish to install a WAS secondary node profile,

you must edit these attributes to match the host name and soap port of the Deployment manager you wish to have your WAS profile managed by
If this secondary node is a standalone node, then edit these attributes to match the secondary node's hostname and soap port values.

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.dmhost' value='DMHost'/>

<data key="user.com.ibm.lotus.sametime.proxyserver.Dmgr.WASDMSoapPort" value="8603"/>

<data key="user.com.ibm.lotus.sametime.proxyserver.Dmgr.WASHost" value="DMHost"/>
<!--

Uncomment this attribute if you wish the WAS profile to use a hostname other than the default for this machine

if you do specify a hostname it should be the fully qualified domain name (e.g. "servername.domain.com" not "servername")
<data key='user.com.ibm.lotus.sametime.proxyserver.hostname' value='HOSTNAME'/>

-->

<!--

The WAS profiles created by the installer will be protected by File based security 

for the System Console. The following two XML elements specify the username & password

that can be used to login to the WAS System Console.It is not allowed to omit these elements.

For security reasons you do not store the password in plain text. Instead you can use

the generateEncodedPassword utility contained in the kit to generated an encoded form of the password

For example 'zN39fpCc9SqIryGJM7+02A==' is equivalent to a password of 'wasadmin'
It is not possible to choose LDAP security during install, but this can be

configured after install 

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.was.adminid' value='wasadmin'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.was.password' value='zN39fpCc9SqIryGJM7+02A=='/>

<!--

The default is to name the WAS cell <HOSTSHORTNAME>ProxyCell 

and to name the WAS node <HOSTSHORTNAME>ProxyNode

but you can over-ride the defaults by un-commenting the following parameters
<data key='user.com.ibm.lotus.sametime.proxyserver.cellname' value='HOSTNAMEProxyCell'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.nodename' value='HOSTNAMEProxyNode'/>

-->
<!--

These are the names that will be assigned to the WAS profiles created by this installer

It is safe to simply use the default values or to change them to anything you want so long as the profile names are unique

 -->

<data key='user.com.ibm.lotus.sametime.proxyserver.was.serverprofile' value='STPAppProfile'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.was.dmprofile' value='STPDMgrProfile'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.was.snserverprofile' value='STPSNAppProfile'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.was.appserver' value='STProxyServer'/>

<!--Port the Deployment Manager will run on. This should not be changed from the default of 8600 -->

<data key='user.com.ibm.lotus.sametime.proxyserver.was.dmport' value='8600'/>

<!--Port the Deployment Manager will use for SOAP connections (the default is 8603) -->

<data key='user.com.ibm.lotus.sametime.proxyserver.was.dm.soapport' value='8603'/>

<!--

This attribute specifies whether or not you wish to install from a SSC deployment plan

It is case-sensitive and the only valid values are TRUE and FALSE
If you change this to TRUE, you will also need to specify values for

user.com.ibm.lotus.sametime.proxyserver.SSChostname, user.com.ibm.lotus.sametime.proxyserver.SSCportnumber, 

user.com.ibm.lotus.sametime.proxyserver.SSCusername, user.com.ibm.lotus.sametime.proxyserver.SSCpassword and 

user.com.ibm.lotus.sametime.proxyserver.SSC_SSLEnabled

see the other sample response file for sample values for these parameters

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.SSCEnabled' value='FALSE'/>
<!-- 

This attribute specified is we are doing an upgrade or a fresh install

It is case-sensitive and valid values are 'INSTALL' or 'UPDATE'.

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.job' value='INSTALL'/>
<!-- version of Sametime to install, 10.0.0.0 -->  

<data key='user.com.ibm.lotus.sametime.proxyserver.offVersion' value='10.0.0.0'/>
<!--

This attribute specifies if the Proxy server will co-exist with a SCC server

Set this value to TRUE if SSC is installed on same machine as Proxy Server 

  It is case-sensitive and the only valid values are TRUE and FALSE

-->  

<data key='user.com.ibm.lotus.sametime.proxyserver.ssc.coexist' value='FALSE'/>
<!-- 

On non-windows platforms this attribute should point to a temporary directory that

can be used for storage of files used by the installer. I needs to be on a disk with

greater than 1GB of free space

-->

<data key='user.nonWin.temp' value='C:/IBM/WebSphere/STtemp'/>
<!--

This is the Language code for the language version you wish to install

e.g. de for German, zh for Simplified Chinese

-->

<data key='cic.selector.nl' value='en'/>
<!--

    These parameters specifies database credentials and hostname. Get the encoded DB password using the generateEncodedPassword utility

available in the download package. The default port for DB2 is 50000.

-->

<data key='user.com.ibm.lotus.sametime.proxyserver.dbport' value='50000'/>    

<data key='user.com.ibm.lotus.sametime.proxyserver.dbhostname' value='@@DBHOST@@'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.dbname' value='@@DBNAME@@'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.dbuser' value='@@DBUSER@@'/>

<data key='user.com.ibm.lotus.sametime.proxyserver.dbpass' value='@@ENCODEDDBPASSWORD@@'/>

</profile>
<install modify='false'>

<!--

The features parameter of the offering element identifies what features are to be installed.

If installing with SSC, it should be set to 'CEMain,nossc,ssc'

If installing without SSC, it should be set to 'CEMain,nossc'

The id parameter should not be altered of the installer won't work

In the profile name the ® character is represented by ® 

This representation is safer if the file is edited with a tool which does not support Unicode properly

-->

<offering profile='IBM® Sametime® Server Platform' features='CEMain,nossc' id='com.ibm.lotus.sametime.proxyserver'/>

</install>
<!--

This next parameter specifies the location of the temporary cache where

Installation Manager stores files during install e.g. C:/Program Files/IBM/IMShared

-->

<preference value='C:/IBM/IMShared' name='com.ibm.cic.common.core.preferences.eclipseCache'/>

<!--

This parameter specifies whether or not the installer should search for possible updates on the web at ibm.com

Since Sametime does not maintain a fix repository on the web this should always be set to false because if there is no

connection to the Internet and it is set to true users may get error messages when it looks for the repository

(and the repository will not contain fixes anyway)

-->

<preference value='false' name='offering.service.repositories.areUsed'/>

</agent-input>

Installing Sametime 10 Limited Use

A slight delay on this blog, which I promised to write last week, but I have a good excuse. My original plan was to do an in-place upgrade as Sametime 10 is still 32bit-only enabling it to be upgraded on top of your existing Community Server so long as the underlying Domino install is at least 9.0.1 FP10 32bit. However the system requirements support Windows 2012 and Windows 2016 servers, but not Windows 2008 which was what the server I planned to upgrade was installed on. Yeah, it’s been a while since we had a major Sametime upgrade 🙂

Change of plan. I decided to start over first by building a standalone Community server and then adding the Proxy Server all without a System Console.

The documentation for this install is a PDF file that is linked to on this page https://www-01.ibm.com/support/docview.wss?uid=ibm10887399. My write-up below is based on my own install using that documentation.

Installing Domino

I built a new Windows 2016 Standard server and installed a new Domino 32 bit 9.0.1 FP10 server on it. IBM Fix Central still has FP10 accessible on it so you can still get it from there.

  • Windows 2016 is a supported server operating system for Domino 9.0.1, so I went with that as it will also support future upgrades.
  • We know that the next major Sametime version is meant to be 64bit-only, so we’re not going to be able to upgrade over 32 bit Domino without uninstalling and re-installing but at least I won’t have to rebuild the entire machine. However, for a customer I’d probably do a side-by-side upgrade so I could have a fallback to the existing Community Server.
  • I like to install Sametime in its own domain as Sametime uses LDAP for authentication and even if that LDAP is a Domino Directory you can’t point to the Directory running the Sametime server.
  • Having the Sametime Community server in its own domain means I can upgrade Domino and upgrade Sametime without touching anything else such as my mail or application domain.

My Domino server then is built as an Enterprise server (which doesn’t need to be licensed separately if you’re only using it for Sametime) in its own domain and with minimal services (remove things like Calendar, Scheduler, RNRMgr from starting up) but leave HTTP.

Installing MongoDB

MongoDB is new to the Sametime requirements and also temporary. It is needed for storing of chat history, but HCL have said they will be changing the DB platform from Mongo in v11 to something else and providing tools to move the data to the new database store at that point. You don’t need to be a MongoDB expert for this step.

First we need to download the right version of MongoDB and separately the Compass add-on.

MongoDB Installer

Compass Addon

The supported MongoDB version is 3.6.5 and fixpacks specifically. Once downloaded we run the installer, make sure you uncheck the box to install “Compass” during the MongoDB install or the install will fail.

Choose “Complete” on this screen

Un-Check this box which is checked by default or the install will fail.

Once completed we then run the Compass installer separately

Now we’re almost ready to start and configure MongoDB. When the server starts it will expect to find the data and log directories so these need to be created in advance or the start will fail. The default locations for those are c:\data\db and c:\program files\mongodb\logs\log.txt (I don’t use these in general but created them purely for the first server start).

Run the mongod.exe first from the install directory which by default is

C:\Program Files\MongoDB\Server\3.6\bin

from a command window run “mongod.exe” which will start the server instance and end up listening on port 27017. Do not shut the server window if running in the foreground at this point!

Now we want to issue commands to the server via the Mongo command line tool which we run using “mongo.exe” from the same directory.

Out of interest at this point if you type the following in a command window you should see both the server listening and command window established connections:

netstat -an |find /i “27017”

Checking The Schema

We now need to make sure the schema version for MongoDB is set correctly to support Sametime’s requirements. From the mongo command window type

use admin
db.system.users.remove({})
db.system.version.find()

my request returned “{“_id”: “featureCompatibilityVersion”, “version”: “3.6”}” so I used the command below to set it correctly then restarted the Mongodb server.

db.system.version.insert({ “_id” : “authSchema”, “currentVersion” : 3 })

The documentation now wants us to add MongoDB as a service using the following instructions however I had problems with it as although it installed as a service it then refused to start returning a 1053 error. After a couple of hours trying to fix that which seems to be Windows related I went another way to get a working service.

This is what the documentation said to use (sc is found in c:\windows\system32 if your path can’t find it)
sc.exe create MongoDB binPath=”\”C:\Program Files\MongoDB\Server\3.6\bin\mongod.exe\” -service — config=\”C:\Program Files\MongoDb\Server\3.6\mongod.cfg”” DisplayName= “MongoDB” start= “auto”

I ended up removing that service since it wouldn’t start (sc delete MongoDB) and adding it using a different syntax from the Mongo bin directory itself

mongod -directoryperdb -dbpath C:\data\mongodb\ -logpath C:\data\mongodb\log\mongo.log -logappend -service -install

That installed the service which started with no problems, including writing to the log file in the new directory I configured on the command line (again the directories have to exist first).

So now Mongodb is installed and running, let us move onto configuration.

Configuring MongoDB

Launch the Mongo Compass Community from your Windows start menu and connect to your locally-running server.

I don’t need to change any of these settings - just press “Connect”
Once connected I am shown a list of existing databases. These are all system databases so don’t delete any of them!

Now we need to create a new database for chatlogging and collections within that new database. Click the “Create Database” button and complete the dialog as shown below. Keep the naming and capitalisation identical to what is shown here, Database Name “chatlogging”, Collection Name “SESSIONS” . If you don’t you will need to modify the sametime.ini later.

Once the database is created we can click on it for further configuration

The documentation wants us to create an additional Collection called “EVENTS”, so we click on “Create Collection” and do that.

Having created the new chatlogging db we need to set up a Mongo user account and give that admin access. We do all this from the Mongodb console again which is accessed by running mongo.exe.

The documentation specifies using the user name “sametimeUser” and the password “sametime”. If you use a values you will need to edit the MongoDB settings in Sametime.ini once you complete the Sametime configuration steps.

The command below not only creates the user but also gives that new user admin rights to the chatlogging database.

use admin

db.createUser({user:”sametimeUser”,pwd: “sametime”,roles:[{role:”readWrite”, db:”chatlogging”},{role: “userAdminAnyDatabase”, db: “admin”}]})

Installing Sametime Standalone

With Domino shut down we can now install the Sametime Community Server. We don’t need a System Console in place before doing this.

Configuring Chat Logging

There are three components to configuring chat logging on the Sametime server:

  1. Enable Chat Logging through stconfig.nsf so the server knows to support it.
  2. Configure the policy files to enable chat logging.
  3. Point the Sametime chat logging to the MongoDB.

Open STConfig.nsf on the server and find the communityservices document then edit that and change the Chat Logging parameter to “relax”

Now we need to modify the XML files that control the policies. We can find those in the Domino program directory and can open them with a text editor.

  • In policies.server.xml set the “current-value” of offlinemsgs.community.enabled to “1” instead of “0”
  • In ‘OfflineMessages.xml’ set the ‘current-value’ of offlinemsgs.community.enabled to “1” instead of “0”
  • In ‘policies.users.xml’ file set the ‘current-value’ of im.enableOfflineMessages to “1”. There will be multiple instances of that value, one for the default and one for the anonymous policies that are available on a standard install.

I will be talking more about manually setting and creating policies in a later blog. The documentation provides a reference URL to detailed policy settings but that URL now returns a 404 so I’m checking with HCL for an updated link.

broken link: https://www.ibm.com/support/knowledgecenter/en/SSKTXQ_9.0.1/admin/admin/admin_policy _im_ids.html

In addition, whilst you have policies.users.xml open you can enable the persistent chat feature by setting im.persistentChatEnabled to “1”.

Our last step in this configuration is making sure Sametime knows how to reach MongoDB and your new chatlogging database. If you used the default settings in the documentation and as I’ve documented here and MongoDB is installed locally to the Sametime server then you should not need to change any values, but for reference, and in case you do, the default settings in Sametime.ini are as follows:

[stconvomap]
MONGO_URI=mongodb://127.0.0.1:27017
MONGO_DB=chatlogging
MONGO_COLLECTION=EVENTS 

Chatlogging.ini
CL_MONGO_HOST=127.0.0.1
CL_MONGO_PORT=27017
CL_MONGO_USER=sametimeUser
CL_MONGO_PASSWORD=sametime 
CL_MONGO_AUTH_DBNAME=admin CL_MONGO_DBNAME=chatlogging 

Now give everything a restart and confirm that it all works!

In my next blog I’ll be looking at installing the Sametime Proxy Server again without using the System Console so that I can get access to the new web client and the mobile cilents which are part of the Limited Use entitlement.


Domino Server Health - Monitoring and Managing @ Engage

This is my session on Domino Server Health given at Engage in Brussels last week.

If you’re a Domino administrator how do you decide what to monitor on your servers and how to manage them ? What are the key things to monitor? How do good practice management tools such as statistics reporting, DDM, cluster symmetry, database repair and policy settings make your work lighter and faster. Finally we’ll talk about some of the “must dos” in the day, week and month of a Domino admin.

Face/Off Domino vs Exchange On Premises @ Engage

Here is my presentation discussing how Exchange and Outlook on premises differs from Domino and Notes given at Engage in Brussels last week.

I hope you find it useful, this was my first presentation pulling together my ideas from the past few years of working with Exchange on premises integration projects.

How do Exchange on premises and the various Outlook clients line up against Domino on premises and its clients? In this session we’ll look at the configuration options and management interfaces for each server as well as the client options and client behaviours. We’ll also discuss the general ecosystems, considerations for migrating or co-existing and lessons learned. A great session for Domino admins who want to know more about the other side.

Exchange 2019 On Prem Install

In a couple of weeks time I’ll be in Brussels presenting at Engage and one of my sessions is Face/Off Domino vs Exchange On Premises (Weds at 8am).  I have an Exchange 2016 install but since Exchange 2019 shipped last October I wanted to update my install with that so I could use the latest version to demo.  In truth very little has changed in Exchange on premises since 2008 but I don’t like using an old version in my presentations.  So this is the story of the 4 days it took me to complete the install.

Four. Days.

Day 1: My big mistake.  I decided to uninstall Exchange 2016 instead of upgrading it. I wanted an entirely clean server to demonstrate.  The uninstall failed half way through.  It wouldn’t uninstall and it was still listed under installed programs.  Several hours of trial and error and internet research confirmed this is a common problem with Exchange uninstalls and the “fix” is to flatten the machine and start over.  The problem was the Exchange install was on the same box as the Active Directory 2016 Domain Controller which I really really didn’t want to flatten.

Day 2: Being Stubborn.  I’d do just about anything to avoid flattening the entire box and rebuilding so some more internet research took me to several blogs that talked about manually removing registry entries in order to clean up the install.  Hundreds of registry entries.  After doing that I still couldn’t delete or rename the folder despite no services being present so then it was into safe mode to do the rename.  That worked and I started the upgrade to Windows 2019 (the only supported platform for Exchange 2019). You can now do an inplace Windows upgrade from 2016 to 2019 and that worked maintaining all my Active Directory settings.

Day 3: Accepting the inevitable. Off I go with an Exchange 2019 install once more which started to install then prompted me for the Exchange installer disk.  It wouldn’t take the mounted disk I had started the installer from.  After a few hours’ research I realised this is a common red herring error that basically means the server can detect some old installation files and won’t complete.  At this point there were no services, no directory, nothing listed under installed programs.  Sometimes you have to accept you’ve strayed too many hacks from your starting point it’s best to startover and do it properly.  Windows 2019 install #2 this time letting it blat the server and rebuilding Active Directory from scratch (luckily it’s just my demo machine and I could do that but good luck if it’s your production environment).

At the end of day 3 I had a new Windows 2019 Domain Controller fully patched and I was ready to start my Exchange 2019 install.

Day 4: The Long Road.  Before Exchange will install the installer program will verify you have all the pre-requisites required on the operating system.  There are many from IIS management tools to .Net 4.7.1 to the basic authentication system.  A scrolling page of missing features is shown with URL links explaining them.  Since 90% of those features were actually Windows features you go to add/remove features to install I don’t know why the Exchange installer doesn’t just offer to install them for me because it took some time to work out where in the multi level hierarchy of features each one was.  In addition serveral of the URLs brought up 404 pages on the Microsoft site refering to Exchange 2003 and that link not being available(!).  Anyway finally after a few hours of digging around, downloading libraries, installing features and restarting it agreed to install Exchange 2019 and I was done.

If you take one lesson from this it should be that the Microsoft solution to many problems seems to be “flatten and start over”.  For that reason I wouldn’t put Exchange on any machine you wouldn’t be happy to flatten and start over or replace.

 

More Apple Cloud Issues & The Solution

If you saw my earlier post on my trouble with my  Apple iCloud account it looked by the end of it I had found a resolution (original post here https://turtleblog.info/2019/03/08/the-painful-journey-to-abandoning-icloud/).

In summary it appeared (from my investigation) that my iCloud account somehow got corrupted or modified in such a way that it became unusable and I ended up disabling iCloud storage for documents and desktop amongst other things to fix it.

Except it didn’t entirely fix it.

For the past few weeks I’ve had endless problems with Finder being slow. Spinning ball and up to 10 seconds to change folders, open a fie dialog in any application - longer for dragging and dropping files.  In addition Spotlight was broken, it would let me type a few characters then take around 10 seconds to fill in any suggestions which I then had to wait another 5 seconds or so to click on.

I tried all the fixes I could find including rebuilding Spotlight, removing the CloudMetadata.xml files and resetting preferences.  Things were definitely “usable” but not the way I found acceptable ie. how it used to be.  So last week I had a call with an Apple “genius”.  Two resets, a boot into safe mode (which took 45 minutes and caused hin to exclaim “oh my god”) and finally I caved and reinstalled the OS entirely (which leaves everything else in place).  The guy ended up ghosting me when I pointed out that he wasn’t actually identifying the problem, just hitting it with ever bigger hammers in the hopes it would go away and that if this final hammer (reinstalling the OS) didn’t work he better have a plan.

So this week I was back on my own with a Mac that was still slow and convinced even flattening and rebuiding wouldn’t necessarily help if the issue was related to my iCloud account.  Then I read a few community comments that put me on the right path - I removed all internet accounts from under system preferences.  In my case all I had was my iCloud account.  To remove it it had to save a copy of all my iCloud data locally and remove all my credit cards.  Then I removed the Cloud metadata (rm ~/Library/Caches/CloudKit/CloudKitMetadata*;killall cloudd) and restarted.

I reattached my iCloud account and configured what I wanted to sync, added my credit cards and went to bed.  This morning everything is working as it should including Finder and Spotlight so we’re back to the status before the iCloud corruption.  It seems the single only fix I needed was to remove the iCloud account from system preferences entirely and then re-add it.

 

The Painful Journey To Abandoning iCloud

As some of you know I’m very committed to the Mac ecosystem.  I have Mac laptops, an iPad with over 4000 books, iPhone (not the latest because who needs that), a watch Apple TVs x 4 etc etc.  I’m also extremely risk averse and cloud wary.  I gave in and let Apple put all my books in the cloud just because iTunes sucks for syncing and cloud syncing worked across all my devices however I also had a lengthy open support call last year with Apple wanting to know where my books were now stored on my Mac so I could find them and back them up

“they are all in the cloud”

“yes I get that but they are also on my laptop so where are they”

“no they are only in the cloud”

“well that’s not true because here I go, switching off wifi and hey I can still read my books in ibooks so they are here somewhere”

..>> pause for several weeks whilst this is escalated>>>>

“they are on your Macbook but stored in a way you can’t find them or access them”

(please no advice on this one, I found my own workaround to find them and backup un-DRM copies)

So.. iCloud. I agreed about 18 months’ ago to let my Documents and Desktop folders sync to iCloud.  My only reason for that was so that I could get at files if I needed to on my iPad or by logging into any browser but tbh I rarely used it.  Still it worked and seemed a decent idea.

Then one Saturday about two weeks’ ago it all went horribly wrong..

I was sat working when I got an alert saying facetime had been added to my watch.  Which was odd.  My watch is 18 months old and was on my wrist and nothing had changed. The watch itself had no alert.  So off I go digging and I find under my account and devices a list of my current watch and an old watch I wiped and sold to a friend to give to his wife 18 months ago.  Well still odd but no big deal.  They hadn’t done anything so clearly just an odd gremlin.  Just in case I removed that old watch from my devices.

Then I got alerts saying my credit cards had been removed from my watch.  Except they hadn’t been removed from the watch on my wrist and the other watch was flattened before I handed it over 18 months ago.

I did some research, found nothing nefarious and let it go.   I did notice I had been logged out of all my Apple accounts on all my devices and things like Sonos had to be re-authorised again.  Weird and annoying but a side effect of whatever happened I assume**

Then a few days later I restarted my laptop.  I probably only restart it every two weeks so this was the first time since that alert.  The laptop restarts but finder and anything that uses finder like spotlight or even terminal were entirely non responsive.  They would briefly work long enough for me to type 2 characters or click on a folder then there would be a spinning ball for about 25 seconds before it would respond.  That gradually got slower and slower over a few minutes.  So off we go to research because I now have a broken laptop.

After several hours research we found this article which gave a bit of a clue as it pointed to a cloud corruption problem http://osxdaily.com/2015/04/17/fix-slow-folder-populating-cloudkit-macosx/

Unlike some of the other Finder troubles, the Finder process usually doesn’t eat much CPU or crash repeatedly, it’s just inordinately slow when loading folder views, populating files, and opening folders.

So I followed the instructions and deleted the files they specify and immediately my laptop was more responsive.  OK.. Well that was a scary afternoon and I’ll just go ahead and disable cloud syncing so that never happens again.

Did you know Apple doesn’t let you do that?  If you disable cloud syncing for Documents anbd Desktop it actually deletes the contents of those folders and keeps the files in the cloud for 30 days in case you want them back.  So that’s dumb.  I decided to move the contents of both folders to temporary folders, disable cloud syncing then move them back but my laptop was working and I was busy so I parked that for later.

Later…. about a week later again the Finder sluggishness came back but this time I knew how to fix it.  Once it was fixed I went ahead and moved the contents of both Documents and Desktop to temporary folders, disabled cloud syncing and moved them back.  My laptop immediately started working, finder was faster than it had been for a very long time and I’ve had no more problems.

Now I wonder if that first alert about this non existent “watch” was a precursor to some cloud corruption on my account.  That cloud corruption caused all the authentication for my account to be lost and also corrupted the authentication for my cloud data which only tried to reconnect when I signed back into the OS.

** for anyone who was wondering if I had asked an apple “genius” about this. Yes I did. No they had no clue what I was talking about since most of them are “iphone experts” in store now and the one who called me back seemed to think I made it up.

Lesson learned. Apple iCloud for all but my books is now disabled.