Domino Server Health – Monitoring and Managing @ Engage

This is my session on Domino Server Health given at Engage in Brussels last week.

If you’re a Domino administrator how do you decide what to monitor on your servers and how to manage them ? What are the key things to monitor? How do good practice management tools such as statistics reporting, DDM, cluster symmetry, database repair and policy settings make your work lighter and faster. Finally we’ll talk about some of the “must dos” in the day, week and month of a Domino admin.

Face/Off Domino vs Exchange On Premises @ Engage

Here is my presentation discussing how Exchange and Outlook on premises differs from Domino and Notes given at Engage in Brussels last week.

I hope you find it useful, this was my first presentation pulling together my ideas from the past few years of working with Exchange on premises integration projects.

How do Exchange on premises and the various Outlook clients line up against Domino on premises and its clients? In this session we’ll look at the configuration options and management interfaces for each server as well as the client options and client behaviours. We’ll also discuss the general ecosystems, considerations for migrating or co-existing and lessons learned. A great session for Domino admins who want to know more about the other side.

Exchange 2019 On Prem Install

In a couple of weeks time I’ll be in Brussels presenting at Engage and one of my sessions is Face/Off Domino vs Exchange On Premises (Weds at 8am).  I have an Exchange 2016 install but since Exchange 2019 shipped last October I wanted to update my install with that so I could use the latest version to demo.  In truth very little has changed in Exchange on premises since 2008 but I don’t like using an old version in my presentations.  So this is the story of the 4 days it took me to complete the install.

Four. Days.

Day 1: My big mistake.  I decided to uninstall Exchange 2016 instead of upgrading it. I wanted an entirely clean server to demonstrate.  The uninstall failed half way through.  It wouldn’t uninstall and it was still listed under installed programs.  Several hours of trial and error and internet research confirmed this is a common problem with Exchange uninstalls and the “fix” is to flatten the machine and start over.  The problem was the Exchange install was on the same box as the Active Directory 2016 Domain Controller which I really really didn’t want to flatten.

Day 2: Being Stubborn.  I’d do just about anything to avoid flattening the entire box and rebuilding so some more internet research took me to several blogs that talked about manually removing registry entries in order to clean up the install.  Hundreds of registry entries.  After doing that I still couldn’t delete or rename the folder despite no services being present so then it was into safe mode to do the rename.  That worked and I started the upgrade to Windows 2019 (the only supported platform for Exchange 2019). You can now do an inplace Windows upgrade from 2016 to 2019 and that worked maintaining all my Active Directory settings.

Day 3: Accepting the inevitable. Off I go with an Exchange 2019 install once more which started to install then prompted me for the Exchange installer disk.  It wouldn’t take the mounted disk I had started the installer from.  After a few hours’ research I realised this is a common red herring error that basically means the server can detect some old installation files and won’t complete.  At this point there were no services, no directory, nothing listed under installed programs.  Sometimes you have to accept you’ve strayed too many hacks from your starting point it’s best to startover and do it properly.  Windows 2019 install #2 this time letting it blat the server and rebuilding Active Directory from scratch (luckily it’s just my demo machine and I could do that but good luck if it’s your production environment).

At the end of day 3 I had a new Windows 2019 Domain Controller fully patched and I was ready to start my Exchange 2019 install.

Day 4: The Long Road.  Before Exchange will install the installer program will verify you have all the pre-requisites required on the operating system.  There are many from IIS management tools to .Net 4.7.1 to the basic authentication system.  A scrolling page of missing features is shown with URL links explaining them.  Since 90% of those features were actually Windows features you go to add/remove features to install I don’t know why the Exchange installer doesn’t just offer to install them for me because it took some time to work out where in the multi level hierarchy of features each one was.  In addition serveral of the URLs brought up 404 pages on the Microsoft site refering to Exchange 2003 and that link not being available(!).  Anyway finally after a few hours of digging around, downloading libraries, installing features and restarting it agreed to install Exchange 2019 and I was done.

If you take one lesson from this it should be that the Microsoft solution to many problems seems to be “flatten and start over”.  For that reason I wouldn’t put Exchange on any machine you wouldn’t be happy to flatten and start over or replace.

 

Notes pre 10.0.1 crashing on Mojave

A customer reported this week that their Macs were suddenly all crashing when trying to send mail , reply or forward.  Which is odd.. until I found this technote with the following statement updated on April  1st

As of April 1, 2019, IBM does not recommend upgrading to 10.14.4 for customers running pre-Notes 10.0.1 releases (i.e. Notes 9.0.1.x).  We are currently working on resolving a crash issue that occurs after upgrading to this latest OS via SPR #ZNDNBAPSZB.  This is documented under TN #0879431.

TN #0879431 is dated April 2nd and says a fix will be provided soon.

If you’re having issues with Notes on Mojave 10.14.4 and you’re running a version prior to 10.0.1 you might want to consider upgrading or let’s hope HCL provides a fix soon.

 

 

A No-Brainer For Domino Admins

I made this as short as I could – it should take you 3 minutes at most to read.

Do you want to be able to get a free audit report of your Notes clients, including what hardware they are using, what versions of Notes, what memory and disk each machine has, and what databases are on their workspaces?

Would you like to easily set notes.ini, any other ini, and even windows registry settings without the user noticing or being involved?

Would you like to deploy files to client workstations silently?

All of those things are now part of Notes and Domino 10.0.1 free of charge and with virtually no effort on your part.

Many of you will have heard of MarvelClient from Panagenda. Some of you may have heard that a licensed version of MarvelClient is free of charge with Domino 9.0.1 and later.  Why am I only talking about it now? Well MarvelClient now ships with Notes and Domino 10.0.1, which means that if you install either Domino or Notes 10.0.1 then the library files and databases needed to run MarvelClient are already installed for you.

Let me explain, prior to 10.0.1 you needed to deploy a library file to the clients that you wanted to use MarvelClient on, and although that could be done in a variety of ways – including postopen scripts, buttons and mail triggers – most of them involved some degree of user interaction and deployment configuration, and that hurdle was often too high for many customers to take on.  Now that hurdle is gone.

I want to talk about what you get with MarvelClient Essentials and here I’m going to be brief because I want you to deploy it, you want to deploy it, and so I want to be as clear as I can to get you there.  To install MarvelClient Essentials, you do this:

  1. Install Domino 10.0.1 * – that gives you the MC databases in a folder on the server called panagenda.  There is a configuration database for configuring what you want it to do and an Analyze database to show you the results.
  2. Open the desktop policy for your users and add a single notes.ini setting for EXTMGR_ADDINS
  3. Run your Notes 10.0.1 clients and watch the good stuff roll in.

Here’s the IBM whitepaper that goes into a little more detail, but not much because there isn’t much more you need to know to get started, although there’s a lot more you can learn when you’re ready to do more.

So that’s what I did.  I spent less than 10 minutes setting up the ACLs of the two server databases, signing them and updating the desktop policy and immediately the information started to come in.  These are some of the results that showed for Tim and I accessing our development environment using Macs.  What is great about MarvelClient is that it gives me a view and management over the client environment which I can’t see any other way – for instance:-

What directories Notes is installed in, where the program files are and where the notes.ini file is.

Screenshot 2019-02-07 at 21.22.23

Notes.ini settings for each user (note some are “2” where we both have then set and some are “1” where only one of us has it set).  These can be set, changed and deleted by MarvelClient as well.

Screenshot 2019-02-07 at 21.20.14

Notes client preferences, by preference and by person / machine.

Screenshot 2019-02-07 at 21.28.10

Resources, disk capacity / free, memory / free.

Screenshot 2019-02-07 at 21.19.30

Below is the full list of things you can do with Marvelclient Essentials.  On the right is the additional features you can get by upgrading to the Basic version which is chargeable.  I think it’s very clear just how much Panagenda and HCL are providing to you at no cost.  Very few of my customers are able to provide a good audit of their client environment and even fewer able to easily make changes to that environment.   It’s a testament to HCL’s commitment to lowering the TCO of Notes that they have provided all this functionality in 10.0.1.

Now what are you waiting for?

Essentials

Basic

Analyze
Desktop, Bookmarks

notes.ini, User Preferences

Mailfile Details

IBM Notes Version and Installation Information
OS and HW Overview
Local Databases / Replicas
Eclipse and Plugin Details
Eclipse Settings (incl Sametime)
ECL
Server<->Client Latency
ID File Details
Locations, Connections, Accounts, Certificates
HW / SW Inventory
Mail Archives
Windows Application Usage
Configuration
notes.ini and MC Config Variables
Any .ini File
User Preferences
Windows Registry
Up-/Download
Upload Data for Analyze
Upload Backups for Rollback
File Deployment
Smart File Downloader
Roll Back User Configuration
Run
Run Programs
Run Notes Processes
Run Agents
Run Notes Formulas
Copy, Move, Delete Files
Compact Desktop
Manage
Workspace Pages
Desktop Icons
Local Replicas
Replicator Page
Bookmarks & Bookmark Folders
Locations
Connections
ECL
ID File Management
Profile Documents
Switch Location
Migrate
Mass Change to Update Database Links
Mass Delete to Remove Database Links

 

Domino – Exchange On Premises Migration Pt2: Wrestling the Outlook Client

In part 1 of my blog about Exchange on premises migration from Domino I talked about the challenges of working with Exchange for someone who is used to working with Domino.  If only that were all of it but now I want to talk about the issues around Outlook and other Exchange client options that require those of us used to working with Domino to change our thinking.

In Domino we are used to a mail file being on the server and regardless of whether we used Notes or a browser to see that client, the data is the same.  Unless we are using a local replica, but the use of that is very clear when we are in the database as it visibly shows “on Local” vs the server name.

We can also easily swap between the local and server replicas and even have both open at the same time.

In Outlook you only have the option to open a mailbox in either online or cached mode.

So let’s talk about cached mode because that’s the root of our migration pains. You must have a mail profile defined in Windows in order to run Outlook. The default setting for an Outlook profile is “cached mode” and that’s not very visible to the users. The screenshot below is what the status bar shows when you are accessing Outlook in cached mode.

connectedtoexchange

In cached mode there is a local OST file that syncs with your online Exchange mailbox.  It’s not something you can access or open outside of Outlook.

datafiles

Outlook will always use cached mode unless you modify the settings of the data file or the account to disable it.

cachedsettings

As you can see from the configuration settings below, a cached OST file is not the same as a local replica and it’s not designed to be.   The purpose of the cached mail is to make Outlook more efficient by not having everything accessed on the server.

cachedoffline

Why does this matter during a migration?  Most migration tools can claim to be able to migrate directly to the server mailboxes but in practice the speed of that migration is often unworkably slow.  If that can be achieved it’s by far the most efficient but Exchange has its own default configuration settings that work against you doing that including throttling of activity and filtering / scanning of messages.   Many / most migration tools do not expect to migrate “all data and attachments” which is what we are often asked to do.  If what we are aiming for is 100% data parity between a Domino mail file and an Exchange mailbox then migrating that 5GB, 10GB, 30GB volume directly to the server isn’t an option.  In addition if a migration partially runs to the server and then fails it’s almost impossible to backfill the missing data with incremental updates.  I have worked with several migration tools testing this and just didn’t have confidence in the data population directly on the server.

In sites where I have done migrations to on premises servers I’ve often found the speed of migration to the server mailbox on the same network makes migration impossible so instead I’ve migrated to a local OST file.  The difference between migrating a 10GB file to a local OST (about an hour) vs directly to Exchange (about 2.5 days) is painfully obvious. Putting more resources onto the migration machine didn’t significantly reduce the time and in fact each tool either crashed (running as a Domino task) or crashed (running as a Windows desktop task) when trying to write directly to Exchange.

An hour or two to migrate a Domino mail file to a local workstation OST isn’t bad though right?  That’s not bad at all, and if you open Outlook you will see all the messages, folders, calendar entries, etc, all displaying.  However that’s because you’re looking at cached mode. You’re literally looking at the data you just migrated.  Create a profile for the same user on another machine and the mail file will be empty because at this point there is no data in Exchange, only in the local OST.  Another thing to be aware of is that there is no equivalent of an All Documents view in Outlook so make sure your migration tool knows how to migrate unfoldered messages and your users know where to find them in their new mailbox.

Now to my next struggle.  Outlook will sync that data to Exchange.  It will take between 1 and 3 days to do so.  I have tried several tools to speed up the syncing and I would advise you not to bother.  The methods they use to populate the Exchange mailbox from a local OST file sidestep much of the standard Outlook sync behaviours meaning information is often missing or, in one case, it sent out calendar invites for every calendar entry it pushed to Exchange.  I tried five of those tools and none worked 100%. The risk of missing data or sending out duplicate calendar entries/emails was too high.  I opted in the end to stick with Outlook syncing.  Unlike Notes replication I can only sync one OST / Outlook mailbox at a time so it’s slow going unless I have multiple client machines. What is nice is that I can do incremental updates quickly once the initial multi-GB mailbox has synced to Exchange.

So my wrestling with the Outlook client boils down to

  • Create mail profiles that use cached mode
  • Migrate to a local OST
  • Use Outlook to sync that to Exchange
  • Pay attention to Outlook limits, like a maximum of 500 folders*
  • Be Patient

*On Domino mailboxes we migrated that pushed up against the folder or item limits we found Outlook would run out of system memory repeatedly when trying to sync.

One good way to test whether the Exchange data matches the Domino data is to use Outlook Web Access as that is accessing data directly on the Exchange server.  Except that’s not as identical to the server data as we are used to seeing with Verse or iNotes.  In fact OWA too decides to show you through a browser what it thinks you most need to see versus everything that’s there.  Often folders will claim to be empty and that there is no data when in fact that data is there but hasn’t been refreshed by Exchange (think Updall).  There are few things more scary in OWA than an empty folder and a link suggesting you refresh from the server.  It just doesn’t instill confidence in the user experience.

Finally we have Outlook mobile or even using the native iOS mail application.  That wasn’t a separate configuration and unless you configure Exchange otherwise the default is that mobile access will be granted to everyone.   In one instance a couple of weeks ago, mobile access suddenly stopped working for all users who hadn’t already set up their devices.  When they tried to log in they got invalid name or password.  I eventually tracked that down to a Windows update that had changed permissions in Active Directory that Exchange needed set.  You can see reference to the issue here, and slightly differently here, although note it seems to have been an issue since Exchange 2010 and still with Exchange 2016.  I was surprised it was broken by a Windows update but it was.

I know (and have used) many workarounds for the issues I run into but that’s not for here.  Coming from a Domino and Notes background I believe we’ve been conditioned to think in a certain way about mailfile structure, server performance, local data, and the user experience, and expecting to duplicate that exactly is always going to be troublesome.

#DominoForever