Domino – Exchange On Premises Migration Pt2: Wrestling the Outlook Client

In part 1 of my blog about Exchange on premises migration from Domino I talked about the challenges of working with Exchange for someone who is used to working with Domino.  If only that were all of it but now I want to talk about the issues around Outlook and other Exchange client options that require those of us used to working with Domino to change our thinking.

In Domino we are used to a mail file being on the server and regardless of whether we used Notes or a browser to see that client, the data is the same.  Unless we are using a local replica, but the use of that is very clear when we are in the database as it visibly shows “on Local” vs the server name.

We can also easily swap between the local and server replicas and even have both open at the same time.

In Outlook you only have the option to open a mailbox in either online or cached mode.

So let’s talk about cached mode because that’s the root of our migration pains. You must have a mail profile defined in Windows in order to run Outlook. The default setting for an Outlook profile is “cached mode” and that’s not very visible to the users. The screenshot below is what the status bar shows when you are accessing Outlook in cached mode.

connectedtoexchange

In cached mode there is a local OST file that syncs with your online Exchange mailbox.  It’s not something you can access or open outside of Outlook.

datafiles

Outlook will always use cached mode unless you modify the settings of the data file or the account to disable it.

cachedsettings

As you can see from the configuration settings below, a cached OST file is not the same as a local replica and it’s not designed to be.   The purpose of the cached mail is to make Outlook more efficient by not having everything accessed on the server.

cachedoffline

Why does this matter during a migration?  Most migration tools can claim to be able to migrate directly to the server mailboxes but in practice the speed of that migration is often unworkably slow.  If that can be achieved it’s by far the most efficient but Exchange has its own default configuration settings that work against you doing that including throttling of activity and filtering / scanning of messages.   Many / most migration tools do not expect to migrate “all data and attachments” which is what we are often asked to do.  If what we are aiming for is 100% data parity between a Domino mail file and an Exchange mailbox then migrating that 5GB, 10GB, 30GB volume directly to the server isn’t an option.  In addition if a migration partially runs to the server and then fails it’s almost impossible to backfill the missing data with incremental updates.  I have worked with several migration tools testing this and just didn’t have confidence in the data population directly on the server.

In sites where I have done migrations to on premises servers I’ve often found the speed of migration to the server mailbox on the same network makes migration impossible so instead I’ve migrated to a local OST file.  The difference between migrating a 10GB file to a local OST (about an hour) vs directly to Exchange (about 2.5 days) is painfully obvious. Putting more resources onto the migration machine didn’t significantly reduce the time and in fact each tool either crashed (running as a Domino task) or crashed (running as a Windows desktop task) when trying to write directly to Exchange.

An hour or two to migrate a Domino mail file to a local workstation OST isn’t bad though right?  That’s not bad at all, and if you open Outlook you will see all the messages, folders, calendar entries, etc, all displaying.  However that’s because you’re looking at cached mode. You’re literally looking at the data you just migrated.  Create a profile for the same user on another machine and the mail file will be empty because at this point there is no data in Exchange, only in the local OST.  Another thing to be aware of is that there is no equivalent of an All Documents view in Outlook so make sure your migration tool knows how to migrate unfoldered messages and your users know where to find them in their new mailbox.

Now to my next struggle.  Outlook will sync that data to Exchange.  It will take between 1 and 3 days to do so.  I have tried several tools to speed up the syncing and I would advise you not to bother.  The methods they use to populate the Exchange mailbox from a local OST file sidestep much of the standard Outlook sync behaviours meaning information is often missing or, in one case, it sent out calendar invites for every calendar entry it pushed to Exchange.  I tried five of those tools and none worked 100%. The risk of missing data or sending out duplicate calendar entries/emails was too high.  I opted in the end to stick with Outlook syncing.  Unlike Notes replication I can only sync one OST / Outlook mailbox at a time so it’s slow going unless I have multiple client machines. What is nice is that I can do incremental updates quickly once the initial multi-GB mailbox has synced to Exchange.

So my wrestling with the Outlook client boils down to

  • Create mail profiles that use cached mode
  • Migrate to a local OST
  • Use Outlook to sync that to Exchange
  • Pay attention to Outlook limits, like a maximum of 500 folders*
  • Be Patient

*On Domino mailboxes we migrated that pushed up against the folder or item limits we found Outlook would run out of system memory repeatedly when trying to sync.

One good way to test whether the Exchange data matches the Domino data is to use Outlook Web Access as that is accessing data directly on the Exchange server.  Except that’s not as identical to the server data as we are used to seeing with Verse or iNotes.  In fact OWA too decides to show you through a browser what it thinks you most need to see versus everything that’s there.  Often folders will claim to be empty and that there is no data when in fact that data is there but hasn’t been refreshed by Exchange (think Updall).  There are few things more scary in OWA than an empty folder and a link suggesting you refresh from the server.  It just doesn’t instill confidence in the user experience.

Finally we have Outlook mobile or even using the native iOS mail application.  That wasn’t a separate configuration and unless you configure Exchange otherwise the default is that mobile access will be granted to everyone.   In one instance a couple of weeks ago, mobile access suddenly stopped working for all users who hadn’t already set up their devices.  When they tried to log in they got invalid name or password.  I eventually tracked that down to a Windows update that had changed permissions in Active Directory that Exchange needed set.  You can see reference to the issue here, and slightly differently here, although note it seems to have been an issue since Exchange 2010 and still with Exchange 2016.  I was surprised it was broken by a Windows update but it was.

I know (and have used) many workarounds for the issues I run into but that’s not for here.  Coming from a Domino and Notes background I believe we’ve been conditioned to think in a certain way about mailfile structure, server performance, local data, and the user experience, and expecting to duplicate that exactly is always going to be troublesome.

#DominoForever

 

 

 

 

 

 

Whooomf – All Change. HCL Buys The Shop…

According to this Press Release as of mid June 2019, HCL take ownership of a bunch of IBM products including Notes, Domino and Connections on premises. Right now and since late 2017 there has been a partnership with IBM on some of the products such as Notes, Domino, Traveler and Sametime* so this will take IBM out of the picture entirely. Here are my first “oh hey it’s 4am” thoughts on why that’s not entirely surprising or unwelcome news ..

HCL are all about leading with on premises, not cloud. The purchase of Connections is for on premises and there are thousands of customers who want to stay on premises. Every other provider is either entirely Cloud already or pushing their on premises customers towards it by starving their products of development and support (waves at Microsoft). *cough*revenue stream*cough*

HCL have shown in 2018 that they can innovate (Domino’s TCO offerings, Notes on the iPad, Node integration etc) , develop quickly and deliver on their promises. That’s been a refreshing change.

They must be pleased with the current partnership products to buy them and more outright.

When HCL started the partnership with IBM they brought on some of the best of the original IBM Collaboration development team and have continued to recruit at high speed. It was a smart move and one I hope they repeat across not just development but support and marketing too.

HCL already showed with “Places” that they have ideas for how collaboration tools could work (see this concept video https://youtu.be/CJNLmBkyvMo) and that’s good news for Connections customers who gain a large team and become part of a bigger collaboration story in a company that “gets it”.

Throughout 2018 HCL have made efforts to reach out repeatedly to customers and Business Partners, asking for our feedback and finding out what we want. From sponsoring user group events (and turning up in droves) around the world to hosting the factory tour in June at their offices in Chelmsford where we had two days of time with the developers and their upcoming technologies. I believe they have proven they understand what this community is about and how much value comes from listening and – yes – collaborating.

Tonight I am more optimistic for the future of these products and especially Connections than I have been in a while. HCL, to my experience, behave more like a software start up than anything else, moving fast, changing direction if necessary and always trying to lead by innovating. I hope many of the incredibly smart people at IBM (yes YOU) who have stood alongside these products for years do land at HCL if that’s what they want, it would be a huge loss if they don’t.

*HCL have confirmed that Sametime is included

Are You Ready: Domino #Perfect10

In today’s edition of my #Perfect10 webcast I discuss some steps in finding your Domino servers, reviewing their dependencies and auditing database access.   This is a 15 minute presentation which I’ve tried just as slides with my voice instead of video.

As always please let me know what you think and anything else you would find useful.

Next Up: Are you ready – Traveler, Sametime and Web Mail

 

So What About Domino @ IBM Connect? Review Post #2

Domino was very visible at Connect this year, not only in both of the opening sessions but in about 40% of the sessions overall.   The ones I picked to attend were talking about strategy and futures so that’s what I wanted to talk about here.

Verse on premises which shipped at the end of Dec 2016 is a very nice browser mail client right now which is easy to install on your Domino server (and you should) but it’s missing an updated calendar interface,  so I was pleased to hear the commitment to deliver that and other functionality to bring on premises in line with Verse in the cloud.  If you don’t have Verse installed on premises now on your Domino servers you need to be looking at it as your path forward.

Feature packs continue to be the strategic path with updates coming via FP installers but with template updates slipstreamed in optionally and separately downloadable through Fix Central.  I wouldn’t look for the templates to ship in step with the feature packs so you’re going to have to plan to subscribe to fix central for updates if you aren’t already.

From Barry Rosen’s strategy presentation here are a couple of snapshots showing planned feature pack features including those for FP8 which should ship soon.

Notes Feature Pack highlights screen-shot-2017-02-26-at-20-51-26

Domino Application Development feature pack highlights (FP8 shipping soon)screen-shot-2017-02-26-at-20-51-41

 

For application design the path IBM appear to be on is one we and many other Business Partners have been pursuing for some time with Domino as a back end data store and a web based UI on whatever platform you choose.  To that end the really good news is that we will finally be getting some extensions to the existing REST APIs including ones for

  • Directory
  • Mail Contacts
  • Mail File Search
  • Polling for changes in databases

In addition the application modernisation story at the conference was focused around partner solutions.  Of particular interest is Panagenda’s ApplicationInsights tool coming in a freemium model to all maintenace customers in Q2.  That version I believe will allow you to analyse your most prominent existing applications and instances to see what is being used by who and how. More information about it can be found here.

So lots of Domino sessions, lots of talk of future client and server developments, lots of confirmation of support at least to 2021.  For a nearly 30 year old product that’s not bad going.  With the investment in Verse and the introduction of cognitive features in on premises applications as well as a cognitive plugin for Notes, I’m feeling positive about where we are and the support IBM are offering.

Oh and my watch word for 2017 continues to be “Hybrid”

 

 

Before second guessing IBM try a CTRL-F

A new press release just appeared from IBM announcing extending support for Domino 9, Notes 9, Traveler 9, Sametime 9.0 and Designer 9.0.1 amongst others.

Now do me a favour, before you do anything else,  press CTRL-F and look for the word “END”.  You won’t find it.  This is extending not ending support.

Now could IBM have done better by using the words “at least” – in my opinion yes but since I assume the document was minutely inspected by IBM lawyers, it can’t make any open ended promises.

We live in a world of fast changing technology and many of us work with technologies that are 20+ years old.  Who knows what will happen next year, in 2 years or in 5 years.  That’s a good thing.  We should embrace changing technologies that match how we, our environment and our work evolves. Every change offers an opportunity but today and for the foreseeable future it should be enough that Notes and Domino aren’t dead and they aren’t predicted to die anytime soon.

Not even in 2021.

 

From F to A In A Day

As I went to bed last night I set the alarm early, I have a lot to do this week especially since I’ll be at Icon UK for 2 days of it and I wanted to get started early.  So of course today was the day my work went out of the window and I lost 10 hrs debugging one of my own servers. Let’s back up…

This weekend I was prepping my presentations for Icon UK this Thursday.  One is called “Domino In The Back, Party In The Front” so I’m going to be talking about all the client options available to you using Domino as a back end.

On Sunday I had the idea of installing IMSMO (IBM Mail Services For Microsoft Outlook) on one of my lab machines.  I had a customer wanting to deploy and I wanted to try and mirror their setup, plus it meant I’d have something to demo from.  The lab server was already running 9.0.1 FP6 with a SHA2 SSL certificate delivering TLS1.2.  I hadn’t used any web services on it in a couple of weeks so I went ahead and added IF3 (required by IMSMO) and installed the application addin service.  It actually installs as a variant of Traveler (and I’ll be blogging on that later).  I completed the install and Outlook worked fine.  Unfortunately it was the only HTTPS service that worked.  Everything failed.  By failed I mean the browser – any browser – refused to connect.

So off I went to investigate why the browsers wouldn’t connect.  I started with testing via SSLLabs and that reported AN F as apparently the server was demanding SSLv3 instead of TLS 1.2   Of course just about every browser will refuse to accept a negotiation of SSLV3.  But why was the server suddenly demanding it when it had never done so before?

Well 10 hrs later I’d exhausted everything I could think of:

  • verified notes.ini had no additional unexpected settings
  • forced Disable_SSLV3=1 even though that server had been fine serving TLS 1.2 previously
  • disabled internet site documents and reproduced using web configuration
  • recreated the internet site and web rule documents
  • generated a new keyfile from my wildcard certificates
  • uninstalled IF3
  • uninstalled IMSMO including all the cleanup
  • scanned for anything that could be hijacking HTTPS
  • restarted and restarted and restarted http and clear cache upon cache upon cache
  • bothered Darren Duke for a sanity check – I believe the words “I don’t know what the hell is going on” came up
  • uninstalled Domino (around hour 8) because I couldn’t spend any more time troubleshooting

After uninstalling Domino. Reinstalling up to FP6, copying in the databases and templates and restarting.  I was back with TLS 1.2 again and suddenly SSLLabs was giving me an A+. 

Of course then I did what I should have done in the first place (saving time is never a time saver), I built a new lab server purely for IMSMO.  Installed FP6 and IF3 and the addin and everything worked perfectly including TLS1.2.

I have no idea what part of the IMSMO install , the addin or IF3, conflicted with my existing lab server configuration or what it did to force the server to only serve SSLV3 no matter how I tried to push it otherwise – but an uninstall and clean install ended up being my only fix in the time I had.  Someone somewhere knows the setting that made it do that.  I’d love to know what.

Now it’s 4.15am and I’m back where I thought I was at 11pm Sunday night.  The 4 days work I had to fit in 2 days , I have to fit in 1 day.  This week’s lesson. Never start something new when you barely have time to get the existing things completed.

See you at Icon UK

 

 

Last week in Eindhoven…

We were in Eindhoven last week at the Engage conference.. over 400 attendees, speakers and IBM’ers gathered for two days of learning, talking and cleaning out the hotel bar of tonic water.. I’ve been to several of the past Engage conferences and Theo always puts on a great event but this was bigger and better than ever.  So why?

IBM sent a lot of executives to Engage with the Opening General Session being given by the new ICS general manager (appointed at Connect in January) Inhi Cho Suh and with product strategy presented by Suzanne Livingston , Sara Gibbons and Chris Crummey.  The first thing Inhi announced was that things are going to change – starting with the Orlando conference which moves to February 22nd at Moscone West in San Francisco.  That’s a big decision and commitment – serious tech companies have conferences in SF and that’s where ICS (IBM Collaboration Services) need to be if they are going to innovate, lead and grow as opposed to maintain.   Inhi also let us know that she has asked the product team to work on a 2020 strategy and that it will include IBM Verse on premise.

Then we got the demo of Verse , Toscana and the thinking behind ICS design.  It’s a shame the OGS wasn’t recorded as Suzanne’s background to their design thinking and Sara & Chris’ demo were both much more detailed (and further advanced) than at Connect in January.  However if you want some idea of what we saw take a look at the OGS video from January (from about 90 seconds in to 20 mins in) here

Aside from the OGS the entire IBM team (of which there were more than 30 in attendance) were everywhere wanting to hear about problems, wanting to listen, wanting to change their relationship with partners, with customers with development for the better.   It’s hard not to be taken up with the positivity and enthusiasm.  I’m an optimistic person but I don’t consider myself naive – I feel that I recognise honesty and intent when people talk to me and I what I heard that ICS was important, investable and part of the core IBM development strategy.

In short I choose to believe until I’m proved wrong.

There were of course plenty of great sessions to attend and, as usual, I missed many of the ones I wanted.  Partly because there were also lots of round table discussions too which I found very interesting.  Apparently I’m still the 8 year old in class first to put her hand up with a question.

My session on SHA2 and SSL vulnerabilities was against Mat Newman’s User Blast and Sara Gibbons’ with Toscana.   We were all along the same corridor and I watched person after person go past my room on their way to Mat or Sara’s , so thank you to everyone who chose to hear about security instead and filled out my room.  I hope you found it useful  (and the hand puppets helpful).  For anyone who wasn’t there I have added it to slideshare 

On the final evening of the event Theo invited speakers to a dinner preceded by a surprise.  The surprise was that 32 of us were sent into the Escape Rooms.. you are locked in a themed room for an hour and have to decode lots of puzzles to find the code to get out.  I’ve always wanted to try an Escape Room and I chose the “Tomb” which was an Egyptian tomb and went in with a team including Tim and Mike, Sue Smith, Bill Malchisky, Mat Newman, Rene Winkelmeyer and Carl Tyler.  We didn’t make it out in time – we were soooooo close.. but a few things to bear in mind

  • The tomb was entirely dark except for a small flashlight Tim found hidden in a basket in a corner and some candles.  My night vision varies from “bad” to “crappy”
  • Having multiple alpha males in a small space all shouting instructions at each other may not be the best way to get out quickly
  • There was sand everywhere.  Everywhere.  My shoes may never recover
  • Tim is great at puzzles but apparently in the dark, without his glasses (which he forgot to bring in) and with 7 people shouting at him to hurry up – not so much
  • There was a really cool effect where we completed a puzzle and lasers appeared out of the eyes of a skull on the wall and we had to position 7 different mirrors around the room to bounce the lasers around to hit a small hole on the wall.  We got so excited doing that we didn’t notice we had completed the puzzle and a new “door” had opened for about 10 mins.
  • I was given a cryptex to decode and open.  I broke it by pulling the end off.
  • With only 1 light source we could only do one thing at a time so some of us spent a lot of time kneeling in the sand feeling around fake skeletons for clues

In the end it was great fun and I’d definitely want to do it again.

All of that plus a chance to talk to lots of customers and see lots of friends – some of which came along just to meet up.

I hope you’re recovered Theo – because we’re all up to do it again next year.

 

 

 

 

 

A Statement From IBM On El Capitan and iOS9 Support

IBM have today released a statement explaining why some applications will be unable to connect to Domino servers from iOS9 and El Capitan devices due to Apple removing support for Elliptic curve technology (no – me either) and enhanced transport security.  This doesn’t affect only IBM but it’s something you need to be aware of.  There will be an interim fix for Domino 9.0.1 FP4 and also a new FP5 to resolve these issues (eta end Sept) but there will be no fix for Domino 8.5.x servers.

The full statement and explanation is here but the key summary is

Additionally, IBM is working on an Interim Fix for 9.0.1 Fix Pack 4 (and the upcoming 9.0.1 Fix Pack 5) that will implement Elliptic Curve cipher support for TLS 1.2 and TLS 1.0 that remedies this issue and implements Elliptic Curve support for the following protocols: HTTP/HTTPS, LDAP/LDAPS, SMTP, IMAP, and POP3. Currently, the ETA for the Interim Fix posting is end of September 2015.

Elliptic Curve support will not be available for Domino 8.5.x releases since the specification requires updated cryptographic libraries that are available only in Domino 9.0 and above.

Domino 9.0.1 FP4 Crashes With HTTP On Linux and AIX

I discovered this on a customer site this weekend.  Their servers are running SLES Linux 64bit and already had Domino 9.0.1 FP2 on them.  I upgraded to  FP4 but only one of the clustered mail servers runs iNotes – that server kept crashing as soon as someone tried to access their mail.  The other server was stable and if I disabled HTTP the crashing server stayed up.

Turns out the IBM installer for FP4 on Linux and AIX is setting the ownership of the dojo folder incorrectly which causes the crash.  The dojo folder is under <notesdatadirectory>/domino/js and the ownership was set to invalid names.  From the js directory (which just has the dojo folder in it) I ran

chown notes:notes * -R

which told Linux to change the ownership of the dojo folder and everything beneath to the account / group used to run Domino.

There is a technote dated 28th August that i’ll post here but the fix on the technote is incorrect.  On their fix they say the permissions are wrong and need changing to 755 using chmod but that’s not true, they are already 755 in my installs but the actual ownership is wrong.  Maybe they’ll fix the technote but the background is here http://www-01.ibm.com/support/docview.wss?uid=swg21964549