More Adventures In *** RHEL Configuration

I know I shouldn’t have blogged on Saturday – as soon as I think I have a problem fixed the universe rises up and slaps me roundly about the head.  So fast forward to the end, it’s Sunday night and I’m installing Connections on RHEL 7 so that’s good.  However to get there I had more hurdles which I’ll note here both for myself and for anyone else

I configured and enabled VNC and SSH for access which worked fine on the same network but not from any other network (“Connection Refused”).  The obvious first guess is that the firewall on the server hasn’t been disabled.  It’s always the first thing I do since I have perimeter firewalls between networks and I don’t like to use OS ones. So Saturday and Saturday night was an adventure in checking, double checking and checking again that I had the firewall disabled.  RHEL 7 has replaced iptables with firewalld but iptables still exists so my worry was that I had something enabled somewhere.  I didn’t think it could be my perimeter firewall since I had built the server with the same ip as an earlier server that already worked. My other guess was VNC being accidentally configured with –nolisten but that wasn’t true either.

By the time I went to bed Sunday morning I had ruled out it being the OS and was going to start fresh a few hours later.  I’d also noticed that although I could connect via VNC it was slow as hell despite having a ton of resources.

Sunday morning I decided to delete all the entries referring to that server’s ip on our Sonicwall perimeter device and recreate them.  That fixed the network access. The one thing I didn’t build from scratch was the one thing that was broken. *sigh*.

At this point I did consider switching to Windows 2016 on a new box but I already planned to use that for another server component and wanted to build with mixed OS. Also #stubborn.

So now I have VNC and SSH access but the GUI is awful. I can’t click on most of the menus and it keeps dropping out.  I’m running GNOME 3 and I can find endless posts about problems with GNOME 3 and Cent OS or Redhat so I bite the bullet and install KDE because all I want is a GUI.  KDE is as bad, slow, menus not clickable.  I make sure SELINUX is set to “Disabled” but still no luck.   I try installing NoMachine as an alternative method but that has the same problem with the GUI – slow, unresponding, menus unclickable and eventually a crash with “Oh no!, Something has gone wrong”.  Which isn’t nearly as entertaining the 100th time you see it.  Along the way I disable IPV6 entirely and found and fixed this bug

https://bugzilla.redhat.com/show_bug.cgi?id=912892

and this one

https://bugzilla.redhat.com/show_bug.cgi?id=730378

oh and this irritating setting

https://access.redhat.com/solutions/195833 “Authentication is required” prompt

Throughout Sunday I’m continually working with /etc/systemd/system/vncserver@:1.0 to modify the settings, create new instances, create new VNC users but no matter what I try it proves unworkable.

I’m using the Red Hat instructions from here which has a configurator you can use to automatically create the file vncserver@ file according to your settings.  I’m suspicious of that file because it has settings I don’t normally use like  -RANDR so eventually I edit the file and change

ExecStart=/sbin/runuser -l turtlevnc -c \”/usr/bin/vncserver %i -extension RANDR -geometry 1024×768\”
PIDFile=~turtlevnc/.vnc/%H%i.pid

To

ExecStart=/sbin/runuser -l turtlevnc -c “/usr/bin/vncserver %i -geometry 1024×768”
PIDFile=~turtlevnc/.vnc/%H%i.pid
Cleared the /tmp/X11.unix/X? directories and restart once more.  Everything including GNOME 3 works and it’s zippy zippy fast.

 

So. Note to self. Next time remove that RANDR setting and win yourself an entire day back.

 

Me vs Technology (spoiler: I win)

Yesterday Connections 6 shipped and although I was in meetings all day my goal for last night was to get everything downloaded and in place on a VM and have that VM built with a configured and hardened OS.  That was the plan.  I thought it might be fun to share my 4pm – 4am battle against technology and maybe it will help someone else.  It might also explain all the “other” work that tends to take up my time before I  ever get to the actual stuff I’m meant to be installing.

All my servers are hosted in a data centre and mostly I run ESXi boxes with multiple servers on them. I have 5 current ESXi boxes. So first things first, create a new virtual machine on a box with capacity so I can download the software.  All of this is done from a Windows VM on my Mac which connects to Turtle’s data centre

Vsphere lets me create the machine then gives me VMRC disconnected when I try and open a console.  After some checking I realise it’s the older ESXi boxes that are throwing that error for every VM and only since I upgraded to Windows 10.  If I can’t open a console on the VM I can’t do anything so I search the internet for various random advice which included

  • Disable anti virus
  • Remove Vsphere
  • Install latest Vsphere (which keeps being overwritten with an older one each time I connect to an older machine)
  • Uninstall VMware Converter (which I had forgotten was even there) – that required me booting into safe mode in my VM which only worked if I used msconfig to get it to restart in safe mode
  • Downgrade Windows
  • Create a new clean desktop VM to install Vsphere into

This is a bigger problem than just this install because I also can’t manage any of my servers on those boxes.  I rarely connect to them via the console so I don’t know how long it’s been like that but it can’t stay like that.

Several hours later.. still no luck. Vsphere lets me do everything to a virual machine except open a console.  I could use another ESXi box but I’m being stubborn at this point. I want to use this box

Then I find reference to VGC – Virtual Guest Console  https://labs.vmware.com/flings/vgc.  Created in VMWare labs in 2010 and still in “beta” it does one thing I need which is open a console.  So now I have VSphere where I can create and manage the instances and the VGC to open a console I’m ready to install and OS.

But which OS?  The host boxes have ISOs on them I already use but those are Windows 2012 R2 and RHEL 6.4.  I want either Windows 2016 or RHEL 7.1  Again I could use Windows 2012 but #stubborn.

I download Windows 2016 to my Mac and it’s over 5GB.  That’s going to take a few hours to upload to the datastore and I’m optimistically thinking I don’t have a few hours to waste.  So Plan B is that I take an existing RHEL 6.4 ISO and use that to install then upgrade it to 7.1 in place since you can now do that with Redhat if you’re moving from the latest 6.x to 7.x.  Top tip – it would have been quicker to upload Windows 2016.

I start building the new VM using RHEL 6.4 and eventually I get to the point where I can tell it to get all updates and off it goes.  It’s now 1am and it’s showing 19/1934 updates.  So.. I go to bed taking my iPad with me and leaving my laptop downstairs.  Once I’m in bed I can use Jump on the iPad to connect to my laptop which is on the same network and Terminus and the VPN on the iPad to open a putty session to the data centre.  The 6.4 updates finish and now I need to get it to 7.1  First thing I need to do is download 7.1 directly to that new VM which I can do easily because I installed a browser so I download the 3GB ISO directly to the VM which only takes 3 minutes and I’m ready to install.

Except not quite.  Redhat requires to you run their pre upgrade utility before doing an inplace upgrade.  In fact the upgrade won’t even run until you run pre-upgrade.  So I do that and as expected it fails a bunch of stuff that I don’t care about because this is a new machine and I’m not using anything yet so I’m not bothered if something stops working.  Except the upgrade still won’t run because it spots I failed the pre upgrade test.  That’s where “redhat-upgrade-tool -f” comes in.  Around 4am I left that running and got some sleep.

Incidentally this is a great document on upgrading but I think you may need a login to read it https://access.redhat.com/solutions/637583

At 7am I found it completed at RHEL 7.1 and then ran one more update to make sure everything was on the latest patches,  added the GUI and configured the firewall.

I’m NOW ready to download Connections 6

Session from InterConnect – IoT In The Enterprise

Firstly I’d like to thank Chris Miller from Connectria who wrote and submitted the original abstract then kindly let me have the session when he had a scheduling conflict.  Any issues or problems with the content are down to me not Chris so please don’t hold him responsible 🙂

The original abstract was

Enabling Internet of Things (IoT) so your employees and your customers can have a simplified experience with new services and products sounds exciting. In this session, we will dig into the top ten risks that come with the IoT experience. Due to the rapidly evolving nature of IoT and associated threats, there are risks in allowing access to your enterprise resources. Custom firmware, embedded operating systems and wi-fi connectivity of IoT devices offer many possible areas for exploits and misuse. Come explore current security offerings and get a first look at best practices. Walk away with an immediate checklist to benefit your enterprise as it deploys and offers IoT access.

There are several aspects to IoT in the Enterprise which are important to the world of collaborative working

  1. IoT devices generate a huge amount of data. That data has to be analysed and actioned.  In a presentation at InterConnect IBM made the point that 80% of data analysts’ time is spent on cleanup and scrubbing not analysis.  Although we have had access to big data for many years, most companies simply haven’t gotten their heads around how to work with it.  That’s going to become more and more critical as IoT devices start to appear in companies.
  2. Security is a huge issue with IoT devices that are still primarily designed for consumer use.  Most devices still transfer data over HTTP (even authentication data) and security has not been a priority.  The introduction of blockchain technologies such as the one IBM has developed is the best chance for having secure IoT devices but we’re not there yet.
  3. IoT is really the beginning of Industry 4.0 with 3.0 being “the internet” 2.0 being “the conveyor belt” and 1.0 being “steampower”.  Consider that your company is on the precipice of the beginning of the internet. You’ve heard of it, you wonder where it’s going to take you, you might be considering something called email.  Well IoT is going to change your business and give you the same kind of opportunities to leap ahead of your competitors as the Internet did.  This isn’t something you can choose to ignore.
  4. The technology might not yet be there but now is the time to consider how you would change your business processes if you could access any data and use it in any way.  Again, consider the changes in processes pre Internet and now.
  5. Being able to analyse data , redesign business processes on the fly and take action is all in the DNA of those of us who have worked for years in the ICS community.
    Data Analysis = WATSON
    Business Process Action = WATSON APIs

I will be presenting (hopefully with Chris) on this at Engage in Antwerp on May 9th. You can register for that here

Watson Work Services – Connect Review #4

I know it’s a bit late in the day but I have a couple more things I want to talk about post Connect and with preparations for Interconnect and trying to tie up work before I go away – well these got pushed back.

Watson Work Services, what is it?  WWS (not sure if anyone else is using that acronym but let’s go with it) is not a product, it’s a platform. It is designed to connect to Watson’s APIs and leverage those for language, search, and data. The results can then be fed back to your application and used to trigger actions.  If you’ve seen Watson Workspace (formerly known as “Toscana”) then you might know that it is underpinned by Watson Work Services.   I stole this screenshot from Marc Pagnier’s presentation which I think explains the role WWS is intended to play.

Screen Shot 2017-03-14 at 23.37.37

So why is this good news? Well most of us have heard of IBM’s Watson efforts and understand some of the things Watson can do but for the majority the idea of accessing Watson’s APIs or applying its intelligence to our data appeared out of reach. I mean it’s not like you’re going to install Watson on site.  WWS gives any size company or even single developer access to those Watson APIs without installing anything on site and without investing a lot of money.  In fact WWS works within Bluemix and so your application, whether on premises or in the cloud, can call a query to WWS to feed it data and get results back you can then store and act on.  The cost is calculated in pennies each time you run a WWS query so , as an application designer, that is entirely within your control.   With that model you can easily and quickly experiment with integrating cognitive logic and intelligent behaviour into your applications.

To get started with WWS go to https://developer.watsonwork.ibm.com and to access example applications visit http://github.com/watsonwork . To stimulate  your creative brain here’s another screenshot I stole that shows some of Watson’s APIs and you can find out more about what they can do here Screen Shot 2017-03-14 at 23.53.36

For a start we already have several ideas for our customers who generate a lot of data and would benefit from integrating  intelligent analysis and action triggers into their applications.

 

 

What Kept Me Busy In 2016 and Where Am I Going Now?

I think this post might be just under the wire for 2016 reviews so let’s talk about what I was working on and learning for the past year.  I always need to be learning, if I’m not I feel like I’m standing still and last year most of my learning moved outside of the core IBM products simply because there was little new to learn.

So what kind of projects did I work on?

  • Security reviews of Domino, Connections, HTTP environments
  • Single Sign On projects including deploying SAML using ADFS and TFIM as well as lots of Kerberos / IWA integration projects
  • Designing hybrid environments for customers moving mail to the cloud
  • Lots of TLS configurations on lots of different products
  • IBM Connections upgrades to 5.5
  • IBM Sametime deployments from sites that had 8.5.2
  • Domino consolidation, maintenance and hardware migrations
  • High Availability for Traveler, Domino HTTP and Sametime

What was I learning?  I’m always looking for interesting and challenging technologies that can make a difference to those smaller customers who need to stretch a tight budget.  It’s how I got involved with Notes originally in the early 90s – It allowed me to make big changes quickly for smaller customers.  This year that has meant staying on top of cloud and hybrid security issues and single sign on products and technologies.  Beyond that I have become really interested in data visualisation and have been working with products like Tableau and some of its cheaper competitors to see what they can offer.

Then in December I signed up for a Lynda.com subscription to ensure I have a good grounding in wider technologies and how they can work together.  Of course signing up and actually making time to learn are two different things so that takes us to 2017.

Goals for 2017

  • More data visualisation tools / learning cool things to do with Tableau
  • Building myself a Lynda training plan
  • Deploying Verse on Premise for existing Domino customers and introducing those without Connections to that integration piece
  • More work with database technologies around performance and security
  • Identify ways to deploy docker solutions with better stability and security
  • Improving my languages (I’ve been working on Italian and want to learn Spanish)
  • Working on interesting projects or ones that make a difference

As you can see my “goals” are fairly loose, I am always open to new ideas for technologies to learn (except development languages – blech).  It may be my review of 2017 will be nothing like my goals list and I won’t consider that a failure.

 

 

From F to A In A Day

As I went to bed last night I set the alarm early, I have a lot to do this week especially since I’ll be at Icon UK for 2 days of it and I wanted to get started early.  So of course today was the day my work went out of the window and I lost 10 hrs debugging one of my own servers. Let’s back up…

This weekend I was prepping my presentations for Icon UK this Thursday.  One is called “Domino In The Back, Party In The Front” so I’m going to be talking about all the client options available to you using Domino as a back end.

On Sunday I had the idea of installing IMSMO (IBM Mail Services For Microsoft Outlook) on one of my lab machines.  I had a customer wanting to deploy and I wanted to try and mirror their setup, plus it meant I’d have something to demo from.  The lab server was already running 9.0.1 FP6 with a SHA2 SSL certificate delivering TLS1.2.  I hadn’t used any web services on it in a couple of weeks so I went ahead and added IF3 (required by IMSMO) and installed the application addin service.  It actually installs as a variant of Traveler (and I’ll be blogging on that later).  I completed the install and Outlook worked fine.  Unfortunately it was the only HTTPS service that worked.  Everything failed.  By failed I mean the browser – any browser – refused to connect.

So off I went to investigate why the browsers wouldn’t connect.  I started with testing via SSLLabs and that reported AN F as apparently the server was demanding SSLv3 instead of TLS 1.2   Of course just about every browser will refuse to accept a negotiation of SSLV3.  But why was the server suddenly demanding it when it had never done so before?

Well 10 hrs later I’d exhausted everything I could think of:

  • verified notes.ini had no additional unexpected settings
  • forced Disable_SSLV3=1 even though that server had been fine serving TLS 1.2 previously
  • disabled internet site documents and reproduced using web configuration
  • recreated the internet site and web rule documents
  • generated a new keyfile from my wildcard certificates
  • uninstalled IF3
  • uninstalled IMSMO including all the cleanup
  • scanned for anything that could be hijacking HTTPS
  • restarted and restarted and restarted http and clear cache upon cache upon cache
  • bothered Darren Duke for a sanity check – I believe the words “I don’t know what the hell is going on” came up
  • uninstalled Domino (around hour 8) because I couldn’t spend any more time troubleshooting

After uninstalling Domino. Reinstalling up to FP6, copying in the databases and templates and restarting.  I was back with TLS 1.2 again and suddenly SSLLabs was giving me an A+. 

Of course then I did what I should have done in the first place (saving time is never a time saver), I built a new lab server purely for IMSMO.  Installed FP6 and IF3 and the addin and everything worked perfectly including TLS1.2.

I have no idea what part of the IMSMO install , the addin or IF3, conflicted with my existing lab server configuration or what it did to force the server to only serve SSLV3 no matter how I tried to push it otherwise – but an uninstall and clean install ended up being my only fix in the time I had.  Someone somewhere knows the setting that made it do that.  I’d love to know what.

Now it’s 4.15am and I’m back where I thought I was at 11pm Sunday night.  The 4 days work I had to fit in 2 days , I have to fit in 1 day.  This week’s lesson. Never start something new when you barely have time to get the existing things completed.

See you at Icon UK

 

 

User Denied Access To Files and Wikis

Another PMR this week on a new 5.5 side by side build. Once built everything looked OK except a couple of users in IT who received access denied errors when going to Files or Wikis, everything else worked.  Those two applications have databases with pretty much the same schema so we often see matching problems in both applications.

Checking the application security I could see that both were set to All Authenticated so there was no reason why those users couldn’t get at files.  The browser error contained

Identifier: LC6C54CE35BA4D41BF8CB2461634B9EAE6 EJPVJ9275E: Unable to add a group with the directory ID [E7F267C7-8811-D8EC-8025-7E57004A5278, 4339D1D3-2F37-ACDB-8025-7E57004A5285, C0085F47-7A84-EFD4-8025-7E57004A51FA, 4DB58BD6-77EA-80AC-8525-6B700078923E, A5456CF5-9FA0-E49B-8025-7E57004A5316, 54578802-623A-2E18-8025-7E57004A5289, 4EEFAFD1-A098-4155-8025-7F1D00522430, 0D1FD4C5-F61E-CB15-8025-7E57004A51F6, 5A3E2519-52BE-F072-8025-7E57004A527B, 04CE2967-BD15-B84D-8025-7E57004A52F1, 0D162A8C-223C-33C3-8025-7EB4002F6ADF, 6DCCEAE9-6A16-2A75-8025-7E57004A5377, 2B5D0EBA-B225-BA42-8025-7E57004A52DF, C6B296E2-5D27-0F89-8025-7E57004A532A].

If I count how many directory IDs are listed there, there are 14 which matched the number of groups that user was a member of when doing an LDAP query.  Still we weren’t using groups for any access and this exact configuration was working for the same users in 5.0.

In the SystemOut.log I could also see

CWWIM4546E  Duplicate entries were found in the external identifier ‘d68bb54dea77ac8085256b700078923e’ in repository ‘d68bb54dea77ac8085256b700078923e’.

That ID (formatted in various ways) would not resolve to any group in Files or Wikis never mind to duplicates.

Eventually David McCarthy @ IBM got me to change the wimconfig.xml file on the deployment manager which fixed the problem.  My configuration didn’t exactly match the documentation which said to change

<config:baseEntries name=”o=ORGX” nameInRepository=”o=ORGX”/>
to
<config:baseEntries name=”” nameInRepository=””/>

my configuration only had <config:baseEntries name=”” – no ORGX and no nameInRepository at all.  I believe that’s because we use  Domino for LDAP and “root” as the base entry so my federated repository looks like this – a configuration that results in no entry for nameInRepository in wimconfig.xml.

Screen Shot 2016-06-29 at 14.52.26

Once more this isn’t a problem in 5.0 but possibly due to a change in WebSphere behaviour in a newer version, I had to manually edit wimconfig.xml to add the nameInRepository=”” value.

At IBM’s request I also added the Group Membership Attribute which is used for resolving nested group memberships.  This customer uses Domino for LDAP and doesn’t really use nested groups in Connections so in 5.0 it was empty and worked fine however 5.5 may have been struggling with resolving group memberships for some individuals.  In 5.5 having it set to empty could have been contributing to the access problem.

The screenshot below is from 5.0. Screen Shot 2016-06-28 at 19.13.56this is how I changed it in 5.5 (same LDAP source, same users, same everything else)

Screen Shot 2016-06-29 at 15.06.31

Resyncing and restarting then fixed the problem and the users concerned could suddenly access Files and Wikis.

Not sure why it didn’t work for those users before the changes but it could have been something to do with one particular group and its nesting or maybe even a replication conflict which I couldn’t find.

Go figure.