Tag Archives: WebSphere

03.30.16
by

Last week in Eindhoven…

We were in Eindhoven last week at the Engage conference.. over 400 attendees, speakers and IBM’ers gathered for two days of learning, talking and cleaning out the hotel bar of tonic water.. I’ve been to several of the past Engage conferences and Theo always puts on a great event but this was bigger and better than ever.  So why?

IBM sent a lot of executives to Engage with the Opening General Session being given by the new ICS general manager (appointed at Connect in January) Inhi Cho Suh and with product strategy presented by Suzanne Livingston , Sara Gibbons and Chris Crummey.  The first thing Inhi announced was that things are going to change - starting with the Orlando conference which moves to February 22nd at Moscone West in San Francisco.  That’s a big decision and commitment - serious tech companies have conferences in SF and that’s where ICS (IBM Collaboration Services) need to be if they are going to innovate, lead and grow as opposed to maintain.   Inhi also let us know that she has asked the product team to work on a 2020 strategy and that it will include IBM Verse on premise.

Then we got the demo of Verse , Toscana and the thinking behind ICS design.  It’s a shame the OGS wasn’t recorded as Suzanne’s background to their design thinking and Sara & Chris’ demo were both much more detailed (and further advanced) than at Connect in January.  However if you want some idea of what we saw take a look at the OGS video from January (from about 90 seconds in to 20 mins in) here

Aside from the OGS the entire IBM team (of which there were more than 30 in attendance) were everywhere wanting to hear about problems, wanting to listen, wanting to change their relationship with partners, with customers with development for the better.   It’s hard not to be taken up with the positivity and enthusiasm.  I’m an optimistic person but I don’t consider myself naive - I feel that I recognise honesty and intent when people talk to me and I what I heard that ICS was important, investable and part of the core IBM development strategy.

In short I choose to believe until I’m proved wrong.

There were of course plenty of great sessions to attend and, as usual, I missed many of the ones I wanted.  Partly because there were also lots of round table discussions too which I found very interesting.  Apparently I’m still the 8 year old in class first to put her hand up with a question.

My session on SHA2 and SSL vulnerabilities was against Mat Newman’s User Blast and Sara Gibbons’ with Toscana.   We were all along the same corridor and I watched person after person go past my room on their way to Mat or Sara’s , so thank you to everyone who chose to hear about security instead and filled out my room.  I hope you found it useful  (and the hand puppets helpful).  For anyone who wasn’t there I have added it to slideshare 

On the final evening of the event Theo invited speakers to a dinner preceded by a surprise.  The surprise was that 32 of us were sent into the Escape Rooms.. you are locked in a themed room for an hour and have to decode lots of puzzles to find the code to get out.  I’ve always wanted to try an Escape Room and I chose the “Tomb” which was an Egyptian tomb and went in with a team including Tim and Mike, Sue Smith, Bill Malchisky, Mat Newman, Rene Winkelmeyer and Carl Tyler.  We didn’t make it out in time - we were soooooo close.. but a few things to bear in mind

  • The tomb was entirely dark except for a small flashlight Tim found hidden in a basket in a corner and some candles.  My night vision varies from “bad” to “crappy”
  • Having multiple alpha males in a small space all shouting instructions at each other may not be the best way to get out quickly
  • There was sand everywhere.  Everywhere.  My shoes may never recover
  • Tim is great at puzzles but apparently in the dark, without his glasses (which he forgot to bring in) and with 7 people shouting at him to hurry up - not so much
  • There was a really cool effect where we completed a puzzle and lasers appeared out of the eyes of a skull on the wall and we had to position 7 different mirrors around the room to bounce the lasers around to hit a small hole on the wall.  We got so excited doing that we didn’t notice we had completed the puzzle and a new “door” had opened for about 10 mins.
  • I was given a cryptex to decode and open.  I broke it by pulling the end off.
  • With only 1 light source we could only do one thing at a time so some of us spent a lot of time kneeling in the sand feeling around fake skeletons for clues

In the end it was great fun and I’d definitely want to do it again.

All of that plus a chance to talk to lots of customers and see lots of friends - some of which came along just to meet up.

I hope you’re recovered Theo - because we’re all up to do it again next year.

 

 

 

 

 

07.13.15
by

The IBM Docs Dilemma

IBM Docs is a really nice add on to IBM Connections, what’s more it’s not particularly hard to install.  It does have one requirement, a big one, a show stopping one, a requirement that prevented my customer build from working for about four weeks until IBM and I came up with an agreement for how it could work.  Hopefully this will help you fast forward through that four weeks yourself ..

IBM Docs Infrastructure - The Simple Version

IBM Docs has four component WebSphere servers with applications stored on each

The servers also need access to three data shares; the standard Connections share, a new share for IBM Docs data and a new share for IBM Docs Viewer.  I created the two new shared on the Linux server that currently hosted the CIFS Connections share and installed Samba to enable a Windows server to access them.

I had one problem where it consistently failed during install if I didn’t use capital letters for the mapped drives.  It didn’t refuse to accept lower case letters, it just failed the install.  If your install fails make sure you aren’t using lower case letters.

Challenges

The key requirement for IBM Docs to actually work is that

1. The shares must use mapped drive letters and those drives letters must exist prior to the IBM Docs elements being started

2. The IBM recommendation for achieving this is to create a batch file on the IBM Docs OS (which must be partially if not wholly Windows) to do the drive mapping and have that load in Windows task scheduler on startup.

3. The WAS servers must then be run as services not using a system account but using a named Windows account that matches the one assigned to run the batch file in task scheduler

This solution had two problems, I hated it, and it didn’t work.

I hated this idea because my customer doesn’t run AD at all and their share was a samba share on a Linux box using CIFS.  That means there is no account that can be used to start the services that can also be used to map the drives. There is no easy way to have Windows pass credentials to mount the shares without storing both the name and password that samba recognises in the batch file - like this

net use m: \\hubshared\ibmdocsdata sambapassword /user:sambaaccount
net use n: \\hubshared\ibmdocsview sambapassword /user:sambaaccount
net use l: \\hubshared\conntestshare sambapassword /user:sambaaccount

Unfortunately after several weeks of different ideas from L3 support we admitted defeat to allow me to move on with the install.  I have minimised risk by ensuring the account isn’t a linux account and only has access to the samba shares.

The second part of the solution is the assumption that if you map the drives through task scheduler owned by a Windows user and that same Windows user starts the WAS services - the WAS services will be able to see the mapped drives.  To everyone’s disappointment that absolutely didn’t work because Microsoft kindly mapped the drives from the batch file in a different session to the one where it started the WAS services.  The servers couldn’t see the mapped drives.

So the install was simple but getting everything running securely and without the customer having to manually do anything held us up for weeks.  In the end I opted for a solution where I created a batch file to both map the drives and then start the WAS servers in a scheduled startup script.  That worked beautifully and this is what it looks like

net use m: \\hubshared\ibmdocsdata sambapassword /user:sambaaccount
net use n: \\hubshared\ibmdocsview sambapassword /user:sambaaccount
net use l: \\hubshared\conntestshare sambapassword /user:sambaaccount

Call “c:\IBM\WebSphere\AppServer\profiles\IBMDocs\bin\startnode”
Call “c:\IBM\WebSphere\AppServer\profiles\IBMConversion\bin\startnode”
Call “c:\IBM\WebSphere\AppServer\profiles\IBMViewer\bin\startnode”
Call “c:\IBM\WebSphere\AppServer\profiles\IBMDocsProxy\bin\startnode”

As you can see I only start the nodeagents. The servers themselves and the applications on them are bootstrapped to the start of those. To do that modify the server’s monitoring policy which is found under Java and Process Management for each server

Then set the “Node Restart State” to “RUNNING”

bootstrap nodeagents

05.25.15
by

The Recurring WebSphere Bug And Connections External Users

I wanted to share a recurring WebSphere bug that I noticed over a year ago because although it was irritating then, if it occurs now it can actually prevent you from deploying Connections external users the way you want.

Here’s the scenario (and it’s fairly common for me).

IBM Connections 5 CR2 on WebSphere 8.5.5 FP3

Primary LDAP is a Domino server

Secondary LDAP for external users is a separate Domino server in an isolated domain

When we want external users to access our Connections environment, the most secure approach is to have a dedicated LDAP server or branch for external users to appear in.  Especially if (as we do) you have a self registration / password reset process for those users.  The problem occurs because we want to use Domino as our LDAP.  LDAP servers other than Domino are built with hierarchical entries so on the WebSphere configuration screen where we are asked for the “unique distinguished name of base entries” that’s very easy, we just select the top level of the hierarchy.  Unfortunately in Domino LDAP we don’t always have a hierarchy - we have flat names and we have flat groups so if we try and use a O= xx value - those names and groups aren’t picked up.

We used to use C=US which would trick WebSphere into querying a level above O= and that would work but since WebSphere 7.0.0.23 we have been using the word “root” which validates both flat names and all hierarchies on the server.

So far so great.

Now we want to add another LDAP server which will be a Domino server where people will register.  We’ll have two TDI processes one connecting to the internal Domino server for internal users and another to the external Domino server for external user access.  It’s Domino so we want to use “root” as our base entry but since WebSphere requires all federated repositories to have unique base entries and since we already use “root” for our internal server, I have to fake a hierarchy for external users just so I can add the 2nd LDAP.  It’s ugly but not unworkable.  It’s also not our problem.

 

The problem is that once I add the second Domino server or even a third.  My federated repositories in WebSphere look like this

Can you see what’s wrong?  That table reads from the underlying wimconfig.xml file found under the Deployment Manager profile /config/cells/<cellname>/wim/config.  That wimconfig.xml is fine which is why if I click on Manage Repositories they are all there.  I just can’t edit them from this screen, I can only edit from the previous screen and that one links to the last LDAP entry I added.

managerepositories

So that’s part of our problem.  It’s been there for a few years but since we could manually edit the wimconfig.xml to overwrite settings it was workable.  This is caused by the “root” base entry on the first LDAP.  That word “root” translates to an empty baseentries name= value in wimconfig.

Here’s the internal LDAP with baseEntries name=””

Here’s the external LDAP where I have defined a base entry of o=turtlehost

wimconfigexternal

 

The additional side effect of this bug (and I’m not sure we can call it a WebSphere bug since expecting hierarchical LDAP is a fairly standard thing) is that in the latest version of WebSphere, it refused to search the second external directory.  No error. Nothing.  Just refused to search it which meant those users couldn’t login.

I edited wimconfig.xml and added a O=Turtle to replace the baseentries name=”” etc and that fixed both the WebSphere view and the ability of users to login.

So where does that leave us.  Well it’s a problem because I want to use Domino. I don’t want to have to force a single hierarchy.  C=xx doesn’t work anymore to trick WebSphere.  “root” breaks both WebSphere and authentication.  That means I can’t have a secondary Domino server for external users and still use a “root” base entry for the internal server.  Without that “root” value, the flat Domino groups will be ignored.

That leaves me with a few options

1. Force a fake hierarchy on groups so I can have a base entry value that works and not use root

2. Use Directory Assistance and “root” but that allows external users to authenticate against my internal directory.  I don’t like that

3. Use an LDAP attribute to separate external from internal users instead of a dedicated LDAP server.  For security reasons i’m no fan of that either

4. Don’t use Domino for both LDAPs, only for one of them.  One “root” defined Domino server will work fine

 

03.25.15
by

Upgrading Filenet - Something To Watch Out For

Last night I was working on a Connections CR2 upgrade that included upgrading Filenet.    I was using this very nice piece of documentation from IBM on a lab walkthrough of upgrading to CR2 here .  To date my biggest problem with FileNet has been finding all the files needed and that’s especially true of the CR2 update files but I got there.  What tripped me up was upgrading the CE Client and FNCS (section 3.4.5 on the document).  The sample command is

update-fncs-ceclient.bat -was.dm.path “C:\IBM\WebSphere\AppServer\profiles\Dmgr01” - was.admin.user wasadmin -was.admin.password Passw0rd -fncs.fp.installer.location “c:\Downloads\FNCR2\IBM_CONTENT_NAVIGATOR-2.0.3.2-FP002- WIN.exe” -doInstallFNCS y -ceclient.fp.installer.location “C:Downloads\FNCR2\Filenet cr2\5.2.1-P8CPE-CLIENT-WIN.EXE” - doInstallCEClient y

The problem is that this script ran for about 6 minutes then bombed out.  I checked the fn-fncs-ceclient-update.log file in c:\ibm\connections\ccm\ccm\ccm\scripts (where the script is run from) and it showed a failure to make a SOAP connection (after it had already successfully made one earlier in the script).  I then checked wsadmin.traceout in the Dmgr profile under logs and saw references to “WASX7198W websphere The configuration service is not running” and other failures logging in like “empty credentials”.

OK so the problem was my login credentials used in the script for admin account and password right? Wrong.  Because that works fine earlier in the script. In fact the first thing it does is make a SOAP connection to the Dmgr server.  OK.  So the problem is that my password for the admin account contained a ! - we all know that WAS doesn’t like encoding and decoding special characters right?  Wrong. I tried another account, and another,  I tried wasadmin but that failed (and I wondered if that was because it was a FileNet update and FileNet wants an LDAP account).  At this point i’m bouncing between accounts trying to get this working.  I get the customer to remove the ! and give me a password with no special characters.  Still no luck.

Then I think a bit more.  What if the script isn’t using the values I gave it to run but instead is assuming they are already there in soap.client.props and is attempting to read them from that file. I go to check soap.client.props under Dmgr\properties and there is no value stored there at all.  I add my admin credentials in there and the script runs perfectly.

Today’s lesson is don’t assume a script you are running is using the credentials you give it!

 

09.03.14
by

Access Denied - Me vs OS and WebSphere Security

Today I went to apply a patch to a customer’s Sametime Proxy server.  This is a server that’s been around for a few weeks.  I’ve logged into the SSC countless times in that time.  I launch Installation Manager (using “run as administrator”) and when it gets to the “sign on to SSC” part it fails saying it can’t connect.  I check the logs in /users/myname/appdata/local/temp/SSCLogs and find the error saying it can’t resolve <sschostname>:9443/console/deployment/login.  So I try that URL in a browser myself  and sure enough it does fail.

Well I can guess what that is and it’s an easy fix.  In Sametime we map virtual hosts for each application including the SSC containing the hostnames and ports used by that application.  So I went to check that the default_host virtual host used by the SSC had 9443 in it.

Go to SSC on the Deployment Manager server through a browser, try and login using my file repository account.  Login failed.  Try again. and again.  and again. and again. Type into notepad to make sure there’s no caps lock or language issues.  Failed again. This is worrying, no-one else has access right now so no-one has changed any password. I check the SystemOut.log for dmgr and there are errors in there and in the FFDC files saying Password is wrong.  OK.  No need to panic.  I’ve seen this before when Dmgr gets low on memory so first things first, let’s restart the box.  If in doubt, reboot WebSphere.  Server comes back up and still I can’t login.

OK so now I start to worry.  I go find the security.xml file in the config for the cell and decode the password stored in there (don’t ask how because I shouldn’t be able to but it’s possible).  The password says it’s what I think it is.  I really really don’t want to go down the path of changing that password even though I can disable security and do that because that’s going to have knock on effects all over the place….So - deep breath - let’s try this again from another machine.  I go to the SSC from my desktop this time instead of a browser on the DMGR server and it logs in perfectly first time using the name and password that was failing when I tried from the DMGR server.  Back to the browser on the server, login still fails.   This makes no sense.

So the issue isn’t the “wrong password” at all.  The issue is that the security on the SSC OS is preventing me logging in via a browser - I assume preventing the browser accessing the files on the file system in some way.  In addition the SSC was unable to sync any nodes or restart any servers (this was new) although it could tell status - until I restarted everything manually under my account.  This appears to be a problem with the services on the SSC accessing the file system on any of the OS even its own.  The customer is looking into all of that since the environment is tightly locked down and I can’t see anything.

When I finally got in (and yes I could use the LDAP alternative accounts I had in there) I added 9443 and 9080 to default_host under the hostname of the SSC and the Installation Manager ran fine.

Today’s lesson learned..DON’T PANIC!

 

 

 

06.22.14
by

When bad wasadmins go missing

Working yesterday on deploying a new application in a test Connections environment I was logged into the ISC using wasadmin for hours. Eventually I finish my work and restart everything to test.  I go to login to the deployment manager and no account will work, not wasadmin nor any of the LDAP administrative accounts set up.  So what do I do?  Well first I need to work out what’s going wrong and I check SystemOut.log when trying to login and see this error as a root cause

CWWIM2009E The principal ‘AnonymousUser’ does not have the role ‘administrator’ required for the operation ‘GET CONFIGURATION’

Well OK, let’s back up ,since it happened after a reboot the change could have been made any time since the previous restart and wasn’t necessarily related to the work I was doing at all.  First I need to get into the ISC and to do that I need to disable ISC security so I can get in.  I edit security.xml in the /profiles/dmgr/config/cells/<cellname> directory and find the first enabled=”true” in the security tag and change that to enabled=”false” (make sure you save a copy of this file first).  Then stop the dmgr and start it again. I have trouble stopping it as the authentication isn’t working so , since the dmgr is the only WAS server running , I just terminate java.exe from task manager.  Having done that the URL for the dmgr  <hostname>:9043/ibm/console no longer asks for a password and lets me login using just a user name.  and I’m IN - albeit with no security so no way to start servers.

I go look at the Administrative users configured in the system and sure enough the LDAP admin accounts are there but wasadmin is gone.  I can’t add wasadmin because security is disabled and it can’t find the account.  I can work around it but a better solution is to tell the ISC to use the LDAP realm instead of the defaultWimFileBasedRealm (which contains wasadmin).  I go to Global Security, re-enable security from that screen (it was disabled by my earlier security.xml change) and then go into the federated repository and change the realm name from o=defaultWIMFileBasedRealm to whatever my LDAP realm is (in this case “root”) and then change the Primary administrative user name to one of my LDAP admin accounts (in this case gabdavis).

Now I can restart dmgr and login to the ISC with the name gabdavis (my ldap account) and its ldap password.  Once in there I can go to Administrative Users and re-add wasadmin with all the roles I need then (if I wanted to) go back to Global Security and revert the realm and primary administrative account back to what was set originally (above).

And that’s it.  I hope this is useful for anyone else who has a wasadmin go astray…Backup your deployment manager profile regularly people !