Tag Archives: TDI

06.14.16
by

Severe TDI Issue In Connections 5.5

I have been working with a customer who is migrating to Connections 5.5 from Connections 5.0.   When I do a migration I like to do it properly and create clean data by using dbt.jar to migrate content to new databases.  I know a lot of people are happy with the backup/restore of databases idea but for me that leaves too much scope for bad data to carry over from old system to new.

Everything was going fine, the profiles data migrated and then I tried a sync all dns to sync the ldap data to the database.  Something we schedule daily at this customer.  It failed as it tried to hash the database tables.  The error in the ibmdi.log was

Error: The sort page size property - source_ldap_sort_page_size= - must be greater than 10 if it is not 0. Aborting.

That’s a value that is set in profiles_tdi.properties and it was already set to 0.  So why was it aborting?

I decided to troubleshoot just with a cutdown list of names in collect.dns and using populate_from_dn_file function.  Again it failed but with the strangest error that would find the user in LDAP, get all their values then fail to find the user in the database and fail to update.

In SyncUpdates.log I could see the following error no matter what user I chose for populate_from_dn_file.

ERROR [com.ibm.di.log.FileRollerAppender.bc9c35a0-aae5-416e-9a99-1d418c3c564c] - [callSyncDB_mod] [ProfileConnector] null
java.lang.IndexOutOfBoundsException
at java.util.Collections$EmptyList.get(Collections.java:87)

I then tried copying the collect.dns to my 5.0 production environment and running there and it worked fine, found the users as duplicates and didn’t update them which is the correct behaviour.

I compared the map_dbrepos_from_source.properties files in 5.0 with 5.5 and it all looked pretty much the same.  So I opened a PMR which was eventually escalated to development. As soon as they received it they knew what the problem was - apparently a known but not documented bug that was fixed in CR1 with files that you have to manually deploy (we were already at CR1).

Development’s report of the problem was

log4j.logger.com.ibm.lconn.profiles.internal.service=ALL           
                                                                       
in log4j.properties causes TDI populate and sync commands to crash if an
EMPLOYEE is altered        

Well the crashing was true but the value  log4j.logger.com.ibm.lconn.profiles.internal.service=ALL was # out and unused so it wasn’t related to that particular log setting in my case.

The fix was to go find the two files

lc.profiles.core.service.impl.jar
lc.profiles.core.service.api.jar

in the Connections install and copy them to your TDI\lib directory in your tdisol environment.  In my case I had created a folder called TDISOL55 and under that I had a TDI directory with all the properties, script etc files in and the lib subdirectory full of jar files.  That came from the D1 (day 1) release download of Wizards which contained updated TDIPopulation directory and was dated 18th Dec.  There was no new tdisol with CR1 but clearly there should have been.

I found the files in my Websphere Application profile directory for the profiles application server under the directory

D:\IBM\WebSphere\AppServer\profiles\AppSrv01\installedApps\conn55Cell01\Profiles.ear

I copied those two files over and it al-most worked.  I had one more problem.  The value source_ldap_sort_attribute in profiles_tdi.properties which was initially set to empty (not null but = ) had been changed at the request of L2 to source_ldap_sort_attribute=mail which matched the 5.0 properties we were using.  They asked me to change it for exact comparison and that broke the updates.  Once I took out the “mail” mapping the scripts, both populate_from_dn_file and sync_all_dns ran perfectly.

The new environment does use different LDAP servers (but the same source data) and I don’t know if attempting to tell the server to sort the LDAP results failing is a problem with that server configuration (both environments are Domino 9.0.1) or 5.5 itself. I’ll investigate that and update.

So my two fixes were

  • copy the two jar files from your CR1 installedapps directory to your TDISol directory (lib subdirectory).
  • make sure source_ldap_sort_attribute= in profiles_tdi.properties
09.16.13
by

Adventures in TDI - Connections and Updating Profiles

Recently (well the past couple of months) I have been working on building custom Assemblylines to sync Connections data held in DB2 to LDAP data held in Domino.  I really struggled with finding good documentation for doing this and that’s because the best documentation was written for 3.0.1 and hasn’t been updated since.  Thanks to help from some people at IBM (who I’d name publicly here but I’m not sure they want emails from everyone) I managed to get hold of some draft updated documentation and get answers to some questions as I went along so I thought it would be helpful to share what I’ve learnt although this is a very streamlined description, hopefully it will give you some pointers.

Firstly if you are using Connections you need to populate profiles.  Populating profiles requires you to have installed TDI (and the right fixpack, don’t try it without) but once you have done that you have 3 techniques for population, two of which involve you never seeing a TDI configuration screen.

1. The population wizard which is a graphical tool supplied by IBM for pulling data from LDAP into your Connections data source (DB2 in this case).  The population wizard is easy to use and meets that needs of I’d say 60%+ of users who are working with Connections.  It’s certainly where most people begin to get that information populated in the first instance.

2. Underlying the population wizard are XML and properties files.  If you run the wizard you’ll find it has written to  properties and XML files on the file system and then it uses IBM supplied scripts to run the import.   What you can do is take a copy of the TDISOL directory which contains all the properties, xml files and scripts (and is updated on Fix Central occasionally) and use those to create you own custom syncing.  The files you will be working with are

tdienv.bat/sh  This is where you tell TDI to find its own installed files

profiles_tdi.properties  This is where you tell TDI how to find the LDAP and DB2 sources as well as how to authenticate with them

map_dbrepos_from_source.properties This is where you tell TDI how and what to map from LDAP (Domino) to Connections (DB2)

map_dbrepos_to_source.properties  This is where you tell TDI how and what to map from DB2 to LDAP if required.

You can then schedule a batch file called sync_all_dns.bat /sh which will bi-directionally update the information according to those files.  I would always recommend a) doing this in a test environment first b) backing up PEOPLEDB before starting c) running collect_dns.bat/sh first to ensure your search scope for LDAP returns the users you expect

3. So if you’re still with me and you want something even more advanced you’ll need to create an Assemblyline using the TDI Configuration Editor. For instance sync_all_dns does a complete bi-directional sync so for 50k users it was taking nearly an hour and for 25k users 20 minutes because it has to check every record in both LDAP and DB2.  That was taking too long so we wanted something that ran faster, in another case we wanted to pull in data from another data source to populate profiles alongside the LDAP data.  Using TDI and creating a custom assemblyline allows you unlimited scope to pull data from any LDAP or JDBC source (amongst others) into your profiles.

You’ll hear a lot of talk about Assemblylines being “real time” but in fact that’s very difficult to do in a Connections environment.  There is no real time monitor of generic LDAP you can use.  There are change control monitors for both Domino and Active Directory if you are using those as your LDAP source you can use them to  continuously monitor the servers and trigger on any change.  I have found however there is a risk associated with a continuously running and monitoring service and, being risk averise,  I prefer to schedule my Assemblylines to run.     There is also a change connector for RDBMS you could use to monitor DB2 but again, I prefer to use the standard Connector with a SQL statement telling it to look for things modified in the past x minutes or whatever.  I then export the project  and create a batch file to call the Assemblylines I want, scheduling that to run repeatedly in Task Scheduler (for Windows) or Linux.

The hard and fast rule when creating an Assemblyline is that you must use IBM’s supplied files.  They have provided Assemblylines that can be copied and modified to do just about anything you need.  To begin with you will need to complete the tdienv.bat/sh and profiles_tdi.properties files which will be used by the Configuration Editor when it’s launched.  Once launched you’re going to create a new project by importing the profiles_tdi.xml file and that will present you with all the Connectors and starter Assemblylines you need.  In particular there is a Connector called the ProfilesConnector which contains everything needed to write to a profile.

Without going into 40 pages more detail, I hope this helped.  This paper from IBM is the absolute best source for setting up your environment, although significantly out of date it will set you on the right path .  I am also anonymising a simple Domino - DB2 and DB2 - Domino Assemblyline  that I am happy to share with people once it’s complete.  I can’t support it and you take it at your own risk but it may help you to see something configured.