Using Node.js to access Domino

You will be pleased to hear that the Domino 10 module for Node.js is now in beta (you can request early access here) and in this article I would like to show you how easy it will be to use.

Before we get started using the Domino module in Node, we do need to do some admin stuff on our Domino server. It has to be running Domino 10 and we have to install the Proton add-in, and we also have to create the Design Catalog including at least one database. (The Proton add-in listens on its own port, by default 3002, and is separate from HTTP.)

More detail on these admin tasks will be covered in a companion blog, but here I would like to focus on the app-dev side.

Lets build a simple Node.js system that will read some Domino documents and get field values, and also create some new documents. In my next post we can integrate this into the basic Node stack we developed in my previous post to create an actual API.

Assuming our Domino server is all set up, the first thing we do is install the new dominodb module. When this goes live later in the year, we will do this with the usual  ‘npm install dominodb’, but while we are still in beta we install it from the downloaded beta package:

npm install ../packages/domino-domino-db-1.0.0.tgz --save

Once we have the dominodb connector installed, we can use it in our Node server.js code. The sequence is almost exactly the same as in LotusScript. First, we connect to Domino and open the Server (sort of equivalent to a NotesSession), then from this we open our Database, and then using the Database we can access and update Documents.

We start the same way as with all other Node.js modules, with a ‘require’:

const { useServer } = require('@domino/domino-db');
This ‘useServer’ is a function and is going to create our server connection. We give it the Domino server’s hostname and the Proton port, and it connects to our server like this:
serverConfig = {
    hostName: 'server.mydomain.com',
    connection: { port:'3002'}
};
useServer( serverConfig )
The useServer function returns a ‘server’ object. This is inside a javascript promise (I talked about promises in an earlier post), so we can use the server object inside the promise to open our database like this:
useServer( serverConfig ).then( async server => {
    const database = await server.useDatabase( databaseConfig );
The databaseConfig contains the filepath of our database on the server:
const databaseConfig = {
    filePath: 'orders.nsf'
};

Notice how we are using ‘async await’ to simplify the asynchronous nature of getting the database. This code looks very similar to the LotusScript equivalent.

So we have created a server connection and opened a database. Now we want to get some documents.

The domino-db module uses the new Domino Query Language (DQL) which comes with Domino 10. This can perform very efficient high-performance queries on Domino databases.

It is very much like getting a document collection with a db.search( ) in LotusScript, and the query syntax is similar to selection formulas, for example:

const coll = await database.bulkReadDocuments({
    query: "Form = 'Order' and Customer = 'ACME'"
});
This returns a collection of documents in a JSON array. These documents do not automatically contain all the items from the Notes documents. By default they only have some basic metadata, i.e. unid, created date, and modified date:
{
  "documents":[
    {
      "@unid":"A2504056F3AF6EFE8025833100549873",
      "@created": {"type":"datetime","data":"2018-10-25T15:24:00.51Z"},
      "@modified": {"type":"datetime","data":"2018-10-25T15:24:00.52Z"}
    }
  ],
  "errors":0,
  "documentRange":{"total":1,"start":0,"count":1}
}
To get field data from the documents, you need to specify which fields you want returned, and this is done with an array of itemNames:
const coll = await database.bulkReadDocuments({
  query: "Form = 'Order' and Customer = 'ACME'",
  itemNames: ['OrderNo', 'Qty', 'Price']
});
Then you get the field data included in your query results:
{
"documents":[
  {
    "@unid":"A2504056F3AF6EFE8025833100549873",
    "@created":{"type":"datetime","data":"2018-10-25T15:24:00.51Z"},
    "@modified":{"type":"datetime","data":"2018-10-25T15:24:00.52Z"},
    "OrderNo":"001234",
    "Qty":2,
    "Price":25.49
  }
],
"errors":0,
"documentRange":{"total":1,"start":0,"count":1}
}
We can output the document field data with JSON dot notation in a console.log (with the JSON.stringify method to format it properly):
console.log( "Order No: " + JSON.stringify( coll.documents[0].OrderNo ) );

So this is how to read a collection of documents. Now lets look at creating a new document, which is even easier.

First we build our document data in an ‘options’ object, including the Form name and all our other item values:

const newDocOptions = {
  document: {
    Form: 'Order',
    Customer: 'ACME',
    OrderNo: '000343',
    Qty: 10,
    Price: 49.99
  }
};
Then we simply call the ‘createDocument’ method, like this:
const unid = await database.createDocument( newDocOptions );

We don’t need to call a ‘save’ on the document, it is all handled in one operation. The return value is the unid of the newly-created document, so we can act on it again to update it if we want to.

To update a document, we get it by its unid with the ‘useDocument’ method, and then we can call ‘replaceItems’ on it.

This takes the new values in a ‘replaceItems’ parameter, but it only needs to contain the fields to update:

const document = await database.useDocument({
  unid: coll.documents[0]['@unid']
});
await document.replaceItems({
  replaceItems: { Qty: 11 }
});

Here we are using the ‘@unid’ value from the collection we got earlier. This is a bit fiddly because ‘@’ is a reserved symbol in JSON, but we can use the JSON square bracket notation to get around this.

We have got documents and created and updated documents. In addition to all these, the domino-db module provides a variety of methods for reading, creating and updating documents, allowing you to do anything you would need.

Also, the DQL syntax has sophisticated search facilities that can return large numbers of documents and can even search in views and columns.

There are couple of things to be aware of:

The first is that by default the connection to Domino is unsecured, but you can easily make it use TLS/SSL. You may need help from your Domino admin to provide you with the certificate and key files, but this is all explained in the documentation.

The second thing is that currently this access is either Anonymous or can be set to a single Domino user account in TLS/SSL by the client certificate (which maps to a Domino person document). So in the beta there is no user authentication per se, but this will be coming later with OAuth support.

I hope you found this both useful and exciting. In my next article I plan to show how to build these domino-db methods into a Node HTTP server and create an API gateway.

 

Deploying The AppDev Pack – An Admins Guide

Over here on the blog is Tim’s next entry talking about Node development and Domino, this time he explains how to use the early release of the app dev package to access (read and write) Domino data via Node.  However I don’t let developers do Domino admin so this is the bit where I explain how to configure Domino.  It’s all very easy and also all still early release so things may well change for GA.

First you will need to request the early release package which you can do here. What you’ll then get is a series of .tgz files including one entitled ‘domino-appdev-docs-site.tgz’ which, once extracted, gives you the index.html with instructions for installing.

You need to bear in mind that at least initially this only runs on Linux and Domino 10 and that Domino 10 on Linux 64bit officially means RHEL 7.4 or higher, or SLES 12. I went with RHEL 7.5.

Next we need to install  “Proton” so it can be run as a Domino server task which just means extracting the file ‘proton-addin.tgz’ into the /opt/ibm/domino/notes/latest/linux directory.   There is also some checking to make sure files are present and setting permissions but I don’t want to repeat the install instructions here as I would rather you refer to the latest official version of those.  Suffice it to say this is a 5 minute job at most.

Once the files are in place you can start and stop Proton as you would any other Domino task by doing “load Proton”, “tell Proton quit”, etc.

Then there are a few notes.ini settings you can choose to set including:

PROTON_SSL
= if you want the traffic between the Proton task and Node server to be encrypted (0/1).

PROTON_LISTEN_PORT= what port you want Proton to listen and be accessed by Node on (default 3002 ).

PROTON_LISTEN_ADDRESS= if you want Proton to listen on a specific address on your Domino server such as 127.0.0.1 which would require Node to be installed locally or 0.0.0.0 which will listen on any available address.

PROTON_AUTHENTICATION= how Proton handles authentication.  There are currently two options, client_cert or anonymous.  With authentication set to anonymous all requests that come from the Node application are done as an “anonymous” Domino user and your Domino application must allow Anonymous rights in the ACL.

The “client_cert” option requires the Node application to present a client certificate to the Proton task and for the Domino administrator to have already mapped that certificate to a specific person document by importing it.  Note that “client_cert” still means that all activity from that Node application will be done as a single identified user that must be in the ACL but does mean you need not allow anonymous access.  You can also use different identities in different Node applications.

Of course, what we all want is OAuth or an authentication model that allows individual user identities and this is hopefully why the product is still considered “early release”.   Both the “anonymous” and “client_cert” models are of limited use in production.

PROTON_KEYFILE
= the keyfile to use if you want PROTON to be communicating using SSL.  This isn’t releated to the Domino keyfile (although it could be) and since this is only for communication between your Node server and your Domino Proton task and never for client-facing traffic you could use entirely internally-generated keys since they only need to be shared with the Node server itself.

HCL have kindly provided scripts to generate all the certificates you need for your testing.

Finally we need to create a design catalog for Proton to use.  You can add individual databases to the design catalog and the first one you add actually creates the catalog.  There must be a catalog with at least one database in it for Proton to work at all.

The catalog contains an index of all the design elements in a Domino database so to add a new database to the catalog you would type:
load updall <database> -e

This isn’t dynamically maintained though, so if you change the design of a database you must update its entry in the catalog if you want to have new design elements added or updated, like this:
load updall <database path> -d

The purpose of the catalog is to speed up DQL’s access to the Domino data.  It’s not required that every database be catalogued but obviously doing so speeds up access and opens up things like view scanning using the <‘View or folder name’>.<Columnname> syntax.

Proton

So that’s my very quick admin guide to what I did that enabled Tim to do what he does. It’s very possible (even probable) that this entire blog will be obsolete when the GA release ships but hopefully this and Tim’s blog help you get started with the early release.

Perfect10 – Building A Test Lab

In the 10th edition of my Perfect 10 webcast I explain how and why to build a test lab so you can get to deploying those products you downloaded today. You did download them right?

I was asked recently to share the slides for all these Perfect10 presentations and to be honest I hadn’t thought of it but it’s a good idea so I’ll be sharing them all this week.

Next up: Let’s Talk Domino v10 & Admin

11 mins

All the 10s, Let’s Make Things Simple..

Announced at the Domino 10 launch today: if you have let your licensing for Domino lapse you can renew it until the end of the year saving up to 50%, and if you have licensing and need more then IBM will discount new licensing by up to 20% until the end of the year.  I don’t sell licenses but that seems like a good deal to me.  Why should you? Well, why shouldn’t you?  If you have Domino in your environment even with a lapsed license you are keeping those servers around because of all the data that’s on them – don’t you want to use that data?  Let’s talk about what you get if you are licensed:

  • access to your Domino data using Node and modern javascript development tools
  • a new query language for Domino (DQL) which allows you to access Domino data from external platforms (buh-bye ODBC!)
  • access to your Notes applications (yes even the really really old ones) on an iPad
  • entitlement to not only use instant messaging in your Notes and web clients but also on mobile devices
  • a ton of TCO features including a new 256GB db size limit, auto database repair, cluster symmetry (automatically populating entire directories from one server to another and keeping them in sync), publishing of stats to New Relic and other cloud based reporting tools, new full text indexing engine, …

I’m not mentioning a ton of other features too.  I’m giving credit to the team who did such a great job today and especially Luis Gurigay whose presentation was seamless and showed how your back-end Domino applications work with no required code changes on an iPad, how your Domino data and web apps can be integrated into O365 and (this which he showed live and I took a screenshot) of showing a Notes document being created and then published in Slack, Microsoft Teams and Watson Workspace concurrently.

The code for this will be made available via the IBM Destination Domino site tomorrow.

Screenshot 2018-10-09 at 11.05.53

So Domino 10 is out tomorrow.  The beta for the application development pack which includes the Node module is due out this week, and the beta for Notes on the iPad is due out this month.  If you want to sign up for the betas go to the destination domino site and if you want to talk licensing and you don’t have someone at IBM to talk to let me know and I’ll see who I can point you at (and then step away because I do not want to do licensing :-))

Building a stack in Node.js

[Update: Since I posted this article, I have been informed that the domino-db node integration will be available in beta with Domino10 in a week’s time!]

With Domino 10 nearly upon us, and the Node integration hopefully following soon after, I thought I would talk about building a full-stack application in Node.js, covering how modern JavaScript UI frameworks can be built on top of Node.js and integrated with Domino in the background as a datastore.

This is all part of the IBM and HCL strategy of having Node.js as a parallel development platform alongside the standard Domino development tools, with Node providing a way for web developers to extend existing Domino apps and datastores. However, you don’t have to wait for the Domino node module to start learning about this.

If you consider the development stack acronyms of DEAN and DERN (or NERD), the UI framework is the ‘A’ and the ‘R”. These initials refer to Angular and React respectively, but generally apply to any JavaScript framework, and there are many very good ones.

The main advantage of a development stack is that the UI layer can be independent of the middleware, server, and datastore layers and so you can replace or modify the UI without impacting the rest of the architecture. As an example, you might want to do this to extend an existing web app onto a mobile platform which may require a different UI.

A review of popular frameworks and their features and advantages is beyond the scope if this post, and I may return to that later, but for now I would like to get into the broader topic of how this all fits together, i.e. how does a JavaScript UI sit ‘on top of’ Node?

The first thing to understand is that UI frameworks such as Angular, React, et al, are nothing to do with Node.js. They are not part of Node.js and do not require Node.js to work. They all run perfectly happily on any web server, including Domino. When we use a UI framework with Node, Node is essentially acting as a web server, serving the framework web pages to a browser which then loads the pages and runs the framework code.

You can run framework components inside a regular Domino web form on a Domino server, but the advantage of using the JavaScript development stack is two-fold. First, the stack is all JavaScript, so it makes it easy to talk between layers because all the data is JSON. Second, we are opening up Domino to a new development arena, with an established community, support resources and third-party products.

JavaScript UI frameworks come in all sizes and flavours, but they all work essentially the same way. You embed some code in your HTML web pages and then call the framework library (usually a js file) to enable the framework within these pages. The key thing is that everything is held in the usual web files and folders which are stored on the web server file system. Your server (whether it is Domino, Node, or Apache) simply serves up these files when a browser hits it.

In my earlier post, I talked about how you can use Express to provide middleware routes in your Node server. Here is an example which uses a static route to serve up the contents of the ‘public’ folder on the server:

const express = require('express');
const app = express();
app.use(express.static(__dirname + '/public'));
const hostname = '127.0.0.1';
const port = 3000;
app.listen(port, hostname, () => {
   console.log(`Server running at http://${hostname}:${port}/`);
});

By default, the server opens the index.html file in that folder. Luckily, this is the default filename used in most UI frameworks, so it is very easy to put your framework web app and all its files and folders in the public folder and run it from your website. The folder structure could look something like this:

node
|   server.js
|   package.json
|
+---node-modules
|   \---(node stuff...)
|
+---public
|      index.html
|      other framework files...
|      +---framework folders...
|      \---etc

The Node server runs from the server.js (using modules such as Express which are installed under the node-modules folder), and serves up the framework UI starting with the index.html in the public folder.

So we have a nice UI running on our node server. Now we need to connect this UI to our data backend. In a Domino web form we would typically make a call to a web agent which would return whatever Domino data we need. Here in Node we do pretty much the same thing, specifically our UI will make a call to an API to get the data.

Node is great at two important roles – being a web server and being an API gateway. So we can add middleware routes to our Node server to handle these API calls.

Let’s suppose we want to get a list of people to display on-screen. In our UI framework we will make a call to an API on the same Node server, something like this:

let uri = myDomain + "/api/people";
request.get( uri ).then( displayTheResults );

So this request would hit our Node server with the route “/api/people”. Currently, our server doesn’t know what to do with this, so let’s add in a new Express route to handle it.

app.get('/api/people', (req, res) => {
   let myResults = doLookup();
   res.json( myResults );
})

This will catch requests coming in to “myDomain.com/api/people”, and then our middleware code does the lookup and sends the results back in the response as JSON.

Notice how these methods handle all the JSON stuff for us, making it easy to pass the data back and forth.

This is all we have to do to to get our server responding to the API call. Now we need to look at how we get data from the backend, i.e. what happens in our doLookup().

If we suppose we are going to be using the upcoming domino-db module, then we can use its methods to run queries on Domino data. The details of exactly how the domino-db module works are still under NDA right now, but it could be something like this:

function doLookup() {
const { domServer } = require('domino-db');
   domServer(serverConfig).then(
      async (server) => {
         const db = await server.useDatabase(dbConfig);
         const docs = await db.bulkReadDocuments(DQLquery);
         return docs;
   });
} 

This example function would run a Domino Query Language query on a Domino database which returns a JSON array of document objects, i.e. ‘docs’.

We pass this back as the return value of our doLookup function and this will be sent out as the response from our API route.

Back in our front-end framework UI, we receive this JSON data in our ‘request.get’ call and we can then go ahead and call our ‘displayTheResults’ function.

This really is pretty much all there is to it. We can easily get data, in a standard JSON format, all the way from the datastore to the UI front-end, without needing fiddly data manipulation and all in only a few lines of code.

Also, what is great is that the UI framework is separate from the Express routes and these are separate from the datastore. They can all be on different servers, and we can even have different UIs accessing the same API routes to get at the same data, for example separate web apps and mobile apps.

I hope this gives you an idea of how we will be able to about go about building a full-stack application in a DEAN/DERN environment. In my next blog I plan to expand on how we can use Express to build useful routes to do all sort of things, such as performing CRUD operations and incorporating business logic.