The story of Async in JavaScript

By Tim Davis – Director of Development

In my last post I talked about some Javascript concepts that will be useful when starting out with Node.js. This time I would like to talk about a potentially awkward part of JavaScript, i.e. asynchronous (async) operations. It is a bit of a long story, but it does have a happy ending.

So what is an asynchronous operation? Basically, it means a function or command that goes off and does its own thing while the rest of the code continues. It can be really useful or really annoying depending on the circumstances.

You may have used async code if you ever did AJAX calls to Domino web agents for lookups on web pages. The rest of the page loads while the lookup to the web agent comes back, and the user is happy because part of the page updates in the background. This is brilliant and is the classic use case for an async function.

This asynchronous behaviour is built into JavaScript through and through and you need to bear it in mind when you do any programming in Node.

So how does this async behaviour manifest itself? Lets look at an example. Suppose we have an asynchronous function that goes and does a lookup somewhere.

function doAsyncLookup() {
    ... do the lookup ...
    console.log("got data");
}

Then suppose we call this function from our main code, something like this:

console.log("start");
doAsyncLookup();
console.log("finish");

The output will be this:

start
finish
got data

By the time the lookup has completed it is too late, the code has moved on.

So how do you handle something like this? How can you possibly control your processes if things finish on their own?

The original way JavaScript async functions allowed you to handle this was with ‘callbacks’.

A callback is a function that the async function calls when it is finished. So instead of your code continuing after the async function is called, it continues inside the async function.

In our example a callback could look something like this:

function myCallback() {
    console.log("finish");
}

console.log("start");
doAsyncLookup( myCallback );

Now, the output would be this:

start
got data
finish

This is much better. Usually, the callback function receives the results of the async function as a parameter, so it can act on those results. So in examples of callbacks around the web, you might see something like:

function myCallback( myResults ) { 
    displayResults( myResults );
    console.log("finish"); 
} 

console.log("start"); 
doAsyncLookup( myCallback );

Often the callback function doesn’t need to be defined separately and is defined inside the async function itself as a sort of shorthand, so you will probably see a lot of examples looking like this:

console.log("start"); 
doAsyncLookup( function ( myResults ) { 
    displayResults( myResults ); 
    console.log("finish"); 
} );

This is all great, but the problem with callbacks is that you can easily get a confusing chain of callbacks within callbacks within callbacks if you want to do other asynchronous stuff with the results.

For example, suppose you do a lookup to get a list, then want to look up something else for each item in the list, and then maybe update a record based on that lookup, and finally write updates to the screen in a UI framework. In a JavaScript environment it is highly likely that each of these operations is asynchronous. You end up with a confusing chain of functions calling functions calling functions stretching off to the right, with all the attendant risk of coding errors that you would expect:

console.log("start"); 
doAsyncLookup( function ( myResults ) { 
    lookupItemDetails( myResults, function ( myDetails ) {
        saveDetails( myDetails, function ( saveStatus ) {
            updateUIDisplay( saveStatus, function ( updatedOK ) {
                console.log("finish");
            } );
        } );
    } );    
} );

It gets even worse if you add in error handling. We may have solved the async problem, but at the penalty of terrible code patterns.

Well, after putting up with this for a while the JavaScript world came up with a better version of callbacks, called Promises.

Promises are much more readable than callbacks and have some useful additional features. You pass the results of each function to the next with a ‘then’, and you can just add more ‘thens’ on the end if you have more async things to do.

Our nightmare-indented example above becomes something like this (here I am using the popular arrow notation for functions, see my previous article for more on them):

console.log("start"); 
doAsyncLookup()
.then( (myResults) => { return lookupItemDetails(myResults) } )
.then( (myDetails) => { return saveDetails(myDetails) } )
.then( (saveStatus) => { return updateUIDisplay(saveStatus) } )
.then( (updatedOK) => { console.log("finish") } );

This is much nicer. We don’t have all that ugly nesting.

Error handling is easier, too, because you can add a ‘catch’ to the end (or in the middle if you need) and it is all still much more clear and understandable:

console.log("start"); 
doAsyncLookup() 
.then( (myResults) => { return lookupItemDetails(myResults) } ) 
.then( (myDetails) => { return saveDetails(myDetails) } ) 
.then( (saveStatus) => { return updateUIDisplay(saveStatus) } ) 
.then( (updatedOK) => { console.log("finish") } )
.catch( (err) => { ... handle err ... } );

What is really neat is that you can create your own promises from existing callbacks, so you can tidy up any older messy async functions.

Promises also have some great added features which help with other async problems. For example, with ‘Promises.all’ you can force a list of async calls to be made in order.

So promises solved the callback nesting problem, but The Gods of JavaScript were still not satisfied.

Even with all these improvements, this code is still too ‘asynchronous’. It is still a chain of function after function and you have to pay attention to what is passed from one to the next, and remember that these are all asynchronous and be careful with your error handling.

Once upon a time, Willy Wonka gave us ‘square sweets that look round’, and so now TGOJ have given us ‘asynchronous functions that look synchronous’.

The latest and greatest advance in async handling is Async/Await.

All you need to do is make your main function ‘async’, and you can ‘await’ all your promises:

async function myAsyncStuff() {
    console.log("start"); 
    let myResults = await doAsyncLookup();
    let myDetails = await lookupItemDetails(myResults);
    let saveStatus = await saveDetails(myDetails);
    let updatedOK = await updateUIDisplay(saveStatus); 
    console.log("finish");
 }

How cool is this? Each asynchronous function runs in order, with no messy callbacks or chains of ‘thens’. They all sit in their own line of code just like regular functions. You can do things in between them, and you can wrap them in the usual try/catch error handling blocks. All the async handling stuff is gone, and this is done with just two little keywords.

Plus, the functions are all still promises, so you can do promise-y things with them if you want to, and you can create and ‘await’ your own promises to refactor and revive old callback code.

Async/Await is fully supported by Node.js, by popular UI frameworks like Angular and React, and by all modern browsers.

One of the biggest headaches in JavaScript development now has an elegant and usable solution and they all lived happily ever after.

I hope you enjoyed this little story. I told you it had a happy ending.

A Solution For Time Tracking

I’ve always struggled with tracking time.  It’s partly because of my work where I’ll often leap between three or four things at once “oh compact is running, whilst that continues I’ll just do this..” and partly because I disappear down a rabbit hole and forget to “stop” whatever timer I start.  I  have had various time tracking tools integrated to my Mac where a key press starts a timer and another stops it.  It’s the stopping that’s the problem. Tim is much more diligent and carefully logs all his time in a tiny moleskine (analog ftw!) and our friend Mark Myers has long advocated for the pomodoro method where time is broken out into 25 minute chunks.  In fact Mark’s approach to work most closely aligns to mine in that he is often juggling multiple phone calls and pieces of development at the same time and so when he showed me his new Zei tool from Timeular I was convinced enough that we bought one for Tim.  Within a week of him getting it I decided to get one for myself.

Long story short.. it’s the first time tracking tool that I find easy to use and fits into how I work.  The Zei is actually an octohedran (or a D8 for those of you that way inclined) that fits nicely into your hand, you simply label each side,  tell the app what each side is labelled for and then turn that side “face up” when you start work on that project.  The bluetooth connection to my Mac or phone detects when I turn another side face up and stops the timer on the previous project, starting it on the new one.  There are apps for just about every platform and the reporting is really nice.  The cost is 49 Euro and then 9 Euro a month for unimited projects but we both spent 99 Euro as a one off cost that limits you to 8 projects on the device which suits us fine.  You can (and we do) erase and rewrite the sides of the device regularly (well every few weeks) as work changes.

You can see on my timeular below that I’m currently writing a blog – the terrible scrawl is mine and I have erased / written over badly a few times.  I have left one side entirely blank so if I’m doing “nothing” such as shopping for books online I just turn that face up and I’m not logging time.  You can add more activities in the actual app but I like having the 7 most important ones to track.  When you turn a side face up the app detects that and stops the previous timer, startimg a new one for the new face up side but if you turn it again in less than a minute it doesn’t log anything at all.  The battery on the Zei is meant to last 10,000 hours

IMG_0623

This a sample report from the app on my phone showing activities logged.  I’ve obviously  badly replaced the actual ones with sample ones so I can share it here.  You can see how well it tracks even small amounts of time.  If I’m answering emails I wouldn’t usually start a timer but I can easily turn the timeular face up and it will log all the small increments of time that usually disappear.  For my own reference even on fixed price projects I find it incredibly useful.

IMG_2347

As an additional bonus I know Mark has spoken to them several times about features he’d like and they are very keen to work with customers. So..

a beautifully built product

from a small company that cares about their solution

at a good price point

that actually works

I can’t say better than that http://timeular.com

My Collabsphere 2018 video goes live: “What is Node.js?”

By Tim Davis – Director of Development

Last month, I presented a session at Collabsphere 2018 called ‘What is Node.js?’

In it I gave an introduction to Node and covered how to set it up and create a simple web server. I also talked about how Domino 10 will integrate with it, and about some cool new features of JavaScript you may not be aware of.

Luckily my session was recorded and the video is now available on the YouTube Collabsphere channel.

The slides from this session are also available on slideshare.

If you are interested in learning about Node.js (especially with the npm integration coming up in Domino 10) then its worth a look.

Many thanks to Richard Moy and the Collabsphere team for putting on such a great show!

File Repair v10 #Domino10 #Domino2025

If you follow this blog you know that v10 of Domino, Sametime, Verse on Premises, Traveler etc are all due out this year and I want to do some – very short – blog pieces talking about new features and what my use case would be for them.

So let’s start with FILE REPAIR (or whatever it’s going to be called)

The File Repair feature for Domino v10 is designed to auto repair any corrupted databases in a cluster. Should Domino detect a corruption on any of its databases that are clustered, it automatically removes the corrupted instance and pulls a new instance from a good cluster mate. Best of all this happens super fast, doesn’t use regular replication to repopulate, doesn’t require downtime and the cluster manager is fully aware of the database availability throughout.

I can think of plenty of instances where I have had a corrupted database that I can’t replace or fix without server downtime.  No more, and another good reason to cluster your servers.

 

Definitely different – a few days looking into the future with HCL (and IBM)

If this blog is tl:dr then here’s your takeaway

I can’t thank everyone at HCL enough for throwing open the doors and leaving them open. Together we will continue to innovate great things for customers

Last week Tim and I were invited to the 1st CWP Factory tour held by HCL at their offices in Chelmsford.  “CWP” stands for “Collaboration Workflow Platform” and includes not only the products HCL took over from IBM late last year such as Domino, Traveler, Verse on Premises and Sametime but also new products that HCL are developing as extensions of those.  These (that I can talk a little bit about) such as HCL Nomad (Notes for iPad) and HCL Places (a new client runnvetaing against Domino 10 and providing integrated collaborative services such as chat, AV , web and Notes applications) will be leapfrogging Domino far over its competitors.

I want to start by thanking HCL for inviting us inside to see their process. We met and made our voices heard with more than 30 developers and executives, all of who wanted to know “do you like this?”  “what are we missing?”.  I came away from the two days with a to-do list of my own at the request of various people to send in more details of problems or requirements I had mentioned when there.  John Paganetti, who is also a customer advocate at HCL, hosted the “ask the developers” impromptu session (we had so many questions so they threw one into the agenda on day 2).  We were told to get to know and reach out directly to the teams with our feedback and questions.  If you don’t have a route to provide feedback and want one then please reach out.

Back in February I attended a Domino Jam hosted by Andrew Manby (@andrewmanby) from IBM in London.  These were held all over the world and attendees were pushed to brainstorm around features that were missing or needed.  That feedback was used to create priorities for v10 and many of the features requested at my session and others have appeared in the current beta and are committed to a v10 release.  At the end of the 2nd day of the factory tour we again had a Domino Jam hosted by Andrew Manby but this time for Domino 11 features – wheeeeeeee!   With the Jams and the Destination Domino blog as well as the #domino2025 hashtag activity, IBM are really stepping up to the products in a way they haven’t in several years.  I want to recognise the hard work being done by Andrew, by Uffe Sorensen, and by Mat Newman amongst others, to make this IBM/HCL relationship work.

So what was the factory tour? It was a 2 day conference held at HCL’s (still being built) offices. I am pleased to say it was put together very informally, we were split into groups of about 10 (hi Daniel, Francie, Julian, Richard, Paul, Nathan, Devin, Fabrice!) and one by one the development teams came and took our feedback on the work they are doing.  We worked with the Verse (on premises) team, the TCO team (looking at the Domino and Sametime servers), the Notes client team, the Nomad team and the Application Development team.  It was an intense day in a good way with so much information being shared with us and questions being asked of us.  It was also good to be told that the majority of what we saw and discussed could be shared publicly.

A few highlights (out of many) from the two days that were new to me:

  • The new database repair and folder sync features in Domino 10 (shame on me for not remembering what they are called). The database repair feature will detect when a database is corrupted and replace it whilst the server is running with a new instance from a working cluster mate (another good reason to cluster).  The folder sync feature will keep any  Domino database files or NLOs in any listed folders in sync.  This stuff is so cool and exactly what Domino clustering needed so we asked for them to extend the sync feature to include any files in the HTML directory such as HTML CSS and CGI scripts and they are considering that (v10 is a tight delivery timeline right now so no guarantees of anything).
  • Some very candid discussions (I think repeated multiple times by everyone there) about getting rid of WebSphere for Sametime in the future and how to better provide Sametime services purely under Domino.
  • HCL Places looking much evolved even in the few weeks since it was first shown at Engage – this is going to be a game changer client when it comes out.
  • The Domino General Query Facility (DGQF) available in Domino 10 is the biggest investment in Notes/Domino code in 10 years. A query language accessible outside Domino that doesn’t require any  knowledge of Domino design by a developer.  Using DGQF you can rapidly query collections of documents represented by any criteria not necessarily views or forms.  Using DGQF a regular web developer would be able to build a Node application, for instance, using back-end Domino data without ever having to learn the structure of the Domino database or touch Domino Designer.  Here’s a sneaky picture I took of the positioning for DGQF.John Curtis who is the lead designer behind DGQF has been very responsive on twitter to questions about how it will work (@john_d_curtis)IMG_0313
  • A lot of stuff Nomad and Node related which is still NDA but you’ll hear more about them at Collabsphere in Ann Arbor – HCL will be out in force as will IBM speaking, showing and listening so if you can you need to get yourself there.   Turn out and turn up – there’s still time to get your voice heard.

 

Watson Workspace Design Event in Dublin

Here’s a good thing.. an interesting (free) 2 day event being held by IBM in Dublin on Workspace development and IBM Design Thinking.  If Watson Workspace is something you are working with or interested in they will have designers and developers on hand to help you work through ideas.  Unfortunately I will be in the US then or I’d definitely be signed up.

Take a look and register here