Debugging Next.js with GoogleBot

Debugging a contemporary JavaScript application can be somewhat of a black art. The asynchronous nature of the language and implementation bring on race conditions and the likes. These have been the mainstay of JS developers since the beginning of time and are well documented.

Next.js is a capable Universal JavaScript framework that builds on the React view library. Getting started with Next.js is very easy and writing a simple hello world with GraphQL or other similar backend is quite straightforward. The groundwork for building Universal JavaScript applications is here.

Universally rendered JavaScript, where the code renders both on the server and the client, takes the complexity of development up a notch or two. Some capabilities like the DOM are simply not available on the server side Node.js JavaScript Runtime, but this is easy to spot during development.

When you go into more dynamic components, like Bing or Google Maps components which only run on the browser, you can use a simple trick to determine wheter the request is rendered by the server or the browser. Below is an example which will work as a dynamic definition based on existance of the window object:

const isBrowser = typeof window !== "undefined";

With this approach you can simply do some if-else logic in your JSX markup, leaving out components that are not intended to be rendered on the server as shown below using the ternary operator:

<div>
  {isBrowser ? (
    <RichMapComponent />
  ) : null}
</div>

This works well, but you will probably run into more surprises when you start deploying your Universal JavaScript that is also executed by search engine crawlers capable of executing JS, but where you have no access to debugging.

Debugging GoogleBot JavaScript errors

In 2017 GoogleBot is running a dated version (~41) of their Chrome browser for rendering views. In general you'll want to keep polyfills in place to help Chrome 41.0 transpile to ES5 JavaScript and other older technologies such as Async/Await or Fetch will need polyfills for optimal search engine visibility.

An unexpected error has occurred.

The above is a generic 500 error generated by Next.js by default in production mode. If the application works on a modern browser (Chrome, Firefox, Edge, Firefox…) and you can view the site with a text browser like Lynx, then the issue is how the GoogleBot Chrome version handles JavaScript.

In production mode you will have no access to debug, so if you want to get more information on the error, you can run the Next.js application in the dev mode (e.g. npm run dev) in a public facing environment. You can then hit your application with either the Google Webmaster Tools' "Fetch as Google" feature. Alternatively and easier way is to use the Page Speed Insights tool which is publicly available and you won't need to verify your temporary application.

In development mode you might be able to get useful debug information for some cases from your production environment, but if the issue lies on the client side support of JavaScript by GoogleBot - this becomes a whole lot trickier. You won't have easy access to the native error console of the headless browser that GoogleBot essentially nowadays is. You can still try some methods to get live JavaScript debugging information from crawlers or other clients whose console you don't have access to.

Google itself recommends using the global onerror handler to catch errors in browsers and then logging them either on screen or relaying them forward to a third party. Below you can see an example code snippet that enables logging of errors from GoogleBot:

window.addEventListener('error', function(e) {
    var errorText = [
        e.message,
        'URL: ' + e.filename,
        'Line: ' + e.lineno + ', Column: ' + e.colno,
        'Stack: ' + (e.error && e.error.stack || '(no stack trace)')
    ].join('
');

    // Example: log errors as visual output into the host page.
    // Note: you probably don’t want to show such errors to users, or
    //       have the errors get indexed by Googlebot; however, it may
    //       be a useful feature while actively debugging the page.
    var DOM_ID = 'rendering-debug-pre';
    if (!document.getElementById(DOM_ID)) {
        var log = document.createElement('pre');
        log.id = DOM_ID;
        log.style.whiteSpace = 'pre-wrap';
        log.textContent = errorText;
        if (!document.body) document.body = document.createElement('body');
        document.body.insertBefore(log, document.body.firstChild);
    } else {
        document.getElementById(DOM_ID).textContent += '

' + errorText;
    }

    // Example: log the error to remote service.
    // Note: you can log errors to a remote service, to understand
    //       and monitor the types of errors encountered by regular users,
    //       Googlebot, and other crawlers.
    var client = new XMLHttpRequest();
    client.open('POST', 'https://example.com/logError');
    client.setRequestHeader('Content-Type', 'text/plain;charset=UTF-8');
    client.send(errorText);

});

You'll need to tailor the script to your own needs, but at least it provides a way of debugging errors in Search Engines Robots' JavaScript console.

-- Jani Tarvainen, 14/11/2017