Adapt Service without Web API

Maciej Laskowski Knot.x 1.2.0

Overview

Hello Knot.x users!

In this post we will show you how easy it is to inject data coming directly from a database into an HTML template. When developing advanced systems on the Web, we are often asked to integrate some external services and use the data our clients provide to render some information on a page. It is not a rare case when the data source we integrate with has no Web API or even can't have it because of security reasons. This is the case we will study over the course of this tutorial.

What you're going to learn:

  • How to implement a simple Service Adapter and start using it with Knot.x.
  • How to use Vert.x to easily access your database in a very performant way.

If you want to skip the configuration part and simply run the demo, please checkout github/adapt-service-without-webapi and follow the instructions in README.md to compile and run the complete code.

Solution Architecture

So, we have a data source but no Web API to integrate with at the front-end layer.

We have two options now:

  1. Implement a Web API layer to access the database and then integrate with it using e.g. AJAX or an HTTP adapter.

  2. Implement a Knot.x Service Adapter.

Option (1) may be quite expensive to implement or even not possible due to security reasons. In this article, we will focus on option (2) and omit additional Web API layer. We are going to connect to the database directly from Knot.x and inject the data into an HTML template.

The architecture of our system will look like this:

Solution architecture

Data and page template

In this example, we create a page that lists information about books and authors retrieved from a database. Page markup will look like following snippet:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Knot.x</title>
  <link href="https://bootswatch.com/superhero/bootstrap.min.css" rel="stylesheet"/>
</head>
<body>
<div class="container-fluid">
  <div class="row">
    <div class="col-sm-12">
      <div class="panel panel-default">
        <div class="panel-heading">Books list</div>
        <div class="panel-body">
          This section lists books from the database.
        </div>
      </div>
    </div>
  </div>
  <div class="row">
    <!-- list all books here -->
  </div>

  <div class="row">
    <div class="col-sm-12">
      <div class="panel panel-default">
        <div class="panel-heading">Authors list</div>
        <div class="panel-body">
          This section lists authors from the database.
        </div>
      </div>
    </div>
  </div>
  <div class="row">
    <!-- list all authors here -->
  </div>
</div>
</body>
</html>

Set up the project

We will show you how to create a custom adapter project using Maven archetype - feel free to use any other favourite project build tool. To build and run this tutorial code you need Java 8 and Maven.

Follow the instructions from here to create a project structure for a custom adapter (archetype knotx-adapter-archetype). You can set the requested parameters to whatever you like, but we used these in tutorial:

  1. groupId: io.knotx.tutorial
  2. artifactId: custom-service-adapter
  3. version: 1.2.0
  4. package name: io.knotx.tutorial
  5. project name: First custom service adapter

Finally the command you use to setup project may look like this:

mvn archetype:generate -DarchetypeGroupId=io.knotx.archetypes -DarchetypeArtifactId=knotx-adapter-archetype -DarchetypeVersion=1.2.0 -DgroupId=io.knotx.tutorial -DartifactId=custom-service-adapter -Dversion=1.2.0 -DpackageName=io.knotx.tutorial -DprojectName="First custom service adapter"

Created pom.xml file will have dependencies on knotx-core and knotx-adapter-common with scope set to provided. This is because we will have those dependencies on the classpath provided by knotx-standalone-1.2.0.fat.jar (there are also other dependencies, but for the purpose of this exercise we need only those two). Additionally, we will use also vertx-jdbc-client and hsqldb driver. The <dependencies> section of your project's pom.xml should contain the following dependencies:

  <dependencies>
    <dependency>
      <groupId>io.knotx</groupId>
      <artifactId>knotx-core</artifactId>
      <version>${knotx.version}</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>io.knotx</groupId>
      <artifactId>knotx-adapter-common</artifactId>
      <version>${knotx.version}</version>
      <scope>provided</scope>
    </dependency>

    <!-- custom adapter dependencies -->
    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-jdbc-client</artifactId>
      <version>${vertx.version}</version>
    </dependency>
    <dependency>
      <groupId>org.hsqldb</groupId>
      <artifactId>hsqldb</artifactId>
      <version>2.3.4</version>
    </dependency>
    ...
  </dependencies>

You may simply download a ready pom.xml file from the tutorial codebase.

Implementing the Adapter

In order to integrate with Knot.x we need to create a Verticle. The easiest way to do it is to extend the AbstractVerticle class provided by RXJava Vert.x.

The Adapter's Heart - Verticle

There is already ExampleServiceAdapter class created in /src/main/java/io/knotx/tutorial/adapter/example/ which extends AbstractVerticle:

package io.knotx.tutorial.adapter.example;

import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.serviceproxy.ServiceBinder;


public class ExampleServiceAdapter extends AbstractVerticle {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);

  private MessageConsumer<JsonObject> consumer;

  private ExampleServiceAdapterConfiguration configuration;

  private ServiceBinder serviceBinder;

  @Override
  public void init(Vertx vertx, Context context) {
    super.init(vertx, context);
    configuration = new ExampleServiceAdapterConfiguration(config());
  }

  @Override
  public void start() throws Exception {
    LOGGER.info("Starting <{}>", this.getClass().getSimpleName());

    //register the service proxy on event bus
    serviceBinder = new ServiceBinder(getVertx());
    consumer = serviceBinder
        .setAddress(configuration.getAddress())
        .register(AdapterProxy.class, new ExampleServiceAdapterProxy());
  }

  @Override
  public void stop() throws Exception {
    serviceBinder.unregister(consumer);
  }
}

Configuration

Now we will need a simple configuration for our custom code. The configuration file defines a Verticle that will initialise the whole Service Adapter and enable us to pass properties to our custom adapter.

This configuration file named io.knotx.tutorial.adapter.example.ExampleServiceAdapter.json already exists in /src/main/resources/:

{
  "main": "io.knotx.tutorial.adapter.example.ExampleServiceAdapter",
  "options": {
    "config": {
      "address": "knotx.adapter.service.example",
      "params": {
        "message": "Hello Knot.x"
      }
    }
  }
}

This configuration file is prepared to run the custom Service Adapter, starting the io.knotx.tutorial.adapter.example.ExampleServiceAdapter Verticle and listening at the address knotx.adapter.service.example on the event bus.

Now we will implement a Java model to read the configuration:

package io.knotx.tutorial.adapter.example;

import io.vertx.core.json.JsonObject;

public class ExampleServiceAdapterConfiguration {

  private String address;

  private JsonObject clientOptions;

  public ExampleServiceAdapterConfiguration(JsonObject config) {
    address = config.getString("address");
    clientOptions = config.getJsonObject("clientOptions", new JsonObject());
  }

  public JsonObject getClientOptions() {
    return clientOptions;
  }

  public String getAddress() {
    return address;
  }
}

Registering a Service Proxy

The next step would be to register an AdapterProxy to handle incoming requests. The simplest way to achieve this is to create a class that extends AbstractAdapterProxy. We have it already created in /src/main/java/io/knotx/tutorial/adapter/example/. It is called ExampleServiceAdapterProxy.

package io.knotx.tutorial.adapter.example;

import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;

public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);

  @Override
  protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
    final String message = adapterRequest.getParams().getString("message");
    LOGGER.info("Processing request with message: `{}`", message);
    /**
     * In a real scenario, one would connect to an external service here
     */
    return prepareResponse(message);
  }

  private Single<AdapterResponse> prepareResponse(String message) {
    final AdapterResponse response = new AdapterResponse();
    final ClientResponse clientResponse = new ClientResponse();
    clientResponse.setBody(Buffer.buffer("{\"message\":\"" + message + "\"}"));
    response.setResponse(clientResponse);
    return Single.just(response);
  }

}

Now we should register this AdapterProxy in the start() method of our ExampleServiceAdapter and set it up with the following configuration:

package io.knotx.tutorial.adapter.example;

import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.reactivex.ext.jdbc.JDBCClient;
import io.vertx.serviceproxy.ServiceBinder;


public class ExampleServiceAdapter extends AbstractVerticle {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);

  private MessageConsumer<JsonObject> consumer;
  private ExampleServiceAdapterConfiguration configuration;
  private ServiceBinder serviceBinder;

  @Override
  public void init(Vertx vertx, Context context) {
    super.init(vertx, context);
    LOGGER.debug("Initializing <{}>", this.getClass().getSimpleName());
    // using config() method from AbstractVerticle we simply pass our JSON file configuration to Java model
    configuration = new ExampleServiceAdapterConfiguration(config());
  }

  @Override
  public void start() throws Exception {
    LOGGER.info("Starting <{}>", this.getClass().getSimpleName());

    //create JDBC Clinet here and pass it to AdapterProxy - notice using clientOptions property here
    final JDBCClient client = JDBCClient.createShared(vertx, configuration.getClientOptions());

    //register the service proxy on the event bus, notice using `getVertx()` here to obtain non-rx version of vertx
    serviceBinder = new ServiceBinder(getVertx());
    consumer = serviceBinder
        .setAddress(configuration.getAddress())
        .register(AdapterProxy.class, new ExampleServiceAdapterProxy(client));
  }

  @Override
  public void stop() throws Exception {
    // unregister adapter when no longer needed
    serviceBinder.unregister(consumer);
    LOGGER.debug("Stopped <{}>", this.getClass().getSimpleName());
  }
}

Fetching Data from the Database

Now, as we have our adapter ready, we can implement the data querying logic in ExampleServiceAdapterProxy:

package io.knotx.tutorial.adapter.example;

import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.json.JsonArray;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.ext.sql.ResultSet;
import io.vertx.reactivex.ext.jdbc.JDBCClient;

public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);

  //we will need JDBC Client here to perform DB queries
  private final JDBCClient client;

  public ExampleServiceAdapterProxy(JDBCClient client) {
    this.client = client;
  }

  @Override
  protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
    final String query = adapterRequest.getParams().getString("query");
    LOGGER.debug("Processing request with query: `{}`", query);
    return client.rxGetConnection()
        .flatMap(
            sqlConnection -> sqlConnection.rxQuery(query)
        )
        .map(this::toAdapterResponse);
  }

  private AdapterResponse toAdapterResponse(ResultSet rs) {
    final AdapterResponse adapterResponse = new AdapterResponse();
    final ClientResponse clientResponse = new ClientResponse();
    clientResponse.setBody(Buffer.buffer(new JsonArray(rs.getRows()).encode()));
    adapterResponse.setResponse(clientResponse);
    return adapterResponse;
  }
}

What we do here is:

  • When there is a request in processRequest, the first thing we do is to get the query from the request object.
  • Then we create a Single from the previously configured JDBC Client, which gives us a SQLConnection object that will be used to perform the next operation asynchronously.
  • Next we perform a flatMap operation on the SQLConnection and execute the query.
  • The last thing to do is to perform map a ResultSet obtained from the query execution to an AdapterResponse, as required by the processRequest method's contract. To do this, we simply put all query results in the body of the ClientResponse.

Integration

We have our custom Adapter. Now it's time to integrate it with Knot.x and the database.

Set up the Database

For the purpose of demonstration, we're going to use an HSQL database in this example.

Follow this tutorial in order to set up the database. To create tables with data, use the script provided in the db folder of this tutorial.

When you have your database configured, update the clientOptions property in io.knotx.tutorial.adapter.example.ExampleServiceAdapter.json to point at the database. If you followed the tutorial and your database runs at port 9001, the configuration file should look like configuration shown below:

{
  "main": "io.knotx.tutorial.adapter.example.ExampleServiceAdapter",
  "options": {
    "config": {
      "address": "knotx.adapter.service.example",
      "clientOptions": {
        "url": "jdbc:hsqldb:hsql://localhost:9001/",
        "driver_class": "org.hsqldb.jdbcDriver"
      }
    }
  }
}

The last thing to do is to remove ExampleServiceAdapterTest from adapter. After that, build your custom adapter using the Maven command: mvn clean package. The build should result with a file called custom-service-adapter-1.2.0-fat.jar (fat jar is a jar which contains all project class files and resources packed together with all it's dependencies) being created in the target directory.

Set up Knot.x

Create a folder where we will start Knot.x and the custom Adapter. It should contain the following files:

├── knotx-standalone.json  (e.g. download from Maven Central knotx-standalone-1.2.0.json, maven archetype creates it for us)
├── logback.xml (e.g. download from Maven Central knotx-standalone-1.2.0.logback.xml, maven archetype creates it for us)
├── app
│   ├── knotx-standalone-1.2.0.fat.jar (download from Maven Central)
├── content
│   ├── local
│       ├── books.html (Contains markup of a page - see "Data and page template" section)

You may download Knot.x files from the Maven Central Repository

  1. Knot.x standalone fat jar
  2. JSON configuration file
  3. Log configuration file

Plug in the Custom Adapter

All you need to do now to get the adapter up and running is to copy custom-service-adapter-1.2.0-fat.jar to the app directory and update the knotx-standalone-1.2.0.json configuration file to add new services:

{
  "modules": [
    "knotx:io.knotx.KnotxServer",
    "knotx:io.knotx.FilesystemRepositoryConnector",
    "knotx:io.knotx.FragmentSplitter",
    "knotx:io.knotx.FragmentAssembler",
    "knotx:io.knotx.ServiceKnot",
    "knotx:io.knotx.HandlebarsKnot",
    "knotx:io.knotx.tutorial.adapter.example.ExampleServiceAdapter"
  ],
  "config": {
    "knotx:io.knotx.ServiceKnot": {
      "options": {
        "config": {
          "services": [
            {
              "name": "books-listing",
              "address": "knotx.adapter.service.example",
              "params": {
                "query": "SELECT * FROM books"
              }
            },
            {
              "name": "authors-listing",
              "address": "knotx.adapter.service.example",
              "params": {
                "query": "SELECT * FROM authors"
              }
            }
          ]
        }
      }
    }
  }
}

There are two services available thanks to the above configuration:

  • books-listing which will initiate service at knotx.adapter.service.example (our Custom Adapter) with additional query parameter: SELECT * FROM books. This query selects all records from the books table.
  • authors-listing that initiates the same service but passes another query: SELECT * FROM authors which selects all records from the authors table.

Prepare the template

The last thing left for us to build is a template configuration. We want the template to display data from books-listing and authors-listing services. This can be achieved by creating a couple of simple Handlebars templates in books.html:

    <script data-knotx-knots="services,handlebars"
            data-knotx-service="books-listing"
            type="text/knotx-snippet">
            {{#each _result}}
              <div class="col-sm-4">
                <div class="card">
                  <div class="card-block">
                    <h2 class="card-title">{{this.TITLE}}</h2>
                    <h4 class="card-title">{{this.ISBN}}</h4>
                    <p class="card-text">
                      {{this.SYNOPSIS}}
                    </p>
                  </div>
                </div>
              </div>
            {{/each}}
    </script>

This tells Knot.x to call the books-listing service and make the data available in the _result scope. We iterate over _result since it is a list of all books fetched from the database.

    <script data-knotx-knots="services,handlebars"
            data-knotx-service="authors-listing"
            type="text/knotx-snippet">
            {{#each _result}}
              <div class="col-sm-4">
                <div class="card">
                  <div class="card-block">
                    <h2 class="card-title">{{this.NAME}}</h2>
                    <h4 class="card-title">{{this.AGE}}</h4>
                  </div>
                </div>
              </div>
            {{/each}}
    </script>

This makes Knot.x call the authors-listing service and expose the data in the _result scope. We iterate over the entries in _result since it is a list of all authors fetched from the database.

The final markup of the template can be downloaded from our GitHub repository for this tutorial.

Run the example

Now we have all the parts ready and can run the demo. The application directory should now contain the following artifacts:

├── knotx-standalone.json
├── knotx-standalone.logback.xml
├── app
│   ├── custom-service-adapter-1.2.0-fat.jar
│   ├── knotx-standalone-1.2.0.fat.jar
├── content
│   ├── local
│       ├── books.html

You can run the Knot.x instance using the following command:

java -Dlogback.configurationFile=knotx-standalone.logback.xml -cp "app/*" io.knotx.launcher.LogbackLauncher -conf knotx-standalone.json

When you visit the page http://localhost:8092/content/local/books.html, you will see books and authors from the database listed. Now, when you add new books to database just refresh the page - new records will be visible immediately with no additional configuration.

The complete code of this whole tutorial is available in the Knot.x tutorials GitHub repository.