Adapt Service without Web API

Maciej Laskowski
1.2.0 1.3.0

Overview

Hello Knot.x users!

In this post we will show you how easy it is to inject data coming directly from a database into an HTML template. When developing advanced systems on the Web, we are often asked to integrate some external services and use the data our clients provide to render some information on a page. It is not a rare case when the data source we integrate with has no Web API or even can't have it because of security reasons. This is the case we will study over the course of this tutorial.

What you're going to learn:

  • How to implement a simple Service Adapter and start using it with Knot.x.
  • How to use Vert.x to easily access your database in a very performant way.

Solution Architecture

So, we have a data source but no Web API to integrate with at the front-end layer.

We have two options now:

  1. Implement a Web API layer to access the database and then integrate with it using e.g. AJAX or an HTTP adapter.

  2. Implement a Knot.x Service Adapter.

Option (1) may be quite expensive to implement or even not possible due to security reasons. In this article, we will focus on option (2) and omit additional Web API layer. We are going to connect to the database directly from Knot.x and inject the data into an HTML template.

The architecture of our system will look like this:

Solution architecture

Prerequisites

You will need working Knot.x Stack in order to deploy module created in this tutorial. Please see Getting Started with Knot.x Stack tutorial in order to setup your Knot.x Stack.

Data and page template

In this example, we create a page that lists information about books and authors retrieved from a database. Page markup will look like following snippet:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Knot.x</title>
  <link href="https://bootswatch.com/superhero/bootstrap.min.css" rel="stylesheet"/>
</head>
<body>
<div class="container-fluid">
  <div class="row">
    <div class="col-sm-12">
      <div class="panel panel-default">
        <div class="panel-heading">Books list</div>
        <div class="panel-body">
          This section lists books from the database.
        </div>
      </div>
    </div>
  </div>
  <div class="row">
    <!-- list all books here -->
  </div>

  <div class="row">
    <div class="col-sm-12">
      <div class="panel panel-default">
        <div class="panel-heading">Authors list</div>
        <div class="panel-body">
          This section lists authors from the database.
        </div>
      </div>
    </div>
  </div>
  <div class="row">
    <!-- list all authors here -->
  </div>
</div>
</body>
</html>

Create file db-books.html in your stack instance under: content/db-books.html.

Set up the project

We will show you how to create a custom adapter project using Maven archetype - feel free to use any other favourite project build tool. To build and run this tutorial code you need Java 8 and Maven.

Follow the instructions from here to create a project structure for a custom adapter (archetype knotx-adapter-archetype). You can set the requested parameters to whatever you like, but we used these in tutorial:

  1. groupId: io.knotx.tutorial
  2. artifactId: custom-service-adapter
  3. version: 1.3.0
  4. package name: io.knotx.tutorial
  5. project name: First custom service adapter

Finally the command you use to setup project may look like this:

mvn archetype:generate -DarchetypeGroupId=io.knotx -DarchetypeArtifactId=knotx-adapter-archetype -DarchetypeVersion=1.3.0 -DgroupId=io.knotx.tutorial -DartifactId=custom-service-adapter -Dversion=1.3.0 -DpackageName=io.knotx.tutorial -DprojectName="First custom service adapter"

Created pom.xml file will have dependencies on knotx-dependencies, knotx-core and knotx-adapter-common. knotx-dependencies is Bill of Materials pattern that delivers all dependencies and versions required to build Knot.x modules. Additionally, we will use also vertx-jdbc-client and hsqldb driver. The <dependencies> section of your project's pom.xml should contain the following dependencies:

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.knotx</groupId>
        <artifactId>knotx-dependencies</artifactId>
        <version>${knotx.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>io.knotx</groupId>
      <artifactId>knotx-core</artifactId>
    </dependency>
    <dependency>
      <groupId>io.knotx</groupId>
      <artifactId>knotx-adapter-common</artifactId>
    </dependency>
    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-service-proxy</artifactId>
    </dependency>
    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-rx-java2</artifactId>
    </dependency>
    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-codegen</artifactId>
    </dependency>

    <!-- custom adapter dependencies -->
    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-jdbc-client</artifactId>
      <version>${vertx.version}</version>
    </dependency>
    <dependency>
      <groupId>org.hsqldb</groupId>
      <artifactId>hsqldb</artifactId>
      <version>2.3.4</version>
    </dependency>
    ...
  </dependencies>

Implementing the Adapter

In order to integrate with Knot.x we need to create a Verticle. The easiest way to do it is to extend the AbstractVerticle class provided by RXJava Vert.x.

The Adapter's Heart - Verticle

There is already ExampleServiceAdapter class created in /src/main/java/io/knotx/tutorial/adapter/example/ which extends AbstractVerticle:

package io.knotx.tutorial.adapter.example;

import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.serviceproxy.ServiceBinder;

public class ExampleServiceAdapter extends AbstractVerticle {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);

  private MessageConsumer<JsonObject> consumer;

  private ExampleServiceAdapterOptions configuration;

  private ServiceBinder serviceBinder;

  @Override
  public void init(Vertx vertx, Context context) {
    super.init(vertx, context);
    configuration = new ExampleServiceAdapterOptions(config());
  }

  @Override
  public void start() throws Exception {
    LOGGER.info("Starting <{}>", this.getClass().getSimpleName());

    //register the service proxy on event bus
    serviceBinder = new ServiceBinder(getVertx());
    consumer = serviceBinder
        .setAddress(configuration.getAddress())
        .register(AdapterProxy.class, new ExampleServiceAdapterProxy());
  }

  @Override
  public void stop() throws Exception {
    serviceBinder.unregister(consumer);
  }
}

Configuration

Now we will need a simple configuration for our custom code. The configuration file defines a Verticle that will initialise the whole Service Adapter and enable us to pass properties to our custom adapter.

Together witch example java code, also example configuration was generated by the archetype. Look at conf folder:

├── exampleStack.conf
└── includes
    └── exampleAdapter.conf

exampleStack.conf contains all configurations that should be added to the stack conf/application.conf. Update it, final config file (conf/application.conf) should look like:

modules = [
  "server=io.knotx.server.KnotxServerVerticle"
  "httpRepo=io.knotx.repository.http.HttpRepositoryConnectorVerticle"
  ...
  "exampleAdapter=io.knotx.tutorial.adapter.example.ExampleServiceAdapter"
]
global {
  serverPort = 8092
  ...
}

global {
  exampleAdapter {
    address = knotx.adapter.service.example
  }
}

config.exampleAdapter {
  options.config {
    include required(classpath("includes/exampleAdapter.conf"))
  }
}

Now copy includes/exampleAdapter.conf to the stack conf/includes directory. It contains configuration of our custom adapter e.g. setup of the Event Bus address to knotx.adapter.service.example.

Now we will implement a Java model to read the configuration:

package io.knotx.tutorial.adapter.example;

import io.vertx.codegen.annotations.DataObject;
import io.vertx.core.json.JsonObject;

@DataObject(generateConverter = true, publicConverter = false)
public class ExampleServiceAdapterOptions {

  private String address;

  private JsonObject clientOptions;

  public ExampleServiceAdapterOptions(JsonObject config) {
    ExampleServiceAdapterOptionsConverter.fromJson(config, this);
  }

  public String getAddress() {
    return address;
  }

  public ExampleServiceAdapterOptions setAddress(String address) {
    this.address = address;
    return this;
  }

  public JsonObject getClientOptions() {
    return clientOptions;
  }

  public ExampleServiceAdapterOptions setClientOptions(JsonObject clientOptions) {
    this.clientOptions = clientOptions;
    return this;
  }

  /**
   * Convert to JSON
   *
   * @return the JSON
   */
  public JsonObject toJson() {
    JsonObject json = new JsonObject();
    ExampleServiceAdapterOptionsConverter.toJson(this, json);
    return json;
  }
}

Here we use Vert.x codegen to generate Data Object which is a simple POJO following several rules (e.g. public fluent setters). Thanks to this annotation src/main/generated/io/knotx/tutorial/adapter/example/ExampleServiceAdapterOptionsConverter.java will be generated during next build.

Registering a Service Proxy

The next step would be to register an AdapterProxy to handle incoming requests. The simplest way to achieve this is to create a class that extends AbstractAdapterProxy. We have it already created in /src/main/java/io/knotx/tutorial/adapter/example/. It is called ExampleServiceAdapterProxy.

package io.knotx.tutorial.adapter.example;

import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;

public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);

  @Override
  protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
    final String message = adapterRequest.getParams().getString("message");
    LOGGER.info("Processing request with message: `{}`", message);
    /**
     * In a real scenario, one would connect to an external service here
     */
    return prepareResponse(message);
  }

  private Single<AdapterResponse> prepareResponse(String message) {
    final AdapterResponse response = new AdapterResponse();
    final ClientResponse clientResponse = new ClientResponse();
    clientResponse.setBody(Buffer.buffer("{\"message\":\"" + message + "\"}"));
    response.setResponse(clientResponse);
    return Single.just(response);
  }

}

Now we should register this AdapterProxy in the start() method of our ExampleServiceAdapter and set it up with the following configuration:

package io.knotx.tutorial.adapter.example;

import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.reactivex.ext.jdbc.JDBCClient;
import io.vertx.serviceproxy.ServiceBinder;


public class ExampleServiceAdapter extends AbstractVerticle {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);

  private MessageConsumer<JsonObject> consumer;
  private ExampleServiceAdapterOptions configuration;
  private ServiceBinder serviceBinder;

  @Override
  public void init(Vertx vertx, Context context) {
    super.init(vertx, context);
    LOGGER.debug("Initializing <{}>", this.getClass().getSimpleName());
    // using config() method from AbstractVerticle we simply pass our JSON file configuration to Java model
    configuration = new ExampleServiceAdapterOptions(config());
  }

  @Override
  public void start() throws Exception {
    LOGGER.info("Starting <{}>", this.getClass().getSimpleName());

    //create JDBC Clinet here and pass it to AdapterProxy - notice using clientOptions property here
    final JDBCClient client = JDBCClient.createShared(vertx, configuration.getClientOptions());

    //register the service proxy on the event bus, notice using `getVertx()` here to obtain non-rx version of vertx
    serviceBinder = new ServiceBinder(getVertx());
    consumer = serviceBinder
        .setAddress(configuration.getAddress())
        .register(AdapterProxy.class, new ExampleServiceAdapterProxy(client));
  }

  @Override
  public void stop() throws Exception {
    // unregister adapter when no longer needed
    serviceBinder.unregister(consumer);
    LOGGER.debug("Stopped <{}>", this.getClass().getSimpleName());
  }
}

Fetching Data from the Database

Now, as we have our adapter ready, we can implement the data querying logic in ExampleServiceAdapterProxy:

package io.knotx.tutorial.adapter.example;

import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.json.JsonArray;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.ext.sql.ResultSet;
import io.vertx.reactivex.ext.jdbc.JDBCClient;

public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {

  private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);

  //we will need JDBC Client here to perform DB queries
  private final JDBCClient client;

  public ExampleServiceAdapterProxy(JDBCClient client) {
    this.client = client;
  }

  @Override
  protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
    final String query = adapterRequest.getParams().getString("query");
    LOGGER.debug("Processing request with query: `{}`", query);
    return client.rxGetConnection()
        .flatMap(
            sqlConnection -> sqlConnection.rxQuery(query)
        )
        .map(this::toAdapterResponse);
  }

  private AdapterResponse toAdapterResponse(ResultSet rs) {
    final AdapterResponse adapterResponse = new AdapterResponse();
    final ClientResponse clientResponse = new ClientResponse();
    clientResponse.setBody(Buffer.buffer(new JsonArray(rs.getRows()).encode()));
    adapterResponse.setResponse(clientResponse);
    return adapterResponse;
  }
}

What we do here is:

  • When there is a request in processRequest, the first thing we do is to get the query from the request object.
  • Then we create a Single from the previously configured JDBC Client, which gives us a SQLConnection object that will be used to perform the next operation asynchronously.
  • Next we perform a flatMap operation on the SQLConnection and execute the query.
  • The last thing to do is to perform map a ResultSet obtained from the query execution to an AdapterResponse, as required by the processRequest method's contract. To do this, we simply put all query results in the body of the ClientResponse.

Integration

We have our custom Adapter. Now it's time to integrate it with Knot.x and the database.

Set up the Database

For the purpose of demonstration, we're going to use an HSQL database in this example.

Follow this tutorial in order to set up the database. To create tables with data, use the script provided here.

When you have your database configured, update conf/includes/exampleAdapter.conf to point at the database. If you followed the tutorial and your database runs at port 9001, the configuration file should look like configuration shown below:

# Event bus address of the example Knot
address = ${global.exampleAdapter.address}

clientOptions {
  url = jdbc:hsqldb:hsql://localhost:9001/
  driver_class = org.hsqldb.jdbcDriver
}

The last thing is updating ExampleServiceAdapterTest to test custom adapter. For the clarity of this tutorial we will skip this section and simply remove this test.

Build your custom adapter using the Maven command: mvn clean package. The build should result with a file target/custom-service-adapter-1.3.0.jar.

Deploy your module

Copy target/custom-service-adapter-1.3.0.jar to the Knot.x Stack folder. The structure of your stack instance should look like:

├── bin
│   └── knotx                     // shell script used to resolve and run knotx instance
├── conf                          // contains application and logger configuration files
│   ├── application.conf          // defines all modules that Knot.x instance is running, provides configuration for Knot.x Core and global variables for other config files
│   ├── bootstrap.json            // config retriever options, defines application configuration stores (e.g. points to `application.conf` - the main configuration)
│   ├── default-cluster.xml       // basic configuration of Knot.x instance cluster
│   ├── includes                  // additional modules configuration which are included in `application.conf`
│   │   ├── actionKnot.conf
│   │   ├── hbsKnot.conf
│   │   ├── httpRepo.conf
│   │   ├── server.conf
│   │   ├── serviceAdapter.conf
│   │   └── serviceKnot.conf
│   │   └── fileSystemRepo.conf
│   │   └── exampleAdapter.conf
│   └── logback.xml          // logger configuration
├── content
│   └── db-books.html
├── knotx-stack.json         // stack descriptor, defines instance libraries and dependencies
├── lib                      // contains instance libraries and dependencies, instance classpath
│   ├── ...
│   ├── custom-service-adapter-1.3.0.jar
│   ├── ...

Plug in the Custom Adapter

All you need to do now to get the adapter up and running is to update the conf/includes/serviceKnot.conf configuration file to add new services entry:

...
services = [
  {
    name = books-listing
    address = ${global.exampleAdapter.address}
    params.query = "SELECT * FROM books"
  },
  {
    name = authors-listing
    address = ${global.exampleAdapter.address}
    params.query = "SELECT * FROM authors"
  }
]

There are two services available thanks to the above configuration:

  • books-listing which will initiate service at knotx.adapter.service.example (our Custom Adapter) with additional query parameter: SELECT * FROM books. This query selects all records from the books table.
  • authors-listing that initiates the same service but passes another query: SELECT * FROM authors which selects all records from the authors table.

Prepare the template

The last thing left for us to build is a template configuration. We want the template to display data from books-listing and authors-listing services. This can be achieved by creating a couple of simple Handlebars templates in content/db-books.html:

    <script data-knotx-knots="services,handlebars"
            data-knotx-service="books-listing"
            type="text/knotx-snippet">
            {{#each _result}}
              <div class="col-sm-4">
                <div class="card">
                  <div class="card-block">
                    <h2 class="card-title">{{this.TITLE}}</h2>
                    <h4 class="card-title">{{this.ISBN}}</h4>
                    <p class="card-text">
                      {{this.SYNOPSIS}}
                    </p>
                  </div>
                </div>
              </div>
            {{/each}}
    </script>

This tells Knot.x to call the books-listing service and make the data available in the _result scope. We iterate over _result since it is a list of all books fetched from the database.

    <script data-knotx-knots="services,handlebars"
            data-knotx-service="authors-listing"
            type="text/knotx-snippet">
            {{#each _result}}
              <div class="col-sm-4">
                <div class="card">
                  <div class="card-block">
                    <h2 class="card-title">{{this.NAME}}</h2>
                    <h4 class="card-title">{{this.AGE}}</h4>
                  </div>
                </div>
              </div>
            {{/each}}
    </script>

This makes Knot.x call the authors-listing service and expose the data in the _result scope. We iterate over the entries in _result since it is a list of all authors fetched from the database.

Run the example

Now we have all the parts ready and can run the demo. The application directory should now contain the following artifacts:

You can run the Knot.x instance using the following command from the stack folder:

bin/knotx run-knotx

When you visit the page http://localhost:8092/content/local/db-books.html, you will see books and authors from the database listed. Now, when you add new books to database just refresh the page - new records will be visible immediately with no additional configuration.