Overview
Hello Knot.x users!
In this post we will show you how easy it is to inject data coming directly from a database into an HTML template. When developing advanced systems on the Web, we are often asked to integrate some external services and use the data our clients provide to render some information on a page. It is not a rare case when the data source we integrate with has no Web API or even can't have it because of security reasons. This is the case we will study over the course of this tutorial.
What you're going to learn:
- How to implement a simple Service Adapter and start using it with Knot.x.
- How to use Vert.x to easily access your database in a very performant way.
Solution Architecture
So, we have a data source but no Web API to integrate with at the front-end layer.
We have two options now:
Implement a Web API layer to access the database and then integrate with it using e.g. AJAX or an HTTP adapter.
Implement a Knot.x Service Adapter.
Option (1) may be quite expensive to implement or even not possible due to security reasons. In this article, we will focus on option (2) and omit additional Web API layer. We are going to connect to the database directly from Knot.x and inject the data into an HTML template.
The architecture of our system will look like this:
Prerequisites
You will need working Knot.x Stack in order to deploy module created in this tutorial. Please see Getting Started with Knot.x Stack tutorial in order to setup your Knot.x Stack.
Data and page template
In this example, we create a page that lists information about books and authors retrieved from a database. Page markup will look like following snippet:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Knot.x</title>
<link href="https://bootswatch.com/superhero/bootstrap.min.css" rel="stylesheet"/>
</head>
<body>
<div class="container-fluid">
<div class="row">
<div class="col-sm-12">
<div class="panel panel-default">
<div class="panel-heading">Books list</div>
<div class="panel-body">
This section lists books from the database.
</div>
</div>
</div>
</div>
<div class="row">
<!-- list all books here -->
</div>
<div class="row">
<div class="col-sm-12">
<div class="panel panel-default">
<div class="panel-heading">Authors list</div>
<div class="panel-body">
This section lists authors from the database.
</div>
</div>
</div>
</div>
<div class="row">
<!-- list all authors here -->
</div>
</div>
</body>
</html>
Create file db-books.html
in your stack instance under: content/db-books.html
.
Set up the project
We will show you how to create a custom adapter project using Maven archetype - feel free to use any other favourite project build tool. To build and run this tutorial code you need Java 8 and Maven.
Follow the instructions from here
to create a project
structure for a custom adapter (archetype knotx-adapter-archetype
). You can set the requested parameters to whatever you like,
but we used these in tutorial:
- groupId:
io.knotx.tutorial
- artifactId:
custom-service-adapter
- version:
1.3.0
- package name:
io.knotx.tutorial
- project name:
First custom service adapter
Finally the command you use to setup project may look like this:
mvn archetype:generate -DarchetypeGroupId=io.knotx -DarchetypeArtifactId=knotx-adapter-archetype -DarchetypeVersion=1.3.0 -DgroupId=io.knotx.tutorial -DartifactId=custom-service-adapter -Dversion=1.3.0 -DpackageName=io.knotx.tutorial -DprojectName="First custom service adapter"
Created pom.xml
file will have dependencies on knotx-dependencies
, knotx-core
and knotx-adapter-common
.
knotx-dependencies
is Bill of Materials pattern that delivers all dependencies and versions
required to build Knot.x modules.
Additionally, we will use also vertx-jdbc-client
and
hsqldb
driver. The <dependencies>
section of your project's pom.xml
should contain the following dependencies:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.knotx</groupId>
<artifactId>knotx-dependencies</artifactId>
<version>${knotx.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>io.knotx</groupId>
<artifactId>knotx-core</artifactId>
</dependency>
<dependency>
<groupId>io.knotx</groupId>
<artifactId>knotx-adapter-common</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-service-proxy</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-rx-java2</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-codegen</artifactId>
</dependency>
<!-- custom adapter dependencies -->
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-jdbc-client</artifactId>
<version>${vertx.version}</version>
</dependency>
<dependency>
<groupId>org.hsqldb</groupId>
<artifactId>hsqldb</artifactId>
<version>2.3.4</version>
</dependency>
...
</dependencies>
Implementing the Adapter
In order to integrate with Knot.x we need to create a Verticle.
The easiest way to do it is to extend the AbstractVerticle
class provided by RXJava Vert.x.
The Adapter's Heart - Verticle
There is already ExampleServiceAdapter class created in /src/main/java/io/knotx/tutorial/adapter/example/
which extends AbstractVerticle
:
package io.knotx.tutorial.adapter.example;
import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.serviceproxy.ServiceBinder;
public class ExampleServiceAdapter extends AbstractVerticle {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);
private MessageConsumer<JsonObject> consumer;
private ExampleServiceAdapterOptions configuration;
private ServiceBinder serviceBinder;
@Override
public void init(Vertx vertx, Context context) {
super.init(vertx, context);
configuration = new ExampleServiceAdapterOptions(config());
}
@Override
public void start() throws Exception {
LOGGER.info("Starting <{}>", this.getClass().getSimpleName());
//register the service proxy on event bus
serviceBinder = new ServiceBinder(getVertx());
consumer = serviceBinder
.setAddress(configuration.getAddress())
.register(AdapterProxy.class, new ExampleServiceAdapterProxy());
}
@Override
public void stop() throws Exception {
serviceBinder.unregister(consumer);
}
}
Configuration
Now we will need a simple configuration for our custom code. The configuration file defines a Verticle that will initialise the whole Service Adapter and enable us to pass properties to our custom adapter.
Together witch example java code, also example configuration was generated by the archetype. Look at conf
folder:
├── exampleStack.conf
└── includes
└── exampleAdapter.conf
exampleStack.conf
contains all configurations that should be added to the stack conf/application.conf
.
Update it, final config file (conf/application.conf
) should look like:
modules = [
"server=io.knotx.server.KnotxServerVerticle"
"httpRepo=io.knotx.repository.http.HttpRepositoryConnectorVerticle"
...
"exampleAdapter=io.knotx.tutorial.adapter.example.ExampleServiceAdapter"
]
global {
serverPort = 8092
...
}
global {
exampleAdapter {
address = knotx.adapter.service.example
}
}
config.exampleAdapter {
options.config {
include required(classpath("includes/exampleAdapter.conf"))
}
}
Now copy includes/exampleAdapter.conf
to the stack conf/includes
directory.
It contains configuration of our custom adapter e.g. setup of the Event Bus address to knotx.adapter.service.example
.
Now we will implement a Java model to read the configuration:
package io.knotx.tutorial.adapter.example;
import io.vertx.codegen.annotations.DataObject;
import io.vertx.core.json.JsonObject;
@DataObject(generateConverter = true, publicConverter = false)
public class ExampleServiceAdapterOptions {
private String address;
private JsonObject clientOptions;
public ExampleServiceAdapterOptions(JsonObject config) {
ExampleServiceAdapterOptionsConverter.fromJson(config, this);
}
public String getAddress() {
return address;
}
public ExampleServiceAdapterOptions setAddress(String address) {
this.address = address;
return this;
}
public JsonObject getClientOptions() {
return clientOptions;
}
public ExampleServiceAdapterOptions setClientOptions(JsonObject clientOptions) {
this.clientOptions = clientOptions;
return this;
}
/**
* Convert to JSON
*
* @return the JSON
*/
public JsonObject toJson() {
JsonObject json = new JsonObject();
ExampleServiceAdapterOptionsConverter.toJson(this, json);
return json;
}
}
Here we use Vert.x codegen to
generate Data Object which is
a simple POJO following several rules (e.g. public fluent setters).
Thanks to this annotation src/main/generated/io/knotx/tutorial/adapter/example/ExampleServiceAdapterOptionsConverter.java
will
be generated during next build.
Registering a Service Proxy
The next step would be to register an AdapterProxy
to handle incoming requests. The simplest way to achieve this is to create a class
that extends AbstractAdapterProxy
.
We have it already created in /src/main/java/io/knotx/tutorial/adapter/example/
. It is called ExampleServiceAdapterProxy.
package io.knotx.tutorial.adapter.example;
import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);
@Override
protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
final String message = adapterRequest.getParams().getString("message");
LOGGER.info("Processing request with message: `{}`", message);
/**
* In a real scenario, one would connect to an external service here
*/
return prepareResponse(message);
}
private Single<AdapterResponse> prepareResponse(String message) {
final AdapterResponse response = new AdapterResponse();
final ClientResponse clientResponse = new ClientResponse();
clientResponse.setBody(Buffer.buffer("{\"message\":\"" + message + "\"}"));
response.setResponse(clientResponse);
return Single.just(response);
}
}
Now we should register this AdapterProxy
in the start()
method of our ExampleServiceAdapter
and set it up
with the following configuration:
package io.knotx.tutorial.adapter.example;
import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.reactivex.ext.jdbc.JDBCClient;
import io.vertx.serviceproxy.ServiceBinder;
public class ExampleServiceAdapter extends AbstractVerticle {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);
private MessageConsumer<JsonObject> consumer;
private ExampleServiceAdapterOptions configuration;
private ServiceBinder serviceBinder;
@Override
public void init(Vertx vertx, Context context) {
super.init(vertx, context);
LOGGER.debug("Initializing <{}>", this.getClass().getSimpleName());
// using config() method from AbstractVerticle we simply pass our JSON file configuration to Java model
configuration = new ExampleServiceAdapterOptions(config());
}
@Override
public void start() throws Exception {
LOGGER.info("Starting <{}>", this.getClass().getSimpleName());
//create JDBC Clinet here and pass it to AdapterProxy - notice using clientOptions property here
final JDBCClient client = JDBCClient.createShared(vertx, configuration.getClientOptions());
//register the service proxy on the event bus, notice using `getVertx()` here to obtain non-rx version of vertx
serviceBinder = new ServiceBinder(getVertx());
consumer = serviceBinder
.setAddress(configuration.getAddress())
.register(AdapterProxy.class, new ExampleServiceAdapterProxy(client));
}
@Override
public void stop() throws Exception {
// unregister adapter when no longer needed
serviceBinder.unregister(consumer);
LOGGER.debug("Stopped <{}>", this.getClass().getSimpleName());
}
}
Fetching Data from the Database
Now, as we have our adapter ready, we can implement the data querying logic in ExampleServiceAdapterProxy
:
package io.knotx.tutorial.adapter.example;
import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.json.JsonArray;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.ext.sql.ResultSet;
import io.vertx.reactivex.ext.jdbc.JDBCClient;
public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);
//we will need JDBC Client here to perform DB queries
private final JDBCClient client;
public ExampleServiceAdapterProxy(JDBCClient client) {
this.client = client;
}
@Override
protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
final String query = adapterRequest.getParams().getString("query");
LOGGER.debug("Processing request with query: `{}`", query);
return client.rxGetConnection()
.flatMap(
sqlConnection -> sqlConnection.rxQuery(query)
)
.map(this::toAdapterResponse);
}
private AdapterResponse toAdapterResponse(ResultSet rs) {
final AdapterResponse adapterResponse = new AdapterResponse();
final ClientResponse clientResponse = new ClientResponse();
clientResponse.setBody(Buffer.buffer(new JsonArray(rs.getRows()).encode()));
adapterResponse.setResponse(clientResponse);
return adapterResponse;
}
}
What we do here is:
- When there is a request in
processRequest
, the first thing we do is to get thequery
from the request object. - Then we create a
Single
from the previously configured JDBC Client, which gives us aSQLConnection
object that will be used to perform the next operation asynchronously. - Next we perform a
flatMap
operation on theSQLConnection
and execute the query. - The last thing to do is to perform
map
aResultSet
obtained from the query execution to anAdapterResponse
, as required by theprocessRequest
method's contract. To do this, we simply put all query results in the body of theClientResponse
.
Integration
We have our custom Adapter. Now it's time to integrate it with Knot.x and the database.
Set up the Database
For the purpose of demonstration, we're going to use an HSQL database in this example.
Follow this tutorial
in order to set up the database.
To create tables with data, use the script provided here
.
When you have your database configured, update conf/includes/exampleAdapter.conf
to point at the database.
If you followed the tutorial and your database runs at port 9001
, the configuration
file should look like configuration shown below:
# Event bus address of the example Knot
address = ${global.exampleAdapter.address}
clientOptions {
url = jdbc:hsqldb:hsql://localhost:9001/
driver_class = org.hsqldb.jdbcDriver
}
The last thing is updating ExampleServiceAdapterTest
to test custom adapter.
For the clarity of this tutorial we will skip this section and simply remove this test.
Build your custom adapter using
the Maven command: mvn clean package
. The build should result with a file target/custom-service-adapter-1.3.0.jar
.
Deploy your module
Copy target/custom-service-adapter-1.3.0.jar
to the Knot.x Stack folder.
The structure of your stack instance should look like:
├── bin
│ └── knotx // shell script used to resolve and run knotx instance
├── conf // contains application and logger configuration files
│ ├── application.conf // defines all modules that Knot.x instance is running, provides configuration for Knot.x Core and global variables for other config files
│ ├── bootstrap.json // config retriever options, defines application configuration stores (e.g. points to `application.conf` - the main configuration)
│ ├── default-cluster.xml // basic configuration of Knot.x instance cluster
│ ├── includes // additional modules configuration which are included in `application.conf`
│ │ ├── actionKnot.conf
│ │ ├── hbsKnot.conf
│ │ ├── httpRepo.conf
│ │ ├── server.conf
│ │ ├── serviceAdapter.conf
│ │ └── serviceKnot.conf
│ │ └── fileSystemRepo.conf
│ │ └── exampleAdapter.conf
│ └── logback.xml // logger configuration
├── content
│ └── db-books.html
├── knotx-stack.json // stack descriptor, defines instance libraries and dependencies
├── lib // contains instance libraries and dependencies, instance classpath
│ ├── ...
│ ├── custom-service-adapter-1.3.0.jar
│ ├── ...
Plug in the Custom Adapter
All you need to do now to get the adapter up and running is to update
the conf/includes/serviceKnot.conf
configuration file to add new services
entry:
...
services = [
{
name = books-listing
address = ${global.exampleAdapter.address}
params.query = "SELECT * FROM books"
},
{
name = authors-listing
address = ${global.exampleAdapter.address}
params.query = "SELECT * FROM authors"
}
]
There are two services available thanks to the above configuration:
books-listing
which will initiate service atknotx.adapter.service.example
(our Custom Adapter) with additionalquery
parameter:SELECT * FROM books
. This query selects all records from thebooks
table.authors-listing
that initiates the same service but passes another query:SELECT * FROM authors
which selects all records from theauthors
table.
Prepare the template
The last thing left for us to build is a template configuration. We want the template to display data from books-listing
and
authors-listing
services. This can be achieved by creating a couple of simple
Handlebars templates in content/db-books.html
:
<script data-knotx-knots="services,handlebars"
data-knotx-service="books-listing"
type="text/knotx-snippet"></script>
This tells Knot.x to call the books-listing
service and make the data available in the _result
scope.
We iterate over _result
since it is a list of all books fetched from the database.
<script data-knotx-knots="services,handlebars"
data-knotx-service="authors-listing"
type="text/knotx-snippet"></script>
This makes Knot.x call the authors-listing
service and expose the data in the _result
scope.
We iterate over the entries in _result
since it is a list of all authors fetched from the database.
Run the example
Now we have all the parts ready and can run the demo. The application directory should now contain the following artifacts:
You can run the Knot.x instance using the following command from the stack folder:
bin/knotx run-knotx
When you visit the page http://localhost:8092/content/local/db-books.html, you will see books and authors from the database listed. Now, when you add new books to database just refresh the page - new records will be visible immediately with no additional configuration.