Overview
Hello Knot.x users!
In this post we will show you how easy it is to inject data coming directly from a database into an HTML template. When developing advanced systems on the Web, we are often asked to integrate some external services and use the data our clients provide to render some information on a page. It is not a rare case when the data source we integrate with has no Web API or even can't have it because of security reasons. This is the case we will study over the course of this tutorial.
What you're going to learn:
- How to implement a simple Service Adapter and start using it with Knot.x.
- How to use Vert.x to easily access your database in a very performant way.
If you want to skip the configuration part and simply run the demo, please checkout
github/adapt-service-without-webapi
and follow the instructions in README.md
to compile and run the complete code.
Solution Architecture
So, we have a data source but no Web API to integrate with at the front-end layer.
We have two options now:
Implement a Web API layer to access the database and then integrate with it using e.g. AJAX or an HTTP adapter.
Implement a Knot.x Service Adapter.
Option (1) may be quite expensive to implement or even not possible due to security reasons. In this article, we will focus on option (2) and omit additional Web API layer. We are going to connect to the database directly from Knot.x and inject the data into an HTML template.
The architecture of our system will look like this:
Data and page template
In this example, we create a page that lists information about books and authors retrieved from a database. Page markup will look like following snippet:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Knot.x</title>
<link href="https://bootswatch.com/superhero/bootstrap.min.css" rel="stylesheet"/>
</head>
<body>
<div class="container-fluid">
<div class="row">
<div class="col-sm-12">
<div class="panel panel-default">
<div class="panel-heading">Books list</div>
<div class="panel-body">
This section lists books from the database.
</div>
</div>
</div>
</div>
<div class="row">
<!-- list all books here -->
</div>
<div class="row">
<div class="col-sm-12">
<div class="panel panel-default">
<div class="panel-heading">Authors list</div>
<div class="panel-body">
This section lists authors from the database.
</div>
</div>
</div>
</div>
<div class="row">
<!-- list all authors here -->
</div>
</div>
</body>
</html>
Set up the project
We will show you how to create a custom adapter project using Maven archetype - feel free to use any other favourite project build tool. To build and run this tutorial code you need Java 8 and Maven.
Follow the instructions from here
to create a project
structure for a custom adapter (archetype knotx-adapter-archetype
). You can set the requested parameters to whatever you like,
but we used these in tutorial:
- groupId:
io.knotx.tutorial
- artifactId:
custom-service-adapter
- version:
1.2.0
- package name:
io.knotx.tutorial
- project name:
First custom service adapter
Finally the command you use to setup project may look like this:
mvn archetype:generate -DarchetypeGroupId=io.knotx.archetypes -DarchetypeArtifactId=knotx-adapter-archetype -DarchetypeVersion=1.2.0 -DgroupId=io.knotx.tutorial -DartifactId=custom-service-adapter -Dversion=1.2.0 -DpackageName=io.knotx.tutorial -DprojectName="First custom service adapter"
Created pom.xml
file will have dependencies on knotx-core
and knotx-adapter-common
with scope
set to provided
. This is because we will have those dependencies on the classpath provided by knotx-standalone-1.2.0.fat.jar
(there are also other dependencies, but for the purpose of this exercise we need only those two).
Additionally, we will use also vertx-jdbc-client
and
hsqldb
driver. The <dependencies>
section of your project's pom.xml
should contain the following dependencies:
<dependencies>
<dependency>
<groupId>io.knotx</groupId>
<artifactId>knotx-core</artifactId>
<version>${knotx.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.knotx</groupId>
<artifactId>knotx-adapter-common</artifactId>
<version>${knotx.version}</version>
<scope>provided</scope>
</dependency>
<!-- custom adapter dependencies -->
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-jdbc-client</artifactId>
<version>${vertx.version}</version>
</dependency>
<dependency>
<groupId>org.hsqldb</groupId>
<artifactId>hsqldb</artifactId>
<version>2.3.4</version>
</dependency>
...
</dependencies>
You may simply download a ready pom.xml
file from the tutorial codebase.
Implementing the Adapter
In order to integrate with Knot.x we need to create a Verticle.
The easiest way to do it is to extend the AbstractVerticle
class provided by RXJava Vert.x.
The Adapter's Heart - Verticle
There is already ExampleServiceAdapter class created in /src/main/java/io/knotx/tutorial/adapter/example/
which extends AbstractVerticle
:
package io.knotx.tutorial.adapter.example;
import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.serviceproxy.ServiceBinder;
public class ExampleServiceAdapter extends AbstractVerticle {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);
private MessageConsumer<JsonObject> consumer;
private ExampleServiceAdapterConfiguration configuration;
private ServiceBinder serviceBinder;
@Override
public void init(Vertx vertx, Context context) {
super.init(vertx, context);
configuration = new ExampleServiceAdapterConfiguration(config());
}
@Override
public void start() throws Exception {
LOGGER.info("Starting <{}>", this.getClass().getSimpleName());
//register the service proxy on event bus
serviceBinder = new ServiceBinder(getVertx());
consumer = serviceBinder
.setAddress(configuration.getAddress())
.register(AdapterProxy.class, new ExampleServiceAdapterProxy());
}
@Override
public void stop() throws Exception {
serviceBinder.unregister(consumer);
}
}
Configuration
Now we will need a simple configuration for our custom code. The configuration file defines a Verticle that will initialise the whole Service Adapter and enable us to pass properties to our custom adapter.
This configuration file named io.knotx.tutorial.adapter.example.ExampleServiceAdapter.json
already exists
in /src/main/resources/
:
{
"main": "io.knotx.tutorial.adapter.example.ExampleServiceAdapter",
"options": {
"config": {
"address": "knotx.adapter.service.example",
"params": {
"message": "Hello Knot.x"
}
}
}
}
This configuration file is prepared to run the custom Service Adapter, starting the
io.knotx.tutorial.adapter.example.ExampleServiceAdapter
Verticle and listening at the address knotx.adapter.service.example
on the event bus.
Now we will implement a Java model to read the configuration:
package io.knotx.tutorial.adapter.example;
import io.vertx.core.json.JsonObject;
public class ExampleServiceAdapterConfiguration {
private String address;
private JsonObject clientOptions;
public ExampleServiceAdapterConfiguration(JsonObject config) {
address = config.getString("address");
clientOptions = config.getJsonObject("clientOptions", new JsonObject());
}
public JsonObject getClientOptions() {
return clientOptions;
}
public String getAddress() {
return address;
}
}
Registering a Service Proxy
The next step would be to register an AdapterProxy
to handle incoming requests. The simplest way to achieve this is to create a class
that extends AbstractAdapterProxy
.
We have it already created in /src/main/java/io/knotx/tutorial/adapter/example/
. It is called ExampleServiceAdapterProxy.
package io.knotx.tutorial.adapter.example;
import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);
@Override
protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
final String message = adapterRequest.getParams().getString("message");
LOGGER.info("Processing request with message: `{}`", message);
/**
* In a real scenario, one would connect to an external service here
*/
return prepareResponse(message);
}
private Single<AdapterResponse> prepareResponse(String message) {
final AdapterResponse response = new AdapterResponse();
final ClientResponse clientResponse = new ClientResponse();
clientResponse.setBody(Buffer.buffer("{\"message\":\"" + message + "\"}"));
response.setResponse(clientResponse);
return Single.just(response);
}
}
Now we should register this AdapterProxy
in the start()
method of our ExampleServiceAdapter
and set it up
with the following configuration:
package io.knotx.tutorial.adapter.example;
import io.knotx.proxy.AdapterProxy;
import io.vertx.core.Context;
import io.vertx.core.Vertx;
import io.vertx.core.eventbus.MessageConsumer;
import io.vertx.core.json.JsonObject;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.reactivex.core.AbstractVerticle;
import io.vertx.reactivex.ext.jdbc.JDBCClient;
import io.vertx.serviceproxy.ServiceBinder;
public class ExampleServiceAdapter extends AbstractVerticle {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapter.class);
private MessageConsumer<JsonObject> consumer;
private ExampleServiceAdapterConfiguration configuration;
private ServiceBinder serviceBinder;
@Override
public void init(Vertx vertx, Context context) {
super.init(vertx, context);
LOGGER.debug("Initializing <{}>", this.getClass().getSimpleName());
// using config() method from AbstractVerticle we simply pass our JSON file configuration to Java model
configuration = new ExampleServiceAdapterConfiguration(config());
}
@Override
public void start() throws Exception {
LOGGER.info("Starting <{}>", this.getClass().getSimpleName());
//create JDBC Clinet here and pass it to AdapterProxy - notice using clientOptions property here
final JDBCClient client = JDBCClient.createShared(vertx, configuration.getClientOptions());
//register the service proxy on the event bus, notice using `getVertx()` here to obtain non-rx version of vertx
serviceBinder = new ServiceBinder(getVertx());
consumer = serviceBinder
.setAddress(configuration.getAddress())
.register(AdapterProxy.class, new ExampleServiceAdapterProxy(client));
}
@Override
public void stop() throws Exception {
// unregister adapter when no longer needed
serviceBinder.unregister(consumer);
LOGGER.debug("Stopped <{}>", this.getClass().getSimpleName());
}
}
Fetching Data from the Database
Now, as we have our adapter ready, we can implement the data querying logic in ExampleServiceAdapterProxy
:
package io.knotx.tutorial.adapter.example;
import io.knotx.adapter.AbstractAdapterProxy;
import io.knotx.dataobjects.AdapterRequest;
import io.knotx.dataobjects.AdapterResponse;
import io.knotx.dataobjects.ClientResponse;
import io.reactivex.Single;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.json.JsonArray;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import io.vertx.ext.sql.ResultSet;
import io.vertx.reactivex.ext.jdbc.JDBCClient;
public class ExampleServiceAdapterProxy extends AbstractAdapterProxy {
private static final Logger LOGGER = LoggerFactory.getLogger(ExampleServiceAdapterProxy.class);
//we will need JDBC Client here to perform DB queries
private final JDBCClient client;
public ExampleServiceAdapterProxy(JDBCClient client) {
this.client = client;
}
@Override
protected Single<AdapterResponse> processRequest(AdapterRequest adapterRequest) {
final String query = adapterRequest.getParams().getString("query");
LOGGER.debug("Processing request with query: `{}`", query);
return client.rxGetConnection()
.flatMap(
sqlConnection -> sqlConnection.rxQuery(query)
)
.map(this::toAdapterResponse);
}
private AdapterResponse toAdapterResponse(ResultSet rs) {
final AdapterResponse adapterResponse = new AdapterResponse();
final ClientResponse clientResponse = new ClientResponse();
clientResponse.setBody(Buffer.buffer(new JsonArray(rs.getRows()).encode()));
adapterResponse.setResponse(clientResponse);
return adapterResponse;
}
}
What we do here is:
- When there is a request in
processRequest
, the first thing we do is to get thequery
from the request object. - Then we create a
Single
from the previously configured JDBC Client, which gives us aSQLConnection
object that will be used to perform the next operation asynchronously. - Next we perform a
flatMap
operation on theSQLConnection
and execute the query. - The last thing to do is to perform
map
aResultSet
obtained from the query execution to anAdapterResponse
, as required by theprocessRequest
method's contract. To do this, we simply put all query results in the body of theClientResponse
.
Integration
We have our custom Adapter. Now it's time to integrate it with Knot.x and the database.
Set up the Database
For the purpose of demonstration, we're going to use an HSQL database in this example.
Follow this tutorial
in order to set up the database.
To create tables with data, use the script provided in the db
folder of this tutorial.
When you have your database configured, update the clientOptions
property in io.knotx.tutorial.adapter.example.ExampleServiceAdapter.json
to point at the database. If you followed the tutorial and your database runs at port 9001
, the configuration
file should look like configuration shown below:
{
"main": "io.knotx.tutorial.adapter.example.ExampleServiceAdapter",
"options": {
"config": {
"address": "knotx.adapter.service.example",
"clientOptions": {
"url": "jdbc:hsqldb:hsql://localhost:9001/",
"driver_class": "org.hsqldb.jdbcDriver"
}
}
}
}
The last thing to do is to remove ExampleServiceAdapterTest from adapter. After that, build your custom adapter using
the Maven command: mvn clean package
. The build should result with a file called custom-service-adapter-1.2.0-fat.jar
(fat jar is a jar which contains all project class files and resources packed together with all it's dependencies)
being created in the target
directory.
Set up Knot.x
Create a folder where we will start Knot.x and the custom Adapter. It should contain the following files:
├── knotx-standalone.json (e.g. download from Maven Central knotx-standalone-1.2.0.json, maven archetype creates it for us)
├── logback.xml (e.g. download from Maven Central knotx-standalone-1.2.0.logback.xml, maven archetype creates it for us)
├── app
│ ├── knotx-standalone-1.2.0.fat.jar (download from Maven Central)
├── content
│ ├── local
│ ├── books.html (Contains markup of a page - see "Data and page template" section)
You may download Knot.x files from the Maven Central Repository
Plug in the Custom Adapter
All you need to do now to get the adapter up and running is to copy custom-service-adapter-1.2.0-fat.jar
to the app
directory and update the knotx-standalone-1.2.0.json
configuration file to add new services
:
{
"modules": [
"knotx:io.knotx.KnotxServer",
"knotx:io.knotx.FilesystemRepositoryConnector",
"knotx:io.knotx.FragmentSplitter",
"knotx:io.knotx.FragmentAssembler",
"knotx:io.knotx.ServiceKnot",
"knotx:io.knotx.HandlebarsKnot",
"knotx:io.knotx.tutorial.adapter.example.ExampleServiceAdapter"
],
"config": {
"knotx:io.knotx.ServiceKnot": {
"options": {
"config": {
"services": [
{
"name": "books-listing",
"address": "knotx.adapter.service.example",
"params": {
"query": "SELECT * FROM books"
}
},
{
"name": "authors-listing",
"address": "knotx.adapter.service.example",
"params": {
"query": "SELECT * FROM authors"
}
}
]
}
}
}
}
}
There are two services available thanks to the above configuration:
books-listing
which will initiate service atknotx.adapter.service.example
(our Custom Adapter) with additionalquery
parameter:SELECT * FROM books
. This query selects all records from thebooks
table.authors-listing
that initiates the same service but passes another query:SELECT * FROM authors
which selects all records from theauthors
table.
Prepare the template
The last thing left for us to build is a template configuration. We want the template to display data from books-listing
and
authors-listing
services. This can be achieved by creating a couple of simple
Handlebars templates in books.html
:
<script data-knotx-knots="services,handlebars"
data-knotx-service="books-listing"
type="text/knotx-snippet"></script>
This tells Knot.x to call the books-listing
service and make the data available in the _result
scope.
We iterate over _result
since it is a list of all books fetched from the database.
<script data-knotx-knots="services,handlebars"
data-knotx-service="authors-listing"
type="text/knotx-snippet"></script>
This makes Knot.x call the authors-listing
service and expose the data in the _result
scope.
We iterate over the entries in _result
since it is a list of all authors fetched from the database.
The final markup of the template can be downloaded from our GitHub repository for this tutorial.
Run the example
Now we have all the parts ready and can run the demo. The application directory should now contain the following artifacts:
├── knotx-standalone.json
├── knotx-standalone.logback.xml
├── app
│ ├── custom-service-adapter-1.2.0-fat.jar
│ ├── knotx-standalone-1.2.0.fat.jar
├── content
│ ├── local
│ ├── books.html
You can run the Knot.x instance using the following command:
java -Dlogback.configurationFile=knotx-standalone.logback.xml -cp "app/*" io.knotx.launcher.LogbackLauncher -conf knotx-standalone.json
When you visit the page http://localhost:8092/content/local/books.html, you will see books and authors from the database listed. Now, when you add new books to database just refresh the page - new records will be visible immediately with no additional configuration.
The complete code of this whole tutorial is available in the Knot.x tutorials GitHub repository.