A Connector-Based RAG System

Instead of preparing data upfront, we simply fetch it ad-hoc from Confluence using the official REST API.

1. User asks a question
2. The LLM generates search keywords from the question
3. A Java client queries Confluence via its REST-API
4. Matching pages are fetched
5. Their content becomes the RAG context

Just a connector → Confluence → the LLM. It’s a “just-in-time RAG” without any preprocessing step.

1. ConfluenceClient.java

First, we need to build our way to read Confluence. So this class will do two things:

• Perform a CQL (Confluence Query Language) search
• Fetch a page’s content (HTML), which is then stripped and cleaned to plain text

Below is a shortened version of the code. In the end, it’s just a Java client which calls a REST-API. To do so, Confluence offers the so-called personal access token.

public class ConfluenceClient {

  private final String baseUrl = "https://my-confluence.de/confluence";
  private final String pat = "PERSONAL-ACCESS-TOKEN";

  public JsonNode search(String query) {
    String cql = buildCql(query);
    String url = baseUrl + "/rest/api/search?cql=" + encode(cql);
    HttpRequest req = request(url);
    String body = send(req);
    return MAPPER.readTree(body);
  }

  public JsonNode getPage(String id) {
    var url = baseUrl + "/rest/api/content/" + id + "?expand=body.storage";
    var req = request(url);
    var body = send(req);
    return MAPPER.readTree(body);
  }

  private String buildCql(String query) {
    // Example output:
    //   space in ("MY-SPACE","YOUR-SPACE") AND type = "page" AND (text ~ "animal" OR text ~ "human")
    ...
  }

  private HttpRequest request(String url) { ... }
  private String send(HttpRequest req) { ... }
}

2. LLM Generates Search Keywords

But before we even hit Confluence, the LLM produces 3–5 high-quality keywords from the user’s question. This step avoids sending full questions with stopwords to the Confluence search API.

public String[] extractSearchKeywords(String question) {

  var systemPrompt = """
      You get a question related to our internal Confluence.
      Produce 3-5 relevant search terms.
      No stopwords, no sentences, only domain terminology.
      Return a comma-separated list.
      """;

  var response = callOpenAI(systemPrompt, question);
  return response
      .toLowerCase()
      .replaceAll("[^a-z0-9öäüß, ]", " ")
      .split("[, ]+");
}

3. Putting it all together: answerQuestion()

Now we can put it all together:

• We have implemented a simple Java client to read Confluence
• We use the LLM to convert the user’s questions to search keywords
• Now we can query Confluence and build a RAG context with its search results

public String answerQuestion(String question) throws Exception {

  // 1) Let the LLM decide which keywords to search for
  String[] keywords = extractSearchKeywords(question);
  String keywordQuery = String.join(" ", keywords);

  // 2) Search Confluence
  JsonNode results = confluence.search(keywordQuery).path("results");

  if (results.isEmpty()) {
    return "No relevant Confluence data found.";
  }

  // 3) Build context by loading up to 10 pages
  StringBuilder ctx = new StringBuilder();
  for (JsonNode r : results) {
    String id = r.path("content").path("id").asText();
    JsonNode page = confluence.getPage(id);

    String html = page.path("body").path("storage").path("value").asText();
    String text = stripHtml(html);

    ctx.append("=== ").append(page.path("title").asText()).append(" ===\n");
    ctx.append(text).append("\n\n");
  }

  // 4) Ask the LLM based on this context
  String systemPrompt = """
      You answer strictly based on the Confluence context.
      If the answer is not in the context, say:
      "The Confluence data does not contain the answer."
      """;

  String userPrompt = """
      QUESTION:
      %s

      CONTEXT:
      %s
      """.formatted(question, ctx.toString());

  return callOpenAI(systemPrompt, userPrompt);
}

4. Main class to run it

Now we can go ahead and ask a question:

public class Main {
  public static void main(String[] args) throws Exception {
    AiAgent agent = new AiAgent();
    String answer = agent.answerQuestion("What is a report?");
    System.out.println(answer);
  }
}

The Magic: No Vector Store Needed

This approach is super simple:

• The LLM itself creates the search query
• Confluence returns only relevant pages
• The pages become the RAG context
• The LLM answers based on that context

✅ Advantages

• Easy to build: No embedding pipeline, no vector database, no chunking logic.
• Always up to date: The answers always use live Confluence data.
• Minimal infrastructure: Just Java + HTTP + your LLM endpoint. Great for enterprise environments: Works even if you cannot store data locally due to compliance restrictions.

❌ Disadvantages

• You are limited by the Confluence search API: CQL quality determines response quality. If CQL can’t find it, the LLM can’t answer it.
• Potentially slow for large queries: Each request triggers multiple Confluence API calls.
• No semantic search: You rely on keyword-based text search, not embeddings.
• Rate limits: Large teams or heavy usage may hit Confluence rate limits.

When should you use this approach?

Use connector-based RAG when you want:

• fast development
• zero data preprocessing
• live data from Confluence
• minimal maintenance

Use local vector-based RAG (as in my previous post) when:

• you need semantic search
• you want to customize chunking, scoring, ranking
• you want low latency and high throughput
• you need strong control over your retrieval logic

Best regards,
Thomas

You don’t need a big platform or a six-month project to get started with AI on your internal knowledge base.

You can build a fully local, privacy-friendly Retrieval-Augmented Generation (RAG) system in just a couple of hours. No external dependencies. No cloud vector databases. No proprietary frameworks. Just a local embedding model, your internal documents, and a Large Language Model (LLM) endpoint.

This post shows you how.

Demo Project on GitHub

You can find a demo project to this topic on my GitHub account:

https://github.com/tuhrig/local-rag-java-gradle

What We Are Building

So what is RAG? RAG systems use an LLM, after retrieving the most relevant pieces of your own documents. Our documents can be anything from text files, to code or PDFs. But in a typical company, it will pretty much be Confluence. That’s where your knowledge of the last couple of years is buried. So let’s give it back some life.

In practice, this results in a system like this:

1. You ask a question
2. Your system searches through your documents
3. It picks the most relevant text chunks
4. It builds a huge context prompt including the found document parts
5. It asks the LLM
6. You get a context-grounded answer

Technically, we will go through the following steps:

                     ┌────────────────────────┐
                     │   1. Extract Content   │
                     │  (Confluence API, PDFs)│
                     └─────────────┬──────────┘
                                   ▼
                     ┌────────────────────────┐
                     │     2. Clean & Chunk   │
                     │   HTML → text → chunks │
                     └─────────────┬──────────┘
                                   ▼
            ┌──────────────────────────────────────────┐
            │         3. Embed & Store Chunks          │
            │  local embeddings → JSON vector files    │
            └────────────────────────┬─────────────────┘
                                     ▼
                     ┌────────────────────────┐
                     │   4. Similarity Search │
                     └─────────────┬──────────┘
                                   ▼
          ┌────────────────────────────────────┐
          │      5. Build Prompt & Query LLM   │
          │  (inject context → call LLM)       │
          └──────────────────────────┬─────────┘
                                     ▼
                     ┌────────────────────────┐
                     │      Final Answer      │
                     └────────────────────────┘

Step 1: Extract Your Internal Content

Most companies use something like Confluence to share their internal knowledge. Whatever you have — if it has an API or can export PDFs, you can get the content. But for Confluence in particular, it’s very convenient because of its REST-API:

GET /rest/api/content?spaceKey=ABC&limit=...&expand=body.storage

Using this endpoint, you can paginate through all pages of a space and download them as JSON. Here’s a minimal Java example that retrieves every page from a Confluence space and stores the results as individual JSON files:

var client = HttpClient.newHttpClient();
var mapper = new ObjectMapper();

var baseUrl = "https://your-confluence-domain/rest/api/content";
var spaceKey = "ABC";
int limit = 50;
int start = 0;

while (true) {
    var url = baseUrl 
            + "?spaceKey=" + spaceKey 
            + "&limit=" + limit 
            + "&start=" + start 
            + "&expand=body.storage";

    var req = HttpRequest.newBuilder()
            .uri(URI.create(url))
            .header("Authorization", "Bearer <YOUR_TOKEN>")
            .build();

    var resp = client.send(req, HttpResponse.BodyHandlers.ofString());
    var root = mapper.readTree(resp.body());
    var results = root.get("results");

    if (results == null || !results.isArray() || results.size() == 0) {
        break;
    }

    for (var page : results) {
        var id = page.get("id").asText();
        var out = new File("raw_pages/" + id + ".json");
        mapper.writerWithDefaultPrettyPrinter().writeValue(out, page);
    }

    int size = results.size();
    if (size < limit) {
        break; // no more pages
    }
    start += limit;
}

After running this script, your local download folder might look like this:

raw_pages/
  123456.json
  123457.json
  123458.json
  123459.json
  ...

Each file contains a full Confluence page in JSON format — including its ID, title, metadata, and HTML content (body.storage.value).

Step 2: Clean the HTML Content

Before we can embed the text, we should convert the HTML into clean, readable plain text:

• remove tags
• remove boilerplate (menus, macros, metadata)
• normalize whitespace
• keep only the meaningful textual content

The easiest way to do this in Java is using Jsoup, a lightweight HTML parser.

Here is a minimal snippet that takes the downloaded Confluence JSON files, extracts the HTML, and converts it to plain text:

var inputDir = new File("raw_pages");
var outputDir = new File("clean_pages");
outputDir.mkdirs();

for (var file : inputDir.listFiles((d, n) -> n.endsWith(".json"))) {
    var root = mapper.readTree(file);
    var id = root.get("id").asText();
    var body = root.path("body").path("storage").path("value");
    var html = body.isMissingNode() ? "" : body.asText();
    var cleanText = Jsoup.parse(html).text();
    cleanText = cleanText.replaceAll("\\s+", " ").trim();
    var out = new File(outputDir, id + ".txt");
    var.writeString(out.toPath(), cleanText);
}

After running this step, your directory structure looks like this:

raw_pages/
  123456.json
  123457.json
  ...

clean_pages/
  123456.txt
  123457.txt
  ...

Each .txt file now contains a clean, normalized text representation of the corresponding Confluence page. This content is ready for chunking and embedding in the following steps.

Step 3: Chunk the Documents

Big documents don’t embed well. So we split them into small pieces — for example 300–600 characters each. Every chunk gets stored locally:

{pageId}_{title}_chunk_{n}.json

To do so, we can also use some Jsoup and simple Java:

var inputDir = new File("clean_pages");   // contains 12345.txt etc.
var chunkDir = new File("chunks");
chunkDir.mkdirs();

int chunkSize = 600;
int overlap = 100;

for (var file : inputDir.listFiles((d, n) -> n.endsWith(".txt"))) {

    var pageId = file.getName().replace(".txt", "");
    var text = Files.readString(file.toPath());

    text = text.replaceAll("\\s+", " ").trim();

    int index = 0;
    int start = 0;

    while (start < text.length()) {

        int end = Math.min(start + chunkSize, text.length());
        String chunk = text.substring(start, end).trim();

        var node = mapper.createObjectNode();
        node.put("pageId", pageId);
        node.put("chunkIndex", index);
        node.put("text", chunk);

        var out = new File(chunkDir, pageId + "_chunk_" + index + ".json");

        mapper.writerWithDefaultPrettyPrinter().writeValue(out, node);

        index++;
        start = end - overlap;  // sliding window
        if (start < 0) start = 0;
    }
}

This becomes your “source library”. Each chunk_*.json file contains:

• the pageId
• the chunkIndex
• the cleaned text snippet

In the next step, we will embed these files as the foundation for your vector store.

clean_pages/
  12345.txt
  12346.txt
  ...

chunks/
  12345_chunk_0.json
  12345_chunk_1.json
  12345_chunk_2.json
  12346_chunk_0.json
  ...

Step 4: Create Embeddings Locally

To perform semantic search (and find the most relevant documents to our question), we need a numerical representation (an embedding) for each text chunk. We can easily run an embedding model locally, entirely offline. For this example, we use the lightweight and well-established all-MiniLM-L6-v2 embedding model, which is small, fast, and works great for document search. You can download a full copy of the model as a ZIP file here:

👉 https://www.kaggle.com/datasets/sircausticmail/all-minilm-l6-v2zip

After downloading, unzip it into a folder like:

D:/embedding_models/all-MiniLM-L6-v2/

Once the model is available locally, we expose it via a tiny Python Flask service. This allows any application (e.g., our Java tooling) to request embeddings via a simple HTTP call.

from flask import Flask, request, jsonify
from sentence_transformers import SentenceTransformer

app = Flask(__name__)

# Path to your downloaded model folder
MODEL_PATH = r"D:\embedding_models\all-MiniLM-L6-v2"

print("Loading model from:", MODEL_PATH)
model = SentenceTransformer(MODEL_PATH)

@app.route("/embed", methods=["POST"])
def embed():
    data = request.get_json()
    text = data.get("text", "")
    embedding = model.encode(text).tolist()
    return jsonify({"embedding": embedding})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5005)

Run the server:

python embedding_service.py

To embed all your cleaned and chunked documents, you can call the local this service from Java. This script iterates over all chunk files, sends each chunk’s text to the local embedding service, and writes the result into a corresponding .embedding.json file.

var chunkDir = new File("chunks");
var embedDir = new File("embeddings");
embedDir.mkdirs();

for (var file : chunkDir.listFiles((d, n) -> n.endsWith(".json"))) {

    var root = MAPPER.readTree(file);
    var pageId = root.get("pageId").asText();
    int chunkIndex = root.get("chunkIndex").asInt();
    var text = root.get("text").asText();

    var body = MAPPER.createObjectNode();
    body.put("text", text);

    var req = HttpRequest.newBuilder()
            .uri(URI.create("http://localhost:5005/embed"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(body.toString()))
            .build();

    var resp = CLIENT.send(req, HttpResponse.BodyHandlers.ofString());
    var embedJson = MAPPER.readTree(resp.body());

    var out = MAPPER.createObjectNode();
    out.put("pageId", pageId);
    out.put("chunkIndex", chunkIndex);
    out.set("embedding", embedJson.get("embedding"));

    var outFile = new File(embedDir, pageId + "_chunk_" + chunkIndex + ".embedding.json");

    MAPPER.writerWithDefaultPrettyPrinter().writeValue(outFile, out);
}

After running this step, your directory structure looks like this:

chunks/
  12345_chunk_0.json
  12345_chunk_1.json

embeddings/
  12345_chunk_0.embedding.json
  12345_chunk_1.embedding.json

Each embedding file will contain a large number of vectors returned from the local embedding model. Now we have created the data basis for our RAG system: We have download all documents from Confluence, we have clean and chunked them and finally we have converted them to vectors.

Step 5: Similarity Search

Once every chunk has an embedding, the next step is to find the chunks that are most relevant to the user’s question.

The process is simple:

• Embed the user’s query (using the same local embedding service)
• Compare this query embedding with all stored chunk embeddings
• Compute similarity between them
• Sort the results
• Pick the top k chunks (e.g., 10–20)

List<Double> embed(String text) {
    // call local embedding server
    var body = "{\"text\": " + MAPPER.writeValueAsString(text) + "}";
    var req = HttpRequest.newBuilder()
            .uri(URI.create("http://localhost:5005/embed"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
    var resp = CLIENT.send(req, HttpResponse.BodyHandlers.ofString());
    return MAPPER.readTree(resp.body())
                 .get("embedding").findValuesAsText(null)
                 .stream().map(Double::valueOf).toList();
}

double cosine(List<Double> a, List<Double> b) {
    double dot=0, na=0, nb=0;
    for (int i=0; i<a.size(); i++) {
        dot += a.get(i)*b.get(i);
        na  += a.get(i)*a.get(i);
        nb  += b.get(i)*b.get(i);
    }
    return dot / (Math.sqrt(na)*Math.sqrt(nb));
}

// --- Similarity Search ---
var query = embed("How does the booking logic work?");
var scores = new HashMap<>();

for (var f : new File("embeddings").listFiles()) {
    var vec = ... load embedding from JSON ...;
    scores.put(f, cosine(query, vec));
}

scores.entrySet().stream()
      .sorted((a,b)->Double.compare(b.getValue(), a.getValue()))
      .limit(5)
      .forEach(e -> System.out.println(e.getValue() + " -> " + e.getKey()));

Now we have the chunks that are semantically closest to the user’s question. They will become the input context for the LLM in the next step.

📁 Example Output

0.8421  ->  12345_chunk_1.embedding.json
0.8012  ->  12345_chunk_0.embedding.json
0.7950  ->  98765_chunk_2.embedding.json
...

Step 6: Build the Prompt and Ask the LLM

Once we have identified the most relevant chunks through similarity search, we can assemble the final RAG prompt. This prompt gives the LLM the exact pieces of information it needs to answer the user’s question — without hallucination and without relying on its general training.

The structure is simple:

You are an internal assistant.
Answer the question using only the context below.
If the information is missing, say so.

### Context
[Chunk 1]
<text>

[Chunk 2]
<text>

...

### Question
<user question>

This prompt is then sent to your LLM endpoint (e.g., Azure OpenAI or any other model you have access to). Because the LLM receives the exact, clean, and relevant context, it can generate accurate, grounded answers based on your internal documentation.

var prompt =
    "You are an internal assistant...\n\n" +
    "### Context\n" +
    topChunks.map(c -> c.text).collect(joining("\n\n")) +
    "\n\n### Question\n" +
    userQuestion;

var response = callAzureOpenAI(prompt);  // or any other LLM endpoint

System.out.println(response);

The result is a precise, context-aware answer grounded entirely in your own documents — no hallucinations, no guesswork, and no external data leakage.

Why This Works So Well

Most LLM tools focus on fancy interfaces, cloud services, and integrations. But at its core, RAG is extremely simple.

You need:

• a way to download your data (from Confluence REST-API)
• a way to clean and chunk text (Jsoup)
• a way to embed text (e.g. all-MiniLM-L6-v2)
• a way to compare embeddings
• an LLM endpoint (e.g. Azure OpenAI)

Final Thoughts

LLMs are powerful, but the real magic comes when you mix them with your own knowledge. And the best part: you don’t need a huge infrastructure to get there. A local RAG system is one of the fastest and most effective ways to bring AI into everyday work.

If you want to try it yourself, start small: Pick one space, one folder, or one project — and build your own AI assistant around it.

It’s easier than you think. And highly addictive once it works.

Also check my demo project on GitHub:

https://github.com/tuhrig/local-rag-java-gradle

Best regards,
Thomas

But not only am I leaving, Bringmeister as a company has already closed its doors. Since Saturday, the website is offline and Bringmeister is out of the race of online grocery delivery.

The Story of Bringmeister

(Source: web.archive.org)

Although Bringmeister looks like a typical start-up (and felt like!), it has a long history. It was founded in 1997 (!) by Kaisers-Tengelmann in Berlin and was operating based on paper-catalogs. You could order by phone or fax.

After Kaisers-Tengelmann closed their doors, Bringmeister went to EDEKA along with many stationary shops. This was the time I joined the team in 2018. EDEKA invested a lot of money and modernized old structures - including the software backend. The day I started, there were still laptops lying around in the office running production code. But things were about to change! EDEKA hired a lot of new people, and we re-wrote large parts of the old code base. And it was state of the art! Kotlin, Spring Boot, AWS, events - we did it the right way!

However, EDEKA was always reluctant to scale and grow the business. In my personal opinion, having a delivery service interfered with their local shops. So we were able to improve a lot of things, but we failed to grow.

Then came Corona 😷

Although this crisis was tough for everyone, from a business perspective it was a big accelerator. Orders went through the roof and the numbers looked real good. That was when EDEKA took their chance and sold Bringmeister to Rockaway Capital. For EDEKA it was the right timing, but not for Bringmeister.

Rockaway Capital promised a bright future. Growth! Scaling! New office! More people! Billion-Dollar-Company! A location in every big city! Everything sounded real good. Too good. But except of the office, nothing ever happened. After the initial welcome-pitches, nobody from Rockaway Capital was ever seen again. And a couple of months later, Bringmeister was desperately looking for a new investor to jump on board. Eventually Rockaway Capital sold Bringmeister again.

This time, Bringmeister went to Rohlik which is the owner of Knuspr - a direct competitor. And this takeover was even more unpleasant as the one before. The employees didn’t know anything about what was to happen. There was dead silence about any plans. From one day to the other, the whole C-level management was gone and Rohlik announced to layoff about 60% of the staff. A month later, the remaining employees followed and the message was through the door: Bringmeister will be closed.

Why Bringmeister failed

(Please keep in mind that I’m a software developer, not a business analyst and that the following is all my personal opinion.)

Bringmeister has always been a losing business (as many other online grocery delivery services, too). You need to invest a lot of money in order to get this kind of business started. And we actually did! Over the years, Bringmeister managed to get a positive contribution margin 1. But the scaling was missing. We never managed to get a foot into any new city or location besides of Berlin and Munich. EDEKA didn’t push it because of the competition with their local stores, Rockaway Capital didn’t push it because there was no genuine interest, and Rohlik just wanted to get rid of a competitor. After so many missed chances, it was time for a market consolidation.

Our success

The field of online grocery services is a tough battleground. Many have tried, many have failed. But Bringmeister did it as one of the pioneers for a long time. We managed to build a great web-shop, a great app and a great company behind the scenes. We made over 100 million euros of sales in 2023 and shipped over 20.000 orders per week. More than 20.000 products have been listed in our shop and we have been recognized by Stiftung Warentest as Testsieger among food delivery services. We created a state-of-the-art software backend fully on AWS with more than 50 microservices. And at least for my personal development, the last six years have been a big success.

Thank you

The last thing for me, is to say thank you to all of my old colleagues. For the vast majority of the time, Bringmeister was a great place to work and a big part of my life. I met extraordinary people, learned a lot of new things and had the personal space to grow and take ownership. I will miss those times for sure!

Read more

• https://www.supermarktblog.com/2023/08/15/auslaufmodell-bringmeister-lieferdienst-mit-angezogener-handbremse
• https://excitingcommerce.de/2024/04/18/bringmeister-wollte-2023-die-100-mio-euro-marke-knacken/

Best regards, Thomas.

• The frontend is a big blob. A single application, developed and deployed as one large package.
• Every service has its own little frontend, but they are not connected in any way.

The second was the case for our internal admin UIs at my current company. Every other service has its own little UI and everybody maintains a list of bookmarks to find things again.

This post shows a simple approach to solve this problem. You will find the following below:

• Build a simple Spring Boot app with Vaadin
• Integrate multiple (Vaadin) UIs via IFrames
• Communicate between IFrames via window.postMessage()
• Discussion of various aspects and alternative approaches

Although this example uses a certain tech-stack (Spring Boot, Vaadin, Kotlin), the shown principles are simple and technology-agnostic.

Spring Boot with Vaadin

Setting up a Spring Boot app with Vaadin is really simple. We can use the Spring Initializr at https://start.spring.io to setup the basic project skeleton.

After that, we can create a simple Vaadin view and we are almost done with the first step.

@Route("")
class SimpleVaadinView : VerticalLayout() {
    init {
        this.add(Html("<h1>Hello!</h1>"))
    }
}

However, there’s a small but important detail for the example we want to implement. Our final goal is, to have multiple Vaadin apps running in the same browser tab via IFrames. All of those apps will run under the same host (localhost). So we must ensure two things:

• ⚠️ Every Vaadin app gets its own unique port
• ⚠️ The JSESSIONID cookie must have a unique name

We can achieve both by using the application.properties:

server.port=8080
server.servlet.session.cookie.name=JSESSIONID_MY_SIMPLE_VIEW

Having a unique port is obvious as you cannot run multiple applications on the same port. Renaming the JSESSIONID is necessary, because every app has its own session. And since all apps will run in the same tab and under the same host, the cookie would be overwritten. This would result in expired sessions, because only the last JSESSIONID would be stored in the cookie. By renaming the cookie, we ensure that every app can handle its sessions correctly.

Integration via IFrames

We want to integrate multiple independent UIs on a single page. The example we want to implement looks like this:

   +=======================================================+
   |                    << Browser Tab >>         x - *    |
   +=======================================================+
   |   Main-View                                           |
   |                                                       |
   |         << IFrame >>             << IFrame >>         |
   |   +---------------------+   +---------------------+   |
   |   | Left-View           |   | Right-View          |   |
   |   |                     |   |                     |   |
   |   |                     |   |                     |   |
   |   |                     |   |                     |   |  
   |   +---------------------+   +---------------------+   |
   |                                                       |
   +=======================================================+

We want to implement three independent apps:

• A left-view which will run on port 8081
• A right-view which will run on port 8082
• And a main-view which will run on port 8080 and which will integrate the other views

For the example (which you can find on GitHub) we implement the following:

• The left-view shows a list of programming languages. The user can click on a language and select it.

• The right-view shows a short description of a programming language. The user cannot click anything. The language can only be selected using a URL parameter (like http://localhost:8082/languages/Kotlin)

• The main-view finally includes the other views via IFrames. It also provides a nice heading on top of it.

Doing this is quite simple. The main-view looks like this:

@Route("languages")
class MainVaadinView : VerticalLayout(), HasUrlParameter<String?> {

    private val heading = Html("<h1>Choose a programming language!</h1>")
    private val leftIFrame = IFrame("http://localhost:8081/languages")
    private val rightIFrame = IFrame("http://localhost:8082/languages")

    init {

        val splitLayout = SplitLayout(leftIFrame, rightIFrame)
        splitLayout.setSizeFull()

        this.add(heading)
        this.add(splitLayout)
        this.setSizeFull()
    }

    override fun setParameter(event: BeforeEvent, @OptionalParameter language: String?) {
        if (!parameter.isNullOrBlank()) {
            heading.setHtmlContent("<h1>What is ${language}?</h1>")
            leftIFrame.src = "http://localhost:8081/languages/$language"
            rightIFrame.src = "http://localhost:8082/languages/$language"
        }
    }
}

What do we have right now?

• The main-view integrates both other views via IFrames
• Depending on the URL parameter, a language is selected (e.g. /languages/Java)
• The language is passed on to the other views by setting the src of the IFrame accordingly

However, the important part is still missing: the interaction. A click on the left-view should change what the right-view is showing. How can we achieve this?

Communication between IFrames

Usually, IFrames are isolated and protected by the same-origin policy. Only code from the same origin (protocol + host + port) can interact.

However, there’s an exception: We can use window.postMessage() (see here) for cross-origin communication. If we obtain a reference to a window object, this API enables us to post a message to the window. The window in return can listen to the event and react accordingly.

The basic idea goes like this:

// The Main-View has a listener for messages
window.addEventListener("message", (event) => {
  console.log(event.data)
});

// The IFrame can post a message to its parent
window.top.postMessage("Hello there!", "*");

Based on this simple approach, we can implement a communication pattern between the IFrames:

1. The user selects something in the left IFrame. By using window.top.postMessage(...) the IFrame can send an event to its parent (the top window). Note that it is not possible to send a message directly to the other IFrame since there is no reference to this window object.
2. The parent IFrame has a window.addEventListener to listen for the event. The event has some predefined format which is the protocol between the IFrames. This can be anything, for example: {"language":"Kotlin"}.
3. The parent IFrame broadcasts the event to all of its children. It is the only place where we can obtain a reference to all window objects.
4. Every (child) IFrame can handle or ignore the event as it wants to.

As you can see, communication between IFrames only requires a bit of vanilla JavaScript. But in case of Vaadin, we need an additional step, because all logic resides on the server-side. So we must transfer the JavaScript events back to the server in order to handle them.

To do so, we can use Vaadin’s @ClientCallable annotation (see here). It lets us implement a listener method to send data from the JavaScript frontend to the Kotlin backend.

@ClientCallable
fun receiveFrontendEvent(event: String) {
    log.info("Received event from frontend: {}", event)
    ...
}

We can hook this method with some JavaScript to our window.addEventListener:

element.executeJs("""
    window.addEventListener("message", (event) => {
        ${'$'}0.${'$'}server.receiveFrontendEvent(event.data);
    });
""".trimIndent(), element)

See the example on GitHub for the complete implementation.

Discussion

• Are IFrames bad? - IFrames are just another tool in the box. They are simple, provide good isolation (especially in sandbox mode) and are designed to embed content to a page. The window.postMessage(...)-API makes communication save and easy. For simple micro-frontends with a decent amount of embedded elements, they are a good choice in my opinion.
• Any alternatives? - You can find a discussion on different approaches at martinfowler.com. Besides IFrames, the article lists server-side techniques and web-components.
• Any specialties for Vaadin? - Vaadin provides a WebComponentExporter (see here) to export web-components. You can find an example right here. A drawback for me is the need for a shared JavaScript bundle to use in the browser.

More

• https://github.com/tuhrig/micro-ui-with-vaadin
• https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage
• https://martinfowler.com/articles/micro-frontends.html

Best regards, Thomas.

Stability

Often times End-2-End tests are not as stable as unit tests. While the context of a unit test is very small and well-defined, End-2-End tests have many pitfalls.

• Is every service deployed with the expected version?
• Are there any changes to the infrastructure?
• Is there a lot of traffic or network issues on the environment?
• Are there timing issues such as race-conditions in the tests?
• Do we have any regressions?
• Do we have some actual bugs?
• Have the acceptance tests been update to reflect all recent changed?

To put it simply, a lot can go wrong. And to be honest, our End-2-End tests will fail at least once a day. For sure.

Visualizing the rate of failed tests

If a single End-2-End tests fails, that’s not a big deal for us. As I described above, there are a bunch of (good) reasons for this to happen. However, the overall rate of failing tests might be an indicator for issues such as underlying bugs or technical debt. But how can we visualize this failing rate?

Bitbucket REST-API

First of all, we must have access to our build results. Let’s take Bitbucket as an example which offers a REST-API to do so. The following GET request will query the first 100 build results from our pipeline. It has five parameters which are important:

• (a) the name of the user / organisation / company
• (b) the name of the repository
• (c) the page to get (starting with 1)
• (d) the page length (max 100)
• (e) the trigger type

https://api.bitbucket.org/2.0/repositories/my-user-name/acceptance-test/pipelines/?page=1&pagelen=100&sort=-created_on&trigger_type=SCHEDULED
                                           |----------| |-------------|                 |         |-|                               |-------|
                                                a               b                       c          d                                    e

If you run the request, you will get a result like this. It contains everything we need to visualize our build results over time.

{
   "page":1,
   "pagelen":2,
   "values":[
      {
         "state":{
            "result":{
               "name":"SUCCESSFUL"
            }
         },
         "build_number":33308,
         "created_on":"2023-03-07T07:48:01.848598Z",
         "trigger":{
            "name":"SCHEDULE",
            "type":"pipeline_trigger_schedule"
         },
         "duration_in_seconds":390
      },
      {
         "state":{
            "result":{
               "name":"SUCCESSFUL"
            }
         },
         "build_number":33307,
         "created_on":"2023-03-07T06:48:00.912946Z",
         "trigger":{
            "name":"SCHEDULE",
            "type":"pipeline_trigger_schedule"
         },
         "duration_in_seconds":379
      }
   ]
}

Scripting to collect all build results

We can query the build results page by page with a max page length of 100. To get a huge number of build results, we can use some simple scripting. In Kotlin this could look like this:

// Authentication via Basic Auth header: user + app password
val headers = HttpHeaders().apply {
    val base = Base64.encodeBase64("$user:$appPassword".toByteArray())
    val asString = String(base)
    set("Authorization", "Basic $asString")
}

// Get page 1 to 3
val pagesWithPipelineRuns: List<JsonNode> = (1..3).map { page ->
    val uri = "https://api.bitbucket.org/2.0/repositories/$company/$repository/pipelines/?page=$page&pagelen=100&sort=-created_on&trigger_type=SCHEDULED"
    val result = restTemplate.exchange(uri, GET, HttpEntity<String>(headers), String::class.java)
    objectMapper.readTree(result.body).get("values")
}

Authentication via Basic Auth

The authentication works with basic auth. However, you must create an app password in order to do so. Go to your personal settings in Bitbucket, click on “App passwords” and create a new one. You must use this password together with your username for Basic Auth.

Create a CSV file

At this point, we have a list of as many test results as we want. We can query page after page for the last couple of years. To plot the results we can map the JSON results to a simple CSV file.

First, we map the JSON from Bitbucket to a simple Kotlin data class. This makes it much easier for us to create the CSV.

val allPipelineResults = pagesWithPipelineRuns.flatMap { it.mapNotNull(::jsonToTestResult) }

fun jsonToTestResult(jsonOfTestResult: JsonNode): TestResults {
    val success = jsonOfTestResult.get("state").get("result").get("name").asText()
    return TestResults(
        build = jsonOfTestResult.get("build_number").asInt(),
        success = if (success == "SUCCESSFUL") 0 else 1,
        duration = jsonOfTestResult.get("duration_in_seconds").asInt(),
        createdOn = jsonOfTestResult.get("created_on").asText().asOffsetDateTime()
    )
}

data class TestResults(
    val build: Int,
    val success: Int,
    val duration: Int,
    val createdOn: OffsetDateTime
)

Now we can map all test results to a simple CSV file. Since there might be many test results, I decided to group all results by day and to take the average.

File("results.csv").printWriter().use { out ->
    allPipelineResults
        .distinct()
        .sortedBy { it.createdOn }
        .groupBy { it.createdOn.toLocalDate() } // group by day
        .forEach {
            val success = it.value.map { it.success }.average() // average per day
            out.println("${it.key};${success};") // e.g. 2023-03-06;0.452;
        }
}

The final CSV file could look like this:

2023-02-26;0.142;
2023-02-27;0.125;
2023-02-28;0.33;
2023-03-01;0.5;
2023-03-02;0.125;
2023-03-03;0.04166;
2023-03-04;0.0;
2023-03-05;0.0;
2023-03-06;0.0;

Plot with Google Sheets

We can plot the CSV file easily with Google Sheets. Create a new spreadsheet an import the CSV file.

Now you can mark both columns and insert a simple line chart. The plotting should work without any modifications out of the box. Just hit the button.

Interpreting the chart

So what can we see with such a chart? Let’s take a look at a real life example from our team. The chart below shows the test results from one of our End-2-End tests for the last year. Starting at January 2022 until March 2023 (aka today).

I’ve marked three sections in the chart which are interesting.

• (A) During this time we implemented a new search engine. This was a big topic and took several weeks until everything was working. You can see this very good in the chart. The error rate goes up as soon as the new project was startet. It drops only again weeks later when all issues have been fixed.
• (B) At the beginning of section B we started a big organisational change. We made new teams and changed our development process. This lead to an increase of technical dept for multiple weeks.
• (C) In section C we started to tackle this problem. We actively reduced technical debt and put our focus on bug-fixing. The rate of failing End-2-End tests dropped as a consequence.

As you can see, we can (try to) correlate the increase of the failure rate with some kind of events. This helps us to spot problematic features or changes we did in the past.

Summary

End-2-End tests are great and can give you much confidence in your system. But of course, they will fail from time to time for whatever reason. The rate of failure can be a good measurement for technical debt, underlying bugs or regressions. By visualizing this failure rate (for example by plotting test results) you can correlate it with events. Such events can be features or projects, but also other things such as organizational changes.

Best regards, Thomas.

@KafkaListener(topics = "orderSubmitted")
public void handle(String event) {
    // ...
}

@KinesisListener(stream = "order-submitted-event")
public void handle(OrderSubmittedEvent event) {
    // ...
}

@JmsListener(destination = "orderSubmitted")
public void handle(OrderSubmitted event) {
    // ...
}

Defining a custom annotation

For this example, we introduce a custom annotation. In Kotlin this would look like this. The annotation is called @MyEventListener and takes a single argument - the name of the event to listen for.

@Target(AnnotationTarget.FUNCTION)
@Retention(AnnotationRetention.RUNTIME)
annotation class MyEventListener(val event: String)

A complete event listener would look like this:

@Component
class EventListeners {
    
    @MyEventListener(event = "order-submitted-event")
    fun handle(event: OrderSubmittedEvent) {
        // ...
    }

    @MyEventListener(event = "order-cancelled-event")
    fun handle(event: OrderCancelledEvent) {
        // ...
    }
}

So how can we pick up those methods?

Finding all beans

The first step is to get a list of all beans managed by Spring. We can do this by using the ApplicationContext which gives us access to all available beans. The only thing we must be careful with, is to wait until the ApplicationContext is ready to use. In Kotlin, this would look like this:

@EventListener
fun applicationReady(event: ApplicationReadyEvent) {
    val allAvailableBeans = getAllBeans()
    println(allAvailableBeans)
}

private fun getAllBeans(): List<Any> {
    return applicationContext
        .beanDefinitionNames
        .map { applicationContext.getBean(it) }
}

Finding beans with annotated methods

After we have the list of all beans, we can search for methods with our annotation.

val beansWithOurMethodAnnotation = allAvailableBeans.filter {
    AopUtils
        .getTargetClass(it)
        .methods
        .any { it.isAnnotationPresent(MyEventListener::class.java) }
}

This looks obvious except of one step: what is AopUtils.getTargetClass(it)? To answer this question, we must take a closer look at how the Spring framework works.

Spring uses proxies to handle cross-cutting concerns such as transactions. This means that a bean is not called directly, but by using a proxy. The proxy wraps the actual bean and takes care of things such as transactions (before and after calling the actual class).

(Picture taken from Spring Docs 3.0.0.M3)

In your debugger this looks something like this:

The tricky thing is, that the proxy does not have a method annotated with our annotation. Only the target class wrapped by the proxy has this annotation. So we need to unwrap the class inside the proxy before looking for methods with our annotation.

Invoking our annotated methods

Great, we found all beans which have a method annotated with our customer annotation! But what can we do with them now? Of course, we can call them!

val eventName = "order-submitted-event"
val eventJson = """{ "id":"o-2023-01", "customer":"c-21331", "items":[ ... ] }"""

allAvailableBeans.forEach { bean ->
    AopUtils
        .getTargetClass(bean)
        .methods
        .filter { it.isAnnotationPresent(MyEventListener::class.java) }
        .filter { it.getAnnotation(MyEventListener::class.java).event == eventName }
        .forEach { method ->
            val eventClass = method.parameterTypes[0]
            val event = objectMapper.readValue(eventJson, eventClass)
            method.invoke(bean, event)
        }
}

An example use-case

An example would be to use this technique to provide REST-controllers for event listeners (such as Kinesis, Kafka or ActiveMQ). The event listeners are called by any incoming record from Kinesis. But sometimes it’s good to have a simple way for providing test data and debugging. So we implemented a generic REST-controller to invoke any Kinesis event listeners. It looks like this. The REST-controller makes it super easy to send some JSON to a Kinesis event listener. It’s very convenient.

@RestController
class KinesisListenerController(private val objectMapper: ObjectMapper) {

    private var listeners: List<Any> = emptyList()
    
    @PutMapping("/kinesis/streams/{stream}")
    fun event(@PathVariable stream: String, @RequestBody json: String) {
        listeners
            .forEach { bean ->
                AopUtils
                    .getTargetClass(bean)
                    .methods
                    .filter { it.isAnnotationPresent(KinesisListener::class.java) }
                    .filter { it.getAnnotation(KinesisListener::class.java).stream == stream }
                    .forEach { method ->
                        val eventClass = method.parameterTypes[0]
                        val event = objectMapper.readValue(json, eventClass)
                        method.invoke(bean, event)
                    }
            }
    }

    @EventListener
    fun applicationReady(event: ApplicationReadyEvent) {
        listeners = getAllBeans()
            .filter {
                AopUtils
                    .getTargetClass(it)
                    .methods
                    .any { it.isAnnotationPresent(KinesisListener::class.java) }
            }
    }

    private fun getAllBeans(): List<Any> {
        return applicationContext
            .beanDefinitionNames
            .map { applicationContext.getBean(it) }
    }
}

More

• Spring Application Context Events
• Spring AOP Proxies

Best regards, Thomas.

Graph Databases

Graph databases store information - as the name implies - as a graph. They create a network between different nodes by defining relations. This makes it easy to see how entities stand to each other. Every relation has a semantic. In a traditional SQL database, this would be implemented by JOIN operations which is usually complex and slow.

Note that there are two types of graph databases:

• RDF based graph databases which store everything as simple triplets
• Property Graphs (like Neo4J) which store information as nodes and edges with properties

(More here)

Example

Let’s make an example. Below you can see a dependency graph from the demo project I posted on GitHub.

• There are three shops (EDEKA, ATU and Media Markt)
• Each shop has a couple of locations
• Shops might have the same location (maybe they are in the same shopping-center)
• There are products which are sold by shops
• Products are compatible with each other or respectively require another product

If we try to break down the data model, we will have the following:

• 3 entities (shop, location and product)
• 4 relations (located_at, sold_by, compatible_with, requires)

In a relational SQL database this would look like this:

• 3 tables for the entities (shop, location and product)
• 4 tables for the relations (many-to-many relationship)

The main difference between graph databases and relational databases is the following:

• Building relations in SQL is hard, but easy in Graph databases. Relations in SQL means to use JOIN operations which are complex and not efficient.
• SQL databases focus on single tables with a strong schema and transaction handling. So while relation databases are super efficient with single entries, graph databases are efficient with relations.

So how does a JOIN operation look like in Neo4J?

MATCH (p:product)-[r:SOLD_BY]->(s:shop)
WHERE EXISTS {
    MATCH (s)-[:LOCATED_AT]->(:location {city: 'Karlsruhe'})
}
RETURN p.name, s.name

This query selects all products which are sold in “Karlsruhe” (including the shop name). The result looks like this:

| p.name              | s.name
+---------------------+----------
| "USB Cabel"         | "ATU"
| "Cleaning Spray"    | "ATU"

Neo4J with Spring Boot

I prepared a simple demo project which shows how to use Neo4J with Spring Boot. You can find the project on GitHub. It shows a simple example with some entities (see example above), REST-controllers and boilerplate-code.

• Define entities and relations
• Create data
• Query nodes and relations

https://github.com/tuhrig/neo4j-demo

More

• https://memgraph.com/blog/graph-database-vs-relational-database
• https://neo4j.com/developer/spring-data-neo4j
• https://www.baeldung.com/spring-data-neo4j-intro
• https://www.wisecube.ai/blog/knowledge-graphs-rdf-or-property-graphs-which-one-should-you-pick

Best regards, Thomas.

Monitoring latency in a Microservice Architecture

Background

At Bringmeister, all software we developed during the last five years is composed of microservices. Information drops into the system at one point, moves from service to service and eventually reaches its goal.

A typical dataflow would look like this:

 ------------------------- direction of data flow ----------------------->

 +-----+
 | ERP | -- Master Data ----v
 +-----+            +-------------+        +--------------+    
                    | Aggregation | -----> | Search Index | 
                    |   Service   |        |    Service   | 
                    +-------------+        +--------------+ 
 +-----+                     ^                    |
 | PIM | -- Product Data ----+                    v
 +-----+                                   +-------------+     +----------+
                                           | SaaS Search | --> | Web Shop |
                                           |    Index    |     +----------+  
                                           +-------------+

An architecture like this brings up a couple of questions:

• How long does it take for data from end to end? For example from our ERP to the search index?
• Which service takes the most time?
• How does a change to the system affect the time?

Steffen’s Bachelor thesis tries to give an answer to those questions.

Measuring latency

Steffen implemented a microservice (called Chronos) to measure the time between two points in the system (ERP and search index). The service receives all incoming data simultaneously to the regular system. Based on a sampling algorithm it decides which request should be taken for measurement:

• A fixed list of IDs
• Every tenth or hundredth request
• A certain quota (for example 100 per 5 minutes)
• …

The service then measures the time until the data arrives in the search index. It does so by actively polling the search index until it returnes the expected data.

This approach gave us a couple of advantages:

• We did not relay on logs. Logs can be late (latency until they arrive in the logging system) or misleading (e.g. written before or way after an operation).
• We did not relay on events from the monitored systems themselves. For example an event that states that the data was written to the search index, does not mean that the data is already indexed and live, too.
• We had the opportunity to do a lot of customizing. For example, we implemented a CSV download of the results but also a Micrometer-based metric for DataDog.
• We could easily change the measuring or parts of the system taken into account. For example, we compared two different search engines in parallel under production load.

The results

Steffen’s work brought up a lot of interesting results for us.

• The latency from end to end is between 10 and 15 seconds most of the time. So an update from ERP is live in the shop in about 15 seconds.
• The latency does depend on the load, but not as much as we thought. Random peaks and outliers are actually bigger as the correlation to the load.
• Single systems have a huge impact. The search index for example is responsible for most of the time needed to get an update live. So switching the search engine (SaaS) has a huge impact.

Thank you

I would like to thank Steffen for his great thesis and work.

Congratulations on your Bachelor degree!

🏆🎉🍀

Download

You can download Steffen’s Bachelor thesis right here. It’s written in German.

Best regards, Thomas.

https://github.com/mmnaseri/spring-data-mock

Spring Data

Spring Data works by implementing repository interface during runtime. Here’s an example. Let’s say you have a User object which should be saved. You could write the following repository interface:

public interface UserRepository extends JpaRepository<User, Long> {
    User findByName(String name);
}

During runtime, Spring Data would provide an implementation based on this interface. Every method of the interface follows a certain convention, so Spring Data knows how to implement it.

Dynamic Proxies

Technically, this is solved by using Dynamic Proxies. Dynamic Proxies let us create a proxy for an interface during runtime. The proxy will receive every method invocation on the interface and can handle it. Spring uses it a lot, not only for repositories, but for all kinds of cross-cutting concerns such as transactions or caches.

In Kotlin, this could look like this:

val proxy = Proxy.newProxyInstance(
    this::class.java.classLoader,
    arrayOf<Class<*>>(UserRepository::class.java),
    MyProxyClass()
) as UserRepository

val user = proxy.findByName("Thomas")

class MyProxyClass : InvocationHandler {
    override operator fun invoke(proxy: Any?, method: Method, args: Array<Any?>?): Any? {
        if (method.name == "findByName") {
            // ...
        } else {
            // ...
        }
    }
}

How we usually test

Let’s consider the following example:

class UserServiceTest {

    @Test
    fun `should create invoice for user`() {

        doReturn(user).whenever(userRepo).findByName("Thomas")
        doReturn(order).whenever(orderRepo).find("O-202201-0002")
        
        val invoice = userService.createInvoice("Thomas", "O-202201-0002")
        
        assertThat(invoice).isEqualTo(/*...*/)
    }
}

This could be a typical test case for a UserService. We need to mock the behaviour of two repositories used by the UserService:

• doReturn(user).whenever(userRepo).findByName("Thomas")
• doReturn(order).whenever(orderRepo).find("O-202201-0002")

Another way would be using Docker containers or an in-memory database (like H2). But every solution has its drawbacks:

• Mocking is time-consuming and requires deep white-box-knowledge of the code under test.
• Docker or in-memory databases are slow and require additional setup.

However, we could also go into another direction and provide some in-memory implementations by our own.

class TestInMemoryUserRepository: UserRepository {
    private val users = mutableListOf<User>()
    override fun findByName(name: String): User? {
        return users.find { it.name == name }
    }
}

Instead of mocking or using Docker, we have a super simple “in-memory” implementation which will just act as the normal repository.

But as always, there’s also a drawback to this solution: we probably need to implement this kind of class over and over again.

Imagine an example like this:

interface UserRepository {
    fun find(id: String): User?
    fun save(user: User)
}

interface OrderRepository {
    fun find(orderNumber: String): Order?
    fun save(order: Order)
    fun delete(orderNumber: String)
}

interface AddressRepository {
    fun find(id: String): Address?
    fun save(address: Address)
    fun findAll(userId: String): List<Address>
}

interface InvoiceRepository {
    // ...
}

Using Dynamic Proxies for testing

However, we can use Dynamic Proxies to make this a bit more pleasuring. If you are already working with Spring Data, @mmnaseri’s GitHub library gives you a quick start: Using his implementation, mocking a repository is really simple:

UserRepository repository = RepositoryFactoryBuilder.builder().mock(UserRepository.class);
repository.save(new User());

But what if you don’t use Spring Data? Take a look at the interfaces from the example above - they don’t use Spring Data! In fact, nearly all the software I developed during the last years, does not use Spring Data. Instead, we use DynamoDB (on AWS) with its SDK directly.

In such a case, we can write a Dynamic Proxy on our own. The most complicated part is the InvocationHandler. You can find an example below.

Note that the most important thing is, that your interfaces stick to a naming-convention. If your method is sometimes called find(...), sometimes findById(...) and then get(...) or load(...) again, you will have a hard time to write a generic Dynamic Proxy. But if you stick to a naming-convention, you can re-use the Dynamic Proxy for different interfaces.

class MyGenericRepositoryProxy : InvocationHandler {
    private val store = mutableListOf<Any>()
    override operator fun invoke(proxy: Any?, method: Method, args: Array<Any?>?): Any? {
        when (method.name) {
            "findByName" -> {
                val nameToFind = args?.first()!!
                return store
                    .find { obj ->
                        obj::class.declaredMemberProperties.any {
                            it.name == "name" && it.getter.call(obj).toString() == nameToFind
                        }
                    }
            }
            "findAll" -> {
                return store
            }
            "save" -> {
                val objToSave = args?.first()!!
                store.add(objToSave)
                return null
            }
            else -> {
                // ...
            }
        }
    }
}

We can wrap the creation of the Dynamic Proxy in a nice factory method:

object RepoMockFactory {
    fun <T> mock(javaClass: Class<T>): T {
        return Proxy.newProxyInstance(
            this::class.java.classLoader,
            arrayOf<Class<*>>(javaClass),
            MyGenericRepositoryProxy()
        ) as T
    }
}

Using our mocks

After we created our Dynamic Proxy, we can use it in our tests. No Docker containers, no H2-database and no doReturn(xy).whenever(xy).xy(...). We can just use our mocks like real implementations. This makes the test clean and easy to read.

class UserServiceTest {

    @Test
    fun `should create invoice for user`() {

        val mockedUserRepo = RepoMockFactory.mock(UserRepository::class.java)
        val mockedOrderRepo = RepoMockFactory.mock(OrderRepository::class.java)
        
        mockedUserRepo.save(User("Thomas"))
        mockedOrderRepo.save(Order("O-202201-0002"))
        
        val userService = UserService(mockedUserRepo, mockedOrderRepo)
        val invoice = userService.createInvoice("Thomas", "O-202201-0002")
        
        assertThat(invoice).isEqualTo(/*...*/)
    }
}

More

• Dynamic Proxies in Java
• Java Dynamic proxy mechanism and how Spring is using it
• Spring Data Mock on GitHub
• Dynamische Proxys mit dem JDK umsetzen (German)

Best regards, Thomas.

Right from the beginning, I made an obvious decision: I installed Wordpress together with a couple of plugins and themes to get started. And to get things clear, Wordpress did a great job for more than a decade for me!

Why I want to migrate away from Wordpress

However, since a couple of years, I have the feeling that Wordpress is growing over my head. I must take care of installing updates, moderating comments, optimize the performance and manage a bunch of plugins which I think I need. And in the end, it comes all down to this:

Wordpress is far too powerful for the simple tasks I want to achieve.

What’s the alternative?

As obvious as Wordpress was back in 2011 for me, so is GitHub pages right now. I’m a software developer and working with Git and Markdown is easy for me. The concept of generating static HTML based on templates (Jekyll) feels like a light-weight alternative for a full-fledged CMS like Wordpress. Exactly what I want!

Migrating from Wordpress to Jekyll

Migrating from Wordpress to Jekyll was a mix of automated tasks and manual work. Here’s a rough outline of what I did:

1. Exporting all of my posts from Wordpress as a huge XML-file (Tools > Export > Download Export File). After 10 years of blogging I got about 3,3 MB of pure XML.
2. Converting the XML-file to Markdown by using wordpress-export-to-markdown. This will also download all images. I ended up with about 30 MB of Markdown and images.
3. Choosing a pre-made Jekyll theme that looks clean and fork it.
4. Creating the folder structure for posts and images in the new theme. I made a folder per year (2022, 2021…) and inside a folder per month (01, 02…).
5. Copying the generated Markdown files to the new folder structure and fix their naming. The Wordpress export will name every file index.md, but the name schema should be yyyy-mm-dd-my-post-name.md.
6. Manually checking each file to see if the export/conversion has broken something. I also fixed typos and rewrote some misleading sentences. This part took the longest.
7. Commit and push everything to https://github.com/tuhrig/tuhrig.github.io.

At this point, my blog was live on GitHub Pages. However, I wanted to use my existing domain tuhrig.de. So I decided to deploy the Jekyll build via FTP to my own webspace. The alternative was to use a 308 Permanent Redirect which I don’t prefer.

Here’s what I did:

1. I created a GitHub action to run the Jekyll-build. See here.
2. After the Jekyll-build, a simple FTP upload too my own webspace is executed. See here.

You can find the current build script right here.

You can find a very good tutorial about migrating from Wordpress to Jekyll right here:

https://haralduebele.github.io/2021/02/10/Moving-my-Blog-from-Wordpress-to-Github-Pages

No more comments

I decided to remove all comments from my blog. While I received some useful comments in the past, there were not many. The amount of spam was always very high and I had to moderate the comments regularly. Often comments contained questions about old posts which I was unable to answer since everything was outdated for years. So in the end, the value of the comments was very low to me.

Instead, I encourage everyone to get in touch with me via email or any of my social media profiles.

Final thoughts

Blogging with Jekyll instead of Wordpress feels much more understandable for me. No more plugins, pingbacks, WYSIWYG-editors, admin sections and setting pages. Everything seems to be more under control.

The actual migration on the other side was a bunch of work. My old blog has been growing over the years and it accumulated a lot of “technical debt”. Even before the migration, there have been broken posts, dead links and missing images. I’m happy that it’s done, but it wasn’t all fun.

More

• Tutorial on how to migrate from Wordpres to Jekyll
• GiHub FTP upload action
• Run Jekyll-build in an own GitHub action

Best regards, Thomas.