Opinions expressed in this post are totally my own and not necessarily that of my employer.

Wednesday started with the infamous “scriptbowl”, a competition between various scripting languages. This year the contenters were JRuby, Groovy, Scala, and Clojure. I wondered whether Scala considers itself a scripting language but obviously they either do or just seek the opportunity to be on stage.

To keep a long story short: Groovy has won this event for the third time in a row! This year the race was tied with Scala. Guillaume presented Groovy in the typical Groovy-idomatic style and explained every single line of his concurrent visual analyzer for Google+ postings. Dick Wall presented only non-idomatic Scala code. I interpret this as: to make Scala appealing you have to make it look like Groovy. Furthermore, he presented Kojo, which is a great interactive learning environment written in Play/Scala. In contrast to all other presentations, this was not specifically created for the scriptbowl, nor was it written by the presenter, nor was it clear how much effort went into it, nor did the audience see a single line the implementation code. How much this skewed the comparison, I leave to everybody’s judgement. The show was good, though.

I felt a bit sorry for Clojure. It is a great language and deserves a presentation that is more visually appealing to convince the crowd.

Afterwards, I attended a hands-on lab for “rapid enterprise development with netbeans”, which was essentially creating a Swing app for database CRUD actions. If I remember correctly, I did the exact same task 1997 with JBuilder. It left me with the feeling of “Yes, it works” but it is not less complex than it was 13 years ago.

Early afternoon Gerrit Grunwald (better known as @hansolo_) presented his work on simplified custom components for Swing. Given that he speaks about an activity that is both utterly important and highly underadvertised he would really deserve speaking at the center stage.

Graeme Rocher’s great session about Grails, polyglot datastores (hibernate, jpa, redis, mongodb, …), and the cloud was overshadowed by the news that Steve Jobs has died. Accidentally, the demo application was about showing a BBC News stream, which displayed this information live on stage. Both the presenter and the audience were equally touched.

The day officially ended with a big event at treasure island. I decided to not go there, though, and meet the former Canooey Denis Antonioli in Berkely where we had a great evening.

Dierk Koenig

Finally, late in the evening I joined Dan Sline’s talk on WebServices in the Groovy space. The major take-away for me was a repercussion of the well-known advice: “keep it simple”.

Throughout the day, a lot of people approached me to tell how much they liked my talks yesterday. That was a really nice experience. Last year I had the very last talk of the conference and only this year I recognized how much of a difference the scheduling of the talks make.

Dierk Koenig

This screencast demonstrates the small language changes that are part of Open JDK 7, which is available from the Open JDK website. It demonstrates multi-catch, try with resources, strings in switch statements, underscores in literals, and the diamond operator.

If you have any issues watching the video below, then you may have better luck viewing it on the JetBrains.tv site.

I’ve made a lot of screencasts and blog posts over the years. If you like this, then there are many ways to see the other stuff I’ve done: 

• My main blog: http://hamletdarcy.blogspot.com
• My other JetBrains.tv posts: http://tv.jetbrains.net/tags/hamlet
• IDEA related posts on my blog: http://hamletdarcy.blogspot.com/search/label/IDEA
• My screencasts on YouTube: http://www.youtube.com/user/HamletDRC
• IDEA related Posts on my work blog: http://www.canoo.com/blog/tag/idea/
• Or follow me on Twitter: @HamletDRC

The screencast was created with Ubuntu 10.04, PiTiVi, Audicity, gtk-RecordMyDesktop, IntelliJ IDEA, and LibreOffice. OS from top to bottom.

Thanks for watching, and leave a comment!

I got a chance to sit down and talk to Java Champion Antonio Goncalves about the past, present, and future of Java EE. I’ve been working for the last six months in a heavy EE/SOA stack, and it’s been interesting to see the advantages and disadvantages. I definitely come from the other side of the world where specifications aren’t seen as an inherent sign of quality, and frameworks not sanctioned by Sun/Oracle are not to be feared. It was fun to get his opinions about this stuff.

The full interview is on the JetBrains Zone at DZone. We’re both JetBrains Academy Members and we’re slowly interviewing each other.

P.S. This is the first post I’ve ever made that mentioned Java Enterprise Edition. I suspect the next time EE is mentioned will be in another few years

Well, well, I made another screencast. This time I’m taking on IntelliJ IDEA code inspections, and writing your own static code analysis rule (and quick fix!) using Structural Search & Replace. Not bad for under 5 minutes.

If you have any trouble viewing the video then perhaps you should watch it directly on the JetBrains site. And if you’re feeling generous, then clicky clicky to upvote at DZone.

Check out these pages for more information on IDEA inspections: 

• JetBrains’ Code Inspection page 
• IDEA’s Webhelp for inspections

Read these pages to learn more about Structural Search and Replace: 

• JetBrains’ Structural Search and Replace Documentation
• IDEA’s Webhelp for Structural Search and Replace

I’ve made a lot of screencasts and blog posts over the years. If you like this, then there are many ways to see the other stuff I’ve done: 

• My main blog: http://hamletdarcy.blogspot.com
• My other JetBrains.tv posts: http://tv.jetbrains.net/tags/hamlet
• IDEA related posts on my blog: http://hamletdarcy.blogspot.com/search/label/IDEA
• My screencasts on YouTube: http://www.youtube.com/user/HamletDRC
• IDEA related Posts on my work blog: http://www.canoo.com/blog/tag/idea/
• Or follow me on Twitter: @HamletDRC

Phew, that’s a lot of self-promotion

Thanks for watching, and leave a comment! 

If you don’t know it yet, this is the fourth part of a series about Apache Camel. The previous posts can be found here: first part, second part and third part.
In this fourth article, we are going to show how to properly use the Spring framework in the sample application, while adding some extra functionality to it.

Extracting RSS feeds

If you remember what we have done so far, the sample application is capable of, given the URL of a web page, extract the text content of the body paragraphs. To make our application a little more powerful, let’s add some functionality to extract the contents of web pages listed in RSS feeds. To accomplish this, add a new class called “RssExtractorRoutes” with the following content:

public class RssExtractorRoutes extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("rss:http://feeds.bbci.co.uk/news/rss.xml?splitEntries=false")
                .marshal().rss()
                .marshal().string()
                .split(xpath("//item/link/text()"))
                .log("Link: '${body}'")
                .to(PAGE_EXTRACTOR_EP);
    }
}

Because we have added a new component, we need to add a new dependency to our Maven pom file:

<dependency>
     <groupId>org.apache.camel</groupId>
     <artifactId>camel-rss</artifactId>
     <version>${camel.version}</version>
</dependency>

public class Part4 {
    public static void main(String[] args) throws Exception {
        DefaultCamelContext camelContext = new DefaultCamelContext();
        camelContext.addRoutes(new PageExtractorRoutes());
        camelContext.addRoutes(new HtmlImproverRoutes());
        camelContext.addRoutes(new RssExtractorRoutes());
        camelContext.start();

        Thread.sleep(100000);
        camelContext.stop();
    }
}

public class ElasticSearchBean {
    public static final int DEFAULT_MAX_RESULTS = 50;

    public static final String INDEX_METHOD = "index";
    public static final String SEARCH_METHOD = "search";
    public static final String ID_HEADER = "search_id";

    private final Client fClient;
    private final Node fNode;
    private final String fIndex;
    private final String fType;
    private final String fField;
    private int fMaxResults = DEFAULT_MAX_RESULTS;

    public ElasticSearchBean(String index, String type, String field) {
        fIndex = index;
        fType = type;
        fField = field;
        fNode = nodeBuilder().local(true).node();
        fClient = fNode.client();
    }

    public String getIndex() {
        return fIndex;
    }

    public String getType() {
        return fType;
    }

    public String getField() {
        return fField;
    }

    public int getMaxResults() {
        return fMaxResults;
    }

    public void setMaxResults(int maxResults) {
        fMaxResults = maxResults;
    }

    public List<String> search(String query) {
        SearchResponse searchResponse = search(
                getField(), query,
                getMaxResults(), getIndex()
        );

        List<String> results = new ArrayList<String>();

        for (SearchHit hit : searchResponse.getHits()) {
            results.add(hit.id());
        }

        return results;
    }

    public void index(Exchange exchange) {
        Message in = exchange.getIn();
        String id = in.getHeader(ID_HEADER, String.class);
        String content = in.getBody(String.class);
        try {
            index(getIndex(), getType(), id, getField(), content);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    public void close() {
        fNode.close();
        fClient.close();
    }

    private IndexResponse index(String index, String type, String id, String fieldName, String fieldValue) throws IOException {
        XContentBuilder item = jsonBuilder()
                .startObject()
                .field(fieldName, fieldValue)
                .endObject();
        return fClient.prepareIndex(index, type, id)
                .setSource(item)
                .execute()
                .actionGet();
    }

    private SearchResponse search(String fieldName, String query, int maxResults, String... indexes) {
        return fClient.prepareSearch(indexes)
                .setSearchType(SearchType.DEFAULT)
                .setQuery(termQuery(fieldName, query))
                .setFrom(0).setSize(maxResults).setExplain(true)
                .execute()
                .actionGet();
    }
}

To use it in the RSS extractor, adjust the code like this:

public class RssExtractorRoutes extends RouteBuilder {
    private static final String ARTICLES_INDEX = "articles";
    private static final String ARTICLE_CONTENT_FIELD = "content";
    private static final String ARTICLE_TYPE = "article";
    private static final ElasticSearchBean ELASTIC_SEARCH_BEAN = new ElasticSearchBean(ARTICLES_INDEX, ARTICLE_TYPE, ARTICLE_CONTENT_FIELD);

    @Override
    public void configure() throws Exception {
        from("rss:http://feeds.bbci.co.uk/news/rss.xml?splitEntries=false")
                .marshal().rss()
                .marshal().string()
                .split(xpath("//item/link/text()"))
                .setHeader(ID_HEADER, body())
                .log("Link: '${body}'")
                .to(PAGE_EXTRACTOR_EP)
                .bean(ELASTIC_SEARCH_BEAN, INDEX_METHOD);
    }
}

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>0.15.2</version>
</dependency>

• If we would like to test this route, we cannot supply a different implementation of the bean.
• In case that we want to change the name of the index, the name of the field or any other configurable value, we have to do it in the code and recompile the application.
• When the bean is created, the Elastic Serach node is correctly initialized but, when is the “close” method called to shutdown the instance properly? This is, of course, the problem I mentioned.

To correct these issues, we can take advantage of the Spring framework and the excellent Spring support that Camel offers.

First of all, let’s include the Spring dependency in our Maven pom file:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-spring</artifactId>
    <version>${camel.version}</version>
</dependency>

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
    <camelContext xmlns="http://camel.apache.org/schema/spring">
        <packageScan>
            <package>com.canoo.camel</package>
        </packageScan>
    </camelContext>
</beans>

import org.apache.camel.spring.Main;

public class Part4 {
    public static void main(String[] args) throws Exception {
        Main main = new Main();
        main.enableHangupSupport();
        main.start();
    }
}

Now that we have configured Spring, let’s move the Elastic Search bean into the Spring beans file (“camel-context.xml”):

<bean class="com.canoo.camel.beans.ElasticSearchBean" id="elasticSearchBean" scope="singleton" destroy-method="close">
    <constructor-arg value="articles"/>
    <constructor-arg value="article"/>
    <constructor-arg value="content"/>
</bean>

@Override
public void configure() throws Exception {
    from("rss:http://feeds.bbci.co.uk/news/rss.xml?splitEntries=false")
            .marshal().rss()
            .marshal().string()
            .split(xpath("//item/link/text()"))
            .setHeader(ID_HEADER, body())
            .log("Link: '${body}'")
            .to(PAGE_EXTRACTOR_EP)
            .beanRef("elasticSearchBean", INDEX_METHOD);
}

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="location">
        <value>elasticsearch.properties</value>
    </property>
</bean>

<bean class="com.canoo.camel.beans.ElasticSearchBean" id="elasticSearchBean" scope="singleton"
      destroy-method="close">
    <constructor-arg value="${index_name}"/>
    <constructor-arg value="${content_type}"/>
    <constructor-arg value="${index_field}"/>
</bean>

To end this article doing something useful with our indexed contents, let’s create a search web service that allows us executing a search query and displays a web page with the matching article links.

For this, create a new route builder:

public class SearchServiceRoutes extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("jetty://http://0.0.0.0:8080/search")
                .setBody(header("query"))
                .to("direct:search")
                .bean(HtmlFormatterBean.class, AS_SEARCH_RESULTS_PAGE);

        from("direct:search")
                .log("Searching: '${body}'.")
                .beanRef("elasticSearchBean", SEARCH_METHOD)
                .bean(HtmlFormatterBean.class, AS_LINKS)
                .log("Found results: '${body}'.");
    }
}

We need also to extend the functionality of the class “HtmlFormatterBean” as follows:

public class HtmlFormatterBean {
    public static final String AS_EXTRACTED_RESULTS_PAGE = "asExtractedResultsPage";
    public static final String AS_SEARCH_RESULTS_PAGE = "asSearchResultsPage";
    public static final String AS_LINKS = "asLinks";

    public String asExtractedResultsPage(List<String> contents) {
        return asPage(contents, "Extracted contents:");
    }

    public String asSearchResultsPage(List<String> contents) {
        return asPage(contents, "Search results:");
    }

    private String asPage(List<String> contents, String title) {
        StringBuffer stringBuffer = new StringBuffer();
        stringBuffer.append(String.format("<html><body><h1>%s</h1><ul>", title));
        for (String content : contents) {
            stringBuffer.append("<li>").append(content).append("</li>");
        }
        stringBuffer.append("</ul></body></html>");
        return stringBuffer.toString();
    }

    public List<String> asLinks(List<String> urls) {
        List<String> result = new ArrayList<String>();
        for (String url : urls) {
            result.add(String.format("<a href='%1$s'>%1$s</a>", url));
        }
        return result;
    }
}

To test our new search service, start the application and point to the URL: http://localhost:8080/search?query=news. If some of the indexed articles contains the word “news”, you should see a list with their links.

The camel ride ends here. I hope that you enjoyed reading the articles as much as I enjoyed writing them. I also hope that it has helped you in getting to know how Camel works and that you can found some use cases where to apply this slightly different way of building integration applications.

The code of this fourth part is here, to execute the application just unzip the file, change to the directory where the pom file is and type ‘mvn compile exec:java -Dexec.mainClass=”com.canoo.camel.Part4″‘ in the console.

Hope to see you soon in another post!

In the first part of this series we saw the basics of Apache Camel routing and contexts, and in the second part we learned about error handling and web services. Today, in the third part, we will add more functionality to our Camel extractor application and we will do it in a modular way. We will also have a look at how to effectively test our functionality.

Improving our HTML cleaner

In a perfect world, our application would never need any bug fixing or improvements. But because we know that the world is not perfect, it happens often that we discover limitations in our applications and we must make some improvements to correct them. If you try extracting a URL from DZone (“http://www.dzone.com“) with the application, then you will see an exception in the console indicating that “TagSoup” cannot fix a problem related with the DOM structure (TagSoup is used by the “TidyMarkup” marshaller to clean our HTML) . We could configure “TagSoup” differently, but instead, let’s assume that we have to use another tool to do the work. In this case, we are going to use “HtmlCleaner” and we’ll integrate it into our application using a Java “bean”.

Add a new Java class to your project containing this code:

public class HtmlCleanerBean {
    public Document cleanHtml(String html) {
        CleanerProperties properties = new CleanerProperties();
        properties.setNamespacesAware(false);
        HtmlCleaner cleaner = new HtmlCleaner(properties);
        TagNode node = cleaner.clean(html);
        try {
            return new DomSerializer(properties).createDOM(node);
        } catch (ParserConfigurationException e) {
            throw new RuntimeException(e);
        }
    }
}

This class uses “HtmlCleaner” and therefore requires the following Maven dependency:

<dependency>
    <groupId>net.sourceforge.htmlcleaner</groupId>
    <artifactId>htmlcleaner</artifactId>
    <version>2.2</version>
</dependency>

We could add more steps to our extractor route, but it is starting to become a little verbose and unclear. So instead, let’s better add a new route for our improved html functionality:

public class HtmlImproverRoutes extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("direct:html_improver")
                .onException(CamelException.class)
                    .continued(true)
                    .log("Tidying failed, trying with Cleaning.")
                    .bean(HtmlCleanerBean.class)
                .end()
                .unmarshal().tidyMarkup();
    }
}

And replace the tidy markup marshalling with a detour through the new created route:

        from("direct:page_extractor")
                .streamCaching()
                .onException(HttpOperationFailedException.class)
                .onWhen(bean(HttpErrorHelperBean.class).isEqualTo(true))
                    .log("Fetching URL failed: '${exception.message}', trying with relocation: '${exception.redirectLocation}'.")
                    .handled(true)
                    .setBody(simple("${exception.redirectLocation}"))
                    .to("direct:page_extractor")
                .end()
                .setHeader(Exchange.HTTP_URI, body())
                .setBody(constant(null))
                .log("Extracting content from: '${header." + Exchange.HTTP_URI + "}'")
                .pipeline("http:extractor", "direct:html_improver")
                .log("Html from: '${body}'")
                .split(xpath("//body//p/text()"), new SplitterAggregationStrategy("(?s).*[A-Za-z0-9].*"))
                    .log("Text chunk: '${body}'.")
                .end();

To get this route created in the Camel context, adjust the main method in the “main” class like this:

public static void main(String[] args) throws Exception {
    DefaultCamelContext camelContext = new DefaultCamelContext();
    camelContext.addRoutes(new PageExtractorRoutes());
    camelContext.addRoutes(new HtmlImproverRoutes());
    camelContext.start();

    ProducerTemplate template = camelContext.createProducerTemplate();
    String result = template.requestBody("direct:page_extractor", "http://www.dzone.com", String.class);
    System.out.printf("Extracted: '%s'.\n", result);

    Thread.sleep(100000);
    camelContext.stop();
}

We have also performed some changes in the page extractor route. First, we have activated the “streamcaching” to avoid exceptions when the inputstream returned by the “http” extractor is consumed several times. Also, after setting the URL to extract in the “CamelHttpUri” header, we are now resetting the body of the message sent to the “http” endpoint. If not done, the body causes some problems when accessing certain web servers because the endpoint uses the HTTP POST method (used when the body is not empty) instead of the more suitable GET. The last change is to route the result of the “http” endpoint to the newly created route. To enforce a little bit more the sequential character of this process, we have applied the “pipeline” enterprise integration pattern instead of the default “to” construct of the Java DSL.

If you run the application now, you should see how it detects the parsing error and continues the processing of the content using “HtmlCleaner” instead of “TagSoup”.
This functionality (the “html improver”) has been moved from the initial route builder to an specific one, what contributes a to better modularization and re-usability of the code in Apache Camel. Because the endpoints are named with string constants and now the strings: “direct:page_extractor” and “direct:html_improver” appear in different files, it is usually a good practice to extract these values as constants in the route builders. This, makes more difficult to get the code broken in case of a change in the endpoint name or type, and it plays the role of an interface.

Testing the routes

Apache Camel offers mock testing support for testing the routes in an effective way. But, while in most of the examples out there, a mock endpoint is added explicitly to the route, let’s see how to do it in a programmatic way to avoid altering the route that we want to test.

First, add these two dependencies to your Maven pom file:

...
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-test</artifactId>
    <version>${camel.version}</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.8.1</version>
    <scope>test</scope>
</dependency>
...

Second, add this class with some run-time support for mock testing:

public class CamelMockTestSupport extends CamelTestSupport {
    private MockEndpoint fMockEndpoint;

    public MockEndpoint getMockEndpoint() {
        if (null == fMockEndpoint) {
            fMockEndpoint = new MockEndpoint("mock:_mock_");
        }
        return fMockEndpoint;
    }

    protected void sendBodyToMock(String endpoint, final Object body) {
        Exchange exchange = template.request(endpoint, new Processor() {
            public void process(Exchange exchange) throws Exception {
                exchange.getIn().setBody(body);
            }
        });
        exchange.getIn().copyFrom(exchange.getOut());
        template.send(getMockEndpoint(), exchange);
    }
}

Third, create a test based on this approach:

...
import static com.canoo.camel.routes.HtmlImproverRoutes.HTML_IMPROVER_EP;

public class HtmlImproverRoutesTest extends CamelMockTestSupport {
    @Override
    protected RouteBuilder createRouteBuilder() throws Exception {
        return new HtmlImproverRoutes();
    }

    @Test
    public void testHtmlImproverRoute_withBadHtml() throws InterruptedException {
        MockEndpoint mock = getMockEndpoint();
        mock.expectedMessageCount(1);
        mock.expectedBodiesReceived("<html xmlns:html=\"http://www.w3.org/1999/xhtml\">\n" +
                "<body>\n" +
                "<h1>Bad Html</h1>\n" +
                "</body>\n" +
                "</html>\n");

        sendBodyToMock(HTML_IMPROVER_EP, "<html><h1>Bad Html</html>");

        mock.assertIsSatisfied();
    }
}

As summary of this third part, we have shown how to organize the project routes using several route builders and how you can use mocks to test and, therefore, improve the quality and stability of your Apache Camel project.

In the next Camel “ride”, we will introduce the Spring framework and unveil what to do with the extracted data. Hope to see you there!

Source code of this third part for download. To run the application, just expand the file, change to the folder where the pom file is and execute: ‘mvn compile exec:java -Dexec.mainClass=”com.canoo.camel.Part3″‘

The first part of this series showed you how to start a Camel context, write a simple route, and then stop the context. You’re starting to see how to do some things in the “Camel way”. Let’s go for a second Camel ride and see how far we can go.

Error Handling in Camel

As promised, we’re going to add some error handling to our route and see how to deal with the usual problems. To see what I mean, go to the main class from the first article and substitute the URL of the W3C site (“http://www.w3.org”) with this alternate version: “http://www.w3c.org”. Run the application once again and now you should see the application failing to retrieve the page with a “HttpOperationFailedException” error. The problem is that the new URL is a redirect to the old one, but our route doesn’t know how to deal with this.

If you have a look at the “http” endpoint documentation, you will notice how flexible and configurable it is. You’ll also see that some HTTP response codes are thrown as exceptions by default. In plain Java, you can instruct a connection to follow redirections in a transparent way. With the “http” configuration options, we could probably do the same by configuring the “HttpClient” and modifying the “http” endpoint it uses internally. Instead of this, let’s learn some error handling strategies in Camel.

We want some sepcial logic for when an “HttpOperationFailedException” is thrown: we should grab the redirection location present in the exception and retry the request. For this, modify the code of the extractor route in the following way:

@Override
public void configure() throws Exception {
    from("direct:page_extractor")
            .onException(HttpOperationFailedException.class)
                .log("Fetching URL failed: '${exception.message}', trying with relocation: '${exception.redirectLocation}'.")
                .handled(true)
                .setBody(simple("${exception.redirectLocation}"))
                .to("direct:page_extractor")
            .end()
            .setHeader(Exchange.HTTP_URI, body())
            .log("Extracting content from: '${body}'")
            .to("http:extractor")
            .unmarshal().tidyMarkup()
            .log("Html from: '${body}'")
            .split(xpath("//body//p/text()"), new SplitterAggregationStrategy("(?s).*[A-Za-z0-9].*"))
                .log("Text chunk: '${body}'.")
            .end();
}

What we introduced at the beginning of the route is an “onException” clause that, in case of an exception of the given type (or types) being thrown within this route, executes the steps contained in the clause. Please, notice the “end” method call separating what should be done in case of such an exception from the normal route steps.

In the error handling code, the first action we take is to log some information about what is about to happen. Then, we mark that the exception has already been handled and, after setting in the body the relocation URL, we send the message back to the endpoint to be retried.

The “onException” clause has a lot of options and the topic “error handling” is very well described in the chapter 5 from Camel in Action.

Fine grained logic with Java beans

If you need functionality that you cannot express with the Camel Java DSL, then using Java or any other languages supported in Camel is pretty easy. One option is to use the “process” method from the Java DSL. This method accepts a “Processor” parameter in through which you have access to the complete message (“Exchange” in Camel). While this seems pretty easy and practical, you will be tempted to pollute your routes with anonymous Java classes containing logic. You should also notice that, following this approach, you need to do almost the same mapping every time to access the body of the “in” exchange and set the result back into it. Another drawback of anonymous classes is that you cannot easily test the logic on them and, if you want to test them, you must create top-level classes that are both more complicated than simple Java beans and coupled to the Camel API through the “Exchange” class.

A better option is to use plain Java classes (“beans”) with the least possible logic, and let Camel do the default mapping using the signature of the methods invoked. If mapping the body of the exchange is not enough, you can use annotations to get access to some other parts of the exchange or even the whole exchange in your bean methods (this time relying on the Camel API because of the specific annotations).

To see the problem that we want to solve with this option, try now the application with the following URL: “http://www.w3.org/notfound”. It does not exist and you will notice that Camel retries the operation with an empty URL obtained from the redirect location property in the “HttpOperationFailedException” exception. The application fails with a “UnknowHostException” produced by the now empty URL.
Using the “bean” option, let’s see how to distinguish between any HTTP response code and a HTTP redirection code and avoid retrying the fetch unnecessarily. We can read this information by inspecting the “HttpOperationFailedException” properties. For this, create a class named “HttpErrorHelperBean” with the following content:

public class HttpErrorHelperBean {
    public boolean isRedirectionError(Exception exception) {
        return (exception instanceof HttpOperationFailedException) &&
                (((HttpOperationFailedException) exception).getRedirectLocation() != null);
    }
}

...
    .onException(HttpOperationFailedException.class)
    .onWhen(bean(HttpErrorHelperBean.class).isEqualTo(true))
        .log("Fetching URL failed: '${exception.message}', trying with relocation: '${exception.redirectLocation}'.")
...

Did you also notice something strange in the “bean” method construction? We indicated no method name or bean instance, but Camel seems to have instantiated the class and called the method we expected. What Camel does is try to find an instance of the passed class in the “registry” or instantiate the class if it could not find an instance (for this, the class needs to have a default no arguments constructor). For guessing which method to call, Camel applies a matching algorithm based on the method signature and the message payload type. If you want to know how it exactly works, you can find a detailed description in the chapter 4.4 of Camel in Action.

Exposing our extractor service as a Web service

Now let’s expose our logic as a web service. But first let me clarify: the kind of Web service we expose here is a plain and simple HTTP&HTML service (not a REST or a SOAP based Web Service).

To implement a web service, create a class named “HtmlFormatterBean” with the following code:

public class HtmlFormatterBean {
    public String asHtmlPage(List<String> contents) {
        StringBuffer stringBuffer = new StringBuffer();
        stringBuffer.append("<html><body><h1>Extracted contents:</h1><ul>");
        for (String content : contents) {
            stringBuffer.append("<li>").append(content).append("</li>");
        }
        stringBuffer.append("</ul></body></html>");
        return stringBuffer.toString();
    }
}

Add a new dependency in the Maven pom file:

...
    <dependency>
        <groupId>org.apache.camel</groupId>
        <artifactId>camel-jetty</artifactId>
        <version>${camel.version}</version>
    </dependency>
...

And modify the route builder adding this second route within the “configure” method:

from("jetty://http://0.0.0.0:8080/page_extractor")
        .setBody(header("extractUrl"))
        .to("direct:page_extractor")
        .bean(HtmlFormatterBean.class);

Wow! It was a short route but it had a big impact! By using the “jetty” endpoint in this way, Camel creates an embedded instance of the Jetty web server and starts listening to all the local IP interfaces, publishing the endpoint under the “page_extractor” web application context (“http://0.0.0.0:8080/page_extractor”). When you access this URL from a web browser adding the parameter “extractUrl=http://www.w3c.org”, the route creates a new message populating its headers with the different URL parameters and their values, and routes the message to the next step which happens to be the page extractor.

To send the request to our page extractor, we have to first set the value of the message body to the URL that we want to extract (which has been saved as a message header under the name of the HTTP parameter: “extractUrl”). To accomplish this, use the “setBody” method and, after this, the modified message is routed to the page extractor. When it comes back from the extractor, its body contains a list with the text chunks. The last step is to translate this payload into HTML, which is returned as a response to the HTTP request received in the “jetty” endpoint. If you do not like return HTML then instead of using a Java class you could also use the “velocity” Camel component.

This is the end of the second part of this series. In the next one, we introduce new functionality in a modular way and show how to test with the Camel testing framework.

I hope that you enjoyed the reading and that you come back for a new Camel ride in the next part.

Update: added source code for download. To run the application, just expand the file, change to the folder where the pom file is and execute: ‘mvn compile exec:java -Dexec.mainClass=”com.canoo.camel.Part2″‘

If you are a fan of “Enterprise Integration Patterns” or you work in data integration projects, you have probably heard about Apache Camel. If not, this is a good moment to discover it.
And what has Apache Camel to do with LEGOs? LEGOs offer a fixed set of blocks that you can combine to build some working gadgets. In that sense, Camel offers you Java implementations of the EIPs that you can mix and match to get some functionality done, without having to construct the blocks yourself. If you are open and eager to learn new ways of building Java applications, let’s start with some practical Camel examples.

The Minimal Camel Application

There are several different ways of deploying Camel: as plain Java application, as Spring-enabled application, as Java Web application, as OSGi module, etc. Let’s begin with a plain Java application and make our life easier by using Maven to grab the dependencies (the default option for Camel).

You will need a new Maven-based project and a “main” Java class:

public class Part1 {
    public static void main(String[] args) throws Exception {
        DefaultCamelContext camelContext = new DefaultCamelContext();
        camelContext.start();
        Thread.sleep(10000);
        camelContext.stop();
    }
}

The Maven pom file with the minimal dependencies required looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.canoo.camel</groupId>
    <artifactId>part1</artifactId>
    <version>1.0</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-core</artifactId>
            <version>${camel.version}</version>
        </dependency>
    </dependencies>

    <properties>
        <camel.version>2.6.0</camel.version>
    </properties>
</project>

First feature: the “web page extractor”

We are going to create a first route capable of downloading the page at a URL and transforming the contents into plain text. Basically, what we want do is grab the HTML, fetch all the paragraph elements in the page body, extract the text they contain and clean them by filtering all pure non-alphanumeric strings.
For that, we need a new class extending “RouteBuilder” that can be applied to the Camel context and that creates the route (or routes). The “PageExtractorRoutes” class would look like this:

public class PageExtractorRoutes extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("direct:page_extractor")
                .setHeader(Exchange.HTTP_URI, body())
                .log("Extracting content from: '${body}'")
                .to("http:extractor")
                .unmarshal().tidyMarkup()
                .log("Html from: '${body}'")
                .split(xpath("//body//p/text()"), new SplitterAggregationStrategy("(?s).*[A-Za-z0-9].*"))
                .log("Text chunk: '${body}'.")
                .end();
    }
}

Suppose that in the body of the message we receive the URL of a web page. We want to send this URL to an endpoint of type “http” that will access the page and grab its content passing the body of the message to the next step. The “http” endpoint can be set with the value of URL to access or, in case that the header “Exchange.HTTP_URI” is present in the message, it will use this value as the URL to grab. To put the URL in the expected header, we have used “setHeader” copying the body value. When using a dynamic URL set in the header, the name of the “http” endpoint is irrelevant (in this case we used “extractor”).

Once the “http” endpoint has fetched the content, we have to take care of some details before we can use XPath to select the text of the body paragraphs:

• Some endpoint types in Camel use streaming to avoid loading big pieces of data in memory. This means that the payload of the message is of type “InputStream” and, by default, can only be consumed once. This is the case with the “http” endpoint. If, as in our case, we know that the fetched web content is not going to be big, then we can convert it to a string that is consumable as many times as necessary (you can use “convertBodyTo(‘String.class’)” from the Java DSL) or we can activate the streaming cache using the method “streamCaching()” from the Java DSL at the beginning of the route.
• Because XPath needs well formed XML documents, we need to assure that the HTML content fulfills this condition. By using the “TidyMarkup” marshaller we achieve that and automatically our input stream gets converted to a DOM node that is a perfect fit for the XPath expression in the next step. Tidied markup can also be consumed multiple times.

If everything works as expected, we can now split the DOM document into text chunks. Therefore, we use a splitter (“split” Java DSL method) that takes as first parameter an expression and as second an “AggregationStrategy” (this is only one of the many forms of the split method). The expression is responsible for returning the pieces that the split method controls, and the aggregation strategy has in this case two roles: to merge the split pieces in the required way and to filter the unwanted pieces (non-alphanumeric text chunks in our case). The “.end()” method call belongs to the splitter and is used to indicate where the pieces should be joined, returning finally a unique message whose body is the aggregation of the previously split pieces.

Ideally, we would separate these two concerns by using the Java DSL method “filter”. But it seems the splitter is unable to detect when to finish the split process if you dropped some pieces with the filter, which we do. As a workaround for this, do all the steps together in the aggregation strategy. Note that the XPath expression is applied implicitly to the message body.

The “SplitterAggregationStrategy” class looks like this:

public class SplitterAggregationStrategy implements AggregationStrategy {
    private final String fFilter;

    public SplitterAggregationStrategy(String filter) {
        fFilter = filter;
    }

    public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
        List<String> result = null == oldExchange ? result = new ArrayList<String>() : oldExchange.getIn().getBody(List.class);

        Node node = newExchange.getIn().getBody(Node.class);
        String content = node.getNodeValue().trim();
        if (content.matches(fFilter)) {
            result.add(content.trim());
        }

        newExchange.getIn().setBody(result);
        return newExchange;
    }
}

public static void main(String[] args) throws Exception {
    final DefaultCamelContext camelContext = new DefaultCamelContext();
    camelContext.addRoutes(new PageExtractorRoutes());
    camelContext.start();

    ProducerTemplate template = camelContext.createProducerTemplate();
    String result = template.requestBody("direct:page_extractor", "http://www.w3.org", String.class);
    System.out.printf("Extracted: '%s'.\n", result);

    Thread.sleep(10000);
    camelContext.stop();
}

Last but not least, we need to complete the dependencies in our Maven pom file like this:

...
<dependencies>
    <dependency>
        <groupId>org.apache.camel</groupId>
        <artifactId>camel-core</artifactId>
        <version>${camel.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.camel</groupId>
        <artifactId>camel-http</artifactId>
        <version>${camel.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.camel</groupId>
        <artifactId>camel-tagsoup</artifactId>
        <version>${camel.version}</version>
    </dependency>
</dependencies>
...

To make this post short enough, we will end here and continue in a second part of a series. In the next part, we will introduce the following topics and features:

• Using error handling to control HTTP redirections and Java Beans for fine-grained logic.
• Exposing our extractor functionality as a Web service.

If you enjoyed the reading, I hope to see you in the second part.

Note: if you are interested in learning more Camel Enterprise Integration Patterns, maybe you should have a look at DZone Refcard: The Top Twelve Integration Patterns for Apache Camel.

Update: added source code for download. To run the application, just expand the file, change to the folder where the pom file is and execute: ‘mvn compile exec:java -Dexec.mainClass=”com.canoo.camel.Part1″‘