Say Goodbye to HTTP URI Query Parameters

New Year is approaching fast but there is still some time to make one more New Year resolution.

Traditionally, when it comes to expressing the search requirements with HTTP URI, one uses URI query name and value, for example:

1. "/search/people?age=30&age=40"
2. "/search/people?ageFrom=30&ageTill=40"

First query can be read like this: "Find all people who are either 30 or 40 years old", the second - "Find all people older than 30 but younger than 40".

Plain query parameters are very widely used and actually very 'capable', one can invent a query parameter such as "ageFrom" to indicate the conditional requirement. It is not quite perfect in that the actual operator is "=" but the name implies it is not "equal to" but "greater or equal to", but it works, and probably reads better in some cases.

That said, the bigger the number of terms, the more tedious it can become to support creating the parameters like "ageFrom" and writing the actual code for figuring out which parameter means whether it is indeed the equality check as in "age=30" or say a "greater than" check as in "ageFrom=30". Supporting the finer-level checks such as "greater or equal to", as well as combining multiple type of checks can indeed become difficult to manage at the code level.

Consider always using FIQL expressions:

1. "/search/people?_s=age==30;age=40"
1.1 "/search/people?age==30;age=40"
1.2 "/search/people/age==30;age=40"

2. "/search/people?_s=age=ge=30;age=lt=40"
2.1 "/search/people?age=ge=30;age=lt=40"
2.2 "/search/people/age=ge=30;age=lt=40"

Does it look more complex ? I don't think it does but I can agree it may look a bit unusual at first, though the syntax is actually very easy to understand after typing it  just for a bit of time.

Note a number of possible variations - in fact I guess I like options 1.2 and 2.2 most, the "optionality" aspect of URI query components does not make the whole URI represent a specific application state very cleanly (options 1, 1.1, 2, 2.1).

Also note the fine-level checks in 2.1/2.2 which are tricky to express with the plain queries, example, in this case it is "Find all people 30 years old or older but younger than 40 (exclusive)".

If you think about it, you have nothing to lose by starting using FIQL, for many cases it is nearly as simple and primitive as using plain query name and value pairs but the bonus comes immediately once you start expressing something more involved. As noted at the beginning of this post, plain queries are also capable in this regard - but the more complex the search requirements are - the less easy it becomes working with the plain queries.

Those Apache CXF users who have started experimenting with FIQL can utilize the existing FIQL converters, which is another plus point, consider the task of writing the code for converting plain query parameters to JPA2 expressions - it can be an involved task indeed - but with CXF JPA2 converter it is a walk in the park; as a side note - JPA2 converter now supports "count" extensions, for example, "find all the people living in London with more than 6 children".

Well, you may not be convinced that it is time to drop plain query name and value pairs. Never mind, just get CXF converting them internally to FIQL (which is very easy to do for CXF given that FIQL is a richer language, and still rely on the handy FIQL converters of your choice to interact with the data stores and completely generalize the search processing logic along the way. Offer different search endpoints for your users to enjoy working with, one accepting the traditional queries and another one accepting FIQL queries, with both endpoints using the same FIQL converters.

Still plenty of options to innovate in this well-explored space :-)!

Update: See also this earlier post from Abhishek Jain.

Happy New Year !

FIQL and JPA2 Queries In Action

I've been focusing quite a lot recently on enhancing CXF Search extension module, by improving the existing converters and creating the new ones, making sure the parser is configurable, flexible and capable of mapping arbitrary property names to the properties of the bean capturing the search expression, and improving the documentation.

Andy Michalec created a FIQL parser quite a long time ago, it's been around for a while really, but it is only since Jeff Wang provided an initial FIQL to JPA2 converter patch that it kind of struck me how important it was to ensure the utility converters for mainstream and most popular technologies for querying the data stores were available. 

With all the documentation and code improvements, it is still not quite good till it is actually demonstrated somehow, and this is exactly what I've spent the last few days upon, on enhancing the existing Talend ESB jaxrs-advanced demo to show how the client can type the queries with SearchConditionBuilder without having to type FIQL expressions manually and get the expressions seamlessly converted to JPA2 TypedQuery and CriteriaQuery, with Tuple mixed in, and the matching data returned to the client, all with the help of FIQL to JPA2 converter.

Let me comment on the actual demo enhancements in more detail.

First check the client code, the "useSearchService" function, starting from the line 194 and more specifically from line 205. IMHO expressing the query with the help of the fluent query builder (also created by Andy) is quite cool, it is definitely more descriptive with respect to what is actually expected, and it is the specific query language agnostic, it is FIQL by default but can be something else.
The builder can build whatever advanced query expression is needed, though practically speaking I would not expect massive expressions being created.

Now lets move to the server code. SearchService is one which actually handles the client requests and all it does it delegates to PersonInfoStorage which deals with querying the data. Note, SearchService works with CXF SearchContext  by expecting it to extract the search expression from the current URI query component (default mode) or getting the expression from URI path and submitting it to SearchContext.

The main and in fact the single demo domain entity is Person but it is the one which is recursive, with children and parents, ancestors and descendants linked to. I haven't modified these relationships to get CXF JPA2 converter working properly, the only thing I did I added a couple of missing Person setters, added JPA2 annotations and also a hack required for JAXB RI capable of dealing with recursive structures (see the end of Person code).

The Person model is initialized in PersonInfoStorage init method (see the end of the file), where the injected JPA2 EntityManager is used - it could've been done elsewhere but was good enough for me for the purpose of the demo.

Next check getTypedQueryPerson (line 68) - see how straightforward it is  to get the FIQL expression transparently converted to JPA2 TypedQuery, the comments in the code should make it very easy to understand.

Note, Person is a recursive and possibly a very deep structure and one may not necessarily want to fetch all the Person representation back to the client. A number of options exists for dealing with this issue including using the intrusive JAXB XmlTransient annotation but using JPA2 Tuple is one of the most elegant ones.

Using Tuple is one of JPA2 options for having the response shaped into simpler or different structures and getCriteriaQueryPerson  method shows how it can be easily done by having the query returning the data sufficient for initializing a simple PersonInfo representation. One needs to get OpenJPA auto-generating the
metamodel classes for the tuple selections working and you can see how it can be done here, this tip helped me a lot. 

Also note that PersonInfoStorage has the injected bean properties passed to SearchContext. Why this may have to be done is explained here but in short it lets to get the property names used in query language completely decoupled from the corresponding properties of the capturing bean, for example, in the demo case, the client can type "builder.is('childName').equalTo('Fred')" and have it working with the 'childName'  correctly mapped to "children.name", where "children" points to a Person collection of children. This makes it simpler and easier to work with for the client, while keeping the internals of how the properties are actually linked to each other completely opaque. That is cool, I can hear you saying :-)

Finally have a look at how JPA2 EntityManager is initialized and injected. I started with arguably a simpler approach, I basically used Spring ORM module to get the entity manager wired in, see this beans.xml.  This actually back-fired on me when I tried to make the demo working in Karaf - Spring ORM needs to see persistence.xml and is not capable on its own of inspecting OSGI Meta-Persistence property set up in the common module's pom.xml so I just ended up duplicating persistence.xml in both common and service modules.

On the TODO list is to follow an excellent tutorial from Christian and make the demo working with OSGi JPA Service. Perhaps someone from the community will be interested in doing a related pull request ?
  
So this is it and hope you'll find something interesting in this demo, enjoy !  

Use FIQL to query LDAP and OSGI containers

It has taken me about 90 minutes to write an initial FIQL to LDAP converter which outputs the query formatted according to RFC 4515 and document it here.

I have copied most of the boiler-plate code from the FIQL to SQL converter and given how simple the LDAP query is it was very easy to finish it off fast. The point is that other custom converters (example FIQL to CQL) can likely be written even faster by copying and pasting the LDAP converter :-).

More work will likely need to be done to get RFC-4515 supported completely, but the converter should already be capable to support many simple or composite LDAP queries which do not use complex characters, etc.

You may want to ask why would not one support LDAP query language by directly encoding it into request URIs. There are two issues here, first, the LDAP language is not quite URI friendly, the second one is that it will leak the details about the data technology supporting a given REST endpoint.

This is why FIQL or other dedicated language like OData is the way to go for doing medium complexity queries over HTTP. It is easy to read and it lets the users completely encapsulate the data source details.

I think with this latest update CXF Search extension has enough 'material' for users to start experimenting with using FIQL on top of the well-known and traditional data sources.

I can nearly see some of you starting thinking of creating some cool UI search module which will let users do more interesting search queries over the containers such as LDAP stores or OSGI runtimes with this new converter and may be even sharing the tested queries with the other users who may find it useful.

The next phase is to get FIQL a bit closer to all those No SQL data stores and it should be quite interesting.

Watch this space :-)

[OT] Just Say Yes to CXF

This is a regular, once or so per year, totally off-topic post dedicated to linking the music I listen to to CXF :-)

The New Year is coming soon, time to start thinking about the next year, about the decisions which will make the professional life of developers working with web services front-ends something to really look forward to.

It has never been easy to choose which framework to use to get web services up and running. It is tough. At a decision time like this one, when it is cold outside, it is good to listen to Snow Patrol, which is one of my favorite bands.

Some of their music is absolutely brilliant. Listen to it, think about it, and Just Say Yes to CXF.

I've had some fun writing this post :-), hope you'll have some fun too !

How to refresh OAuth2 access tokens in CXF

OAuth2 Refresh Token grant lets OAuth2 clients owning an access token refresh it with a new access token if the current attempt to access the end user's resources has failed.

Refresh tokens offer an advanced support for the OAuth2-protected applications to force the clients to re-authenticate regularly without forcing them to go via the re-authorization step involving the end user which may not always be practical. 

They can also help with effectively getting the original access tokens revoked and replaced with the modified access tokens, example, with the new opaque scopes limiting or extending the current client permissions or even replacing the token type itself, example, replacing Bearer with MAC, all dynamically.

During the refresh process, the actual refresh token may also be 'renewed' with the new value by having the so-called key-rotation scheme implemented.

One possible approach is to have a short-lived access token and long-lived refresh token. Whenever the access token expires the client is forced to refresh with a valid refresh token and re-authenticate along the way.

I'd like to encourage developers to check OAuth2 archives on some interesting insights from the security experts on why using refresh tokens might benefit a particular OAuth2 application.

All this sounds good but you may want to ask, how to actually work with the refresh token in practice, specifically with CXF ?

It is a child's play though I guess I should not refer to such an advanced feature like that :-).

Whenever your custom  OAuthDataProvider returns a new ServerAccessToken, a client-centric ClientAccessToken representation is returned to the client. At this point of time the data provider may choose to offer a refresh token too, which will be made available to the client.

On the server the refresh tokens are bound to a specific Client instance,  I'm considering updating the Client model class to actually keep a list of refresh tokens, I'm not exactly sure yet if it will make it easier for the implementers or not, but in meantime one would need to keep a dedicated table joining client id plus the refresh and access token pairs.  

So, the client will use the access token to access the end user's resources and at some point the access request will fail due to the access token expiring. All one needs to do now is to issue a Refresh Token grant request to OAuth2 Access Token Service, and then do a single retry using a newly obtained access token.
This is all to it.

Please check the Client-side support section for the new example showing this retry attempt pseudo code, as well as review on how well-known OAuth2 grants can be currently supported in CXF.

Enjoy!
    

Latest WADL To Java enhancements in CXF

Dan has released CXF 2.7.0 which has some major enhancements including the addition of the asynchronous HTTP conduit and initial support for most parts of JAX-RS 2.0 (the topic of the next post on this blog).

What I'd like to mention in this post is the few enhancements to CXF wadl-to-java code generator, added thanks to the colleagues from Talend ESB team who have been stressing the generator to the limits :-).

One of the issues with processing complex WADL documents is that it is not obvious whether (URI) path parameters attached to one of the parent resource elements have to be inherited when generating the method signatures for descendant resources, for example:

<resource>
<param name="id" style="template"/>
<method name="GET">
</method>
<resource>
<resource>
<method name="POST">
</method>
</resource>
</resource>
</resource> 

Does the user want a generated method signature corresponding to "POST" contain an 'id' parameter or only have it added when a "GET" method available directly under the parent resource is processed ?  By default CXF does not add the current parameters to all the descendant methods. Note, WADL provides a mechanism to link to global parameter declarations from within individual methods but if restructuring a given document is not ideal then a new "inheritResourceParams" property will help.

Here is another issue one may face. Suppose you have an XML representation in the schema and the generated code will reference a JAXB-generated class name, say, "books.Book" class. What if you'd like a JAXP Source be used instead ? And what if you have a representation which has "multipart/form-data" payload, it is not 'fair' having no reference to whatever class may be able to handle this payload added to the generated code.

No problems, it is going to be easy with a new "repMap" (the representation map) property, for example, it may have pairs like "application/xml : javax.xml.transform.Source" and "multipart/form-data : org.apache.cxf.jaxrs.ext.multipart.MultipartBody". Note this property is effective when a given representation has no actual reference to a schema element, which is the case for all non-XML representations.

Now, suppose you have a jaxb binding document overriding the name of the class to be generated by JAXB and the generator missing on it,  or you have to 'override' an XML representation linking to the scheme element for the latter be handled by JAXP Source. This is where a new "tMap" parameter will become handy.

Please experiment with the above options, I know some users have already tried them and hope they will be of real help.

As far as WADL itself is concerned...As you know it has not been made a standard yet. However with the original wadl-to-java project at Oracle being under the active development and the CXF generator being regularly enhanced it is likely to become more and more useful tool for testers and for users of WADL-'aware' RESTful endpoints. Don't be concerned too much that WADL is not going to accepted at the universal mechanism for describing and working with RESTful services - it has its niche and hopefully it will make it into becoming a final recommendation one day.
   



 

OAuth2 MAC Access Token support in CXF

OAuth2 offers a clear differentiation between token grants and token types. Grant is what Access Token service will verify before issuing a token, and the best thing about it is that the same code path is used irrespectively of whatever grant or token is used.

OAuth2 mentions simple bearer tokens as default token types, but of course the bearer is not the only token type possible.

Eran Hammer-Lahav wrote a MAC Access Authentication draft  which introduces a MAC authentication scheme and describes how it can be used in OAuth2.

OAuth2 experts are considering whether this effort has to be completed or not. I'd like to encourage those who are interested to check OAuth2 mail archives for different opinions expressed regarding the MAC scheme.

Here is my opinion for whatever it is worth:

- What is important is that OAuth2 offers a pluggable mechanism for different token types so I think users should be encouraged to experiment with new token types, MAC in this case, and see if it makes the flows more secure, and 'contribute' to the idea of completing the effort from Eran by talking about possible issues or advantages of this scheme (such as: is it simpler to operate than tokens like JWT, or is it a reasonable alternative to JWT, etc) - remember, it is still the same code path for client and server applications, only a different token type.

- Having it completed will encourage OAuth1.0 users migrate to OAuth2.0, because OAuth2.0 with MAC is a much simpler version of OAuth1.0 - MAC scheme offers a simple holder-of-key support and combined with HTTPS it makes the exchanges between clients and OAuth2 services more secure.

- It will be a good idea to complete the effort any way, some implementers will ignore it but some will like.

Sasi M. has contributed the initial MAC Token implementation to Apache CXF, thanks! The documentation is available here.  Please try this token type in the applications, it is really easy to work with. Also, we'd be open to contributing the MAC-specific code to a dedicated 3rd party module if it will be useful to have the one - the details can be discussed at the CXF dev list. That said - it is just a couple of utility classes that are needed to get a MAC token created and validated. Sasi also provided a patch for a MAC nonce validator - this is not at the trunk yet but will make it there in time.

Enjoy!

OAuth2 Demo in Talend ESB

Talend ESB ships many interesting, advanced examples demonstrating CXF and Camel in action. The demos attempt to show something interesting, something that one may try to do in the production.

JAX-RS OAuth2 demo has been evolving as a POC demo with the main goal to stress  CXF OAuth2 services and make sure that they can cope with what one might want to consider as a medium complexity OAuth2 deployment.

The demo shows 4 parties cooperating with or depending on each other, where Social.com service offers the registered users an option to reserve a table at the favorite restaurant with the help of Restaurant Reservations service (with the latter depending on its own partner).  A classical 3-leg OAuth flow (Authorization Code Flow in OAuth2) is demonstrated with most of the demo depending on CXF alone to do most of the work (OAuth2, Presentations, etc).

The end user (individual Social.com resource owner) can approve or disapprove a client (Restaurant Reservations) request to read and possibly update the user's calendar.  The demo can be run as a simple OAuth2 demo with all the endpoints running within the same container or as an advanced demo showing how a user can interact with the application supported by many application containers, with the SAML SSO feature introduced to improve the user experience - this advanced option will be covered in detail in one of the future posts.

Please check this presentation I did at JAX-2012 in Mainz for more information.
I'd like to encourage users to try the demo, provide the feedback and git pull requests :-)

Enjoy.

Master Kerberos Security with Apache CXF

Kerberos is a well-known security protocol, originally developed at MIT and has been a major authentication protocol on Windows.

Why would you want to learn about or experiment with Kerberos today, when developing web services ?

One may want to do it if we have a web service which needs to expose the information available from the internal Kerberos-protected store or when a Single Sign-On service needs to use  Kerberos servers to keep the principal details or when Kerberos is deemed to be the best authentication protocol which can protect the given complex application exposed as a web service. 

The decision by Hadoop developers to support Kerberos will undoubtedly make it more important for developers to understand what Kerberos is about, due to the fact the Big Data is becoming so important these days.

In Apache CXF, Kerberos has been supported on a number of levels for a while. Colm has published a two-part series about the way WS-Security Kerberos is supported and tested in CXF, and Christian has implemented a client-side support for the HTTP Negotiate authorization scheme. 

Starting from CXF 2.6.2 (to be released soon), the JAX-RS frontend offers an additional server and client side support for making it very easy to support the Kerberos authentication for RS endpoints and clients. 

After installing Kerberos packages, the next thing you can do is to run JAXRSKerberosBookTest or add server and/or client Kerberos handlers to your own application as documented at the wiki and see what actually happens.

Have you been thinking of getting started with Kerberos for Web Services ?
Do it today with Apache CXF :-)

Jettison 1.3.2 is out

Jettison 1.3.2 has been released this week, please check the Download page.

Those who try to customize the way Jettison works should find it easier to override various Jettison classes, for example, in CXF I've been able to remove about 50 lines of code I had to copy earlier on to get large Jettison sequences optionally restricted.

Jettison will no longer require a namespace map set up for the serialization to work, in cases when it is not configured to ignore namespaces, as I moved a fix provided by Benson from CXF to Jettison.

A number of minor performance enhancements have also been implemented thanks to the proposals from Fabian Lange.

During the next  few interactions we will continue minimizing a number of outstanding issues, however I'd also like to encourage the community to provide patches - they will get an immediate attention during the release, as it's been the case during the last few releases. I haven't resolved one issue with the available patch to do with treating XML attributes as elements, in CXF we can manage it by providing a custom XML writer, so I needed more time on investigating if that issue can be resolved at the current Jettison implementation level effectively or not.

At some point in the future an optional streaming support will have to be introduced, either on read or write sides, even if certain restrictions will have to be there.

In meantime, please keep stressing Jettison and provide the patches :-) 

CXF Log Browser Demo

A Log Browser demo has been available in the CXF distributions  for more than a year now. This demo is based on the brilliant contribution from Thomasz Opanovicz done as part of his GSOC project.

What I would like to do is explain what exactly the CXF Log Browser can do right now, and suggest some ideas on how it can be enhanced.

At the moment, the browser can be used to poll the Atom-enabled management endpoints and display the available log entries, this can be done per every CXF service, JAX-WS or JAX-RS one. The Atom endpoints can be set up as described in this section and shown in the demo. Have a look at this image on how the browser can show the log entries, I think it is quite cool.  Note the entries are paged, example, one can go to the next, previous, first or last pages, from the current page.

At the moment the browser polls every time the "refresh" button is pressed - this is one minor thing that can be customized further, example, a browser can be asked to poll every 60 seconds.

The browser offers an option to restrict which log entries are displayed by letting users to set FIQL search queries.

Typically, the Atom endpoint (which handles the browser queries) captures the log events, transforms them to Atom feeds and keeps them in memory or overflows to a registered ReadWriteLogStorage if needed but what is interesting it can also be configured from the start  to read the log entries from the existing log files.

For example, it can be configured to check the directory where Karaf gets the log files created. CXF offers a file-based ReadableLogStorage interface implementation  which can check the existing log files matching a given naming pattern, figure out which log file is newer or older and get the log entries in whatever format is used to format them using the simple configuration involving no regular expressions at all. For example, have a look at this test which works with the log files available here as well as this configuration file (check the bean with id "storage3").

All of it needs to be more extensively tested but Log Browser is already offering a fairly involved support for working with the available log entries.

However, quite a few enhancements can be done which can make it more useful and eventually turn the browser into a management console of its own.

One obvious thing which is missing is the ability to capture events immediately, with a browser acting as a receiver. For example, it would be good to get Atom push endpoints sending the events to it too. Implementing an enhancement like this one would very likely lead to Web Sockets supported in CXF. The browser would be configured to support either push or pull style on the startup or perhaps it would keep the pull  style by default and get the real-time events displayed in the running line at the bottom of the window.

The browser can also offer an option to get the log files from the selected location downloaded to the local disk. 

We can also have at least two more tabs available. One would show the JMX statistics related to CXF, another one would show the message exchange details.

I'd like to encourage users to try the demo, and consider helping us with enhancing the browser further.  

Advanced queries involving multiple entities

As I've mentioned a number of times, FIQL can help with expressing the advanced search conditions in a compact and easy to understand syntax.

The queries like "find all the books published before a given date" are very easy to type in FIQL and extending this query with a restriction like "and having the page count between 80 and 100 pages or less than 20" is quite straightforward too, manually, or with the help of the client FIQL builder.

However, what if one would like to do a query involving multiple entities, for example, "find the library in a preferred location where all the books published before a given date and written by more than 2 authors are available" ? Or imagine or more interesting query if you'd like.

This bit, "where all the books published before a given date and written by more than 2 authors" is an easy one to support with the CXF search extension. The question is, how to structure the advanced query such that Book entities matching the arbitrary complex query are selected first and then a Library entity is checked per every Book on whether it matches its own selection criteria or not.

Another question is how to express the requirement that it is actually "the list of books",  "published before a given date and written by more than 2 authors and available in one of the libraries in a preferred location", that has to be returned to a user.

As you can see, there are at least two main requirements to deal with. First one is how to let users choose which entity (among a number of entities being matched) is actually returned which affects the response representation. Second one is how to do an advanced query involving multiple entities at the level of the JAX-RS application handling such a query.

The good news is that the first requirement can be easily managed with the help of URI path and/or query components. For example, an expression like "/books[date=le=2012-02-11]/library" can be used to find all the libraries having the matching books, and the one like "/library(dist=lt=10;dist=gt=5)/books(date=le=2012-02-11)" can get all the books matching a given search criteria available in a library within a 10 km range from some well-known location (such as the city center) but not closer than 5 (given that the parking is not free within a 5 km range).

It is really up to the designer of the service how to get the actual expressions captured, which characters to use to mark them, example "[" and "]", "(" and ")" or ";" and how to support the selective retrieval of the entities involved in the multi-entity search. See this section for more examples.

Yet another good news is that a complex multi-entity query can be managed pretty easily at the JAX-RS resource level, see this section for a number of options on how it might be done. Note how multiple FIQL expressions can be handled.

What is mentioned there is that it can be more optimal to get say JPA to execute a JOIN like query at a database level in one go as opposed to getting it to find a list of entities matching the first FIQL expression, then doing the follow-up in-memory match against the selected entities.

Right now CXF can not automate the process of creating a JPA TypedQuery or SQL expression which can 'span' the multiple entities - this is the next task, and in meantime please experiment with the demonstrated approach and see how it fares against the one involving the composite JOIN-like queries. Alternatively, introspect the captured search conditions as shown in this section and build a composite query in the code.

Please also have a look at the OData4J project and check how the advanced queries can be managed there. 

Finally, I'd like to answer on the following question: why one would worry about offering such a search interface to users, would it not be simpler to offer a Google-like search interface where one enters a few words and gets the list of matching data pages ? I think that when one creates a service for exposing the data with the known properties and relationships, the opportunity is there to offer an optimized search engine for users to get the customized search experience.

JMS Transport support for CXF JAX-RS clients

I blogged about the support for JMS by CXF JAX-RS endpoints two years ago.

The main reason behind making the JAX-RS frontend (associated by most users with supporting HTTP-based communications) JMS-aware was to do with getting the most from the 'investment' made into implementing the RESTful services on top of CXF JAX-RS.

If one has the resource code relying on the JAX-RS runtime to make the inbound data delivered to the right method and easily consumable in the form of a given JAXB bean instance, then the possibility is that this code can work equally well when the data comes to this resource handler either via HTTP or JMS or indeed some other transport, example, CXF Local Transport.    

CXF is super-flexible in the way it can support multiple transports, and it does make sense to get JAX-RS based endpoints optionally supporting non-HTTP transports.

And it is exactly for the same reason that CXF JAX-RS proxies (and I have to admit, WebClient :-) - simply because proxies and WebClient are using the same base code) also optionally support JMS transport now, thanks to the contribution from Willem.

So why would one want to use CXF JAX-RS client code with JMS ? As noted above, it is primarily about making the same code, client one in this case, re-usable in different contexts. I could've noted again that REST principles can probably be applied even to RS-232 endpoints if really really needed :-) but I'd rather focus on the re-usability aspect.

For example, consider a newly introduced HTTPSPStateManager implementation for making it easy for users to quickly set up a centralized distributed SSO cache endpoint. You've tested it and seen it working with HTTP. Now the time has come up for you to move to a more sophisticated distributed cache implementation for it to scale really well in the production. But then you realize that by simply getting HTTPSPStateManager proxies and the endpoint addresses use  a JMS scheme you can get an entirely different cache support, still using the same configuration, so you are going to give it a go and see what happens.

Or perhaps you have a Camel JAX-RS endpoint, supported by Camel transport and jaxrs:endpoint or CXFRS bean or endpoint, supporting both HTTP and JMS. It would be cool to get the same Camel CXFRS client to be able to talk to the endpoints using either HTTP or JMS.

I'm hoping users can come up with more interesting cases which will be possible thanks to the optional end-to-end support for JMS in the JAX-RS frontend.

Enjoy!    

How to test CXF JAX-RS endpoints

Users have been asking during the last couple of years how to test CXF JAX-RS endpoints. One of the users from the CXF community would always point to either a blog entry or paste a code example showing how the endpoints can be tested easily enough.

The problem has been all the time that there was no any documentation on the CXF JAX-RS wiki specifically describing the steps required to get the tests set-up and running.

A new wiki page has been added recently and it documents how the endpoints and indeed the consumers can be tested easily by using either the embedded Jetty or the newly added support for CXF Local Transport. The latter option is quite cool is that it does let to test the complete end-to-end invocation without spending the time on setting up HTTP mocks, by only having an address value to include a "local:" URI scheme.

This wiki page will be expanded to show how the tests can be run in the different test environments. Let us know please if you have some material ready to be contributed to the wiki :-)

From FIQL expressions to typed JPA queries

Imagine you are working on the RESTful service implementation that relies on JPA2 but having a bit of a difficulty coming up with an interface that can neatly let users search the service data with queries allowing to express something more interesting than just "find all the data that have a given property equal to a given value".

Yes, FIQL is fantastic and its ability to express complex queries in a fairly simple and compact way is hard to beat.  CXF  offers an option to convert the captured FIQL expressions to other query languages and representations and now JPATypedQueryVisitor for converting FIQL queries to JPA2 TypedQuery or CriteriaQuery objects has been introduced. This can help users with creating typed JPA2 queries easily and getting the most of their JPA2-based services.

Major thanks to Jeff Wang for driving this enhancement with his contributions.

Next we will try to support hierarchical queries, for example, find all the chapters with a given content from books with a given set of properties or find all the books with the given properties that only have the chapters with a given content, etc. You can expect other useful converters, example for Casandra CQL and Lucene query languages, being added in time. Finally, the support for the alternative URI query languages is also on the map.

Watch this space :-)

SAML Web SSO for CXF JAX-RS endpoints

Last week Colm announced that support for SAML Web SSO profile was available starting from CXF 2.6.1. 

We have also created a new CXF wiki page dedicated to describing the way the CXF JAX-RS applications can be protected with SAML SSO filters.

Colm has done a comprehensive testing against many popular IDP implementations which support SAML SSO and we believe that CXF offers one of the most configurable and interoperable SP SAML SSO implementations, even at this early stage.

I think it is a pretty major milestone for CXF JAX-RS and for CXF Security overall. CXF already offers a production quality WS-Trust STS implementation, Oliver has just finished the first release of Fediz. Having  a quality Service Provider SAML SSO support will help users get CXF JAX-RS endpoints integrated with the enterprise security services which is very important.

This SSO support and other SSO implementations to be supported by CXF will also make it easier to build more distributed OAuth2 applications.

We are planning  to work on a demo demonstrating SAML Web SSO in action. Stay tuned :-)

RESTful Data Updates with Talend Studio

I created a presentation few months ago which showed how one can create a simple DB browser for exploring the arbitrary databases using Talend Open Studio for ESB.

The Customers DB explorer we created last time was supporting GET queries allowing to return the list of all the customers and the individual customer records.

I've worked recently on enhancing the job that we created last time for supporting more involved GET queries as well as data updates with POST, PUT and DELETE verbs. The presentation is available here.

The demo shows the original ReadCustomers DB job, as well as ReadCustomersAndOrders one which additionally manages GET requests for all the orders of the individual customers and the individual orders only.

The CRUDOrders  job shows how to create the new orders for specific customers, update and delete the existing orders, and query the list of all the orders for all the customers. Check how easy it is to configure the job to return a status like 400 in case of attempting to add an order for a non-existent customer.

Note that in this demo I did not really focus on showing how the individual jobs can be created from scratch, in order to keep it shorter in time. There are many other presentations available from Talend Channel, in series like this one, which show how various Talend components can work together, please follow one of those presentations or this one to get more information about it.

In this demo I really wanted to highlight that REST components can manage different HTTP requests (queries, updates, etc), with different, possibly overlapping URI paths and data payloads.

The tRESTRequest component linking HTTP verbs and URI templates is based on the JAX-RS matching algorithm and you can see how flexible the configuration can be, with as many capturing templates as needed added to the relevant HTTP mappings.

More work will be needed for it to support the richer set of media types, in addition to XML, JSON and forms. More flexible support for identifying newly created resources will be needed too.  We will be showing the relevant presentations in time.

I'd like to encourage those who are interested to play with creating RESTful jobs and provide us with the feedback.

In meantime I'll work on creating a demo showing how a RESTful job can be packaged as an OSGI bundle and deployed into Karaf with HTTPS and Basic Authentication supported by the container.

Stay tuned !

Controlling Large Payloads in CXF

You may have already read that so called hash-collision attacks may affect many of the existing Web applications. A massive form, XML or JSON payload with specially constructed keys can be posted to the service and cause the denial of service situation due to the fact many of XML, JSON or form payload processors depend internally on the map implementations that are not capable of dealing with such data effectively.

It does not have to be a specially ill-prepared payload. Huge XML or say multipart/form-data application payloads with open-ended list of child elements or parts may significantly slow down the services too.

There has been a number of announcements from the major projects on the workarounds that may need to be taken, for example, please check this Tomcat issue.

Security has always been and going to be  a major story in Apache CXF and obviously we spent some time on making sure CXF endpoints can be adequately protected in such cases too.

For a start, we followed the Tomcat's lead and introduced a "maxFormParameterCount" endpoint property then can be used to restrict a number of form name/value pairs passed in the HTTP POST body.

We also introduced a DepthRestrictingStreamInterceptor that can be used to enforce a number of limits on the incoming XML payloads such as: the total number of elements, the maximum number of child elements and the maximum stack level. More fine-grained, element-specific limits may be supported in the future. Note it was already possible to protect CXF WS endpoints with the relevant system properties before the introduction of this interceptor.

Additionally, CXF JAX-RS endpoints and JAXB-based providers can be individually configured with the limits that will be imposed on the incoming payloads. Note that even JSON payloads can be controlled by the default Jettison-based JSONProvider.

HTTP 413 will be returned whenever the limit is reached.

Finally, as has already been mentioned on this blog, we had a high-quality contribution which made it possible to control the attachment limits by using an "attachment-max-size" property.

Please check the CXF Security and JAX-RS DataBinding sections for more information, test with the recently released CXF 2.6.0, 2.5.3, 2.4.7 and 2.3.10 distributions and provide the feedback.

Custom JAX-RS Contexts in CXF 2.6.0

CXF 2.6.0 is due to be released soon, with several new features likely to get the developers interested.

CXF is about to become much more OSGI-friendly which will open the way for many new interesting enhancements to come thanks to Dan and Christian leading this major refactoring effort.

The JAX-RS frontend has benefited (as usual :-)) from the core CXF improvements. One of the positive side-effects was that the initial go at splitting the fairly big JAXRS module was attempted.

Most of the optional JAX-RS providers were moved with all their optional dependencies to the new cxf-rt-rs-extension-providers module, Christian's refactoring of the clustering feature helped to drop the JAX-RS specific extension with JAX-RS clients now being able to use same fail-over feature configuration as their JAX-WS brothers :-), and the code to do with the WADL to Java code generation made its way to its real home, the cxf-tools-wadlto-jaxrs module. The CORS code now lives in cxf-rt-rs-security-cors, it needs to as enforcing the CORS across multiple servers will very likely require more enhancements to the current filter and the couple of annotations.

Finally the FIQL search extension code got moved to the new cxf-rt-rs-extensions-search module. I think this extension has a lot of offer and more enhancements will start coming in sooner or later due to the power and simplicity of FIQL.

Moving the FIQL code presented a challenge, how to get the core JAX-RS frontend to populate the SearchContext offering an optimized access to the FIQL queries ? The name of this custom JAX-RS Context class is hard-coded within the frontend in the earlier CXF versions but with the extension now moving away it was not an option any more.

The new ContextProvider extension was to be the answer I was looking for and here is the implementation which creates a SearchContext instance by relying on the CXF Message class which has all the information about the current request. The last thing that needs to be done is to get the ContextProvider registered with JAX-RS endpoints.

It is actually quite a major enhancement, now the users can inject whichever contexts they like. For example, JAX-RS 1.1 HttpHeaders context offers a number of utility methods for accessing the HTTP headers. Lets say you'd like to help the application developer to handle Origin headers but HttpHeaders can not help. Well, write a custom OriginHeader ContextProvider, extract the Origin out of the message and make it really easy for the application developer to access various Origin parameters. You can use ContextProviders whenever you'd like to offer an optimized access to some of the information available in the current request.

I believe Jersey was offering a similar extension probably from the very early start. One can not deny Jersey was an absolute star :-) at the start of the JAX-RS, and they continue to be the major JAX-RS implementation, but CXF just keeps catching up even though it took us a bit of time to get to the ContextProvider.

Note ContextProvider looks similar to JAX-RS ContextResolver but they actually serve different purposes with the latter meant to simplify processing the data possibly involving the custom media types, see this method. I believe it was originally introduced to handle custom JAXBContexts. After thinking a bit about reusing ContextResolver I decided not to in order to avoid possible conflicts.

Finally, having to register the ContextProvider providing SearchContext instances led to another enhancement request. In CXF one needs to explicitly register custom JAX-RS providers and I believe it works well most of the time due to the flexibility offered by the explicit configuration approach. However, having an option to get simple basic providers auto-discovered would be useful too - thus a new enhancement request to support the optional class scanning is now pending.

It is always useful to refactor and simplify code - more often than not it has the positive side-effect of the new features added :-)

All Roads Lead to Mainz

They say all the roads lead to Mainz. Indeed, this year it is the home to JAX-2012, a very popular conference for Java developers in Germany.

The Talend team which includes Bernd Trops, Principal Consultant with Talend, Christian, Oliver and myself will be there. We are going to present few sessions during the Apache Integration Day on 18th April, so be there if you can :-)

I'm a bit nervous given that my German is not perfect :-), but with Bernd, Christian and Oliver all talking German I should be fine :-)

Simple DB Browser in Talend Open Studio

The screen-cast showing how to create a basic DB browser within Talend Open Studio is now available on the Talend Channel.

This presentation is more involved than the one showing how to create a simple REST endpoint. This time a basic database containing a list of customer records is exposed via HTTP and we go through a number of steps showing how the task of creating an arbitrary DB browser can be approached.

Initially the complete list of records is returned and then the optimized one containing smaller records but with links pointing back to complete individual records is returned.

What are we going to present next is the job which also supports the various data updates with a bit more complex database involved. Some options for dealing with exceptions will be shown. I haven't mentioned that JSON is also supported by default so we'll show the way it is supported out of the box too.

As I've already mentioned before our REST Tooling project is an ongoing effort and there will be quite a few things enhanced and improved over the near future. We are brainstorming with my colleagues various ideas such as how to make it feasible to bind the data to views in the most transparent way, how to secure the services, how to make the integration with other Talend components more complete, etc.

So I expect us producing quite a few exciting demos over the time :-)

In meantime, please view the presentation and provide us with the feedback.

Distributed OSGi RI 1.3 is out!

The signs are that the fortunes of Distributed OSGI are looking good.

Distributed OSGI RI based on Apache CXF (Apache CXF DOSGi RI) has been around for a while, and quite a few OSGI developers have experimented with and built custom applications on top of it successfully.

However, it's been more than a year since DOSGi RI 1.2 has been released and this project has been inactive recently. In meantime, two more Distributed OSGi implementations have been announced by two OSGI heavyweights, one by my colleague JB, and another one by Guillaume Nodet.

Now, as far as Apache CXF DOSGi RI is concerned, we are seeing users asking the questions quite regularly and this is a sign this implementation and the whole idea of the Distribured OSGI is of interest to some OSGI developers, more on it below.

So after getting some of issues reported against DOSGi RI 1.2 for the last couple of months, we have released Apache CXF DOSGi 1.3. Please see the release notes for more information (note there is a minor typo in the release notes, it is CXF 2.5.2 which this release is based upon, not CXF 2.5.1).

The major improvement in this release is that it is now possible to register custom CXF interceptors (pre-configured if needed) as service properties with the underlying JAX-WS and JAX-RS frontends.

WSDL-first approach is also supported now which is a good news for SOAP developers, see this project for an example. Of course, the JAX-RS frontend was trying to offer something similar :-), so a new org.apache.cxf.rs.wadl.location property has been added. Please see this updated page for more information on all the new properties.

If you are an existing user of the DOSGI RI then please try this new release.

If you have never tried it and wonder what is the story with DOSGI then try it too. DOSGI RI is quite sophisticated in that not only the basic endpoint and consumer creation is supported but also a mechanism for the distributed discovery is wired in.

But it is this fact that the OSGI programming model is used to drive the creation of the web service endpoints and consumers which is appealing to some developers and that is what one should focus upon first when experimenting with DOSGi.

If you think about it, the way to create a new web service endpoint or stop the old one in OSGI is typically to deploy a new bundle or stop the existing one from the shell or possibly from the UI management console. I guess it is quite rare that the custom application bundle will deal with updating the bundles itself.

In DOSGI, the creation if web services endpoints and consumers is actively driven by the typical OSGI BundleContext and ServiceTracker calls. If this style of managing the web services indirectly by the custom application registering or looking for OSGI services does appeal to you then DOSGI could become a perfect fit for your project.

In DOSGI 1.3 we fixed some basic blockers to get the project active again. The future releases will likely focus on making the distributed discovery working really well and also on improving the way the custom configuration can be applied.

One more thing which I'd like to mention is that if you are interested in OSGI in general and possibly in DOSGi and looking for a way to get involved in the open-source project and make a difference then please think of contributing to this project.

OAuth without the end user explained

One is the most confusing things in OAuth is a so-called 2-leg OAuth flow where an explicit authorization step involving the end user pressing an Allow or Deny button is not taken.

There are many resources on the web explaining what is an OAuth 2-leg flow. Most of those explanations are effectively describing the process where the 3rd party consumer accesses its own space on the resource server, possibly with the end user itself 'hiding' behind such a consumer.

But the classical OAuth is about the 3rd party consumer being able to access one way or another the resources of the end user. How does a 2-leg flow gets into the picture ?

Please read this blog entry. This is the best explanation I've seen so far and it was so good I had to stop doing my current task immediately and quickly update the CXF OAuth 1.0 code to be able to handle all the variations of the 2-leg flows better.

I think this 'pure' 2-leg flow described by Andrew is really close to a client credentials flow in OAuth 2.0. Without a pre-authorized access token (authorization code) the options are limited for a 2-leg flow.

Please see the updated documentation for more information on how CXF supports OAuth and its 2-leg flows in particular.

Make your Application Server CXF JAX-RS friendly

Now and then I'm seeing users reporting issues on the forums to do with deploying web applications with CXF JAX-RS libraries into some of the popular Java EE application servers.

So I thought, while investigating a problem reported on the CXF users list to do with using a CXF Redirection feature in WebLogic, that it was also worth giving it a try and experimenting with deploying a complete OAuth demo web application packaged as a war archive that we are working upon into several popular application servers.

Initially I focused on testing Glassfish, JBoss and WebLogic and you can see the notes on how to overcome various deployment issues here.

It was a rather interesting exercise and I had few observations at a time.

WebLogic was the simplest to work with, as far as deploying the application was concerned without having to tweak anything at the container level. The management console of WebLogic is quite sophisticated and seems like this application server is still very capable. I only had to tweak the servlet configuration to get a CXFServlet using a wildcard URI pattern redirecting to WebLogic specific JSP engine properly as advised by the user who reported the issue.

Glassfish is OSGI-enabled and it was interesting to see Apache Felix serving as the default OSGI framework, which is a good news for the OSGI community at large, as it should drive the fixes back to this Apache project.

I had hard time though getting past the Jersey filter trying to deal with this application - it was failing eventually due to it not able to inject a CXF-specific JAX-RS Context instance. I only managed to get it work after removing one of Jersey jars from the available libs - one would only have to do it or may be something simpler :-) if the war has JAX-RS Application implementations and the OAuth demo has 5.

JBoss 7 was easy enough to deploy to. RestEasy also tries to load Applications but it was much easier to deal with it in JBoss. As a side note I thought the way JBoss 7 managed to move away from having to dump all the libs into common folders was very impressive. It is somewhat similar to the Apache Karaf's features mechanism, with Karaf having the repository of libraries and features linking those libraries together. In JBoss the repository itself has some additional metadata.

One thing I thought about after finishing this exercise was that the fact that the deployed applications are eagerly scanned for JAX-RS interfaces exposes a possible issue with these scanners. And the issue is that these scanners need to check if a deployed war contains a /META-INF/services/javax.ws.rs.ext.RuntimeDelegate resource or not and if it points to some alternative implementation then just leave this web application alone.

It would make it simpler not only for CXF JAX-RS users who may want to work with Glassfish or JBoss but for Jersey and RestEasy users too who may want to try the containers where either of this popular implementations is not natively supported. Besides, it would meet the general expectation that a self-contained war should be deployable to any Java EE container.

In meantime, please consider contributing the tips to this section. The tips in the WebLogic section have been provided on the CXF users list, so please keep them coming.

CORS Support in CXF

Cross-Origin Resource Sharing (CORS) is a W3C specification (Working Draft) which "defines a mechanism to enable client-side cross-origin requests". Please see the Introduction for more information.

We had several users asking questions about how CORS could possibly be supported in CXF. Then at some point of time a mysterious :-) sergkorney offered his help with prototyping an initial code for a CORS filter, and the process moved forward.

Benson took charge and wrote a comprehensive and well-documented filter implementation which follows the specification very closely.

This filter can do the preflight and simple request checks. It interacts with the JAX-RS runtime by relying on its selection algorithm to confirm that an application has a resource method which is capable of dealing with the current request in cases when the filter does not block. If a JAX-RS resource method which can handle the HTTP OPTIONS verb indicates via a dedicated annotation that it will handle the preflight check, then the filter will delegate to it. JAX-RS root resources and individual methods can be customized for them to take part in the CORS process.

It is likely to grow into a more complex security feature in time.

Please review this initial documentation (with the link to the package.html), start experimenting and provide the feedback.

RESTful endpoint in 60 seconds in Talend Open Studio

I briefly mentioned in the end of this post that we are working on the tooling for creating RESTful endpoints in Talend Open Studio for ESB.

I'd like to offer to your attention my first screen-cast available on the Talend Channel. It shows how one create and test a basic HTTP Service endpoint echoing the POST payload back to the client. I timed it all for some fun, so it actually did took me 60 seconds to create a service and about 60 seconds to test it.

Of course one would spend much more time on developing a more involved service but the idea was to give you a glimpse of how easy you can create a RESTful service as well as to point to the flexibility of the tooling as far as wiring the extra components in is concerned.

Future presentations will show how to create more involved services, those supporting multiple HTTP verbs, with more components facilitating the access to the real data.

Please listen to this short presentation, download the Studio and try creating your own HTTP service and also check what it offers with respect to working with SOAP services and Camel routes.

Oh yes, one more thing, if you are into languages and you'd like to hear what an Irish-Belorussian accent is about :-), then please listen indeed to this screen-cast

Enjoy !

Maven archetype for creating CXF JAX-RS applications

A number of Maven plugins that CXF JAX-RS users could try for generating the initial code they could build upon for creating working applications was close to zero not too long ago, in fact it was actually 0.

Then we added a wadl2java plugin so the numbers started to look better :-).

And now starting from CXF 2.5.1 an archetype plugin for creating Spring-based JAX-RS applications is also available, all thanks to Benson.

Please see this page for more information.
The generated project will get the integration tests running too, so having a simple end-to-end application created and tested in less than a minute is really cool.