Converting FIQL expressions into SQL queries in CXF JAXRS

I've done some more work on the search extensions so that users can easily introspect a given SearchCondition instance representing either a primitive or complex query and also added a utility SearchCondition.toSQL(String tableName, String... columnNames) which can be used to convert this search condition into an equivalent SQL query to be subsequently executed against a database.

For example,

// find all conditions with names starting from 'ami' 
// and levels greater than 10
SearchCondition sc = 
   fiqlParser.parse("name==ami*;level=gt=10");
assertEquals("SELECT * FROM table 
              WHERE 
              name LIKE 'ami%' 
              AND 
              level > '10'",
              sq.toSQL("table"));

// find all conditions with names starting from 'foo' 
// and those with names not ending with 'bar' or
// levels greater than 10 
SearchCondition sc2 = 
   fiqlParser.parse(
   "name==foo*;(name!=*bar,level=gt=10)");
assertEquals("SELECT * FROM table 
              WHERE 
              (name LIKE 'foo%') 
              AND 
              ((level > '10'") 
                OR 
               (name NOT LIKE '%bar')),
              sq.toSQL("table"));

Here are some clarifications. First, note that the above code shows that a FIQL parser creates a search condition but it is a unit test code, the actual user code will use a SearchContext instead in a type safe way, providing a bean class to it. SearchContext will create an instance and will inject into it the property values, those provided and extracted from the original FIQL expression.

This provides for a possibility to protect against possible SQL injection attacks at the Java and the bean validation levels. For example, if a Book bean has an integer 'id' property then there's no way an unsafe FIQL query such as 'id==DROP%20users' can result in a 'DROP users' value injected due to a NumberFormatException.

Additionally, bean setters may have some custom validation logic and the CXF FIQL parser may also get enhanced to check for the bean validation annotations.

Perhaps you might want to give it a try and see how safe this option of auto-converting FIQL expressions into SQL queries is ?

In the end of the day, the good thing is that you actually do not have to use
SearchCondition.toSQL to get the SQL query. Suppose you are still concerned about the security or see that this method is not going to help you with building the advanced SQL queries which you actually use in the production, may be you are using java.sql.PreparedStatements or perhaps you don't even use a relational database but rather work with say Cassandra. If it is the case then just introspect a SearchCondition by getting to all the subconditions it has, and build a query which suits best.

XOP support in CXF JAXRS

An XML-binary Optimized Packaging (XOP) is a W3C specification which is mainly associated with a SOAP Message Transmission Optimization Mechanism (MTOM) but is equally applicable to plain XML.

XOP is actually a very good specification which provides for the inclusion of binary data in XML payloads via links to co-located or external parts and thus facilitates the processing of such payloads by technologies like XSLT or even XMLSecurity, this article gives a pretty good description.

A CXF user has recently tried to have a single Java service class serving both SOAP MTOM and 'basic' HTTP multipart-related requests. Now, it is possible to do it, but I found it was not quite appealing so to say for users having to create either MTOM or multipart/related (with no links from the root part) payloads for the whole idea to work out. In addition, users have to add CXF JAXRS Multipart annotations to every method parameters corresponding to individual multipart parts which was not ideal given that the same user was also experimenting with a document-first approach.

So it took me virtually 1 day to add a XOP support to CXF JAXRS, thanks, as usual, to the fact that CXF has it all, specifically, attachment interceptors already deal with MTOP/XOP and its JAXB databinding has all the related utility code required for serializing/deserializing XOP payloads. One would need to set a CXF Message.MTOM-ENABLED property on CXF JAXRS clients/endpoints, but I think it is ok even though a property name is SOAP related, but it is really just an indicator to CXF as to whether do XOP or not.

I think it is a good enhancement indeed :

- SOAP users doing multiparts (MTOM) can have a simpler option for reusing the same code base when trying to do HTTP/REST as well, the only restriction at the moment is that CXF JAXRS will not unwrap the complex data types, example, just have a single EmailPayload method parameter, as opposed to many parameters corresponding to EmailPayload parts, such as text, attachments, etc; it is better anyway IMHO

- it is just a nice to have feature for CXF JAXRS users to play with, as far the multiparts support is concerned

The only regret I have is that I started working on it late enough and thus did not have it merged into soon to be released 2.2.7.

WSDL - first, CXF JAXRS model - first !

So you're a SOAP developer who prefers starting the development of a new service from creating a WSDL document first. It is a WSDL-first approach that works for you, not a Java-first one. You are also interested in giving it a try and seeing what advantages following the RESTful style can give the users of your application, but would also like to keep a single code base.

If so then please read on this long post...

CXF makes it easy for developers to expose a single Java class as a SOAP and REST service, by sharing a service bean between multiple jaxws and jaxrs endpoints. The problem is, as far as you're concerned, that it requires one to follow a Java-first approach.

One option is for CXF to offer users an option to do a WADL-first development. CXF JAXRS is not quite there just yet but nothing prevents it now from providing a code gen command line utility and a maven plugin. This is an ongoing effort though and some UI tooling support would also be required. So, eventually, one would be able to use WSDL for developing SOAP services and WADL for developing RESTful services. This may or may not become a viable approach for users preferring a document-first style. Working on different documents (WADL and WSDL) might not make much sense, and creating a shared service class will not be possible.

The other advice which you, the SOAP developer, are most likely to hear today is as follows : you'd be better off migrating to REST/JAXRS altogether rather than sticking with SOAP. So the issue of figuring out how to do SOAP and REST the document-first style at the same time would be resolved. Even though there is some evidence that a number of individual and corporate developers may have done just that, I think this advice may not be the most effective option, as far as convincing those who have invested in SOAP and actually liking it no matter what that there could be a better option awaiting them. It is enhancing the runtime for it to be able to do the advanced stuff SOAP developers can do today with respect to the message-level security, multiparts and the document-first development and providing the easy path for SOAP developers to experiment with REST in the live production is what can convince the most conservative developers over time that the other approach (REST) may be adopted, partially at least.

Here is another option CXF could try: implement WSDL2, thus letting users define both SOAP and REST bindings in a single document which can be quite effective from the development point of view. Now, WSDL2 is said to be a better language than WSDL 1.1, but the question is : would an investment into implementing it be cost-effective ? Will it bring new users to CXF? And is it worth investing into it just for the sake of letting users do both SOAP and REST development the document-first way ? It does not seem it can do better that WADL in describing a RESTful application, given that it is more likely WADL can be enhanced even further while WSDL2 will not(not touching here on the interface vs resource centric approaches issue). Finally, as Dan Kulp says, a user demand is just not there, at least at the CXF users list. It can be quite a difficult thing to do, deciding on whether a given feature should be implemented depending on the existing or anticipated demand from users, sometimes expecting the users/customers always indicate to you what should be done next can stifle the innovation, but in case of WSDL2 it seems like the right thing to do.

This post has been long enough but I've tried to do some analysis there...So here is what can be done right now the real cost effective way (thanks to one of CXF users trying this option first) : generate your SOAP and JAXB classes from WSDL and then
'attach' some RESTful behavior to generated interfaces by relying on the annotation-free feature and finally have both jaxws and jaxrs endpoint sharing an implementation bean, with the jaxrs endpoint including the model describing the interface only. Example, imagine that no JAXRS annotations exist in this (generated) source interface. Next, just describe this interface similarly to the way it is done here. Finally add a jaxrs endpoint (see the one with a 'bookservice3' id) referring to the model describing this interface and the bean which implements it and you're done.

It would be mostly about making your code a bit more HTTP aware as opposed to magically turning it into a RESTful application, but it is an easy start and you may want to give it a try. And if it works for you then you might want to ask your friends from the UI team to have a tool generating CXF JAXRS models for you :-)

CXF JAXRS Search Extensions

What is it that defines a user experience on the WEB ? Reading the information, submitting some information, following the links and searching.

How to let the users do the advanced search of the data your application makes available publicly ? One way is to let users type or transform their requirements into XQuery expressions, another option is to build a URI-friendly query language which will let users easily type it and link to the resulting URIs.

This post provides a good comparison between two approaches used by Microsoft Astoria and Google Base Data respectively. As you can see one can do some fairly sophisticated queries but they seem to be quite complicated for users be able to type them manually and in some cases are not easy on the eye. Both approaches provide for advanced search capabilities (to do with hierarchies, etc) but which may not be easy to implement.

Andy Michalec and myself have worked recently on implementing a search extension which will allow users to use a Feed Item Query Language (FIQL) to do the advanced search queries against CXF JAXRS endpoints. FIQL is a simple, very intuitive and easy on the eye URI-friendly language which lets users express fairly advanced queries.

Mark Nottingham writes simple and practical specifications. CXF now implements two specifications authored by Mark, RFC-5005 (partially) and the FIQL draft and I think CXF users will start getting some real value out of it.

Even though FIQL is a language originally meant to be used for searching Atom feeds, it is written such that non-Atom data can be searched too. So after Andy has spent an hour or so :-) on writing a quality FIQL parser, I just wired in a SearchContext into the CXF JAXRS runtime and now users can do the following :

WebClient wc =
WebClient.create("http://localhost:9080/books?"
+ "_s=name==CXF*;id=ge=123;id=lt=555");
List<Book> books = wc.getCollection(Book.class);

Here we create a FIQL expression allowing us to get all the books with names starting from "CXF" and ids greater or equal to 123 but less than 555. FIQL can let you do much more powerful queries but I think it is just a brilliant effort from Mark.

Note how simple the query is, here are the rules in a nutshell : it is only the first '=', the one which follows an _s query name (for the record, _search is also supported) which acts as a usual separator between a query name and its value, '==' means 'equals', while '=' in '=gt=', etc helps easily identify a condition such as 'greater than'. A user can quite easily type such an expression in a browser without some (web )UI gadget hiding the complexity of the language.

And here is the server side test code :

@Path("books")
public class Books {

private Map<Long, Book> books;
@Context
private SearchContext context;

@GET
public List<Book> getBook() {

SearchCondition<Book> sc =
   searchContext.getCondition(Book.class);
List<Book> found = sc.findAll(books.values());
return found;
}
}

An injected SearchContext returns you a SearchCondition which can be a composite one and encapsulates a given FIQL expression, though nothing prevents us transparently supporting other languages in the future, such as XQuery, etc.

Now, at the moment the assumption is made that the data are kept in memory, but often enough the search requirements are passed down to the database layer. You can try to push SearchCondition as close as possible to the db layer but SearchCondition, having captured the FIQL expression can let you easily transform it into an SQL language expression if needed, ex, it can let you check all the conditions it encapsulates, and the values which will be used during the checks, etc.

I'm not sure yet if I'll merge it into 2.2.7-SNAPSHOT or not. May be I'll keep it in 2.3-SNAPSHOT and experiment a bit more and then merge it into 2.2.8-SNAPSHOT once the interfaces are deemed capable enough. Will also update the Atom pull server to let users search for very specific log entries, that what FIQL was created for :-)

Enjoy !

On the way to WADL-first development in CXF JAXRS

Yesterday I added a CXF JAXRS request filter capable of generating the Java code for consuming a given JAXRS resource on demand. I was about to finish it two weeks ago but had to postpone it a bit. Here's a test code fragment showing how to get a zip containing the source :

// Get the link to the source
WebClient wc =
 WebClient.create("http://localhost:9080/petstore");
XMLSource source = wc.query("_code")
                    .query("_os", getOs())
                    .get(XMLSource.class);
String link = source.getValue(
  "ns:html/ns:body/ns:ul/ns:a/@href",
  Collections.singletonMap("ns",
    "http://www.w3.org/1999/xhtml"));
// download a zip file          
WebClient wcZip = WebClient.create(link);
InputStream is = wcZip.accept("application/zip")
                .get(InputStream.class);
// unzip and compile it

Please see a testPetStoreSource test for more details. The _os query is optional, it is there to ensure you can get OS-specific line separators added to some of the generated classes (ex, the server is running on Windows, you'd like to run a test client on Unix). You can also add a _language query but it is only Java which is supported for now.

What is interesting is that a newly introduced CodeGeneratorProvider generates the source from a WADL instance which another provider, WADLGenerator, creates after introspecting a given JAXRS resource. The zipped source will contain the JAXB-generated classes from a WADL grammar section (if any) and the proxies (interfaces) representing WADL resources, including the subresources, with the resource methods linking to corresponding JAXB classes.

You can request that classes be generated from a WADL grammar section only by adding a _codeType=grammar query. Two other possible values here are 'proxy' and 'web', the former value is a default one at the moment with the latter one being not supported yet. I need to think a bit more about what sort of code can be generated for a _codeType=web. Suppose we have a resource named BookStore. What is the difference between a method called getBook (in case of proxies) and a method called get(in case of web clients) ? I'm also not sure yet what to do with cases where WADL resource contains multiple methods like GET but with different mediaType values, need a bit more time. In meantime if the word 'proxy' makes you nervous :-) then please just use _codeType=grammar to download the classes representing schema elements and proceed from there by using your JAXRS runtime's web client api.

How does CodeGeneratorProvider generate the proxies given that WADL does not provide for it ? Internally, it requests a WADLGenerator to add 'id' attributes to WADL resource elements representing root or subresource resources and to WADL resource/method elements. Example, wadl:resource/@id="{org.apache.cxf.test.RootResource}" and wadl:resource/wadl:method/@id="getBook".

This is really all what 'facilitates' the WADL to Java code generation which is exactly what CodeGeneratorProvider does. In fact, I haven't actually even thought originally about introducing such a provider but then I thought that a future command-line codegen tool allowing for a server-side code be generated from a WADL document will do exactly what CodeGeneratorProvider does, albeit it will generate the resource class skeletons as opposed to proxies. In fact, given that CodeGeneratorProvider is a regular CXF JAXRS provider, you can set a false 'generateInterfaces' property on it and you'll get a *server-side* code downloaded, perhaps you might want to use it to write your own added-value service interposing the original one.

CXF 2.2.7 will be out soon so please experiment with this new feature.

I'll need to think a bit more about the format for resource ids, perhaps it can be just "BookStore" but a future codegen tool can ask users to provide a package name. We can have JAXB classes acting as subresources that is why I've introduced an extended QName format initially. At the moment WADL does not allow internal method elements to contain ids (as opposed to global ones), hope this restriction can be lifted, alternatively it would be easy for a (UI) tool allowing users to create WADL documents to ensure resource methods are declared globally and then linked to from internal resources.

Seems like using resource and method ids for the purpose of allowing WADL-first development might make sense, it will preserve the simplicity of WADL the language.

Finally, why worry about the WADL-first development ? It is simple : IMHO it might help in making the RESTful approach toward developing web services being more attractive to many developers who prefer starting with building a picture of the future application first. So imagine your UI tool showing a nice view of how resources relate/link to each other and what formats and HTTP methods they support and schemas they're aware of, would not it be cool to be able to press a button next and have most of the code being generated for you ?

CXF is my favourite game

I was listening to the radio in the car this morning and I heard 'And I'm losing my favourite game' by The Cardigans. For some reasons, I'm often associating certain phrases from the songs I hear at a given moment of time with what I'm thinking or doing at the moment. So the idea of "CXF is my favourite game" sprang to mind and here I am, with all the time on my hands, blogging about it.

The last week was quite interesting. For the most of the last 20 years I've been at work virtually every day doing something, writing some code, discussing something, thus being presented with an opportunity (this is how I'd like to see our severance with the Fuse team) to rethink and take a break poses quite a challenge. But there's definitely something in it. For example, I've been to London many times but I felt the trip last week was the best one : you kind of feel the energy of the city and realize the world is so rich and there are definitely opportunities lurking out there.

I've been focusing mainly on CXF for the last couple of years. I don't know yet where my next career move will bring me to, but I'm feeling there could more coming from CXF, I'll try to keep contributing whenever a get a chance. The software world is big but I'd like to see CXF being a continuous success.