Wednesday, July 4, 2012

LDAP recovery

If you have suffered from LDAP wreckage then probably you already know how difficult it is to make it work again. In my case all problems started with such message in system log:
slapd[11705]: bdb(dc=xxx,dc=com): PANIC: fatal region error detected; run recovery

Note: my bdb version is 4.7, yours may differ;
slapd stands for Stand-alone LDAP Daemon, your service name may be ldap


At first I tried to recover LDAP using
db4.7_verify -h /var/lib/ldap
db4.7_recover -v -h /var/lib/ldap 

But LDAP got broken again and again.
Thus I decided to recreate whole LDAP folder. Here is the scenario:

  1. Stop the LDAP server
    service slapd stop
  2. Dump the directory structure to a text file
    slapcat -l /etc/ldap/backup/ldap_old.ldif
  3. Recover LDAP
    db4.7_recover -v -h /var/lib/ldap
  4. Dump the directory structure to a text file
    slapcat -l /etc/ldap/backup/ldap.ldif
  5. Verify that ldap.ldif contains your entries. If it does not, or if slapcat returned errors in step 3, try running db4.7_recover in catastrophic mode:
    db4.7_recover -v -h /var/lib/ldap -c
    and repeat step 3
  6. Delete the LDAP directory
    rm -fr /var/lib/ldap/*
  7. Load LDAP directory from the file you have create in step 3
    slapadd -l /etc/ldap/backup/slap.ldif
  8. Make openldap the owner of the LDAP directory
    chown -R openldap:openldap /var/lib/ldap/*
  9. Start the LDAP server
    service slapd start

Saturday, June 9, 2012

AuthorityLabs Partner API

If you need to get SERP(search engine results page) rank information on some keywords, there is a very good solution - AuthorityLabs Partner API. I have successfully used it on one of my projects and I want to share with you some tips on developing functionality to work with it.

AuthorityLabs Partner API is as a REST service. The main API methods are for POSTing keywords that you want to get info on and GETting the information from POSTed keywords. There are 2 different ways of work with POST requests:
  • Instant POST request(expensive)
  • Delayed POST request(cheaper)
We'll work with Delayed requests.
At first we should think how are we going to work with the Web service. I have chosen to use Apache HttpComponents because it's straightforward and easy to use. To download the library I used the following Maven dependencies:
<dependency>
 <groupId>org.apache.httpcomponents</groupId>
 <artifactId>httpcore</artifactId>
 <version>4.2</version>
</dependency>
 <dependency>
 <groupId>org.apache.httpcomponents</groupId>
 <artifactId>httpclient</artifactId>
 <version>4.1.3</version>
</dependency>

Now let's develop method to POST keywords to the AuthorityLabs Partner API.
public class AuthorityLabsTest {
 private static final String AUTHORITYLABS_AUTH_TOKEN = "XXX";
 private static final String AUTHORITYLABS_URL_POST_KEYWORD = "http://api.authoritylabs.com/keywords";

 
  /**
     * Post keyword to the delayed queue of the AuthorityLabs API
     * @param keyword
     * @param engine
     * @param locale
     * @return
     */
 static boolean postKeyword(String keyword, String engine, String locale) {
  HttpClient client = new DefaultHttpClient();
  
  HttpPost post = new HttpPost(AUTHORITYLABS_URL_POST_KEYWORD);
  try {
   List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(1);
   nameValuePairs.add(new BasicNameValuePair("auth_token", AUTHORITYLABS_AUTH_TOKEN));
   nameValuePairs.add(new BasicNameValuePair("keyword",keyword));
   nameValuePairs.add(new BasicNameValuePair("engine", engine));
   nameValuePairs.add(new BasicNameValuePair("locale", locale));  
   post.setEntity(new UrlEncodedFormEntity(nameValuePairs));
   System.out.println("Post keyword; parameters - " + nameValuePairs);
 
   HttpResponse response = client.execute(post);
   BufferedReader rd = new BufferedReader(new InputStreamReader(
     response.getEntity().getContent()));
   String line = "";
   while ((line = rd.readLine()) != null) {
    System.out.println(line);
   }

  } catch (IOException e) {
   e.printStackTrace();
   return false;
  }
  return true;
 }
}

Next we need method to GET keyword info.
public class AuthorityLabsTest {
 private static final String AUTHORITYLABS_AUTH_TOKEN = "XXX";
 private static final String AUTHORITYLABS_URL_POST_KEYWORD = "http://api.authoritylabs.com/keywords";
 private static final String AUTHORITYLABS_URL_GET_RESULTS = "http://api.authoritylabs.com/keywords/get.json?keyword={0}&auth_token={1}&engine={2}&locale={3}&rank_date={4}";

 static boolean postKeyword(String keyword, String engine, String locale) {
    ...
 }
 
 /**
  * Gets JSON results from the AuthorityLabs API
  * @param keyword
  * @param engine
  * @param locale
  * @param stringPostKeywordDate
  * @return
  */
 static InputStream getKeywordResults(String keyword, String engine, String locale, String stringPostKeywordDate) {
  String urlPattern = AUTHORITYLABS_URL_GET_RESULTS;
  String url = MessageFormat.format(urlPattern, keyword, AUTHORITYLABS_AUTH_TOKEN, 
    engine, locale, stringPostKeywordDate);
  //Prepare URL (e.g. replace spaces with %20 etc)
  try {
   url = URIUtil.encodeQuery(url);
  } catch (URIException e1) {
   e1.printStackTrace();
   return null;
  }
  System.out.println("Get keyword results; url - " + url);
  
  HttpClient client = new DefaultHttpClient();
  HttpGet get = new HttpGet(url); 
  HttpResponse response;
  try {
   response = client.execute(get);
   System.out.println(response.getStatusLine());   
   return response.getEntity().getContent();
  } catch (Exception e) {
   e.printStackTrace();
  }   
  return null;
 }
}

Note that we need to encode URL in order to be able to get results. To choose correct engine and locale for that engine please check the documentation on AuthorityLabs API.

That's the core of the work with AuthorityLabs API.

Friday, January 6, 2012

MySQL - SELECT ... INTO OUTFILE ... - ERROR 1 (HY000)

There is a very useful functionality in MySQL for exporting data into a file. The syntax is like:
SELECT <FIELDS> INTO OUTFILE <FILE_NAME> FROM <TABLE_NAME>;


If you use Linux then when you execute this export command with FILE_NAME that is not absolute than the file will be saved in the database folder (e.g. FILE_NAME = '1.csv'); if you specify to use /tmp folder then it will be saved there(e.g. FILE_NAME='/tmp/1.csv').
But if you specify some other directory (e.g. FILE_NAME='/home/anton/mysql/1.csv') then you will get error:
ERROR 1 (HY000): Can't create/write to file '/home/anton/mysql/1.csv' (Errcode: 13)

Even if you will set MySQL user as an owner of the directory and will provide all access to it, this error will appear nevertheless.

The reason lays in the Linux kernel security module AppArmor. This module restricts programs possibilities and we have stumbled upon one of those. Thus the solution is pretty obvious - we need to configure AppArmor. For this we will add at the end (but before the closing brace) of the file /etc/apparmor.d/usr.sbin.mysqld (for Ubuntu) such lines:
/home/anton/mysql/ w,
/home/anton/mysql/* rwk,

Then we need to reload this configuration file with command (for Ubuntu):
sudo /etc/init.d/apparmor reload

Now check that the directory that you want to use (/home/anton/mysql) has 755 rights and mysql us is the owner of it. Also make sure that there is enough free space on the volume.
Repeat export SELECT command

Tuesday, November 1, 2011

Creating mock OFX server

Nowadays more and more financial establishments use modern Web technologies to provide user with better experience. One of such technologies is OFX.

OFX stands for Open Financial Exchange, it is the format, that is based on XML, for data exchange between financial institutions. It was released in 1997 and evolved from Microsoft's Open Financial Connectivity (OFC) and Intuit's Open Exchange file formats.

In Java there are not many OFX libraries, the best one is OFX4J. Using this library you can develop both client and server. It has classes for every possible OFX (XML) element. On my opinion, the biggest flaw of this library is the bad documentation. Unfortunately, there are also not many examples online on usage of OFX4J. So here is my example that may help someone.

Let's write the simple JAX-WS OFX server. As always at first we need to download the library or specify Maven dependency:
<dependency>
 <groupId>net.sf.ofx4j</groupId>
 <artifactId>ofx4j</artifactId>
 <version>1.5</version>
</dependency>

Now we need the web-service interface:
package com.test.ofx.server.service;

import javax.activation.DataHandler;
import javax.jws.WebMethod;
import javax.jws.WebService;

@WebService
public interface OFXServerService {    
    @WebMethod
    DataHandler getResponse(DataHandler request);    
}

As you can see there was nothing difficult. Now it's time for the implementation:

package com.test.ofx.server.service.impl;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.TreeSet;
import java.util.UUID;

import javax.activation.DataHandler;
import javax.jws.WebService;

import net.sf.ofx4j.domain.data.ApplicationSecurity;
import net.sf.ofx4j.domain.data.RequestEnvelope;
import net.sf.ofx4j.domain.data.RequestMessage;
import net.sf.ofx4j.domain.data.RequestMessageSet;
import net.sf.ofx4j.domain.data.ResponseEnvelope;
import net.sf.ofx4j.domain.data.ResponseMessageSet;
import net.sf.ofx4j.domain.data.banking.AccountType;
import net.sf.ofx4j.domain.data.banking.BankAccountDetails;
import net.sf.ofx4j.domain.data.banking.BankStatementRequestTransaction;
import net.sf.ofx4j.domain.data.banking.BankStatementResponse;
import net.sf.ofx4j.domain.data.banking.BankStatementResponseTransaction;
import net.sf.ofx4j.domain.data.banking.BankingResponseMessageSet;
import net.sf.ofx4j.domain.data.common.Status;
import net.sf.ofx4j.domain.data.common.Status.Severity;
import net.sf.ofx4j.io.AggregateMarshaller;
import net.sf.ofx4j.io.AggregateUnmarshaller;
import net.sf.ofx4j.io.OFXParseException;
import net.sf.ofx4j.io.OFXWriter;
import net.sf.ofx4j.io.v2.OFXV2Writer;

import com.sun.xml.ws.util.ByteArrayDataSource;
import com.test.ofx.server.service.OFXServerService;

@WebService(endpointInterface = "com.test.ofx.server.service.OFXServerService", name = "ofxWebService")
public class OFXServerServiceImpl implements OFXServerService {
    
    @Override
    public DataHandler getResponse(DataHandler request) {
        try {
            //parse request            
            AggregateUnmarshaller<requestenvelope> unmarshaller = new AggregateUnmarshaller<requestenvelope>(
                    RequestEnvelope.class);
            RequestEnvelope requestEnvelope = unmarshaller.unmarshal(request.getInputStream());

            ResponseEnvelope responseEnvelope = getResponse(requestEnvelope);

            //prepare response to send
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            OFXWriter writer = new OFXV2Writer(baos);
            AggregateMarshaller marshaller = new AggregateMarshaller();
            
            marshaller.marshal(responseEnvelope, writer);
            writer.close();
            DataHandler dataHandler = new DataHandler(new ByteArrayDataSource(baos.toByteArray(),
                    "application/octet-stream"));
            return dataHandler;
        } catch (IOException e) {
            throw new RuntimeException(e);
        } catch (OFXParseException e) {
            throw new RuntimeException(e);
        }
    }

    private ResponseEnvelope getResponse(RequestEnvelope requestEnvelope) {
        //declare variables to generate response
        ResponseEnvelope responseEnvelope = new ResponseEnvelope();            
        BankingResponseMessageSet responseMessageSet = new BankingResponseMessageSet();
        List<bankstatementresponsetransaction> transactions = new ArrayList<bankstatementresponsetransaction>();
        BankStatementResponse bankStatementResponse = new BankStatementResponse();
        BankAccountDetails bankAccountDetails;
        Random rand = new Random();

        //generate response transactions
        for (RequestMessageSet requestMessageSet : requestEnvelope.getMessageSets()) {
            for (RequestMessage requestMessage : requestMessageSet.getRequestMessages()) {
                //specify bank account details
                bankAccountDetails = new BankAccountDetails();
                bankAccountDetails
                        .setAccountNumber(((BankStatementRequestTransaction) requestMessage)
                                .getMessage().getAccount().getAccountNumber()
                                + "-" + rand.nextInt());
                bankAccountDetails.setBankId("Bank ID");
                bankAccountDetails.setAccountType(AccountType.CHECKING);
                
                //specify and add bank statment response transaction
                bankStatementResponse.setAccount(bankAccountDetails);
                BankStatementResponseTransaction bankStatementResponseTransaction = new BankStatementResponseTransaction();
                bankStatementResponseTransaction.setMessage(bankStatementResponse);
                Status status = new Status();
                status.setSeverity(Severity.INFO);        
                bankStatementResponseTransaction.setStatus(status);
                bankStatementResponseTransaction.setUID(UUID.randomUUID().toString());
                transactions.add(bankStatementResponseTransaction);
            }
        }

        //create response envelope
        responseMessageSet.setStatementResponses(transactions);
        responseEnvelope.setMessageSets(new TreeSet<responsemessageset>());
        responseEnvelope.getMessageSets().add(responseMessageSet);
        responseEnvelope.setSecurity(ApplicationSecurity.TYPE1);
        responseEnvelope.setUID(UUID.randomUUID().toString());
        return responseEnvelope;
    }
}
There are many classes in the library and at first the code may seem pretty incomprehensible. But you have to remember that almost all this variety of classes do nothing more than just wrap XML element. Every time you are not sure about meaning of the current operation feel free to look the library code. It is not very complicated.

By the way at first I've tried to get RequestEnvelope and send ResponseEnvelope in the web service, but I've stumbled upon implementation of ResponseEnvelope. It is dependable on some class that do not have public default constructor(UnknownStatusCode). And due to limitations of JAX-WS it was impossible to create stubs or use the common interface with the client.

That's all about the server part. In the next article we'll create the OFX client.

Sunday, October 30, 2011

Full working runtime annotation example

Annotations are metadata(data about data) that allows you to keep an additional information right in your code. Annotations have been introduced into Java in 5th edition.

Before annotations almost all configuration data had to be kept in XML format. And as the application grew, the number and complexity of its XML-files had been increasing exponentially. They called it XML hell.

And now when we have annotations we can keep the configuration of the code right with it. Besides annotations are tested and verified by the compiler.

But the use of annotations is not limited to configuration, annotations allow you to get rid of the boilerplate code (e.g. cross-cutting concerns such as transactions and security).

The boon of annotations can be very easily noted using Hibernate Annotation package, Spring 2.5 and newer, Seam. But how do they write the annotations of their own?

So here is the definition of our annotation @Testing:
package com.test.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Testing { }
As you can see this definition is not much unlike of the definition of an interface. And really it compiles to the class file as any other Java interface. As you can see the annotation definition requires @Target and @Retention annotations:
  • @Target defines where you can use your annotation. In this example it can be used only with methods. Here are all possible values and places where you can use annotations:
    • TYPE - class, interface (including annotation type), or enum declaration 
    • FIELD - field declaration (includes enum constants)
    • METHOD - method declaration
    • PARAMETER - parameter declaration
    • CONSTRUCTOR - constructor declaration
    • LOCAL_VARIABLE - local variable declaration
    • ANNOTATION_TYPE - annotation type declaration 
    • PACKAGE - package declaration
  • @Retention defines when the annotation can be available:
    • SOURCE - in the source code (discarded by the compiler)
    • CLASS - in the class files (not retained at run time). This is the default behavior.
    • RUNTIME - at run time (can be read reflectively)
Often annotations contain some values (as @Target and @Retention above). These values are represented as interface methods in the definition. If an annotation does not have any element than it is called marker annotation.

We will use proxy pattern to allow our processor intercept the calls to the methods annotated with @Testing. Java provides implementation of the proxy with the java.lang.reflect.Proxy. It is very powerful tool but unfortunately it does not provide possibility to create proxies just for the class, you have to provide the list of interfaces to create the proxy (if you need to proxy classes and not just interfaces then you should use CGLib).

So let's create the annotation processor:
package com.test.annotation.processor;

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;

import com.test.annotation.Testing;

public class TestingInvocationHandler implements InvocationHandler {
 private Object proxied;
 
 public TestingInvocationHandler(Object proxied) {
  this.proxied = proxied; } 

 @Override
 public Object invoke(Object proxy, Method method, Object[] args)
   throws Throwable {
  Method m = proxied.getClass().getMethod(method.getName(), method.getParameterTypes());
  if (m.isAnnotationPresent(Testing.class)) {
   System.out.println("\tIn the annotation processor");   
  } 
  return method.invoke(proxied, args);
 }
}
The annotation processor implements InvocationHandler interface. This interface is responsible for handling the proxy object invocation.
As you can see we only print additional line before invoking the original method.

Here is the helper class with the static method to get the proxy object:
package com.test.annotation.processor;
import java.lang.reflect.Proxy;

public class TestingProxy {
 
 public static Object getNewProxy(Object proxied, Class<?> interfaze) {
  Object proxy = Proxy.newProxyInstance(
      TestingInvocationHandler.class.getClassLoader(),
      new Class[] {interfaze}, 
      new TestingInvocationHandler(proxied));
  return proxy;
 }

}
Let's define the interface
package com.test.service;

public interface Observable {
 void print1();
 
 void print2(); 
}
and the class of the proxied objects
package com.test.service;

import com.test.annotation.Testing;
import com.test.annotation.processor.TestingProxy;

public class PrintService implements Observable {
 private static Observable instance;
 private PrintService(){
  
 }
 public static Observable getInstance() {
  if (instance == null) {   
   instance = (Observable) TestingProxy.getNewProxy(new PrintService(),
     Observable.class);   
  }
  return instance;
 }
 
 @Testing
 public void print1() {
  System.out.println("1 - Text in annotated method");
 }
 
 public void print2() {
  System.out.println("2 - Just ususal text");
 }
}
This class is implemented as a simple singleton and uses the proxy object in getInstance() method. Also this class implements both methods of the interface. One of this methods is annotated and the other is not.

Finally let's create the class with the main method:
package com.test;

import com.test.service.Observable;
import com.test.service.PrintService;

public class Application {
 public static void main(String[] args) {
  Observable printService = PrintService.getInstance();  
  printService.print1();  
  printService.print2();
 }
}
That's all.