Cannot run Spring application: BeanDefinitionParsingException

When you try to run Spring based application you can see the exception

org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Unable to locate Spring NamespaceHandler for XML schema namespace

The problem may be caused by not having all neccessary Spring jars or by jar conflicts.

First of all check that you have all jars you need. Here is the list of possible jars with explanations for Spring 3.0.5 (the original of this list of dependencies is Spring Blog):

<!-- Shared version number properties -->
<properties>
    <org.springframework.version>3.0.5.RELEASE</org.springframework.version>
</properties>
 
<!--
    Core utilities used by other modules.
    Define this if you use Spring Utility APIs (org.springframework.core.*/org.springframework.util.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-core</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Expression Language (depends on spring-core)
    Define this if you use Spring Expression APIs (org.springframework.expression.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-expression</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Bean Factory and JavaBeans utilities (depends on spring-core)
    Define this if you use Spring Bean APIs (org.springframework.beans.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-beans</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Aspect Oriented Programming (AOP) Framework (depends on spring-core, spring-beans)
    Define this if you use Spring AOP APIs (org.springframework.aop.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-aop</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Application Context (depends on spring-core, spring-expression, spring-aop, spring-beans)
    This is the central artifact for Spring's Dependency Injection Container and is generally always defined
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-context</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Various Application Context utilities, including EhCache, JavaMail, Quartz, and Freemarker integration
    Define this if you need any of these integrations
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-context-support</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Transaction Management Abstraction (depends on spring-core, spring-beans, spring-aop, spring-context)
    Define this if you use Spring Transactions or DAO Exception Hierarchy
    (org.springframework.transaction.*/org.springframework.dao.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-tx</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    JDBC Data Access Library (depends on spring-core, spring-beans, spring-context, spring-tx)
    Define this if you use Spring's JdbcTemplate API (org.springframework.jdbc.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-jdbc</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Object-to-Relation-Mapping (ORM) integration with Hibernate, JPA, and iBatis.
    (depends on spring-core, spring-beans, spring-context, spring-tx)
    Define this if you need ORM (org.springframework.orm.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-orm</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Object-to-XML Mapping (OXM) abstraction and integration with JAXB, JiBX, Castor, XStream, and XML Beans.
    (depends on spring-core, spring-beans, spring-context)
    Define this if you need OXM (org.springframework.oxm.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-oxm</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Web application development utilities applicable to both Servlet and Portlet Environments
    (depends on spring-core, spring-beans, spring-context)
    Define this if you use Spring MVC, or wish to use Struts, JSF, or another web framework with Spring (org.springframework.web.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-web</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Spring MVC for Servlet Environments (depends on spring-core, spring-beans, spring-context, spring-web)
    Define this if you use Spring MVC with a Servlet Container such as Apache Tomcat (org.springframework.web.servlet.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-webmvc</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Spring MVC for Portlet Environments (depends on spring-core, spring-beans, spring-context, spring-web)
    Define this if you use Spring MVC with a Portlet Container (org.springframework.web.portlet.*)
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-webmvc-portlet</artifactId>
  <version>${org.springframework.version}</version>
</dependency>
 
<!--
    Support for testing Spring applications with tools such as JUnit and TestNG
    This artifact is generally always defined with a 'test' scope for the integration testing framework and unit testing stubs
-->
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-test</artifactId>
  <version>${org.springframework.version}</version>
  <scope>test</scope>
</dependency>

If this does not help then try Maven Shade plugin to solve the most probable conflicts:

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-shade-plugin</artifactId>
 <version>1.4</version>
 <executions>
  <execution>
   <phase>package</phase>
   <goals>
    <goal>shade</goal>
   </goals>
   <configuration>
    <transformers>
     <transformer
      implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
      <resource>META-INF/spring.handlers</resource>
     </transformer>
     <transformer
      implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
      <resource>META-INF/spring.schemas</resource>
     </transformer>
    </transformers>
   </configuration>
  </execution>
 </executions>
</plugin>

Read more...

Work with annotations in Spring Framework

Annotations are a very useful feature in Java programming language and if you use Spring Framework than it can be also very easy to create annotation processor.

I expect you to be familiar with basics of annotations (read here). In this tutorial we'll create annotation for SLF4J. Here is the annotation code:

package uay.log;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Log {

}

Now let's create annotation processor using Spring Framework capabilities:

package uay.log;

import java.lang.reflect.Field;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.config.BeanPostProcessor;
import org.springframework.stereotype.Component;
import org.springframework.util.ReflectionUtils;
import org.springframework.util.ReflectionUtils.FieldCallback;

@Component
public class LogProcessor implements BeanPostProcessor {
 public Object postProcessAfterInitialization(Object bean, String beanName)
            throws BeansException {
        return bean;
    }

    public Object postProcessBeforeInitialization(final Object bean, String beanName)
            throws BeansException {
        ReflectionUtils.doWithFields(bean.getClass(), new FieldCallback() {
            public void doWith(Field field) throws IllegalArgumentException, IllegalAccessException {
                if (field.getAnnotation(Log.class) != null) {
                    Logger log = LoggerFactory.getLogger(bean.getClass());
                    field.setAccessible(true);
                    field.set(bean, log);
                    field.setAccessible(false);
                }
            }
        });
        return bean;
    }
}

And it's that easy. Now we can use this annotation in our code:

@Log
Logger log;

Read more...

Quickstart with SLF4J

In this tutorial we will configure SLF4J logging with log4j implementation.

SLF4J is a facade of abstractions for various logging frameworks(log4j, java.util.logging, commons logging, logback). It is really easy to swap between logging frameworks when you use SLF4J. Here is what you have to add to your Maven pom.xml to use SLF4J with log4j implementaion:

<properties>
 <slf4j.version>1.6.6</slf4j.version>
</properties>
<dependencies>    
 <!-- Logging -->
 <dependency>
  <groupId>log4j</groupId>
  <artifactId>log4j</artifactId>
  <version>1.2.17</version>
 </dependency>
 <dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>${slf4j.version}</version>
 </dependency>
 <dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-log4j12</artifactId>
  <version>${slf4j.version}</version>
 </dependency>
</dependencies>

Run

mvn install

Now we need to configure log4j as usual. There are two ways to do it: via xml or via properties file. Let's use properties file. We need to create log4j.properties in the classpath. Here is the code of the file:

# Root logger option
log4j.rootLogger=INFO, file, stdout
 
# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.DatePattern = '.'yyyy-MM-dd
log4j.appender.file.Append = true
log4j.appender.file.File=logs/main.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n
 
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{dd-MM-yy HH:mm:ss,SSS} %5p [%c{1}:%M:%L] - %m%n

According to this configuration we defined the lowest log priority to INFO and set the output to Console and to the logs/main.log file(logs/main.log file will be rolled by days automatically).

Now we can test how the logging works:

package uay.test;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class App 
{
    private static Logger log = LoggerFactory.getLogger(App.class);
    public static void main( String[] args )
    {
     log.info("test");
    }
}

If we run the code the Console output would be:

01-12-12 23:12:24,840  INFO [App:main:11] - test

The file output would be:

23:12:24,840  INFO App:11 - test

Thus, we have configured the code to use log4j via SLF4J. As a further improvement of code you should think about using annotations to make the code look like this:

@Log
private Logger log;

This tutorial will definitely help you with it. Later I will show how to implement this annotation even easier using Spring Framework.

Read more...

Simple analysis and improvement of MySQL schema

Here is some basic analysis and advises on improvement of MySQL database performance. From time to time you should check you MySQL schema status, I use the following request for it:

SELECT                                                                                                                                                                                                                                                                         
    table_name AS table_name,                                                                                                                                                                                                                                                  
    engine,                                                                                                                                                                                                                                                                    
    ROUND(data_length/1024/1024,2) AS total_data_size_mb,                                                                                                                                                                                                                      
    ROUND(index_length/1024/1024,2) AS total_index_size_mb,                                                                                                                                                                                                                    
    ROUND((data_length+index_length)/1024/1024,2) AS total_size_mb,                                                                                                                                                                                                            
    ROUND(data_free/1024/1024,2) AS total_free_size_mb,                                                                                                                                                                                                                        
    ROUND(data_free/(data_length/100)) AS defragment_percent,                                                                                                                                                                                                                  
    table_rows                                                                                                                                                                                                                                                                 
FROM                                                                                                                                                                                                                                                                           
    information_schema.tables                                                                                                                                                                                                                                                  
WHERE                                                                                                                                                                                                                                                                          
    table_schema=DATABASE();

Field defragment_percent value should be 0, but it may be even higher than 100(e.g. you loaded 3MB of data to the table and then deleted 2MB, the table will still be 3MB with real data of 1MB). The most prone to fragmentation are VARCHAR fields so pay attention to the tables where you use VARCHAR a lot. To optimize tables you can use MySQL command:

OPTIMIZE [NO_WRITE_TO_BINLOG | LOCAL] TABLE mi_tabla1 [, mi_tabla2] ...;

To optimize the whole schema you can use:

mysqlcheck -o <databasename>

Field total_index_size_mb should be less than total total_data_size_mb. If it is not so then review your indexes.

Also I'd recommend to use MySQLTuner. MySQLTuner is a script written in Perl that allows you to review a MySQL installation quickly and make adjustments to increase performance and stability. To get it you can use:

wget http://mysqltuner.com/mysqltuner.pl

Then run it with

perl mysqltuner.pl

This tool will provide you with analysis of your MySQL server and some advises on how to improve its performance.

Read more...

Execute scripts in multiple databases

Have you ever needed to execute the same script on several databases? I have met such need and developed a simple algorithm for it. This algorithm I'd like to share.

At first I tried to develop it using only JDBC but unfortunately it has the really big limitation: it cannot execute at once several calls, which is pretty common for SQL scripts. So I decided to run scripts merely from a command line.

Here is the code:

package uay.loader;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileFilter;
import java.io.FileReader;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.log4j.Logger;

public class BulkScriptLoader {

    private static final String   DB_PREFIX     = "ua";
    private static final String[] DB_SKIP = { "ua_1", "ua_2" };
    private static final String   FOLDER_NAME   = "scripts";
    private static final String   FILE_EXT      = ".sql";
    private static final String   URL           = HOST + ":" + PORT;
    private static final String   USER_NAME     = "xxx";
    private static final String   PASSWORD      = "xxx";

    private final static Logger   LOGGER        = Logger.getLogger(BulkScriptLoader.class);

    public static void main(String[] args) {
        Connection connection;
        try {
            connection = DriverManager.getConnection(URL, USER_NAME, PASSWORD);
        } catch (SQLException e1) {
            LOGGER.error(e1);
            throw new RuntimeException(e1);
        }

        List<String> dbNames = getDbNames(connection);
        File[] scriptFiles = getScriptFiles();
        for (File file : scriptFiles) {
            LOGGER.info("Work with " + file.getName());
            BufferedReader bf;
            StringBuilder statement = new StringBuilder();
            try {
                bf = new BufferedReader(new FileReader(file));
                String line = "";
                while ((line = bf.readLine()) != null) {
                    statement.append(line).append("\n");
                }
            } catch (IOException e) {
                LOGGER.error(e, e);
            }
            if (statement.toString().length() > 0) {
                execute(connection, dbNames, statement.toString(), URL, USER_NAME, PASSWORD);
            }
        }
    }

    public static File[] getScriptFiles() {
        File folder = new File(FOLDER_NAME);
        if (folder.exists() && folder.isDirectory()) {
            return folder.listFiles(new FileFilter() {
                @Override
                public boolean accept(File pathname) {
                    return pathname.getName().toLowerCase().endsWith(FILE_EXT.toLowerCase());
                }
            });
        }
        return null;
    }

    public static void execute(Connection connection, List<String> dbNames,
            String generalStatement, String url, String userName, String password) {
        for (String dbName : dbNames) {
            String cmdCommand = String.format("mysql -u %s -p%s -h %s -P %s -D %s",userName, password, 
              parseHost(url), parsePort(url), dbName);
            LOGGER.info(cmdCommand);
            try {
                Process process = Runtime.getRuntime().exec(cmdCommand);
                Writer w = new OutputStreamWriter(process.getOutputStream());
                // System.out.println(statement);
                w.write(generalStatement);
                w.flush();
                w.close();
            } catch (IOException e) {
                LOGGER.error(e, e);
            }
        }
    }

    public static String parseHost(String url) {
        return url.substring(0, (url.indexOf(":") > 0 ? url.indexOf(":"): url.length()))

    }

    public static String parsePort(String url) {
        Pattern p = Pattern.compile("(\\d\\d\\d\\d)");
        Matcher m = p.matcher(url);
        if (m.find()) {
            return m.group(0);
        }
        return "3306";
    }

    public static List<String> getDbNames(Connection connection) {
        List<String> dbNames = new ArrayList<String>();
        try {
            PreparedStatement preparedStatement = connection.
               prepareStatement("select schema_name from information_schema.schemata");
            if (preparedStatement.execute()) {
                ResultSet rs = preparedStatement.getResultSet();
                while (rs.next()) {
                    String dbName = rs.getString(1);
                    if (dbName.startsWith(DB_PREFIX) && !Arrays.asList(DB_SKIP).contains(dbName)) {
                        dbNames.add(dbName);
                    }
                }
            }
        } catch (SQLException e) {
            LOGGER.error(e, e);
        }
        return dbNames;
    }
}

As you can see it's really simple. All scripts that I'd like to execute I place in 'scripts' folder with '.sql' extension. I configure database connection, prefix of databases that I want to update and specify the list of databases that I want to skip with this update.

Read more...

Search and replace text in multiple files

It's very common task to find some text in some files and replace it. Here is my recipe to deal with it

In this example I will find all 'varchar' text in my '.sql' files and replace it with 'char'.
At first let's search for the files that contain the text:

grep -i 'varchar' *.sql

In the output you can see the list of files with corresponding match within it.
Now let's take this list of files and replace the searched text within with the new one:

grep -il 'varchar' *.txt | xargs sed -i 's/varchar/char/gi'

This was a case-insensitive case. But if you need to be case-sensitive then use:

grep -l 'varchar' *.txt | xargs sed -i 's/varchar/char/g'

Read more...

Exclude SVN directories from Eclipse search

If you perform search in Eclipse IDE and use SVN in your project then probably you have encountered the situation that the search tool looks also in .svn.

To fix this situation you can:

1. Open Project settings
2. Select Resoursec / Resource Filter
3. Click Add... button
4. Select exclude all, applies to folders, all children
5. Type .svn

And that is all you need. The next search will no include svn directories.

Read more...

Never use JBoss Seam version below 2.2.2.Final

Today it will be the small post about very significant security hole in JBoss Seam versions below 2.2.2.Final.

It is possible to execute malware code on your server through Seam application using only browser's address bar. To check the issue add this to your GET parameters(works for Linux):

actionOutcome=/pwn.xhtml?pwned%3d%23{expressions.getClass().forName('java.lang.Runtime').getDeclaredMethods()[6].invoke(expressions.getClass().forName('java.lang.Runtime')).exec('mkdir%20/tmp/pwned')}

This code will create 'pwned' directory in /tmp/ directory.

To close this vulnerability just update your JBoss Seam to 2.2.2.Final.

In this post I used the material of this article.To read more check these links:

1. JBoss Seam Framework remote code execution
2. JBoss Seam2 privilege escalation caused by EL interpolation in FacesMessages
3. Abusing JBOSS
4. Good Bye Critical Jboss 0day

Read more...

Debug remote application with Eclipse

When you have remote application with enabled debug it is easy to connect Eclipse IDE and debug the application. Here is the list of steps for it:

1. Open menu Run, select item Debug Configurations.
2. In the left area of the appeared window double click on Remote Java Application
3. Enter all required information about your server. In the screenshot below you can see configurations for the server configured here
4. Click Apply or Debug

Read more...

JBoss remote debugging

As all you well know debugging is very important part of the development process. In this post I will show you how to configure your JBoss AS to allow debugging deployed applications.

We need to configure JBoss's bin/run.conf file. Add the following line at the end:

JAVA_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n $JAVA_OPTS"

This options mean:

• -Xdebug asks JBoss to work with debugging support
• -Xrunjdwp loads JDWP (Java Debug Wire Protocol). This option has its own options:
  1. transport=dt_socket means that sockets will be used for transport
  2. address=8787 means the address(in this case it's the local machine's port 8787) where the socket will be opened
  3. server=y if it is set to 'y' then it means that JBoss will listen for debugger to attach; if it is set to 'n' then it means that JBoss will attach to the debugger at the specified address
  4. suspend=n if it is set to 'y' then it means that JBoss will be launched in the suspended mode and will stay suspended until the debugger is connected

You may also want to check how to debug remote application with Eclipse

Read more...

LDAP Authentication and Search

LDAP is a very widespread way for authentication. In this post I would like to show you how to connect to LDAP server, authenticate user and perform search.

We will use only standard Java classes. So we don't need any dependency. In the example you can see my configurations. Yours may differ (pay attention to Context.SECURITY_PRINCIPAL). Here is the code:

import java.util.Properties;

import javax.naming.Context;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import javax.naming.directory.Attribute;
import javax.naming.directory.Attributes;
import javax.naming.directory.SearchControls;
import javax.naming.directory.SearchResult;
import javax.naming.ldap.InitialLdapContext;

public class LdapTest {
    
    public static void processRequest(InitialLdapContext ctx, String userContext, String filter, String attribute) {
        SearchControls searchControls = new SearchControls();
        searchControls.setSearchScope(SearchControls.SUBTREE_SCOPE);

        NamingEnumeration<SearchResult> searchResults;
        try {
            searchResults = ctx.search(userContext, filter, searchControls);
            
            while (searchResults.hasMoreElements()) {
                SearchResult sr = searchResults.next();
                Attributes attributes = sr.getAttributes();
                Attribute a = attributes.get(attribute);
                if (a != null) {
                    String attrValue = (a.get().toString());
                    System.out.println(attrValue);
                } else {
                    System.out.println("Cannot get data");
                }
            }
        } catch (NamingException e) {
            e.printStackTrace();
        }
    }
        
    private static InitialLdapContext initialiaseLdapContext(String server, int port, String username, String password, String contextDN) {
        boolean initialised = false; 
        Properties properties = new Properties();
        properties.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
        properties.put(Context.PROVIDER_URL, "ldap://" + server + ":"+ port);
        properties.put(Context.SECURITY_AUTHENTICATION, "simple");
        properties.put(Context.SECURITY_PRINCIPAL, "uid=" + username + "," + contextDN); //YOU MAY NEED TO CHANGE CODE HERE
        properties.put(Context.SECURITY_CREDENTIALS, password);

        InitialLdapContext ctx = null;
        try {
            // Create initial context
            ctx = new InitialLdapContext(properties, null);
            initialised = true;
        } catch (NamingException e) {
            initialised = false;
        } finally {
            if (initialised) {
                System.out.println("Initialization success");
            } else {
                System.out.println("Initialization fail");
            }
        }
        return ctx;
    }

    public static void main(String[] args) {
        String contextDN = "dc=test,dc=com";
        InitialLdapContext ctx = initialiaseLdapContext("localhost", 389, "test@test.com", "test" , contextDN);
        try {
            if (ctx != null) {
                processRequest(ctx, contextDN, "(uid=qwertTest@test.com)", "street");
                ctx.close();
            }
        } catch (NamingException e) {
            e.printStackTrace();
        }
    }
}

Read more...

Fast bulk loading to the database

Whenever you have to insert a big amount of data consider using not usual 'Insert Into', but the load command. For MySQL and Oracle it's 'Load Data Infile'. Here is the small example how to import usual CSV file with the header to MySQL:

load data local infile 'load.csv'
into table someTable
character set utf8 
fields terminated by ',' 
optionally enclosed by '"'
lines terminated by '\n'
ignore 1 lines

For more information view official documentation

Read more...

Fast way to kill many MySQL processes

I think everyone who works for some time with MySQL has been in trouble with suspended processes or reaching the limit of connections. The fastest way to fix it is to restart MySQL service

service mysql restart

But if it's the production server and you cannot afford restarting the service then here is the tip for you.

At first we need to find out what processes are causing troubles. Let's login to MySQL:

mysql -uroot -ptoor

And watch the list of all processes

select * from information_schema.processlist;

or

show processlist;

Let's assume we need to delete processes that are active more then 1000 seconds. To watch them we need this query:

select * from information_schema.processlist where TIME > 1000;

Let's format the output and save it to the file (if you have problems saving to the file then check this post)

select concat('kill ',id,';') 
into outfile '/home/anton/mysql/kill.sql' 
from information_schema.processlist 
where TIME > 1000;

After this let's load the created file

source /home/anton/mysql/kill.sql;

Thus we have deleted multiple MySQL processes with our own criteria.

Read more...

Reading XML files using XPath

Nowadays declarative programming has took a well-deserved place in imperative languages. In Java we can declare what we want in annotations or in XML. The main plus of XML is that we can reconfigure our application in one place and update it without rebuilding.

There a lot of ways to read data from XML. The most common ones are: XPath, XSLT and XQuery.

• XPath consists of path expressions, conditions and some XPath functions
• XSLT(Extensible Stylesheet Language (with) Transformation) consists of XPath and some transformations
• XQuery consists of XPath and a Query Language. This query language is pretty powerful but it does not have strong underlying mathematics like SQL.

In this tutorial we will develop a simple XPath XML file parser. But as you probably know, there are different kind of parsers:

• DOM(Document Object Model) parsers. DOM API creates a DOM tree in memory for a XML document
• SAX(Simple API For XML) parsers. They are event driven(as they parse documents they invoke the callback methods)

We will develop DOM parser. For this purpose we will use only standard Java classes without any additional dependencies.
As an example XML file we will use JBoss 5.1 datasource file. Here it is:

<?xml version="1.0" encoding="UTF-8"?>

<datasources>
    <xa-datasource>
        <jndi-name>DevDbDS</jndi-name>
 <!--
        <use-java-context>false</use-java-context>
 -->
 <xa-datasource-class>
  com.mysql.jdbc.jdbc2.optional.MysqlXADataSource
 </xa-datasource-class>
 <xa-datasource-property name="URL">
  URL:jdbc:mysql://localhost:3306/dev_db
 </xa-datasource-property>

        <user-name>root</user-name>
        <password>root</password>
 <!--
        <security-domain>VocdmsDSEncryptedLogon</security-domain>
 -->
    <xa-datasource-property name="characterEncoding">UTF-8</xa-datasource-property>

 <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter</exception-sorter-class-name>
        <min-pool-size>5</min-pool-size>
        <max-pool-size>20</max-pool-size>
        <blocking-timeout-millis>30000</blocking-timeout-millis>
        <idle-timeout-minutes>15</idle-timeout-minutes>
        <prefill>true</prefill>
        <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
        <connection-property name="characterEncoding">UTF-8</connection-property>
        <connection-property name="autoReconnect">true</connection-property>
        <connection-property name="maxReconnects">4</connection-property>
        <connection-property name="initialTimeout">3</connection-property>
        <metadata>
            <type-mapping>mySQL</type-mapping>
        </metadata>
    </xa-datasource>
</datasources>

Next we need to remind ourselves how to write XPath expressions. You can do it here

Now finally we are ready to write the code to get some information from the datasource. In this example let's assume we need user name, password and URL(you may need this data to establish java.sql.Connection). So here is our code:

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import java.io.*;

public class XmlParser {
 //Immutable object for data storage
 private static class ConnectionInfo {
  private final String url;
  private final String userName;
  private final String password;
  
  public ConnectionInfo(String url, String userName, String password) {
   this.url = url;
   this.userName = userName;
   this.password = password;
  }
 
  public String getUrl() {
   return url;
  }
  public String getUserName() {
   return userName;
  }
  public String getPassword() {
   return password;
  } 
  @Override
  public String toString() {
   return "URL: " + url +
    "\nuserName: " + userName + 
    "\npassword: " + password;
  }
 }

 public static void main(String[] args) {
  System.out.println(parseDataSource("mysql-ds.xml"));
 }

 public static ConnectionInfo parseDataSource(String fileName) {
  try {   
      DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance();
      domFactory.setNamespaceAware(true); // never forget this!
      DocumentBuilder builder = domFactory.newDocumentBuilder();
      Document doc = builder.parse(new FileInputStream(fileName));

      XPathFactory factory = XPathFactory.newInstance();
      XPath xpath = factory.newXPath();
      String url = xpath.evaluate("//datasources/xa-datasource/xa-datasource-property[@name='URL']/text()", doc).trim();     
      String userName = xpath.evaluate("//datasources/xa-datasource/user-name/text()", doc).trim();
      String password = xpath.evaluate("//datasources/xa-datasource/password/text()", doc).trim();
      return new ConnectionInfo(url, userName, password);
  } catch (Exception ex) {
      ex.printStackTrace(); 
      return null;
  }
     } 
}

Read more...

Google Analytics API usage example

It's very useful in SEO business to keep track of different user's action on your site. One of the best tools for such kind of analysis is Google Analytics. Furthermore it's free and provides the API. In this post I will show you the simple example of usage of this API.

As usual let's start with the Maven dependencies. But unfortunately Google do not provide jars that we need in any repository. Thus we need to download and install them to Maven by ourselves. Google for "gdata core jar" and "gdata analytics jar". When you will have those files execute in command line following code to install these jars:

mvn install:install-file -Dfile=gdata-core-1.0.jar -DgroupId=com.google.gdata -DartifactId=gdata-core -Dversion=1.0 -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -Dfile=gdata-analytics-2.1.jar -DgroupId=com.google.gdata -DartifactId=gdata-analytics -Dversion=2.1 -Dpackaging=jar -DgeneratePom=true

Add to pom.xml following Maven dependencies:

<dependency>
    <groupId>com.google.gdata</groupId>
    <artifactId>gdata-core</artifactId>
    <version>1.0</version>
</dependency>
<dependency>
    <groupId>com.google.gdata</groupId>
    <artifactId>gdata-analytics</artifactId>
    <version>2.1</version>
</dependency>

Also you may need Guava(which is Google's library with different basic and universal functions.

<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>10.0.1</version>
</dependency>

After maven fetches us our new dependencies we may continue and develop Java source code for Google Analytics API data fetching.

public static DataFeed getDataFeed(String dimensions, String metrics, String userName, String password, 
        String tableId, String startDate, String endDate, int startIndex) throws IOException, ServiceException  {
    System.out.println("Building data feed");
    // Configure GA API.
    AnalyticsService as = new AnalyticsService("My application name");

    // Client Login Authorization.
    as.setUserCredentials(userName, password);
    DataQuery query = new DataQuery(new URL("https://www.google.com/analytics/feeds/data"));
    query.setIds(tableId);
    query.setDimensions(dimensions);
    query.setMetrics(metrics);
    query.setSort("-" + metrics);
    query.setMaxResults(10000);        
    query.setStartDate(startDate);
    query.setEndDate(endDate);
    query.setStartIndex(startIndex);
    URL url = query.getUrl();
    
    System.out.println("URL: " + url.toString());

    // Send our request to the Analytics API and wait for the results to
    // come back.
    DataFeed feed = as.getFeed(url, DataFeed.class);
    return feed;
}

As a parameters you should provide

• dimensions and metrics (you can get both from Google Analytics API Reference)
• Google Analytics credentials (user name and password)
• Google Analytics Profile ID(also called table ID)
• date range for quering
• start index(We need to use start index because Google Analytics service won't return result set with more then 10000 records)

Let's save data that we got to CSV file using opencsv:

public static void saveAnalyticsFile(DataFeed feed, String fileName) {
 System.out.println("Saving to file");
    try {
        CSVWriter writer = new CSVWriter(new FileWriter(fileName), ',' ,CSVWriter.DEFAULT_QUOTE_CHARACTER, '\\');
        
        DataEntry singleEntry = feed.getEntries().get(0);
        List<String> feedDataNames = new ArrayList<String>();
        List<String> dataRowValues = new ArrayList<String>(LoaderConstants.GA_FILE_HEADER.length);

        // Put all the dimension and metric names into an array.
        for (Dimension dimension : singleEntry.getDimensions()) {
            feedDataNames.add(dimension.getName());
        }
        for (Metric metric : singleEntry.getMetrics()) {
            feedDataNames.add(metric.getName());
        }

        //write header
        writer.writeNext(LoaderConstants.GA_FILE_HEADER);

        for (DataEntry entry : feed.getEntries()) {         
         //assuming that the first entry is a keyword and others are numeric values
            if (!entry.stringValueOf(feedDataNames.get(0)).equals("(not provided)") && 
                    !entry.stringValueOf(feedDataNames.get(0)).equals("(not set)")) {
             dataRowValues.add(entry.stringValueOf(feedDataNames.get(0))); //keyword
             for (int i = 1; i < feedDataNames.size(); i++) {
                 Double d = Double.parseDouble(entry.stringValueOf(feedDataNames.get(i)));
                 String googleVal = new Long(d.longValue()).toString();
                 dataRowValues.add(googleVal); 
             }
                writer.writeNext(dataRowValues.toArray(new String[0]));
             
                dataRowValues.clear();
            }
        }
        writer.close();        
        System.out.println("Saving is done");
        
    } catch (Exception e) {
        System.out.println("Cannot save file ");
        throw new RuntimeException(e);
    }
}

And finally here is the main method:

public static void main(String[] args) {
 int startIndex = 1;
 boolean fetch = true;
 while (fetch) {
     DataFeed dataFeed = getDataFeed("ga:keyword", "ga:visits", "userName", "password", 
                "profileId", "2012-08-13", "2012-08-19", startIndex);
        CsvWriteUtils.saveAnalyticsFile(dataFeed, "test");
        startIndex += 10000; //Google Analytics Max result value
        //continue fetching until we receive all data
        if (startIndex > dataFeed.getTotalResults()) {
         fetch = false;
        }
 }
}

Read more...

LDAP recovery

If you have suffered from LDAP wreckage then probably you already know how difficult it is to make it work again. In my case all problems started with such message in system log:

slapd[11705]: bdb(dc=xxx,dc=com): PANIC: fatal region error detected; run recovery

Note: my bdb version is 4.7, yours may differ;
slapd stands for Stand-alone LDAP Daemon, your service name may be ldap

At first I tried to recover LDAP using

db4.7_verify -h /var/lib/ldap
db4.7_recover -v -h /var/lib/ldap 

But LDAP got broken again and again.
Thus I decided to recreate whole LDAP folder. Here is the scenario:

1. Stop the LDAP server
   

   service slapd stop

2. Dump the directory structure to a text file
   

   slapcat -l /etc/ldap/backup/ldap_old.ldif

3. Recover LDAP
   

   db4.7_recover -v -h /var/lib/ldap

4. Dump the directory structure to a text file
   

   slapcat -l /etc/ldap/backup/ldap.ldif

5. Verify that ldap.ldif contains your entries. If it does not, or if slapcat returned errors in step 3, try running db4.7_recover in catastrophic mode:
   

   db4.7_recover -v -h /var/lib/ldap -c

   and repeat step 3
6. Delete the LDAP directory
   

   rm -fr /var/lib/ldap/*

7. Load LDAP directory from the file you have create in step 3
   

   slapadd -l /etc/ldap/backup/slap.ldif

8. Make openldap the owner of the LDAP directory
   

   chown -R openldap:openldap /var/lib/ldap/*

9. Start the LDAP server
   

   service slapd start

Read more...

AuthorityLabs Partner API

If you need to get SERP(search engine results page) rank information on some keywords, there is a very good solution - AuthorityLabs Partner API. I have successfully used it on one of my projects and I want to share with you some tips on developing functionality to work with it.

AuthorityLabs Partner API is as a REST service. The main API methods are for POSTing keywords that you want to get info on and GETting the information from POSTed keywords. There are 2 different ways of work with POST requests:

• Instant POST request(expensive)
• Delayed POST request(cheaper)

We'll work with Delayed requests.
At first we should think how are we going to work with the Web service. I have chosen to use Apache HttpComponents because it's straightforward and easy to use. To download the library I used the following Maven dependencies:

<dependency>
 <groupId>org.apache.httpcomponents</groupId>
 <artifactId>httpcore</artifactId>
 <version>4.2</version>
</dependency>
 <dependency>
 <groupId>org.apache.httpcomponents</groupId>
 <artifactId>httpclient</artifactId>
 <version>4.1.3</version>
</dependency>

Now let's develop method to POST keywords to the AuthorityLabs Partner API.

public class AuthorityLabsTest {
 private static final String AUTHORITYLABS_AUTH_TOKEN = "XXX";
 private static final String AUTHORITYLABS_URL_POST_KEYWORD = "http://api.authoritylabs.com/keywords";

 
  /**
     * Post keyword to the delayed queue of the AuthorityLabs API
     * @param keyword
     * @param engine
     * @param locale
     * @return
     */
 static boolean postKeyword(String keyword, String engine, String locale) {
  HttpClient client = new DefaultHttpClient();
  
  HttpPost post = new HttpPost(AUTHORITYLABS_URL_POST_KEYWORD);
  try {
   List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(1);
   nameValuePairs.add(new BasicNameValuePair("auth_token", AUTHORITYLABS_AUTH_TOKEN));
   nameValuePairs.add(new BasicNameValuePair("keyword",keyword));
   nameValuePairs.add(new BasicNameValuePair("engine", engine));
   nameValuePairs.add(new BasicNameValuePair("locale", locale));  
   post.setEntity(new UrlEncodedFormEntity(nameValuePairs));
   System.out.println("Post keyword; parameters - " + nameValuePairs);
 
   HttpResponse response = client.execute(post);
   BufferedReader rd = new BufferedReader(new InputStreamReader(
     response.getEntity().getContent()));
   String line = "";
   while ((line = rd.readLine()) != null) {
    System.out.println(line);
   }

  } catch (IOException e) {
   e.printStackTrace();
   return false;
  }
  return true;
 }
}

Next we need method to GET keyword info.

public class AuthorityLabsTest {
 private static final String AUTHORITYLABS_AUTH_TOKEN = "XXX";
 private static final String AUTHORITYLABS_URL_POST_KEYWORD = "http://api.authoritylabs.com/keywords";
 private static final String AUTHORITYLABS_URL_GET_RESULTS = "http://api.authoritylabs.com/keywords/get.json?keyword={0}&auth_token={1}&engine={2}&locale={3}&rank_date={4}";

 static boolean postKeyword(String keyword, String engine, String locale) {
    ...
 }
 
 /**
  * Gets JSON results from the AuthorityLabs API
  * @param keyword
  * @param engine
  * @param locale
  * @param stringPostKeywordDate
  * @return
  */
 static InputStream getKeywordResults(String keyword, String engine, String locale, String stringPostKeywordDate) {
  String urlPattern = AUTHORITYLABS_URL_GET_RESULTS;
  String url = MessageFormat.format(urlPattern, keyword, AUTHORITYLABS_AUTH_TOKEN, 
    engine, locale, stringPostKeywordDate);
  //Prepare URL (e.g. replace spaces with %20 etc)
  try {
   url = URIUtil.encodeQuery(url);
  } catch (URIException e1) {
   e1.printStackTrace();
   return null;
  }
  System.out.println("Get keyword results; url - " + url);
  
  HttpClient client = new DefaultHttpClient();
  HttpGet get = new HttpGet(url); 
  HttpResponse response;
  try {
   response = client.execute(get);
   System.out.println(response.getStatusLine());   
   return response.getEntity().getContent();
  } catch (Exception e) {
   e.printStackTrace();
  }   
  return null;
 }
}

Note that we need to encode URL in order to be able to get results. To choose correct engine and locale for that engine please check the documentation on AuthorityLabs API.

That's the core of the work with AuthorityLabs API.

Read more...

MySQL - SELECT ... INTO OUTFILE ... - ERROR 1 (HY000)

There is a very useful functionality in MySQL for exporting data into a file. The syntax is like:

SELECT <FIELDS> INTO OUTFILE <FILE_NAME> FROM <TABLE_NAME>;

If you use Linux then when you execute this export command with FILE_NAME that is not absolute than the file will be saved in the database folder (e.g. FILE_NAME = '1.csv'); if you specify to use /tmp folder then it will be saved there(e.g. FILE_NAME='/tmp/1.csv').
But if you specify some other directory (e.g. FILE_NAME='/home/anton/mysql/1.csv') then you will get error:

ERROR 1 (HY000): Can't create/write to file '/home/anton/mysql/1.csv' (Errcode: 13)

Even if you will set MySQL user as an owner of the directory and will provide all access to it, this error will appear nevertheless.

The reason lays in the Linux kernel security module AppArmor. This module restricts programs possibilities and we have stumbled upon one of those. Thus the solution is pretty obvious - we need to configure AppArmor. For this we will add at the end (but before the closing brace) of the file /etc/apparmor.d/usr.sbin.mysqld (for Ubuntu) such lines:

/home/anton/mysql/ w,
/home/anton/mysql/* rwk,

Then we need to reload this configuration file with command (for Ubuntu):

sudo /etc/init.d/apparmor reload

Now check that the directory that you want to use (/home/anton/mysql) has 755 rights and mysql us is the owner of it. Also make sure that there is enough free space on the volume.
Repeat export SELECT command

Read more...