Monday, August 9, 2010

How to load/write XML using JAXB?

Every now and then you need to load or write some XML files. It is very effective way to put data in machine-readable form. XML is text format file which is well formatted using standardized rules.

So to demonstrate use of JAXB we will:
1) create XML file with some data (bookmarks.xml)
2) create XML schema file to validate our XML file (bookmarks.xsd)
3) create XML object model using xjc utility (xjc.txt)
4) load that XML file into JAXB object (XMLUnmarshall.java)
5) write JAXB object back over original XML file (XMLMarshaller.java)
6) create main java class to call these functions in appropriate order (JAXBLoader,java)

Create XML file
Lets create a simple xml file for storing bookmark data organized through sets. It will contain root element bookmarks with bookmark elements inside and list of attribute elements for storing href values. Also for every bookmark element there will be a attribute set which will identify current bookmark set. Example of this xml can be found here: bookmarks.xml.

Create XML schema file
To validate bookmark.xml we need to create schema file. Example for this example can be found here: bookmarks.xsd.

Create object model
To create object model for our xml we need to run xjc utility and supply it with xsd schema file as argument at command line.

zlaja@orion:~/NetBeansProjects/JAXBLoader> xjc ./data/bookmarks.xsd -p com.blogspot.zetaorionis.bookmarks.model -d ./src
parsing a schema...
compiling a schema...
com/blogspot/zetaorionis/bookmarks/model/AttributeType.java
com/blogspot/zetaorionis/bookmarks/model/Bookmark.java
com/blogspot/zetaorionis/bookmarks/model/Bookmarks.java
com/blogspot/zetaorionis/bookmarks/model/ObjectFactory.java
com/blogspot/zetaorionis/bookmarks/model/package-info.java
zlaja@orion:~/NetBeansProjects/JAXBLoader>

Unmarshalling / loading XML data to object model instance
This utility class will help you read xml file into it's object representation. It will work with every xml file. You just need to supply it with uri path to xml file:
data/bookmarks.xml

and root element class of object model:
Bookmarks.class

Here is XMLUnmarshaller.java utility class:

package com.blogspot.zetaorionis.util.xml;

import java.io.FileNotFoundException;
import java.io.InputStream;
import java.io.IOException;
import java.net.URI;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;
import javax.xml.transform.stream.StreamSource;

import org.apache.log4j.Logger;

/**
*
* @author zlaja
*/
public class XMLUnmarshaller {

private static final Logger logger = Logger.getLogger(XMLUnmarshaller.class);
/**
* The bookmark file.
*/
protected URI uri;

/**
* Create an xml based loader of bookmarks.
* @param uri the bookmark file
*/
public XMLUnmarshaller(final URI uri) {
this.uri = uri;
}

//
// BookmarksXMLLoader
//
/**
* Load bookmarks into a action list.
*
* @throws IOException
* If an I/O error occurred.
* @throws FileNotFoundException
* If the resource was not found.
*/
public T load(Class docClass) throws IOException {
final InputStream in = getClass().getResourceAsStream("/" + uri.getPath());

if (in == null) {
throw new FileNotFoundException("Cannot find resource: " + uri);
}

try {
return load(docClass, in);
} finally {
in.close();
}
}

protected T load(Class docClass, final InputStream in) {
T o = null;
try {
o = unmarshal(docClass, in);
} catch (JAXBException ex) {
logger.error("Error while unmarshalling.", ex);
}
return o;
}

public T unmarshal(Class docClass, InputStream inputStream)
throws JAXBException {
String packageName = docClass.getPackage().getName();
JAXBContext jc = JAXBContext.newInstance(packageName);
Unmarshaller u = jc.createUnmarshaller();
JAXBElement doc = u.unmarshal(new StreamSource(inputStream), docClass);
return doc.getValue();
}
}

Marshalling / writing data to xml from object model instance
This utility class will help you write data from object representation to xml file. It will work with every xml file. You just need to supply it with uri path to xml file:
data/bookmarks.xml

and root element class of object model:
Bookmarks.class

Here is XMLMarshaller.java utility class:
package com.blogspot.zetaorionis.util.xml;

import java.io.FileNotFoundException;
import java.io.OutputStream;
import java.io.FileOutputStream;
import java.net.URI;
import java.io.IOException;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;

import org.apache.log4j.Logger;

/**
*
* @author zlaja
*/
public class XMLMarshaller {

private static final Logger logger = Logger.getLogger(XMLUnmarshaller.class);
/**
* The output XML file.
*/
protected URI uri;

/**
* Create an xml based writer for specified jaxbObject.
* @param uri - uri for output XML file
*/
public XMLMarshaller(final URI uri) {
this.uri = uri;
}

//
// XMLMarshaller
//
/**
* Write JAXBElement representation of object to XML file.
*
* @param jaxbObject - object for marshalling to xml, converted to JAXBElement.
* Conversion is done using function inside ObjectFactory.java which is
* created with xjc utility
* @param docClass - class for object that is going to be marshalled to XML
*
* @throws IOException
* If an I/O error occurred.
* @throws FileNotFoundException
* If the resource was not found.
*/
public void write(final JAXBElement jaxbObject, Class docClass) throws IOException {
final OutputStream os = new FileOutputStream(uri.getPath());

if (os == null) {
throw new FileNotFoundException("Cannot create resource: " + uri);
}

try {
write(jaxbObject, docClass, os);
} finally {
os.close();
}
}

protected void write(final JAXBElement jaxbObject, Class docClass, final OutputStream os) {
try {
marshall(jaxbObject, docClass, os);
} catch (JAXBException ex) {
logger.error("Error in marshalling to XML.", ex);
}
}

private void marshall(final JAXBElement jaxbObject, Class docClass, final OutputStream os)
throws JAXBException {
String packageName = docClass.getPackage().getName();
JAXBContext context = JAXBContext.newInstance(packageName);
Marshaller m = context.createMarshaller();
m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
m.marshal(jaxbObject, os);
}
}

Call utility classes inside main class
package com.blogspot.zetaorionis.jaxbloader;

import com.blogspot.zetaorionis.bookmarks.model.Bookmarks;
import com.blogspot.zetaorionis.bookmarks.model.ObjectFactory;
import com.blogspot.zetaorionis.util.xml.XMLMarshaller;
import com.blogspot.zetaorionis.util.xml.XMLUnmarshaller;
import java.net.URI;
import org.apache.log4j.Logger;

/**
*
* @author zlaja
*/
public class JAXBLoader {

private static final String LOG4J_PROPERTIES = "data/log4j.properties";
private static final Logger logger = Logger.getLogger(JAXBLoader.class);
private Bookmarks bookmarks = new Bookmarks();

public JAXBLoader() {
Log4J.init(LOG4J_PROPERTIES);
}

/**
* Loads XML data to object
*/
private void loadBookmarks() {
try {
final String path = "data/bookmarks.xml";
final URI uri = new URI(path);

final XMLUnmarshaller xmlBookmarks = new XMLUnmarshaller(uri);
this.bookmarks = xmlBookmarks.load(Bookmarks.class);
logger.info("Info: Bookmarks loaded successfuly.");
} catch (Exception ex) {
logger.error("Error: Loading bookmarks XML file failed", ex);
}
}

/**
* Writes object data to XML
*/
private void writeBookmarks() {
try {
final String path = "data/bookmarks.xml";
final URI uri = new URI(path);

final XMLMarshaller xmlBookmarks = new XMLMarshaller(uri);
ObjectFactory of = new ObjectFactory();
xmlBookmarks.write(of.createBookmarks(this.bookmarks), Bookmarks.class);
logger.info("Info: Bookmarks written successfuly.");
} catch (Exception ex) {
logger.error("Error: Writing bookmarks XML file failed", ex);
}
}

/**
* @param args the command line arguments
*/
public static void main(String[] args) {
JAXBLoader loader = new JAXBLoader();
loader.loadBookmarks();
loader.writeBookmarks();
}
}

Saturday, February 20, 2010

Use yubikey for safer and less painful browsing

What is yubikey?
Yubikey is usb powered piece of hardware with one action button to generate OTP (one time passwords).

Main advantages:
- There is no need to remember credentials for different sites on the web. Just plug in the yubikey and press action button which will create random OTP password with unique header number of that particular yubikey.
- It will guarantee user inviolability and thus disable fishing attacks and credential information interception.

Main disadvantages:
- You have to carry it with you :), but you can access your web portals without using it in old fashion way by typing your credentials.

Settings things up:
You need to register your account at KeyGenius:
http://kg.yubico.com/

You can use basic or standard account type. Basic will not ask you for any password when logging and all you need is one touch on your yubikey for every login on the web. You are released of typing. Standard account is more secure. It will ask you on first access for your KeyGenius credentials, and after supplying it you can use yubikey like in basic account for every other portal on the web.

Under your account you need to supply information like username, passsword and domain for every domain and credential information.

So it's like this. You want to login on e.g. facebook.com. You will open facebook in browser. Your username will be remembered by your browser. Instead of typing passwords you plug your yubikey and touch button. Login process will continue automatically. First it will send request to KeyGenius to return real password for facebook and after receiving password browser will log you to facebook.

Browser will need to know how to get password from KeyGenius. That is accomplished with java script which can be installed to browser using GreasyMonkey addon. Script can be found here:
http://kg.yubico.com/keygenius.user.js

And thats it. Enjoy your safe browsing... :)

Database synchronization using TableSyncer

Prerequisites:
ruby
ruby-devel
ruby-mysql
rubygems
libmysqlclient
libmysqlclient-devel
libopenssl-devel
zlib-devel

Installation:
sudo gem install mysql
sudo gem install table_syncer

Settings:
cd /usr/lib64/ruby/gems/1.8/gems/table_syncer-0.3.1/lib
cp table_syncer.rb table_syncer.rb.orig
vi table_syncer.rb

Add and change these lines:
local_source_db = {:host => '127.0.0.1', :user => 'user', :password => 'password', :db => 'SourceDatabase'}
local_test_db = {:host => '127.0.0.1', :user => 'user', :password => 'password', :db => 'test'}

Executing
table_syncer --from=local_source_db --to=local_test_db --tables=account

Reference:
http://code.google.com/p/ruby-roger-useful-functions/wiki/TableSyncer
http://www.freelinuxtutorials.com/quick-tips-and-tricks/sync-mysql-tables-via-ruby-gem-tablesyncer/
http://forums.mysql.com/read.php?116,178217,198518#msg-198518

SCP & SSH

At local machine:
ssh-keygen -t rsa
cd ~/.ssh
cp id_rsa.pub authorized_keys
scp -p ~/.ssh/authorized_keys username@remoteMachine:.ssh/

Note that before executing scp you need to make sure there is ~/.ssh at remote machine also and create it if neccecery with
mkdir ~/.ssh

At remote machine:
cd
ls -ld . .ssh .ssh/authorized_keys
drwxr-xr-x 36 username username 4096 Jul 25 02:24 .
drwxr-xr-x 2 username username 512 Apr 10 02:30 .ssh
-rw-r--r-- 1 username username 1674 Apr 10 02:29 .ssh/authorized_keys

cd
chmod go-w . .ssh .ssh/authorized_keys

At local machine:
scp -p file username@remoteMachine:path/to/file

Reference:
http://kimmo.suominen.com/docs/ssh/

Mounting remote windows station

1) Using mount command:
mount -t cifs //192.168.0.4/Movies ~/Desktop/movies/

2) Using /etc/fstab file:
//192.168.0.4/Movies /media/remote/movies cifs file_mode=0777,dir_mode=0777,password=******** 0 0

Mounting iso file image:
mkdir /media/iso
mount -o loop -t iso9660 ~/file.iso /media/iso

Monday, March 30, 2009

Installing Plone True Gallery

Problem
It should be simple as editing buildout.cfg and runing buildout again.
Edit buildout.cfg with:
$ joe buildout.cfg

Under eggs and zcml sections add collective.plonetruegallery string like:
eggs =
collective.plonetruegallery
...
zcml =
collective.plonetruegallery

Run buildout again with:
$ ./bin/buildout -v

This will give you the error:
Error: Couldn't find a distribution for 'gdata.py>=1.2.3'.

Problem is gdata.py package doesn't exists because it is now known as gdata only.

Solution
$ wget http://pypi.python.org/packages/source/c/collective.plonetruegallery/collective.plonetruegallery-0.6b2.4.tar.gz
$ tar xfz collective.plonetruegallery-0.6b2.4.tar.gz
$ cd collective.plonetruegallery-0.6b2.4
$ joe setup.py
Change gdata.py to gdata under dependency part so it looks like this:
install_requires=[
'setuptools',
'gdata>=1.2.3',
'flickrapi>=1.2',
'simplejson',
'elementtree'
],
$ sudo python2.4 setup.py install
$ joe buildout.cfg

Add zcml slug like:
zcml =
collective.plonetruegallery

Run buildout again with:
$ ./bin/buildout -v

Run your instance with:
$ ./bin/instance fg

Now, it should be under 'add on products' so you can install it.

Sunday, March 29, 2009

Managing projects using buildout

Directories in the buildout
Before we dive into buildout.cfg, let us take a quick look at the directories that buildout has created for us:
bin/

Contains various executables, including the buildout command, and the instance Zope control script.
eggs/

Contains eggs that buildout has downloaded. These will be explicitly activated by the control scripts in the bin/ directory.
downloads/

Contains non-egg downloads, such as the Zope source code archive.
var/

Contains the log files (in var/log/) and the file storage ZODB data (in var/filestorage/Data.fs). Buildout will never overwrite these.
src/

Initially empty. You can place your own development eggs here and reference them in buildout.cfg. More on that later.
products/

This is analogous to a Zope instance's Products/ directory (note the difference in capitalisation). If you are developing any old-style Zope 2 products, place them here. We will see how buildout can automatically download and manage archives of products, but if you want to extract a product dependency manually, or check one out from Subversion, this is the place to do so.
parts/

Contains code and data managed by buildout. In our case, it will include the local Zope installation, a buildout-managed Zope instance, and Plone's source code. In general, you should not modify anything in this directory, as buildout may overwrite your changes.

The main [buildout] section
The [buildout] section is the starting point for the file. It lists a number of "parts", which are configured in separate sections later in the file. Each part has an associated recipe, which is the name of an egg that knows how to perform a particular task, e.g. build Zope or create a Zope instance. A recipe typically takes a few configuration options.

Our global settings are as follows:
[buildout]
parts =
plone
zope2
productdistros
instance
zopepy
find-links =
http://dist.plone.org
http://download.zope.org/ppix/
http://download.zope.org/distribution/
http://effbot.org/downloads
eggs =
elementtree
develop =

This specifies that the parts plone, zope2, productdistros, instance and zopepy will be run, in that order. Then, we tell buildout that it can search one of a number of URLs when it is looking for eggs to download. In addition, it will always search the Cheese Shop.

Next, we can list any eggs that buildout should download and install for us. This may include version specifications. For example, if you want sqlalchemy 0.3, but not 0.4, you could list;
eggs =
elementtree
sqlalchemy>=0.3,<0.4dev

Finally, we can list development eggs, by specifying a directory where the egg is extracted in source format. For example:
eggs =
elementtree
my.package
develop =
src/my.package

This presumes that there is an egg called my.package in the src/ directory. We will learn how to create such eggs a little later in this tutorial. Notice how we must also list my.package as an actual egg dependency: development eggs are not automatically added to the "working set" of eggs that are installed for Zope.

The [plone] section
This is very simple - it just uses plone.recipe.plone to download Plone's products and eggs.
[plone]
recipe = plone.recipe.plone

It will use the latest release available. Version numbers for plone.recipe.plone correspond to version numbers for Plone itself. Therefore, to make sure you always get a 3.0.x release, but not a 3.1, you can do:
[plone]
recipe = plone.recipe.plone>=3.0,<3.1dev

When the recipe is run, Plone's products will be installed in parts/plone. The eggs are made available via buildout variable ${plone:eggs}, which we will reference in the [instance] section later, and the URL of a "known good" version of Zope is available in the variable ${plone:zope2-url}.

The [zope2] section
This part builds Zope 2, using plone.recipe.zope2install. If you specified an existing Zope installation, you will not have this part. Otherwise, it looks like this:
[zope2]
recipe = plone.recipe.zope2install
url = ${plone:zope2-url}

Here, we reference the download location for Zope as emitted by the [plone] part. This ensures that we always get the recommended version of Zope. You could specify a download URL manually instead, if you wanted to use a different version of Zope.

When the recipe is run, Zope 2 is installed in parts/zope2. The Zope software home becomes parts/zope2/lib/python.

The [productdistros] section
This uses the plone.recipe.distros recipe, which is able to download distributions (archives) of Zope 2 style products and make them available to Zope. It is empty to begin with:
[productdistros]
recipe = plone.recipe.distros
urls =
nested-packages =
version-suffix-packages =

However, you can list any number of downloads. The recipe is also able to deal with archives that contain a single top-level directory that contains a bundle of actual product directories (nested-packages), or packages that have a version number in the directory name and thus need to be renamed to get the actual product directory (version-suffix-packages).

Consider the following distributions:

# A typical distribution

ExampleProduct-1.0.tgz
|
|- ExampleProduct
| |
| |- __init__.py
| |- (product code)

# A version suffix distribution

AnotherExampleProduct-2.0.tgz
|
|- AnotherExampleProduct-2.0
| |
| |- __init__.py
| |- (product code)

# A nested package distribution

ExampleProductBundle-1.0.tgz
|
|- ExampleProductBundle
| |
| |- ProductOne
| | |- __init__.py
| | |- (product code)
| |
| |- ProductTwo
| | |- __init__.py
| | |- (product code)

Here is what the part would look like if we try to install the three distributions above:
[productdistros]
recipe = plone.recipe.distros
urls =
http://example.com/dist/ExampleProduct-1.0.tgz
http://example.com/dist/AnotherExampleProduct-2.0.tgz
http://example.com/dist/ExampleProductBundle-1.0.tgz
nested-packages = ExampleProductBundle-1.0.tgz
version-suffix-packages = AnotherExampleProduct-2.0.tgz

You can specify multiple downloads on separate lines. When the recipe is run, the product directories for downloaded products are found in parts/productdistros.

The [instance] section
The instance section pulls it all together: It configures a Zope instance using the plone.recipe.zope2instance script. Here is how it looks:
[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
verbose-security = on
eggs =
${buildout:eggs}
${plone:eggs}
zcml =
products =
${buildout:directory}/products
${productdistros:location}
${plone:products}

Here, we reference the Zope 2 installation from the [zope2] part - if you specified a location yourself when creating the buildout, you would see that one here. Then, we specify the initial admin user and password, and the port that Zope will be bound to. We also turn on debug mode and verbose security. These options are used to generate an appropraite zope.conf file for this instance. See the recipe page in the Cheese Shop for more details on the options available.

Next, we specify which eggs that will be made available to Zope. This references the "global" eggs from the [buildout] section, as well as the eggs specified by Plone. You could add additional eggs here, though it is generally easier to specify these at the top of the file, so that they get included in the ${buildout:eggs} working set.

As explained previously, Zope 3 configure.zcml files are not loaded automatically for eggs or packages not the Products namespace. To load ZCML files for a regular package, we can make buildout create a ZCML slug by listing the package under the zcml option:
zcml =
my.package
my.package-overrides

This assumes that my.package was previously referenced in the buildout. This would load both the main configure.zcml and the overrides.zcml file from this package.

Finally, we list the various directories that contain Zope 2 style products - akin to the Products/ directory in a traditional instance. Notice how the products/ directory in the main buildout directory comes first, followed by the products downloaded with the [productdistros] part, followed by the products downloaded by the [plone] part. This means that even if Plone ships with a product, you could override it (e.g. with a newer product) by putting a product with the same name in the top-level products/ directory.

When the recipe is run, the Zope instance home will be parts/instance, and a control script is created in ./bin/instance.

The [zopepy] section
This final section creates a Python interpreter that has all the eggs and packages (but not Zope 2 style products) that Zope would have during startup. This can be useful for testing purposes.
[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

Here, we copy the eggs from the [instance] section, and include in the pythonpath the Zope instance home.

When the recipe is run, the script will be created in ./bin/zopepy.

Managing ZCML files
It is important to realize that Zope will not load configure.zcml files automatically for packages that are not in the Products.* namespace. Instead, you must explicitly reference the package. Buildout can create such a reference (known as a ZCML slug) with the zcml option under the [instance] part. Here is how to ensure that borg.project is available to Zope:
[buildout]
...
eggs =
elementtree
borg.project
...
[instance]
...
zcml =
borg.project

Should you need to load an overrides.zcml or a meta.zcml, you can use a syntax like:
zcml =
some.package
some.package-overrides
some.package-meta


Resources:
http://plone.org/documentation/tutorial/buildout/tutorial-all-pages