Python cmd.exe UnicodeEncodeError

October 11th, 2010

Problem

I have written a script that calls SVN command using Popen. It looked something like this:

up = subprocess.Po­pen([SVN, „up“, file, ‚-r‘, revision], cwd=dest, stdout=subpro­cess.PIPE).com­municate()[0]

It seemed to work OK until I got a bug report including this trace:

Traceback (most recent call last):
  File "E:\zdir\checkout.py", line 212, in <module>
    up = subprocess.Popen([SVN, "up", file, '-r', revision], cwd=dest,
stdout=subprocess.PIPE).communicate()[0]
  File "C:\Python27\lib\subprocess.py", line 672, in __init__
    errread, errwrite)
  File "C:\Python27\lib\subprocess.py", line 882, in _execute_child
    startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position
103: ordinal not in range(128)

The string that did not get processed was Änderungen and the problem occurred only if the script was run from cmd.exe (it worked OK from Linux shell and Windows PyDev console).

Solution

Having tried many things which all ended in UnicodeEncodeError. The only solution I found working was to use encode method. locale.getpre­ferredencodin­g() seems to be the right choice to keep the program portable. If you don't provide the encode() method with this argument, it will again end up in some UnicodeEncodeError.

The result code looks like this:

up = subprocess.Po­pen([SVN, „up“, file.encode(lo­cale.getprefe­rredencoding()), ‚-r‘, revision], cwd=dest, stdout=subpro­cess.PIPE).com­municate()[0]

Rational Team Concert Workflow Visualiser

April 9th, 2010

Overview

Example workflow diagram Workflow Visualiser is a tool that takes the RTC process configuration XML and exports graphical representation of the workflows included in the configuration.

Motivation

The most typical task I'm facing at new RTC deployments is analysis of their workflow and its implementation in RTC. The usual procedure is:

  1. Meet with the customer to try to understand the workflow (states, transitions)
  2. Draw the workflow in Microsoft Visio and discuss the Visio files with customer
  3. Implement the workflow in RTC

This would be all great it there wasn't number 4 – implement the changes the customer finds after the implementation in RTC, which usually repeats several times. In this moment the Visio diagrams and the RTC implementation get out of sync and it is not easy to keep them synchronized. Workflow Visualiser helps in this step as I can export the real diagram and either send it to the customer for review or redraw it in Visio.

Another case in which this tool comes handy are big workflows (with 10< states). Although the transition table in RTC workflow designer is straightforward, checking such a workflow is not an easy task. Seeing a graphical representation of the workflow makes it clear.

Running the program

The program has to be invoked from command line and the command line arguments are:

Usage: java -jar visualiser.jar [options] source-xml-file
 Options:
  -h|--help Print help and exit
  --version Print help and exit
  -d|--dot Location of dot executable (defaults to /usr/bin/dot)
  -o|--out Output directory (defaults to /tmp/)
  -f|--format Output format (see the output formats of dot) (defaults to pdf)
  -s|--source Save the DOT source file in the output directory

Examples

Windows example PDF
  • JAR stored in C:\visualiser.jar
  • Process XML configuration file saved in process.xml
  • Graphviz installed in standard location
  • Export diagrams in PDF format to „C:\temp“
java -jar "C:\visualiser.jar" -o "C:\temp" -d "C:\Program Files\Graphviz2.26.3\bin\dot.exe" process.xml

Windows example PNG
  • JAR stored in C:\visualiser.jar
  • Process XML configuration file saved in process.xml
  • Graphviz installed in standard location
  • Export diagrams in PNG format to „C:\temp“
java -jar "C:\visualiser.jar" -o "C:\temp" -d "C:\Program Files\Graphviz2.26.3\bin\dot.exe" -f png process.xml

Linux example
  • JAR stored in ~/visualiser.jar
  • Process XML configuration file saved in process.xml
  • Graphviz installed in standard location (/usr/bin/dot)
  • Export diagrams in PDF format to ~/diagrams
java -jar ~/visualiser.jar -o ~/diagrams process.xml
Download

Download

Prerequisites

The program uses Graphviz for diagram drawing. You must have Graphviz installed on your computer to be able to run the program. It can be freely downloaded from http://www.graphviz.org/Download.php

Binary

Download Runnable JAR

Source code

Source code is available at GitHub: https://github.com/…w-visualiser. It includes pom.xml with the Maven project dependencies.

License

Pro program is distributed under Apache License, Version 2.0. For the license agreement of the used libraries see the appropriate license agreement included in the distribution.

Version history

  • 2010–04–09 Initial version
  • 2010–04–09.2 Resolution edges corrected

Comments, bug reports

Any comments or bug reports are welcome. Send them either in the comments under this article or via email to honza@stastnarodina.com

RQM on 64bit Ubuntu

February 7th, 2010

I installed IBM Rational Quality Manager 2.0.0.1 on the same machine as I installed Rational Requirements Composer before. Some steps from installation tips may be needed.

The official product documentation does not give the exact DB2 command line command. If you use the command from RTC manual, the table creation will fail. This is caused by the pagesize being too small. Use this command to create the database instead:

db2 create database RQM using codeset UTF-8 territory en PAGESIZE 16384

RRC on 64bit Ubuntu

February 6th, 2010

Another post, where the main audience is me :)

Installation

To install IBM Rational Requirements Composer 2.0.0.1 on 64bit Linux, you need 32bit libraries installed. Strange error message is displayed otherwise when trying to run the installer:

root@rtc:/opt/rrc-install/RRCS_SETUP/InstallerImage_linux# ./install
bash: ./install: No such file or directory

To install the needed libraries run:

apt-get install ia32-libs

The installation should run successfully after this.

Startup

RRC needs swt-gtk to run. Install it by:

apt-get install libswt-gtk-3.4-jni

In addition, RRC needs a X screen to start. There is an IBM Technote with solution for RHEL and Suse. To install Xvfb on Ubuntu do:

apt-get install xvfb

The rest of the procedure (starting the server) is the same as on RHEL or SLES (see the technote).

The server is started by server.startup. This script has to be called from its directory – you need to cd first. The init script can look for example like this:

#!/bin/bash
#
# rrc        Startup script for the RRC
#
# chkconfig: - 90 16
# description: Rational Requirements composer  \
# processname: java
# pidfile: /var/run/rrc.pid
#

prog=rrc
pidfile=${PIDFILE-/var/run/rrc.pid}
lockfile=${LOCKFILE-/var/lock/rrc}
rrchome=/opt/rrc/server
RETVAL=0

start() {
        echo -n $"Starting $prog: "
        # export GTK_PATH=/usr/lib32/gtk-2.0
        Xvfb :1 -screen 0 800x600x24&
        cd ${rrchome}
        DISPLAY=localhost:1.0 ./server.startup
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch ${lockfile}
        return $RETVAL
}

stop() {
        echo -n $"Stopping $prog: "
        ${rrchome}/server.shutdown
        RETVAL=$?
        killall Xvfb
        echo
        [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}

# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        status $prog
        RETVAL=$?
        ;;
  restart)
        stop
        start
        ;;
  *)
        echo $"Usage: $prog {start|stop|restart}"
        RETVAL=3
esac

exit $RETVAL

OS details: Ubuntu 9.04, Server edition, 64bit

IBM Installation Manager RHEL 4 64bit

January 8th, 2010

I'm currently installing Jazz CALM solution on CentOS 4 64bit (equivalent to RHEL 4 64bit). I got stuck when installing both Rational Quality Manager (RQM) and Rational Requirements Composer (RRC). Both need to be installed via IBM Installation Manager. RRC installation package includes IBM Installation Manager, whereas RQM doesn't.

To install IBM Installation manager on 64bit RHEL 4 you first need to install these:

yum install gtk2.i386 libgcc.i386

I first installed libgcc.i386 only to get rid of the error message

libgcc_s.so.1 must be installed for pthread_cancel to work

After this, there was no error message, but the installer returned exit code 13 (which I haven't found documented anywhere). Installing 32bit gtk library solved the problem.

The IBM Installation Manager itself can be downloaded from ftp://public.dhe.ibm.com/…alsdp/v7/im/

I hope this helps somebody save some time of desperate googling for solution…

RTC 2.0 on Ubuntu

October 31st, 2009

Environment

  • RTC Server version: 2.0.0.1
  • OS: Ubuntu 9.04, 64bit. Server edition
  • HW: VMWare guest, 64bit, 1GB RAM

Problem

Installation of Rational Team Concert is in general as easy as unzipping the downloaded archive. I did this recently on Ubuntu 9.04 64bit, server edition. Later I have encountered errors when generating any report (for example Floating License Report). I got this message:

CREATE_EXTENSION_FAIL ( 1 time(s) )
 detail : org.eclipse.birt.report.engine.api.EngineException: CREATE_EXTENSION_FAIL
 at org.eclipse.birt.report.engine.executor.ExecutionContext.addException(ExecutionContext.java:1121)
 ....

The post at jazz.net forums suggested, that there are missing libraries required by file */jazz/server/jre/bin/­libfontmanager­.so You can find out which libraries are missing by running

ldd jazz/server/jre/bin/­libfontmanager­.so

My output was:

linux-vdso.so.1 =>  (0x00007fff5efad000)
libdl.so.2 => /lib/libdl.so.2 (0x00007fc04eed9000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007fc04ecbd000)
libwrappers.so => not found
libawt.so => not found
libstdc++.so.5 => not found
libm.so.6 => /lib/libm.so.6 (0x00007fc04ea36000)
libX11.so.6 => /usr/lib/libX11.so.6 (0x00007fc04e72f000)
libc.so.6 => /lib/libc.so.6 (0x00007fc04e3bd000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc04f26f000)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00007fc04e1a0000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x00007fc04df9d000)
libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x00007fc04dd98000)

Solution

The system library that was missing in my system (using default installation of Ubuntu 9.04 64bit, Server edition) was libstdc++.so.5. This library is included in the package libstdc++5. It can be installed by running:

apt-get install libstdc++5

The RTC server has to be restarted afterwards.

jazz/server/ser­ver.shutdown jazz/server/ser­ver.startup

Ubuntu: change default encryption algorithm

July 12th, 2009

While fighting with DB2 denying logins I had to change the default hash algorithm used to store passwords in /etc/shadow.

As far as I know (and as grep -i -r "sha512" /etc/* tells me :-) ), there are two files that carry this information. Namely /etc/pam.d/common-password and /etc/login.defs.

To change the algorithm from sha512 (which shifts as the default in the new Ubuntu releases) change this:

/etc/pam.d/common-password

password [success=1 default=ignore] pam_unix.so obscure sha512

to

password [success=1 default=ignore] pam_unix.so obscure md5

/etc/login.defs

ENCRYPT_METHOD SHA512

to

ENCRYPT_METHOD MD5

HOWTO: add disk to VMware Ubuntu guest without reboot using LVM

July 10th, 2009

This howto is mostly built on the great tutorial available at http://www.comnetslash.com/…hout-reboot/. My spot tries to be more Ubuntu and LVM specific. This manual was tested on Ubuntu 9.04 (jaunty) Server edition guest OS.

Add Virtual Hard Disk

add-disk1 A new hard disk can be added easily to a running virtual machine in VMWare Infrastructure Web Access by clicking Add Hardware and following the wizard.

Note: I assume that all of the commands given are run under superuser (root) rights. To become root type

sudo su root

Detect new SCSI Disk

The package scsitools provides the script rescan-scsi-bus.sh, which does exacly what we need. Install it and run it:

apt-get install scsitools
rescan-scsi-bus.sh

This should detect the newly added hard disk and make it available in /dev/ – /dev/sdb in the case you are adding a new drive for the first time. The output of fdisk -l should include the newly detected disk.

Format the new disk

Format the newly detected drive using fdisk

fdisk /dev/sdb
    n (for new partition)
    p (for primary partition)
    1 (partition number)
    (keep the other values default)
    w (write changes)

Change the type of the partition to LVM Linux:

fdisk /dev/sdb
    t (change the partition type)
    8e (for Linux LVM)
    w (write changes)

LVM

Initialize LVM Physical Volume

pvcreate /dev/sdb1

Add Physical Volume to Volume Group

List the Volume groups available at your machine.

vgdisplay

Replace VolGroupName with what you read from the output of the last command.

vgextend VolGroupName /dev/sdb1

You should see the newly aquired free space in the output of

vgdisplay

Resize Logical Volumes

After adding some free space to your Volume Group, you can distribute it to the Logical Volumes. The list of Logical Volumes can be obtained by running

lvdisplay

Run the following commands for each Logical Volume you want to resize (add space to). Again – replace /dev/VolGroupNa­me/Name with the Logical Volume name you can read from the output of the last command. The +10G means Add 10 GB

lvextend -L +10G /dev/VolGroupName/Name

The last step is to resize the ext3 filesystem (use different tool is you use different filesystem):

resize2fs /dev/VolGroupName/Name

Shibboleth Ubuntu

June 23rd, 2009

SP2 Installation

I switched one server from Fedora Core to Ubuntu yesterday. Shibboleth 2 SP installation is a bit more tricky under Ubuntu than Fedora Core. The packages are not available in the Ubuntu repositories, but Satya's blog serves a working howto.

failed to bind to socket

Shibboleth seemed to be working well after installation, but today it was down dumping

2009–06–23 12:51:53 ERROR Shibboleth.Listener : socket call resulted in error (2): no message 2009–06–23 12:51:53 CRIT Shibboleth.Listener : failed to bind to socket.

into the /var/log/shibboleth/shibd.log. I luckily found a solution in quite a short time. The problem was that the shibd was trying to create socket in /var/run/shibboleth directory, which did not exist. Creating it once does not help because Ubuntu removes content of /var/run/ at every reboot. To fix it this has to be added to /etc/init.d/shibd

if [ ! -e /var/run/shibboleth ]
then
   mkdir /var/run/shibboleth
fi

DB2: SQL30082N Security processing failed with reason “15″

May 11th, 2009

If you are getting

SQL30082N Security processing failed with reason „15“ („PROCESSING FAILURE“)

when connecting to the db2 database using db2 CONNECT TO dbname USER username or

Connection authorization failure occurred. Reason: Local security service non-retryable error. ERRORCODE=-4214, SQLSTATE=28000

when connecting remotely (via JDBC driver) read on. I finally found what causes this error and a solution!

[ad]

Environment

Problem

Tho whole problem is in the format of the passwords in /etc/shadow. DB2 doesn't seem to like the passwords generated when changing password using the standard passwd command. In Fedora Core 10 the password is hashed using sha-512 and the entry for user username looks like:

username:$6$ef­WWOYRY$z5DNL1kLQ­U4AmPkFBqbQh6LOh­0Qjxq654dS9jE46iP­Np8Zces8I4bP8GLZ3G3RWLo/­6o.LYOV5neYSKxXbL­.M1:14375:0:99999:7:::

Solution

DB2 works fine with passwords hashed with standard crypt function. Password in the desired format can be obtained by calling openssl passwd desiredPassword. The output of openssl can be passed to usermod --password The complete command to change user's password then look like:

usermod --password `openssl passwd desiredPassword` usename

Bigger image

I tried to hand-edit /etc/shadow and insert salted MD5 password, which can be obtained by openssl passwd -1 desiredPassword and the connection was sucessfuly established. To sum it up, the whole problem in the end seems to be that DB2 doesn't like SHA-512 hashes in /etc/shadow. On the other hand, it works fine with hashes generated by crypt and MD5.

Changing default algorithm in Fedora Core

Fedora Core contains a nice tool authconfig. To change the default hashing algorithm to MD5 run

authconfig –passalgo md5 –update

All the passwords inserted in the password database will be stored in MD5 from now on. It will, of course, not change the hashes of the current passwords.

Ubuntu

Ubuntu ships with sha512 as default hash algorithm for passwords in /etc/shadow. The easiest way to change the default algorithm is to edit the file /etc/pam.d/common-password and change the line that reads

password [success=1 default=ignore] pam_unix.so obscure sha512

to

password [success=1 default=ignore] pam_unix.so obscure md5

The information about the algorith is also included in the file /etc/login.defs, which is used by chpasswd for example. Therefore change the line

ENCRYPT_METHOD SHA512

to

ENCRYPT_METHOD MD5