Robs Blog

Blog containing all sorts of info and help from my day to day life

Categories

Mosquitto 1.0 python changes

Looks like there have been some changes to the Mosquitto v1.0.1 python libraries. So far only hit the one but it was quite a major one. The connect method has been changed. If you try and run your code you will get

Traceback (most recent call last):
File "test.py", line 13, in <module>
mqttc.connect(broker, port, 60, True)
TypeError: connect() takes at most 4 arguments (5 given)

In 1.0 the connect function changed from:
def connect(self, hostname="localhost", port=1883, keepalive=60, clean_session=True):
to:
def connect(self, host, port=1883, keepalive=60):

So all it needed was to remove the true flag from the connect statement and then the script worked.

August 17th, 2012 by robconvery
Posted in linux | 1 Comment »

Installing Mosquitto 1.0.1 in Raspbian on Raspberry Pi

Mosquito have just released their 1.0 release..well actually its already 1.01 ūüôā

Currently the Raspbian repositories don’t seem to have it but its nice and easy to get working

wget http://mosquitto.org/files/source/mosquitto-1.0.1.tar.gz
tar -zxvf mosquitto-1.0.1.tar.gz
cd mosquitto-1.0.1
make
sudo make install
sudo ldconfig

Jobs a goodun. It can be checked by running “moqsuitto -v”. If you are already running a previous version you will want to restart the service using “service mosquitto restart”

August 16th, 2012 by robconvery

Using node.js WebSockets to get live MQTT updates

To keep some of my pages up to date with the live sensor data I am using websockets to get MQTT publications.

On the server there are two parts. A Python script which is monitoring a serial port for data and then publishing it accordingly. Information about this can be found at  XRF Temperature Sensor -> XRF Slice of Pi

The second part on the server is a node.js application which is runs a websocket server. For further information about installing Node.js take a look at Install Node.js v0.8.2 on Raspbian The server application creates an mqtt subscription for every connection, it subscribes to the topic according to the url that the client has come in on i.e. xxxx.xxx.xxx.xx/MyTopic will cause a subscription on MyTopic.

Node.js server code

#!/usr/bin/env node
var util   = require('util');
sys = require("sys");
url = require("url");
var WebSocketServer = require('websocket').server;
var http = require('http');
spawn = require('child_process').spawn;

var args = { /* defaults */
    port: '8000',
    debug: 'true'
};
/* Parse command line options */
var pattern = /^--(.*?)(?:=(.*))?$/;
process.argv.forEach(function(value) {
    var match = pattern.exec(value);
    if (match) {
        args[match[1]] = match[2] ? match[2] : true;
    }
});
var port = parseInt(args.port, 10);
var debug = (args.debug == 'true');

var server = http.createServer(function(request, response) {
    console.log((new Date()) + ' Received request for ' + request.url);
    response.writeHead(404);
    response.end();
});
server.listen(port, function() {
    console.log((new Date()) + ' Server is listening on port ' + port);
});

wsServer = new WebSocketServer({
    httpServer: server,
    // You should not use autoAcceptConnections for production
    // applications, as it defeats all standard cross-origin protection
    // facilities built into the protocol and the browser.  You should
    // *always* verify the connection's origin and decide whether or not
    // to accept it.
    autoAcceptConnections: false
});

function originIsAllowed(origin) {
  // put logic here to detect whether the specified origin is allowed.
  return true;
}

wsServer.on('request', function(request) {
    if (!originIsAllowed(request.origin)) {
      // Make sure we only accept requests from an allowed origin
      request.reject();
      console.log((new Date()) + ' Connection from origin ' + request.origin + ' rejected.');
      return;
    }
    var myProcs=new Array();

    var connection = request.accept('currentcostdata', request.origin);
    console.log((new Date()) + " connection accepted from " + connection.remoteAddress + " via " + request.origin);
    var topicString = request.resource.substring(1);
    console.log((new Date()) + ' Connection Topic: ' + topicString);
    var mosquittopid = request.key;

	console.log((new Date()) + ' Connection ID: ' + mosquittopid);
	mosq = spawn('mosquitto_sub',['-t',topicString]);
	mosq.stdout.setEncoding('utf8')
	console.log((new Date()) + 'PID of mosquitto_sub for ' + topicString + ' ConID: ' + mosquittopid + ' is ' +mosq.pid);
	myProcs[mosquittopid] = mosq.pid;

	mosq.stdout.on('data', function (data) {
		connection.sendUTF(data)
		data=data.replace("\r","").replace("\n","");
		data=data.replace("\r\n","");
		console.log((new Date()) + 'Topic: ' + topicString + ' - ' + data);
	});

    connection.on('message', function(message) {
        if (message.type === 'utf8') {
            console.log('Received Message: ' + message.utf8Data);
            connection.sendUTF(message.utf8Data);
        }
    });

    connection.on('close', function(reasonCode, description) {
		//console.log((new Date()) + ' Peer ' + connection.remoteAddress + ' disconnected.');
		console.log((new Date()) + ' ' + mosquittopid + ' disconnected');
		console.log((new Date()) + ' ' + topicString + ' Closed: pid: ' + myProcs[mosquittopid] + '.');
		process.kill(myProcs[mosquittopid]);
		delete myProcs[mosquittopid];
    });
});

On the client side there is some JavaScript which opens the websockets connection to the server and then listens for updates and acts accoridingly

<script type="text/javascript">// <![CDATA[
		$(document).ready(function() {
			/* Define handler for websocket debug information*/
			function debug(str) {
				$("#debug_txt").html("

"+str+"

");
			};
			function rtrim(str) {
				return str.replace(/\s+$/,"");
			}
			/* Create a websocket connection */
			ws = new WebSocket("ws://myip:8000/CurrentCostTemp","currentcostdata");
			/* Define websocket handlers */
			ws.onmessage = function(evt) {
				var data = rtrim(evt.data);
				var temptitle = data +'\u00B0C';
				$("#TempValue").text(temptitle);
				debug('Temp - ' + temptitle);
			};
			ws.onclose = function() {
				debug("socket closed");
			};
			ws.onopen = function() {
				debug("connected...");
			};
		});

// ]]></script>

August 8th, 2012 by robconvery

Installing MQ / libstdc++.so.5 pre-req on RHEL6

One of the pre-reqs that if needed by MQ7 (or more specifically gsk7bas) is that it needs – libstdc++.so.5.

If you try and find this package its often not available. To install it run the following commands

yum groupinstall “Compatibility libraries”
yum install libstdc++.so.5

If you want to install MQ 7.0.1.5 you will also need

yum install libstdc++.so.6

December 21st, 2011 by robconvery
Posted in linux | No Comments »

MQ on RHEL6 FDC – MSGAMQ6119 Resource temporarily unavailable’ from pthread_create

After configuring a RHEL6 box with MQ 7.0.1.5 we hit a problem where by when we had a large number of connections (118) it would then fail to allow any more connections. You could also not perform any operations on the box. If you attempted to make a change you got the following error

  • fork: retry: Resource temporarily unavailable

After lots of searching this was fixed by doing the following

Disabling SELinux

  • setenforce Permissive
  • reboot

Changing the users file limitations. Go to /etc/security/limits.conf, change this user’s limitation:

username         hard    nproc         unlimited
username         soft    nproc           unlimited

After making these changes we were able to make many more client connections

August 17th, 2011 by robconvery
Posted in linux | No Comments »

Apply MQ Service on Ubuntu – 7.0.1.0 to 7.0.1.5

The first thing to do before upgrading MQ is to stop all queue managers. If you are running any message brokers then stop these as well.

i.e. endmqm CSIM

Use the command “dspmq” to check that all local queue managers are stopped

Download the fixback from the IBM service web site – i.e. for 64-bit Ubuntu 7.0.1-WS-MQ-LinuxX64-FP0005.tar.gz

First we must remove the old versions of GSK as these clash with the new ones

rpm -e –noscripts –nodeps gsk7bas-7.0-4.23.i386
rpm -e –noscripts –nodeps

We can now install the latest version of GSK and the MQ updates

rpm -ivh –nodeps –force-debian gsk7*
rpm -ivh –nodeps –force-debian MQSeries*

Check that the install has gone OK by running “dspmqver”. The output should look like

Name:        WebSphere MQ
Version:     7.0.1.5
CMVC level:  p701-105-110419
BuildType:   IKAP Р(Production)

You can now start up your queue managers using strmqm as per usual and they should be running at 7.0.1.5

July 12th, 2011 by robconvery

Ubuntu 11.04 Upgrade…won’t boot – GRUB prompt

Aftert upgrading to Ubuntu 11.04 my server would not boot. It looks like there is a bug in the update such that if you upgrade from Ubuntu 10.10 which itself had been upgraded from Ubuntu 10.04, and grub was never reinstalled then you just get a GRUB prompt after bootup.

To fix this you need to manually boot the system using grub commands. Then reinstall grub. For me the following worked

set root=(hd0,1)
linux /boot/vmlinux-2.6.38-8-generic root=/dev/sda1 ro
initrd /boot/initrd.img-2.6.38-8-generic
boot

Once booted then run

sudo grub-install /dev/sda
sudo update-grub

And then it should all reboot nicely

June 20th, 2011 by robconvery

Install MQ 7.0.1.0 onto Ubuntu 10.4 (or greater)

Ubuntu is not officially supported on MQ 7.0.1.0 but it runs fine except you need a few tweeks to get it installed. As ubuntu normally uses Apt-Get to install packages we need to install the rpm packages. This lets you use the rpm command like you would on other linux OSs

sudo apt-get install rpm

Download the install file and extract a temp directory

tar -zxvf CZ4VEML.tar.gz -C /InstallFiles/MQ7010

Accept the license agreement using the normal method

sudo ./mqlicense.sh

Unless you need all the languange messages these can be deleted

sudo rm /InstallFiles/MQ7010/MQSeriesMsg_*

You can now install the other files using rpm

sudo rpm -ivh –nodeps –force-debian MQSeriesClient-7.0.1-0.x86_64.rm
sudo rpm -ivh –nodeps –force-debian MQSeriesJava-7.0.1-0.x86_64.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesKeyMan-7.0.1-0.x86_64.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesMan-7.0.1-0.x86_64.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesRuntime-7.0.1-0.x86_64.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesSamples-7.0.1-0x86_64i386.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesServer-7.0.1-0.x86_64.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesSDK-7.0.1-0.x86_64.rpm
sudo rpm -ivh –nodeps –force-debian gsk7bas-7.0-4.23.x86_64.rpm

You can now start using the MQ server like you normally would. If you would like to install the GUI then this can be done as follows

sudo rpm -ivh –nodeps –force-debian MQSeriesJRE-7.0.1-0.i386.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesEclipseSDK33-7.0.1-0.i386.rpm
sudo rpm -ivh –nodeps –force-debian MQSeriesConfig-7.0.1-0.i386.rpm

Some of the eclipse plugins don’t get copied into the correct folder so this is a manual fix

sudo cp -r /opt/mqm/eclipse/* /opt/mqm/eclipseSDK33/eclipse
sudo chown -R mqm:mqm /opt/mqm

To start MQ GUI (eclipse) run the following command

strmqcfg

 

If you need to uninstall MQ Uninstall MQSeries then use the following commands

sudo rpm -qa | grep MQSeries | xargs sudo rpm -e –noscripts
sudo rpm -qa | grep gsk7bas | xargs sudo rpm -e –noscripts

 

June 10th, 2011 by robconvery
Posted in linux | No Comments »

Creating a VM from a Linux System

Create a VM with similar characteristics i.e. Same OS & HDD setup as real machine.

Real Machine

  • IBM HS20 Blade
  • 2x Xeon Processors
  • Filesystem Size Used Avail Use% Mounted on
    /dev/sda5 48G 3.5G 43G 8% /
    /dev/sda1 99M 53M 41M 57% /boot
    /dev/sdb5 49G 4.5G 42G 10% /db2data
    none 2.0G 0 2.0G 0% /dev/shm
    /dev/sdb2 7.7G 1.5G 5.9G 21% /home

For this example I have created a VM system as follows

  • RHEL4-U4 VM
  • 100MB /dev/sda1 (/boot)
  • 10 GB /dev/sda2 (/)
  • 2GB /dev/sda3 (sawp)
  • 25 GB /dev/ sdb1 (/db2data)

Boot VM using Knoppix

Fix Network – Menu -> KNOPPIX -> Network/Internet -> Network Card Configuration

Enable SSH

  • cd /etc/ssh
  • sudo ssh-keygen -t rsa -f ssh_host_rsa_key -C ‘ ‘ -N ‘ ‘
  • sudo chmod 600 ssh_host_rsa_key
  • sudo chmod 644 ssh_host_rsa_key.pub
  • sudo ssh-keygen -t dsa -f ssh_host_dsa_key -C ‘ ‘ -N ‘ ‘
  • sudo chmod 600 ssh_host_dsa_key
  • sudo chmod 644 ssh_host_dsa_key.pub
  • sudo /etc/init.d/ssh restart

Set the root password on the VM System

  • sudo passwd
  • <somepassword>
  • <somepassword>

Check you can now SSH into the VM from the real Linux system

  • ssh root@x.x.x.x
    The authenticity of host ‘x.x.x.x (x.x.x.x)’ can’t be established.
    RSA key fingerprint is 46:cf:45:0f:40:9d:47:e8:f1:45:97:27:86:f4:3e:04.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added ‘x.x.x.x’ (RSA) to the list of known hosts.
    Password:
    Welcome to Knoppix!

Check the VM discs are mounted with Knoppix. On my system it appears as follows

  • sudo mount /dev/sda1 /mnt/sda1
  • sudo mount /dev/sda2 /mnt/sda2
  • sudo mount /dev/sdb1 /mnt/sdb1

Work out what filesystems you need to copy across.

# df -vh
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 48G 3.4G 43G 8% /
/dev/sda1 99M 66M 29M 70% /boot
/dev/sdb5 49G 18G 29G 38% /db2data
none 2.0G 0 2.0G 0% /dev/shm
/dev/sdb2 7.7G 51M 7.3G 1% /home

Obviously this section depends on how your Linux system is setup.
On my real system /home is on a separate drive but for when I move it to the VM
I am going to make this just part of the system disc as its not really used.

Attempt to Copy across some of the file systems i.e.

  • rsync -ave ssh –delete –exclude=/db2data/* –exclude=/u/* –exclude=/boot/* / root@x.x.x.x:/mnt/sda2/
    • –exclude=/db2data/* is used so it does not copy across the files stored in /db2data as I want them to be stored on a different disk
    • –exclude=/u/* is used as this directory is mounted from a remote machine
    • –exclude=/boot/* is used as this directory is mounted to another partition
  • rsync -ave ssh –delete /boot/ root@x.x.x.x:/mnt/sda1/
  • rsync -ave ssh –delete /home root@x.x.x.x:/mnt/sda2/home
  • rsync -ave ssh –delete /db2data/ root@x.x.x.x:/mnt/sdb1/

The next step is to exit the fstab file to make sure the files are mounted correctly

Next step – reboot the VM and see what happens.

Issues…..

  • Depending on how grub is setup you might need to reinstall grub onto the main disk. If you need to do this follow the instructions below
    1. Boot from a Live CD, like Ubuntu Live, Knoppix, Mepis, or similar.
    2. Open a Terminal. Go SuperUser (that is, type “su”). Enter root passwords as necessary.
      Type “grub” which makes a GRUB prompt appear.
    3. Type “find /boot/grub/stage1”. You’ll get a response like “(hd0)” or in my case “(hd0,0)”. Use whatever your computer spits out for the following lines.
    4. Type “root (hd0,0)”.
    5. Type “setup (hd0,0)”. This is key. Other instructions say to use “(hd0)”, and that’s fine if you want to write GRUB to the MBR. If you want to write it to your linux root partition, then you want the number after the comma, such as “(hd0,3)”.
    6. Type “quit”.
  • If you have changes /boot from being a partition to mounted within / you will find that GRUB cannot find the kernel files. The easiest solution is to reboot the vm using the Knoppix CD and then mount the filesystem with /boot on it i.e. sudo mount /dev/sda2 /mnt/sda2 then edit the /boot/grub.conf file from something like
    • title Red Hat Linux (2.4.18-0.12)
      root (hd0,0)
      kernel /vmlinuz-2.4.18-0.12 ro root=LABEL=/
      initrd /initrd-2.4.18-0.12.img

to

  • title Red Hat Linux (2.4.18-0.12)
    root (hd0,0)
    kernel /boot/vmlinuz-2.4.18-0.12 ro root=/dev/sda1
    initrd /boot/initrd-2.4.18-0.12.img
December 17th, 2007 by robconvery