Current OS = Linux Mint 13

INTRO

The aim of this blog. You know how something challenges you and you google away, find a fix with some 'trial and error' and then in the future someone asks about how you did it, or you need to alter/re-do it at a later date but you have forgotten what little trick you did to accomplish it ? Well my aim is to keep a track of what I am working on and methods I have used here. And now, I can access it easily, it can be google indexed for others and I will have a URL to send others for problems I cant recall off hand how I fixed them. I hope you find this site useful.

26-11-2014 18:34

MySQL multi-master replication with GTID on version 5.6

It turns out that multi-master replication is nothing more than a "criss-cross" master-slave replication setup. To explain that better with an example, server A is the master for server B and server B is the master of server A. This obviously mean any changes on either is replicated to the other. So you simply get master-slave replication working in one way, then mimic that on the other side once satisfied.

The first step in getting this working is to set up the config files. See here:

[mysqld]
# Needed for masters and slaves
server-id       = 3
log_bin         = /var/log/mysql/mysql-bin.log
binlog_format=row
gtid_mode=on
enforce_gtid_consistency=true
log_slave_updates=true

# Needed for the slaves
# Better to filter here than using binlog_do_db on the master etc
# http://www.mysqlperformanceblog.com/2009/05/14/why-mysqls-binlog-do-db-option-is-dangerous/
# more here: http://dev.mysql.com/doc/refman/5.1/en/replication-options-binary-log.html#option_mysqld_binlog-do-db
replicate-wild-ignore-table=mysql.%
replicate-wild-ignore-table=information_schema.%
replicate-wild-ignore-table=performance_schema.%

relay_log         = /var/log/mysql/mysql-relay-bin.log
# http://jonathonhill.net/2011-09-30/mysql-replication-that-hurts-less/
auto-increment-offset = 1
auto-increment-increment = 4
read_only=OFF

I simply add this to /etc/mysql/conf.d/ ensuring the file name ends in .cnf. Restart mysql and that part is done. Next we connect to one of the servers, let's call this one A. Step 1 is to create a dump or a snapshot of your data.

 mysqldump -u root -p --all-databases --flush-privileges --single-transaction --master-data=2 --flush-logs --triggers --routines --events --hex-blob  |bzip2 > $HOSTNAME-$(date +%F).sql.bz2
Next step is to add the replication user on server A. You could use root, but best not to as you will be setting up the user (password and all) on another server.
 GRANT REPLICATION SLAVE ON *.* TO 'rep'@'slave_ip' IDENTIFIED BY 'some_secret';
Now on server B, import the dump from A.
 bzcat $HOSTNAME-$(date +%F).sql.bz | mysql -u root -p
Now we tell server B where the master server is and the user to use (the one we just setup).
 change master to master_host='master_ip", master_port=3306, master_user='rep', master_password='some_secret', master_auto_position=1; 
start slave; 
We should be up and running. Check with the "show slave status\G" command on the slave. Now we simply repeat this but the other way around (skipping the dump and restore) in order to set up multi-master. Setup a replication user on B, then run the "change master to" command on server A so it is now a slave of B. All should be done now. See the references for troubleshooting. I actually use stunnel to connect my MySQL servers over the internet. Also, you can create a ~/.my.cnf file with your login info to save having to pass that to the mysql command every time. Contents would like so:
[client]
user=root
password=some_secret
Ref: http://fromdual.com/gtid_in_action?_ga=1.204815570.317472044.1416546765 http://fromdual.com/replication-troubleshooting-classic-vs-gtid https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication http://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/

Posted by DaveQB | Permanent Link | Categories: IT

03-10-2014 15:55

How the MySQL ALTER command works.

What does an ALTER command do in MySQL? Quoting from this StackOverflow thread:

In most cases, ALTER TABLE works by making a temporary copy of the original table. The alteration is performed on the copy, and then the original table is deleted and the new one is renamed. While ALTER TABLE is executing, the original table is readable by other sessions. Updates and writes to the table are stalled until the new table is ready, and then are automatically redirected to the new table without any failed updates.
This makes a lot of sense. Using the "show processlist;" command, we can see the state an ALTER command is in. Such as "copy to tmp table ALTER TABLE mdl_sessions2 ENGINE=InnoDB". This gives one confident to cancel the operation while nothing has actually been changed. There might be a small window between checking the state of the command to actually cancelling it. But if it is a long running query (which would be the only way it would be humanly possible to cancel it) then the next step is altering the tmp table, so again, no issue if that is cancelled. So cancel away on an ALERT command while it is still copying. If it is in a state of altering the tmp table, but be careful you don't cancel it as it deletes the original table.


Posted by DaveQB | Permanent Link | Categories: IT

27-09-2014 23:26

Flash in Chromium on Ubuntu based OS

sudo apt-add-repository ppa:skunk/pepper-flash
sudo apt-get update
sudo apt-get install pepflashplugin-installer -y
sudo su -c "echo '. /usr/lib/pepflashplugin-installer/pepflashplayer.sh' >> /etc/chromium-browser/default"

This works on any Ubuntu based distro. I ran it on my Mint 13 desktop and it worked after restarted flash.


Posted by DaveQB | Permanent Link | Categories: IT

21-07-2014 12:13

Networking tips and tricks

This is a good article for basics using network manager. A good read for anyone using Network Manager. The one issue is the hosts file editing. It is better to put the FQDN first after the IP address and then any short names after that as the first name after the IP is what the system uses to resolve it's own hostname. So the command hostname -f won't work if you have your systems FQDN second or later on it's 127.0.0.1 line. It is just a good habit to be in.


Posted by DaveQB | Permanent Link | Categories: IT

23-06-2014 15:57

AWS Sydney Region network throughput

I did some iperf testing on the two Availability Zones (AZ) in the Sydney AWS region. I have 4 gluster servers a m1.medium and m3.large in ap-southeast-2a and a m1.medium and m3.large in ap-southeast-2b. So I did a cross AZ iperf test between like servers.

m1.medium 2a > m1.medium 2b

[ ID] Interval Transfer Bandwidth 
[ 4] 0.0-10.0 sec 220 MBytes 184 Mbits/sec 
[ 7] 0.0-10.0 sec 188 MBytes 157 Mbits/sec 
[ 6] 0.0-10.0 sec 193 MBytes 161 Mbits/sec 
[ 8] 0.0-10.1 sec 262 MBytes 218 Mbits/sec 
[SUM] 0.0-10.1 sec 863 MBytes 719 Mbits/sec 

[ 4] 0.0-10.1 sec 219 MBytes 181 Mbits/sec 
[ 6] 0.0-10.2 sec 227 MBytes 187 Mbits/sec 
[ 7] 0.0-10.2 sec 197 MBytes 162 Mbits/sec 
[ 3] 0.0-10.2 sec 221 MBytes 182 Mbits/sec 
[SUM] 0.0-10.2 sec 864 MBytes 712 Mbits/sec 


m3.large 2a > m3.large 2b

[ ID] Interval Transfer Bandwidth 
[ 7] 0.0-10.0 sec 222 MBytes 185 Mbits/sec 
[ 3] 0.0-10.0 sec 66.0 MBytes 55.1 Mbits/sec 
[ 6] 0.0-10.1 sec 96.6 MBytes 80.6 Mbits/sec 
[ 8] 0.0-10.1 sec 56.8 MBytes 47.3 Mbits/sec 
[SUM] 0.0-10.1 sec 441 MBytes 368 Mbits/sec 

[ 7] 0.0-10.1 sec 140 MBytes 116 Mbits/sec 
[ 3] 0.0-10.1 sec 162 MBytes 134 Mbits/sec 
[ 6] 0.0-10.2 sec 79.1 MBytes 65.3 Mbits/sec 
[ 5] 0.0-10.2 sec 57.4 MBytes 47.4 Mbits/sec 
[SUM] 0.0-10.2 sec 438 MBytes 362 Mbits/sec

Not that good, not gigabit speeds, but not terrible.

Now keeping the test inside the same AZ.

m1.medium 2b > m3.large 2b

[ ID] Interval Transfer Bandwidth 
[ 7] 0.0-10.0 sec 105 MBytes 87.6 Mbits/sec 
[ 6] 0.0-10.0 sec 161 MBytes 135 Mbits/sec 
[ 5] 0.0-10.1 sec 93.9 MBytes 78.3 Mbits/sec 
[ 8] 0.0-10.1 sec 86.4 MBytes 71.9 Mbits/sec 
[SUM] 0.0-10.1 sec 446 MBytes 371 Mbits/sec 

[ 5] 0.0-10.2 sec 143 MBytes 118 Mbits/sec 
[ 4] 0.0-10.2 sec 93.5 MBytes 77.0 Mbits/sec 
[ 7] 0.0-10.2 sec 119 MBytes 98.2 Mbits/sec 
[ 6] 0.0-10.2 sec 89.0 MBytes 73.3 Mbits/sec 
[SUM] 0.0-10.2 sec 445 MBytes 366 Mbits/sec 

Not much better.


Posted by DaveQB | Permanent Link | Categories: IT

21-05-2014 13:32

How to generate a HmacSHA512 from the shell

This is ripped right from here but I am copying here for my own (and others) reference.

I realise this isn't exactly what you're asking for, but there's no point in reinventing the wheel and writing a bash version. You can simply use the openssl command to generate the hash within your script.

[me@home] echo -n "value" | openssl dgst -sha1 -hmac "key"
57443a4c052350a44638835d64fd66822f813319
Or simply:
[me@home] echo -n "value" | openssl sha1 -hmac "key"
57443a4c052350a44638835d64fd66822f813319
Remember to use -n with echo or else a line break character is appended to the string and that changes your data and the hash. That command comes from the OpenSSL package which should already be installed (or easily installed) in your choice of Linux/Unix, Cygwin and the likes. Do note that older versions of openssl (such as that shipped with RHEL4) may not provide the -hmac option. As an alternative solution, but mainly to prove that the results are the same, we can also call PHP's hmac_sha1() from the command line:
[me@home]$ echo '' | php
57443a4c052350a44638835d64fd66822f813319
Edit: One could use printf rather than echo -n.


Posted by DaveQB | Permanent Link | Categories: IT

12-03-2014 17:51

AWS-CLI

I have been running all the different cli API tools for EC2 access at work. It is handy and needed for scripting purposes, but so slow, even the command line completion is slow. For example, doing an ec2-describe-instances was taking 1min and 14 seconds, with the new aws-cli tools it takes 1.87 seconds.. Well worth the hassle.


Posted by DaveQB | Permanent Link | Categories: IT

07-03-2014 11:25

Denyhosts ignores some whitelisting

I had issues at work with some IPs that fall into an IP wildcard range that was in my denyhots whitelist being blocked by denyhosts. It turns out they were Class B networks and denyhosts wildcard white listing only accepts Class C networks. Although written incorrectly here you undestand what it is saying. The examples do only show Class C networks, but it doesn't explicitly say it ONLY accepts Class C networks, hence why it was hard for me to troubleshoot this issue.


Posted by DaveQB | Permanent Link | Categories: IT

25-11-2013 18:34

Time calculations in shell

I have had needs where I needed to calculate the time something has taken to run. It can be hard when going over a date in the calendar. To add to that, some shells don't allow floating point calculation too. So I sat down and worked out a scrpt that when given a start second and a finish second (like seconds since epoch), it can show the difference in HOURS:MINUTES. Feel free to modify as you please, GPL3 of course.

#!/bin/bash

# Capture the epoch start and then finish time.
#SLEEPSTOP=1385183548
#SLEEPSTART=1385180248

# Or give it the absolute seconds.
SLEEPSTOP=0
SLEEPSTART=1440

SLEEP=$(($SLEEPSTOP-$SLEEPSTART))
HR=$(($SLEEP/60/60))
MINS=$(echo $(($SLEEP/60*100/60))|rev|cut -c-2|rev)
MINS=$((MINS*60/100))

echo ${HR}:${MINS}


Posted by DaveQB | Permanent Link | Categories: IT

25-10-2013 17:59

Allow root ssh from selected machines

Sometimes you need to allow root ssh access. But this is not a great idea; I disallow root ssh on all my systems, instead finding alternative (often much more difficult) solutions such as running two or more ssh daemons and their config files and control access to them with iptables. Convoluted.

Recently I found out how you can use the Match statement to selectively allow root ssh (and even limit it to PubkeyAuthentication only). It is rather simple and reading the sshd_config man page explains more. Below is a self explaining example. The without-password actually means PubkeyAuthentication only and password auth is disabled. It does look a bit scary if you didn't know that. I did find that only having the Address 127.0.0.1 didn't allow root ssh access to localhost.

Match Address 127.0.0.1,23.4.76.129,8.8.8.8
        PermitRootLogin without-password

Match Host localhost
        PermitRootLogin without-password

Posted by DaveQB | Permanent Link | Categories: IT
HERE!