Electricmonk

Ferry Boender

Programmer, DevOpper, Open Source enthusiast.

Blog

Ansible-cmdb v1.20: Generate a host overview of Ansible facts.

Monday, February 13th, 2017

I've just released ansible-cmdb v1.20. Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page containing system configuration information. It supports multiple templates (fancy html, txt, markdown, json and sql) and extending information gathered by Ansible with custom data.

This release includes the following changes:

  • Support for installation in a VirtualEnv.
  • New columns in the html_fancy template for the number of network interfaces and physical disk sizes.
  • Improved error reporting.
  • Added a "Reset settings" button to the html_fancy template that resets localStorage
  • The user that generated the overview and the host it was generated on are now shown in the html_fancy output.
  • Various minor cosmetic changes to the html_fancy template.
  • Work properly with disk sizes of 0 bytes.
  • Host detail information in the html_fancy template can now be collapsed. Use the -p collapse=1 flag to automatically collapse all sections..

As always, packages are available for Debian, Ubuntu, Redhat, Centos and other systems. Get the new release from the Github releases page.

Here's a screenshot of the collapsing host details:

ScriptForm v1.1: Webserver that automatically generates forms to serve as frontends to scripts

Monday, February 6th, 2017

ScriptForm is a stand-alone webserver that automatically generates forms from JSON to serve as frontends to scripts. It takes a JSON file which contains form definitions, constructs web forms from this JSON and serves these to users over HTTP. The user can select a form and fill it out. When the user submits the form, it is validated and the associated script is called. Data entered in the form is passed to the script through the environment.

Some links for further information:

Github repository / sourcehttps://github.com/fboender/scriptform
Download releaseshttps://github.com/fboender/scriptform/releases, packages are available for Debian, Redhat and other Linux-bases systems.
Screenshotshttps://github.com/fboender/scriptform/tree/master/doc/screenshots
Tutorialhttps://github.com/fboender/scriptform/blob/master/doc/MANUAL.md#tutorial
Manualhttps://github.com/fboender/scriptform/blob/master/doc/MANUAL.md

This release (v1.1) features the following changes:

  • Improved form configuration loading error messages.
  • Passwords containing a ':' caused problems.
  • When incorrect redirects were done, or if the user specified an URL starting with two slashes, Scriptform would show a 500 error.
  • When stopping or restarting Scriptform, a harmless error about missing arguments to _cleanup() would be shown in the log file.
  • Various updates to the documentation and examples.

Ansible-cmdb v1.19: Generate a host overview of Ansible facts.

Monday, January 9th, 2017

I've just released ansible-cmdb v1.19. Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page containing system configuration information. It supports multiple templates (fancy html, txt, markdown, json and sql) and extending information gathered by Ansible with custom data.

This release includes the following bugfixes;:

  • Always show stack trace on error and include class name.
  • Exit with proper exit codes.
  • Exclude certain file extensions from consideration as inventories.
  • Improved error reporting and lookups of templates.
  • Improved error reporting when specifying inventories.

As always, packages are available for Debian, Ubuntu, Redhat, Centos and other systems. Get the new release from the Github releases page.

Bexec: Execute script in buffer and display output in buffer. Version 0.10 released.

Monday, January 9th, 2017

After almost a year of no releases, I've made a new release of Bexec today. It's a minor feature release that brings a new setting to Bexec: bexec_splitsize. This settings controls the default size of the output window. You can set it in your .vimrc as follows:

let g:bexec_splitsize=20

This will always make the output window 20 lines high.

It's been almost exactly ten years since the first version of Bexec was released. Version 0.1 was uploaded to vim.org on Januari 30th of 2007. That makes about one release per year on average for Bexec ;-) Perhaps it's time for v1.0 after all this time…

Multi-repo Git status checking script

Tuesday, December 6th, 2016

I've got a whole bunch of Git repositories in my ~/Projects/ directory. All of those may have unstaged, uncommitted or unpushed changes. I find this hard to keep track of properly, so I wrote a script to do this for me. The output looks like this:

mgitstatus

As you can see, it shows:

  • Untracked files: File that are new, are unknown to git and have not been ignored.
  • Uncommitted changes: Files that are known to git and have changes which are not committed.
  • Needs push: New local commits which have not been pushed to the remove origin. 

The script scans for .git directories under the given path. It will only scan a certain level deep. The default for this is 2, which means "all directories directly under this directory". A value of '3' would scan two directories deep.

Full usage:

Usage: git-status.sh <DIR> [DEPTH=2]
​Scan for .git dirs under DIR (up to DEPTH dirs deep) and show git status

Get it from the Github project page.

USB 3 interferes with wireless radio-controlled keyboards and mice

Tuesday, December 6th, 2016

I bought an Intel NUC Kit NUC5CPYH as a replacement for my Raspberry Pi 2b media center and file server. One of the main advantages of the NUC over the Raspberry is that it has USB v3 ports, which greatly increases the throughput speeds to the external USB disks and doesn't require me to have external power on some of them.

I use a radio-controlled keyboard as my remote control my media center. When I plugged it into the NUC, I suddenly found the range of the keyboard greatly reduced. Even when I placed it in exactly the same place as the Raspberry Pi, the range was still horrible further than about 1.5 meters.

Turns out USB 3.x ports interfere with 2.4 GHz devices, which the radio-controlled wireless keyboard operates on.

The solution was to plug in a USB extension cord and putting the RC receiver in that and placing it about 30 cm from the NUC.

On a side-note: Disable UEFI boot if you can't seem to install any Linux distro on your NUC.

Very fast MySQL slave setup with zero downtime using rsync

Sunday, November 6th, 2016

Most online tutorials for setting up a slave replication database involve dumping and loading the database on the master with mysqldump. Unfortunately, loading data with mysqldump can be very slow. My friend Cris suggest a much faster method using rsync.serveimage

Benefits

The benefits of this method are:

  • Very fast setup of a slave by avoiding having to load a logical dump into MySQL.
  • Virtually no downtime for the master. There is a small window (seconds, usually) during which writes are temporary blocked.
  • If the slave was already set up and has become corrupt for whatever reason, rsync will ensure we won't have to copy all the data again that's already on the slave.

In this article I'm going to assume you've already correctly configured the master and slave for replication. If not, please see this guide on how to configure MySQL. You can skip the step where they perform a mysqldump, since that's what this article is replacing with rsync in the first place.

How it works

 Roughly speaking, we'll be doing the following steps:

  1. Stop the slave
  2. Rsync the binary files from the master to the slave while the master is running.
  3. Set a write lock on the master, record the master log position and do another rsync from the master to the slave.
  4. Unlock the master write lock.
  5. On the slave, we change the master log position and start replication again.

The benefits are obvious. Since we first do a sync to the slave without a write lock, the master can keep receiving writes. However, this leaves the slave in a potentially corrupt state, since we might have copied the master data in the middle of a transaction. That's why, after the initial bulk sync, we set a write lock on the master and perform another sync. This only needs to synchronize the new data since the last sync, so it should be fast. The master will only need to be write-locked for a short amount of time. Meanwhile, we record the master log position for the slave. 

Do note that this will only work if your master and slave database are the architecture and run the same MySQL version. If not, this will end in disaster.

Walkthrough

Let's see a practical example:

On the SLAVE, stop MySQL:

root@slave# sudo /etc/init.d/mysql stop

Configure the SLAVE not to automatically start replication on startup. We need this so we can modify the master log pos after starting the slave. Add the 'skip-slave-start' setting to the [mysqld] section:

root@slave# vi /etc/mysql/my.cnf

[mysqld]
skip-slave-start

Now we rsync the MASTER binary data to the SLAVE. For this you'll need to have root ssh access enabled. Depending on your setup, there are a few files you'll have to exclude. For example, if you don't want to overwrite the users on your slave, you'll have to exclude the 'mysql' directory. You should experiment a bit with what you should and shouldn't exclude.

root@master# rsync -Sa --progress --delete --exclude=mastername* --exclude=master.info --exclude=relay-log.info /var/lib/mysql root@192.168.57.3:/var/lib

We've now synchronized the bulk of the master data to the slave. Now we need another sync to put the slave in a valid state. We do this by locking the MASTER to prevent write actions:

root@master# mysql
mysql> flush tables with read lock;
mysql> show master status;
+------------------+----------+--------------+---------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB                            |
+------------------+----------+--------------+---------------------------------------------+
| mysql-bin.000001 |    16242 |              | mysql information_schema performance_schema |
+------------------+----------+--------------+---------------------------------------------+

Record the value of the Position field, since we'll need it later on the slave. The master is now locked against writes.

Keep this MySQL session open and run the next rsync in a different terminal:

root@master# rsync -Sa --progress --exclude=mastername* --exclude=master.info --exclude=relay-log.info /var/lib/mysql root@192.168.57.3:/var/lib

As soon as the final rsync is done, we can unlock the master. Go back to the terminal containing your MySQL session and unlock the tables:

mysql> unlock tables;

The master can now receive writes again. Time to start the SLAVE:

root@slave# /etc/init.d/mysql start

On the slave, we'll change the master log position and start replication:

root@slave# mysql
mysql> change master to master_log_pos=16242;
mysql> start slave;

Verify that the slave is working properly:

mysql> show slave status\G

That's it, your slave is now up and running!

Disclaimer

And now for the big disclaimer:

Copying binary files is always more risky than doing a logical dump and load. By necessity, there are differences between a master and a slave and how the handle the binary files. While in theory the above method should work, I have to caution against using this method if your slave is going to be vital to your setup. If your slave has been set up with the method above, be extra careful when relying on that slave for making logical backups and such.

Ansible-cmdb v1.17: Generate a host overview of Ansible facts.

Saturday, October 8th, 2016

I've just released ansible-cmdb v1.17. Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page containing system configuration information. It supports multiple templates (fancy html, txt, markdown, json and sql) and extending information gathered by Ansible with custom data.

This release includes the following bugfixes;:

  • A bug that caused whitespace to be added each time a character was typed into the search box was fixed.
  • Fix for HP-UX missing netmask (patch by Randy Rasmussen)
  • SQL template improvements for BSD.

As always, packages are available for Debian, Ubuntu, Redhat, Centos and other systems. Get the new release from the Github releases page.

Disable "New release available" emails on Ubuntu

Monday, August 15th, 2016

We have our Ubuntu machines set up to mail us the output of cron jobs like so:

$ cat /etc/crontab 
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=admin@example.com

# m h dom mon dow user    command

This is encrible useful, since cronjobs should never output anything unless something is wrong.

Unfortunately, this means we also get emails like this:

/etc/cron.weekly/update-notifier-common:
New release '16.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

You can fairly easily disable these by modifying the corresponding cronjob /etc/cron.weekly/update-notifier-common:

 #!/bin/sh
 
 set -e
 
 [ -x /usr/lib/ubuntu-release-upgrader/release-upgrade-motd ] || exit 0
 
 # Check to see whether there is a new version of Ubuntu available
-/usr/lib/ubuntu-release-upgrader/release-upgrade-motd
+/usr/lib/ubuntu-release-upgrader/release-upgrade-motd > /dev/null

Now you'll no longer receive these emails. It's also possible to remove the cronjob entirely, but then an upgrade is likely to put it back, and I have no idea if the cronjob has any other side effects besides emailing.

Update: Raf Czlonka emailed me with the following addendum:

I wanted to let you know that there's still one place which makes you see the annoying prompt:

/etc/update-motd.d/91-release-upgrade

an easier way to get rid if the "issue" is simply to run this:

# sed -i 's/^Prompt.*/Prompt=never/' /etc/update-manager/release-upgrades

Thanks Raf! 

Ansible-cmdb v1.15: Generate a host overview of Ansible facts.

Thursday, July 28th, 2016

I've just released ansible-cmdb v1.15. Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page containing system configuration information. It supports multiple templates and extending information gathered by Ansible with custom data.

This release includes the following bugfixes and feature improvements:

  • Improvements to the resilience against wrong, unsupported and missing data.
  • SQL template. Generates SQL for use with SQLite and MySQL.
  • Minor bugfixes.

As always, packages are available for Debian, Ubuntu, Redhat, Centos and other systems. Get the new release from the Github releases page.

The text of all posts on this blog, unless specificly mentioned otherwise, are licensed under this license.