Using Mattermost as a receiver for Alertmanager

While working to move to Prometheus and Alertmanager for our monitoring lately, my team discovered that the generic webhook_config in Alertmanager does not work with a Mattermost webhook. The JSON included in the POST from Alertmanager causes Mattermost to return a 400, and the lack of flexibility in Alertmanager leaves a lot to be desired. Luckily, Mattermost webhooks translate Slack webhooks natively (according to the documentation here), and Alertmanager natively supports Slack with a slack_config.

receivers:
  - name: mattermost
    slack_configs:
    - api_url: http://{your-mattermost-site}/hooks/xxx-generatedkey-xxx

Then, customize the alert routing and message as much as you want. Further reading on custom templating which I found helpful is available on the Prometheus Blog.

Graphing SNMP data with Prometheus and Grafana

This summer, I’ve been an intern on the Infrastructure team at CoreOS, and this week marks the halfway point. I’ve been learning lots and doing some really interesting work. My latest project revolves around getting some metrics about the office network set up, using SNMP, Prometheus and Grafana.

SNMP

The first step was learning about SNMP. For those of you who haven’t heard of SNMP before, it’s a protocol that came about in the late 80’s, used for getting and modifying information about a managed device. It uses OIDs, which are allocated to different organizations, such as Cisco, who then in turn define their subset of OIDs in a MIB, which SNMP uses to know how to parse the information. Here’s an example of a request to a host with some OID

$ snmpget -v 2c -c public hostname 1.3.6.1.2.1.2.2.1.10.21
IF-MIB::ifInOctets.21 = Counter32: 17202468

Using snmpget (from the net-snmp package), version 2c of SNMP and the community string ‘public’, we request information associated with the OID 1.3.6.1.2.1.2.2.1.10.21. I had to look up this OID, and I’m just using it as an example of what an OID looks like. Usually, the string of the OID is easier for individual requests. But you can see below the command, in the response, what that OID resolved to. IF-MIB::ifInOctets.21 breaks down to the Interface MIB, or IF-MIB, then using that MIB, to the number of octets on some interface, in this case physical port 21 (this particular host is a switch). If you have some host that supports SNMP version 2 or 3, try the following command to see most of the OIDs your host knows about, assuming that the default community string is ‘public’ (which is standard): snmpwalk -v 2c -c public hostname.

Prometheus

With a base of SNMP understanding, now we can start getting some metrics. We’ll need Prometheus and the Prometheus SNMP Exporter. I want to run them in docker, and have them talk to each other. This is simple with something like Kubernetes, but for my proof-of-concept, I set up a docker bridge network, and put both containers on it.

$ docker network create --driver bridge prometheus
$ docker run --network=prometheus -d -p 9116:9116 quay.io/taylor_fahlman/prom_snmp_exporter
$ docker run --network=prometheus -d -p 9090:9090 -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

And here’s the prometheus.yml:

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:
  - job_name: 'snmp'
    static_configs:
      - targets:
        - hostname # Target SNMP device.
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.18.0.3:9116  # Docker IP with SNMP exporter.

I found the correct bridge IP for the SNMP exporter container by running docker network inspect prometheus. Just replace 172.18.0.3 whatever the IP is. Now I could see info about the scrapes Prometheus was making at http://dockerhostname:9090/consoles/snmp.html. It showed the device, which should be the same as the host or IP under targets, if it was up, Ports Up, Ports Total, Mb/s in, out, discard and errors. That’s a lot of metrics just from a default setup! There’s a great tool that goes along with SNMP exporter that generates the config so if you want to make your own module that handles certain Cisco MIBs and Supermicro MIBs, its easy. You can find that tool here.

Now I have metrics about the target host in a time series. What better way to analyze that data then by graphing it?

Grafana

This part is super easy. Spin up a Grafana container on the same bridge:

$ docker run --network=prometheus -d -p 3000:3000 grafana/grafana

Follow the docs to log in and set up a Prometheus data source. Then, import this as a Dashboard. Assuming everything was set up right, there should be a graph for each port on your device, automatically labeled, and probably a few data points already graphed. In my case, it worked out of the gate for most hosts, but I had a few that required SNMP v1 for some reason so I had to tinker. But for SNMP 2 and 3, that template just works at time of writing.

Conclusion

SNMP is actually really useful, even if most people think its old. Using that data, I can now easily tell when certain ports on a switch are spiking in traffic, which ones get more inbound traffic, which get more outbound traffic, and in general make better hypothesis about network issues in the office. Prometheus and Grafana were a breeze to set up, and are incredibly powerful tools that I’ve barely scratched the surface of.

Travis and SFML

I currently work at the Open Source Lab as a Student System Admin, and will soon be interning on the Infrastructure team at CoreOS . So I like automating things, especially testing and building. One of the more annoying things about using a framework like SFML is that there is not a lot of documentation on testing them. Don’t get me wrong, I’m really loving SFML and its community, especially the IRC channel. But in asking if anyone there used anything like Travis to automate their builds, there were only a few that did, and even then, it was all hacked together and not easily replicable.

I want to share my working Travis config for my SFML project so that anyone else out there could have at least a starting point.

language: cpp
dist: trusty
sudo: required
compiler:
  - gcc

before_script:
  - wget https://www.sfml-dev.org/files/SFML-2.3.2-linux-gcc-64-bit.tar.gz -O /tmp/sfml.tar.gz
  - tar -xvf /tmp/sfml.tar.gz
  - sudo cp -r SFML-2.3.2/* /usr/
  - wget http://www.bromeon.ch/libraries/thor/download/v2.0/thor-v2.0-linux64-make.tar.gz -O /tmp/thor.tar.gz
  - tar -xvf /tmp/thor.tar.gz
  - sudo cp -r thor-v2.0-linux64-make/* /usr/
  - sudo apt-get install -y libjpeg62-dev libsndfile1-dev libglew1.5 libglew1.5-dev libfreetype6 libjpeg-turbo8 libjpeg8 libopenal-data libopenal1  libxrandr2 libxrender1 libsoil1 libxcb-image0 libxcb-randr0 libudev1
  - sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
  - sudo apt-get update
  - sudo apt-get install g++-4.9
  - cd src

script: make travis

So lets break this down a little.

laguage: cpp
dist: trusty
sudo: required
compiler:
  - gcc

The language is CPP, assuming that’s what you’re using for SFML. The distro is trusty, because there are a few packages that don’t exist or work properly on 12.04 (default for travis) that we need. Sudo is required due to some commands you’ll see in the next block, and I’m using gcc, (well, really g++). Nothing there too out of the ordinary.

before_script:
  - wget https://www.sfml-dev.org/files/SFML-2.3.2-linux-gcc-64-bit.tar.gz -O /tmp/sfml.tar.gz
  - tar -xvf /tmp/sfml.tar.gz
  - sudo cp -r SFML-2.3.2/* /usr/
  - wget http://www.bromeon.ch/libraries/thor/download/v2.0/thor-v2.0-linux64-make.tar.gz -O /tmp/thor.tar.gz
  - tar -xvf /tmp/thor.tar.gz
  - sudo cp -r thor-v2.0-linux64-make/* /usr/

Ubuntu does have an SFML package, but its too low of a version for me, so I manually install the version I want. I’m also using Thor, a library built for SFML, so I also manually install that. This is the method I recommend for installing the SFML binaries because it ensures exactly what version you’ll get. You can also compile it yourself if you really want to, but I won’t cover that here.

- sudo apt-get install -y libjpeg62-dev libsndfile1-dev libglew1.5 libglew1.5-dev libfreetype6 libjpeg-turbo8 libjpeg8 libopenal-data libopenal1  libxrandr2 libxrender1 libsoil1 libxcb-image0 libxcb-randr0 libudev1
- sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
- sudo apt-get update
- sudo apt-get install g++-4.9

First, we install all the dependencies for SFML and Thor. libudev1 and the xcb packages gave me a lot of trouble, which is why I tried using trusty. So here’s where I banged my head against the wall for a while. I kept getting errors about udev and undefined references on travis that I wasn’t getting locally. Turns out, even on trusty, the default g++ is 4.8, but 4.9 or greater is needed for udev to work (I guess?). Installing 4.9 fixed everything and my project compiled cleanly.

script: make travis

Now, just run your make script or whatever your equivalent is. I made a separate target that specifically uses g++-4.9, but other than that, the target is identical to my default target.

Hopefully, this will help someone out there who wants to automate their SFML build.

Animation

An animation handler is decently hard to write, especially when working with objects instead of directly with sprites, so I opted to use the Thor library. It has some great tools and tutorials. I think I’ll also use the action handling component of Thor in the future, but for now I’m just using the Animation component.

Before this, I manually moved the sprite’s position every time I pressed an arrow key. It looked something like

case sf::Event::KeyPressed:
{
        if(sf::Keyboard::isKeyPressed(sf::Keyboard::Down)){
                if(y < 600)
                        y += 5;
                playerSprite.setTexture(this->textureManager.getRef("down"));
        }

        ...(add other directions here)

}

playerSprite.setPosition(x, y);

As you can see, there is a lot that can go wrong here, and animation without some sort of counter and change to how I handled textures was going to be hard and tedious. But Thor’s animation makes this much easier. First you load the animations

Thor::FrameAnimation playerDown;
Thor::Animator animator;

loadAnimations() {
        playerDown.addFrame(1.f, sf::IntRect(0, 0, 400, 600));
        playerDown.addFrame(1.f, sf::IntRect(400, 0, 400 ,600));
        playerDown.addFrame(1.f, sf::IntRect(800, 0, 400, 600));
        playerDown.addFrame(1.f, sf::IntRect(1200, 0, 400, 600));
        animator.addAnimation("down", playerDown, sf::seconds(1.f));
}

We use an animator from Thor to keep track of all the animations we need, with a reference by a string similar to how the texture manager works, and a duration. Now our input handling block will play the animation

case sf::Event::KeyPressed:
    {
        if(sf::Keyboard::isKeyPressed(sf::Keyboard::Down)){
            if(y < 600)
                y += 5;
            animator.playAnimation("down");
        }

        ...(add other directions here)

    }

    playerSprite.setPosition(x, y);
    animator.update(clock.restart());
    animator.animate(playerSprite);

And that’s all there is to get the animations I need for player movement as of now. Once we load the frames we want played and ensure the clock gets restarted, we just need to set the animation we want, and tell the animator to handle it, and Thor takes care of it all for us. This is essentially what I wanted my animation handler to do, but Thor did it much more elegantly and efficiently than I could.

After adding this in, I saw a glaring flaw in the input handling: manually manipulating the sprite position does not scale. Time to implement an entity-component system.

Sprite Rendering and State Handling

After a long hiatus (thanks to a 19 credit term), I’m back to working on the game. Textures and sprites now can be loaded and drawn pretty easily. Chaning between the states is now smoothly handled after I removed a bug. Now I can start the game in a menu, click on a button, transition to the ‘gameplay’ and render a sprite that changes when pressing the direction keys. Now time to explain how it all works, after a short GIF!

(GIF Here)

The first part of the architecture I worked on was state handling. The most basic thing one can do, at least in my game, is go from the menu, to the game or to the options, and of course there has to be a pause screen too. In order to manage these states, I made a StateManager class. This class has a stack of GameStates, which is the parent class of all possible states. Along with this stack, there are functions to push, pop, peek and change states, which is just a combination of pop and push. The state manager also runs the game loop, which handles input, updates the state based on the input, and draws all the changes. It does this by using a parent GameState objet, and running the respective functions of whatever the state actually is.

The GameState parent class isn’t used directy in the game, but instead provides the basic functions that each state will need to implement themselves, and lets the StateManager not care which state is currently on top of the stack. The MenuState currently has a blue background and a placeholder button. The input handling function only looks for mouse click within the parameters of the drawn sprite. The PlayState’s input handling fucntion instead check if the arrow keys are pressed. I haven’t implemented other states yet, but this is the general way they work.

For textures, I’ve created a TextureManager. When a texture is needed to be loaded in a state, the TextureManager takes a string, image file, and coordinates. It then takes the resulting slice of the image file and add it to a std::map with the string as its id. So when applying a texture, it looks something like

playerSprite.setTexture(this->textureManager.getRef("right"));

The ‘right’ reference then returns the corresponding texture. This makes it really easy to have multiple textures that you can apply to a single sprite, or to resuse a texture across mutliple sprites.

Unfortunately, this doesn’t work great for animation. I’m currently in the process of writing the animation componenet. Once that is done, I hope to work on making events easier to handle, rather than a big case statement looking for certain keys to be pressed. Until next time!

Rust Syntax Highlighting in Vim

I use Vim on a daily basis, and now that I’m learning Rust, I wanted syntax highlighting. There are some great tools out there, but the instructions to get them to work are broken up a bit. Here are the best instructions I found in one place.

First, you’ll need Vundle if you don’t have it. Vundle is a great tool for Vim, managing plugins and making them configurable in your .vimrc.

Make sure that your .vim directory has a bundle subdirectory. Then clone the Vundle repo there:

git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim

Now you can install and configure pluins within your .vimrc. But first, we need a little setup.

This little snippet needs to go at the top of your .vimrc. You have to make sure that filetype is set to off and that Vundle is running before we can load the Rust plugin.:

set nocompatible
filetype off

set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

Plugin 'VundleVim/Vundle.vim'

call vundle#end()
filetype plugin indent on

Awesome, so now Vundle starts up, loads itself, and then it ends. After this snippet, put whatever other setting for Vim you want. Run this from the command line to ensure all is working well:

vim +PluginInstall +qall

Now to get the Rust highlighting. I’m using this repo, which has worked pretty great for me so far. All you need to do to use it is to add this line before the call to vundle#end:

Plugin 'wting/rust.vim'

Now run the above command again to make sure everything works:

vim +PluginInstall +qall

And now you should have beautiful Rust syntax highlighting!

Enable Rust on OSU Servers

Rust was recently installed on flip{1-3} at Oregon State. In order to use it, you must first add the path to the library to your LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/usr/local/apps/rust/lib:$LD_LIBRARY_PATH

Now add the Rust binary directory path to your $PATH. This is needed for cargo to work:

export PATH=$PATH:/usr/local/apps/rust-1.4.0/bin/

Now you just need to alias rustc and cargo to the full paths:

alias rustc="/usr/local/apps/rust-1.4.0/bin/rustc"
alias cargo="/usr/local/apps/rust-1.4.0/bin/cargo"

Happy Rusting

Useful OpenSSL commands

Here a few useful commands I found when working with certs lately

Covert a pfx file into a pem:

openssl pkcs12 -in foo.pfx -out foo.pem -nodes

Extract the key out of a pem:

openssl pkey -in foo.pem -out foo.key

Extrct CA cert chain out of a pem:

openssl crl2pkcs7 -nocrl -certfile foo.pem | openssl pkcs7 -print_certs -out foo.chain.crt

Extract cert out of a pem:

openssl x509 -in foo.pem -outform DER -out foo.crt