IoT Data Management with the TIG Stack

The TIG stack is a modification of the TICK stack, which is a group of applications provided by InfluxData. It is comprised of applications that receive, store, visualize, and act on time-based data. We’ll only be using the “T” and “I” components of the TICK stack (Telegraf and InfluxDB) and we’ll be adding the “G” as Grafana for our visualization.

This tutorial assumes you’ve either set up a NodeMCU with sensors like we did in Part II of my IoT tutorial, or you already some other device publishing data to an MQTT server.

Install Telegraf and Grab the NodeMCU Data

Telegraf is the tool that will grab our MQTT data and pass it along to InfluxDB for storage.

  1. Install Telegraf  – follow the instructions on the InfluxData downloads page to complete the install.
  2. Now we need to generate a configuration file for Telegraf that will contain the inputs and outputs we want Telegraf to handle. In this case we’ll want and MQTT input, and an InfluxDB output. First find out where your telegraf.conf file is located – on Linux it is typically under/etc/telegraf/telegraf.conf
  3. Now generate a new configuration file that will include the mqtt_consumer input and the InfluxDB output:
    telegraf -sample-config -input-filter mqtt_consumer -output-filter influxdb > telegraf.conf
    Note: you may need to sudo su before you run the above command in order to be able to output to the /etc folder. You could also generate the conf file in a folder you have permission to, and then use sudo to copy it into /etc/telegraf
  4. Now edit the configuration file, and find the “Input Plugins” section – here we’ll add our MQTT server info and a single topic:# Read metrics from MQTT topic(s)
    [[inputs.mqtt_consumer]]
    servers = [“localhost:1883”]
    ## MQTT QoS, must be 0, 1, or 2
    qos = 0

    ## Topics to subscribe to
    topics = [
    “/newdevice/#”,
    ]

    Here we’ve set the server to localhost because Telegraf is running on the same server as Mosquitto. In the topics, we’ve added a single subscription to “/newdevice/#” – newdevice is the name of our NodeMCU with sensors, and the hash/pound sign tells Telegraf to subscribe to everything that is published by that device. Now jump down towards the bottom of the MQTT config section, you need to edit two more lines:

    data_format = "value"
    data_type = "float"

     

  5. Save your configuration file, but don’t start Telegraf just yet.

Install InfluxDB and Start ’em Up!

Now we’ll install InfluxDB which will store our data, and start out services.

  1. Follow the instructions to install InfluxDB.
  2. Now start Influx, then Telegraf
    sudo systemctl start influxdb
    sudo systemctl start telegraf
  3. Give everything a minute to get subscribed and get some data flowing. Now we’ll check and see if our data is showing up in InfluxDB:[zenden:~$] influx
    Connected to http://localhost:8086 version 1.2.0
    InfluxDB shell version: 1.2.0
    > use telegraf
    Using database telegraf
    > show measurements
    name: measurements
    name
    —-
    mqtt_consumer

    select * from mqtt_consumer
    name: mqtt_consumer
    time host topic value
    —- —- —– —–
    1488122840639096875 zenden-server /newdevice/DHT/Temperature 69.44
    1488122840642429794 zenden-server /newdevice/DHT/Humidity 33.7
    1488122845619173945 zenden-server /newdevice/BH1750/Lux 54612.5
    1488122846645082191 zenden-server /newdevice/DHT/Temperature 69.44
    1488122846648567285 zenden-server /newdevice/DHT/Humidity 32.8

  4. Look at that! Now you’ve got a database storing your sensor values. Awesome! You can add more MQTT subscriptions to the telegraf.conf file – for instance you could log when you turn your relays on/off.

Install and Configure Grafana

While the TICK stack does have a visualization tool called Chronograf, the project is not quite far along enough to be fully usable as a dashboard solution. We’ll use our own option, Grafana, for visualization.

  1. Install Grafana on your server using the appropriate installation for your OS.
  2. Once Grafana is installed and running, you should be able to access it on port 3000 by default, navigate your browser to http://myipordomainname:3000/
  3. Login with “admin” as the username and password.
  4. Once you’re logged into Grafana, the first thing we need to do is connect to our InfluxDB instance. From the top-left Grafana dropdown menu, select “Data Sources” and click “Add Data Source”
  5. Fill in the data source info for your InfluxDB database:
    Be sure you set the URL of your InfluxDB Server (usually http://localhost:8086). Click “Save & Test” – a successful connection will show a nice happy success message.
  6. Now let’s create a new dashboard. Navigate from the top-left Grafana icon to Dashboards->New
  7. You should now be at a blank dashboard:
  8. Let’s add a graph! Click the “Graph” icon to add a new graph to the dashboard.
  9. You’ll get a blank graph in your row:
  10. Near where it says “Panel Title” left-click your mouse, and you’ll get a dropdown. Select “Edit”
  11. Now we need to build a query. Click near the letter “A” to drop down the query builder:
  12. Here we’ll select our humidity data. Use the dropdowns in the query builder to select the humidity column. The graph above the query builder should refresh every time you change the configuration. If you select the right data, you should see your humidity data populate:

    A couple notes here: instead of plotting EVERY single data point (that’s a LOT of data) we can tell Grafana/Influx to do some work make the data a bit more reasonable. By default, Grafana will want to group your data points into more manageable blocks of time. In the example image above, you’ll see we are selecting the mean() value of our data point, grouped by a time interval. Look a little further down, and you’ll see I’ve set my time interval to 1m. This will tell Influx to group all of my data points into 1-minute chunks, and send back to Grafana the mean value of all the data points in each 1-minute chunk. This makes our charts quite a bit more readable, and also will run a lot faster because we’re not plotting thousands of points. Adjust this based on your needs – even a 5-minute time interval would probably suffice, especially if you want to view days of data at a time on your chart.
  13. When you’re done, click the “X” in the query builder to exit and return back to your dashboard. At this point we haven’t saved our dashboard, so click the little disk icon at the top to save it.
  14. Now you can go to town! Here’s my completed dashboard:

2 Replies to “IoT Data Management with the TIG Stack”

  1. Hi Andy. THANK YOU VERY MUCH for this tutorial. Firstly I tried to repeat this job at Ubuntu within VirtualBox. But I had a lot of issues with VirtualBox. So I made all your steps at my Windows 7 x64:
    1. Mosquitto Broker v 3.1 (run as service)
    2. Telegraf (run as service)
    3. InfluxDB (I don’t know how to run as service, so it runs as app)
    4. Grafana (runs as app so far, although there is some manual to run as service)
    Now I have goo looking charts for my sensors! Thank you very much again for opening MQTT world for me.
    With the best wishes from Ukraine.

Leave a Reply

Your email address will not be published. Required fields are marked *