TSDB tutorial - 2023/11/16

This unconventional approach can be used even when you do not have admin rights on the computer: all software runs in user mode and uses non-privileged ports. Obviously you should

IMPORTANT NOTE for Windows it seems you need to locate the database in a directory located on the Windows filesystem (i.e. not in the pseudo fs emulated for $HOME). So: make sure you are in your User directory on Windows (the prompt should display something like /mnt/c/Users/username/. Create a tsdb directory somewhere in this tree (for instance /mnt/c/Users/username/Desktop/tsdb to find it easily) and make sure to always run the commands from there.

Download the Influxdb (v 2.7.3), Telegraf software and the InfluxDB v2 Cloud CLI (as static binaries*), *for Linux 64-bit (even on Windows since we will use WSL).

Uncompress the archives, and run the server in a terminal (again: make sure you are in the right directory):

./influxdb2-2.7.3/usr/bin/influxd --bolt-path=data/bolt --engine-path=data/engine

Then access the administrative URL at the address http://localhost:8086/

Create a new user, with a password (do not use your existing account password). Use tp for initial organization name (it corresponds to a group), and local for initial bucket name.

Note in a file the API key it generates to authentify against the admin user. Then click on Quick start to setup local metric collection. It will automatically create sensors for multiple measurements linked to the InfluxDB database itself (number of users, of queries, of databases…) so that you can monitor its behaviour. The associated dashboard InfluxDB 2.0 OSS Metrics will also be created so that you can visualize the data.

[Question] Open the InfluxDB 2.0 OSS Metrics dashboard. How many buckets are defined in the system.

Let's configure the authentication of the command-line tool, so that we can use it more conveniently.

You should have kept the API token generated for the admin user. If not, you will to generate a new token to grant access to the database. In the web GUI, got to the Load Data / API Tokens. Click on Generate API token and select All Access API token (for simplifying things in the context of this tutorial, do not do this at home). Click on the created token and copy its textual value into a file to preserve it.

Open another terminal (the first one is used to have the daemon run) and set the environment variables so that you can use them more easily:

export INFLUX_TOKEN=<token value>
export INFLUX_ORG=tp
export INFLUX_BUCKET=local

Then configure the authentication for the CLI with the config command:

./influx config create --config-name tpconfig \
  --host-url http://localhost:8086 \
  --org $INFLUX_ORG \
  --token $INFLUX_TOKEN \
  --active

You can now use influx on the command line to interact with your database. Do not use the web interface for the moment.

[Question] What is the influx command that allows you to list the available buckets?

[Question] What are the names of the available buckets?

For the moment, our InfluxDB is configured to monitor itself. We will configure a system resource monitor, in order to monitor the computer system. In the main page, click on Load your data button.

The telegraf component is used to collect data and send it to the InfluxDB database. In the Load data interface, select the Telegraf tab and create a new configuration. Select the System configuration, select all measurements (cpu, disk, etc) and create the configuration.

A specific token is created so that the Telegraf component can only write its data to a specific bucket. Copy it to the same file as the first token to keep a copy. A URL endpoint for writing data is also given. They are indicated in the dialog displayed after the creation.

Open another terminal (you should already have one for the database and one for running the influx client), set the INFLUX_TOKEN environment variable generated for the Telegraf component and start the telegraf server (note that it is in usr/bin in the extracted telegraf directory) with the command-line given by Influx. We will let it run for a while so that it collects some data, before exploring it.

We will use a sample database. Download sample data. The grep part is to keep only lines that define measurements.

curl https://s3.amazonaws.com/noaa.water-database/NOAA_data.txt | grep h2o > NOAA_data.txt

Have a look at the loaded data to make sure everything is ok. Check the timestamp values and determine their unit. Hint: in python, time.localtime returns a time structure from a number of seconds.

Using the command-line interface, create a new bucket named noaa and import the NOAA data into it (influx binary, write command). Important Do not forget to specify the precision (time unit) used in your file.

• [Question] Give the command-line used to import the data file.

We will first use the command-line interface influx to start exploring the data, and make queries using the Flux language.

The Flux language often requires to filter against a time range. We first have to determine the appropriate time range for our dataset. For this, we will do some queries with a very large time range (starting in 1900) so that we know that we should get the appropriate info.

The following command runs the query that outputs the first measurement for all series:

./influx query -r 'from (bucket: "noaa") |> range(start: 1900-01-01, stop: now()) |> first() |> keep(columns: [ "_time", "_measurement" ])'

[Question] What is the beginning datetime of measurements? Note: you will need this information in the next questions

Modify the above query to find the time of the last measurement of noaa.

[Question] What is the new query, and what is the end datetime of measurements?

Now that we know the appropriate time range, we can investigate our data structure. Influx provides a package (functions defined using Flux) to help with this, in the schema package. See https://docs.influxdata.com/influxdb/v2/query-data/flux/explore-schema/ for its documentation.

Note that many of the exploring functions need a start parameter expressing a duration (the default -30 days if fine for real-time data, but is too short in our case).

• [Question] What measurements are present in the noaa bucket? What is the query to list them?
• [Question] What are the fields defined in the h2o_feet measurements, and the query to find them?
• [Question] What are the tags keys defined in the h2o_feet measurements, and the query to find them?
• [Question] What are the different values for the location tag in h2o_feet measurements, and the query to find them?

Using the Flux syntax, answer the following questions:

• [Question] How many measurement points of water level are there in the h2o_feet measurement for each location? (hint: count). Give the query used.
• [Question] Convert the h2o_feet measurement into a new h2o_meter measurement by converting the water_level values to meters (hint: map for mapping values, to for storing the new measurement into noaa). Give the query used.