InfluxDB 2¶

references¶

assumptions¶

Your Raspberry Pi is running full 64-bit Raspberry Pi OS Debian GNU/Linux 11 (bullseye).
- DockerHub does not have a 32-bit image for InfluxDB 2 so you can't run this container until you have upgraded.
- Running full 64-bit is not the same as enabling the 64-bit kernel in /boot/config.txt. User-mode needs to be 64-bit capable as well. You must start from a full 64-bit image.
Node-RED is your principal mechanism for feeding data to InfluxDB 1.8.
- You may have other services feeding data to InfluxDB 1.8 (eg Telegraf). The steps documented here will migrate all your existing data but do not discuss how to adapt services other than Node-RED to feed new data to InfluxDB 2.
Grafana is your principle mechanism for creating dashboards based on data stored in InfluxDB 1.8.
- You may have other visualisation tools. You may gain insights from studying how Grafana needs to be changed to run Flux queries against InfluxDB 2 buckets but this documentation does not explore alternatives.
Node-RED, InfluxDB 1.8 and Grafana are all running in non-host mode on the same Docker instance, and that it is your intention to deploy InfluxDB 2 in non-host mode as well.
- If you are running any containers in host mode or have distributed the services across multiple Docker instances, you will have to adapt appropriately.

terminology: database vs bucket¶

InfluxDB 1.8 and InfluxDB 2 are both database management systems (DBMS), sometimes referred to as "engines", optimised for storage and retrieval of time-series data. InfluxDB 1.8 uses the term database to mean a collection of measurements. InfluxDB 2 uses the term bucket to mean the same thing.

When an InfluxDB 1.8 database is migrated, it becomes an InfluxDB 2 bucket. You will see this change in terminology in various places, such as the InfluxDB-out node in Node-RED. When that node is set to:

Version 1.x, the user interface has a "Database" field which travels with the connection. For example:
- [v1.x] influxdb:8086/power (set up in the connection sheet)
This implies that you need one connection per database.
Version 2.0, the user interface has a "Bucket" field which is independent of the connection. For example:
- [v2.0] influxdb:8086 (set up in the connection sheet)
- Bucketpower/autogen (set up in the node)
This implies that you need one connection per engine. It is a subtle but important difference.

reference service definition¶

The InfluxDB 2 service definition is added to your compose file by the IOTstack menu.

influxdb2:
  container_name: influxdb2
  image: "influxdb:latest"
  restart: unless-stopped
  environment:
    - TZ=Etc/UTC
    - DOCKER_INFLUXDB_INIT_USERNAME=me
    - DOCKER_INFLUXDB_INIT_PASSWORD=mypassword
    - DOCKER_INFLUXDB_INIT_ORG=myorg
    - DOCKER_INFLUXDB_INIT_BUCKET=mybucket
    - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token
    - DOCKER_INFLUXDB_INIT_MODE=setup
  # - DOCKER_INFLUXDB_INIT_MODE=upgrade
  ports:
    - "8087:8086"
  volumes:
    - ./volumes/influxdb2/data:/var/lib/influxdb2
    - ./volumes/influxdb2/config:/etc/influxdb2
    - ./volumes/influxdb2/backup:/var/lib/backup
  # - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro
  healthcheck:
    test: ["CMD", "influx", "ping"]
    interval: 30s
    timeout: 10s
    retries: 3
    start_period: 30s

As an alternative to using the menu, you can copy and paste the service definition into your compose file from the template at:

~/IOTstack/.templates/influxdb2/service.yml

required edits¶

Edit the service definition in your compose file to change the following variables:

TZ=«country»/«city»
DOCKER_INFLUXDB_INIT_USERNAME=«username»

This name becomes the administrative user. It is associated with your «password» and «token».
DOCKER_INFLUXDB_INIT_PASSWORD=«password»

Your «username» and «password» form your login credentials when you administer InfluxDB 2 using its web-based graphical user interface. The strength of your password is up to you.
DOCKER_INFLUXDB_INIT_ORG=«organisation»

An organisation name is required. Examples:
- myorg
- my-house
- com.mydomain.myhouse
DOCKER_INFLUXDB_INIT_BUCKET=«bucket»

A default bucket name is required. The name does not matter because you won't actually be using it so you can accept the default of "mybucket". You can delete the unused bucket later if you want to be tidy.
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=«token»

Although you can let InfluxDB 2 generate your access token for you, it will keep things simple if you generate your own. Here are some possible approaches:
1. use a universally-unique ID:
```
$ uuidgen
4fef85b4-2f56-480f-b143-fa5cb6e8f18a
```
2. use GnuPG to generate a random string:
```
$ gpg --gen-random -a 0 25
bYS3EsnnY0AlRxJ2uk44Hzwm7GMKYu5unw==
```
3. use a password-generator of your choosing.

Note:

Unless a container's documentation explicitly states that it is supported, you should never use quote marks to encapsulate the values you supply via environment variables. InfluxDB 2 treats quotes as being part of the value (eg a password of "fred" is the 6-character string that includes the quotes). If you put quote marks around anything as you were editing, please go back and remove them.

Table 1: mode-specific directives¶

InfluxDB 2 operates in three distinct modes which are controlled by the DOCKER_INFLUXDB_INIT_MODE environment variable. The table below summarises the variables and volumes mappings that need to be active in each mode.

Table 1

initialising InfluxDB 2¶

If you have only just included the template service definition in your compose file and performed the required edits, then you can follow the initialisation process below.

However, if you want to re-initialise the container, go to re-initialising InfluxDB 2.

To initialise InfluxDB 2:

Confirm that the service definition directives are set as per the "setup" column of Table 1.
Be in the correct directory (assumed throughout):
```
$ cd ~/IOTstack
```
Start the InfluxDB 2 container:
```
$ docker-compose up -d influxdb2
```
InfluxDB 2 will notice the following environment variable:
```
DOCKER_INFLUXDB_INIT_MODE=setup
```
This instructs the container to initialise the database engine structures based on a combination of defaults and the values you provide via the other environment variables.
Confirm that the InfluxDB 2 container is not in a restart loop and isn't reporting errors by using commands like:
```
$ docker ps
$ docker logs influxdb2
```

If you don't need to migrate any data from InfluxDB 1.8 you can go straight to running InfluxDB 2, otherwise follow the data-migration procedure instructions below.

data-migration procedure¶

Successful migration depends on the following assumptions being true:

The InfluxDB 2 container is running and has just been initialised as per initialising InfluxDB 2.
The InfluxDB 1.8 container is running, and is based on the IOTstack service definition (or reasonable facsimile) at:
```
~/IOTstack/.templates/influxdb/service.yml
```

To migrate your InfluxDB 1.8 data:

Be in the correct directory (assumed throughout):
```
$ cd ~/IOTstack
```
InfluxDB 1.8 runs as root and its persistent store is owned by root but not all files and folders in the persistent store are group or world readable. InfluxDB 2 runs as user ID 1000 (user "influxdb" inside the container). Because of this, you need to give InfluxDB 2 permission to read the InfluxDB 1.8 persistent store.

It is not a good idea to interfere with a persistent store while a container is running so best practice is to stop InfluxDB 1.8 for long enough to make a copy of its persistent store:
```
$ sudo rm -rf ./volumes/influxdb.migrate
$ docker-compose rm --force --stop -v influxdb
$ sudo cp -a ./volumes/influxdb ./volumes/influxdb.migrate
$ docker-compose up -d influxdb
$ sudo chown -R 1000:1000 ./volumes/influxdb.migrate/data
```
In words:
1. Ensure any previous attempts at migration are removed. Always be extremely careful with any sudo rm command. Check your work before you press return.
2. Stop InfluxDB 1.8.
3. Make a copy of the InfluxDB 1.8 persistent store.
4. Start InfluxDB 1.8 again.
5. Change ownership of the copy of the InfluxDB 1.8 persistent store.
Edit your compose file as per the "upgrade" column of Table 1. The changes you need to make are:
1. Change the initialisation mode from setup to upgrade:
  - before editing:
    12 13
    - DOCKER_INFLUXDB_INIT_MODE=setup # - DOCKER_INFLUXDB_INIT_MODE=upgrade
  - after editing:
    12 13
    # - DOCKER_INFLUXDB_INIT_MODE=setup - DOCKER_INFLUXDB_INIT_MODE=upgrade
2. Activate the volume mapping to give InfluxDB 2 read-only access to the copy of the InfluxDB 1.8 persistent store that you made in step 2:
  - before editing:
    20
    # - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro
  - after editing:
    20
    - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro
Save your work but do not execute any docker-compose commands.
InfluxDB 2 creates a "bolt" (lock) file to prevent accidental data-migrations. That file needs to be removed:
```
$ rm ./volumes/influxdb2/data/influxd.bolt
```
The InfluxDB 2 container is still running. The following command causes the container to be recreated with the edits you made in step 3:
```
$ docker-compose up -d influxdb2
```
InfluxDB 2 will notice the following environment variable:
```
DOCKER_INFLUXDB_INIT_MODE=upgrade
```
This, combined with the absence of the "bolt" file, starts the migration process. You need to wait until the migration is complete. The simplest way to do that is to watch the size of the persistent store for InfluxDB 2 until it stops increasing. Experience suggests that the InfluxDB 2 persistent store will usually be a bit larger than InfluxDB 1.8. For example:
- reference size for an InfluxDB 1.8 installation:
```
$ sudo du -sh ./volumes/influxdb
633M    ./volumes/influxdb
```
- final size after migration to InfluxDB 2:
```
$ sudo du -sh ./volumes/influxdb2
721M    ./volumes/influxdb2
```
Data migration is complete once the folder size stops changing.

Proceed to running InfluxDB 2 below.

running InfluxDB 2¶

The container now needs to be instructed to run in normal mode.

Be in the correct directory (assumed throughout):
```
$ cd ~/IOTstack
```

Edit your compose file as per the "(omitted)" column of Table 1. The changes are:

Deactivate all DOCKER_INFLUXDB_INIT_ environment variables. After editing, the relevant lines should look like:

  # - DOCKER_INFLUXDB_INIT_USERNAME=me
  # - DOCKER_INFLUXDB_INIT_PASSWORD=mypassword
  # - DOCKER_INFLUXDB_INIT_ORG=myorg
  # - DOCKER_INFLUXDB_INIT_BUCKET=mybucket
  # - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token
  # - DOCKER_INFLUXDB_INIT_MODE=setup
  # - DOCKER_INFLUXDB_INIT_MODE=upgrade

Deactivate the volume mapping if it is active. After editing, the line should look like:
20
# - ./volumes/influxdb.migrate/data:/var/lib/influxdb:ro

Save your work.

The InfluxDB 2 container is still running. The following command causes the container to be recreated with the edits you have just made:
```
$ docker-compose up -d influxdb2
```
The absence of an active DOCKER_INFLUXDB_INIT_MODE variable places InfluxDB 2 into normal run mode.
If you have just performed a data migration, you can remove the copy of the InfluxDB 1.8 persistent store:
```
$ sudo rm -rf ./volumes/influxdb.migrate
```
always be extremely careful with any sudo rm command. Always check your work before you press return.

re-initialising InfluxDB 2¶

If you need to start over from a clean slate:

Be in the correct directory (assumed throughout):
```
$ cd ~/IOTstack
```

Terminate the InfluxDB 2 container:

$ docker-compose rm --force --stop -v influxdb2

Remove the persistent store:
```
$ sudo rm -rf ./volumes/influxdb2
```
always be extremely careful with any sudo rm command. Always check your work before you press return.

Edit your compose file as per the "setup" column of Table 1. After editing, the relevant lines should look like this:

    - DOCKER_INFLUXDB_INIT_USERNAME=me
    - DOCKER_INFLUXDB_INIT_PASSWORD=mypassword
    - DOCKER_INFLUXDB_INIT_ORG=myorg
    - DOCKER_INFLUXDB_INIT_BUCKET=mybucket
    - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token
    - DOCKER_INFLUXDB_INIT_MODE=setup
  # - DOCKER_INFLUXDB_INIT_MODE=upgrade

Go to initialising InfluxDB 2.

exploring InfluxDB 2 data¶

browse data

Launch a browser and connect it to port 8087 on your Raspberry Pi. For example:
```
http://raspberrypi.local:8087
```
You can also use the IP address or domain name of your Raspberry Pi. In this context, 8087 is the external port number from the left hand side of the port mapping in the service definition:
14 15
ports: - "8087:8086"
Sign in to the InfluxDB 2 instance using your «username» and «password».
Click on "Explore" in the left-hand tool strip. That is marked [A] in the screen shot. In the area marked [B] you should be able to see a list of the buckets that were migrated from InfluxDB 1.8 databases.

In the screen shot, I clicked on other fields to create a query:
- In area [B], I selected the "power/autogen" bucket;
- In area [C], I selected the "hiking2" (electricity meter) measurement;
- In area [D], I selected the "voltage" field;
- The bucket in this test is a migrated copy of an InfluxDB 1.8 database. It was not ingesting live data so I also needed to change the duration popup menu [E] to a time-span that included the most-recent insertions;
- Then I clicked the "Submit" button [F]; and
- The result was the graph in [G].

You can explore your own tables using similar techniques.

Flux queries via point-and-click¶

Grafana does not (yet) seem to have the ability to let you build Flux queries via point-and-click like you can with InfluxQL queries. Until Grafana gains that ability, it's probably a good idea to learn how to build Flux queries in InfluxDB, so you can copy-and-paste the Flux statements into Grafana.

Once you have constructed a query in the "Query Builder", click the "Script Editor" button [H] to switch to the editor view.

For this example, the query text is:

from(bucket: "power/autogen")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "hiking2")
  |> filter(fn: (r) => r["_field"] == "voltage")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

Two important things to note here are:

The bucket name: power/autogen; and
The measurement name: hiking2.

example: adapting Node-RED¶

Node-RED flow models

Assume you have an existing flow (eg a fairly standard 3-node flow) which is logging to an InfluxDB 1.8 database. Your goal is to modify the flow to log the same data to the recently-migrated InfluxDB 2 bucket.

Start Node-RED if it is not running:

$ cd ~/IOTstack
$ docker-compose up -d nodered

Use a web browser to connect to your Node-RED instance.
Drag a new InfluxDB-out node onto the canvas:
- This is exactly the same InfluxDB-out node that you have been using to write to your InfluxDB 1.8 databases. There isn't a different node or package for InfluxDB 2.
- Always drag a new InfluxDB-out node from the palette onto the canvas. Do not make the mistake of re-using an existing InfluxDB-out node (eg via copy and paste) because that is a very good way of breaking your flows.
Double-click the InfluxDB-out node to open it:
- At [A], give the node a sensible name.
- Click the pencil icon [B] adjacent to the Server field:
  - Leave the Name field [C] blank. This ensures that the title in the popup menu [D] automatically reflects the version and connection URL.
  - Change the Version popup menu [E] to "2.0".
  - Set the URL [F] to point to your InfluxDB 2 instance:
```
http://influxdb2:8086
```
    In this context, "influxdb2" is the container name and 8086 is the container's internal port. Node-RED communicates with InfluxDB 2 across the internal bridged network (see assumptions).
  - Paste your «token» into the Token field [G].
  - Click "Update" [H].
- Set the Organisation field [I] to your «organisation».
- Set the Bucket [J] to the correct value. You can get that from either:
  - area [B] in the Influx Explorer screen shot; or
  - the bucket name from the saved Flux query.
  In this example, the bucket name is "power/autogen".
- Set the Measurement [K] to the measurement name. You can get that from either:
  - area [C] in the Influx Explorer screen shot; or
  - the measurement name from the saved Flux query.
  In this example, the measurement name is "hiking2".
- Click Done [L].
Connect the outlet of the Change node to the inlet of the InfluxDB-out node.
Click Deploy.
Watch the debug panel to make sure no errors are being reported.
Go back to the InfluxDB 2 Data Explorer and click the refresh button "I". If everything has gone according to plan, you should see recent observations added to your graph.

You may need to wait until your sensor has sent new data.

example: adapting Grafana¶

defining an InfluxDB 2 data source¶

Start Grafana if it is not running:

$ cd ~/IOTstack
$ docker-compose up -d grafana

Use a web browser to connect to your Grafana instance and login as an administrator.
Hover your mouse over the "gear" icon in the tool-strip on the left hand side, and choose "Data sources".
Click the "Add data source" button.
Select the "InfluxDB" option.
Configure as follows:
- Change the Name [A] to a meaningful title that reflects the bucket you are going to query. For example, "InfluxDB 2.0 power".
- Change the Query Language popup menu [B] to "Flux".
  
  Ignore the advice about Flux support being in beta.
- Change the URL [C] to point to your InfluxDB 2 instance:
```
http://influxdb2:8086
```
  In this context, "influxdb2" is the container name and 8086 is the container's internal port. Grafana communicates with InfluxDB 2 across the internal bridged network (see assumptions).
- Turn off all the switches in the "Auth" group [D].
- Set the Organisation [E] to your «organisation».
- Paste your «token» into the Token field [F].
  
  ignore the fact that the prompt text says "password" - you need the token!
- Set the Default Bucket [G] to the bucket (database) you want to query. You can get that from either:
  - area [B] in the Influx Explorer screen shot; or
  - the bucket name from the saved Flux query.
  In this example, the value is "power/autogen".
- Click Save & Test [H].

using an InfluxDB 2 data source in a dashboard¶

Find the + icon in the tool-strip on the left hand side, hover your mouse over it and choose "Create » dashboard".
Click "Add a new panel".
Change the "Data source" popup to the bucket connection you created earlier ("InfluxDB 2.2 power").
The editor automatically switches into Flux mode.
Paste the query text you saved earlier from the InfluxDB 2 query inspector.
If necessary, change the duration to a period that is likely to contain some data to display.
Click the Refresh button.
Click Apply.

In the side-by-side screen shots below, observations before the straight-line (missing data) segment were imported from InfluxDB 1.8 while observations after the straight-line segment were inserted by the new InfluxDB-out node in Node-RED.

compare results

odds and ends¶

Forgot your token:
```
$ docker exec influxdb2 influx auth ls
```

Create a new user, password and token:

$ docker exec influxdb2 influx user create --name «username» --password «password»
$ docker exec influxdb2 influx auth create --user «username» --all-access

List available buckets:

$ docker exec influxdb2 influx bucket ls

Delete the default «bucket»:

$ docker exec influxdb2 influx bucket delete --org «organisation» --name «bucket»

migration strategy¶

From the fact that both InfluxDB 1.8 and InfluxDB 2 can run in parallel, with Node-RED feeding the same data to both, it should be self-evident that you can repeat the data-migration as often as necessary, simply by starting from re-initialising InfluxDB 2.

This implies that you can concentrate on one database at a time, adjusting Node-RED so that it writes each row of sensor data to both the InfluxDB 1.8 database and corresponding InfluxDB 2 bucket.

Having the data going to both engines means you can take your time adjusting your Grafana dashboards to be based on Flux queries. You can either retrofit InfluxDB 2 bucket sources and Flux queries to existing dashboards, or build parallel dashboards from the ground up.