Tools: evaluating the binary logs

The bike computer writes its measurements as a sequence of fixed binary records to the SD card (format v2, see src/LogRecords.h). Tools/ holds everything that works with them offline.

Format	Record size	Content
v1	56 bytes	ride data only (every 5 s) -- older logs, still read
v2	64 bytes	type byte at offset 30: 0 = ride data, 1 = road quality (per interval, 1..10 s), 2 = shock, 3 = manual road label, 4 = ride state

The version number is in every record at offset 31. Unknown record types are skipped (and counted in bikelog info).

Manual road labels (type 3, added without a new version): surface (1 asphalt, 2 gravel, 3 forest track, 4 field track, 5 cobbles, 6 other) and quality 1 (best) to 4, set on the road-label screen or by rq label <surface>,<quality>. One record at every change, repeated every 60 s to be safe and at start/stop of a raw capture; the label applies until the next record. It is also in every shock record and every raw data block. bikelog info compares the labels with the automatic class (distance, median roughness and class distribution per label). The mapping to OSM surface/smoothness is in bikelog/record.py (SURFACE_OSM, LABEL_QUALITY_OSM).

bikelog/            library + CLI (standard library only, no venv needed)
BikeLogService/     service: fetches sessions from the device (mDNS), GPX/CSV (FastAPI, own venv)
pyproject.toml      makes Tools/ installable (for the service, see BikeLogService/install.sh)
tests/              pytest suite for both
ReadTachoBin.py     the well-known CSV converter, now a wrapper around "bikelog csv"
csv2influx.py       unchanged: CSV to InfluxDB
gpxenrich/          GPX track -> route for TrailBridge: turn hints from BRouter, waypoints as
                    named hints (description in the header of gpx_enrich.py)

Files on the SD card

Every boot is a session. It first writes to the working directory /BIKECOMP/CUR/, named by a running number -- at boot the clock is usually not set yet:

File	Content
L_0042.bin	binary log (this format)
D_0042.log, N_0042.log	debug log, Forumslader NMEA
S_0042.bin, R_0042_NN.bin	raw data, see below
T_0042.txt	time hints: start <epochMs> <valid 0/1> <uptimeMs>, then per clock step step <ntp\|gps\|?> <offsetMs> <newEpochMs> <uptimeMs>

After the next boot a background task (src/LogSessions.cpp) clears every older session out of CUR/:

• Start time from T_*; if the session started without a clock (1970), it is corrected by the step with which NTP or the GPS time from TrailBridge made the clock valid.
• Target /BIKECOMP/YYYYMMDD/X_HHMMSS.*, without a determinable time /BIKECOMP/NO_TIME/X_0042.*. In L_*.bin the 1970 timestamps are rewritten; raw data and debug log keep their original times.
• I_HHMMSS.txt: short statistics as key=value lines (distance, duration, moving time, average/maximum speed, share of GPS, number of road-quality intervals, shocks, labels, raw captures; corr_ms = applied time correction). The log file page of the web server shows them per session.
• Empty files are deleted.

Older logs (from before this change) still lie as L0042.bin in NO_TIME/ or dated without I_*.txt.

CLI

Without installation, directly from Tools/:

python3 -m bikelog info -i /path/L0001.bin          # overview of a log
python3 -m bikelog csv  -i /path/L0001.bin -o out.csv [--gps] [--roadq]
python3 -m bikelog csv  -i /path/L0001.bin -o out.csv --roadq-out roads.csv --shocks-out shocks.csv --labels-out labels.csv
python3 -m bikelog gpx  -i /path/L0001.bin -o tour.gpx [--no-shocks] [--min-severity 2]
python3 -m bikelog report -i /path/L0001.bin [--json] [--athlete athlete.json] [-o report.md]
./ReadTachoBin.py -i /path/L0001.bin -o out.csv     # as before

GPX export

"Rich" by default: besides Garmin's TrackPointExtension v2 (heart rate, cadence, temperature, speed, course) every track point carries an extension of its own with gradient, road quality, label etc., label changes become waypoints, the metadata contains the key figures of the ride -- see the log service. --plain leaves out the own extensions, --no-labels the label waypoints.

The XML is the easy part; what matters is what does not go in:

Filter	Default	Option
records without a GPS fix	always out	--
stale fixes (heartbeat repetition in a tunnel)	> 5000 ms out	--max-fix-age
timestamps before 2020 (clock not set by NTP)	always out	--
position exactly (0, 0)	always out	--
fixes that are too inaccurate	off	--max-accuracy
time gap → new <trkseg>	> 60 s	--segment-gap

Altitude source via --ele=auto|baro|gps; auto takes the barometer if its value is plausible, otherwise the GPS altitude. Heart rate 0 and 255 mean "no value" and are left out.

Shocks (v2) are output as <wpt> -- name Stoß 5,2 g, <type> shock-1..3 by severity, in <desc> the second peak (rear wheel) and the speed. The same position and time filters apply to them as to the track points. The road-quality intervals themselves (with all fields) are still only available as CSV (--roadq-out); in the GPX every track point carries class and roughness of the interval it falls into.

Every run reports what was discarded:

195/243 Punkte in 2 Segment(en), 20 verworfen (ohne Fix), 25 verworfen (veraltet)

Session report

bikelog report computes all key figures of a session; --json outputs them as JSON (the input for a text generator, template or LLM), without it a German Markdown report is printed. Code: bikelog/report.py, tests: tests/test_report.py.

Section	Content
meta	time span of the ride proper, format, record counts, clock; idle: seconds cut at the start and end (see below); test: test session (simulator flag or GPS playback suspected, see the log service)
ride	distance, moving time, speeds, elevation (height changes while standing do not count, see below), stops (a logging gap counts as a stop: the device sleeps), temperature, cadence and share of pedalling
climbs	climbs in the barometric profile with the firmware's criteria and categories (climbs): length, height, mean/max. gradient, time, VAM, heart rate, power, position of foot and summit; --hills also lists hills below category 6
profile	barometric elevation profile ([km, m], at most 300 points) for charts
heart	mean/max., coverage, time in zones, TRIMP (Banister with hr_rest, else Edwards), aerobic decoupling (power/heart rate, first against second half, from 40 min; decoupling_basis says whether wind was included or the figure was suppressed because of it, see below), efficiency W/bpm
power	estimated from speed, gradient, mass, CdA and Crr and, with weather data, wind: mean, normalised power, work, best 1/5/20/60 min, W/kg with rider_kg; no_wind repeats mean/NP/work without wind -- for comparing rides, not a measurement
wind	the weather behind it (Open-Meteo, see below): mean wind and direction, gusts, mean headwind component at rider height, share of time with head- and tailwind; or available: false with a reason
road	road classes, manual labels against the automatic class, shocks by severity, the five hardest with position
health	heart-rate dropouts, wheel sensor against GPS (dropouts, wheel circumference), GPS freshness and accuracy, baro against GPS altitude (offset, drift), lost records (gaps in the numbering), cut-off file, unset clock, simulator, IMU flags; plus findings, a list of everything that stood out

--athlete takes a JSON file with rider data, all optional: hr_max, hr_rest, mass_kg (rider + bike, default 85), rider_kg, cda (0.40), crr (0.006), drivetrain (0.97), wind_factor (0.55), zones_pct (upper bounds of zones 1-4 in % of hr_max, default 60/70/80/90), trimp_b (1.92). Times within the ride are seconds since its start (t_s), positions trip kilometres (km).

Idle and standing. The device is often switched on long before the ride and still running after it (indoors, heart-rate strap off, GPS poor). That is not a stop: the report starts at the first record from which the wheel really moves on (50 m within a minute) and ends at the last one it came from (trim_idle()); duration, stops, temperature, GPS accuracy, road quality and shocks only cover that stretch, meta.idle says how much was cut. Stops in the middle stay stops. Height changes while standing (the wheel below 2 km/h) are not climbing -- a lift ride with the bike, or weather drift: ascent, descent, the profile and the climb detection use heights with those jumps taken out (ridestats.corrected_heights(), as the firmware does for the gradient), min/max only look at heights while riding.

Wind. bikelog report --wind fetches the hourly wind (10 m speed, direction, gusts, temperature, pressure) of the ride's day from Open-Meteo (archive API; the "historical forecast" API for days the archive does not have yet) and puts it into the power estimate (--wind-cache DIR, default ~/.cache/bikelog/weather; the log service does the same automatically, see its session report). Only the 0.1 degree cell of the ride and the date go over the line, never the track. The report itself (report.compute(..., weather=)) does no network I/O; the fetch and the cache are bikelog/weather.py.

• The headwind component per step comes from the GPS bearing (else from the fixes around the record); the wind blows from the given direction, so riding west into a wind from 250 degrees is a headwind. Air flows past the rider at speed + headwind; crosswind is ignored.
• Wind at rider height = wind_factor x 10 m wind (default 0.55): the 10 m value is for open terrain, close to the ground (rider at ~1 m) the wind is weaker (logarithmic wind profile, roughly 0.5-0.65 of the 10 m value over fields, less in forest and built-up areas).
• Air density from temperature and pressure of the weather data; the no_wind figures use 1.2 kg/m3.
• Not modelled: shelter (forest, buildings -- there the constant factor overestimates the headwind), gusts, crosswind.
• Decoupling and wind. Power is estimated from speed, so a wind that differs between the two halves (out against it, back with it) puts a number into the decoupling that has nothing to do with the heart. With weather data the decoupling is computed again with the wind factor 40 % lower and higher; if the two results differ by more than 8 percentage points, decoupling_pct is null and decoupling_basis is suppressed (decoupling_nowind_pct, decoupling_wind_pct, decoupling_spread_pts and halves_headwind_kmh stay in the data). A calm day, or wind from the side, keeps the figure (wind); without weather data it is the old no-wind figure (no_wind).

Raw data of the accelerometer

Next to the log of a session L_HHMMSS.bin there can be two raw data files (format: src/RawCapture.h, 400 Hz, raw LSB):

File	Created	Size
R_HHMMSS_NN.bin	on demand: rq raw <s> (serial), buttons on /debug/imu or the record button of the road-label screen, up to 1800 s	about 150 KB/min
S_HHMMSS.bin	always: 0.25 s before to 0.5 s after every logged shock	about 1.8 KB/shock

python3 -m bikelog raw info   -i R_143012_01.bin
python3 -m bikelog raw csv    -i R_143012_01.bin -o raw.csv          # frames in g
python3 -m bikelog raw replay -i R_143012_01.bin shock=2.5 interval=1 --rq-out rq.csv
python3 -m bikelog raw replay -i S_143012.bin                        # recompute every shock

replay computes with the firmware code itself (src/RoadQuality.cpp), not with a Python imitation: on the first call it builds rqreplay/rq_replay.cpp with g++ into .build/ and passes the parameters (python3 -m bikelog raw replay -h lists them). This way thresholds and filters can be tried on real recordings before they go into the firmware. The manual label of every block runs along: raw info lists the label sections, raw replay shows roughness and classes per label -- the direct comparison of "felt" against "measured".

Test data without hardware

python3 -m bikelog fixture synth -o ride.bin --unset-clock 3 [--roadq]
python3 -m bikelog fixture from-gpx tour.gpx -o ride.bin

synth deliberately produces the unpleasant cases (start without a fix, tunnel with stale fixes, break as a time gap, unset clock). from-gpx builds a binary log with realistic geometry from a real tour -- e.g. a Komoot export -- as long as there are no real recordings yet.

Tests

python3 -m venv .venv
.venv/bin/pip install -r BikeLogService/requirements.txt
.venv/bin/python -m pytest

tests/test_logformat.py pins the binary layout to the firmware: test_layout_matches_firmware_header compiles src/LogRecords.h with g++ on the host (logrecords_dump.cpp), has one record per type written and reads it back. If it fails, the firmware structs have changed and bikelog/record.py needs new entries in FORMATS (and the firmware a new FORMAT_VERSION). Without g++ the test is skipped.