# Machinechat JEDI Data Format Converter

This tool upgrades your historical data from the **JEDI 3.x** storage format to the
new **JEDI 4.0** format. Run it once, after upgrading to JEDI 4.0, so that data you
collected with older versions is still visible in JEDI 4.0.

The built program is called **`convert.bin`** (`convert.exe` on Windows).

---

## What you give the tool: two data directories

You give the tool two **JEDI data directories**:

| Option        | What it is                          | The tool uses… |
| ------------- | ----------------------------------- | -------------- |
| `-s` (source) | your **JEDI 3.x** data directory    | reads from `<source>/metrics/byUTC` |
| `-d` (dest)   | your **JEDI 4.0** data directory    | writes to `<dest>/metrics` |

You only point it at the data directories — the tool finds the `metrics/byUTC`
folder inside the source and writes the converted data into the `metrics` folder
inside the destination automatically. You do **not** add `metrics/byUTC` yourself.

> **What is the "data directory"?**
> The data directory is the folder where JEDI keeps everything it stores on disk —
> your collected sensor/device history and other JEDI files. It is the folder you
> configured when you set up JEDI. If you never set one, JEDI uses the folder that
> *contains* the JEDI program folder (the parent of the directory JEDI runs from).
> For `-s`, use the same data directory path you already give to your JEDI 3.x when
> you start it.

**Start JEDI 4.0 in a new, empty folder.** Use that fresh folder as its data
directory, then run this tool with `-s` = your existing JEDI 3.x data directory and
`-d` = the new JEDI 4.0 data directory. 

---

## What changed between JEDI 3.x and 4.0

Only the **folder layout and file format** of the stored history changed. The data
itself is the same — it is just organized differently so that it works better with
tools like Microsoft Excel and Python (pandas).

**JEDI 3.x and earlier** — data was grouped **by metric**, under a `byUTC` folder:

```
<3.x-data-dir>/
└── metrics/
    └── byUTC/
        ├── bme680_temp_c/
        │   └── 2026-07-17/
        │       └── ... .csv
        └── bme680_humidity/
            └── 2026-07-17/
                └── ... .csv
```

**JEDI 4.0** — data is grouped **by date and device**, directly under `metrics`:

```
<4.0-data-dir>/
└── metrics/
    └── 2026/
        └── 07/
            └── 17/
                └── vSensor/
                    ├── 11.csv
                    └── 12.csv
```

In short:

| Version   | Where history is stored          |
| --------- | -------------------------------- |
| JEDI 3.x  | `<data-dir>/metrics/byUTC`       |
| JEDI 4.0  | `<data-dir>/metrics`             |

The converter **reads** your old data from the source's `metrics/byUTC` and
**writes** the upgraded data into the destination's `metrics` folder. Your original
3.x data is **left in place and untouched** — nothing is deleted.

---

## How to use it

Give the tool your JEDI 3.x data directory with `-s` and your (new) JEDI 4.0 data
directory with `-d`.

> ⚠️ **The paths shown below are examples.** `/home/jedi3` and `/home/jedi4`
> (`C:\jedi3` and `C:\jedi4` on Windows) are placeholders — replace them with the
> paths to **your own** JEDI data directories. Do not copy the example paths as-is.

**macOS / Linux:**
```bash
./convert.bin -s /home/jedi3 -d /home/jedi4
```

**Windows (Command Prompt or PowerShell):**
```bat
.\convert.exe -s C:\jedi3 -d C:\jedi4
```

When it finishes you will see `Conversion finished successfully!`, and your history
will appear in JEDI 4.0.

---

## Example runs

### 1. Convert your data

```console
$ ./convert.bin -s /home/jedi3 -d /home/jedi4
Scanning source directory: /home/jedi3/metrics/byUTC...
Discovery complete. Total unique days found: 5
Days remaining to process: 5
Conversion finished successfully!
```

Notice the tool automatically looks inside `.../metrics/byUTC` in the source.

### 2. Windows

```bat
.\convert.exe -s C:\jedi3 -d C:\jedi4
```

### 3. Convert faster on a multi-core machine

Use `-w` to process more days at the same time (here, 8 at once):

```console
$ ./convert.bin -s /home/jedi3 -d /home/jedi4 -w 8
```

### 4. Check how far along it is

```console
$ ./convert.bin --status
Conversion Status (from conversion_state.json):
  Days Processed: 5
```

### 5. Check the tool version

```console
$ ./convert.bin --version
JEDI Data Format Converter version 4.0.1 (dev)
```

---

## Options

| Option              | What it does                                                                              |
| ------------------- | ----------------------------------------------------------------------------------------- |
| `-s, --source`      | **(required)** Your JEDI 3.x data directory. The tool reads `<source>/metrics/byUTC`.      |
| `-d, --dest`        | **(required)** Your JEDI 4.0 data directory. The tool writes to `<dest>/metrics`.          |
| `-w, --workers`     | How many days to convert at the same time (default: `4`). Higher = faster on more CPU cores. |
| `-f, --statefile`   | File used to remember progress (default: `conversion_state.json`).                         |
| `-l, --log`         | File where skipped/corrupted rows are logged (default: `corrupted_rows.log`).              |
| `--status`          | Show how many days have already been converted, then exit.                                 |
| `--version`         | Show the tool version and exit.                                                            |

---

## Checking progress / resuming

The conversion can be safely stopped and restarted. It remembers which days it has
already finished (in the state file), so if it is interrupted you can simply run the
**same command again** and it will pick up where it left off — it will not redo work.

To see how far along it is:

**macOS / Linux:**
```bash
./convert.bin --status
```

**Windows:**
```bat
.\convert.exe --status
```

> **Want to convert everything again from scratch?**
> Delete the `conversion_state.json` file, then run the command again.

---

## Good to know

- **Give data directories, not the `metrics/byUTC` folder.** Point `-s` and `-d` at the
  JEDI data directories; the tool adds `metrics/byUTC` (source) and `metrics` (dest) for
  you. If you pass `.../metrics/byUTC` directly, it will look for
  `.../metrics/byUTC/metrics/byUTC` and find nothing.
- **Your original data is safe.** The tool only reads the 3.x data; it never deletes or
  changes it. The converted 4.0 data is written to the destination.
- **Bad rows are skipped, not fatal.** If a few old records are corrupted, the tool
  logs them to `corrupted_rows.log` and keeps going instead of stopping.
- **"No new days to process" / nothing seems to happen?** That means every day was
  already converted (see the state file). Delete `conversion_state.json` if you want
  to run the whole conversion again.
- **"no legacy data found …"?** Double-check that `-s` points at your JEDI 3.x **data
  directory** (the one that contains a `metrics/byUTC` folder), not at `byUTC` itself.
- **Permission errors?** Make sure you have permission to write inside the destination
  data directory.