# Jitterbug

> Jitter-based congestion inference from RTT time series (PAM 2022)

Jitterbug infers Internet path congestion from Round-Trip Time (RTT) time series. The PyPI distribution is jitterbug-inference; the import package and the CLI are jitterbug. It computes minimum-RTT intervals, detects change points, and classifies each period with a latency jump plus either jitter dispersion or a Kolmogorov–Smirnov test. The method is the one in the PAM 2022 paper (https://doi.org/10.1007/978-3-030-98785-5_7).


# Guides

# 🐞 Jitterbug

A Python framework for inferring Internet path congestion from Round-Trip Time (RTT) measurements. Jitterbug splits an RTT time series at change points, then classifies each period as congested or not by combining a latency-jump test with a jitter test (jitter dispersion or Kolmogorov–Smirnov). It implements the method from *Jitterbug: A New Framework for Jitter-Based Congestion Inference* (PAM 2022) and ships the paper's dataset so you can reproduce it in one command.

Documentation for language models: [`llms.txt`](https://estcarisimo.github.io/jitterbug/llms.txt) (page index) and [`llms-full.txt`](https://estcarisimo.github.io/jitterbug/llms-full.txt) (the same pages in one file).

## ✨ Features

- 📉 **Congestion inference from RTTs alone**: no active probing beyond the pings you already have
- 🔀 **Pluggable change point detection**: `ruptures` out of the box, the paper's Bayesian detector (`bcp`) as an extra
- 📐 **Two jitter tests**: jitter dispersion (moving IQR + moving average) or a Kolmogorov–Smirnov test between periods
- 🧩 **Non-sequential mode**: cluster 15-minute intervals by (latency, jitter) with a Gaussian mixture or k-means and compare each cluster with the baseline, no matter when its intervals occur
- 📁 **Multiple input formats**: CSV, scamper JSON, and InfluxDB queries
- 📊 **Rich terminal output**: summary and per-period tables, plus JSON or CSV results files
- ⚙️ **Typed configuration**: Pydantic models, YAML/JSON config files, `JITTERBUG_*` environment variables
- 🧪 **Reproducible**: the PAM 2022 dataset (47 163 measurements) and its reference results are bundled
- 🐍 **Library and CLI**: use `jitterbug analyze` or call `JitterbugAnalyzer` from your own code

## 🚀 Quick Start

### Installation

From PyPI:

```
pip install jitterbug-inference              # core: ruptures detector, CLI, library
pip install "jitterbug-inference[bcp]"       # + the paper's Bayesian detector
```

> The distribution is named `jitterbug-inference` because `jitterbug` on PyPI is an unrelated project; the import is `import jitterbug` and the command is `jitterbug`. `pip install jitterbug` installs the wrong package.

For development, clone and use [uv](https://docs.astral.sh/uv/):

```
git clone https://github.com/estcarisimo/jitterbug.git
cd jitterbug
uv sync
source .venv/bin/activate   # or prefix every command below with `uv run`
```

See [docs/INSTALLATION.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/INSTALLATION.md) for pip from a clone, pinned versions and troubleshooting.

### Optional back ends

| Extra           | Installs                                                                       | Use it for                                                                                   |
| --------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- |
| `bcp`           | [bayesian-changepoint](https://pypi.org/project/bayesian-changepoint/) + torch | The Bayesian detector used in the paper, and the online mode (`jitterbug stream` / `replay`) |
| `clustering`    | scikit-learn                                                                   | The non-sequential mode (`--mode clustering`)                                                |
| `influx`        | influxdb-client                                                                | Loading RTTs straight from InfluxDB                                                          |
| `zstd`          | zstandard (not needed on Python 3.14+)                                         | Reading and writing `.zst`-compressed files                                                  |
| `visualization` | matplotlib                                                                     | `jitterbug visualize` and the plotting helpers                                               |
| `jupyter`       | JupyterLab, ipykernel                                                          | The notebooks in `examples/`                                                                 |
| `all`           | everything above                                                               |                                                                                              |

```
pip install "jitterbug-inference[bcp,visualization]"   # the paper's setup plus figures
uv sync --extra all                                     # from a clone: everything
```

### System requirements

- Python 3.10 or higher

## 📖 Usage

### Analyze a file

```
# Print the summary and the congestion periods found in the bundled dataset
jitterbug analyze examples/network_analysis/data/raw.csv

# Save the full results
jitterbug analyze examples/network_analysis/data/raw.csv --output results.json

# Reproduce the paper: Bayesian change points + KS test (needs `uv sync --extra bcp`)
jitterbug analyze examples/network_analysis/data/raw.csv --algorithm bcp --method ks_test

# Non-sequential mode: cluster intervals by (latency, jitter) (needs `--extra clustering`)
jitterbug analyze examples/network_analysis/data/raw.csv --mode clustering

# Only the summary table
jitterbug analyze examples/network_analysis/data/raw.csv --summary-only

# CSV instead of JSON (Parquet is also supported if `pyarrow` is installed)
jitterbug analyze rtts.csv --output results.csv --output-format csv

# Zstandard-compressed input and output: a trailing .zst is enough (needs the `zstd` extra
# before Python 3.14)
jitterbug analyze rtts.csv.zst --output results.json.zst
```

### Validate input before analyzing

```
jitterbug validate rtts.csv --verbose
```

### Plot the analysis

```
# Five PNG figures (raw RTT, minimum RTT, change points, verdicts, confidence) at 300 dpi;
# needs `uv sync --extra visualization`
jitterbug visualize examples/network_analysis/data/raw.csv --output-dir plots
```

The figures are described in [docs/VISUALIZATION_USAGE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/VISUALIZATION_USAGE.md), together with the `JitterbugPlotter` API for your own scripts.

### Online mode: stream and replay

```
# Infer congestion as samples arrive: `epoch,rtt` lines on stdin, one JSON event per line;
# needs `uv sync --extra bcp`
my-probe | jitterbug stream --events verdicts

# Replay a recorded dataset through the online pipeline and score it against a reference
jitterbug replay examples/network_analysis/data/raw.csv --method ks_test \
  --reference examples/network_analysis/expected_results/kstest_inferences.csv
```

On the paper dataset congestion onsets are reported 15 min after they start (median; 90 min at most); see [docs/ONLINE_MODE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/ONLINE_MODE.md) for the method, the results on the paper dataset and the `streaming` configuration section.

### Configuration files

```
# Write a template with every option and its default
jitterbug config --template --output config.yaml

# Use it
jitterbug analyze rtts.csv --config config.yaml
```

### Additional commands

```
jitterbug version
jitterbug --help
jitterbug analyze --help
```

### Python API

```
from pathlib import Path

from jitterbug import JitterbugAnalyzer, JitterbugConfig

config = JitterbugConfig()  # or JitterbugConfig.from_file(Path("config.yaml"))
analyzer = JitterbugAnalyzer(config)

results = analyzer.analyze_from_file("examples/network_analysis/data/raw.csv")

for period in results.get_congested_periods():
    print(period.start_timestamp, period.end_timestamp, period.confidence)

summary = analyzer.get_summary_statistics(results)
print(
    f"{summary['congested_periods']} congested periods, "
    f"{summary['congestion_duration_seconds']:.0f} s in total"
)
```

`analyze_from_dataframe(df)` accepts a pandas DataFrame with `epoch` and `values` columns, and `analyze(dataset)` an `RTTDataset` you built yourself.

### Input formats

The contract (columns, units, ordering, what is dropped) is in [docs/INPUT_FORMATS.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/INPUT_FORMATS.md).

**CSV**: one RTT sample per row, epoch seconds and milliseconds.

```
epoch,values
1512144010.0,63.86
1512144010.0,66.52
1512144020.0,85.2
```

**scamper JSON** (`ping` records, one per line):

```
{"type":"ping","src":"192.168.1.1","dst":"8.8.8.8","responses":[{"rtt":1.712,"tx":{"sec":1752855461,"usec":719258}}]}
```

**InfluxDB** (`uv sync --extra influx`):

```
from jitterbug.io import DataLoader

dataset = DataLoader().load_from_influxdb(
    url="http://localhost:8086",
    token="...",
    org="my-org",
    bucket="network-metrics",
    query='from(bucket:"network-metrics") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "rtt")',
)
```

## 🔧 Configuration

Every option lives in a Pydantic model and can be set from a YAML/JSON file or from the CLI flags. `jitterbug config --template` prints the full set; the important ones:

```
analysis_mode: sequential      # sequential | clustering (see docs/CLUSTERING_MODE.md)

change_point_detection:
  algorithm: ruptures          # ruptures | bcp
  threshold: 0.25
  min_time_elapsed: 1800       # seconds between change points
  ruptures_model: rbf
  ruptures_penalty: 10.0

jitter_analysis:
  method: jitter_dispersion    # jitter_dispersion | ks_test
  threshold: 0.25
  moving_average_order: 6
  moving_iqr_order: 4
  significance_level: 0.05

latency_jump:
  threshold: 0.5               # minimum-RTT increase (ms) that counts as a jump

data_processing:
  minimum_interval_minutes: 15 # width of the minimum-RTT intervals
  outlier_detection: true
  outlier_threshold: 3.0

clustering:                    # used when analysis_mode is clustering
  algorithm: gmm               # gmm | kmeans | kmeans_silhouette
  min_period_intervals: 2      # temporal smoothing, in intervals
  min_ks_statistic: 0.1        # smallest KS statistic that counts as a jitter change

output_format: json            # json | csv | parquet (parquet needs pyarrow)
verbose: false
```

Top-level options can also come from environment variables with the `JITTERBUG_` prefix:

```
export JITTERBUG_VERBOSE=true
export JITTERBUG_OUTPUT_FORMAT=csv
```

### How the analysis works

1. **Minimum-RTT intervals**: the raw samples are binned (15 min by default) and the minimum of each bin is kept, which removes most queueing noise.
1. **Change point detection** on the minimum-RTT series marks the boundaries between periods.
1. **Latency jump**: a period is a candidate if its mean minimum RTT rises above the previous period's by more than `latency_jump.threshold`.
1. **Jitter test**: jitter dispersion (variance of the filtered jitter series) or a KS test of the RTT distributions on both sides of the change point.
1. **Congestion inference**: a period is congested when both tests agree; each result carries a confidence and the evidence behind it.

The **clustering mode** (`--mode clustering`) replaces steps 2–3's sequential comparison: intervals are clustered by (minimum RTT, jitter IQR) and each cluster is compared with the lowest-latency one, wherever its intervals fall in time. On the paper dataset the Gaussian mixture recovers all 15 reference congestion periods, plus 2 at the ends of the data where the sequential reference has no verdict. See [docs/CLUSTERING_MODE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/CLUSTERING_MODE.md).

`ruptures` and `bcp` are the two detectors evaluated in the paper; see [docs/ALGORITHM_SELECTION_GUIDE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/ALGORITHM_SELECTION_GUIDE.md) for when to use which.

## 🏗️ Architecture

```
src/jitterbug/
├── analyzer.py             # JitterbugAnalyzer: orchestrates the pipeline below
├── models/                 # Pydantic models
│   ├── rtt_data.py         #   RTTMeasurement, RTTDataset, MinimumRTTDataset
│   ├── analysis.py         #   ChangePoint, LatencyJump, JitterAnalysis, CongestionInference
│   └── config.py           #   JitterbugConfig and the per-stage configs (BaseSettings)
├── detection/              # Change point detection
│   ├── change_point_detector.py   # dispatch on config.algorithm
│   └── algorithms.py       #   RupturesDetector, BayesianChangePointDetector (bcp)
├── analysis/               # Period classification
│   ├── latency_jump_analyzer.py
│   ├── jitter_analyzer.py  #   jitter dispersion and KS test
│   ├── congestion_inference_analyzer.py
│   └── clustering_analyzer.py     # non-sequential mode (GMM / k-means over intervals)
├── streaming/              # Online mode: OnlineJitterbug (Bayesian online change points) and
│                           #   SlidingWindowJitterbug (offline pipeline on a window); replay/score
├── io/                     # DataLoader (CSV, scamper JSON, InfluxDB) and exporters
├── cli/main.py             # Typer CLI: analyze, validate, config, visualize, stream, replay, version
└── visualization/          # JitterbugPlotter (matplotlib): the figures behind `jitterbug visualize`
```

Bundled data: `examples/network_analysis/data/raw.csv` is the PAM 2022 dataset, and `examples/network_analysis/expected_results/` holds the paper's reference output for both jitter methods.

## 🧪 Development

```
# Clone repository
git clone https://github.com/estcarisimo/jitterbug.git
cd jitterbug

# Environment with the dev tools (the `dev` dependency group is installed by default)
uv sync --extra visualization

# Install pre-commit hooks (ruff lint + format on staged files)
uv run pre-commit install
```

### Running tests

```
uv run pytest
uv run pytest --cov=jitterbug --cov-report=term-missing
uv run pytest tests/test_cli.py -v
```

The fast suite runs in about twenty seconds and needs no network; the Bayesian regression tests (`-m slow`) add about two seconds and need the `bcp` extra. Coverage is about 90 % and CI fails below 75 %.

### Code quality

```
uv run ruff check src/ tests/ examples/ tools/
uv run ruff format src/ tests/ examples/ tools/
uv run mypy src/jitterbug
```

`ruff check`, `ruff format --check` and `mypy` are all enforced in CI. See [CONTRIBUTING.md](https://github.com/estcarisimo/jitterbug/blob/main/CONTRIBUTING.md) for the pull request workflow.

### Building

```
uv build
uv pip install dist/*.whl
```

## 📊 Example Output

`jitterbug analyze examples/network_analysis/data/raw.csv` with the default `ruptures` detector:

```
📊 Analysis Summary
┌─────────────────────┬────────────┐
│ Total Periods       │ 22         │
│ Congested Periods   │ 11         │
│ Congestion Ratio    │ 24.44%     │
│ Total Duration      │ 1197000.0s │
│ Congestion Duration │ 292497.0s  │
│ Average Confidence  │ 0.90       │
└─────────────────────┴────────────┘

🔍 Congestion Periods
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃             ┃             ┃          ┃            ┃ Latency    ┃ Jitter      ┃
┃ Start (UTC) ┃ End (UTC)   ┃ Duration ┃ Confidence ┃ Jump       ┃ Change      ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ 2017-12-02  │ 2017-12-02  │ 27000.0s │ 0.90       │ ✓          │ ✓           │
│ 12:00:10    │ 19:30:10    │          │            │            │             │
│ 2017-12-03  │ 2017-12-03  │ 27000.0s │ 0.90       │ ✓          │ ✓           │
│ 11:45:09    │ 19:15:09    │          │            │            │             │
│ 2017-12-04  │ 2017-12-04  │ 18000.0s │ 0.90       │ ✓          │ ✓           │
│ 14:00:10    │ 19:00:10    │          │            │            │             │
│ ...         │ ...         │          │            │            │             │
└─────────────┴─────────────┴──────────┴────────────┴────────────┴─────────────┘
```

The dataset shows the daily congestion episodes the paper analyzes. With the Bayesian detector and the KS test (`--algorithm bcp --method ks_test`) the output matches the reference in `examples/network_analysis/expected_results/`:

Each entry in `results.json` carries the period, the verdict, and the evidence:

```
{
  "start_timestamp": "2017-12-02 12:00:10+00:00",
  "end_timestamp": "2017-12-02 19:30:10+00:00",
  "is_congested": true,
  "confidence": 0.9,
  "latency_jump": {"has_jump": true, "magnitude": 20.62, "threshold": 0.5},
  "jitter_analysis": {"method": "jitter_dispersion", "has_significant_jitter": true, "jitter_metric": 14.55}
}
```

## 🤝 Contributing

Contributions are welcome! Please see the [Contributing Guide](https://github.com/estcarisimo/jitterbug/blob/main/CONTRIBUTING.md) for the development setup, the checks every change must pass, and the pull request workflow.

1. Fork the repository
1. Create a feature branch (`git switch -c feat/amazing-feature`)
1. Commit your changes (`git commit -m 'feat: add amazing feature'`)
1. Push to the branch (`git push origin feat/amazing-feature`)
1. Open a Pull Request

### Project documentation

| Document                                                                                                                  | Contents                                                    |
| ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| [llms.txt](https://estcarisimo.github.io/jitterbug/llms.txt)                                                              | Documentation index for language models                     |
| [CONTRIBUTING.md](https://github.com/estcarisimo/jitterbug/blob/main/CONTRIBUTING.md)                                     | Development setup, workflow, PR expectations                |
| [CHANGELOG.md](https://github.com/estcarisimo/jitterbug/blob/main/CHANGELOG.md)                                           | Release history                                             |
| [AGENTS.md](https://github.com/estcarisimo/jitterbug/blob/main/AGENTS.md)                                                 | Guidance for AI coding agents working in this repo          |
| [SECURITY.md](https://github.com/estcarisimo/jitterbug/blob/main/SECURITY.md)                                             | Vulnerability reporting, and what data this tool handles    |
| [CODE_OF_CONDUCT.md](https://github.com/estcarisimo/jitterbug/blob/main/CODE_OF_CONDUCT.md)                               | Community standards                                         |
| [docs/INSTALLATION.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/INSTALLATION.md)                           | Installation details and troubleshooting                    |
| [docs/INPUT_FORMATS.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/INPUT_FORMATS.md)                         | Input contract: CSV, scamper JSON, DataFrame, InfluxDB      |
| [docs/ALGORITHM_SELECTION_GUIDE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/ALGORITHM_SELECTION_GUIDE.md) | Choosing a change point detector                            |
| [docs/ALGORITHM_USAGE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/ALGORITHM_USAGE.md)                     | Per-detector options and examples                           |
| [docs/VISUALIZATION_USAGE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/VISUALIZATION_USAGE.md)             | `jitterbug visualize`, the five figures, `JitterbugPlotter` |
| [docs/PERFORMANCE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/PERFORMANCE.md)                             | Runtime across releases, and where the speedup comes from   |
| [examples/README.md](https://github.com/estcarisimo/jitterbug/blob/main/examples/README.md)                               | Scripts and notebooks                                       |

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](https://github.com/estcarisimo/jitterbug/blob/main/LICENSE) file for details.

## 🔗 Related Resources

- [Jitterbug paper (PAM 2022)](https://doi.org/10.1007/978-3-030-98785-5_7)
- [bayesian_changepoint_detection](https://github.com/hildensia/bayesian_changepoint_detection) ([docs](https://estcarisimo.github.io/bayesian_changepoint_detection/), PyPI [`bayesian-changepoint`](https://pypi.org/project/bayesian-changepoint/)): the Bayesian detector behind the `bcp` extra. Its 1.2 release made the paper's configuration about 6× faster than Jitterbug 1.0 and more accurate than the version Jitterbug used before; see [docs/PERFORMANCE.md](https://github.com/estcarisimo/jitterbug/blob/main/docs/PERFORMANCE.md#what-changed-in-the-bayesian-library)
- [ruptures](https://centre-borelli.github.io/ruptures-docs/): the default change point library
- [scamper](https://www.caida.org/catalog/software/scamper/): the measurement tool whose JSON output Jitterbug reads

## 🙏 Acknowledgments

Jitterbug is the software behind the following paper. If you use it in your research, please cite it (also available as [CITATION.cff](https://github.com/estcarisimo/jitterbug/blob/main/CITATION.cff) for GitHub's "Cite this repository" button):

**Paper**: *"Jitterbug: A New Framework for Jitter-Based Congestion Inference"* **Authors**: Esteban Carisimo, Ricky K. P. Mok, David D. Clark, and K. C. Claffy **Conference**: Passive and Active Measurement (PAM), March 2022 **Link**: <https://doi.org/10.1007/978-3-030-98785-5_7>

```
@InProceedings{carisimo2022jitterbug,
  author    = {Carisimo, Esteban and Mok, Ricky K. P. and Clark, David D. and Claffy, K. C.},
  title     = {Jitterbug: A New Framework for Jitter-Based Congestion Inference},
  booktitle = {Passive and Active Measurement},
  year      = {2022},
  publisher = {Springer International Publishing},
  pages     = {155--179},
  doi       = {10.1007/978-3-030-98785-5_7}
}
```

Thanks to the authors of `ruptures` and `bayesian_changepoint_detection` for the libraries this framework builds on.

# Installation

The distribution is named `jitterbug-inference` on PyPI (the name `jitterbug` there belongs to an unrelated project); the import is `import jitterbug` and the command is `jitterbug`.

Requirements: Python 3.10 or newer.

## From PyPI

```
pip install jitterbug-inference                            # core
pip install "jitterbug-inference[bcp,visualization]"       # the paper's setup plus figures
uv tool install "jitterbug-inference[bcp]"                  # the CLI in its own environment
```

Pin a version for reproducible runs: `pip install "jitterbug-inference==2.5.0"`.

## From a clone with uv (recommended)

```
curl -LsSf https://astral.sh/uv/install.sh | sh      # once, if you do not have uv
git clone https://github.com/estcarisimo/jitterbug.git
cd jitterbug
uv sync                                               # core: ruptures detector, CLI, library
uv run jitterbug version
```

`uv sync` creates `.venv`, installs the package in editable mode and the `dev` dependency group (pytest, ruff, mypy). Add extras as needed:

| Extra           | Installs                                                                       | Enables                                      |
| --------------- | ------------------------------------------------------------------------------ | -------------------------------------------- |
| `bcp`           | [bayesian-changepoint](https://pypi.org/project/bayesian-changepoint/) + torch | `--algorithm bcp`, the paper's detector      |
| `visualization` | matplotlib                                                                     | `jitterbug visualize`, `JitterbugPlotter`    |
| `clustering`    | scikit-learn                                                                   | `--mode clustering`, the non-sequential mode |
| `influx`        | influxdb-client                                                                | `DataLoader.load_from_influxdb`              |
| `zstd`          | zstandard (Python < 3.14 only)                                                 | `.zst` input and output files                |
| `jupyter`       | JupyterLab, ipykernel                                                          | the notebooks in `examples/`                 |
| `all`           | all of the above                                                               |                                              |

```
uv sync --extra bcp --extra visualization    # the paper's setup plus figures
uv sync --extra all
```

Either activate the environment (`source .venv/bin/activate`) or prefix commands with `uv run`.

## From a clone with pip

```
git clone https://github.com/estcarisimo/jitterbug.git
cd jitterbug
python -m venv .venv && source .venv/bin/activate
pip install -e ".[visualization]"        # extras in brackets, as usual
```

## Directly from GitHub (unreleased changes)

```
pip install "jitterbug-inference[visualization] @ git+https://github.com/estcarisimo/jitterbug.git"
```

Or a tag: `...jitterbug.git@v2.5.0`.

## Notes on the `bcp` extra

- It installs `bayesian-changepoint` (>= 1.2) from PyPI, imported as `bayesian_changepoint_detection`. The older PyPI projects `bayescd` and `bayesian-changepoint-detection` are unmaintained releases of the same library that Jitterbug does not work with. On Linux, uv resolves `torch` from the CPU-only index configured in `pyproject.toml` (`[tool.uv.sources]`), which avoids a multi-gigabyte CUDA download.
- The detector runs on CPU by default (`change_point_detection.bcp_device: cpu`). On Apple Silicon the library would otherwise pick MPS, which is an order of magnitude slower for series of this size. The bundled dataset takes about a second on CPU.
- `--extra bayesian` is a deprecated alias of `--extra bcp` and will go in 3.0.

## Check the installation

```
uv run jitterbug version
uv run jitterbug analyze examples/network_analysis/data/raw.csv --summary-only
uv run jitterbug analyze examples/network_analysis/data/raw.csv --algorithm bcp --method ks_test   # bcp extra
uv run jitterbug visualize examples/network_analysis/data/raw.csv --output-dir plots           # visualization extra
uv run python -c "from jitterbug.detection import get_available_algorithms; print(get_available_algorithms())"
```

`get_available_algorithms()` lists only the detectors whose packages are importable.

## Troubleshooting

| Symptom                                                                           | Cause and fix                                                                       |
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `--algorithm bcp` fails with "bayesian_changepoint_detection package is required" | The extra is not installed: `uv sync --extra bcp`.                                  |
| `bcp` runs for many minutes on a Mac                                              | An older configuration file sets `bcp_device: mps`; use `cpu`.                      |
| `jitterbug visualize` reports that matplotlib is missing                          | `uv sync --extra visualization`.                                                    |
| Plots fail on a server without a display                                          | Set `MPLBACKEND=Agg` before running.                                                |
| `--output-format parquet` fails                                                   | Parquet needs `pyarrow` (`uv pip install pyarrow`).                                 |
| `pip install jitterbug` installed something else                                  | That is the unrelated PyPI project; uninstall it and use one of the commands above. |

## Developer setup

See [CONTRIBUTING.md](https://github.com/estcarisimo/jitterbug/blob/main/CONTRIBUTING.md): `uv sync --extra visualization`, `uv run pre-commit install`, then `uv run pytest -m "not slow"`.

# Input formats

What Jitterbug expects from an RTT time series, how it is read, and what happens to rows that do not fit. The rules are enforced once, when the data is loaded (`DataLoader.load_from_dataframe` for CSV, DataFrames and InfluxDB; the scamper reader applies the same bounds per response), so the analysis code can assume a clean series.

## The contract

One row per RTT sample, from one source to one destination.

| Column        | Required | Type   | Meaning                                                                                                                           |
| ------------- | -------- | ------ | --------------------------------------------------------------------------------------------------------------------------------- |
| `epoch`       | yes      | number | Unix time in **seconds** (UTC). Fractions are fine.                                                                               |
| `values`      | yes      | number | Round-trip time in **milliseconds**. `rtt_value`, `rtt` and `latency` are accepted as synonyms (first match wins, in that order). |
| `source`      | no       | string | Probe or source address; kept as metadata on each sample.                                                                         |
| `destination` | no       | string | Target address; kept as metadata on each sample.                                                                                  |

Other columns are ignored (their names are recorded in `dataset.metadata["original_columns"]`).

Rules, applied in this order:

1. **Missing column or non-numeric value → error.** A missing `epoch` or RTT column, or a cell such as `"12:00"` or `"timeout"` in either, raises `ValueError` naming the column and the first offending value. Fix the file; Jitterbug does not guess.
1. **Missing values are dropped.** A row whose `epoch` or RTT is empty (NaN, `None`, or an empty string) is skipped.
1. **Out-of-range RTTs are dropped.** RTT must be `> 0` and `<= 10 000` ms (`jitterbug.models.MAX_RTT_MS`). Zero or negative values usually encode a timeout or a probe error; values above ten seconds are treated the same way.
1. **Rows are sorted by `epoch`** if they are not already, with a stable sort (ties keep their file order).
1. **Duplicate epochs are kept.** Several samples at the same second are normal for high-rate probing; `jitterbug validate` flags them (and `DataLoader.validate_data` reports `unique_timestamps`).
1. **No rows left → error.**

Every drop and the sort are logged at `WARNING` level and counted in the dataset metadata:

```
dataset.metadata["dropped_rows"]    # {"missing": 2, "non_positive": 0, "too_large": 1}
dataset.metadata["sorted_on_load"]  # True if the rows were reordered
dataset.metadata["rtt_column"]      # which synonym was used
```

Sampling does **not** have to be regular. The first analysis stage takes the minimum RTT per `minimum_interval_minutes` window (15 minutes by default), and change points are detected on that series; gaps and bursts in the raw sampling only affect how many samples fall in each window. `jitterbug validate` prints the average interval and the largest gap so you can judge whether the window is appropriate.

## CSV

```
epoch,values
1512144010.0,63.86
1512144010.0,66.52
1512144020.0,85.2
```

Read with `pandas.read_csv` defaults: a header row, comma separated, `.` decimal point. The bundled `examples/network_analysis/data/raw.csv` (47 163 rows, PAM 2022) is in this form.

```
jitterbug validate rtts.csv --verbose
jitterbug analyze rtts.csv --output results.json
```

## scamper JSON

One JSON object per line, as written by `scamper` with `-O json` (also `.jsonl`). Only `ping` records are used; each response with an `rtt` and a `tx` time becomes one sample, with `src`/`dst` as `source`/`destination`:

```
{"type":"ping","src":"192.168.1.1","dst":"8.8.8.8","responses":[{"rtt":1.712,"tx":{"sec":1752855461,"usec":719258}}]}
```

Records of other types, responses without an `rtt` (timeouts), and lines that are not valid JSON are skipped (invalid lines with a warning). Responses whose `rtt` is outside `(0, 10 000]` ms are dropped with a warning and counted in `metadata["dropped_responses"]`. Samples are sorted by time. A file with no usable response is an error.

## pandas DataFrame

```
from jitterbug import JitterbugAnalyzer, JitterbugConfig

results = JitterbugAnalyzer(JitterbugConfig()).analyze_from_dataframe(df)
```

The frame follows the contract above. CSV and InfluxDB results go through this same path, so anything that can be turned into a frame with `epoch` and `values` columns (Parquet, a database query, a live probe) can be analyzed.

## InfluxDB

With the `influx` extra, `DataLoader.load_from_influxdb(url, token, org, bucket, query)` runs a Flux query and maps `_time` to `epoch` (seconds, UTC, rounded to microseconds) and `_value` (or `rtt`, `latency`, `values`) to the RTT column; the result then follows the DataFrame contract. The token is used only for the connection and is never written to configuration, logs or results.

## Format detection

`load_from_file` picks the reader from the extension (`.csv`; `.json`/`.jsonl`). For any other extension it looks at the first line: a line starting with `{` is scamper JSON, a line containing a comma is a CSV header. Anything else (including binary content) is a `ValueError`; pass `file_format="csv"` or `"json"` explicitly to override.

## Zstandard-compressed files

Large RTT datasets can stay compressed: a file whose name ends in `.zst` is decompressed while it is read, and the extension before it decides the format (`rtts.csv.zst` is CSV, `pings.jsonl.zst` is scamper JSON; `data.zst` falls back to the first line). Results work the same way: `--output results.json.zst` or `results.csv.zst` writes the usual JSON or CSV through Zstandard.

```
zstd rtts.csv                                   # -> rtts.csv.zst
jitterbug analyze rtts.csv.zst --output results.json.zst
zstd -dc results.json.zst | head                # or zstdcat
```

On the bundled PAM 2022 dataset `raw.csv` shrinks from 880 KB to 207 KB. The codec is the standard library's `compression.zstd` on Python 3.14 and later; on older versions install the `zstd` extra (`pip install "jitterbug-inference[zstd]"`), otherwise a `.zst` path raises an `ImportError` that says so. Parquet output compresses its own columns, so `results.parquet.zst` is rejected.

# Change Point Detection Algorithm Selection Guide

This guide helps users select the most appropriate change point detection algorithm for network measurement time series analysis. All algorithms are specifically designed and tuned for RTT (Round-Trip Time) measurements and network congestion detection.

## 📊 Available Algorithms

### 1. Ruptures (Default - Recommended)

**Algorithm**: `ruptures`

**Description**: State-of-the-art change point detection using the ruptures library with multiple kernel functions.

**Strengths**:

- Fast and accurate
- Multiple kernel options (RBF, L1, L2, Normal)
- Well-tested and maintained
- Good performance on various data types
- Configurable penalty parameters

**Best for**:

- Network monitoring in production environments
- Large RTT datasets (>1000 measurements)
- Real-time congestion detection
- When you need reliable, fast results for network analysis

**Configuration**:

```
change_point_detection:
  algorithm: "ruptures"
  threshold: 0.25
  ruptures_model: "rbf"    # Options: "rbf", "l1", "l2", "normal"
  ruptures_penalty: 10.0   # Higher = fewer change points
```

**CLI Usage**:

```
jitterbug analyze data.csv --algorithm ruptures --threshold 0.25
```

### 2. Bayesian Change Point (BCP)

**Algorithm**: `bcp`

**Description**: Classical Bayesian approach using Student-t likelihood and constant priors.

**Strengths**:

- Probabilistic framework
- Good theoretical foundation
- Handles uncertainty well
- Works well with small datasets

**Weaknesses**:

- Slower than ruptures
- Requires external dependency
- Less configurable

**Best for**:

- Network research and analysis
- When you need probabilistic confidence in congestion detection
- Small to medium RTT datasets (\<500 measurements)
- Academic studies requiring theoretical interpretability

**Configuration**:

```
change_point_detection:
  algorithm: "bcp"
  threshold: 0.25
  min_time_elapsed: 1800
```

**CLI Usage**:

```
jitterbug analyze data.csv --algorithm bcp --threshold 0.25
```

## 🎯 Algorithm Selection Matrix

| Criteria             | Ruptures   | BCP        |
| -------------------- | ---------- | ---------- |
| **Speed**            | ⭐⭐⭐⭐⭐ | ⭐⭐⭐     |
| **Accuracy**         | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐   |
| **Ease of Use**      | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐   |
| **Interpretability** | ⭐⭐⭐⭐   | ⭐⭐⭐⭐⭐ |
| **Configurability**  | ⭐⭐⭐⭐⭐ | ⭐⭐⭐     |
| **Memory Usage**     | ⭐⭐⭐⭐   | ⭐⭐⭐⭐⭐ |
| **Dependencies**     | ⭐⭐⭐⭐   | ⭐⭐⭐     |

## 🔍 Detailed Algorithm Comparison

### Performance Characteristics

| Algorithm | Time Complexity | Space Complexity | Min RTT Points |
| --------- | --------------- | ---------------- | -------------- |
| Ruptures  | O(n log n)      | O(n)             | 10             |
| BCP       | O(n²)           | O(n)             | 5              |

### Parameter Sensitivity

#### Ruptures Parameters

- **ruptures_model**:
- `"rbf"`: Radial Basis Function (default, good for most cases)
- `"l1"`: L1 norm (good for sparse changes)
- `"l2"`: L2 norm (good for gradual changes)
- `"normal"`: Normal distribution (good for Gaussian data)
- **ruptures_penalty**: Controls number of change points (1-100, default: 10)

#### BCP Parameters

- **threshold**: Detection sensitivity (0.01-0.5, default: 0.25)
- **min_time_elapsed**: Minimum time between change points (seconds)

## 📈 Use Case Recommendations

### Network Monitoring (Production)

**Recommended**: Ruptures with RBF kernel

```
change_point_detection:
  algorithm: "ruptures"
  ruptures_model: "rbf"
  ruptures_penalty: 10.0
  threshold: 0.25
```

### Research & Analysis

**Recommended**: BCP for interpretability, Ruptures for speed

```
change_point_detection:
  algorithm: "bcp"  # or "ruptures"
  threshold: 0.2    # More sensitive for research
```

### Real-time Applications

**Recommended**: Ruptures with L2 kernel

```
change_point_detection:
  algorithm: "ruptures"
  ruptures_model: "l2"
  ruptures_penalty: 15.0  # Fewer false positives
```

### Noisy Data

**Recommended**: Ruptures with a higher penalty

```
change_point_detection:
  algorithm: "ruptures"
  ruptures_penalty: 20.0
  threshold: 0.3      # Less sensitive
```

## 🛠️ Configuration Examples

### High Sensitivity (Detect More Change Points)

```
change_point_detection:
  algorithm: "ruptures"
  threshold: 0.15
  ruptures_penalty: 5.0
  min_time_elapsed: 900  # 15 minutes
```

### Low Sensitivity (Detect Fewer Change Points)

```
change_point_detection:
  algorithm: "ruptures"
  threshold: 0.35
  ruptures_penalty: 20.0
  min_time_elapsed: 3600  # 1 hour
```

### Balanced Configuration (Default)

```
change_point_detection:
  algorithm: "ruptures"
  threshold: 0.25
  ruptures_penalty: 10.0
  min_time_elapsed: 1800  # 30 minutes
```

## 🎮 Interactive Selection

### Quick Selection Questions

1. **What's your primary goal?**
1. Fast, reliable detection → **Ruptures**
1. Research/interpretability, or reproducing the paper → **BCP**
1. **What's your RTT dataset size?**
1. Small (\<100 measurements) → **BCP**
1. Medium (100-1000 measurements) → **Ruptures**
1. Large (>1000 measurements) → **Ruptures**
1. **What's your tolerance for false positives?**
1. Low (prefer fewer detections) → Higher penalty/threshold
1. High (prefer more detections) → Lower penalty/threshold
1. **What are your computational resources?**
1. Limited → **Ruptures**
1. A few seconds of CPU per series is fine → **BCP**

### Decision Tree

```
RTT Dataset Size?
├── Small (<100 measurements) → BCP
├── Medium (100-1000 measurements)
│   ├── Need interpretability? → BCP
│   └── Need speed? → Ruptures
└── Large (>1000 measurements)
    ├── Reproducing the paper? → BCP (under ten seconds for 1 500 points on CPU)
    └── Production monitoring? → Ruptures
```

## 💡 Best Practices

### General Guidelines

1. **Start with Ruptures**: It's fast, accurate, and well-tested
1. **Use BCP for research**: When you need theoretical grounding
1. **Adjust sensitivity**: Start with defaults, then fine-tune

### Performance Optimization

1. **Large datasets**: Use Ruptures with higher penalty
1. **Real-time**: Use Ruptures with L2 kernel
1. **Batch processing**: Any algorithm works well
1. **Memory constraints**: Use BCP or Ruptures

### Troubleshooting

1. **Too many change points**: Increase penalty/threshold
1. **Too few change points**: Decrease penalty/threshold
1. **Inconsistent results**: Check data quality and preprocessing
1. **Slow performance**: Switch to Ruptures or increase min_time_elapsed

## 📊 Example Configurations for Network Scenarios

### Production Network Monitoring

```
change_point_detection:
  algorithm: "ruptures"
  ruptures_model: "rbf"
  ruptures_penalty: 12.0
  threshold: 0.25
  min_time_elapsed: 1800  # 30 minutes
```

### High-Frequency RTT Analysis

```
change_point_detection:
  algorithm: "ruptures"
  ruptures_model: "l2"
  ruptures_penalty: 8.0
  threshold: 0.2
  min_time_elapsed: 300  # 5 minutes
```

### Network Research Studies

```
change_point_detection:
  algorithm: "bcp"
  threshold: 0.15
  min_time_elapsed: 600  # 10 minutes
```

## 🔬 Advanced Topics

### Custom Algorithm Development

For advanced users who want to implement custom algorithms:

1. Inherit from `BaseChangePointDetector`
1. Implement the `detect()` method
1. Return list of `ChangePoint` objects
1. Register in `ChangePointDetector._create_algorithm()`

### Algorithm Evaluation

`examples/algorithm_benchmark.py` runs both detectors with several configurations on the bundled dataset and writes an HTML report.

This guide should help you select the most appropriate algorithm for your specific use case and data characteristics.

# Jitterbug Algorithm Usage Guide

This guide demonstrates all available ways to run change point detection algorithms with Jitterbug using the example dataset.

## Dataset

All examples use the comprehensive network analysis dataset:

- **File**: `examples/network_analysis/data/raw.csv`
- **Size**: 47,163 RTT measurements
- **Format**: CSV with epoch timestamps and RTT values

## Available Algorithms

Jitterbug supports multiple change point detection algorithms:

1. **Ruptures** (`ruptures`) - Fast and accurate using various models *(included by default)*
1. **Bayesian Change Point** (`bcp`) - The Bayesian detector evaluated in the PAM 2022 paper *(requires the `bcp` extra)*

### Installing Algorithm Dependencies

```
# From a clone (see docs/INSTALLATION.md)
uv sync                    # ruptures only
uv sync --extra bcp        # + the Bayesian detector (bayesian-changepoint, pulls in torch)
uv sync --extra all        # every optional back end
```

## Available Jitter Analysis Methods

1. **Jitter Dispersion** (`jitter_dispersion`) - Analyzes jitter variability changes
1. **Kolmogorov-Smirnov Test** (`ks_test`) - Statistical distribution change detection

______________________________________________________________________

## Command Line Usage

### Basic Usage (Default Settings)

```
# Uses default algorithm (ruptures) and method (jitter_dispersion)
# No additional dependencies required
jitterbug analyze examples/network_analysis/data/raw.csv
```

### Ruptures Algorithm *(No additional dependencies required)*

```
# Basic ruptures with default settings
jitterbug analyze examples/network_analysis/data/raw.csv --algorithm ruptures

# Ruptures with jitter dispersion (default)
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures \
    --method jitter_dispersion

# Ruptures with Kolmogorov-Smirnov test
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures \
    --method ks_test

# Ruptures with custom threshold
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures \
    --threshold 0.15

# Ruptures with high sensitivity
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures \
    --threshold 0.1 \
    --method jitter_dispersion

# Ruptures with low sensitivity
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures \
    --threshold 0.5 \
    --method ks_test
```

### Bayesian Change Point Algorithm *(requires the `bcp` extra)*

```
# First install the extra (from a clone):
uv sync --extra bcp

# Basic Bayesian change point detection
jitterbug analyze examples/network_analysis/data/raw.csv --algorithm bcp

# Bayesian with jitter dispersion
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp \
    --method jitter_dispersion

# Bayesian with Kolmogorov-Smirnov test
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp \
    --method ks_test

# Bayesian with custom threshold
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp \
    --threshold 0.2

# Bayesian with high sensitivity
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp \
    --threshold 0.1 \
    --method jitter_dispersion
```

### All Combinations

```
# Ruptures + Jitter Dispersion (default)
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures --method jitter_dispersion

# Ruptures + KS Test
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures --method ks_test

# Bayesian + Jitter Dispersion
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp --method jitter_dispersion

# Bayesian + KS Test
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp --method ks_test
```

______________________________________________________________________

## Configuration File Usage

### Create Configuration Template

```
# Generate configuration template
jitterbug config --template --output algorithm_config.yaml
```

### Example Configuration Files

#### Ruptures Configuration

```
# ruptures_config.yaml
change_point_detection:
  algorithm: "ruptures"
  threshold: 0.25
  min_time_elapsed: 1800
  ruptures_model: "rbf"
  ruptures_penalty: 10.0

jitter_analysis:
  method: "jitter_dispersion"
  threshold: 0.25
  moving_average_order: 6
  moving_iqr_order: 4

output_format: "json"
verbose: true
```

#### Bayesian Configuration

```
# bayesian_config.yaml
change_point_detection:
  algorithm: "bcp"
  threshold: 0.2
  min_time_elapsed: 1800

jitter_analysis:
  method: "ks_test"
  threshold: 0.25
  significance_level: 0.05

output_format: "json"
verbose: true
```

### Using Configuration Files

```
# Use ruptures configuration
jitterbug analyze examples/network_analysis/data/raw.csv \
    --config ruptures_config.yaml

# Use bayesian configuration
jitterbug analyze examples/network_analysis/data/raw.csv \
    --config bayesian_config.yaml
```

______________________________________________________________________

## Python API Usage

### Basic Usage

```
from jitterbug import JitterbugAnalyzer, JitterbugConfig
from jitterbug.models import ChangePointDetectionConfig, JitterAnalysisConfig

# Default configuration (ruptures + jitter_dispersion)
analyzer = JitterbugAnalyzer(JitterbugConfig())
results = analyzer.analyze_from_file('examples/network_analysis/data/raw.csv')
```

### Ruptures Algorithm

```
# Ruptures with jitter dispersion
config = JitterbugConfig(
    change_point_detection=ChangePointDetectionConfig(
        algorithm="ruptures",
        threshold=0.25,
        ruptures_model="rbf",
        ruptures_penalty=10.0
    ),
    jitter_analysis=JitterAnalysisConfig(
        method="jitter_dispersion",
        threshold=0.25
    )
)

analyzer = JitterbugAnalyzer(config)
results = analyzer.analyze_from_file('examples/network_analysis/data/raw.csv')
```

### Bayesian Algorithm

```
# Bayesian with KS test
config = JitterbugConfig(
    change_point_detection=ChangePointDetectionConfig(
        algorithm="bcp",
        threshold=0.2
    ),
    jitter_analysis=JitterAnalysisConfig(
        method="ks_test",
        significance_level=0.05
    )
)

analyzer = JitterbugAnalyzer(config)
results = analyzer.analyze_from_file('examples/network_analysis/data/raw.csv')
```

### Algorithm Comparison

```
from jitterbug import JitterbugAnalyzer, JitterbugConfig
from jitterbug.models import ChangePointDetectionConfig, JitterAnalysisConfig

# Test all algorithms
algorithms = ['ruptures', 'bcp']
methods = ['jitter_dispersion', 'ks_test']

results = {}

for algorithm in algorithms:
    for method in methods:
        print(f"Testing {algorithm} + {method}...")

        config = JitterbugConfig(
            change_point_detection=ChangePointDetectionConfig(
                algorithm=algorithm,
                threshold=0.25
            ),
            jitter_analysis=JitterAnalysisConfig(
                method=method,
                threshold=0.25
            )
        )

        analyzer = JitterbugAnalyzer(config)
        result = analyzer.analyze_from_file('examples/network_analysis/data/raw.csv')

        results[f"{algorithm}_{method}"] = result

        # Print summary
        summary = analyzer.get_summary_statistics(result)
        print(f"  Congested periods: {summary['congested_periods']}")
        print(f"  Congestion ratio: {summary['congestion_ratio']:.2%}")
        print()
```

______________________________________________________________________

## Output and Saving Results

### Save Results to Different Formats

```
# Save as JSON
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures --output results.json

# Save as CSV
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp --output results.csv

# Save as Parquet
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp --output results.parquet
```

### Verbose Output

```
# Detailed logging
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm ruptures --verbose

# Quiet output
jitterbug analyze examples/network_analysis/data/raw.csv \
    --algorithm bcp --quiet
```

______________________________________________________________________

## Performance Expectations

### Expected Performance for Example Dataset

| Detector | Method            | Runtime (CLI, median) | Peak memory | Periods / congested |
| -------- | ----------------- | --------------------- | ----------- | ------------------- |
| ruptures | KS test           | 1.32 s                | ~255 MiB    | 22 / 11             |
| ruptures | Jitter dispersion | 1.21 s                | ~255 MiB    | 22 / 11             |
| BCP      | KS test           | 2.95 s                | ~470 MiB    | 28 / 14             |
| BCP      | Jitter dispersion | 2.64 s                | ~470 MiB    | 28 / 14             |

Jitterbug 2.3.0 on an Apple M1, Python 3.12, whole `jitterbug analyze` command including start-up, 5 runs. The KS test itself takes about 0.13 s and jitter dispersion 0.03 s; about 1 s of each run is Python start-up and imports (PyTorch for BCP). See [PERFORMANCE.md](https://estcarisimo.github.io/jitterbug/PERFORMANCE/index.md) for the method and for earlier releases.

______________________________________________________________________

## Algorithm Selection Guide

### When to Use Each Algorithm

#### Ruptures

- **Best for**: Fast, accurate detection with good performance
- **Pros**: Fast execution, well-tested, multiple models available
- **Cons**: May miss subtle changes
- **Use when**: You need quick results with good accuracy

#### Bayesian Change Point (BCP)

- **Best for**: Classical statistical approach with uncertainty quantification
- **Pros**: Provides uncertainty estimates, theoretically grounded
- **Cons**: Slower execution, requires more memory
- **Use when**: You need statistical rigor and uncertainty quantification

### Method Selection Guide

#### Jitter Dispersion

- **Best for**: Network congestion detection
- **Pros**: Domain-specific, designed for network measurements
- **Cons**: Less general than statistical tests
- **Use when**: Analyzing network RTT data

#### Kolmogorov-Smirnov Test

- **Best for**: General distribution change detection
- **Pros**: General statistical test, well-established
- **Cons**: May be less sensitive to network-specific patterns
- **Use when**: You want general change detection

______________________________________________________________________

## Troubleshooting

### Common Issues

1. **Algorithm not found**: Install required dependencies

   ```
   uv sync --extra bcp   # the Bayesian detector (bayesian-changepoint + torch)
   ```

1. **Memory issues**: Reduce dataset size or use different algorithm

   ```
   # Use ruptures for lower memory usage
   jitterbug analyze examples/network_analysis/data/raw.csv --algorithm ruptures
   ```

1. **Slow performance**: Use ruptures algorithm for fastest results

   ```
   jitterbug analyze examples/network_analysis/data/raw.csv --algorithm ruptures
   ```

### Getting Help

```
# Show help for analyze command
jitterbug analyze --help

# Show all available options
jitterbug --help
```

# Visualization

Jitterbug ships a small set of matplotlib figures for looking at an analysis: the raw and minimum RTT series, the detected change points, the congestion periods, and summary statistics. They are available from the CLI (`jitterbug visualize`) and from Python (`JitterbugPlotter`).

## Installation

```
uv sync --extra visualization    # adds matplotlib
```

## CLI: `jitterbug visualize`

Runs the same analysis as `jitterbug analyze` and writes five PNG files (300 dpi):

```
jitterbug visualize examples/network_analysis/data/raw.csv --output-dir plots
```

```
✅ 5 plots saved to plots
   • congestion_analysis: jitterbug_congestion_analysis.png
   • change_points: jitterbug_change_points.png
   • confidence_heatmap: jitterbug_confidence_heatmap.png
   • summary_stats: jitterbug_summary_stats.png
   • rtt_timeseries: jitterbug_rtt_timeseries.png

📊 Key Statistics:
   • Total periods: 22
   • Congested periods: 11
   • Congestion ratio: 24.4%
   • Change points: 24
```

Options:

| Option             | Meaning                                                      |
| ------------------ | ------------------------------------------------------------ |
| `--output-dir, -o` | Directory for the PNG files (default `visualization_output`) |
| `--prefix`         | Filename prefix (default `jitterbug`)                        |
| `--algorithm, -a`  | `ruptures` (default) or `bcp`                                |
| `--method, -m`     | `jitter_dispersion` (default) or `ks_test`                   |
| `--threshold, -t`  | Change point detection threshold                             |
| `--config, -c`     | YAML/JSON configuration file, as for `analyze`               |
| `--format, -f`     | Input format (`csv`, `json`); auto-detected if omitted       |

Example with the paper's configuration:

```
jitterbug visualize examples/network_analysis/data/raw.csv \
  --algorithm bcp --method ks_test --output-dir plots_bcp --prefix bcp
```

## The figures

| File                        | Content                                                                                                                                                                                 |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `*_congestion_analysis.png` | Three stacked panels sharing the time axis: raw RTT, minimum RTT per interval, and the congestion verdict per period with congested spans shaded. The figure in the README is this one. |
| `*_change_points.png`       | Minimum RTT series with a vertical line per change point, colored by confidence.                                                                                                        |
| `*_confidence_heatmap.png`  | Confidence of each period's verdict over time.                                                                                                                                          |
| `*_summary_stats.png`       | Counts, congestion ratio, confidence and duration distributions.                                                                                                                        |
| `*_rtt_timeseries.png`      | Raw and minimum RTT series overlaid.                                                                                                                                                    |

Time axes pick their own tick spacing from the span of the data (hours to weeks).

## Python: `JitterbugPlotter`

```
from pathlib import Path

from jitterbug import JitterbugAnalyzer, JitterbugConfig
from jitterbug.visualization import JitterbugPlotter

analyzer = JitterbugAnalyzer(JitterbugConfig())
results = analyzer.analyze_from_file("examples/network_analysis/data/raw.csv")

plotter = JitterbugPlotter()  # JitterbugPlotter(style="ggplot", figsize=(14, 8)) also works

# Everything the CLI writes, in one call
paths = plotter.save_all_plots(
    raw_data=analyzer.raw_data,
    min_rtt_data=analyzer.min_rtt_data,
    results=results,
    change_points=analyzer.change_points or [],
    output_dir=Path("plots"),
    prefix="run1",
)

# Or one figure at a time; each method returns the matplotlib Figure
fig = plotter.plot_congestion_analysis(
    analyzer.raw_data, analyzer.min_rtt_data, results, title="Path A, December 2017"
)
fig.savefig("path_a.png", dpi=300, bbox_inches="tight")
```

Methods, all returning a `matplotlib.figure.Figure` and accepting `title` and `save_path`:

| Method                                                      | Inputs                                                         |
| ----------------------------------------------------------- | -------------------------------------------------------------- |
| `plot_rtt_timeseries(datasets)`                             | `{"label": RTTDataset, ...}`; `show_points=True` draws markers |
| `plot_congestion_analysis(raw_data, min_rtt_data, results)` | the three-panel figure                                         |
| `plot_change_points(dataset, change_points)`                | `MinimumRTTDataset` and `list[ChangePoint]`                    |
| `plot_confidence_heatmap(results)`                          | `CongestionInferenceResult`                                    |
| `plot_summary_statistics(results)`                          | `CongestionInferenceResult`                                    |
| `plot_algorithm_comparison(dataset, algorithm_results)`     | `{"ruptures": [...], "bcp": [...]}` change points per detector |

`analyzer.raw_data`, `analyzer.min_rtt_data` and `analyzer.change_points` are populated by the last `analyze*` call.

### In notebooks

The figures are ordinary matplotlib objects, so `%matplotlib inline` shows them. The two notebooks in `examples/` (`jitter_dispersion_analysis.ipynb`, `kolmogorov_smirnov_analysis.ipynb`) build their own figures with matplotlib directly, which is a good starting point for custom layouts.

### Headless use

Set `MPLBACKEND=Agg` (or `matplotlib.use("Agg")` before importing the plotter) on servers and in tests. The test suite does exactly this in `tests/conftest.py`.

## Comparing detectors

`tools/generate_visualizations.py` runs both detectors on the bundled dataset, writes one congestion analysis figure per detector plus a comparison chart, and regenerates `examples/network_analysis/plots/README.md`. It needs the `bcp` extra and takes a few seconds on CPU (see `bcp_device` in the configuration).

## Troubleshooting

- `matplotlib not found`: `uv sync --extra visualization`.
- Blank or tiny figures in a notebook: call the plotting method after `%matplotlib inline` and do not call `plt.close()` before display.
- Slow on very long series: the plots draw every raw sample; downsample the raw data before plotting or plot only `min_rtt_data`.

# Non-sequential (clustering) mode

The method of the PAM 2022 paper is **sequential**: change point detection splits the minimum-RTT series into periods, and each period is compared with the one right before it. The **clustering** mode drops the time order. Every minimum-RTT interval is described by its latency and its jitter, the intervals are clustered, and each cluster is compared with the uncongested baseline, whether or not their intervals are adjacent in time. A quiet Tuesday morning and a quiet Sunday night end up in the same cluster, and so do the evening peaks of different days.

It needs the `clustering` extra (scikit-learn):

```
pip install "jitterbug-inference[clustering]"      # or: uv sync --extra clustering
```

## How it works

1. **Intervals and features.** The raw RTTs are binned exactly like the sequential mode's minimum-RTT intervals (`data_processing.minimum_interval_minutes`, 15 min by default). Each interval gets two features: its **minimum RTT** and the **interquartile range of its jitter samples** (jitter = difference of consecutive RTTs). Intervals with fewer than two jitter samples are skipped. The features are standardized.

1. **Clustering** with one of three algorithms (`clustering.algorithm`):

   | Algorithm           | Number of clusters                                                                                 |
   | ------------------- | -------------------------------------------------------------------------------------------------- |
   | `gmm` (default)     | Gaussian mixture, 1..`max_clusters` components, chosen by BIC. One component means "no structure". |
   | `kmeans`            | k-means with `n_clusters` clusters (default 2).                                                    |
   | `kmeans_silhouette` | k-means, 2..`max_clusters` clusters, chosen by silhouette score.                                   |

1. **Baseline.** Clusters are numbered by increasing median minimum RTT; cluster `0`, the lowest, is the baseline.

1. **Verdict per cluster**, the paper's two signals against the baseline:

   - *latency jump*: the cluster's median minimum RTT exceeds the baseline's by more than `clustering.latency_threshold`, which defaults to the sequential mode's `latency_jump.threshold` (0.5 ms), **and**
   - *jitter change*: a two-sample Kolmogorov–Smirnov test between the pooled raw jitter samples of the cluster and of the baseline is significant at `jitter_analysis.significance_level` (0.05) **and** its statistic, the largest gap between the two empirical distributions, is at least `clustering.min_ks_statistic` (0.1). The p-value alone is not enough; see *Limitations*.

1. **Periods.** Each interval takes its cluster's verdict. A temporal smoothing (`clustering.min_period_intervals`, default 2) first fills gaps of up to that many non-congested intervals inside congestion, then drops congested runs of up to that many intervals. Consecutive intervals with the same verdict become one period, so the output has the same shape as the sequential mode (`CongestionInferenceResult`), and exporters and plots work unchanged. A gap in the data always ends a period.

## Usage

```
jitterbug analyze examples/network_analysis/data/raw.csv --mode clustering
jitterbug analyze rtts.csv --mode clustering --clustering-algorithm kmeans --output results.json
jitterbug visualize rtts.csv --mode clustering --output-dir plots
```

```
analysis_mode: clustering        # sequential (default) | clustering
clustering:
  algorithm: gmm                 # gmm | kmeans | kmeans_silhouette
  n_clusters: 2                  # kmeans only
  max_clusters: 6                # gmm and kmeans_silhouette
  min_period_intervals: 2        # temporal smoothing; 0 = raw per-interval verdicts
  random_state: 0
  latency_threshold: null        # ms; null = latency_jump.threshold (0.5)
  min_ks_statistic: 0.1          # smallest KS statistic that counts as a jitter change
```

```
from jitterbug import JitterbugAnalyzer, JitterbugConfig

config = JitterbugConfig(analysis_mode="clustering")
config.clustering.algorithm = "gmm"
result = JitterbugAnalyzer(config).analyze_from_file("rtts.csv")

for cluster in result.metadata["clustering"]["clusters"]:
    print(cluster["cluster"], cluster["size"], cluster["latency_jump"], cluster["is_congested"])
```

`result.metadata["clustering"]` holds the algorithm, the number of clusters, the model-selection scores (BIC or silhouette per number of clusters) and, for every cluster, its size, median minimum RTT, median jitter IQR, latency jump, KS statistic and p-value, and verdict. Each period's `latency_jump` and `jitter_analysis` carry the statistics of the cluster that describes it, and the `confidence` of a congested period is the share of its intervals that were congested before smoothing.

## Results on the PAM 2022 dataset

Against the paper's 15 reference congestion periods (`expected_results/kstest_inferences.csv`), with the same overlap criterion as the sequential regression tests:

| Configuration                                   | Periods | Congested | Recovered | Spurious |
| ----------------------------------------------- | ------- | --------- | --------- | -------- |
| `gmm` (BIC → 5 clusters), smoothing 2           | 31      | 16        | **15/15** | 2        |
| `kmeans` (2 clusters), smoothing 2              | 31      | 16        | 14/15     | 2        |
| `kmeans_silhouette` (→ 2 clusters), smoothing 2 | 31      | 16        | 14/15     | 2        |
| `gmm`, no smoothing                             | 61      | 31        | 15/15     | 17       |
| `kmeans`, no smoothing                          | 101     | 51        | 9/15      | 8        |
| *Sequential, BCP + KS test (the paper)*         | *28*    | *14*      | *14/15*   | *0*      |

Each clustering run takes under a second. The first three rows are pinned in `tests/test_paper_regression.py`. The two spurious periods are at the edges of the data; see *Limitations*.

## Limitations

- **The KS p-value does not discriminate; the statistic does.** Clusters pool thousands of jitter samples, so the test is significant for almost any difference: on the paper dataset every non-baseline cluster has p < 1e-49, including one only 0.01 ms above the baseline. Its KS statistic, though, is 0.088, against 0.21–0.32 for the congested clusters, which is why `min_ks_statistic` defaults to 0.1. Any value from 0 to 0.2 gives the same results on this dataset; at 0.25 most of the congestion is lost.
- **The latency threshold is small, but raising it costs recall here.** 0.5 ms suits the sequential comparison of adjacent periods. Here the GMM's cluster 0.86 ms above the baseline (114 intervals on the paper dataset) counts as congested, next to the clear 20+ ms clusters, and it is what recovers the fifteenth reference period: with `clustering.latency_threshold` anywhere from 1 to 10 ms the GMM recovers 14/15 (17 congested periods, still 2 extra), and k-means is unaffected. Raise it if short, shallow episodes are not of interest.
- **Time is ignored until the smoothing step.** Without smoothing a single noisy interval can split a congestion period or create a spurious one, which is why the default window is two intervals (30 minutes).
- **BIC can over-split.** The features are not Gaussian, so the mixture may use several components for one regime (5 on the paper dataset). This is harmless for the verdict as long as the extra components fall on the same side of the latency threshold.
- **"Spurious" periods at the edges of the data.** Both periods counted as spurious on the paper dataset are at its ends: the first 4.5 h (from 12-01 16:00, when the data starts) and the last 8.5 h (until 12-16 16:00, when it ends). The minimum RTT is elevated (20–45 ms over a ~9 ms floor) in both, but the sequential reference cannot label them, because its first period has nothing before it to compare with and its last one has no closing change point. Not being tied to adjacent periods, the clustering mode labels them. They are counted as spurious above only because the reference has no verdict there.
- The sequential mode remains the reference method of the paper and the default.

# Runtime across releases

How long each Jitterbug release takes to analyze the PAM 2022 dataset (`examples/network_analysis/data/raw.csv`, 47 163 RTT samples, 1 440 fifteen-minute intervals), and where the difference comes from.

**In short:** with the Bayesian detector (BCP + KS, the paper's configuration), the current release is **5.9× faster** than 1.0 as each was released (16.6 s → 2.8 s), with the same 28 periods and 14 congested. The gain comes from the Bayesian change point library (`bayesian-changepoint` 1.2), not from Jitterbug's own code: 1.0's code on today's dependencies takes 2.6 s. With ruptures + KS, 2.1 and later are **1.5× faster** than 2.0 (2.05 s → 1.32 s), and that gain is Jitterbug's: the CSV loader went from 0.73 s to 0.11 s. Jitterbug 2.0, with the first PyTorch release of the library, was the slow point: 118 s on the CPU, with a different result (34 periods), and more than an hour on the Apple GPU (MPS), which that library release picked by default; the current release is **about 40× faster** than 2.0 on the CPU. [What changed in the Bayesian library](#what-changed-in-the-bayesian-library) lists the work behind the BCP numbers.

## Setup

- **Machine:** Apple M1 (8 cores), macOS 15.7, Python 3.12.11 for every release, default thread settings (no `*_NUM_THREADS` set). Measured on 2026-09-25.
- **Isolation:** one virtual environment per release and dependency set, built with `uv`.
- **Configurations:** `bcp_ks` (BCP change points, KS test; `jitterbug analyze -a bcp -m ks_test`, or `-c bcp -i ks` in 1.0) and `ruptures_ks` (ruptures, KS test; 2.x only, 1.0 has no ruptures detector), plus `bcp_jd` and `ruptures_jd` with jitter dispersion instead of the KS test (`-m jitter_dispersion`, `-i jd` in 1.0). Default parameters of each release.
- **Dependency sets:**
  - *As released*: dependencies resolved as of the release date (`uv pip install --exclude-newer`), with the BCP implementation the release used: Hildensia's numpy BCP from git for 1.0, the first PyTorch rewrite (BCP 1.0, git) for 2.0, `bayesian-changepoint` 1.2 from PyPI for 2.1 and later.
  - *Current*: every release on the same, current dependencies (NumPy 2.5, pandas 3.0, PyTorch 2.14, `bayesian-changepoint` 1.2.1). This separates changes in Jitterbug's code from changes in its dependencies.
- **Measurements:** *CLI* is the wall-clock time of the command in a fresh process (interpreter start-up, imports, analysis, writing the results file), 5 runs, median. *Stages* is the in-process time of each analysis step (import, load, change points, latency jumps, jitter, inference), 3 runs, median. Before the runs, each environment imports Jitterbug once to compile bytecode and the dataset is read once into the page cache.
- **Device:** every BCP run is on the CPU, except the row marked MPS. Jitterbug 2.0 did not choose a device, and BCP 1.0 then picked Apple's MPS when PyTorch reported it; its CPU rows come from `run --hide-mps`, which makes PyTorch report no MPS device. Since 2.1 Jitterbug passes `bcp_device` (default `cpu`), and `bayesian-changepoint` 1.2 defaults to the CPU as well.
- **Check:** every run's results are counted. All releases give 28 periods / 14 congested with BCP and 22 / 11 with ruptures, with either jitter method, so the timings compare the same analysis. The exception is 2.0 with its own BCP release (34 periods), below.

## Results

### BCP + KS, as released

| Release                  | Median (s) | Min–max (s) | Peak RSS (MiB) | Speedup vs 1.0.0 | Periods / congested |
| ------------------------ | ---------- | ----------- | -------------- | ---------------- | ------------------- |
| 1.0.0                    | 16.62      | 16.52–16.77 | 179            | 1.00×            | 28 / 14             |
| 2.0.0, CPU               | 118        | 111–141     | 372            | 0.14×            | 34 / 14             |
| 2.0.0, MPS (its default) | > 3 600    | –           | –              | < 0.005×         | did not finish      |
| 2.1.0                    | 2.85       | 2.78–8.55   | 470            | 5.83×            | 28 / 14             |
| 2.1.1                    | 2.81       | 2.80–8.81   | 472            | 5.91×            | 28 / 14             |
| 2.2.0                    | 2.82       | 2.76–8.70   | 473            | 5.89×            | 28 / 14             |
| 2.3.0                    | 2.95       | 2.76–8.84   | 470            | 5.64×            | 28 / 14             |

| Release    | import | load  | change points | latency jumps | jitter | other |
| ---------- | ------ | ----- | ------------- | ------------- | ------ | ----- |
| 1.0.0      | 0.61   | 0.013 | **15.51**     | 0.000         | 0.14   | 0.000 |
| 2.0.0, CPU | 0.76   | 0.80  | **110**       | 0.001         | 0.11   | 0.062 |
| 2.1.0      | 0.72   | 0.12  | 0.79          | 0.001         | 0.13   | 0.056 |
| 2.3.0      | 0.78   | 0.12  | 0.80          | 0.001         | 0.13   | 0.056 |

Change point detection is 93% of 1.0's runtime; with `bayesian-changepoint` 1.2 it drops from 15.5 s to 0.8 s (19×). What is left in 2.1+ is mostly fixed cost: the analysis stages add up to about 1.1 s, and the other 1.7 s of the 2.8 s is Python start-up and imports (PyTorch comes with the BCP back end).

Jitterbug 2.0 installed the first PyTorch rewrite of BCP (1.0, July 2025), whose Student-t likelihood looped in Python over every point of every segment, which makes the offline dynamic program cubic in the number of intervals, and ran in float32. On the CPU its change point detection takes 110 s, seven times the NumPy original, and returns 34 periods instead of 28. That release also picked Apple's MPS device when available, and there the same run did not finish within an hour (reported as a lower bound). Both were fixed in the library (next section and [below](#what-changed-in-the-bayesian-library)).

### BCP + KS, current dependencies

| Release | Median (s) | Min–max (s) | Peak RSS (MiB) | vs 1.0.0 | Periods / congested |
| ------- | ---------- | ----------- | -------------- | -------- | ------------------- |
| 1.0.0   | 2.61       | 2.43–2.66   | 410            | 1.00×    | 28 / 14             |
| 2.0.0   | 3.39       | 3.37–9.47   | 474            | 0.77×    | 28 / 14             |
| 2.1.0   | 2.75       | 2.73–8.85   | 471            | 0.95×    | 28 / 14             |
| 2.1.1   | 2.89       | 2.73–8.85   | 473            | 0.90×    | 28 / 14             |
| 2.2.0   | 2.73       | 2.70–8.72   | 471            | 0.96×    | 28 / 14             |
| 2.3.0   | 2.79       | 2.72–8.52   | 469            | 0.94×    | 28 / 14             |

On the same dependencies, Jitterbug's own code is not faster on this path: change point detection takes the same 0.8 s in every release, and 2.x spends about 0.1 s more than 1.0 loading the data into its validated data models (0.12 s against 0.01 s). 2.0 is 0.8 s slower because of its CSV loader (next section).

### ruptures + KS, as released

| Release | Median (s) | Min–max (s) | Peak RSS (MiB) | Speedup vs 2.0.0 | Periods / congested |
| ------- | ---------- | ----------- | -------------- | ---------------- | ------------------- |
| 2.0.0   | 2.05       | 2.01–3.17   | 285            | 1.00×            | 22 / 11             |
| 2.1.0   | 1.34       | 1.30–2.49   | 262            | 1.53×            | 22 / 11             |
| 2.1.1   | 1.33       | 1.31–2.49   | 261            | 1.54×            | 22 / 11             |
| 2.2.0   | 1.34       | 1.30–2.47   | 259            | 1.53×            | 22 / 11             |
| 2.3.0   | 1.32       | 1.31–2.44   | 256            | 1.55×            | 22 / 11             |

| Release | import | load     | change points | latency jumps | jitter | other |
| ------- | ------ | -------- | ------------- | ------------- | ------ | ----- |
| 2.0.0   | 0.62   | **0.73** | 0.058         | 0.001         | 0.12   | 0.060 |
| 2.1.0   | 0.69   | 0.11     | 0.056         | 0.001         | 0.12   | 0.056 |
| 2.3.0   | 0.71   | 0.11     | 0.055         | 0.001         | 0.12   | 0.056 |

The whole difference is loading the CSV (0.73 s → 0.11 s), a Jitterbug change in 2.1; the current-dependency runs show the same 1.5× (1.91 s → 1.31 s). ruptures itself takes 0.06 s, so with this detector the runtime is dominated by start-up and imports.

### Jitter dispersion instead of the KS test

The same picture with the other jitter method. The 2.x runs are 0.1–0.3 s shorter than with the KS test: jitter dispersion takes 0.03 s where the KS test takes 0.13 s. 2.0 with its own BCP release was measured on the CPU only, since on MPS it had already run past an hour with the KS test, and change point detection comes before the jitter method.

| Configuration                  | Release    | Median (s) | Min–max (s) | Speedup | Periods / congested |
| ------------------------------ | ---------- | ---------- | ----------- | ------- | ------------------- |
| BCP + JD, as released          | 1.0.0      | 16.11      | 16.07–16.25 | 1.00×   | 28 / 14             |
|                                | 2.0.0, CPU | 112        | 109–121     | 0.14×   | 34 / 15             |
|                                | 2.1.0      | 2.73       | 2.60–3.19   | 5.90×   | 28 / 14             |
|                                | 2.3.0      | 2.64       | 2.60–2.93   | 6.11×   | 28 / 14             |
| BCP + JD, current dependencies | 1.0.0      | 2.38       | 2.31–2.55   | 1.00×   | 28 / 14             |
|                                | 2.0.0      | 3.27       | 3.23–3.78   | 0.73×   | 28 / 14             |
|                                | 2.3.0      | 2.60       | 2.58–3.09   | 0.91×   | 28 / 14             |
| ruptures + JD, as released     | 2.0.0      | 1.94       | 1.91–1.96   | 1.00×   | 22 / 11             |
|                                | 2.1.0      | 1.20       | 1.18–1.25   | 1.61×   | 22 / 11             |
|                                | 2.3.0      | 1.21       | 1.19–1.25   | 1.60×   | 22 / 11             |

2.1.1 and 2.2.0 are within about 0.1 s of 2.1.0 and 2.3.0 in every configuration; all rows are in the raw data.

## What changed in the Bayesian library

The BCP speedup, and better BCP results, come from work on the Bayesian change point library itself, [hildensia/bayesian_changepoint_detection](https://github.com/hildensia/bayesian_changepoint_detection) (PyPI: [`bayesian-changepoint`](https://pypi.org/project/bayesian-changepoint/), docs: [estcarisimo.github.io/bayesian_changepoint_detection](https://estcarisimo.github.io/bayesian_changepoint_detection/)), which Jitterbug uses through its `bcp` extra. Jitterbug uses the offline detector (Fearnhead 2006) with the Student-t likelihood on the 1 440 minimum-RTT intervals.

| BCP version                              | Used by                                                                                                  | BCP + KS on the paper dataset                                       | Periods / congested |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------- |
| 0.4, NumPy original                      | Jitterbug 1.0                                                                                            | change points 15.5 s, CLI 16.6 s                                    | 28 / 14             |
| 1.0.0, first PyTorch rewrite (July 2025) | Jitterbug 2.0, and `main` until [#31](https://github.com/estcarisimo/jitterbug/pull/31) (September 2026) | change points 110 s, CLI 118 s on the CPU; more than an hour on MPS | 34 / 14             |
| 1.2 (September 2026)                     | Jitterbug 2.1 and later                                                                                  | change points 0.8 s, CLI 2.8 s                                      | 28 / 14             |

From 1.0.0 to 1.2 the change point stage went from 110 s to 0.8 s (**about 140×**) on the same data and CPU. Until September 2026 Jitterbug's lock file pinned a commit one day after the 1.0.0 tag with the same detection code (`2e8b4f6`); the regression test pinned 34 / 14 then. Against the paper's reference output (`examples/network_analysis/expected_results/`, 15 congested intervals), 1.2 recovers 14 with no spurious detections, where 1.0 recovered 13 (Jitterbug [#31](https://github.com/estcarisimo/jitterbug/pull/31)). These are the changes in the library that did it:

| BCP pull request                                                                                                                                                 | What it changed                                                                                                                                                                                                                                                                                                            | Effect on Jitterbug                                                                                                                                                                                    |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [#50](https://github.com/hildensia/bayesian_changepoint_detection/pull/50) (fixes [#47](https://github.com/hildensia/bayesian_changepoint_detection/issues/47))  | Exact closed-form Normal-Gamma segment likelihood from prefix sums, instead of a Python loop over every point of every segment (O(n³) scalar operations) under an approximate predictive; float64 recursion, falling back to the CPU on MPS. 70–155× faster on 250–1 000 points, and it finds changes the old code missed. | CLI from about 136 s to about 8 s when Jitterbug adopted it ([#31](https://github.com/estcarisimo/jitterbug/pull/31)); 34 → 28 periods, 13 → 14 of 15 reference intervals recovered; no more MPS runs. |
| [#76](https://github.com/hildensia/bayesian_changepoint_detection/pull/76)                                                                                       | Full sum over segment ends; `truncate` deprecated, since the inherited truncation rule could drop the dominant term.                                                                                                                                                                                                       | Jitterbug stopped passing `truncate=-40`: same 28 change points, probabilities within 1e-12, same speed.                                                                                               |
| [#96](https://github.com/hildensia/bayesian_changepoint_detection/pull/96) (closes [#55](https://github.com/hildensia/bayesian_changepoint_detection/issues/55)) | Segment statistics exact for data far from zero (errors of up to 44 nats before).                                                                                                                                                                                                                                          | RTTs around 60 ms are data far from zero.                                                                                                                                                              |
| [#100](https://github.com/hildensia/bayesian_changepoint_detection/pull/100)                                                                                     | Inputs keep float64 precision instead of being cast to float32.                                                                                                                                                                                                                                                            | Minimum RTTs reach the detector unrounded.                                                                                                                                                             |
| [#103](https://github.com/hildensia/bayesian_changepoint_detection/pull/103)                                                                                     | CPU by default, accelerators opt-in (the offline detector cannot use Apple's MPS, which lacks float64).                                                                                                                                                                                                                    | The default Jitterbug 2.0 relied on, now CPU; Jitterbug also passes `bcp_device: cpu` since 2.1.                                                                                                       |
| [#106](https://github.com/hildensia/bayesian_changepoint_detection/pull/106)                                                                                     | Changepoint table built in O(J T²) instead of O(T³), with an early stop; up to 19× faster.                                                                                                                                                                                                                                 | Part of the 0.8 s change point stage.                                                                                                                                                                  |
| [#105](https://github.com/hildensia/bayesian_changepoint_detection/pull/105), [#107](https://github.com/hildensia/bayesian_changepoint_detection/pull/107)       | Timings across BCP releases, and detection quality on the Turing Change Point Dataset.                                                                                                                                                                                                                                     | The library-side counterpart of this page.                                                                                                                                                             |

The library's own benchmark ([Performance](https://github.com/hildensia/bayesian_changepoint_detection#-performance)) measures the offline Student-t detector on 1 000 synthetic points at 0.53 s for 1.2.0, 24 s for the NumPy original and 108 s for 1.0.0, which also misses changes there; on the 1 440 RTT intervals here, the same ordering: 0.8 s, 15.5 s and 110 s.

## Limitations

- The MPS result is specific to Apple silicon; on a machine with CUDA, BCP 1.0 would have picked that GPU instead, which was not measured.
- One machine, one dataset. The BCP cost grows faster than linearly with the number of intervals, so longer series widen the gap between BCP implementations.
- The first run in each PyTorch environment is about 6 s slower (the maximum column of the KS tables, which ran first) while macOS loads PyTorch's libraries from disk; the medians are not affected, and the jitter dispersion runs, done later, do not show it.
- Default thread settings, as a user would run it. PyTorch may use several cores; the numpy BCP of 1.0 is single-threaded.
- "As released" resolves dependencies by date on today's Python 3.12, not the exact environment of 2022 (1.0's `requirements.txt` pins NumPy 1.19, which has no wheels for Python 3.12 on Apple silicon). The versions used are listed in `environment.json`.

## Reproducing

The script and the raw measurements are in the repository:

```
uv run python tools/benchmark_versions.py setup --workdir /tmp/jb-bench --deps era current
uv run python tools/benchmark_versions.py run --workdir /tmp/jb-bench --deps era current \
    --repeats 5 --stage-repeats 3 --timeout 3600
uv run python tools/benchmark_versions.py run --workdir /tmp/jb-bench --deps era \
    --releases v2.0.0 --configs bcp_ks bcp_jd --hide-mps --timeout 3600   # the CPU rows
uv run python tools/benchmark_versions.py report --workdir /tmp/jb-bench
```

`examples/network_analysis/benchmarks/` holds `runtime_by_release.csv` (every run and stage; the script writes it as `results.csv` in the work directory) and `environment.json` (machine and package versions) from the measurements above.

# Online mode

Jitterbug's sequential pipeline is retrospective twice over: the offline Bayesian detector sees the whole series, and the verdict for a period needs the *next* change point to close it. The online mode (`jitterbug.streaming`, commands `jitterbug stream` and `jitterbug replay`) runs the same decision rule one RTT sample at a time. Its default back end needs the `bcp` extra; the sliding-window back end runs with any detector. The results on the paper dataset are pinned in `tests/test_paper_regression.py` (`TestOnlineReplay`, `TestOnlineDispersionReplay`, `TestSlidingWindowReplay`); the numbers below come from `jitterbug replay` and the tools described at the end.

## How it works

Two back ends share the same events and the same decision rule; `streaming.backend` picks one. The default, `bocpd`, is the incremental detector described here. The other, `window`, reruns the offline sequential pipeline on a trailing window every few bins and is described under [Back ends compared](#back-ends-compared).

1. **Causal binning.** A minimum-RTT bin (15 min by default) closes when a sample from a later bin arrives.
1. **Online change point detection.** Each closed bin feeds `bayesian_changepoint_detection.streaming.OnlineChangepointDetector` (Adams & MacKay 2007), from the same `bcp` extra the offline detector uses. A change point is declared when the most probable run length drops (`decision: map`, the default). Two fixed-delay rules are available for comparison: `lag` thresholds the posterior that a segment started exactly `lag` bins ago, and `window` thresholds the posterior that it started within the last `lag` bins.
1. **Two-stage verdicts.** The stream start opens the baseline period. When a change point opens a new period, a *provisional* verdict (latency jump against the previous period plus the configured jitter test on the observations so far) is emitted once the open period holds `min_period_samples` jitter samples and, with dispersion, 12 dispersion values (see below). When the next change point closes the period, the *final* verdict is emitted with the whole period: with the KS test it is exactly what the sequential pipeline computes; with dispersion it uses the causal series described below, the offline one delayed by 6 bins. The congestion state carries over between final verdicts as in the sequential mode.

Both jitter methods work online, and `jitter_analysis.method` picks one as in the sequential mode (the config default is `jitter_dispersion`; the results below say which method each row used). The KS test uses consecutive RTT differences and is causal as it stands. Jitter dispersion uses the trailing version of the offline filters: the same moving IQR and moving average, each window ending at the bin it describes instead of being centered on it. The values are the offline ones delayed by 6 bins (1.5 h at the default orders), so the threshold keeps its meaning, but a dispersion value reflects the new period alone only after that delay. A provisional verdict with dispersion therefore waits until the open period holds 12 dispersion values (the delay plus one averaging window, 3 h); with 6 values, 11 of 31 provisional verdicts flipped on the paper dataset, with 12 none.

## Command line

`jitterbug stream` reads `epoch,rtt` lines (seconds, milliseconds; a header line is skipped, so is any line that does not parse) from a file or standard input and prints one JSON object per event: change points, provisional verdicts and final verdicts. `--follow` keeps reading a file as it grows (a half-written line is held until its newline arrives); `--events verdicts` or `--events change-points` filters the output. Samples must arrive in time order; an older sample is dropped. When the input ends (or on Ctrl-C with `--follow`), the open bin is closed and a summary line goes to standard error, so `stream FILE` and `replay FILE` emit the same events.

```
my-probe | jitterbug stream --events verdicts
jitterbug stream rtts.csv --follow --output events.jsonl
```

`jitterbug replay` runs a recorded dataset (any input format `analyze` accepts) through the same pipeline in time order, prints a summary and, with `--reference`, scores the final verdicts against a `starts,ends,congestion` CSV such as the paper's `expected_results`.

```
jitterbug replay examples/network_analysis/data/raw.csv --method ks_test \
  --reference examples/network_analysis/expected_results/kstest_inferences.csv \
  --output events.json
```

Both commands take `--config` and the most common knobs as flags: `--backend`, `--method` (`jitter_dispersion` or `ks_test`), `--decision`, `--hazard-lambda`, `--min-period-samples`, `--min-time-elapsed`.

## Configuration

The online settings live in the `streaming` section of `JitterbugConfig` (`StreamingConfig`). The bin width, latency jump threshold, significance level and device come from the shared sections, so the online and sequential modes make the same decision on the same data.

```
data_processing:
  minimum_interval_minutes: 15     # bin width, shared with the sequential mode
latency_jump:
  threshold: 0.5                   # ms, shared
jitter_analysis:
  method: jitter_dispersion        # jitter_dispersion | ks_test, shared
  threshold: 0.25                  # dispersion rise, ms, shared
  significance_level: 0.05         # KS test, shared
streaming:
  backend: bocpd                   # bocpd (incremental) | window (offline rerun)
  decision: map                    # map | lag | window   (bocpd)
  hazard_lambda: 50                # expected run length, in bins   (bocpd)
  max_run_length: 1000             # run lengths kept by the detector   (bocpd)
  window_hours: 72                 # trailing window   (window)
  rerun_every_bins: 1              # closed bins between reruns   (window)
  stable_runs: 2                   # identical reruns before emitting   (window)
  min_time_elapsed: 3600           # seconds between change points
  min_period_samples: 100          # jitter samples before a provisional verdict
  min_ks_statistic: 0.0            # effect-size guard for the KS test
```

## Python API

```
from jitterbug.models import JitterbugConfig
from jitterbug.streaming import OnlineJitterbug

online = OnlineJitterbug(JitterbugConfig())
for epoch, rtt in stream:  # seconds, milliseconds
    for event in online.push(epoch, rtt):
        if event.kind == "verdict":
            print(event.stage, event.start_epoch, event.is_congested)
```

`jitterbug.streaming.replay(dataset, config)` and `score(events, reference)` are what `jitterbug replay` calls.

## Results on the paper dataset

Replaying `examples/network_analysis/data/raw.csv` in timestamp order with the default online settings (MAP rule, expected run length 50 bins, at least 1 h between change points, 100 jitter samples before a provisional verdict) and the KS test, scored with the metric of `tests/test_paper_regression.py` against the paper's KS reference:

|                                       | Offline pipeline (BCP + KS) | Offline pipeline on growing prefixes                                           | Online mode                          |
| ------------------------------------- | --------------------------- | ------------------------------------------------------------------------------ | ------------------------------------ |
| Periods / congested                   | 28 / 14                     | same boundaries at every cutoff                                                | 31 / 15                              |
| Reference congested periods recovered | 14 of 15                    | 14 of 15                                                                       | 14 of 15                             |
| Spurious congested periods            | 0                           | 0                                                                              | 0                                    |
| Congestion onset first reported       | n/a                         | 1.0 h median, 8.5 h max (2.25 h without the first period)                      | 15 min median, 90 min max            |
| Return to baseline first reported     | n/a                         | 2.0 h median, 13.75 h max                                                      | 2.25 h median, 4.25 h max            |
| Verdict changes                       | n/a                         | 3.9 % for periods under 6 h old, 2.4 % at 6-12 h, 0.3 % at 12-24 h, none older | 0 of 31 provisional verdicts flipped |

Delays are measured from the boundary to the moment it is first reported: the sample that triggers the change point online, or the first hourly cutoff whose prefix run places a boundary within one bin of it. *Onset* is the change point that opens a congested period, *return* the one that closes it. `jitterbug replay` and the tools print these numbers (`onset_delay_min_*` and `return_delay_min_*` in the replay, the "First appearance" block in the prefix experiment). The provisional verdict arrived 11 h before the final one on average and never disagreed with it.

Two findings from the parameter sweep:

- The fixed-delay rules recover 5 to 12 (`lag`) or 5 to 15 (`window`) of the 15 reference periods depending on the setting, but every one of the 45 settings tried produces 3 to 6 spurious congested periods, against 0 for every MAP setting, and they place 5 to 12 (`lag`) or 5 to 14 (`window`) of the 30 reference boundaries within 30 min, against 20 for MAP. The posterior mass of a change spreads over several run lengths, so thresholding one run length, or a short window of them, either misses boundaries or fires on noise.
- The MAP rule refines a boundary a few bins after first reporting it. With 30 min between change points those refinements become duplicate boundaries and tiny periods (38 change points instead of 33); 1 h absorbs them. Thirty jitter samples make the provisional KS p-value noisy (2 flips); 100 samples, about three bins, give none.

Caveats: one path over 15 days, and the metric is agreement with the offline method, not ground truth. The stream's first period is taken as the baseline.

## Back ends compared

The `window` back end is the obvious alternative to a new detector: every `rerun_every_bins` closed bins, run the sequential pipeline (the configured detector and jitter method, so it also works with `ruptures` and with jitter dispersion) on the trailing `window_hours` of samples. A change point is emitted once the detector has placed one within two bins of the same spot in `stable_runs` consecutive reruns; change points come out in time order and are never retracted. Final verdicts form a contiguous chain whose boundaries are the emitted change points: the next one is emitted once the offline pipeline has judged the period at the last emitted boundary identically (same verdict, boundaries within two bins) in `stable_runs` consecutive reruns and the change point that closes it has been emitted. If the detector moves a boundary by a bin or two between reruns, the emitted one stands and the verdict of the offline period that contains it is used; if the window moves past a period before it stabilizes, that period is skipped with a warning and the chain resumes at the next change point. Because change points come out in time order, one that only stabilizes after a later one was emitted is dropped; the chain then waits for the next emitted boundary, up to a window length, before it skips. The window start opens the baseline period, as the stream start does in the incremental back end; once the window has moved past the stream start, anything touching its left edge is ignored. The open period gets a provisional verdict from the same two-period rule with the configured jitter method (trailing filters for dispersion; `jitter_method` on each event says which). On every event, `n_prev` and `n_curr` count the observations the jitter test used: raw jitter samples for the KS test, dispersion values for dispersion. The prefix experiment above showed why this works: the offline boundaries never move and verdicts rarely flip.

BCP on the paper dataset, each row scored against the paper's reference for its jitter method (`jitterbug replay` on the full series; offline, BCP + KS gives 28 periods / 14 congested, 14 of 15 recovered, 0 spurious, and BCP + dispersion the same counts against its own reference):

| Back end                                   | Jitter method       | Periods / congested | Reference periods recovered | Spurious | Boundaries within 30 min | Onset delay, median / max | Return delay, median / max | Provisional flips | Replay wall time |
| ------------------------------------------ | ------------------- | ------------------- | --------------------------- | -------- | ------------------------ | ------------------------- | -------------------------- | ----------------- | ---------------- |
| Incremental Bayesian (`bocpd`, MAP rule)   | KS test             | 31 / 15             | 14 of 15                    | 0        | 20 of 30                 | 15 min / 90 min           | 135 min / 255 min          | 0 of 31           | 2 s              |
| Sliding window, rerun every bin (`window`) | KS test             | 31 / 14             | 14 of 15                    | 0        | 30 of 30                 | 45 min / 120 min          | 113 min / 180 min          | 2 of 31           | 160 s            |
| Sliding window, rerun every 4 bins         | KS test             | 30 / 15             | 14 of 15                    | 1        | 30 of 30                 | 120 min / 225 min         | 180 min / 360 min          | 1 of 30           | 41 s             |
| Incremental Bayesian (`bocpd`, MAP rule)   | dispersion (causal) | 31 / 15             | 14 of 15                    | 0        | 20 of 30                 | 15 min / 90 min           | 135 min / 255 min          | 0 of 31           | 1 s              |
| Sliding window, rerun every 4 bins         | dispersion (causal) | 30 / 15             | 14 of 15                    | 1        | 30 of 30                 | 120 min / 225 min         | 180 min / 360 min          | 1 of 30           | 34 s             |

Both back ends recover the same 14 of 15 reference periods, with either jitter method. The incremental detector reports onsets within one bin at the median (90 min at worst), sooner than the sliding window, whose delay is bounded below by `stable_runs × rerun_every_bins` bins. The sliding window places every reference boundary within 30 min, because it sees the whole window when it decides, but where the detector moves a boundary between reruns it can produce a short spurious period next to a real one (the 4-bin row has one of 1.75 h). Rerunning every bin costs about 0.1 s per bin on this data (BCP on 72 h), cheap for one path and not for thousands; the incremental detector costs a fraction of a millisecond per bin. The table is produced by `tools/compare_online_backends.py`; the 4-bin window row is pinned in `tests/test_paper_regression.py` (`TestSlidingWindowReplay`).

## Reproducing

```
uv sync --extra bcp
uv run jitterbug replay examples/network_analysis/data/raw.csv --method ks_test \
  --reference examples/network_analysis/expected_results/kstest_inferences.csv
uv run jitterbug replay examples/network_analysis/data/raw.csv \
  --reference examples/network_analysis/expected_results/jd_inferences.csv   # dispersion
uv run jitterbug replay examples/network_analysis/data/raw.csv --backend window --method ks_test \
  --reference examples/network_analysis/expected_results/kstest_inferences.csv
uv run python tools/compare_online_backends.py   # the back ends table, about 4 min
uv run python tools/replay_online.py --sweep       # decision rule x lag x hazard x threshold
uv run python tools/replay_online.py --min-time-elapsed 1800   # the 38-change-point figure
uv run python tools/replay_online.py --min-period-samples 30   # the 2-flip figure
uv run python tools/prefix_experiment.py --step-hours 1 --start-hours 24 --output prefix.json
uv run python tools/prefix_experiment.py --from-json prefix.json   # re-score without rerunning
```

`tools/prefix_experiment.py` runs the offline pipeline on growing prefixes of the dataset (about 20 min for hourly cutoffs with BCP) and reports how often, and how long after the fact, its verdicts change, plus the onset and return delays defined above; that is the "rerun on a sliding window" baseline in the table.
# API reference

# Analyzer

## `jitterbug.analyzer.JitterbugAnalyzer`

Main analyzer class that orchestrates the entire Jitterbug analysis pipeline.

This class coordinates data loading, change point detection, jitter analysis, latency jump detection, and final congestion inference.

Parameters:

| Name     | Type              | Description                                              | Default    |
| -------- | ----------------- | -------------------------------------------------------- | ---------- |
| `config` | `JitterbugConfig` | Configuration object containing all analysis parameters. | *required* |

Attributes:

| Name                            | Type                          | Description                       |
| ------------------------------- | ----------------------------- | --------------------------------- |
| `config`                        | `JitterbugConfig`             | Configuration object.             |
| `data_loader`                   | `DataLoader`                  | Data loading component.           |
| `change_point_detector`         | `ChangePointDetector`         | Change point detection component. |
| `jitter_analyzer`               | `JitterAnalyzer`              | Jitter analysis component.        |
| `latency_jump_analyzer`         | `LatencyJumpAnalyzer`         | Latency jump analysis component.  |
| `congestion_inference_analyzer` | `CongestionInferenceAnalyzer` | Congestion inference component.   |

### `analyze_from_file(file_path, file_format=None)`

Analyze RTT data from a file.

Parameters:

| Name          | Type               | Description                                                                                                 | Default    |
| ------------- | ------------------ | ----------------------------------------------------------------------------------------------------------- | ---------- |
| `file_path`   | `Union[str, Path]` | Path to the RTT data file.                                                                                  | *required* |
| `file_format` | `Optional[str]`    | Format of the file ('csv' or 'json'). If None, it is inferred from the extension, then from the first line. | `None`     |

Returns:

| Type                        | Description                                                |
| --------------------------- | ---------------------------------------------------------- |
| `CongestionInferenceResult` | Complete analysis results including congestion inferences. |

### `analyze_from_dataframe(df)`

Analyze RTT data from a pandas DataFrame.

Parameters:

| Name | Type        | Description                                                      | Default    |
| ---- | ----------- | ---------------------------------------------------------------- | ---------- |
| `df` | `DataFrame` | DataFrame containing RTT data with columns 'epoch' and 'values'. | *required* |

Returns:

| Type                        | Description                                                |
| --------------------------- | ---------------------------------------------------------- |
| `CongestionInferenceResult` | Complete analysis results including congestion inferences. |

### `analyze(rtt_data)`

Perform complete Jitterbug analysis on RTT data.

Parameters:

| Name       | Type         | Description              | Default    |
| ---------- | ------------ | ------------------------ | ---------- |
| `rtt_data` | `RTTDataset` | RTT measurement dataset. | *required* |

Returns:

| Type                        | Description                                                |
| --------------------------- | ---------------------------------------------------------- |
| `CongestionInferenceResult` | Complete analysis results including congestion inferences. |

### `save_results(results, output_path, format=None)`

Save analysis results to file.

Parameters:

| Name          | Type                        | Description                                                                                                  | Default    |
| ------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------- |
| `results`     | `CongestionInferenceResult` | Analysis results to save.                                                                                    | *required* |
| `output_path` | `Union[str, Path]`          | Path to save results to. JSON and CSV are compressed with Zstandard when it ends in .zst (results.json.zst). | *required* |
| `format`      | `Optional[str]`             | Output format ('json', 'csv', 'parquet'). If None, uses config default.                                      | `None`     |

Raises:

| Type          | Description                                                         |
| ------------- | ------------------------------------------------------------------- |
| `ValueError`  | If the format is unknown, or Parquet is requested with a .zst path. |
| `ImportError` | If the path ends in .zst and no Zstandard codec is available.       |

### `get_summary_statistics(results)`

Get summary statistics from analysis results.

Parameters:

| Name      | Type                        | Description       | Default    |
| --------- | --------------------------- | ----------------- | ---------- |
| `results` | `CongestionInferenceResult` | Analysis results. | *required* |

Returns:

| Type   | Description         |
| ------ | ------------------- |
| `dict` | Summary statistics. |

# Configuration

## `jitterbug.models.config`

Configuration models using Pydantic for validation and serialization.

### `ChangePointDetectionConfig`

Bases: `BaseModel`

Configuration for change point detection algorithms.

Attributes:

| Name                | Type                         | Description                                       |
| ------------------- | ---------------------------- | ------------------------------------------------- |
| `algorithm`         | `Literal['bcp', 'ruptures']` | Algorithm to use for change point detection.      |
| `threshold`         | `float`                      | Threshold for change point detection sensitivity. |
| `min_time_elapsed`  | `int`                        | Minimum time in seconds between change points.    |
| `max_change_points` | `Optional[int]`              | Maximum number of change points to detect.        |

### `JitterAnalysisConfig`

Bases: `BaseModel`

Configuration for jitter analysis methods.

Attributes:

| Name                   | Type                                      | Description                                    |
| ---------------------- | ----------------------------------------- | ---------------------------------------------- |
| `method`               | `Literal['jitter_dispersion', 'ks_test']` | Method to use for jitter analysis.             |
| `threshold`            | `float`                                   | Threshold for significance testing.            |
| `moving_average_order` | `int`                                     | Order of moving average filter (must be even). |
| `moving_iqr_order`     | `int`                                     | Order of moving IQR filter.                    |
| `significance_level`   | `float`                                   | Statistical significance level for tests.      |

### `LatencyJumpConfig`

Bases: `BaseModel`

Configuration for latency jump detection.

Attributes:

| Name        | Type    | Description                           |
| ----------- | ------- | ------------------------------------- |
| `threshold` | `float` | Threshold for latency jump detection. |

### `DataProcessingConfig`

Bases: `BaseModel`

Configuration for data processing.

Attributes:

| Name                       | Type    | Description                                           |
| -------------------------- | ------- | ----------------------------------------------------- |
| `minimum_interval_minutes` | `int`   | Interval in minutes for computing minimum RTT values. |
| `min_samples_per_interval` | `int`   | Minimum number of samples required per interval.      |
| `outlier_detection`        | `bool`  | Whether to perform outlier detection.                 |
| `outlier_threshold`        | `float` | Z-score threshold for outlier detection.              |

### `ClusteringConfig`

Bases: `BaseModel`

Configuration for the non-sequential (clustering) analysis mode.

Each minimum-RTT interval becomes one point with two features, its minimum RTT and the interquartile range of the jitter samples inside it. The points are clustered without regard to time; the cluster with the lowest median minimum RTT is the baseline, and every other cluster is congested when its median minimum RTT exceeds the baseline by more than the latency threshold and the jitter samples of the two clusters differ: a Kolmogorov-Smirnov test is significant at `jitter_analysis.significance_level` and its statistic is at least `min_ks_statistic`.

Attributes:

| Name                   | Type                                            | Description                                                                                                                                                                                                                                                                                                |
| ---------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `algorithm`            | `Literal['gmm', 'kmeans', 'kmeans_silhouette']` | gmm: Gaussian mixture with the number of components chosen by BIC among 1..max_clusters (one component means no congestion signal). kmeans: k-means with n_clusters clusters. kmeans_silhouette: k-means with the number of clusters in 2..max_clusters that maximizes the silhouette score.               |
| `n_clusters`           | `int`                                           | Number of clusters for kmeans.                                                                                                                                                                                                                                                                             |
| `max_clusters`         | `int`                                           | Largest number of clusters tried by gmm and kmeans_silhouette.                                                                                                                                                                                                                                             |
| `min_period_intervals` | `int`                                           | Temporal smoothing, in intervals: gaps of at most this many non-congested intervals inside congestion are filled, then congested runs of at most this many intervals are dropped. 0 keeps the raw per-interval labels.                                                                                     |
| `random_state`         | `int`                                           | Seed for the clustering algorithms, so results are reproducible.                                                                                                                                                                                                                                           |
| `latency_threshold`    | \`float                                         | None\`                                                                                                                                                                                                                                                                                                     |
| `min_ks_statistic`     | `float`                                         | Smallest Kolmogorov-Smirnov statistic (the largest gap between the two empirical jitter distributions, 0-1) that counts as a jitter change. Clusters pool thousands of samples, so the p-value alone is significant for negligible differences; this bounds the effect size. 0 relies on the p-value only. |

### `StreamingConfig`

Bases: `BaseModel`

Settings of the online (streaming) mode, `jitterbug.streaming.OnlineJitterbug`.

The online mode reuses the sequential decision rule: minimum-RTT bins of `data_processing.minimum_interval_minutes`, the latency jump threshold of `latency_jump.threshold`, the significance level of `jitter_analysis.significance_level` and the device of `change_point_detection.bcp_device`. The fields below are specific to the Bayesian online change point detector and to the two-stage verdicts.

Attributes:

| Name                                   | Type                              | Description                                                                                                                                                                                                                                                                                                                                                                               |
| -------------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backend`                              | `Literal['bocpd', 'window']`      | bocpd (default): incremental Bayesian online change point detection, one bin at a time. window: rerun the offline sequential pipeline on a trailing window of window_hours every rerun_every_bins closed bins and emit a verdict once it has been identical in stable_runs consecutive reruns. The baseline the incremental detector is compared with; heavier per bin, no new algorithm. |
| `window_hours`                         | `float`                           | Length of the trailing window of the window back end, in hours.                                                                                                                                                                                                                                                                                                                           |
| `rerun_every_bins`                     | `int`                             | Closed bins between two reruns of the window back end.                                                                                                                                                                                                                                                                                                                                    |
| `stable_runs`                          | `int`                             | Consecutive reruns in which a change point or a verdict must be identical before the window back end emits it.                                                                                                                                                                                                                                                                            |
| `hazard_lambda`                        | `float`                           | Expected run length in bins (constant hazard 1 / hazard_lambda).                                                                                                                                                                                                                                                                                                                          |
| `decision`                             | `Literal['map', 'lag', 'window']` | Change point rule. map (default) fires when the most probable run length drops; lag thresholds the posterior that a segment started exactly lag bins ago; window thresholds the posterior that it started within the last lag bins and places the change at the most probable run length.                                                                                                 |
| `lag`                                  | `int`                             | Decision delay in bins for the lag and window rules.                                                                                                                                                                                                                                                                                                                                      |
| `threshold`                            | `float`                           | Posterior threshold for the lag and window rules.                                                                                                                                                                                                                                                                                                                                         |
| `max_run_length`                       | `int or None`                     | Run lengths kept by the detector (bounds memory and time per bin); None keeps the exact posterior.                                                                                                                                                                                                                                                                                        |
| `min_time_elapsed`                     | `int`                             | Minimum seconds between change points. The MAP rule refines a boundary a few bins after first reporting it; one hour absorbs those refinements.                                                                                                                                                                                                                                           |
| `prior_alpha, prior_beta, prior_kappa` | `float`                           | Normal-Gamma prior of the Student-t predictive (see online_likelihoods.StudentT).                                                                                                                                                                                                                                                                                                         |
| `prior_mu`                             | `float or None`                   | Prior mean; None uses the first minimum RTT seen.                                                                                                                                                                                                                                                                                                                                         |
| `min_ks_statistic`                     | `float`                           | Smallest Kolmogorov-Smirnov statistic that counts as a jitter change (effect-size guard, 0-1); 0 relies on the p-value only.                                                                                                                                                                                                                                                              |
| `min_period_samples`                   | `int`                             | Jitter samples the open period must hold before a provisional verdict. About three 15-minute bins on the paper dataset; 30 gives noisy p-values. With jitter_analysis.method: jitter_dispersion the verdict also waits for enough causal dispersion values (JitterAnalyzer.causal_dispersion_delay plus moving_average_order bins).                                                       |

### `JitterbugConfig`

Bases: `BaseSettings`

Main configuration class for Jitterbug.

Attributes:

| Name                     | Type                                  | Description                                                                                                                                                                                                                                                                                      |
| ------------------------ | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `analysis_mode`          | `Literal['sequential', 'clustering']` | sequential (default) is the PAM 2022 method: change points split the series and each period is compared with the previous one. clustering groups minimum-RTT intervals by (latency, jitter) regardless of when they occur and compares each cluster with the baseline one; see ClusteringConfig. |
| `change_point_detection` | `ChangePointDetectionConfig`          | Configuration for change point detection.                                                                                                                                                                                                                                                        |
| `jitter_analysis`        | `JitterAnalysisConfig`                | Configuration for jitter analysis.                                                                                                                                                                                                                                                               |
| `latency_jump`           | `LatencyJumpConfig`                   | Configuration for latency jump detection.                                                                                                                                                                                                                                                        |
| `data_processing`        | `DataProcessingConfig`                | Configuration for data processing.                                                                                                                                                                                                                                                               |
| `clustering`             | `ClusteringConfig`                    | Configuration for the clustering mode.                                                                                                                                                                                                                                                           |
| `streaming`              | `StreamingConfig`                     | Configuration for the online (streaming) mode.                                                                                                                                                                                                                                                   |
| `output_format`          | `Literal['json', 'csv', 'parquet']`   | Output format for results.                                                                                                                                                                                                                                                                       |
| `verbose`                | `bool`                                | Whether to enable verbose logging.                                                                                                                                                                                                                                                               |

#### `from_file(config_path)`

Load configuration from file.

Parameters:

| Name          | Type   | Description                                | Default    |
| ------------- | ------ | ------------------------------------------ | ---------- |
| `config_path` | `Path` | Path to configuration file (YAML or JSON). | *required* |

Returns:

| Type              | Description           |
| ----------------- | --------------------- |
| `JitterbugConfig` | Loaded configuration. |

#### `to_file(config_path)`

Save configuration to file.

Parameters:

| Name          | Type   | Description                      | Default    |
| ------------- | ------ | -------------------------------- | ---------- |
| `config_path` | `Path` | Path to save configuration file. | *required* |

# Data models

## `jitterbug.models.rtt_data`

RTT data models using Pydantic for validation and serialization.

### `MAX_RTT_MS = 10000.0`

Largest RTT accepted, in milliseconds (10 s); larger values are treated as bad samples.

### `RTTMeasurement`

Bases: `BaseModel`

Represents a single RTT measurement.

Attributes:

| Name          | Type            | Description                            |
| ------------- | --------------- | -------------------------------------- |
| `timestamp`   | `datetime`      | When the measurement was taken.        |
| `epoch`       | `float`         | Unix timestamp of the measurement.     |
| `rtt_value`   | `float`         | Round-trip time value in milliseconds. |
| `source`      | `Optional[str]` | Source identifier or IP address.       |
| `destination` | `Optional[str]` | Destination identifier or IP address.  |

### `RTTDataset`

Bases: `BaseModel`

Collection of RTT measurements with validation and processing capabilities.

Attributes:

| Name           | Type                   | Description                            |
| -------------- | ---------------------- | -------------------------------------- |
| `measurements` | `List[RTTMeasurement]` | List of RTT measurements.              |
| `metadata`     | `dict`                 | Additional metadata about the dataset. |

#### `validate_measurements_sorted(v)`

Ensure measurements are sorted by timestamp.

#### `to_arrays()`

Convert measurements to numpy arrays.

Returns:

| Type                      | Description                                    |
| ------------------------- | ---------------------------------------------- |
| `tuple[ndarray, ndarray]` | Tuple of (epochs, rtt_values) as numpy arrays. |

#### `to_dataframe()`

Convert measurements to pandas DataFrame.

Returns:

| Type        | Description                                                               |
| ----------- | ------------------------------------------------------------------------- |
| `DataFrame` | DataFrame with columns: timestamp, epoch, rtt_value, source, destination. |

#### `compute_minimum_intervals(interval_minutes=15)`

Compute minimum RTT values over specified intervals.

Parameters:

| Name               | Type  | Description                                      | Default |
| ------------------ | ----- | ------------------------------------------------ | ------- |
| `interval_minutes` | `int` | Interval size in minutes for computing minimums. | `15`    |

Returns:

| Type                | Description                                              |
| ------------------- | -------------------------------------------------------- |
| `MinimumRTTDataset` | Dataset containing minimum RTT values for each interval. |

#### `get_time_range()`

Get the time range of measurements.

Returns:

| Type                        | Description                              |
| --------------------------- | ---------------------------------------- |
| `tuple[datetime, datetime]` | Start and end timestamps of the dataset. |

### `MinimumRTTDataset`

Bases: `RTTDataset`

Dataset containing minimum RTT values computed over intervals.

Attributes:

| Name               | Type                   | Description                                           |
| ------------------ | ---------------------- | ----------------------------------------------------- |
| `measurements`     | `List[RTTMeasurement]` | List of minimum RTT measurements.                     |
| `interval_minutes` | `int`                  | Interval size in minutes used for computing minimums. |

## `jitterbug.models.analysis`

Analysis result models using Pydantic for validation and serialization.

### `ChangePoint`

Bases: `BaseModel`

Represents a detected change point in the time series.

Attributes:

| Name         | Type       | Description                                     |
| ------------ | ---------- | ----------------------------------------------- |
| `timestamp`  | `datetime` | When the change point occurred.                 |
| `epoch`      | `float`    | Unix timestamp of the change point.             |
| `confidence` | `float`    | Confidence score of the change point detection. |
| `algorithm`  | `str`      | Algorithm used to detect the change point.      |

### `LatencyJump`

Bases: `BaseModel`

Represents a detected latency jump between two time periods.

Attributes:

| Name              | Type       | Description                                      |
| ----------------- | ---------- | ------------------------------------------------ |
| `start_timestamp` | `datetime` | Start of the period.                             |
| `end_timestamp`   | `datetime` | End of the period.                               |
| `start_epoch`     | `float`    | Unix timestamp of period start.                  |
| `end_epoch`       | `float`    | Unix timestamp of period end.                    |
| `has_jump`        | `bool`     | Whether a significant latency jump was detected. |
| `magnitude`       | `float`    | Magnitude of the jump in RTT units.              |
| `threshold`       | `float`    | Threshold used for jump detection.               |

### `JitterAnalysis`

Bases: `BaseModel`

Represents results of jitter analysis.

Attributes:

| Name                     | Type                                      | Description                              |
| ------------------------ | ----------------------------------------- | ---------------------------------------- |
| `start_timestamp`        | `datetime`                                | Start of the analysis period.            |
| `end_timestamp`          | `datetime`                                | End of the analysis period.              |
| `start_epoch`            | `float`                                   | Unix timestamp of period start.          |
| `end_epoch`              | `float`                                   | Unix timestamp of period end.            |
| `has_significant_jitter` | `bool`                                    | Whether significant jitter was detected. |
| `jitter_metric`          | `float`                                   | Computed jitter metric value.            |
| `method`                 | `Literal['jitter_dispersion', 'ks_test']` | Method used for jitter analysis.         |
| `threshold`              | `float`                                   | Threshold used for significance testing. |
| `p_value`                | `Optional[float]`                         | P-value if statistical test was used.    |

### `CongestionInference`

Bases: `BaseModel`

Represents the final congestion inference for a time period.

Attributes:

| Name              | Type                       | Description                             |
| ----------------- | -------------------------- | --------------------------------------- |
| `start_timestamp` | `datetime`                 | Start of the congestion period.         |
| `end_timestamp`   | `datetime`                 | End of the congestion period.           |
| `start_epoch`     | `float`                    | Unix timestamp of period start.         |
| `end_epoch`       | `float`                    | Unix timestamp of period end.           |
| `is_congested`    | `bool`                     | Whether congestion was inferred.        |
| `confidence`      | `float`                    | Confidence in the congestion inference. |
| `latency_jump`    | `Optional[LatencyJump]`    | Associated latency jump analysis.       |
| `jitter_analysis` | `Optional[JitterAnalysis]` | Associated jitter analysis.             |

#### `to_dict()`

Convert to dictionary for backward compatibility.

Returns:

| Type   | Description                                     |
| ------ | ----------------------------------------------- |
| `dict` | Dictionary with keys: starts, ends, congestion. |

### `CongestionInferenceResult`

Bases: `BaseModel`

Container for multiple congestion inference results.

Attributes:

| Name         | Type                        | Description                             |
| ------------ | --------------------------- | --------------------------------------- |
| `inferences` | `List[CongestionInference]` | List of congestion inferences.          |
| `metadata`   | `dict`                      | Additional metadata about the analysis. |

#### `to_dataframe()`

Convert to pandas DataFrame for backward compatibility.

Returns:

| Type        | Description                                       |
| ----------- | ------------------------------------------------- |
| `DataFrame` | DataFrame with columns: starts, ends, congestion. |

#### `get_congested_periods()`

Get only the periods identified as congested.

Returns:

| Type                        | Description                |
| --------------------------- | -------------------------- |
| `List[CongestionInference]` | List of congested periods. |

#### `get_total_congestion_duration()`

Get total duration of congestion in seconds.

Returns:

| Type    | Description                           |
| ------- | ------------------------------------- |
| `float` | Total congestion duration in seconds. |

# Change point detection

## `jitterbug.detection.change_point_detector`

Main change point detection interface.

### `ChangePointDetector`

Main interface for change point detection algorithms.

This class provides a unified interface for different change point detection algorithms and handles algorithm selection based on configuration.

Parameters:

| Name     | Type                         | Description                               | Default    |
| -------- | ---------------------------- | ----------------------------------------- | ---------- |
| `config` | `ChangePointDetectionConfig` | Configuration for change point detection. | *required* |

Attributes:

| Name        | Type                         | Description                                |
| ----------- | ---------------------------- | ------------------------------------------ |
| `config`    | `ChangePointDetectionConfig` | Configuration object.                      |
| `algorithm` | `BaseChangePointDetector`    | Selected change point detection algorithm. |

#### `detect(dataset)`

Detect change points in the RTT dataset.

Parameters:

| Name      | Type                | Description                            | Default    |
| --------- | ------------------- | -------------------------------------- | ---------- |
| `dataset` | `MinimumRTTDataset` | Dataset containing minimum RTT values. | *required* |

Returns:

| Type                | Description                     |
| ------------------- | ------------------------------- |
| `List[ChangePoint]` | List of detected change points. |

## `jitterbug.detection.algorithms`

Change point detection algorithm implementations.

### `BaseChangePointDetector`

Bases: `ABC`

Base class for change point detection algorithms.

Parameters:

| Name     | Type                         | Description                               | Default    |
| -------- | ---------------------------- | ----------------------------------------- | ---------- |
| `config` | `ChangePointDetectionConfig` | Configuration for change point detection. | *required* |

#### `detect(dataset)`

Detect change points in the dataset.

Parameters:

| Name      | Type                | Description         | Default    |
| --------- | ------------------- | ------------------- | ---------- |
| `dataset` | `MinimumRTTDataset` | Dataset to analyze. | *required* |

Returns:

| Type                | Description                     |
| ------------------- | ------------------------------- |
| `List[ChangePoint]` | List of detected change points. |

### `RupturesDetector`

Bases: `BaseChangePointDetector`

Change point detection using the ruptures library.

This implementation uses the ruptures library which provides multiple algorithms for change point detection including Pelt, BottomUp, and others.

#### `detect(dataset)`

Detect change points using ruptures library.

Parameters:

| Name      | Type                | Description         | Default    |
| --------- | ------------------- | ------------------- | ---------- |
| `dataset` | `MinimumRTTDataset` | Dataset to analyze. | *required* |

Returns:

| Type                | Description                     |
| ------------------- | ------------------------------- |
| `List[ChangePoint]` | List of detected change points. |

### `BayesianChangePointDetector`

Bases: `BaseChangePointDetector`

Change point detection using Bayesian methods.

This is a modernized version of the original BCP implementation with improved error handling and configuration.

#### `detect(dataset)`

Detect change points using Bayesian change point detection.

Parameters:

| Name      | Type                | Description         | Default    |
| --------- | ------------------- | ------------------- | ---------- |
| `dataset` | `MinimumRTTDataset` | Dataset to analyze. | *required* |

Returns:

| Type                | Description                     |
| ------------------- | ------------------------------- |
| `List[ChangePoint]` | List of detected change points. |

# Latency and jitter analysis

## `jitterbug.analysis.latency_jump_analyzer`

Latency jump analysis implementation.

### `LatencyJumpAnalyzer`

Analyzer for detecting significant latency jumps.

Identifies periods where the baseline latency increases significantly compared to the previous period.

Parameters:

| Name     | Type                | Description                              | Default    |
| -------- | ------------------- | ---------------------------------------- | ---------- |
| `config` | `LatencyJumpConfig` | Configuration for latency jump analysis. | *required* |

#### `analyze(dataset, change_points)`

Analyze latency jumps between change points.

Parameters:

| Name            | Type                | Description                            | Default    |
| --------------- | ------------------- | -------------------------------------- | ---------- |
| `dataset`       | `MinimumRTTDataset` | Dataset containing minimum RTT values. | *required* |
| `change_points` | `List[ChangePoint]` | List of detected change points.        | *required* |

Returns:

| Type                | Description                            |
| ------------------- | -------------------------------------- |
| `List[LatencyJump]` | List of latency jump analysis results. |

#### `get_jump_statistics(latency_jumps)`

Calculate statistics for latency jumps.

Parameters:

| Name            | Type                | Description                   | Default    |
| --------------- | ------------------- | ----------------------------- | ---------- |
| `latency_jumps` | `List[LatencyJump]` | List of latency jump results. | *required* |

Returns:

| Type   | Description                            |
| ------ | -------------------------------------- |
| `dict` | Dictionary containing jump statistics. |

## `jitterbug.analysis.jitter_analyzer`

Jitter analysis implementation.

### `JitterAnalyzer`

Analyzer for jitter-based congestion detection.

Supports two methods:

1. Jitter dispersion analysis
1. Kolmogorov-Smirnov test

Parameters:

| Name     | Type                   | Description                        | Default    |
| -------- | ---------------------- | ---------------------------------- | ---------- |
| `config` | `JitterAnalysisConfig` | Configuration for jitter analysis. | *required* |

#### `causal_dispersion_lag`

Jitter samples consumed before the causal dispersion series yields its first value.

The trailing IQR window holds `2 * moving_iqr_order + 1` jitter samples and the trailing average `moving_average_order` IQR values, so the first dispersion value exists at jitter index `2 * moving_iqr_order + moving_average_order - 1`, and the value at index `t` summarizes jitter samples `t - lag .. t`.

#### `causal_dispersion_delay`

Samples by which the causal dispersion series lags the offline (centered) one.

#### `analyze_jitter_dispersion(dataset, change_points)`

Analyze jitter using dispersion method.

Parameters:

| Name            | Type                | Description                            | Default    |
| --------------- | ------------------- | -------------------------------------- | ---------- |
| `dataset`       | `MinimumRTTDataset` | Dataset containing minimum RTT values. | *required* |
| `change_points` | `List[ChangePoint]` | List of detected change points.        | *required* |

Returns:

| Type                   | Description                      |
| ---------------------- | -------------------------------- |
| `List[JitterAnalysis]` | List of jitter analysis results. |

#### `analyze_ks_test(dataset, change_points)`

Analyze jitter using Kolmogorov-Smirnov test.

Parameters:

| Name            | Type                | Description                          | Default    |
| --------------- | ------------------- | ------------------------------------ | ---------- |
| `dataset`       | `RTTDataset`        | Dataset containing RTT measurements. | *required* |
| `change_points` | `List[ChangePoint]` | List of detected change points.      | *required* |

Returns:

| Type                   | Description                      |
| ---------------------- | -------------------------------- |
| `List[JitterAnalysis]` | List of jitter analysis results. |

#### `compute_causal_jitter_dispersion(epochs, rtt_values)`

Jitter dispersion with trailing filters, for the online mode.

Same filters and window lengths as `_compute_jitter_dispersion` (moving IQR of `2 * moving_iqr_order + 1` samples, then moving average of `moving_average_order` values), but each window ends at the sample it is assigned to instead of being centered on it, so a value never depends on later samples. The values are the offline ones; each is assigned to the last sample of its window instead of the (rounded) middle, so the series is the offline one delayed by `causal_dispersion_delay` samples (6 bins, 1.5 h, by default), and the threshold keeps its meaning.

Parameters:

| Name         | Type      | Description                       | Default    |
| ------------ | --------- | --------------------------------- | ---------- |
| `epochs`     | `ndarray` | Epochs of the minimum-RTT series. | *required* |
| `rtt_values` | `ndarray` | Minimum RTT per bin.              | *required* |

Returns:

| Type                      | Description                                                                                                                                                          |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tuple[ndarray, ndarray]` | (epochs, dispersion): the epoch of the last sample each value depends on, and the dispersion. Empty when fewer than causal_dispersion_lag + 2 samples are available. |

## `jitterbug.analysis.congestion_inference_analyzer`

Congestion inference analysis implementation.

### `CongestionInferenceAnalyzer`

Analyzer for inferring congestion based on latency jumps and jitter analysis.

Combines results from latency jump detection and jitter analysis to make final congestion inferences.

#### `infer(latency_jumps, jitter_analyses)`

Infer congestion periods based on latency jumps and jitter analysis.

Uses the original v1 stateful congestion inference logic:

- Congestion = True when BOTH latency jump AND jitter are detected
- Congestion = False when there's NO latency jump (regardless of jitter)
- Maintains congestion state between periods

Parameters:

| Name              | Type                   | Description                            | Default    |
| ----------------- | ---------------------- | -------------------------------------- | ---------- |
| `latency_jumps`   | `List[LatencyJump]`    | List of latency jump analysis results. | *required* |
| `jitter_analyses` | `List[JitterAnalysis]` | List of jitter analysis results.       | *required* |

Returns:

| Type                        | Description                           |
| --------------------------- | ------------------------------------- |
| `List[CongestionInference]` | List of congestion inference results. |

#### `get_inference_statistics(inferences)`

Calculate statistics for congestion inferences.

Parameters:

| Name         | Type                        | Description                           | Default    |
| ------------ | --------------------------- | ------------------------------------- | ---------- |
| `inferences` | `List[CongestionInference]` | List of congestion inference results. | *required* |

Returns:

| Type   | Description                                 |
| ------ | ------------------------------------------- |
| `dict` | Dictionary containing inference statistics. |

## `jitterbug.analysis.clustering_analyzer`

Non-sequential congestion inference: cluster minimum-RTT intervals by (latency, jitter).

The sequential method (PAM 2022) splits the series at change points and compares each period with the one before it. This mode drops the time order: every minimum-RTT interval is a point with two features (its minimum RTT and the interquartile range of the jitter samples inside it), the points are clustered, and each cluster is compared with the baseline cluster (lowest median minimum RTT), no matter whether their intervals are adjacent in time. A cluster is congested when both signals of the paper hold: its median minimum RTT exceeds the baseline's by more than the latency threshold, and the raw jitter samples of the two clusters differ (a significant Kolmogorov-Smirnov test whose statistic reaches a minimum effect size). Consecutive intervals with the same verdict are merged into periods, after an optional temporal smoothing, so the output has the same shape as the sequential mode.

### `IntervalFeatures`

Minimum-RTT intervals with the features used for clustering.

Attributes:

| Name               | Type            | Description                                                                   |
| ------------------ | --------------- | ----------------------------------------------------------------------------- |
| `starts`           | `ndarray`       | Start of each interval (epoch seconds, UTC), sorted.                          |
| `min_rtt`          | `ndarray`       | Minimum RTT of each interval (ms).                                            |
| `jitter_iqr`       | `ndarray`       | Interquartile range of the jitter samples of each interval (ms).              |
| `jitter`           | `list[ndarray]` | Raw jitter samples (differences of consecutive RTTs) of each interval.        |
| `interval_seconds` | `float`         | Interval length in seconds.                                                   |
| `skipped`          | `int`           | Non-empty intervals left out because they have fewer than two jitter samples. |

### `ClusterSummary`

Statistics and verdict for one cluster.

Attributes:

| Name                | Type    | Description                                                                                         |
| ------------------- | ------- | --------------------------------------------------------------------------------------------------- |
| `cluster`           | `int`   | Cluster index; clusters are numbered by increasing median minimum RTT, so 0 is always the baseline. |
| `size`              | `int`   | Number of intervals in the cluster.                                                                 |
| `median_min_rtt`    | `float` | Median minimum RTT of its intervals (ms).                                                           |
| `median_jitter_iqr` | `float` | Median jitter IQR of its intervals (ms).                                                            |
| `latency_jump`      | `float` | median_min_rtt minus the baseline's (ms).                                                           |
| `ks_statistic`      | \`float | None\`                                                                                              |
| `p_value`           | \`float | None\`                                                                                              |
| `is_congested`      | `bool`  | Whether the cluster is classified as congested.                                                     |

### `ClusteringResult`

Output of :meth:`ClusteringCongestionAnalyzer.analyze`.

Attributes:

| Name                | Type                        | Description                                                                                                                                              |
| ------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `inferences`        | `list[CongestionInference]` | Periods of consecutive intervals with the same verdict, in time order.                                                                                   |
| `clusters`          | `list[ClusterSummary]`      | One summary per cluster, baseline first.                                                                                                                 |
| `labels`            | `ndarray`                   | Cluster of each interval.                                                                                                                                |
| `selection_scores`  | `dict[int, float]`          | Model-selection score per number of clusters tried (BIC for gmm, lower is better; silhouette for kmeans_silhouette, higher is better); empty for kmeans. |
| `intervals`         | `int`                       | Intervals clustered.                                                                                                                                     |
| `intervals_skipped` | `int`                       | Non-empty intervals left out for having fewer than two jitter samples.                                                                                   |

#### `metadata(algorithm)`

Return a JSON-serializable summary for `CongestionInferenceResult.metadata`.

### `ClusteringCongestionAnalyzer`

Infer congestion by clustering minimum-RTT intervals (non-sequential mode).

Parameters:

| Name                 | Type               | Description                                                                                                                                  | Default    |
| -------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `config`             | `ClusteringConfig` | Clustering options.                                                                                                                          | *required* |
| `latency_threshold`  | `float`            | Minimum excess of a cluster's median minimum RTT over the baseline's (ms), from clustering.latency_threshold or else latency_jump.threshold. | *required* |
| `significance_level` | `float`            | Significance level of the KS test, from jitter_analysis.significance_level.                                                                  | *required* |
| `interval_minutes`   | `int`              | Minimum-RTT interval length, from data_processing.minimum_interval_minutes.                                                                  | *required* |

#### `analyze(rtt_data)`

Cluster the intervals of `rtt_data` and classify each cluster.

Parameters:

| Name       | Type         | Description           | Default    |
| ---------- | ------------ | --------------------- | ---------- |
| `rtt_data` | `RTTDataset` | Raw RTT measurements. | *required* |

Returns:

| Type               | Description                                                                                                          |
| ------------------ | -------------------------------------------------------------------------------------------------------------------- |
| `ClusteringResult` | Periods, per-cluster summaries and model-selection scores. With fewer than two usable intervals the result is empty. |

Raises:

| Type          | Description                       |
| ------------- | --------------------------------- |
| `ImportError` | If scikit-learn is not installed. |

### `compute_interval_features(rtt_data, interval_minutes)`

Bin raw RTTs into minimum-RTT intervals and compute the clustering features.

The bins are the same as `RTTDataset.compute_minimum_intervals`. Jitter is the difference between consecutive RTTs, assigned to the interval of the later sample.

Parameters:

| Name               | Type         | Description                 | Default    |
| ------------------ | ------------ | --------------------------- | ---------- |
| `rtt_data`         | `RTTDataset` | Raw RTT measurements.       | *required* |
| `interval_minutes` | `int`        | Interval length in minutes. | *required* |

Returns:

| Type               | Description                                                       |
| ------------------ | ----------------------------------------------------------------- |
| `IntervalFeatures` | Intervals with at least two jitter samples, sorted by start time. |

### `smooth_labels(congested, starts, step, window)`

Remove short gaps and short bursts from per-interval congestion labels.

Within each block of contiguous intervals, runs of at most `window` non-congested intervals that lie between congested ones are relabeled congested; then congested runs of at most `window` intervals are relabeled non-congested. A missing interval (a gap in the data) always ends a block.

Parameters:

| Name        | Type      | Description                                                        | Default    |
| ----------- | --------- | ------------------------------------------------------------------ | ---------- |
| `congested` | `ndarray` | Boolean label per interval, in time order.                         | *required* |
| `starts`    | `ndarray` | Interval start times (epoch seconds), same order.                  | *required* |
| `step`      | `float`   | Interval length in seconds.                                        | *required* |
| `window`    | `int`     | Largest run length to fill or drop; 0 returns a copy of congested. | *required* |

Returns:

| Type      | Description              |
| --------- | ------------------------ |
| `ndarray` | Smoothed boolean labels. |

# Input and output

## `jitterbug.io.data_loader`

Data loading utilities for various input formats.

### `RTT_COLUMNS = ('values', 'rtt_value', 'rtt', 'latency')`

Accepted names for the RTT column, in order of preference. Values are milliseconds.

### `DataLoader`

Data loader for various RTT data formats.

Supports loading from:

- CSV files with epoch timestamps and RTT values
- JSON files from scamper's warts outputs
- InfluxDB query results
- Pandas DataFrames

#### `load_from_file(file_path, file_format=None)`

Load RTT data from a file.

Parameters:

| Name          | Type               | Description                                                                                                                                                                                                            | Default    |
| ------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `file_path`   | `Union[str, Path]` | Path to the data file.                                                                                                                                                                                                 | *required* |
| `file_format` | `Optional[str]`    | Format of the file ('csv' or 'json'). If None, it is inferred from the extension, then from the first line. A trailing .zst means the file is Zstandard-compressed (rtts.csv.zst) and does not count as the extension. | `None`     |

Returns:

| Type         | Description         |
| ------------ | ------------------- |
| `RTTDataset` | Loaded RTT dataset. |

#### `load_from_dataframe(df)`

Load RTT data from a pandas DataFrame.

The input contract (see `docs/INPUT_FORMATS.md`): an `epoch` column with Unix seconds (UTC) and one RTT column named `values`, `rtt_value`, `rtt` or `latency` holding milliseconds; `source` and `destination` are optional. Validation happens here, once, on the whole frame:

- a missing column or a non-numeric value raises `ValueError`;
- rows with a missing epoch or RTT, a non-positive RTT, or an RTT above `MAX_RTT_MS` are dropped with a warning and counted in `metadata["dropped_rows"]`;
- rows are sorted by epoch (stable) if they are not already; duplicate epochs are kept (`validate_data` reports them).

Parameters:

| Name | Type        | Description                    | Default    |
| ---- | ----------- | ------------------------------ | ---------- |
| `df` | `DataFrame` | DataFrame containing RTT data. | *required* |

Returns:

| Type         | Description         |
| ------------ | ------------------- |
| `RTTDataset` | Loaded RTT dataset. |

Raises:

| Type         | Description                                                                       |
| ------------ | --------------------------------------------------------------------------------- |
| `ValueError` | If a required column is missing, a value is not numeric, or no valid rows remain. |

#### `load_from_influxdb(url, token, org, bucket, query)`

Load RTT data directly from InfluxDB.

Parameters:

| Name     | Type  | Description            | Default    |
| -------- | ----- | ---------------------- | ---------- |
| `url`    | `str` | InfluxDB URL.          | *required* |
| `token`  | `str` | InfluxDB token.        | *required* |
| `org`    | `str` | InfluxDB organization. | *required* |
| `bucket` | `str` | InfluxDB bucket.       | *required* |
| `query`  | `str` | Flux query to execute. | *required* |

Returns:

| Type         | Description         |
| ------------ | ------------------- |
| `RTTDataset` | Loaded RTT dataset. |

#### `validate_data(dataset)`

Validate RTT dataset and return quality metrics.

Parameters:

| Name      | Type         | Description          | Default    |
| --------- | ------------ | -------------------- | ---------- |
| `dataset` | `RTTDataset` | Dataset to validate. | *required* |

Returns:

| Type             | Description                             |
| ---------------- | --------------------------------------- |
| `Dict[str, any]` | Validation results and quality metrics. |

## `jitterbug.io.exporters`

Result export utilities.

JSON and CSV outputs are Zstandard-compressed when the path ends in `.zst` (`results.json.zst`); see :mod:`jitterbug.io.compression`.

### `ResultExporter`

Exporter for analysis results to various formats.

#### `export_to_json(results, output_path, pretty=True)`

Export results to JSON format.

Parameters:

| Name          | Type                        | Description                                                           | Default    |
| ------------- | --------------------------- | --------------------------------------------------------------------- | ---------- |
| `results`     | `CongestionInferenceResult` | Analysis results to export.                                           | *required* |
| `output_path` | `Union[str, Path]`          | Path to save JSON file; compressed with Zstandard if it ends in .zst. | *required* |
| `pretty`      | `bool`                      | Whether to format JSON with indentation.                              | `True`     |

#### `export_to_csv(results, output_path)`

Export results to CSV format.

Parameters:

| Name          | Type                        | Description                                                          | Default    |
| ------------- | --------------------------- | -------------------------------------------------------------------- | ---------- |
| `results`     | `CongestionInferenceResult` | Analysis results to export.                                          | *required* |
| `output_path` | `Union[str, Path]`          | Path to save CSV file; compressed with Zstandard if it ends in .zst. | *required* |

#### `export_to_parquet(results, output_path)`

Export results to Parquet format.

Parameters:

| Name          | Type                        | Description                 | Default    |
| ------------- | --------------------------- | --------------------------- | ---------- |
| `results`     | `CongestionInferenceResult` | Analysis results to export. | *required* |
| `output_path` | `Union[str, Path]`          | Path to save Parquet file.  | *required* |

Raises:

| Type         | Description                                                      |
| ------------ | ---------------------------------------------------------------- |
| `ValueError` | If output_path ends in .zst: Parquet compresses its own columns. |

#### `export_summary(results, output_path)`

Export summary statistics to JSON format.

Parameters:

| Name          | Type                        | Description                                                                   | Default    |
| ------------- | --------------------------- | ----------------------------------------------------------------------------- | ---------- |
| `results`     | `CongestionInferenceResult` | Analysis results to export.                                                   | *required* |
| `output_path` | `Union[str, Path]`          | Path to save summary JSON file; compressed with Zstandard if it ends in .zst. | *required* |

### `reject_zstd_parquet(output_path)`

Raise ValueError for `*.zst` Parquet output, which would compress twice.

## `jitterbug.io.compression`

Transparent Zstandard compression for input and output files.

A path whose last suffix is `.zst` (`rtts.csv.zst`, `results.json.zst`) is read and written through a Zstandard stream; any other path is a plain file. The codec is the standard library's `compression.zstd` on Python 3.14 and later, otherwise the `zstandard` package (the `zstd` extra).

### `is_zstd(path)`

Return whether `path` names a Zstandard-compressed file (last suffix `.zst`).

Parameters:

| Name   | Type  | Description | Default    |
| ------ | ----- | ----------- | ---------- |
| `path` | \`str | Path\`      | File path. |

Returns:

| Type   | Description                        |
| ------ | ---------------------------------- |
| `bool` | True for \*.zst, case-insensitive. |

### `inner_suffix(path)`

Return the suffix that describes the content, ignoring a trailing `.zst`.

Parameters:

| Name   | Type  | Description | Default    |
| ------ | ----- | ----------- | ---------- |
| `path` | \`str | Path\`      | File path. |

Returns:

| Type  | Description                                                                                  |
| ----- | -------------------------------------------------------------------------------------------- |
| `str` | Lower-case suffix, e.g. ".csv" for both rtts.csv and rtts.csv.zst; empty when there is none. |

### `open_text(path, mode='r', newline=None)`

Open a text file, through a Zstandard stream when `path` ends in `.zst`.

Parameters:

| Name      | Type  | Description | Default                                                                                                 |
| --------- | ----- | ----------- | ------------------------------------------------------------------------------------------------------- |
| `path`    | \`str | Path\`      | File path.                                                                                              |
| `mode`    | `str` | "r" or "w". | `'r'`                                                                                                   |
| `newline` | \`str | None\`      | Passed to the text layer, as in :func:open. Use "" when handing the stream to the csv module or pandas. |

Returns:

| Type      | Description                               |
| --------- | ----------------------------------------- |
| `IO[str]` | Text stream; use it as a context manager. |

Raises:

| Type          | Description                                               |
| ------------- | --------------------------------------------------------- |
| `ValueError`  | If mode is not "r" or "w".                                |
| `ImportError` | If path ends in .zst and no Zstandard codec is available. |

# Plotting

## `jitterbug.visualization.plotter`

Core plotting functionality for Jitterbug network analysis visualization.

### `JitterbugPlotter`

Main plotting class for Jitterbug network analysis visualization.

This class provides comprehensive plotting capabilities for RTT data, change points, congestion inferences, and analysis results.

#### `plot_rtt_timeseries(datasets, title='RTT Time Series', save_path=None, show_points=False)`

Plot RTT time series data.

Parameters:

| Name          | Type   | Description                              | Default             |
| ------------- | ------ | ---------------------------------------- | ------------------- |
| `datasets`    | `dict` | Dictionary of dataset_name -> RTTDataset | *required*          |
| `title`       | `str`  | Plot title                               | `'RTT Time Series'` |
| `save_path`   | `Path` | Path to save the plot                    | `None`              |
| `show_points` | `bool` | Whether to show individual data points   | `False`             |

Returns:

| Type     | Description        |
| -------- | ------------------ |
| `Figure` | The created figure |

#### `plot_congestion_analysis(raw_data, min_rtt_data, results, title='Congestion Analysis', save_path=None)`

Plot comprehensive congestion analysis (similar to notebooks).

Parameters:

| Name           | Type                        | Description           | Default                 |
| -------------- | --------------------------- | --------------------- | ----------------------- |
| `raw_data`     | `RTTDataset`                | Raw RTT measurements  | *required*              |
| `min_rtt_data` | `MinimumRTTDataset`         | Minimum RTT data      | *required*              |
| `results`      | `CongestionInferenceResult` | Analysis results      | *required*              |
| `title`        | `str`                       | Plot title            | `'Congestion Analysis'` |
| `save_path`    | `Path`                      | Path to save the plot | `None`                  |

Returns:

| Type     | Description        |
| -------- | ------------------ |
| `Figure` | The created figure |

#### `plot_change_points(dataset, change_points, title='Change Point Detection', save_path=None)`

Plot change points overlaid on RTT data.

Parameters:

| Name            | Type                | Description            | Default                    |
| --------------- | ------------------- | ---------------------- | -------------------------- |
| `dataset`       | `MinimumRTTDataset` | RTT dataset            | *required*                 |
| `change_points` | `List[ChangePoint]` | Detected change points | *required*                 |
| `title`         | `str`               | Plot title             | `'Change Point Detection'` |
| `save_path`     | `Path`              | Path to save the plot  | `None`                     |

Returns:

| Type     | Description        |
| -------- | ------------------ |
| `Figure` | The created figure |

#### `plot_confidence_heatmap(results, title='Confidence Heatmap', save_path=None)`

Plot confidence scores as a heatmap.

Parameters:

| Name        | Type                        | Description           | Default                |
| ----------- | --------------------------- | --------------------- | ---------------------- |
| `results`   | `CongestionInferenceResult` | Analysis results      | *required*             |
| `title`     | `str`                       | Plot title            | `'Confidence Heatmap'` |
| `save_path` | `Path`                      | Path to save the plot | `None`                 |

Returns:

| Type     | Description        |
| -------- | ------------------ |
| `Figure` | The created figure |

#### `plot_algorithm_comparison(dataset, algorithm_results, title='Algorithm Comparison', save_path=None)`

Compare different algorithms' change point detection.

Parameters:

| Name                | Type                           | Description                       | Default                  |
| ------------------- | ------------------------------ | --------------------------------- | ------------------------ |
| `dataset`           | `MinimumRTTDataset`            | RTT dataset                       | *required*               |
| `algorithm_results` | `Dict[str, List[ChangePoint]]` | Results from different algorithms | *required*               |
| `title`             | `str`                          | Plot title                        | `'Algorithm Comparison'` |
| `save_path`         | `Path`                         | Path to save the plot             | `None`                   |

Returns:

| Type     | Description        |
| -------- | ------------------ |
| `Figure` | The created figure |

#### `plot_summary_statistics(results, title='Analysis Summary', save_path=None)`

Plot summary statistics and metrics.

Parameters:

| Name        | Type                        | Description           | Default              |
| ----------- | --------------------------- | --------------------- | -------------------- |
| `results`   | `CongestionInferenceResult` | Analysis results      | *required*           |
| `title`     | `str`                       | Plot title            | `'Analysis Summary'` |
| `save_path` | `Path`                      | Path to save the plot | `None`               |

Returns:

| Type     | Description        |
| -------- | ------------------ |
| `Figure` | The created figure |

#### `save_all_plots(raw_data, min_rtt_data, results, change_points, output_dir, prefix='jitterbug')`

Generate and save all standard plots.

Parameters:

| Name            | Type                        | Description            | Default       |
| --------------- | --------------------------- | ---------------------- | ------------- |
| `raw_data`      | `RTTDataset`                | Raw RTT measurements   | *required*    |
| `min_rtt_data`  | `MinimumRTTDataset`         | Minimum RTT data       | *required*    |
| `results`       | `CongestionInferenceResult` | Analysis results       | *required*    |
| `change_points` | `List[ChangePoint]`         | Detected change points | *required*    |
| `output_dir`    | `Path`                      | Output directory       | *required*    |
| `prefix`        | `str`                       | Filename prefix        | `'jitterbug'` |

Returns:

| Type              | Description                          |
| ----------------- | ------------------------------------ |
| `Dict[str, Path]` | Dictionary of plot_name -> file_path |

# Online mode

## `jitterbug.streaming.online_analyzer`

Online (streaming) Jitterbug pipeline.

The sequential pipeline is retrospective twice over: the offline Bayesian detector sees the whole series, and the verdict for period `i` needs change point `i + 1` to close the period. This module runs the same decision rule one RTT sample at a time:

1. Minimum-RTT bins close when a sample from a later bin arrives (causal binning).
1. Each closed bin feeds a Bayesian online change point detector (Adams & MacKay 2007, `bayesian_changepoint_detection.streaming.OnlineChangepointDetector`). A change point is declared when the MAP run length drops (default), or when the posterior that a segment started `lag` bins ago (`lag`) or within the last `lag` bins (`window`) exceeds a threshold. On the paper dataset the MAP rule recovers 14 of the 15 reference congestion periods with no spurious one, like the offline pipeline, and reports congestion onsets one bin (15 min) after they start. The fixed-delay rules recover 5 to 12 (`lag`) or 5 to 15 (`window`) of the 15 depending on the setting but always with 3 to 6 spurious congested periods, because the posterior mass of a change spreads over several run lengths (see `docs/ONLINE_MODE.md`).
1. The stream start opens the baseline period. When a change point opens a new period, a *provisional* verdict is emitted as soon as the open period holds `min_period_samples` jitter samples. When the next change point closes the period, the *final* verdict is emitted with the full period, which is exactly what the sequential pipeline computes.

Both jitter methods work online. The Kolmogorov-Smirnov test uses consecutive RTT differences and is causal as it stands. Jitter dispersion uses the trailing version of the offline filters (`JitterAnalyzer.compute_causal_jitter_dispersion`): same window lengths, each ending at the bin it describes, so its series is the offline one delayed by `JitterAnalyzer.causal_dispersion_delay` bins (6 bins, 1.5 h, by default). A dispersion value reflects the new period alone only after that delay, so a provisional verdict with dispersion waits until the open period holds `delay + moving_average_order` dispersion values (12 bins, 3 h, by default); with fewer, the provisional verdict is still dominated by the previous period's jitter (11 of 31 flipped on the paper dataset with 6, none with 12).

Settings live in `JitterbugConfig.streaming` (`models.config.StreamingConfig`); the bin width, latency jump threshold, significance level and device come from the shared sections of `JitterbugConfig`.

### `StreamingEvent`

Something the online pipeline decided while consuming the stream.

#### `emitted_at`

Epoch of the sample whose arrival triggered the event.

#### `start_epoch`

Change point epoch, or start of the period a verdict is about.

#### `jitter_method = None`

Jitter test behind `has_jitter`; the KS fields are set for `ks_test` only.

#### `jitter_metric = None`

The jitter test's metric: the KS statistic, or the dispersion change.

#### `delay`

Seconds between the event's reference time and its emission.

#### `to_dict()`

Return the event as a plain dictionary (JSON-serializable).

### `OnlineJitterbug`

Streaming congestion inference: feed RTT samples, get events.

Samples must arrive in non-decreasing epoch order; a sample earlier than the last one accepted is dropped (logged at DEBUG), whatever bin it falls in. Bins are indexed by the samples that arrive, so a gap in the stream does not produce empty bins: the detector sees the bins before and after a gap as consecutive, as the offline pipeline does after `dropna` in `RTTDataset.compute_minimum_intervals`.

Parameters:

| Name     | Type              | Description                                                              | Default |
| -------- | ----------------- | ------------------------------------------------------------------------ | ------- |
| `config` | `JitterbugConfig` | Full configuration; config.streaming holds the online-specific settings. | `None`  |

#### `push(epoch, rtt)`

Consume one RTT sample and return the events it triggered.

Parameters:

| Name    | Type    | Description                            | Default    |
| ------- | ------- | -------------------------------------- | ---------- |
| `epoch` | `float` | Unix timestamp of the sample, seconds. | *required* |
| `rtt`   | `float` | Round-trip time in milliseconds.       | *required* |

#### `flush()`

Close the open bin at the end of a finite stream.

### `verdict_event(verdict, stage, emitted_at, start_epoch, end_epoch)`

Wrap a `PeriodVerdict` as a verdict event.

## `jitterbug.streaming.window_backend`

Sliding-window back end: rerun the offline pipeline as the stream grows.

The baseline the incremental detector is measured against. Every `rerun_every_bins` closed minimum-RTT bins, the sequential pipeline (`JitterbugAnalyzer`, with the configured detector and jitter method) runs on the trailing `window_hours` of raw samples. A change point is emitted once the detector has placed one within `TOLERANCE_BINS` of the same bin in `stable_runs` consecutive reruns; change points are emitted in time order and never retracted. Final verdicts form a contiguous chain: the next one is emitted once the offline pipeline has judged the period at the last emitted boundary identically (same verdict, boundaries within the tolerance) in `stable_runs` consecutive reruns and the change point that closes it has been emitted; its boundaries are those emitted change points, so change points and verdicts never disagree. If the window moves past a period before it stabilizes, that period is skipped with a warning and the chain resumes at the next emitted change point. A change point that only stabilizes after a later one was emitted is dropped (time order is kept), which can leave the chain waiting for the next emitted boundary for up to a window length. The open period after the last emitted change point gets a provisional verdict from the shared two-period rule, with the configured jitter method (trailing filters for dispersion), once it holds `min_period_samples` jitter samples, as in the incremental back end.

The window start opens the baseline period, as the stream start does in the incremental back end: when the detector reports no change point within one bin of the first bin, one is inserted there, so the first period after the window start can be judged (the sequential pipeline needs three change points per verdict). Once the window has moved past the stream start, change points and periods touching its left edge are ignored: they are cut by the window and were emitted long before.

### `TOLERANCE_BINS = 2`

Two boundaries this many bins apart or closer are the same boundary (30 min at 15-min bins).

The offline detector moves a change point by a bin or two as the window slides; the sequential pipeline keeps change points at least `change_point_detection.min_time_elapsed` apart (two bins by default), so a larger tolerance would merge real boundaries.

### `SlidingWindowJitterbug`

Streaming congestion inference by rerunning the offline pipeline on a window.

Same interface as `OnlineJitterbug`: `push(epoch, rtt)` returns the events the sample triggered, `flush()` closes the open bin and reruns once, `events` holds everything emitted. Samples must arrive in non-decreasing epoch order.

Parameters:

| Name     | Type              | Description                                                                                                                                                                                               | Default |
| -------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `config` | `JitterbugConfig` | streaming.window_hours, streaming.rerun_every_bins and streaming.stable_runs drive the back end; the detector and jitter method are those of the sequential mode (analysis_mode is forced to sequential). | `None`  |

#### `push(epoch, rtt)`

Consume one RTT sample (seconds, milliseconds) and return the events it triggered.

#### `flush()`

Close the open bin and rerun once, whatever the cadence.

## `jitterbug.streaming.verdict`

The two-period decision rule shared by the online back ends.

Given the minimum-RTT bins and the jitter observations of a period and of the period before it, decide whether the period is congested the way the sequential pipeline does: a latency jump is a rise of the mean minimum RTT above `latency_jump.threshold`; a jitter change is, with `jitter_analysis.method: ks_test`, a significant Kolmogorov-Smirnov test on the raw jitter samples with a statistic of at least `streaming.min_ks_statistic`, and with `jitter_dispersion` a rise of the mean dispersion (trailing filters, see `JitterAnalyzer.compute_causal_jitter_dispersion`) above `jitter_analysis.threshold`; the congestion state carries over: both → congested, no jump → not congested, jump without jitter change → unchanged.

### `PeriodVerdict`

Outcome of `two_period_verdict`.

### `two_period_verdict(prev_bins, cur_bins, prev_jitter, cur_jitter, previous_state, config)`

Classify a period against the one before it.

Parameters:

| Name             | Type              | Description                                                                                                                                               | Default    |
| ---------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `prev_bins`      | `ndarray`         | Minimum RTT per bin of the previous and the current period (ms).                                                                                          | *required* |
| `cur_bins`       | `ndarray`         | Minimum RTT per bin of the previous and the current period (ms).                                                                                          | *required* |
| `prev_jitter`    | `ndarray`         | Jitter observations of the previous and the current period: consecutive RTT differences (ms) for ks_test, causal dispersion values for jitter_dispersion. | *required* |
| `cur_jitter`     | `ndarray`         | Jitter observations of the previous and the current period: consecutive RTT differences (ms) for ks_test, causal dispersion values for jitter_dispersion. | *required* |
| `previous_state` | `bool`            | Congestion state after the previous period's final verdict.                                                                                               | *required* |
| `config`         | `JitterbugConfig` | latency_jump.threshold, jitter_analysis.method, jitter_analysis.threshold, jitter_analysis.significance_level and streaming.min_ks_statistic are used.    | *required* |

Returns:

| Type            | Description                                                                                                                                 |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `PeriodVerdict` | The decision and the statistics behind it. Tests that could not run (too few observations) leave their fields None and the state unchanged. |

## `jitterbug.streaming.replay`

Replay a finite RTT dataset through the online pipeline and score the result.

`replay` feeds every sample of an `RTTDataset` to the configured online back end in timestamp order and returns the events. `score` summarizes them: change points, periods, detection delays, provisional-to-final agreement and, when a reference file is given, agreement with it using the metric of `tests/test_paper_regression.py` (a reference congested period counts as recovered when one of ours overlaps more than half of it) and how many reference period boundaries have a change point within 30 min.

Delays are measured from the change point to the sample that reported it. *Onset* is the change point that opens a congested final period, *return* the one that closes it.

### `replay(dataset, config=None)`

Feed `dataset` sample by sample to the online pipeline.

Parameters:

| Name      | Type              | Description                                                                                                  | Default    |
| --------- | ----------------- | ------------------------------------------------------------------------------------------------------------ | ---------- |
| `dataset` | `RTTDataset`      | Samples to replay; they are sorted by epoch (stable) before being fed.                                       | *required* |
| `config`  | `JitterbugConfig` | Configuration; config.streaming holds the online-specific settings, including which back end runs (backend). | `None`     |

Returns:

| Type                   | Description                                                                 |
| ---------------------- | --------------------------------------------------------------------------- |
| `list[StreamingEvent]` | Every change point and verdict the pipeline emitted, in order, after flush. |

### `load_reference(path)`

Congested periods `(start, end)` of a reference CSV with `starts,ends,congestion`.

### `score(events, reference=None)`

Summarize replay events.

Parameters:

| Name        | Type                   | Description                                                                                                               | Default    |
| ----------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `events`    | `list[StreamingEvent]` | Output of replay.                                                                                                         | *required* |
| `reference` | `Path`                 | CSV with starts,ends,congestion columns (the paper's expected_results). Adds recovered, reference_congested and spurious. | `None`     |

Returns:

| Type   | Description                                                   |
| ------ | ------------------------------------------------------------- |
| `dict` | Counts, delays in minutes and provisional-to-final agreement. |

### `events_to_json(events, path)`

Write `events` as a JSON array of plain dictionaries.
# Project

# Changelog

All notable changes to Jitterbug are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased](https://github.com/estcarisimo/jitterbug/compare/v2.5.0...HEAD)

### Added

- `context7.json` at the repository root so Context7 indexes `docs/` and `examples/`.
- The documentation site serves `/llms.txt` and `/llms-full.txt`, generated by the `mkdocs-llmstxt` plugin (the `docs` dependency group). `llms.txt` indexes the guides and the API reference; `llms-full.txt` is those pages in one file. The README links both.
- PyPI project URLs for the PAM 2022 paper (`Paper`), the changelog (`Changelog`), and `llms.txt`. Keywords also cover latency, round-trip time, congestion inference, change-point detection, and network measurement.

### Changed

- `CITATION.cff` has a top-level `repository-code`, an abstract, and keywords. The code URL was nested under the author, which Citation File Format does not allow on a person. Zenodo copies the abstract and keywords into the software record's description and subjects.
- Locked transitive dependencies that `pip-audit` newly flags: Pillow 13.0.0 on Python 3.11 and newer (Pillow 13 publishes no Python 3.10 wheels, so the visualization extra on 3.10 stays on Pillow 12.3.0) and virtualenv 21.14.13.
- Locked filelock 4.2.4 on Python 3.11 and newer. filelock 3.32.6 is flagged by `pip-audit` (CVE-2026-112907, CVE-2026-112913, CVE-2026-112940); the fixed release, 4.2.1, requires Python 3.11. Python 3.10 stays on filelock 4.1.0, the newest release that still installs there. virtualenv requires filelock >= 3.24.2, already inside the affected range, so that fork cannot take a fixed version. The lint job audits on Python 3.12, where the marker selects 4.2.4.

## [2.5.0](https://github.com/estcarisimo/jitterbug/compare/v2.4.0...v2.5.0) - 2026-10-09

### Added

- Jitter dispersion in the online mode. `JitterAnalyzer.compute_causal_jitter_dispersion` applies the offline filters (moving IQR, moving average) as trailing windows, so a value never depends on later bins; the series is the offline one delayed by 6 bins at the default orders and the threshold keeps its meaning. Both online back ends now follow `jitter_analysis.method` for provisional and final verdicts (the config default, `jitter_dispersion`, included; 2.4.0 always used the KS test online). A provisional verdict with dispersion waits until the open period holds 12 dispersion values. `jitterbug stream` and `jitterbug replay` take `--method`; `tools/replay_online.py` too. On the paper dataset BCP + causal dispersion recovers 14 of the 15 reference periods of the paper's dispersion output with none spurious, like the offline pipeline, with no provisional flips; pinned in `tests/test_paper_regression.py` (`TestOnlineDispersionReplay`) and in the back ends table of `docs/ONLINE_MODE.md`.

### Changed

- The online mode no longer forces the KS test: with the default configuration it uses jitter dispersion, like `jitterbug analyze`. Pass `--method ks_test` (or set `jitter_analysis.method`) for the 2.4.0 behavior.

## [2.4.0](https://github.com/estcarisimo/jitterbug/compare/v2.3.0...v2.4.0) - 2026-10-09

### Added

- Online (streaming) mode. `jitterbug.streaming.OnlineJitterbug` consumes RTT samples one at a time and emits change points and congestion verdicts as events: causal minimum-RTT bins feed the Bayesian online change point detector of `bayesian-changepoint` 1.2 (`bcp` extra), a change point is declared when the MAP run length drops, and each period gets a provisional verdict (latency jump + KS test on the samples so far) and a final one when the next change point closes it. Settings are the `streaming` section of `JitterbugConfig` (`StreamingConfig`); the bin width, latency jump threshold, significance level and device come from the shared sections. `jitterbug stream` infers congestion from `epoch,rtt` lines on a file or standard input (`--follow`, `--events`) and prints JSON events; `jitterbug replay` runs a recorded dataset through the online pipeline and scores it against a reference (`--reference`, `--output`). On the paper dataset it recovers 14 of the 15 reference congestion periods with none spurious, like the offline pipeline, and reports congestion onsets 15 min after they start (median; 90 min max); pinned in `tests/test_paper_regression.py` (`TestOnlineReplay`). See `docs/ONLINE_MODE.md`. Tools: `tools/replay_online.py` (replay and score, `--sweep`, a flag for every online setting) and `tools/prefix_experiment.py` (how retrospective the offline pipeline is).
- A second online back end, `streaming.backend: window` (`jitterbug stream --backend window`): the offline sequential pipeline rerun on a trailing window (`window_hours`, default 72) every `rerun_every_bins` closed bins, emitting a change point once it has been placed within two bins of the same spot in `stable_runs` consecutive reruns, and final verdicts as a contiguous chain whose boundaries are the emitted change points. It works with any detector and jitter method and needs no new algorithm, at about 0.1 s per bin with BCP on the paper dataset. Events carry `jitter_method` and `jitter_metric`. `docs/ONLINE_MODE.md` compares the two back ends (table produced by `tools/compare_online_backends.py`, the 4-bin-cadence row pinned in `tests/test_paper_regression.py`); `jitterbug.streaming.create_online_analyzer` picks the back end, and `two_period_verdict` is the decision rule both share.
- Runtime comparison across releases (`docs/PERFORMANCE.md`) and the script that produces it (`tools/benchmark_versions.py`). On the paper dataset with BCP + KS, 2.3.0 takes 2.8 s against 16.6 s for 1.0.0 (the gain comes from `bayesian-changepoint` 1.2); with ruptures + KS, 2.1 and later are 1.5× faster than 2.0 thanks to the CSV loader (the same holds with jitter dispersion). Raw measurements in `examples/network_analysis/benchmarks/`. The runtime table in `docs/ALGORITHM_USAGE.md` now shows measured numbers.
- `docs/PERFORMANCE.md` credits the Bayesian change point library work behind the BCP numbers (hildensia/bayesian_changepoint_detection #50, #76, #96, #100, #103, #106), with Jitterbug's BCP + KS runtime and results for each library version it used. The README links the library's upstream repository and documentation instead of the fork.
- `docs/PERFORMANCE.md` correction: Jitterbug 2.0 with BCP 1.0 did not "take more than an hour" because of the library alone. On the CPU it takes 118 s (34 periods); the hour was on Apple's MPS, which BCP 1.0 picked by default. `tools/benchmark_versions.py run --hide-mps` measures the CPU case.

## [2.3.0](https://github.com/estcarisimo/jitterbug/compare/v2.2.0...v2.3.0) - 2026-09-24

### Added

- Zstandard (`.zst`) input and output. A path ending in `.zst` is decompressed on read (`jitterbug analyze rtts.csv.zst`; the extension before `.zst` picks the format) and compressed on write (`--output results.json.zst`, `results.csv.zst`, and the same in `ResultExporter` and `JitterbugAnalyzer.save_results`). New `zstd` extra (`zstandard`, only needed before Python 3.14, which has `compression.zstd`), included in `all`. `.parquet.zst` is rejected, since Parquet is already compressed. See `docs/INPUT_FORMATS.md`.

### Changed

- Text inputs and outputs (CSV, scamper JSON, JSON and CSV results) are read and written as UTF-8 instead of the platform's default encoding.

## [2.2.0](https://github.com/estcarisimo/jitterbug/compare/v2.1.1...v2.2.0) - 2026-09-24

### Added

- Non-sequential (clustering) analysis mode: `analysis_mode: clustering` / `jitterbug analyze --mode clustering`. Minimum-RTT intervals are clustered by (minimum RTT, jitter IQR) with a Gaussian mixture (components chosen by BIC), k-means, or k-means with the silhouette-best k (`clustering.algorithm`, `--clustering-algorithm`); every cluster is compared with the lowest-latency one by latency jump and a KS test on raw jitter, wherever its intervals fall in time; a temporal smoothing (`clustering.min_period_intervals`, default 2) merges the per-interval verdicts into periods. Output is the usual `CongestionInferenceResult`, with a `clustering` summary in the metadata. On the PAM 2022 dataset the GMM recovers 15 of the 15 reference congestion periods; its 2 extra periods are at the two ends of the data, where the minimum RTT is elevated but the sequential reference cannot give a verdict (sequential BCP + KS: 14/15, 0 extra). New `clustering` extra (scikit-learn), included in `all`; guide in `docs/CLUSTERING_MODE.md`. The sequential mode stays the default and is unchanged.
- Clustering mode: `clustering.min_ks_statistic` (default 0.1) requires the KS statistic, not only the p-value, to show a jitter change. Clusters pool thousands of samples, so the p-value is significant for negligible differences (p < 1e-49 for every cluster of the PAM 2022 dataset, including one at the baseline's latency, whose statistic is 0.088; the congested clusters are at 0.21–0.32). `clustering.latency_threshold` (default: `latency_jump.threshold`) sets a latency threshold for this mode alone. Results on the PAM 2022 dataset are unchanged.
- Documentation site built with MkDocs (Material theme): the guides in `docs/`, an API reference generated from the docstrings, and the README, changelog, contributing guide and citation included from the root files. CI builds it with `--strict` and validates `CITATION.cff` on every PR; `.github/workflows/pages.yml` publishes it to GitHub Pages from `main`. New `docs` dependency group.
- The documentation is online at <https://estcarisimo.github.io/jitterbug/>: README badge, the `Documentation` project URL (shown on PyPI from the next release) and the repository website point to it.

### Changed

- American English is now a documented convention (`AGENTS.md`, `CONTRIBUTING.md`, the review brief). Existing British spellings in docs, comments, the changelog and one test name (`test_unrecognizable_content_is_an_error`) were corrected; no public function, method, option or message changed.
- README: the extras table lists `jupyter`.

## [2.1.1](https://github.com/estcarisimo/jitterbug/compare/v2.1.0...v2.1.1) - 2026-09-23

### Added

- `.github/workflows/publish.yml`: publishing a GitHub release uploads the tag's sdist and wheel to PyPI with Trusted Publishing (OpenID Connect, no stored token), after checking that the version matches the tag and smoke-testing the wheel, then attaches both files to the release. A manual run uploads an existing tag to TestPyPI or PyPI.

### Changed

- README links and the figure use absolute GitHub URLs, so they work on the PyPI project page.
- First release on PyPI: `pip install jitterbug-inference`. README (with a PyPI badge), `docs/INSTALLATION.md`, `AGENTS.md` install instructions updated accordingly.
- `CODE_OF_CONDUCT.md`: incidents are reported privately through GitHub (maintainer profile, the repository's private reporting form, GitHub's *Report content*); the document no longer lists an email address.

## [2.1.0](https://github.com/estcarisimo/jitterbug/compare/v2.0.0...v2.1.0) - 2026-09-23

Maintenance release: the project is back to a maintainable state, with CI, a protected `main`, tests, and only the components that work. Jitterbug is now a library plus a CLI with two change point detectors (`ruptures` and the paper's Bayesian `bcp`, now installed from PyPI as `bayesian-changepoint`); the REST API, Docker image, experimental detectors and plotly dashboards described in 2.0.0 are gone (see *Removed*). Python 3.10 or newer is required. The distribution is renamed `jitterbug-inference`; installation is from GitHub, there is no PyPI release yet.

### Added

- CLI tests for `validate` (metrics table, `--verbose`, contract violations, missing file), `config --template` (YAML/JSON to stdout and round trip through a file) and the error paths of `analyze` (bad input, unknown method), completing the CLI coverage item of the roadmap.
- `docs/INPUT_FORMATS.md`: the input contract (columns, units, ordering, what is dropped and why) for CSV, scamper JSON, DataFrames and InfluxDB, linked from the README.
- Unit tests for `analysis/` (latency jumps, jitter dispersion, KS test, the congestion state machine) and `io/` (CSV, DataFrame, scamper JSON, format inference, mocked InfluxDB, validation, the JSON/CSV/summary exporters) on small synthetic series. Coverage 23 % → 84 %; CI now fails below 75 %.
- `tests/test_paper_regression.py`: golden counts for both detectors on the PAM 2022 dataset (ruptures + jitter dispersion: 22 periods / 11 congested; BCP + KS test: 34 / 14) and overlap-based agreement with the paper's reference intervals (11/15 and 13/15 recovered, no spurious detections). A dedicated CI job runs the Bayesian half with the `bcp` extra; Linux resolves torch from the CPU wheel index.
- `tests/test_cli.py::test_visualize_writes_the_standard_plots`: first test of the visualization code path (headless, `MPLBACKEND=Agg`).
- `bcp_device` option in `change_point_detection` (default `cpu`). The Bayesian library picks a GPU when it sees one, and on Apple Silicon that made the paper's configuration take over half an hour; on CPU it takes about two minutes. The value is validated (`cpu`, `cuda` or `mps`), and a test with a mocked backend checks that it reaches both the likelihood and the detection call.
- `get_available_algorithms()` now reports only the detectors whose packages are installed (`importlib.util.find_spec`), with tests.
- Continuous integration on GitHub Actions: ruff lint and format checks, mypy (blocking since #19), `pip-audit` over the locked dependency set, tests on Python 3.10–3.13 (Ubuntu) and 3.12 (macOS) with a CLI smoke test on the bundled PAM 2022 dataset, and a build job that installs the wheel in a clean environment. (#4)
- `tests/test_cli.py`: first tests for the command-line interface, run through Typer's `CliRunner`. (#5)
- Extras `rbeast` and `adtk` for the corresponding change point back ends, which the code already supported but no extra declared. `bayesian` is kept as a deprecated alias of `bcp`. (#5)
- Pre-commit configuration (ruff plus file hygiene hooks), Dependabot for GitHub Actions and Python dependencies, and a committed `uv.lock`. (#4)
- Community and policy files: `SECURITY.md`, `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `AGENTS.md`, `CITATION.cff`, `CODEOWNERS`, issue and pull request templates.

### Changed

- The `bcp` extra now installs the Bayesian library from PyPI as **`bayesian-changepoint>=1.2`** (the import name `bayesian_changepoint_detection` is unchanged), replacing the unmaintained upstream release and the git dependency used during development. `git` is no longer needed to install any extra, and a wheel of Jitterbug no longer carries a direct URL dependency. The new release fixes the offline segment likelihoods and the detection performance: on the PAM 2022 dataset the paper's configuration (`--algorithm bcp --method ks_test`) runs in about four seconds end to end (under one second of detection) instead of two minutes, and yields 28 periods / 14 congested (was 34 / 14), recovering 14 of the paper's 15 congested intervals (was 13) with no spurious detections. Golden values, the example plots and summaries, and the timing notes in the docs are updated. The detector no longer passes `truncate=-40`, deprecated in 1.2; the exact sum is as fast and matches the truncated one to 1e-12 on the paper dataset.
- The distribution is now named **`jitterbug-inference`** (`[project] name`); the import package and the `jitterbug` command are unchanged. `jitterbug` on PyPI belongs to an unrelated project, so this is the name a future PyPI release will use. `jitterbug.__version__` reads the metadata of the new distribution name.
- `docs/INSTALLATION.md` rewritten: install from a clone with uv or pip, or directly from GitHub with `pip install "jitterbug-inference[...] @ git+..."`; the extras table; notes on the `bcp` extra; verification commands; a troubleshooting table of symptoms that actually occur. It and `docs/ALGORITHM_USAGE.md` said `pip install jitterbug[...]`, which installs the unrelated project.
- README: `jitterbug visualize` is back in *Usage* (it was left out while broken, #7) and `docs/VISUALIZATION_USAGE.md` is in the documentation table; coverage figure refreshed.
- Build backend requirement raised to `uv_build>=0.12.14,<0.13` (Dependabot). Building from source with an older `uv` CLI still works: it fetches the backend from PyPI.
- `examples/README.md` and `examples/network_analysis/README.md` rewritten to match the directory: setup with `uv sync --extra ...`, the script and the notebooks as the two entry points, the dataset described (dates, sizes, what the paper found and what Jitterbug recovers), and the reference CSV layout explained. The example scripts no longer patch `sys.path`; `results/` and the benchmark outputs are ignored by git.
- Code review policy: PRs are reviewed by an independent, fresh-context session following `.github/REVIEW.md` (today a Claude Sonnet subagent) instead of GitHub Copilot; merge requires `APPROVE` on the final commit. `AGENTS.md`, `CONTRIBUTING.md` and the PR template updated.
- Input validation happens once, at the edge, in `DataLoader.load_from_dataframe` (CSV and InfluxDB go through it too): rows with a missing epoch or RTT, a non-positive RTT, or an RTT above `MAX_RTT_MS` (10 s) are dropped with a warning and counted in `metadata["dropped_rows"]`; unsorted rows are sorted (stable) with a warning instead of being rejected; a non-numeric cell is a `ValueError` naming the column and value. Before, the first bad row surfaced as a Pydantic error from inside the loading loop. The loop itself is vectorized: the bundled dataset loads in 0.1 s instead of 1 s.
- The scamper JSON reader applies the same RTT bounds per response: a `ping` response with an RTT of 0 (a timeout) or above 10 s no longer makes the whole file fail with a Pydantic error; it is dropped with a warning and counted in `metadata["dropped_responses"]`.
- Dependency floors raised to releases that support Python 3.10 (`numpy>=1.24`, `pandas>=2.0`, `scipy>=1.10`, `pydantic>=2.5`, `pydantic-settings>=2.1`, `ruptures>=1.1.9`, `typer>=0.12`, `rich>=13`); `click` and `requests` were declared but never imported and are no longer dependencies.
- `bandit` runs in the CI lint job (clean at the time of writing).
- mypy passes with `disallow_untyped_defs` on the whole package and is now a blocking CI check. Pydantic models use `model_config = ConfigDict(...)` / `SettingsConfigDict` and `model_dump()` instead of the deprecated `class Config` / `.dict()`; the two remaining test warnings are from third-party libraries.
- README rewritten in the project's canonical layout: features, quick start, usage, configuration, architecture, development, example output, citation. The table of contents, the duplicated installation sections and the `pip install jitterbug` instructions are gone; every command was run before being documented. The REST API, Docker and `visualize` command are not documented until they work (#6, #7).
- **Python 3.10 or newer is required** (was 3.8). (#4)
- All temporary ruff ignores are gone except `B008` (Typer's `Option(...)` defaults): `pathlib` everywhere (`Path.open`, `/`, `mkdir`, `iterdir`), `raise ... from` inside every `except`, collapsed nested conditions, no line over 100 characters. `tools/` scripts no longer patch `sys.path` to a directory that does not exist. Behavior unchanged (CLI output on the bundled dataset is identical).
- `pyproject.toml` is the single source of packaging metadata; `setup.py`, `requirements.txt`, `requirements-new.txt` and `install_dev.sh` are gone. The build backend is `uv_build`. Development tools live in the `dev` dependency group, so `uv sync` installs them. (#5)
- `jitterbug.__version__` and the API's reported version are read from package metadata instead of hard-coded strings. (#5)
- Code style is enforced by ruff (line length 100) instead of black, isort and flake8; type annotations use the Python 3.10 syntax (`X | None`, `list[str]`). (#4)
- The Docker image installs from `pyproject.toml`. (#5)

### Fixed

- A file with an unknown extension whose content is not text raises the documented `ValueError("Cannot infer format ...")` instead of leaking `UnicodeDecodeError`.
- `load_from_influxdb` computed epochs as `astype(int) / 1e9`, which assumes nanosecond timestamps; pandas 3 parses `_time` at microsecond resolution, so every epoch was 1000× too small. The conversion is now resolution-independent. Found by running the mocked InfluxDB tests in CI, where the `influx` extra is absent (a stub module now stands in for it).
- `verbose: true` in a configuration file had no effect from the CLI: the command installs a logging handler before reading the file, and the analyzer's second `basicConfig` was a no-op. The analyzer now sets the `jitterbug` logger level.
- The two analysis notebooks in `examples/` run again: they imported `requests` (no longer a dependency), read fields that do not exist on the result models (`start_time`, `confidence_score`, `jitter_ratio`, `ks_statistic`), and their "REST API" sections exercised the server removed in this release. Executed end to end with `nbconvert` before committing.
- Change point timestamps are timezone-aware UTC, like the measurements they come from. They used to be naive local time, so `start_timestamp`/`end_timestamp` in results files and the CLI table depended on the machine's timezone (the epochs were always right). The CLI table columns are now labeled `Start (UTC)` / `End (UTC)`.
- `DataLoader.validate_data()` returns plain Python numbers and booleans (it used to return numpy scalars, which `json.dumps` rejects); CSV datasets record `source: csv` and the file path in their metadata; `.jsonl` is recognized as scamper JSON.
- `jitterbug analyze` and `jitterbug visualize` no longer overwrite the `algorithm`, `method`, `threshold` and `output_format` values of a `--config` file with the CLI's own defaults; a flag now overrides the file only when it is given explicitly. A wrong `--algorithm` value is rejected with Pydantic's message instead of failing later.
- `jitterbug visualize` works again: it now writes the five matplotlib figures through `JitterbugPlotter.save_all_plots` and prints the summary. It used to abort with `'CongestionInference' object has no attribute 'timestamp'` (#7). The `--static-only`/`--interactive-only` flags are gone; `--prefix` is new. When the analysis yields no inferences the confidence heatmap is an empty placeholder instead of an `imshow` error, so the command still writes its five files.
- Time axes in every plot use matplotlib's automatic date locator and concise formatter instead of one labeled tick per hour, which produced an unreadable axis on multi-day series.
- The Bayesian detector no longer swallows exceptions and returns "no change points"; a failure is raised as `RuntimeError` with the cause attached.
- `JitterbugAnalyzer.analyze()` returns the same `metadata` keys (`total_measurements`, `min_intervals`, `change_points`) on its early-return paths as on the full path. (#3)
- Test suite: eight assertions that had drifted from the code, and five tests that failed during the last fifteen minutes of every hour. (#3)

### Removed

- `examples/basic_analysis.py` (synthetic two-hour series on which nothing is detected) and `examples/output_formats_demo.py` (600 lines, crashed on a PyYAML argument, still used Pydantic v1 `.dict()`). `examples/network_analysis/basic_analysis.py` is the script example.
- The `influx` *file* format: `--format influx` and `.flux`/`.influx` files were routed to a placeholder that raised `NotImplementedError`. Loading from an InfluxDB server through `DataLoader.load_from_influxdb()` is unchanged. Format inference now fails with a clear message instead of guessing `influx` for anything it does not recognize.
- Dead code in `analysis/`: `LatencyJumpAnalyzer.analyze_detailed`, `CongestionInferenceAnalyzer._apply_inference_logic` and `_post_process_inferences` implemented alternative rules that nothing called.
- **The plotly dashboard and interactive modules** (`visualization/dashboard.py`, `visualization/interactive.py`), the `plotly` dependency, `examples/visualization_demo.py` and the generated `interactive_bcp_ks/timeline.html`. They had no tests, the dashboard crashed on every real result (#7), and the demo called a method that does not exist. `docs/VERSION_COMPARISON.md`, a 1.x-versus-2.x page that described all of the removed components as features, is gone too.
- **The experimental `torch`, `rbeast` and `adtk` detectors** and their extras. None of them was evaluated in the paper; `torch` was a heuristic rather than a trained model, and `rbeast`/`adtk` silently fell back to an internal statistical method when their package was missing while still labeling the output with the back end's name. The detectors are now `ruptures` (default) and `bcp`. `algorithms.py` shrinks from 1 215 to 318 lines; `examples/interactive_algorithm_selector.py` and the stale plots for the removed detectors are gone, and `examples/network_analysis/plots/` was regenerated (BCP 14/15, ruptures 11/15, unchanged).
- **The REST API server and the Docker image.** `jitterbug.api`, the `api` extra, the `Dockerfile`, `docker-compose.yml`, the entrypoint script, `docs/DOCKER.md` and the two related examples are gone. The server had not been able to start since the 2.0 rewrite (`create_app()` raised on import and `/analyze` called a method that did not exist, #6), had no tests, and the Docker image's only entrypoint was that server. Jitterbug is a library and a CLI; wrap it in your own service if you need HTTP.
- The DockerHub push, PyPI publish and Codecov upload jobs from CI; none of them had working credentials. (#4)
- The `print`-based smoke scripts `test_algorithms.py` and `examples/test_new_implementation.py`, replaced by real tests. (#5)

## [2.0.0](https://github.com/estcarisimo/jitterbug/compare/v1.0.0...v2.0.0) - 2025-07-19

Complete rewrite of the PAM 2022 framework: Pydantic models, five change point detection back ends (ruptures, BCP, PyTorch, Rbeast, ADTK), CSV/JSON/InfluxDB input, Typer CLI, optional REST API and visualization modules. See the [pull request](https://github.com/estcarisimo/jitterbug/pull/2).

## [1.0.0](https://github.com/estcarisimo/jitterbug/releases/tag/v1.0.0) - 2024-03-18

Original implementation accompanying the paper *Jitterbug: A new framework for jitter-based congestion inference* (PAM 2022).

# Contributing to Jitterbug

Thanks for your interest in Jitterbug. This guide covers the development setup, the checks every change must pass, and how a change gets from your machine into `main`.

## Development setup

Jitterbug uses [uv](https://github.com/astral-sh/uv) for everything: environments, dependencies and building.

```
git clone git@github.com:estcarisimo/jitterbug.git
cd jitterbug
uv sync --extra visualization   # runtime + dev tools (the `dev` group is installed by default)
uv run pre-commit install       # git hooks: ruff lint/format, file hygiene
```

Add `--extra bcp` for the Bayesian back end used in the paper (`bayesian-changepoint`), or `--extra all` for every optional back end. See `pyproject.toml` for the full list.

Run the same checks CI runs:

```
uv run ruff check src/ tests/ examples/ tools/
uv run ruff format --check src/ tests/ examples/ tools/
uv run mypy src/jitterbug            # blocking in CI
uv run pytest -m "not slow"          # a few seconds
uv run pytest                        # + the Bayesian regression tests (~2 s more, needs --extra bcp)
uv run jitterbug analyze examples/network_analysis/data/raw.csv --output /tmp/results.json
uv build
uv run --group docs mkdocs build --strict   # the documentation site; warnings fail
```

The documentation site (`mkdocs.yml`, Material theme) is built from `docs/`; its home, changelog, contributing and citing pages include the root files, and the API reference is generated from the docstrings. The same build writes `llms.txt` and `llms-full.txt` at the site root for language models. Preview it with `uv run --group docs mkdocs serve`. Pushes to `main` publish it to GitHub Pages (`.github/workflows/pages.yml`).

`uv run pre-commit run --all-files` runs the lint and hygiene hooks on the whole tree.

## Project conventions

- Python 3.10+ code: `X | None` unions, `list[str]` generics, `match` is fine. Ruff's `UP` rules enforce the 3.10 target.
- Line length 100, ruff formatter, imports sorted by ruff (`I`). The temporary `ignore` entries in `[tool.ruff.lint]` each name a follow-up; remove an entry when its findings are fixed, and do not add new ones.
- `pathlib.Path` for paths, `logging` (module-level `logger = logging.getLogger(__name__)`) instead of `print` in library code. `print` and Rich output belong in `cli/` only.
- NumPy-style docstrings on public functions and classes.
- American English in code and prose: identifiers, docstrings, comments, messages, docs and changelog (`analyze`, `behavior`, `color`, `labeled`, `serialize`).
- Configuration is Pydantic models in `src/jitterbug/models/config.py`. Add a field there and thread it through the CLI (`src/jitterbug/cli/main.py`) rather than adding ad-hoc parameters.
- Type hints on every function; mypy runs with `disallow_untyped_defs` and is blocking in CI.
- Tests are plain pytest functions with fixtures and `parametrize`, one file per module (`tests/test_<module>.py`). Use small synthetic RTT series; the bundled `examples/network_analysis/data/raw.csv` (47 163 rows, PAM 2022) is for smoke and regression tests only. Compare floats with `numpy.testing.assert_allclose`.
- `tests/test_paper_regression.py` pins the current output on the bundled dataset (golden counts) and checks agreement with the paper's reference intervals by overlap. If you change the numbers on purpose, update the goldens and explain in the changelog. The Bayesian half is marked `slow` and runs in its own CI job with the `bcp` extra.
- Every behavior change gets a test under `tests/` and a line in `CHANGELOG.md` under `[Unreleased]`.

## Making a change

`main` is protected. Nobody pushes to it directly; every change lands through a pull request that has passed CI **and** an independent code review from a fresh session (see `.github/REVIEW.md`). This is repository policy: a PR without an `APPROVE` on its final commit is not merged.

1. Create a branch from `main`: `git switch -c <type>/<short-name>` (`feat/`, `fix/`, `docs/`, `chore/`).
1. Commit in small, coherent steps. The pre-commit hooks run ruff on each commit.
1. Push and open a PR: `gh pr create --fill`. The PR template has a checklist.
1. **Iterate until green.** Two things must be true before merging:
1. CI is green: `lint`, every `test (...)` matrix leg, and `build`.
1. A reviewer with no context from your session — a new AI agent session given only `.github/REVIEW.md` and the PR number, or a human — has returned `VERDICT: APPROVE` for the **latest** push. Every finding before that is either fixed or rebutted with evidence in the PR; after each push, start a new reviewer session (never reuse one).

```
gh pr checks <n> --watch          # wait for CI
gh pr view <n> --comments         # every round's verdict is posted here in full
```

1. Merge (squash) once both are green. The branch is deleted automatically.

Stacked PRs: merge the bottom of the stack **without** `--delete-branch`; GitHub then retargets the next PR to `main` on its own. Deleting the base branch first closes the stacked PR. If the retargeted PR shows as `DIRTY`, see the stacked-PR step in `AGENTS.md` for the safe way to update it.

Repository admins can technically bypass the ruleset. Treat that as an emergency-only escape hatch and say so in the PR when it is used.

## Releasing

1. Bump `version` in `pyproject.toml` (the only place the version lives; `jitterbug.__version__` reads it from package metadata).
1. Move the `[Unreleased]` section of `CHANGELOG.md` under a new `[X.Y.Z] - YYYY-MM-DD` heading and add the compare link at the bottom. Update `version` and `date-released` in `CITATION.cff`.
1. Open a PR with those changes and merge it.
1. Tag and publish a GitHub release: `git tag -a vX.Y.Z -m "Jitterbug X.Y.Z"`, `git push origin vX.Y.Z`, then `gh release create vX.Y.Z --verify-tag` with the CHANGELOG section as notes.
1. Publishing the release starts `.github/workflows/publish.yml`, which builds the tag, checks that the version equals the tag, smoke-tests the wheel, uploads to PyPI through Trusted Publishing (no API token) and attaches the sdist and wheel to the release. The upload waits for approval if the `pypi` environment requires a reviewer.

PyPI uploads cannot be replaced or deleted and reused: a broken upload means a new version. To rehearse, run the workflow by hand on TestPyPI first (Actions → Publish → Run workflow, tag `vX.Y.Z`, target `testpypi`); the same form with target `pypi` uploads an existing tag that was not published automatically.

The distribution is named `jitterbug-inference` because `jitterbug` on PyPI belongs to an unrelated project; the import package and the command stay `jitterbug`. The README is the PyPI project page, so its links are absolute URLs.

## Reporting issues

Use the issue templates. For security problems follow `SECURITY.md` instead of opening a public issue.

# Citing Jitterbug

If you use Jitterbug in your research, please cite the PAM 2022 paper:

> Esteban Carisimo, Ricky K. P. Mok, David D. Clark and K. C. Claffy. *Jitterbug: A New Framework for Jitter-Based Congestion Inference.* Passive and Active Measurement (PAM 2022), Springer, pp. 155–179. [doi:10.1007/978-3-030-98785-5_7](https://doi.org/10.1007/978-3-030-98785-5_7)

```
@inproceedings{carisimo2022jitterbug,
  title     = {Jitterbug: A New Framework for Jitter-Based Congestion Inference},
  author    = {Carisimo, Esteban and Mok, Ricky K. P. and Clark, David D. and Claffy, K. C.},
  booktitle = {Passive and Active Measurement (PAM 2022)},
  publisher = {Springer International Publishing},
  pages     = {155--179},
  year      = {2022},
  doi       = {10.1007/978-3-030-98785-5_7}
}
```

The repository's citation metadata, which GitHub shows as *Cite this repository*:

```
cff-version: 1.2.0
message: "If you use Jitterbug in your research, please cite the PAM 2022 paper below."
type: software
title: "Jitterbug: Framework for Jitter-Based Congestion Inference"
abstract: "Jitterbug infers network congestion from Round-Trip Time (RTT) time series. It computes minimum-RTT intervals, detects change points, and classifies each period as congested or not using a latency jump and either jitter dispersion or a Kolmogorov-Smirnov test."
authors:
  - family-names: Carisimo
    given-names: Esteban
    email: esteban.carisimo@northwestern.edu
    affiliation: Northwestern University
keywords:
  - network measurement
  - congestion inference
  - round-trip time
  - jitter
  - change-point detection
license: MIT
version: 2.5.0
date-released: "2026-10-09"
repository-code: "https://github.com/estcarisimo/jitterbug"
preferred-citation:
  type: conference-paper
  title: "Jitterbug: A New Framework for Jitter-Based Congestion Inference"
  authors:
    - family-names: Carisimo
      given-names: Esteban
    - family-names: Mok
      given-names: Ricky K. P.
    - family-names: Clark
      given-names: David D.
    - family-names: Claffy
      given-names: K. C.
  collection-title: "Passive and Active Measurement (PAM 2022)"
  publisher:
    name: Springer International Publishing
  year: 2022
  start: 155
  end: 179
  isbn: "978-3-030-98785-5"
  doi: 10.1007/978-3-030-98785-5_7
```