# Carbon Aware DataStore (Zurich Green IT Hackathon Repo)

[![Go Version](https://img.shields.io/badge/Go-1.18+-blue.svg)](https://golang.org/)

MiniBTreeStore is an experimental key-value (KV) store built in Go, utilizing a B+ Tree data structure for efficient on-disk storage. Its standout feature is a **carbon-aware flushing strategy** designed to reduce the environmental impact of its disk I/O operations.

This project was developed as part of a deep-dive exploration into Green Software Engineering principles, demonstrating how even foundational components like a storage engine can be made more environmentally conscious.

## Quick Start

```bash
# Build the application
go build -o minibtreestore_app .

# Run (Example with carbon-awareness enabled, high intensity start, and visualizer)
./minibtreestore_app -carbonaware=true -mockhigh=true -inserts=500 -buffersize=2 -vizport=:8081 -dir=./test_run_data -interval=1s

Interactive Carbon Intensity Control (During Run)

When the application is running with -carbonaware=true:

Press L (and Enter) in the terminal to set mock carbon emissions to LOW 🟢. Press H (and Enter) in the terminal to set mock carbon emissions to HIGH 🔴.

Viewing the Frontend Visualizer

After running the backend application with a -vizport specified (e.g., :8081):

# Open this file in your web browser
visualizer/frontend/index.html

The dashboard will poll data from http://localhost:PORT/status (e.g., http://localhost:8081/status).

✨ Features

• Core B+ Tree Operations: Supports Insert, Lookup, and Scan operations. Delete functionality is also present.
• Buffer Management: Implements a buffer manager with an LRU (Least Recently Used) page replacement policy to optimize disk access and improve performance.
• Carbon-Aware Flushing Strategy:
  • Monitors (simulated) grid carbon intensity (HIGH 🔴 / LOW 🟢) using a MockIntensityProvider.
  • Intelligently defers non-critical disk page flushes when carbon intensity is reported as HIGH.
  • Processes the queue of deferred flushes when carbon intensity transitions to LOW, leveraging "greener" energy periods.
  • Aims to reduce the effective CO2 footprint of disk write operations.
• Interactive Carbon Simulation:
  • Allows manual toggling of simulated carbon intensity (HIGH/LOW) during runtime via CLI input when main.go is running with carbon-awareness enabled.
• Real-time Visual Feedback:
  • Enhanced CLI Logs: Detailed console output with emojis (e.g., 🟡 Deferred, 🟢 Low Carbon Flush, ❗Forced Flush) to track the system's carbon-aware decisions and actions.
  • Web Visualizer Dashboard: A simple web-based dashboard (visualizer/frontend/index.html) polls a backend endpoint (/status from visualizer/server.go) to display:
    • Current carbon intensity status.
    • Size of the deferred flush queue.
    • Counts of normal, deferred, and forced flushes.
    • Buffer pool statistics.

📂 Project Structure

minibtreestore/
├── carbonaware/             # Carbon intensity provider logic
│   └── intensity_provider.go
├── main.go                  # Main application entry point with CLI flags
├── storage/
│   ├── btree/               # B+ Tree implementation
│   ├── buffer/              # Buffer manager implementation
│   └── page/                # Page formats and constants
├── test/                    # Go tests (e.g., store_test.go)
├── util/
│   └── loader/              # CSV data loader utility (if used)
└── visualizer/
    ├── frontend/            # HTML, CSS, JS for the web dashboard
    └── server.go            # Backend HTTP server for visualizer data

🚀 Getting Started (Detailed)

Prerequisites

• Go (version 1.18 or higher recommended)

Build

Navigate to the project root directory and run:

go build -o minibtreestore_app .

Running

The application main.go serves as the entry point and accepts several flags to configure its behavior and the demo workload.

Key Command-Line Flags (see main.go for all):

• -dir: Storage directory for B-Tree files (default: ./minibtree_data).
• -buffersize: Number of pages in the buffer pool (default: 100). Try small values like 2 or 5 with high insert counts to observe buffer pressure.
• -carbonaware: Enable carbon-aware mode (default: false).
• -carbonregion: Carbon intensity region for simulation (default: "europe-west6").
• -interval: Interval for the carbon watchdog to check deferred queue (default: 30s). Use shorter intervals like 1s for faster demo feedback.
• -mockhigh: Start with mock carbon intensity as HIGH (default: false, starts LOW).
• -vizport: Port for the visualizer HTTP server (e.g., :8081). Set to empty to disable.
• -inserts: Number of key-value pairs to insert for a demo workload.
• -btreename: Name of the B-Tree for demo operations.

Example Run (from Quick Start):

./minibtreestore_app -carbonaware=true -mockhigh=true -inserts=500 -buffersize=2 -vizport=:8081 -dir=./test_run_data -interval=1s

🧪 Testing

The test/ directory contains Go tests (e.g., store_test.go). You can run these using standard Go test commands from the root directory:

go test ./...

To test the carbon-aware features interactively, use the command-line flags provided in main.go to set up different scenarios. The CLI logs and web visualizer are key to observing behavior. The "Quick Start" section above provides an example for running and interacting.

📝 Carbon-Aware Simulation Note

This project currently uses a MockIntensityProvider to simulate grid carbon intensity. This is essential for controlled testing and demonstration of the carbon-aware logic. Future work could involve integrating this with real-world carbon intensity data APIs (e.g., from the Green Software Foundation's Carbon Aware SDK, WattTime, Electricity Maps).

Contributors

• Arka Mitra
• Carmen Parra
• Ismail Mohamed Mohamud
• Jeffrey Koppanyi
• Joe Romeo
• Jonas Ward Van den Bulcke
• Raunak Pillai

License

MIT License

Estimate and track carbon emissions from your computer, quantify and analyze their impact.

Documentation

• About CodeCarbon 💡
• Quickstart 🚀
  • Installation 🔧
  • Start to estimate your impact 📏
    • Monitoring your whole machine
    • In your python code
    • Visualize
• Contributing 🤝
• How To Cite 📝
• Contact 📝

About CodeCarbon 💡

CodeCarbon started with a quite simple question:

What is the carbon emission impact of my computer program? :shrug:

We found some global data like "computing currently represents roughly 0.5% of the world’s energy consumption" but nothing on our individual/organisation level impact.

At CodeCarbon, we believe, along with Niels Bohr, that "Nothing exists until it is measured". So we found a way to estimate how much CO₂ we produce while running our code.

How?

We created a Python package that estimates your hardware electricity power consumption (GPU + CPU + RAM) and we apply to it the carbon intensity of the region where the computing is done.

We explain more about this calculation in the Methodology section of the documentation.

Our hope is that this package will be used widely for estimating the carbon footprint of computing, and for establishing best practices with regards to the disclosure and reduction of this footprint.

So ready to "change the world one run at a time"? Let's start with a very quick set up.

Quickstart 🚀

Installation 🔧

From PyPI repository

pip install codecarbon

From Conda repository

conda install -c codecarbon codecarbon

To see more installation options please refer to the documentation: Installation

Start to estimate your impact 📏

To get an experiment_id enter:

! codecarbon init

You can now store it in a .codecarbon.config at the root of your project

[codecarbon]
log_level = DEBUG
save_to_api = True
experiment_id = 2bcbcbb8-850d-4692-af0d-76f6f36d79b2 #the experiment_id you get with init

Now you have 2 main options:

Monitoring your machine 💻

In your command prompt use: codecarbon monitor The package will track your emissions independently from your code.

In your Python code 🐍

from codecarbon import track_emissions
@track_emissions()
def your_function_to_track():
  # your code

The package will track the emissions generated by the execution of your function.

There is other ways to use codecarbon package, please refer to the documentation to learn more about it: Usage

Visualize 📊

You can now visualize your experiment emissions on the dashboard.

Note that for now, all emissions data send to codecarbon API are public.

Hope you enjoy your first steps monitoring your carbon computing impact! Thanks to the incredible codecarbon community 💪🏼 a lot more options are available using codecarbon including:

• offline mode
• cloud mode
• comet integration...

Please explore the Documentation to learn about it If ever what your are looking for is not yet implemented, let us know through the issues and even better become one of our 🦸🏼‍♀️🦸🏼‍♂️ contributors! more info 👇🏼

Contributing 🤝

We are hoping that the open-source community will help us edit the code and make it better!

You are welcome to open issues, even suggest solutions and better still contribute the fix/improvement! We can guide you if you're not sure where to start but want to help us out 🥇

In order to contribute a change to our code base, please submit a pull request (PR) via GitHub and someone from our team will go over it and accept it.

Check out our contribution guidelines :arrow_upper_right:

Contact @vict0rsch to be added to our slack workspace if you want to contribute regularly!

How To Cite 📝

If you find CodeCarbon useful for your research, you can find a citation under a variety of formats on Zenodo.

Here is a sample for BibTeX:

@software{benoit_courty_2024_11171501,
  author       = {Benoit Courty and
                  Victor Schmidt and
                  Sasha Luccioni and
                  Goyal-Kamal and
                  MarionCoutarel and
                  Boris Feld and
                  Jérémy Lecourt and
                  LiamConnell and
                  Amine Saboni and
                  Inimaz and
                  supatomic and
                  Mathilde Léval and
                  Luis Blanche and
                  Alexis Cruveiller and
                  ouminasara and
                  Franklin Zhao and
                  Aditya Joshi and
                  Alexis Bogroff and
                  Hugues de Lavoreille and
                  Niko Laskaris and
                  Edoardo Abati and
                  Douglas Blank and
                  Ziyao Wang and
                  Armin Catovic and
                  Marc Alencon and
                  Michał Stęchły and
                  Christian Bauer and
                  Lucas Otávio N. de Araújo and
                  JPW and
                  MinervaBooks},
  title        = {mlco2/codecarbon: v2.4.1},
  month        = may,
  year         = 2024,
  publisher    = {Zenodo},
  version      = {v2.4.1},
  doi          = {10.5281/zenodo.11171501},
  url          = {https://doi.org/10.5281/zenodo.11171501}
}

Contact 📝

Maintainers are @vict0rsch @benoit-cty and @SaboniAmine. Codecarbon is developed by volunteers from Mila and the DataForGoodFR community alongside donated professional time of engineers at Comet.ml and BCG GAMMA.

Star History

Comparison of the number of stars accumulated by the different Python CO2 emissions projects: