0ptr/store.md

# The store format
This project uses a custom database format just because we can.

The script reads the entire store on startup. Each object can br altered in memory. A 'dirty' status is set when an object is changed. periodically the script will flush the store, writing all changes back to disk.

The store disk format is optimized for two things:
* Loading the entire contents in memory
* Writing changed objects back to disk

## Objects
First lets discuss what we are storing.

Objects are identified by type and symbol. A symbol is an uppercase alphanumeric string that optionally contains '-' chars. the type is a lowecase alphanumeric string of length 3.

For each type, the store has a class defined that is used to load objects of that type.

An object identifier is its symbol and type joined with a '.' character.

Some examples of object identifiers:

* X1-J84.sys
* X1-J84-0828772.way
* CAPT-J-1.shp

A waypoint is always part of a system. This is also visible because the waypoint symbol is prefixed by the system symbol. However, this relation is not enforced or used by the store. The symbol is an opaque string.

An object has attributes. Values of attributes can be strings, ints, floats, bools, lists, dicts and references. lists and dicts can also only contain the values listed. References are pointers to other objects in the store.

## Indices
An index is a dict with a string as key and a list of objects as value. The dict is built when loading the store. when the index is iterated, each object is re-checked and removed if necessary.

## API

* store.load(fil) loads all objects
* store.get(type, symbol, create=False) fetches the object. If create==False: None if it wasnt present
* store.all(type) generator for all objects of a goven type
* store.purge(obj)
* store.clean() removes all expired objects
* store.flush() writes all dirty objects to disk
* store.defrag() consolidates the store file to minimize storage and loading time

type may be a class or a string containing the name of a class. The type should be a subclass of models.base.Base

# file format
Until specified otherwise, all numbers are stored low-endian 64bit unsigned.

The store file is built up out of chunks. A chunk is either empty or houses exactly one file. If a file is updated and its size fits the chunk, it is updated in-place. If the new content does not fit the chunk, a new chunk is allocated at the end of the file. The old chunk is marked as empty.

A chunk starts with a chunk header. This is just a single field describing the size of the chunk in bytes, not including the header. The first bit of the field is the IN_USE flag. If it is not set, the contents of the chunk are ignored during loading.