64 lines
3.4 KiB
Markdown
64 lines
3.4 KiB
Markdown
# The store format
|
|
This project uses a custom database format just because we can.
|
|
|
|
The script reads the entire store on startup. Each object can br altered in memory. A 'dirty' status is set when an object is changed. periodically the script will flush the store, writing all changes back to disk.
|
|
|
|
The store disk format is optimized for two things:
|
|
* Loading the entire contents in memory
|
|
* Writing changed objects back to disk
|
|
|
|
## Objects
|
|
First lets discuss what we are storing.
|
|
|
|
Objects are identified by type and symbol. A symbol is an uppercase alphanumeric string that optionally contains '-' chars. the type is a lowecase alphanumeric string of length 3.
|
|
|
|
For each type, the store has a class defined that is used to load objects of that type.
|
|
|
|
An object identifier is its symbol and type joined with a '.' character.
|
|
|
|
Some examples of object identifiers:
|
|
|
|
* X1-J84.sys
|
|
* X1-J84-0828772.way
|
|
* CAPT-J-1.shp
|
|
|
|
A waypoint is always part of a system. This is also visible because the waypoint symbol is prefixed by the system symbol. However, this relation is not enforced or used by the store. The symbol is an opaque string.
|
|
|
|
An object has attributes. Values of attributes can be strings, ints, floats, bools, lists, dicts and references. lists and dicts can also only contain the values listed. References are pointers to other objects in the store.
|
|
|
|
## Indices
|
|
An index is a dict with a string as key and a list of objects as value. The dict is built when loading the store. when the index is iterated, each object is re-checked and removed if necessary.
|
|
|
|
## API
|
|
|
|
* store.load(fil) loads all objects
|
|
* store.get(type, symbol, create=False) fetches the object. If create==False: None if it wasnt present
|
|
* store.all(type) generator for all objects of a goven type
|
|
* store.purge(obj)
|
|
* store.clean() removes all expired objects
|
|
* store.flush() writes all dirty objects to disk
|
|
* store.defrag() consolidates the store file to minimize storage and loading time
|
|
|
|
type may be a class or a string containing the name of a class. The type should be a subclass of models.base.Base
|
|
|
|
# file format
|
|
Until specified otherwise, all numbers are stored low-endian 64bit unsigned.
|
|
|
|
The store file is built up out of chunks. A chunk is either empty or houses exactly one file. If a file is updated and its size fits the chunk, it is updated in-place. If the new content does not fit the chunk, a new chunk is allocated at the end of the file. The old chunk is marked as empty.
|
|
|
|
A chunk starts with a chunk header. The header consists of three 8-byte fields.
|
|
|
|
The first field is the magic. Its value is 'ChNkcHnK'. The magic can be used to recover from a corrupted file.
|
|
|
|
The second field is describing the size of the chunk in bytes, not including the header. The first bit of the field is the IN_USE flag. If it is not set, the contents of the chunk are ignored during loading.
|
|
|
|
The third field described how much of the chunk is occupied by content. This is typically less than the size of the chunk because we allocate slack for each object to grow. The slack prevents frequent reallocation.
|
|
|
|
# Future work
|
|
This format is far from perfect.
|
|
|
|
* file corruption sometimes occurs. The cause of this still has to be found
|
|
* Recovery of file corruption has not yet been implemented
|
|
* Diskspace improvements are possible by eliminating slack for non-changing objects such as waypoints and compressing the file
|
|
* Indices have not been implemented although a "member" index keeps track of which objects are in each system.
|