Expiry and defragmentation

Dockerfile

@@ -9,4 +9,4 @@ ADD --chown=user . /app
 RUN chmod +x /app/main.py
 VOLUME /data
 ENTRYPOINT [ "python3", "/app/main.py"]
-CMD ["-s", "/data/"]
+CMD ["-s", "/data/store.npt"]

Readme.md

@@ -0,0 +1,83 @@
+# 0ptr script for spacetraders.io
+This is my script for running spacetraders.io. It has been completely written and operated from an iphone.
+
+Because i'm using a less-than-optimal development platform, some 'best-practice' code standards have been thrown out of the window. For instance, some variables have obscure names just to keep them short. 
+
+The script offers a command-line interface allowing you to do most ST operations by hand. More importantly, it offers the functionality to assign missions to ships and execute them automatically.
+
+# Getting started
+I'm using the [pythonista app](http://omz-software.com/pythonista/) to develop and test the script. Just open main.py and hit run.
+
+To keep the script running for longer periods of time, I'm using a VPS with docker. Start the script like so:
+
+`docker run -v 0ptr:/data -ti 0ptr`
+
+The script will ask for an agent name. Make sure you choose a unique one. After hitting enter, you will be presented with the prompt: `>`
+
+Now you'll want to register. You need to provide your faction:
+
+`> register cosmic`
+
+If all goes well, you should have an account now. 
+
+Next is building up our view of the universe: the atlas. 
+
+`> universe`
+
+This command will iterate over all systems and obtain all waypoint information. **this will take several hours**. However, if you are impatient, you can stop the process after a couple of minutes by hitting enter. It should at least have indexed your local system to continue.
+
+Now, let's see what ships we own.
+
+`> ships refresh`
+
+This will populate our internal cache of present ships and their state. After this, the state is updated continually by each command that alters it. You should be able to omit the 'refresh' after this.
+
+We should also populate the cache of contracts:
+
+`> contracts refresh`
+
+Same here: you can omit the 'refresh' from now on and view your contracts with the bare `contracts` command without sending a http request.
+
+You are now ready to begin operations.
+
+# Operations
+To get a ship to do something, you first have to select it:
+
+```
+> ship 1
+KILO-1 DOCKED [1035/1200] X1-YU85-03282C
+KILO-1> 
+```
+
+This prints the current state of the ship. Notice that the prompt now includes the name of the ship. Any command you now give will be executed by that ship.
+
+A quick, non-complete list of commands:
+
+* go: intra-system travel to another waypoint
+* orbit
+* dock
+* market: show local market data
+* shipyard: show local shipyard data
+* jumps: show local jumpgate data
+* mission: set ships mission type
+	* mine: mine and deliver a resource
+	* probe: find resource prices in nearby markets
+	* survey: generate surveys in a loop
+	* travel: travel to any waypoint in the universe using jumpgates (assuming a route is found)
+	* haul: buy goods at a site and bring them to a destination
+* mset: set a mission parameter
+* purchase: purchase a ship at a shipyard
+* buy: buy a resource at the local market
+* sell: sell a resource at the local market
+* jump: jump to a remote system using the local jumpgate
+* cmine: configures a mining mission for the current contract
+* chaul configures a hauling mission for the current contract
+* cprobe: configures a probing mission for the current contract
+* query: lists nearest markets and buy prices (if known) of a resource
+
+Look in commander.py for more available commands.
+
+# Data storage
+The script stores each object in a separate json file in the data directory. The default location for this directory is in your workingdir. The docker image places the data dir in '/data'
+
+If you want to re-register an agent, or there is a sever reset, just delete all data in that directory.

main.py

@@ -1,8 +1,9 @@
+#!/usr/bin/env python3
 import argparse
 from nullptr.commander import Commander
-
+from nullptr.models.base import Base
 def main(args):
-  c = Commander(args.store_dir)
+  c = Commander(args.store_file)
   c.run()
   
 # X1-AG74-41076A
@@ -10,6 +11,6 @@ def main(args):
 
 if __name__ == '__main__':
   parser = argparse.ArgumentParser()
-  parser.add_argument('-s', '--store-dir', default='data')
+  parser.add_argument('-s', '--store-file', default='data/store.npt')
   args = parser.parse_args()
   main(args)

nullptr/analyzer.py

@@ -1,6 +1,7 @@
 from nullptr.models.marketplace import Marketplace
 from nullptr.models.jumpgate import Jumpgate
 from nullptr.models.system import System
+from nullptr.models.waypoint import Waypoint
 from dataclasses import dataclass
 
 @dataclass
@@ -29,14 +30,42 @@ class Analyzer:
 
   def find_markets(self, resource, sellbuy):
     for m in self.store.all(Marketplace):
-      resources = m.imports if sellbuy == 'sell' else m.exports
-      if resource in resources:
-        yield m
+      if 'sell' in sellbuy and resource in m.imports:
+        yield ('sell', m)
+      
+      elif 'buy' in sellbuy and resource in m.exports:
+        yield ('buy', m)
+      
+      elif 'exchange' in sellbuy and resource in m.exchange:
+        yield ('exchange', m)
+        
+  def find_closest_markets(self, resource, sellbuy, location):
+    if type(location) == str:
+      location = self.store.get(Waypoint, location)
+    mkts = self.find_markets(resource, sellbuy)
+    candidates = []
+    origin = location.system
+    for typ, m in mkts:
+      system = m.waypoint.system
+      d = origin.distance(system)
+      candidates.append((typ, m, d))
+    possibles = sorted(candidates, key=lambda m: m[2])
+    possibles = possibles[:10]
+    results = []
+    for typ,m,d in possibles:
+      system = m.waypoint.system
+      p = self.find_path(origin, system)
+      if p is None: continue
+      results.append((typ,m,d,len(p)))
+    return results
+    
+  def solve_tsp(self, waypoints):
+    # todo actually try to solve it
+    return waypoints
  
   def get_jumpgate(self, system):
     gates = self.store.all_members(system, Jumpgate)
     return next(gates, None)
-
    
   def find_path(self, orig, to, depth=100, seen=None):
     if depth < 1: return None
@@ -54,9 +83,7 @@ class Analyzer:
       for s in jg.systems:
         if s in seen:  continue
         seen.add(s)
-        system = self.store.get(System, s)
-        if system is None: continue
-        dest.add(SearchNode(system, o))
+        dest.add(SearchNode(s, o))
     if len(dest) == 0: 
       return None
     return self.find_path(dest, to, depth-1, seen)

nullptr/api.py

@@ -31,7 +31,7 @@ class Api:
   def request(self, method, path, data=None, need_token=True, params={}):
     try:
       return self.request_once(method, path, data, need_token, params)
-    except ApiLimitError:
+    except (ApiLimitError, requests.exceptions.Timeout):
       print('oops, hit the limit. take a break')
       sleep(10)
       return self.request_once(method, path, data, need_token, params)
@@ -69,6 +69,7 @@ class Api:
     }
     result = self.request('post', 'register', data, need_token=False)
     token = mg(result, 'token')
+    self.agent.update(mg(result, 'agent'))
     self.agent.token = token
   
   def info(self):
@@ -79,7 +80,10 @@ class Api:
   def list_systems(self, page=1):
     data = self.request('get', 'systems', params={'page': page})
     #pprint(self.last_meta)
-    return self.store.update_list(System, data)
+    systems = self.store.update_list(System, data)
+    for s in data:
+      self.store.update_list(Waypoint, mg(s, 'waypoints'))
+    return systems
   
   def list_waypoints(self, system):
     data = self.request('get', f'systems/{system}/waypoints/')
@@ -87,13 +91,12 @@ class Api:
     return self.store.update_list(Waypoint, data)
   
   def marketplace(self, waypoint):
-    system = waypoint.system()
-    symbol = str(waypoint)
+    system = waypoint.system
     data = self.request('get', f'systems/{system}/waypoints/{waypoint}/market')
     return self.store.update(Marketplace, data)
   
   def jumps(self, waypoint):
-    data = self.request('get', f'systems/{waypoint.system()}/waypoints/{waypoint}/jump-gate')
+    data = self.request('get', f'systems/{waypoint.system}/waypoints/{waypoint}/jump-gate')
     symbol = str(waypoint)
     return self.store.update(Jumpgate, data, symbol)
   
@@ -221,8 +224,10 @@ class Api:
       return ship
     
   def jump(self, ship, system):
+    if type(system) == System:
+      system = system.symbol
     data = {
-      "systemSymbol": system.symbol
+      "systemSymbol": system
     }
     data = self.request('post', f'my/ships/{ship}/jump', data)
     if 'nav' in data:

nullptr/atlas_builder.py

@@ -31,12 +31,12 @@ class AtlasBuilder:
       if 'UNCHARTED' in w.traits:
         continue
       if 'MARKETPLACE' in w.traits:
-        self.api.marketplace(w)
         print(f'marketplace at {w}')
+        self.api.marketplace(w)
         sleep(0.5)
       if w.type == 'JUMP_GATE':
-        self.api.jumps(w)
         print(f'jumpgate at {w}')
+        self.api.jumps(w)
         
   def all_waypoints(self, systems):
     for s in systems:

nullptr/central_command.py

@@ -1,10 +1,13 @@
 from nullptr.store import Store
 from nullptr.models.ship import Ship
-from nullptr.mission import *
+from nullptr.missions import create_mission, get_mission_class
 from random import choice
 from time import sleep
 from threading import Thread
 
+class CentralCommandError(Exception):
+  pass
+  
 class CentralCommand:
   def __init__(self, store, api):
     self.missions = {}
@@ -27,14 +30,6 @@ class CentralCommand:
     mission = self.missions[ship]
     mission.step()
     return True
-  
-  def wait_for_stop(self):
-    try:
-      input()
-    except EOFError:
-      pass
-    self.stopping = True
-    print('stopping...')
     
   def run_interactive(self):
     print('auto mode. hit enter to stop')
@@ -43,7 +38,17 @@ class CentralCommand:
     t.start()
     self.run()
     print('manual mode')
-    
+  
+  
+  def wait_for_stop(self):
+    try:
+      input()
+    except EOFError:
+      pass
+    self.stopping = True
+    print('stopping...')
+  
+  
   def run(self):
     self.update_missions()
     while not self.stopping:
@@ -86,6 +91,21 @@ class CentralCommand:
         m = self.missions[s]
         m.next_step = max(s.cooldown, s.arrival)
   
+  def init_mission(self, s, mtyp):
+    if mtyp == 'none':
+      s.mission_state = {}
+      s.mission_status = None
+      s.mission = None
+      return
+    try:
+      mclass = get_mission_class(mtyp)
+    except ValueError:
+      raise CentralCommandError('no such mission')
+    s.mission = mtyp
+    s.mission_status = 'init'
+    s.mission_state = {k: v.default for k,v in mclass.params().items()}
+    self.start_mission(s)
+    
   def start_mission(self, s):
     mtype = s.mission
     m = create_mission(mtype, s, self.store, self.api)

nullptr/command_line.py

@@ -41,7 +41,7 @@ class CommandLine:
     print(f'command not found; {c}')
   
   def handle_error(self, cmd, args, e):
-    logging.error(e, exc_info=type(e).__name__ not in ['ApiError','CommandError'])
+    logging.error(e, exc_info=type(e).__name__ not in ['ApiError','CommandError', 'CentralCommandError'])
   
   def handle_empty(self):
     pass
@@ -90,5 +90,8 @@ class CommandLine:
       except EOFError:
         self.handle_eof()
         break
-      self.handle_cmd(c)
+      try:
+        self.handle_cmd(c)
+      except Exception as e:
+        logging.error(e, exc_info=True)
       

nullptr/commander.py

@@ -2,11 +2,7 @@ from nullptr.command_line import CommandLine
 from nullptr.store import Store
 from nullptr.analyzer import  Analyzer
 import argparse
-from nullptr.models.agent import Agent
-from nullptr.models.system import System
-from nullptr.models.waypoint import Waypoint
-from nullptr.models.marketplace import Marketplace
-from nullptr.models.jumpgate import Jumpgate
+from nullptr.models import *
 from nullptr.api import Api
 from .util import *
 from time import sleep, time
@@ -17,10 +13,8 @@ class CommandError(Exception):
   pass
   
 class Commander(CommandLine):
-  def __init__(self, store_dir='data'):
-    self.store_dir = store_dir
-    self.store = Store(store_dir)
-    self.store.load()
+  def __init__(self, store_file='data/store.npt'):
+    self.store = Store(store_file)
     self.agent = self.select_agent()
     self.api = Api(self.store, self.agent)
     self.atlas_builder = AtlasBuilder(self.store, self.api)
@@ -56,8 +50,23 @@ class Commander(CommandLine):
     agents = self.store.all(Agent)
     agent = next(agents, None)
     if agent is None:
-      symbol = input('agent name: ')
-      agent = self.store.get(Agent, symbol, create=True)
+      agent = self.agent_setup()
+    return agent
+  
+  def agent_setup(self):
+    symbol = input('agent name: ')
+    agent = self.store.get(Agent, symbol, create=True)
+    api = Api(self.store, agent)
+    self.api = api
+    faction = input('faction: ')
+    api.register(faction.upper().strip())
+    print('=== agent:')
+    print(agent)
+    print('=== ships')
+    self.do_ships('r')
+    print('=== contracts')
+    self.do_contracts('r')
+    self.store.flush()
     return agent
   
   def resolve(self, typ, arg):
@@ -82,13 +91,6 @@ class Commander(CommandLine):
   def do_auto(self):
     self.centcom.run_interactive()
   
-  def set_mission(self, arg=''):
-    if arg == 'none':
-      arg = None
-    self.ship.mission = arg
-    self.ship.mission_status = 'init'
-    self.centcom.start_mission(self.ship)
-  
   def print_mission(self):
     print(f'mission: {self.ship.mission} ({self.ship.mission_status})')
     pprint(self.ship.mission_state)
@@ -96,12 +98,15 @@ class Commander(CommandLine):
   def do_mission(self, arg=''):
     if not self.has_ship(): return 
     if arg:
-      self.set_mission(arg)
+      self.centcom.init_mission(self.ship, arg)
     self.print_mission()
   
-  def do_mset(self, args):
+  def do_mreset(self):
+    if not self.has_ship(): return 
+    self.ship.mission_state = {}
+    
+  def do_mset(self, nm, val):
     if not self.has_ship(): return
-    nm, val = args.split(' ')
     self.centcom.set_mission_param(self.ship, nm, val)
   
   def active_contract(self):
@@ -111,27 +116,72 @@ class Commander(CommandLine):
     
   def do_cmine(self):
     if not self.has_ship(): return 
-    site = self.ship.location_str
+    site = self.ship.location
     contract = self.active_contract()
     delivery = contract.unfinished_delivery()
     if delivery is None:
       raise CommandError('no delivery')
     resource = delivery['trade_symbol']
     destination = delivery['destination']
-    self.set_mission('mine')
+    self.centcom.init_mission(self.ship, 'mine')
     self.centcom.set_mission_param(self.ship, 'site', site)
     self.centcom.set_mission_param(self.ship, 'resource', resource)
-    self.centcom.set_mission_param(self.ship, 'destination', destination)
-    self.centcom.set_mission_param(self.ship, 'contract', contract.symbol)
+    self.centcom.set_mission_param(self.ship, 'dest', destination)
+    self.centcom.set_mission_param(self.ship, 'contract', contract)
+    self.print_mission()
+  
+  def do_chaul(self):
+    if not self.has_ship(): return     
+    if len(self.ship.cargo) > 0:
+      raise CommandError('please dump cargo first')
+    contract = self.active_contract()
+    delivery = contract.unfinished_delivery()
+    if delivery is None:
+      raise CommandError('no delivery')
+    resource = delivery['trade_symbol']
+    destination = delivery['destination']
+    m = self.analyzer.find_closest_markets(resource, 'buy', destination)
+    if len(m) == 0:
+      m = self.analyzer.find_closest_markets(resource, 'exchange', destination)
+    if len(m) == 0:
+      print('no market found')
+      return
+    _, m, _, _ = m[0]
+    site = self.store.get(Waypoint, m.symbol)
+    self.centcom.init_mission(self.ship, 'haul')
+    self.centcom.set_mission_param(self.ship, 'site', site)
+    self.centcom.set_mission_param(self.ship, 'resource', resource)
+    self.centcom.set_mission_param(self.ship, 'dest', destination)
+    self.centcom.set_mission_param(self.ship, 'contract', contract)
+    self.print_mission()
+  
+  def do_cprobe(self):
+    if not self.has_ship(): return     
+    contract = self.active_contract()
+    delivery = contract.unfinished_delivery()
+    if delivery is None:
+      raise CommandError('no delivery')
+    resource = delivery['trade_symbol']
+    destination = delivery['destination']
+    m = self.analyzer.find_closest_markets(resource, 'buy,exchange', destination)
+    if len(m) is None:
+      print('no market found')
+      return
+    markets = [ mkt[1] for mkt in m]
+    markets = self.analyzer.solve_tsp(markets)
+    self.centcom.init_mission(self.ship, 'probe')
+    self.centcom.set_mission_param(self.ship, 'hops', markets)
+    self.print_mission()
+  
+  def do_travel(self, dest):
+    dest = self.resolve('Waypoint', dest)
+    self.centcom.init_mission(self.ship, 'travel')
+    self.centcom.set_mission_param(self.ship, 'dest', dest)
     self.print_mission()
     
   def do_register(self, faction):
     self.api.register(faction.upper())
-    site = self.ship.location_str
-    contract = self.active_contract()
-    self.do_mission('mine')
-    self.centcom.set_mission_param(self.ship, 'site', site)
-    self.centcom.set_mission_param(self.ship, 'contract', contract)
+    pprint(self.api.agent)
     
   def do_universe(self, page=1):
     self.atlas_builder.run(page)
@@ -148,13 +198,17 @@ class Commander(CommandLine):
       total += num
       print(f'{num:5d} {nam}')
     print(f'{total:5d} total')
+  
+  def do_defrag(self):
+    self.store.defrag()
     
   def do_waypoints(self, system_str=''):
     if system_str == '':
       if not self.has_ship(): return
-      system = self.ship.location().system()
+      system = self.ship.location.system
     else:
       system = self.store.get(System, system_str)
+    print(f'=== waypoints in {system}')
     r = self.store.all_members(system, 'Waypoint')
     for w in r:
       traits = []
@@ -178,22 +232,22 @@ class Commander(CommandLine):
   def do_jumps(self, waypoint_str=None):
     if waypoint_str is None:
       if not self.has_ship(): return
-      waypoint = self.ship.location()
+      waypoint = self.ship.location
     else:
       waypoint = self.store.get(Waypoint, waypoint_str.upper())
     r = self.api.jumps(waypoint)
     pprint(r)
   
-  def do_query(self):
-    location = self.ask_obj(System, 'Where are you? ')
-    resource = input('what resource?').upper()
-    sellbuy = self.ask_multichoice(['sell','buy'], 'do you want to sell or buy?')
+  def do_query(self, resource):
+    if not self.has_ship(): return
+    location = self.ship.location
+    resource = resource.upper()
     print('Found markets:')
-    for m in self.analyzer.find_markets(resource, sellbuy):
-      system = self.store.get(System, m.system())
-      p = self.analyzer.find_path(location, system)
-      if p is None: continue
-      print(m, f'{len(p)-1} hops')
+    for typ, m, d, plen in self.analyzer.find_closest_markets(resource, 'buy,exchange',location):
+      price = '?'
+      if resource in m.prices:
+        price = m.prices[resource]['buy']
+      print(m, typ[0], f'{plen-1:3} hops {price}')
       
   def do_path(self):
     orig = self.ask_obj(System, 'from: ')
@@ -219,7 +273,7 @@ class Commander(CommandLine):
   
   def do_deliver(self):
     if not self.has_ship(): return 
-    site = self.ship.location_str
+    site = self.ship.location
     contract = self.active_contract()
     delivery = contract.unfinished_delivery()
     if delivery is None:
@@ -227,6 +281,10 @@ class Commander(CommandLine):
     resource = delivery['trade_symbol']
     self.api.deliver(self.ship, resource, contract)
     pprint(contract)
+  
+  def do_fulfill(self):
+    contract = self.active_contract()
+    self.api.fulfill(contract)
     
   def do_ship(self, arg=''):
     if arg != '':
@@ -244,7 +302,7 @@ class Commander(CommandLine):
   
   def do_go(self, arg):
     if not self.has_ship(): return
-    system = self.ship.location().system()
+    system = self.ship.location.system
     symbol = f'{system}-{arg}'
     dest = self.resolve('Waypoint', symbol)
     self.api.navigate(self.ship, dest)
@@ -278,7 +336,7 @@ class Commander(CommandLine):
   def do_market(self, arg=''):
     if arg == '':
       if not self.has_ship(): return
-      waypoint = self.ship.location()
+      waypoint = self.ship.location
     else:
       waypoint = self.resolve('Waypoint', arg)
     r =  self.api.marketplace(waypoint)
@@ -308,13 +366,15 @@ class Commander(CommandLine):
   
   def do_shipyard(self):
     if not self.has_ship(): return
-    location = self.ship.location()
-    pprint(self.api.shipyard(location))
+    location = self.ship.location
+    data = self.api.shipyard(location)
+    for s in must_get(data, 'ships'):
+      print(s['type'], s['purchasePrice'])
   
   def do_jump(self, system_str):
     if not self.has_ship(): return
     if '-' not in system_str:
-      sector = self.ship.location_str.split('-')[0]
+      sector = self.ship.location.system.sector.symbol
       system_str = f'{sector}-{system_str}'
     system = self.resolve('System', system_str)
     self.api.jump(self.ship, system)
@@ -322,7 +382,7 @@ class Commander(CommandLine):
     
   def do_purchase(self, ship_type):
     if not self.has_ship(): return
-    location = self.ship.location()
+    location = self.ship.location
     ship_type = ship_type.upper()
     if not ship_type.startswith('SHIP'):
       ship_type = 'SHIP_' + ship_type

nullptr/missions/__init__.py

@@ -0,0 +1,23 @@
+from nullptr.missions.survey import SurveyMission
+from nullptr.missions.mine import MiningMission
+from nullptr.missions.haul import HaulMission
+from nullptr.missions.travel import TravelMission
+from nullptr.missions.probe import ProbeMission
+
+def get_mission_class( mtype):
+  types = {
+    'survey': SurveyMission,
+    'mine': MiningMission,
+    'haul': HaulMission,
+    'travel': TravelMission,
+    'probe': ProbeMission
+  }
+  if mtype not in types:
+    raise ValueError(f'invalid mission type {mtype}')
+  return types[mtype]
+  
+def create_mission(mtype, ship, store, api):
+  typ = get_mission_class(mtype)
+  m = typ(ship, store, api)
+  return m
+

nullptr/missions/base.py

@@ -2,9 +2,12 @@ from nullptr.store import Store
 from nullptr.models.base import Base
 from nullptr.models.waypoint import Waypoint
 from nullptr.models.contract import Contract
+from nullptr.models.system import System
 from nullptr.models.survey import Survey
 from nullptr.models.ship import Ship
+from nullptr.analyzer import Analyzer
 from time import time
+from functools import partial
 import logging
 from nullptr.util import *
 
@@ -22,8 +25,15 @@ class MissionParam:
       return str(val)
     elif self.cls == int:
       return int(val)
+    elif self.cls == list:
+      if type(val) == str:
+        return [i.strip() for i in val.split(',')]
+      return val
     elif issubclass(self.cls, Base):
-      data = store.get(self.cls, val)
+      if type(val) == str:
+        data = store.get(self.cls, val)
+      else:
+        data = val
       if data is None:
         raise ValueError('object not found')
       return data.symbol
@@ -42,6 +52,7 @@ class Mission:
     self.store = store
     self.api = api
     self.next_step = 0
+    self.analyzer = Analyzer(self.store)
 
   def sts(self, nm, v):
     if issubclass(type(v), Base):
@@ -121,95 +132,21 @@ class Mission:
         self.status(next_step[result])
     print(f'{self.ship} {status} -> {self.status()}')
 
-
-class MiningMission(Mission): 
-  @classmethod
-  def params(cls):
-    return {
-      'site': MissionParam(Waypoint, True),
-      'resource': MissionParam(str, True),
-      'destination': MissionParam(Waypoint, True),
-      'delivery': MissionParam(str, True, 'deliver'),
-      'contract': MissionParam(Contract, False)
-    }
-  
-  def start_state(self):
-    return 'go_site'
-    
-  def steps(self):
-    return {
-      'extract': (self.step_extract, {
-        'done': 'dock',
-        'more': 'extract'
-      }),
-      'dock': (self.step_dock, 'sell'),
-      'sell': (self.step_sell, {
-        'more': 'sell',
-        'done': 'orbit',
-      }),
-      'orbit': (self.step_orbit, 'jettison'),
-      'jettison': (self.step_dispose, {
-        'more': 'jettison',
-        'done': 'extract',
-        'full': 'go_dest'
-      }),
-      'go_dest': (self.step_go_dest, 'dock_dest'),
-      'dock_dest': (self.step_dock, 'unload'),
-      'unload': (self.step_unload, {
-        'done': 'refuel',
-        'more': 'unload'
-      }),
-      'refuel': (self.step_refuel, 'orbit_dest'),
-      'orbit_dest': (self.step_orbit, 'go_site'),
-      'go_site': (self.step_go_site, 'extract')
-    }
-  
-  def get_survey(self):
-    resource = self.st('resource')
-    site = self.rst(Waypoint,'site')
-    # todo optimize
-    for s in self.store.all(Survey):
-      if resource in s.deposits and site.symbol == s.waypoint():
-        return s
-    return None
-    
-  def step_extract(self):
-    survey = self.get_survey()
-    print('using survey:', str(survey)) 
-    result = self.api.extract(self.ship, survey)
-    symbol = sg(result,'extraction.yield.symbol')
-    units  = sg(result,'extraction.yield.units')
-    print('extracted:', units, symbol)
-    self.next_step = self.ship.cooldown
-    if self.ship.cargo_units < self.ship.cargo_capacity:
-      return 'more'
-    else:
-      return 'done'
-    
-  def step_sell(self, except_resource=True):
-    target = self.st('resource')
-    market = self.store.get('Marketplace', self.ship.location_str)
-    sellables = market.sellable_items(self.ship.cargo.keys())
-    if target in sellables and except_resource:
-      sellables.remove(target)
-    if len(sellables) == 0:
-      return 'done'
-    self.api.sell(self.ship, sellables[0])
-    if len(sellables) == 1:
-      return 'done'
-    else:
-      return 'more'
-      
+class BaseMission(Mission):
   def step_go_dest(self):
     destination = self.rst(Waypoint, 'destination')
     if self.ship.location() == destination:
       return
     self.api.navigate(self.ship, destination)
     self.next_step = self.ship.arrival
-
-  def step_dock(self):
-    self.api.dock(self.ship)
-
+  
+  def step_go_site(self):
+    site = self.rst(Waypoint,'site')
+    if self.ship.location() == site:
+      return
+    self.api.navigate(self.ship, site)
+    self.next_step = self.ship.arrival
+    
   def step_unload(self):
     contract = self.rst(Contract, 'contract')
     delivery = self.st('delivery')
@@ -223,55 +160,96 @@ class MiningMission(Mission):
       return 'done'
     else:
       return 'more'
-  
-  def step_refuel(self):
-    self.api.refuel(self.ship)
-    
-  def step_dispose(self):
-    contract = self.rst(Contract, 'contract')
-    typs = self.ship.nondeliverable_cargo(contract)
-    if len(typs) > 0:
-      self.api.jettison(self.ship, typs[0])
-    if len(typs) > 1:
-      return 'more'
-    elif self.ship.cargo_units > self.ship.cargo_capacity - 3:
-      return 'full'
-    else:
+
+  def step_sell(self, except_resource=True):
+    target = self.st('resource')
+    market = self.store.get('Marketplace', self.ship.location_str)
+    sellables = market.sellable_items(self.ship.cargo.keys())
+    if target in sellables and except_resource:
+      sellables.remove(target)
+    if len(sellables) == 0:
       return 'done'
+    self.api.sell(self.ship, sellables[0])
+    if len(sellables) == 1:
+      return 'done'
+    else:
+      return 'more'
   
+  def step_load(self):
+    cargo_space = self.ship.cargo_capacity - self.ship.cargo_units
+    resource = self.st('resource')
+    self.api.buy(self.ship, resource, cargo_space)
   
+  def step_travel(self):
+    traject = self.st('traject')
+    if traject is None or traject == []:
+      return 'done'
+    dest = traject[-1]
+    loc = self.ship.location
+    if dest == loc:
+      self.sts('traject', None)
+      return 'done'
+    hop = traject.pop(0)
+    if type(hop) == Waypoint:
+      self.api.navigate(self.ship, hop)
+      self.next_step = self.ship.arrival
+    else:
+      self.api.jump(self.ship, hop)
+      self.next_step = self.ship.cooldown
+    if traject == []:
+      traject= None
+    self.sts('traject', traject)
+    return 'more'
+  
+  def step_calculate_traject(self, dest):
+    if type(dest) == str:
+      dest = self.store.get(Waypoint, dest)
+    loc = self.ship.location
+    loc_sys = loc.system
+    loc_jg = self.analyzer.get_jumpgate(loc_sys)
+    loc_jg_wp = self.store.get(Waypoint, loc_jg.symbol)
+    dest_sys = dest.system
+    dest_jg = self.analyzer.get_jumpgate(dest_sys)
+    if dest_sys == loc_sys:
+      result = [dest]
+      self.sts('traject', result)
+      return
+    path = self.analyzer.find_path(loc_sys, dest_sys)
+    result = []
+    if loc.symbol != loc_jg.symbol:
+      result.append(loc_jg_wp)
+    result += [s for s in path[1:]]
+    if dest_jg.symbol != dest.symbol:
+      result.append(dest)
+    self.sts('traject', result)
+    print(result)
+    return result
+    
+  def step_dock(self):
+    self.api.dock(self.ship)
+ 
+  def step_refuel(self):
+    if self.ship.fuel_capacity == 0:
+      return
+    if self.ship.fuel_current / self.ship.fuel_capacity < 0.5:
+      try:
+        self.api.refuel(self.ship)
+      except Exception as e:
+        pass
+       
   def step_orbit(self):
     self.api.orbit(self.ship)
-
-  def step_go_site(self):
-    site = self.rst(Waypoint,'site')
-    if self.ship.location() == site:
-      return
-    self.api.navigate(self.ship, site)
-    self.next_step = self.ship.arrival
-    
-class SurveyMission(Mission):
-  def start_state(self):
-    return 'survey'
   
-    
-  def steps(self):
+  def travel_steps(self, nm, destination, next_step):
+    destination = self.st(destination)
+    calc = partial(self.step_calculate_traject, destination)
     return {
-      'survey': (self.step_survey, 'survey')
+      f'travel-{nm}': (self.step_orbit, f'calc-trav-{nm}'),
+      f'calc-trav-{nm}': (calc, f'go-{nm}'),
+      f'go-{nm}': (self.step_travel, {
+        'done': f'dock-{nm}',
+        'more': f'go-{nm}'
+      }),
+      f'dock-{nm}': (self.step_dock, f'refuel-{nm}'),
+      f'refuel-{nm}': (self.step_refuel, next_step)
     }
-  
-  def step_survey(self):
-    result = self.api.survey(self.ship)
-    #pprint(result, 2)
-    self.next_step = self.ship.cooldown
-
-def create_mission(mtype, ship, store, api):
-  types = {
-    'survey': SurveyMission,
-    'mine': MiningMission
-  }
-  if mtype not in types:
-    logging.warning(f'invalid mission type {mtype}')
-    return
-  m = types[mtype](ship, store, api)
-  return m

nullptr/missions/haul.py

@@ -0,0 +1,25 @@
+from nullptr.missions.base import BaseMission, MissionParam
+from nullptr.models.waypoint import Waypoint
+from nullptr.models.survey import Survey
+from nullptr.models.contract import Contract
+class HaulMission(BaseMission):
+  def start_state(self):
+    return 'travel-to'
+  
+  @classmethod
+  def params(cls):
+    return {
+      'site': MissionParam(Waypoint, True),
+      'resource': MissionParam(str, True),
+      'dest': MissionParam(Waypoint, True),
+      'delivery': MissionParam(str, True, 'deliver'),
+      'contract': MissionParam(Contract, False)
+    }
+  
+  def steps(self):
+    return {
+      **self.travel_steps('to', 'site', 'load'),
+      'load': (self.step_load, 'travel-back'),
+      **self.travel_steps('back', 'dest', 'unload'),
+      'unload': (self.step_unload, 'travel-to'),
+    }

nullptr/missions/mine.py

@@ -0,0 +1,80 @@
+from nullptr.missions.base import BaseMission, MissionParam
+from nullptr.models.waypoint import Waypoint
+from nullptr.models.survey import Survey
+from nullptr.models.contract import Contract
+from nullptr.util import *
+
+class MiningMission(BaseMission): 
+  @classmethod
+  def params(cls):
+    return {
+      'site': MissionParam(Waypoint, True),
+      'resource': MissionParam(str, True),
+      'dest': MissionParam(Waypoint, True),
+      'delivery': MissionParam(str, True, 'deliver'),
+      'contract': MissionParam(Contract, False)
+    }
+  
+  def start_state(self):
+    return 'travel-to'
+    
+  def steps(self):
+    return {
+      **self.travel_steps('to', 'site', 'orbit1'),
+      'orbit1': (self.step_orbit, 'extract'),
+      'extract': (self.step_extract, {
+        'done': 'dock',
+        'more': 'extract'
+      }),
+      'dock': (self.step_dock, 'sell'),
+      'sell': (self.step_sell, {
+        'more': 'sell',
+        'done': 'orbit2',
+      }),
+      'orbit2': (self.step_orbit, 'jettison'),
+      'jettison': (self.step_dispose, {
+        'more': 'jettison',
+        'done': 'extract',
+        'full': 'travel-back'
+      }),
+      **self.travel_steps('back', 'dest', 'unload'),
+      'unload': (self.step_unload, {
+        'done': 'travel-to',
+        'more': 'unload'
+      }),
+    }
+  
+  def get_survey(self):
+    resource = self.st('resource')
+    site = self.rst(Waypoint,'site')
+    # todo optimize
+    for s in self.store.all(Survey):
+      if resource in s.deposits and site.symbol == s.waypoint():
+        return s
+    return None
+    
+  def step_extract(self):
+    survey = self.get_survey()
+    print('using survey:', str(survey)) 
+    result = self.api.extract(self.ship, survey)
+    symbol = sg(result,'extraction.yield.symbol')
+    units  = sg(result,'extraction.yield.units')
+    print('extracted:', units, symbol)
+    self.next_step = self.ship.cooldown
+    if self.ship.cargo_units < self.ship.cargo_capacity:
+      return 'more'
+    else:
+      return 'done'
+  
+  def step_dispose(self):
+    contract = self.rst(Contract, 'contract')
+    typs = self.ship.nondeliverable_cargo(contract)
+    if len(typs) > 0:
+      self.api.jettison(self.ship, typs[0])
+    if len(typs) > 1:
+      return 'more'
+    elif self.ship.cargo_units > self.ship.cargo_capacity - 3:
+      return 'full'
+    else:
+      return 'done'
+  

nullptr/missions/probe.py

@@ -0,0 +1,33 @@
+from nullptr.missions.base import BaseMission, MissionParam
+from nullptr.models.waypoint import Waypoint
+
+class ProbeMission(BaseMission):
+  def start_state(self):
+    return 'next-hop'
+  
+  @classmethod
+  def params(cls):
+    return {
+      'hops': MissionParam(list, True),
+      'next-hop': MissionParam(int, True, 0)
+    }
+  
+  def steps(self):
+    return {
+      'next-hop': (self.step_next_hop, 'travel-to'),
+      **self.travel_steps('to', 'site', 'market'),
+      'market': (self.step_market, 'next-hop'),
+ 
+    }
+  
+  def step_market(self):
+    loc = self.ship.location()
+    self.api.marketplace(loc)
+  
+  def step_next_hop(self):
+    hops = self.st('hops')
+    next_hop = self.st('next-hop')
+    hop = hops[next_hop]
+    self.sts('site', hop)
+    self.sts('next-hop', (next_hop+1) % len(hops))
+    

nullptr/missions/survey.py

@@ -0,0 +1,15 @@
+from nullptr.missions.base import BaseMission, MissionParam
+
+class SurveyMission(BaseMission):
+  def start_state(self):
+    return 'survey'
+  
+  def steps(self):
+    return {
+      'survey': (self.step_survey, 'survey')
+    }
+  
+  def step_survey(self):
+    result = self.api.survey(self.ship)
+    #pprint(result, 2)
+    self.next_step = self.ship.cooldown

nullptr/missions/travel.py

@@ -0,0 +1,16 @@
+from nullptr.missions.base import BaseMission, MissionParam
+from nullptr.models.waypoint import Waypoint
+
+class TravelMission(BaseMission):
+  def start_state(self):
+    return 'travel-to'
+  
+  @classmethod
+  def params(cls):
+    return {
+      'dest': MissionParam(Waypoint, True)
+    }
+  
+  def steps(self):
+    return self.travel_steps('to', 'dest', 'done')
+    

nullptr/models/__init__.py

@@ -0,0 +1,12 @@
+from nullptr.models.base import Base
+from nullptr.models.waypoint import Waypoint
+from nullptr.models.sector import Sector
+from nullptr.models.system import System
+from nullptr.models.agent import Agent
+from nullptr.models.marketplace import Marketplace
+from nullptr.models.jumpgate import Jumpgate
+from nullptr.models.ship import Ship
+from nullptr.models.contract import Contract
+from nullptr.models.survey import Survey
+
+__all__ = [ 'Waypoint', 'Sector', 'Ship', 'Survey', 'System', 'Agent', 'Marketplace', 'Jumpgate', 'Contract', 'Base' ]

nullptr/models/agent.py

@@ -8,9 +8,6 @@ class Agent(Base):
   def update(self, d):
     self.seta('credits', d)
     
-  def path(self):
-    return f'{self.symbol}.{self.ext()}'
-  
   @classmethod
   def ext(self):
     return 'agt'

nullptr/models/base.py

@@ -1,17 +1,38 @@
 from copy import deepcopy
 from nullptr.util import sg
 
+class Reference:
+  def __init__(self, typ, symbol, store):
+    self.typ = typ
+    self.symbol = symbol
+    self.store = store
+  
+  @classmethod
+  def create(cls, obj):
+    o = cls(type(obj), obj.symbol, obj.store)
+    return o
+    
+  def resolve(self):
+    return self.store.get(self.typ, self.symbol)
+    
+  def __repr__(self):
+    return f'*REF*{self.symbol}.{self.typ.ext()}'
+
 class Base:
   identifier = 'symbol'
-  symbol: str
-  store: object
   
   def __init__(self, symbol, store):
-    self.disable_dirty = False
+    self.disable_dirty = True
+    self.file_offset = None
     self.store = store
     self.symbol = symbol
     self.define()
-  
+    self.disable_dirty = False
+    
+  @classmethod
+  def ext(cls):
+    raise NotImplementedError('no ext')
+    
   def define(self):
     pass
     
@@ -19,7 +40,13 @@ class Base:
     return hash((str(type(self)), self.symbol))
   
   def __eq__(self, other):
-    return self.symbol == other.symbol and type(self) == type(other)
+    return type(self) == type(other) and self.symbol == other.symbol
+  
+  def get_system(self):
+    parts = self.symbol.split('-')
+    system_str = f'{parts[0]}-{parts[1]}'
+    system = self.store.get('System', system_str, create=True)
+    return system
     
   def seta(self, attr,  d, name=None, interp=None):
     if name is None:
@@ -30,17 +57,30 @@ class Base:
         val = interp(val)
       setattr(self, attr, val)
   
-  def setlst(self, attr, d, name, member):
+  def setlst(self, attr, d, name, member, interp=None):
     val = sg(d, name)
     if val is not None:
-      lst = [sg(x, member) for x in val]
+      lst = []
+      for x in val:
+        val = sg(x, member)
+        if interp is not None:
+          val = interp(val)
+        lst.append(val)
       setattr(self, attr, lst)
     
   def __setattr__(self, name, value):
-    if name not in ['symbol','store','disable_dirty'] and not self.disable_dirty:
+    if name not in ['symbol','store','disable_dirty', 'file_offset'] and not self.disable_dirty:
       self.store.dirty(self)
+    if issubclass(type(value), Base):
+      value = Reference.create(value)
     super().__setattr__(name, value)
   
+  def __getattribute__(self, nm):
+    val = super().__getattribute__(nm)
+    if type(val) == Reference:
+      val = val.resolve()
+    return val
+  
   def update(self, d):
     pass
   
@@ -52,27 +92,15 @@ class Base:
     self.__dict__.update(d)
     self.disable_dirty = False
     
-  def dict(self):
-    r = {}
-    for k,v in self.__dict__.items():
-      if k in ['store']:
-        continue
-      r[k] = deepcopy(v)
-    return r
-    
-  def path(self):
-    raise NotImplementedError('path')
-  
-  @classmethod
-  def ext(self):
-    raise NotImplementedError('extension')
-  
   def type(self):
     return self.__class__.__name__
   
   def __str__(self):
     return self.f()
     
+  def __repr__(self):
+    return self.f()
+    
   def f(self, detail=1):
     r = self.symbol
     if detail > 1:

nullptr/models/contract.py

@@ -18,9 +18,6 @@ class Contract(Base):
   def ext(cls):
     return 'cnt'
   
-  def path(self):
-    return f'contracts/{self.symbol}.{self.ext()}'
-  
   def is_expired(self):
     return time() > self.expires
   

nullptr/models/jumpgate.py

@@ -1,14 +1,17 @@
-from .system_member import SystemMember
+from .base import Base
+from .system import System
 from dataclasses import field
 
-class Jumpgate(SystemMember):
+class Jumpgate(Base):
   def define(self):
     self.range: int = 0
     self.faction: str = ''
     self.systems: list = []
+    self.system = self.get_system()
   
   def update(self, d):
-    self.setlst('systems', d, 'connectedSystems', 'symbol')
+    getter = self.store.getter(System, create=True)
+    self.setlst('systems', d, 'connectedSystems', 'symbol', interp=getter)
     self.seta('faction', d, 'factionSymbol')
     self.seta('range', d, 'jumpRange')
     
@@ -16,13 +19,9 @@ class Jumpgate(SystemMember):
   def ext(self):
     return 'jmp'
   
-  def path(self):
-    sector, system, _ = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'
-  
   def f(self, detail=1):
     r = self.symbol
     if detail > 1:
       r += '\n'
-      r += '\n'.join(self.systems)
+      r += '\n'.join([s.symbol for s in self.systems])
     return r

nullptr/models/marketplace.py

@@ -1,16 +1,23 @@
 
-from .system_member import SystemMember
+from .base import Base
 from time import time
 from nullptr.util import *
 from dataclasses import field
+from nullptr.models import Waypoint
 
-class Marketplace(SystemMember):
+class Marketplace(Base):
   def define(self):
     self.imports:list = []
     self.exports:list = []
     self.exchange:list = []
     self.prices:dict = {}
     self.last_prices:int = 0
+    self.set_waypoint()
+    self.system = self.get_system()
+  
+  def set_waypoint(self):
+    waypoint = self.store.get(Waypoint, self.symbol, create=True)
+    self.waypoint = waypoint
   
   def update(self, d):
     self.setlst('imports', d, 'imports', 'symbol')
@@ -25,6 +32,7 @@ class Marketplace(SystemMember):
         price['symbol'] = symbol
         price['buy'] = mg(g, 'purchasePrice')
         price['sell'] = mg(g, 'sellPrice')
+        price['volume'] = mg(g, 'tradeVolume')
         prices[symbol] = price
       self.prices = prices
   
@@ -43,10 +51,6 @@ class Marketplace(SystemMember):
     if r in self.exchange:
       return 'X'
     return '?'
-    
-  def path(self):
-    sector, system, _ = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'
   
   def f(self, detail=1):
     r = self.symbol

nullptr/models/ship.py

@@ -1,7 +1,7 @@
 from .base import Base
 from time import time
 from nullptr.util import *
-from dataclasses import dataclass, field
+from nullptr.models import Waypoint
 
 class Ship(Base):  
   def define(self):
@@ -10,7 +10,7 @@ class Ship(Base):
     self.status:str = ''
     self.cargo_capacity:int = 0
     self.cargo_units:int = 0
-    self.location_str = ''
+    self.location = None
     self.cooldown:int = 0
     self.arrival:int = 0
     self.fuel_current:int = 0
@@ -22,16 +22,10 @@ class Ship(Base):
   def ext(self):
     return 'shp'
   
-  def location(self):
-    return self.store.get('Waypoint', self.location_str)
-    
-  def path(self):
-    agent = self.symbol.split('-')[0]
-    return f'{agent}/{self.symbol}.{self.ext()}'
-
   def update(self, d):
     self.seta('status', d, 'nav.status')
-    self.seta('location_str', d, 'nav.waypointSymbol')
+    getter = self.store.getter(Waypoint, create=True)
+    self.seta('location', d, 'nav.waypointSymbol', interp=getter)
     self.seta('cargo_capacity', d, 'cargo.capacity')
     self.seta('cargo_units', d, 'cargo.units')
     self.seta('fuel_capacity', d, 'fuel.capacity')
@@ -95,7 +89,7 @@ class Ship(Base):
     if detail > 1:
       r += ' ' + self.status
       r += f' [{self.fuel_current}/{self.fuel_capacity}]'
-      r += ' ' + str(self.location())
+      r += ' ' + str(self.location)
       if self.is_travelling():
         r += f' [A: {arrival}]'
       if self.is_cooldown():

nullptr/models/survey.py

@@ -1,10 +1,10 @@
 from time import time
 from nullptr.util import *
-from .system_member import SystemMember
+from .base import Base
 
 size_names = ['SMALL','MODERATE','LARGE']
 
-class Survey(SystemMember):
+class Survey(Base):
   identifier = 'signature'
   def define(self):
     self.type: str = ''
@@ -17,12 +17,7 @@ class Survey(SystemMember):
   @classmethod
   def ext(cls):
     return 'svy'
-  
-  def path(self):
-    sector, system, waypoint, signature = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'
-    
-  
+
   def is_expired(self):
     return time() > self.expires or self.exhausted
   

nullptr/models/system.py

@@ -17,10 +17,6 @@ class System(Base):
   def ext(self):
     return 'stm'
   
-  def path(self):
-    sector, symbol = self.symbol.split('-')
-    return f'atlas/{sector}/{symbol[0:1]}/{self.symbol}.{self.ext()}'
-  
   def distance(self, other):
     return int(sqrt((self.x - other.x) ** 2 + (self.y - other.y) ** 2))
   

nullptr/models/system_member.py

@@ -1,10 +0,0 @@
-from .base import Base
-
-class SystemMember(Base):
-  @classmethod
-  def ext(cls):
-    return 'obj'
-    
-  def system(self):
-    p = self.symbol.split('-')
-    return f'{p[0]}-{p[1]}'

nullptr/models/waypoint.py

@@ -1,26 +1,24 @@
-from .system_member import SystemMember
+from .base import Base, Reference
+from nullptr.models.system import System
 from nullptr.util import *
-from dataclasses import field
 
-class Waypoint(SystemMember):
+class Waypoint(Base):
   def define(self):
     self.x:int = 0
     self.y:int = 0
     self.type:str = 'unknown'
     self.traits:list = []
     self.faction:str = ''
-  
+    self.system = self.get_system()
+    
   def update(self, d):
     self.seta('x', d)
     self.seta('y', d)
     self.seta('type', d)
     self.seta('faction', d, 'faction.symbol')
     self.setlst('traits', d, 'traits', 'symbol')
-    
+  
   @classmethod
   def ext(self):
     return 'way'
     
-  def path(self):
-    sector, system, _ = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'

nullptr/store.py

@@ -1,30 +1,74 @@
-from nullptr.models.base import Base
-from nullptr.models.waypoint import Waypoint
-from nullptr.models.sector import Sector
-from nullptr.models.system import System
-from nullptr.models.agent import Agent
-from nullptr.models.marketplace import Marketplace
-from nullptr.models.system_member import SystemMember
-from nullptr.models.jumpgate import Jumpgate
-from nullptr.models.ship import Ship
-from nullptr.models.contract import Contract
-from nullptr.models.survey import Survey
+from nullptr.models import *
 from os.path import isfile, dirname, isdir
 import os
 from os.path import basename
 import json
 from .util import *
 from time import time
+import pickle
+from struct import unpack, pack
+from functools import partial
+from io import BytesIO
 
+class StorePickler(pickle.Pickler):
+  def persistent_id(self, obj):
+    return "STORE" if type(obj) == Store else None
+
+class StoreUnpickler(pickle.Unpickler):
+  def __init__(self, stream, store):
+    self.store = store
+    super().__init__(stream)
+    
+  def persistent_load(self, pers_id):
+    if pers_id == "STORE":
+      return self.store
+    raise pickle.UnpicklingError("I don know the persid!")
+     
+        
+class ChunkHeader:
+  def __init__(self):
+    self.offset = 0
+    self.in_use = True
+    self.size = 0
+    self.used = 0
+    
+  @classmethod
+  def parse(cls, fil):
+    offset = fil.tell()
+    d = fil.read(16)
+    if len(d) < 16:
+      return None
+    o = cls()
+    o.offset = offset
+    d, o.used = unpack('<QQ', d)
+    o.size =   d & 0x7fffffffffffffff
+    o.in_use = d & 0x8000000000000000 != 0
+    # print(o)
+    return o
+  
+  def write(self, f):
+    d = self.size
+    if self.in_use:
+      d |= 1 << 63
+    d = pack('<QQ', d, self.used)
+    f.write(d)
+  
+  def __repr__(self):
+    return f'chunk {self.in_use} {self.size} {self.used}'
+    
 class Store:
-  def __init__(self, data_dir):
+  def __init__(self, data_file):
     self.init_models()
-    self.data_dir = data_dir
+    self.fil = open_file(data_file)
     self.data = {m: {} for m in self.models}
     self.system_members = {}
     self.dirty_objects = set()
     self.cleanup_interval = 600
     self.last_cleanup = 0
+    self.slack = 0.1
+    self.slack_min = 64
+    self.slack_max = 1024
+    self.load()
   
   def init_models(self):
     self.models = all_subclasses(Base)
@@ -33,54 +77,113 @@ class Store:
     
   def dirty(self, obj):
     self.dirty_objects.add(obj)
-    
-  def path(self, obj):
-    return os.path.join(self.data_dir, obj.path())
-    
-  def load_file(self, path):
-    if not isfile(path):
-      return None
-    fn = basename(path)
-    ext = fn.split('.')[-1]
-    symbol = fn.split('.')[0]
-    if ext not in self.extensions:
-      return None
-    with open(path) as f:
-      data = json.load(f)
-    
-    typ = self.extensions[ext]
-    obj = self.create(typ, symbol)
-    obj.load(data)
-    obj.store = self
-    return obj
   
+  def dump_object(self, obj):
+    buf = BytesIO()
+    p = StorePickler(buf)
+    p.dump(obj)
+    return buf.getvalue()
+  
+  def load_object(self, data, offset):
+    buf = BytesIO(data)
+    p = StoreUnpickler(buf, self)
+    obj = p.load()
+    obj.file_offset = offset
+    obj.disable_dirty = False
+    self.hold(obj)
+    
   def load(self):
     cnt = 0
     start_time = time()
-    for fil in list_files(self.data_dir, True):
-      self.load_file(fil)
+    total = 0
+    free = 0
+    self.fil.seek(0)
+    offset = 0
+    while (hdr := ChunkHeader.parse(self.fil)):
+      # print(hdr)
+      total += hdr.size
+      if not hdr.in_use: 
+        self.fil.seek(hdr.size, 1)
+        free += hdr.size
+        continue
+      data = self.fil.read(hdr.used)
+      self.load_object(data, offset)
+      self.fil.seek(hdr.size - hdr.used, 1)
+      offset = self.fil.tell()
       cnt += 1
+      
     dur = time() - start_time
     print(f'loaded {cnt} objects in {dur:.2f} seconds')
+    print(f'Fragmented space: {free} / {total} bytes')
   
+  def allocate_chunk(self, sz):
+    used = sz
+    slack = sz * self.slack
+    slack = min(slack, self.slack_max)
+    slack = max(slack, self.slack_min)
+    sz += int(slack)
+    self.fil.seek(0, 2)
+    offset = self.fil.tell()
+    h = ChunkHeader()
+    h.size = sz
+    h.used = used
+    h.offset = self.fil.tell()
+    h.write(self.fil)
+    return offset, h
+    
+  def purge(self, obj):
+    if obj.file_offset is None:
+      return
+    self.fil.seek(obj.file_offset)
+    hdr = ChunkHeader.parse(self.fil)
+    hdr.in_use = False
+    self.fil.seek(obj.file_offset)
+    hdr.write(self.fil)
+    obj.file_offset = None
+    
   def store(self, obj):
-    path = self.path(obj)
-    path_dir = dirname(path)
-    data = obj.dict()
-    if not isdir(path_dir):
-      os.makedirs(path_dir, exist_ok=True)
-    with open(path, 'w') as f:
-      json.dump(data, f, indent=2)
-  
-  def create(self, typ, symbol):
-    obj = typ(symbol, self)
+    data = self.dump_object(obj)
+    osize = len(data)
+    # is there an existing chunk for this obj?
+    if obj.file_offset is not None:
+      # read chunk hdr
+      self.fil.seek(obj.file_offset)
+      hdr = ChunkHeader.parse(self.fil)
+      csize = hdr.size
+      # if the chunk is too small
+      if csize < osize:
+        # free the chunk
+        hdr.in_use = False
+        # force a new chunk
+        obj.file_offset = None
+      else:
+        # if it is big enough, update the used field
+        hdr.used = osize
+      self.fil.seek(hdr.offset)
+      hdr.write(self.fil)
+        
+    if obj.file_offset is None:
+      obj.file_offset, hdr = self.allocate_chunk(osize)
+    # print(type(obj).__name__, hdr)
+    self.fil.write(data)
+    slack = b'\x00' * (hdr.size - hdr.used)
+    self.fil.write(slack)
+    
+  def hold(self, obj):
+    typ = type(obj)
+    symbol = obj.symbol
+    obj.store = self
     self.data[typ][symbol] = obj
-    if issubclass(typ, SystemMember):
-      system_str = obj.system()
-      
+    if hasattr(obj, 'system') and obj.system != None:
+      system_str = obj.system.symbol
       if system_str not in self.system_members:
         self.system_members[system_str] = set()
       self.system_members[system_str].add(obj)
+  
+  def create(self, typ, symbol):
+    obj = typ(symbol, self)
+    self.hold(obj)
+    self.dirty(obj)
     return obj
     
   def get(self, typ, symbol, create=False):
@@ -96,6 +199,11 @@ class Store:
         return None
     return self.data[typ][symbol]
   
+  def getter(self, typ, create=False):
+    if type(typ) == str and typ in self.model_names:
+      typ = self.model_names[typ]
+    return partial(self.get, typ, create=create)
+  
   def update(self, typ, data, symbol=None):
     if type(typ) == str and typ in self.model_names:
       typ = self.model_names[typ] 
@@ -113,6 +221,9 @@ class Store:
       typ = self.model_names[typ]
     
     for m in self.data[typ].values():
+      if m.is_expired():
+        self.dirty(m)
+        continue
       yield m
   
   def all_members(self, system, typ=None):
@@ -121,13 +232,22 @@ class Store:
     
     if type(system) == System:
       system = system.symbol
-  
+    
     if system not in self.system_members:
       return 
+    
+    garbage = set()
     for m in self.system_members[system]:
+      if m.is_expired():
+        self.dirty(m)
+        garbage.add(m)
+        continue
       if typ is None or type(m) == typ:
         yield m
     
+    for m in garbage:
+      self.system_members[system].remove(m)
+      
   def cleanup(self):
     if time() < self.last_cleanup + self.cleanup_interval:
       return
@@ -138,9 +258,8 @@ class Store:
         if o.is_expired():
           expired.append(o)
     for o in expired:
-      path = o.path()
-      if isfile(path):
-        os.remove(path)
+      self.purge(obj)
+      
       del self.data[type(o)][o.symbol]
     dur = time() - start_time
     # print(f'cleaned {len(expired)} in {dur:.03f} seconds')
@@ -151,7 +270,21 @@ class Store:
     start_time = time()
     for obj in self.dirty_objects:
       it += 1
-      self.store(obj)
+      if obj.is_expired():
+        self.purge(obj)
+      else:
+        self.store(obj)
+    self.fil.flush()
     self.dirty_objects = set()
     dur = time() - start_time
     #  print(f'flush done {it} items {dur:.2f}')
+  
+  def defrag(self):
+    nm = self.fil.name
+    self.fil.close()
+    os.rename(nm, nm + '.bak')
+    self.fil = open(nm, 'ab+')
+    for t in self.data:
+      for o in self.all(t):
+        o.file_offset = None
+        self.store(o)

nullptr/util.py

@@ -1,21 +1,16 @@
 from datetime import datetime
 from math import ceil
 import os
-from os.path import isfile
+from os.path import isfile, dirname
 
-def list_files(path, recursive=False):
-  if recursive:
-   for p, dirnames, fils in os.walk(path):
-     for f in fils:
-       fil = os.path.join(p, f)
-       yield fil
+def open_file(fn):
+  d = dirname(fn)
+  os.makedirs(d, exist_ok=True)
+  if isfile(fn):
+    return open(fn, 'rb+')
   else:
-    for f in os.listdir(path):
-      fil = os.path.join(path, f)
-      if not isfile(fil):
-        continue
-      yield fil
-      
+    return open(fn, 'ab+')
+
 def must_get(d, k):
   if type(k) == str:
     k = k.split('.')
@@ -58,7 +53,7 @@ def pretty(d, ident=0, detail=2):
     return d.f(detail)
   r = ''
   idt = ' ' * ident
-  if type(d) == list:
+  if type(d) in [list, set]:
     r += 'lst'
     for i in d:
       r += '\n' + idt + pretty(i, ident + 1, detail)

store.md

@@ -0,0 +1,50 @@
+# The store format
+This project uses a custom database format just because we can.
+
+The script reads the entire store on startup. Each object can br altered in memory. A 'dirty' status is set when an object is changed. periodically the script will flush the store, writing all changes back to disk.
+
+The store disk format is optimized for two things:
+* Loading the entire contents in memory
+* Writing changed objects back to disk
+
+## Objects
+First lets discuss what we are storing.
+
+Objects are identified by type and symbol. A symbol is an uppercase alphanumeric string that optionally contains '-' chars. the type is a lowecase alphanumeric string of length 3.
+
+For each type, the store has a class defined that is used to load objects of that type.
+
+An object identifier is its symbol and type joined with a '.' character.
+
+Some examples of object identifiers:
+
+* X1-J84.sys
+* X1-J84-0828772.way
+* CAPT-J-1.shp
+
+A waypoint is always part of a system. This is also visible because the waypoint symbol is prefixed by the system symbol. However, this relation is not enforced or used by the store. The symbol is an opaque string.
+
+An object has attributes. Values of attributes can be strings, ints, floats, bools, lists, dicts and references. lists and dicts can also only contain the values listed. References are pointers to other objects in the store.
+
+## Indices
+An index is a dict with a string as key and a list of objects as value. The dict is built when loading the store. when the index is iterated, each object is re-checked and removed if necessary.
+
+## API
+
+* store.load(fil) loads all objects
+* store.get(type, symbol, create=False) fetches the object. If create==False: None if it wasnt present
+* store.all(type) generator for all objects of a goven type
+* store.purge(obj)
+* store.clean() removes all expired objects
+* store.flush() writes all dirty objects to disk
+* store.defrag() consolidates the store file to minimize storage and loading time
+
+type may be a class or a string containing the name of a class. The type should be a subclass of models.base.Base
+
+# file format
+Until specified otherwise, all numbers are stored low-endian 64bit unsigned.
+
+The store file is built up out of chunks. A chunk is either empty or houses exactly one file. If a file is updated and its size fits the chunk, it is updated in-place. If the new content does not fit the chunk, a new chunk is allocated at the end of the file. The old chunk is marked as empty.
+
+A chunk starts with a chunk header. This is just a single field describing the size of the chunk in bytes, not including the header. The first bit of the field is the IN_USE flag. If it is not set, the contents of the chunk are ignored during loading.
+