fix the generals problems

Heads had to roll

Dockerfile

@@ -8,5 +8,6 @@ RUN pip3 install -r requirements.txt
 ADD --chown=user . /app
 RUN chmod +x /app/main.py
 VOLUME /data
-ENTRYPOINT [ "python3", "/app/main.py"]
-CMD ["-s", "/data/"]
+#ENTRYPOINT bash
+RUN echo "python3 /app/main.py -d /data" > ~/.bash_history
+CMD ["/bin/sh", "-c", "python3 /app/main.py -d /data ; bash -i"]

Readme.md

@@ -0,0 +1,83 @@
+# 0ptr script for spacetraders.io
+This is my script for running spacetraders.io. It has been completely written and operated from an iphone.
+
+Because i'm using a less-than-optimal development platform, some 'best-practice' code standards have been thrown out of the window. For instance, some variables have obscure names just to keep them short. 
+
+The script offers a command-line interface allowing you to do most ST operations by hand. More importantly, it offers the functionality to assign missions to ships and execute them automatically.
+
+# Getting started
+I'm using the [pythonista app](http://omz-software.com/pythonista/) to develop and test the script. Just open main.py and hit run.
+
+To keep the script running for longer periods of time, I'm using a VPS with docker. Start the script like so:
+
+`docker run -v 0ptr:/data -ti 0ptr`
+
+The script will ask for an agent name. Make sure you choose a unique one. After hitting enter, you will be presented with the prompt: `>`
+
+Now you'll want to register. You need to provide your faction:
+
+`> register cosmic`
+
+If all goes well, you should have an account now. 
+
+Next is building up our view of the universe: the atlas. 
+
+`> universe`
+
+This command will iterate over all systems and obtain all waypoint information. **this will take several hours**. However, if you are impatient, you can stop the process after a couple of minutes by hitting enter. It should at least have indexed your local system to continue.
+
+Now, let's see what ships we own.
+
+`> ships refresh`
+
+This will populate our internal cache of present ships and their state. After this, the state is updated continually by each command that alters it. You should be able to omit the 'refresh' after this.
+
+We should also populate the cache of contracts:
+
+`> contracts refresh`
+
+Same here: you can omit the 'refresh' from now on and view your contracts with the bare `contracts` command without sending a http request.
+
+You are now ready to begin operations.
+
+# Operations
+To get a ship to do something, you first have to select it:
+
+```
+> ship 1
+KILO-1 DOCKED [1035/1200] X1-YU85-03282C
+KILO-1> 
+```
+
+This prints the current state of the ship. Notice that the prompt now includes the name of the ship. Any command you now give will be executed by that ship.
+
+A quick, non-complete list of commands:
+
+* go: intra-system travel to another waypoint
+* orbit
+* dock
+* market: show local market data
+* shipyard: show local shipyard data
+* jumps: show local jumpgate data
+* mission: set ships mission type
+	* mine: mine and deliver a resource
+	* probe: find resource prices in nearby markets
+	* survey: generate surveys in a loop
+	* travel: travel to any waypoint in the universe using jumpgates (assuming a route is found)
+	* haul: buy goods at a site and bring them to a destination
+* mset: set a mission parameter
+* purchase: purchase a ship at a shipyard
+* buy: buy a resource at the local market
+* sell: sell a resource at the local market
+* jump: jump to a remote system using the local jumpgate
+* cmine: configures a mining mission for the current contract
+* chaul configures a hauling mission for the current contract
+* cprobe: configures a probing mission for the current contract
+* query: lists nearest markets and buy prices (if known) of a resource
+
+Look in commander.py for more available commands.
+
+# Data storage
+The script stores each object in a separate json file in the data directory. The default location for this directory is in your workingdir. The docker image places the data dir in '/data'
+
+If you want to re-register an agent, or there is a sever reset, just delete all data in that directory.

main.py

@@ -1,15 +1,23 @@
+#!/usr/bin/env python3
 import argparse
 from nullptr.commander import Commander
-
+import os
+from nullptr.store_analyzer import StoreAnalyzer
+from nullptr.models.base import Base
 def main(args):
-  c = Commander(args.store_dir)
-  c.run()
+  if not os.path.isdir(args.data_dir):
+    os.makedirs(args.data_dir )
+  if args.analyze:
+    a = StoreAnalyzer(verbose=True)
+    a.run(args.analyze)
+  else:
+    c = Commander(args.data_dir, auto=args.auto)
+    c.run()
   
-# X1-AG74-41076A
-# X1-KS52-51429E
-
 if __name__ == '__main__':
   parser = argparse.ArgumentParser()
-  parser.add_argument('-s', '--store-dir', default='data')
+  parser.add_argument('-d', '--data-dir', default='data')
+  parser.add_argument('--analyze', type=argparse.FileType('rb'))
+  parser.add_argument('-a', '--auto', action='store_true')
   args = parser.parse_args()
   main(args)

nullptr/analyzer.py

@@ -3,7 +3,35 @@ from nullptr.models.jumpgate import Jumpgate
 from nullptr.models.system import System
 from nullptr.models.waypoint import Waypoint
 from dataclasses import dataclass
+from nullptr.util import pprint
+from copy import copy
 
+class AnalyzerException(Exception):
+  pass
+
+def path_dist(m):
+  t = 0
+  o = Point(0,0)
+  for w in m:
+    t +=w.distance(o)
+    o = w
+  return t
+  
+@dataclass
+class Point:
+  x: int
+  y: int
+  
+@dataclass
+class TradeOption:
+  resource: str
+  source: Waypoint
+  dest: Waypoint
+  buy: int
+  margin: int
+  dist: int
+  score: float
+  
 @dataclass
 class SearchNode:
   system: System
@@ -24,68 +52,184 @@ class SearchNode:
   def __repr__(self):
     return self.system.symbol
  
-class Analyzer:
-  def __init__(self, store):
-    self.store = store
-
-  def find_markets(self, resource, sellbuy):
-    for m in self.store.all(Marketplace):
-      if 'sell' in sellbuy and resource in m.imports:
-        yield ('sell', m)
-      
-      elif 'buy' in sellbuy and resource in m.exports:
-        yield ('buy', m)
-      
-      elif 'exchange' in sellbuy and resource in m.exchange:
-        yield ('exchange', m)
-        
-  def find_closest_markets(self, resource, sellbuy, location):
-    if type(location) == str:
-      location = self.store.get(Waypoint, location)
-    mkts = self.find_markets(resource, sellbuy)
-    candidates = []
-    origin = self.store.get(System, location.system())
-    for typ, m in mkts:
-      system = self.store.get(System, m.system())
-      d = origin.distance(system)
-      candidates.append((typ, m, d))
-    possibles = sorted(candidates, key=lambda m: m[2])
-    possibles = possibles[:10]
-    results = []
-    for typ,m,d in possibles:
-      system = self.store.get(System, m.system())
-      p = self.find_path(origin, system)
-      if p is None: continue
-      results.append((typ,m,d,len(p)))
-    return results
     
-  def solve_tsp(self, waypoints):
-    # todo actually try to solve it
-    return waypoints
- 
-  def get_jumpgate(self, system):
-    gates = self.store.all_members(system, Jumpgate)
-    return next(gates, None)
-   
-  def find_path(self, orig, to, depth=100, seen=None):
-    if depth < 1: return None
-    if seen is None:
-      seen = set()
-    if type(orig) == System:
-      orig = set([SearchNode(orig,None)])
-    result = [n for n in orig if n.system==to]
-    if len(result) > 0:
-      return result[0].path()
-    dest = set()
-    for o in orig:
-      jg = self.get_jumpgate(o.system)
-      if jg is None: continue
-      for s in jg.systems:
-        if s in seen:  continue
-        seen.add(s)
-        system = self.store.get(System, s)
-        if system is None: continue
-        dest.add(SearchNode(system, o))
-    if len(dest) == 0: 
-      return None
-    return self.find_path(dest, to, depth-1, seen)
+def find_markets(c, resource, sellbuy):
+  for m in c.store.all(Marketplace):
+    if 'sell' in sellbuy and resource in m.imports:
+      yield ('sell', m)
+    
+    elif 'buy' in sellbuy and resource in m.exports:
+      yield ('buy', m)
+    
+    elif 'exchange' in sellbuy and resource in m.exchange:
+      yield ('exchange', m)
+      
+def find_closest_markets(c, resource, sellbuy, location):
+  if type(location) == str:
+    location = c.store.get(Waypoint, location)
+  mkts = find_markets(resource, sellbuy)
+  candidates = []
+  origin = location.system
+  for typ, m in mkts:
+    system = m.waypoint.system
+    d = origin.distance(system)
+    candidates.append((typ, m, d))
+  possibles = sorted(candidates, key=lambda m: m[2])
+  possibles = possibles[:10]
+  results = []
+  for typ,m,d in possibles:
+    system = m.waypoint.system
+    p = find_jump_path(origin, system)
+    if p is None: continue
+    results.append((typ,m,d,len(p)))
+  return results
+  
+def solve_tsp(c, waypoints):
+  wps = copy(waypoints)
+  path = []
+  cur = Point(0,0)
+  while len(wps) > 0:
+    closest = wps[0]
+    for w in wps:
+      if w.distance(cur) < closest.distance(cur):
+        closest = w
+    cur = closest
+    path.append(closest)
+    wps.remove(closest)
+  return path
+
+def get_jumpgate(c, system):
+  gates = c.store.all_members(system, Jumpgate)
+  return next(gates, None)
+
+# dijkstra shmijkstra
+def find_nav_path(c, orig, to, ran):
+  path = []
+  mkts = [m.waypoint for m in c.store.all_members(orig.system, Marketplace)]
+  cur = orig
+  if orig == to:
+    
+    return []
+  while cur != to:
+    best = cur
+    bestdist = cur.distance(to)
+    if bestdist < ran:
+      path.append(to)
+      break
+    for m in mkts:
+      dist = m.distance(to)
+      if dist < bestdist and cur.distance(m) < ran:
+        best = m
+        bestdist = dist
+    if best == cur:
+      raise AnalyzerException(f'no path to {to}') 
+    cur = best
+    path.append(cur)
+  return path
+  
+def find_jump_path(c, orig, to, depth=100, seen=None):
+  if depth < 1: return None
+  if seen is None:
+    seen = set()
+  if type(orig) == System:
+    orig = set([SearchNode(orig,None)])
+  result = [n for n in orig if n==to]
+  if len(result) > 0:
+    return result[0].path()
+  dest = set()
+  for o in orig:
+    jg = get_jumpgate(o)
+    if jg is None: continue
+    for s in jg.connections:
+      if s in seen:  continue
+      seen.add(s)
+      dest.add(SearchNode(s, o))
+  if len(dest) == 0: 
+    return None
+  return find_jump_path(dest, to, depth-1, seen)
+
+def prices(c, system):
+  prices = {}
+  for m in c.store.all_members(system, Marketplace):
+    for r, p in m.prices.items():
+      if not r in prices:
+        prices[r] = []
+      prices[r].append({
+        'wp': m.waypoint,
+        'buy': p.buy,
+        'sell': p.sell,
+        'volume': p.volume,
+        'category': m.rtype(r)
+      })
+  return prices
+
+def find_trade(c, system):
+  max_traders = 3
+  pcs= prices(c, system)
+  occupied_routes = dict()
+  for s in c.store.all('Ship'):
+    if s.mission != 'trade':
+      continue
+    k = (s.mission_state['site'], s.mission_state['dest'])
+    if k in occupied_routes:
+      occupied_routes[k] += 1
+    else:
+      occupied_routes[k] = 1
+  best = None
+  for resource, markets in pcs.items():
+    source = sorted(markets, key=lambda x: x['buy'])[0]
+    dest = sorted(markets, key=lambda x: x['sell'])[-1]
+    swp = source['wp']
+    dwp = dest['wp']
+    margin = dest['sell'] -source['buy']
+    k = (swp.symbol,dwp.symbol)
+    if k in occupied_routes and occupied_routes[k] > max_traders:
+      continue
+    dist = swp.distance(dwp)
+    dist = max(dist, 0.0001)
+    score = margin / dist
+    if margin < 2:
+      continue
+    o = TradeOption(resource, swp, dwp, source['buy'], margin, dist, score)
+    if best is None or best.score < o.score:
+      best = o
+  return best
+
+def find_deal(c, smkt, dmkt):
+  best_margin = 0
+  best_resource = None
+  for r, sp in smkt.prices.items():
+    if not r in dmkt.prices:
+      continue
+    dp = dmkt.prices[r]
+    margin = dp.sell - sp.buy
+    if margin > best_margin:
+      best_margin = margin
+      best_resource = r
+  return best_resource
+
+def best_sell_market(c, system, r):
+  best_price = 0
+  best_market = None
+  for m in c.store.all_members(system, Marketplace):
+    if r not in m.prices: continue
+    price = m.prices[r].sell
+    if price > best_price:
+      best_price = price
+      best_market = m
+  return best_market
+  
+def find_gas(c, system):
+  m = [w for w in c.store.all_members(system, 'Waypoint') if w.type == 'GAS_GIANT']
+  if len(m)==0:
+    raise AnalyzerException('no gas giant found')
+  return m[0]
+  
+def find_metal(c, system):
+  m = [w for w in c.store.all_members(system, Waypoint) if 'COMMON_METAL_DEPOSITS' in w.traits]
+  if len(m) == 0:
+    return None
+  origin = Point(0,0)
+  m = sorted(m, key=lambda w: w.distance(origin))
+  return m[0]
+

nullptr/api.py

@@ -4,9 +4,11 @@ from nullptr.models.waypoint import Waypoint
 from nullptr.models.marketplace import Marketplace
 from nullptr.models.jumpgate import Jumpgate
 from nullptr.models.ship import Ship
+from nullptr.models.shipyard import Shipyard
 from .util import *
-from time import sleep
-class ApiError(Exception):
+from time import sleep, time
+
+class ApiError(AppError):
   def __init__(self, msg, code):
     super().__init__(msg)
     self.code = code
@@ -15,11 +17,11 @@ class ApiLimitError(Exception):
   pass
   
 class Api:
-  def __init__(self, store, agent):
+  def __init__(self, c, agent):
     self.agent = agent
-    self.store = store
+    self.store = c.store
     self.requests_sent = 0
-    self.meta = None
+    self.last_meta = None
     self.last_result = None
     self.root = 'https://api.spacetraders.io/v2/'
     
@@ -30,9 +32,13 @@ class Api:
   
   def request(self, method, path, data=None, need_token=True, params={}):
     try:
-      return self.request_once(method, path, data, need_token, params)
+      start = time()
+      result = self.request_once(method, path, data, need_token, params)
+      dur = time() - start
+      # print(f'api {dur:.03}')
+      return result
     except (ApiLimitError, requests.exceptions.Timeout):
-      print('oops, hit the limit. take a break')
+      # print('oops, hit the limit. take a break')
       sleep(10)
       return self.request_once(method, path, data, need_token, params)
       
@@ -60,7 +66,8 @@ class Api:
     else:
       self.last_error = 0
       return result['data']
-    
+  
+  ######## Account #########
   def register(self, faction):
     callsign = self.agent.symbol
     data = {
@@ -69,38 +76,81 @@ class Api:
     }
     result = self.request('post', 'register', data, need_token=False)
     token = mg(result, 'token')
+    self.agent.update(mg(result, 'agent'))
     self.agent.token = token
   
+  def status(self):
+    try:
+      self.request('get', '')
+    except ApiError:
+      pass
+    return self.last_result
+    
   def info(self):
     data = self.request('get', 'my/agent')
     self.agent.update(data)
     return self.agent
   
+  ######## Atlas #########
   def list_systems(self, page=1):
     data = self.request('get', 'systems', params={'page': page})
     #pprint(self.last_meta)
-    return self.store.update_list(System, data)
+    systems = self.store.update_list(System, data)
+    for s in data:
+      self.store.update_list(Waypoint, mg(s, 'waypoints'))
+    return systems
   
   def list_waypoints(self, system):
     data = self.request('get', f'systems/{system}/waypoints/')
+    tp = total_pages(self.last_meta)
+    for p in range(tp):
+      data += self.request('get', f'systems/{system}/waypoints/', params={'page': p+1})
     # pprint(data)
     return self.store.update_list(Waypoint, data)
   
   def marketplace(self, waypoint):
-    system = waypoint.system()
-    symbol = str(waypoint)
+    system = waypoint.system
     data = self.request('get', f'systems/{system}/waypoints/{waypoint}/market')
     return self.store.update(Marketplace, data)
   
   def jumps(self, waypoint):
-    data = self.request('get', f'systems/{waypoint.system()}/waypoints/{waypoint}/jump-gate')
+    data = self.request('get', f'systems/{waypoint.system}/waypoints/{waypoint}/jump-gate')
     symbol = str(waypoint)
     return self.store.update(Jumpgate, data, symbol)
   
+  def shipyard(self, wp):
+    data = self.request('get', f'systems/{wp.system}/waypoints/{wp}/shipyard')
+    symbol = str(wp)
+    
+    return self.store.update(Shipyard, data, symbol)
+
+  ######## Fleet #########
   def list_ships(self):
     data = self.request('get', 'my/ships')
+    tp = total_pages(self.last_meta)
+    for p in range(1, tp):
+      data += self.request('get', 'my/ships', params={'page': p+1})
     return self.store.update_list(Ship, data)
   
+  def refuel(self, ship, from_cargo=False):
+    fuel_need = ship.fuel_capacity - ship.fuel_current
+    fuel_avail = ship.get_cargo('FUEL') * 100
+    units = fuel_need
+    if from_cargo:
+      units = min(units, fuel_avail)
+    data = {'fromCargo': from_cargo, 'units': units }
+    data =  self.request('post', f'my/ships/{ship}/refuel', data)
+    self.log_transaction(data)
+    if from_cargo:
+      boxes = ceil(float(units) / 100)
+      ship.take_cargo('FUEL', boxes)
+    if 'fuel' in data:
+      ship.update(data)
+    if 'agent' in data:
+      self.agent.update(data['agent'])
+    return data
+    
+  ######## Contract #########
   def list_contracts(self):
     data = self.request('get', 'my/contracts')
     return self.store.update_list('Contract', data)
@@ -110,7 +160,15 @@ class Api:
     if data is not None and 'contract' in data:
       contract = self.store.update('Contract', data['contract'])
       return contract
-  
+      
+  def accept_contract(self, contract):
+    data = self.request('post', f'my/contracts/{contract.symbol.lower()}/accept')
+    if 'contract' in data:
+      contract.update(data['contract'])
+    if 'agent' in data:
+      self.agent.update(data['agent'])
+    return contract
+    
   def deliver(self, ship, typ, contract):
     units = ship.get_cargo(typ)
     if units == 0:
@@ -135,10 +193,12 @@ class Api:
     if 'agent' in data:
       self.agent.update(data['agent'])
     return contract
- 
+  
+  ######## Nav #########
   def navigate(self, ship, wp):
     data = {'waypointSymbol': str(wp)}
     response = self.request('post', f'my/ships/{ship}/navigate', data)
+    ship.log(f'nav to {wp}')
     ship.update(response)
   
   def dock(self, ship):
@@ -150,30 +210,89 @@ class Api:
     data = self.request('post', f'my/ships/{ship}/orbit')
     ship.update(data)
     return data
+  
+  def flight_mode(self, ship, mode):
+    data = {'flightMode': mode}
+    data = self.request('patch', f'my/ships/{ship}/nav', data)
+    ship.update({'nav':data})
+    return data
     
-  def refuel(self, ship):
-    data =  self.request('post', f'my/ships/{ship}/refuel')
-    if 'fuel' in data:
+  def jump(self, ship, waypoint):
+    if type(waypoint) == Waypoint:
+      waypoint = waypoint.symbol
+    data = {
+      "waypointSymbol": waypoint
+    }
+    data = self.request('post', f'my/ships/{ship}/jump', data)
+    if 'nav' in data:
       ship.update(data)
-    if 'agent' in data:
-      self.agent.update(data['agent'])
+    return ship
+    
+  ######## Extraction #########
+  def siphon(self, ship):
+    data = self.request('post', f'my/ships/{ship}/siphon')
+    ship.update(data)
+    amt = mg(data, 'siphon.yield.units')
+    rec = mg(data, 'siphon.yield.symbol')
+    ship.log(f"siphoned {amt} {rec}")
+    ship.location.extracted += amt
+    return data['siphon']
+  
+  def extract(self, ship, survey=None):
+    data = {}
+    url = f'my/ships/{ship}/extract'
+    if survey is not None:
+      data= survey.api_dict()
+      url += '/survey'
+    try:
+      data =  self.request('post', url, data=data)
+    except ApiError as e:
+      if e.code in [ 4221, 4224]:
+        survey.exhausted = True
+      else:
+        raise e
+    ship.update(data)
+    amt = sg(data, 'extraction.yield.units', 0)
+    rec = sg(data, 'extraction.yield.symbol', 'nothing')
+    ship.log(f"extracted {amt} {rec}")
+    ship.location.extracted += amt
     return data
   
-  def accept_contract(self, contract):
-    data = self.request('post', f'my/contracts/{contract.symbol.lower()}/accept')
-    if 'contract' in data:
-      contract.update(data['contract'])
-    if 'agent' in data:
-      self.agent.update(data['agent'])
-    return contract
+  def survey(self, ship):
+    data =  self.request('post', f'my/ships/{ship}/survey')
+    ship.update(data)
+    result = self.store.update_list('Survey', mg(data, 'surveys'))
+    return result
   
-  def sell(self, ship, typ):
-    units = ship.get_cargo(typ)
+    
+  ######## Commerce #########
+  def transaction_cost(self, data):
+    if not 'transaction' in data: return 0
+    act = mg(data,'transaction.type')
+    minus = -1 if act == 'PURCHASE' else 1
+    units = mg(data, 'transaction.units')
+    ppu = mg(data, 'transaction.pricePerUnit')
+    return ppu * units * minus
+    
+  def log_transaction(self, data):
+    if not 'transaction' in data: return
+    typ = mg(data, 'transaction.tradeSymbol')
+    ppu = mg(data, 'transaction.pricePerUnit')
+    shipsym = mg(data,  'transaction.shipSymbol')
+    ship = self.store.get('Ship', shipsym)
+    units = mg(data, 'transaction.units')
+    act = mg(data,'transaction.type')
+    ship.log(f'{act} {units} of {typ} for {ppu} at {ship.location}')
+    
+  def sell(self, ship, typ,units=None):
+    if units is None:
+      units = ship.get_cargo(typ)
     data = {
       'symbol': typ,
       'units': units
     }
     data = self.request('post', f'my/ships/{ship}/sell', data)
+    self.log_transaction(data)
     if 'cargo' in data:
       ship.update(data)
     if 'agent' in data:
@@ -186,6 +305,7 @@ class Api:
       'units': amt
     }
     data = self.request('post', f'my/ships/{ship}/purchase', data)
+    self.log_transaction(data)
     if 'cargo' in data:
       ship.update(data)
     if 'agent' in data:
@@ -202,12 +322,26 @@ class Api:
       'units': units
     }
     data = self.request('post', f'my/ships/{ship.symbol}/jettison', data)
+    ship.log(f'drop {units} of {typ}')
     if 'cargo' in data:
       ship.update(data)
     if 'agent' in data:
       self.agent.update(data['agent'])
     return data
   
+  def transfer(self, sship, dship, typ, amt):
+    data = {
+    'tradeSymbol': typ,
+    'units': amt,
+    'shipSymbol': dship.symbol
+    }
+    data = self.request('post', f'my/ships/{sship.symbol}/transfer', data)
+    sship.log(f'tra {amt} {typ} to {dship}')
+    dship.log(f'rec {amt} {typ} from {sship}', 10)
+    if 'cargo' in data:
+      sship.update(data)
+    dship.put_cargo(typ, amt)
+    
   def purchase(self, typ, wp):
     data = {
       'shipType': typ,
@@ -218,39 +352,4 @@ class Api:
       self.agent.update(data['agent'])
     if 'ship' in data:
       ship = self.store.update('Ship', data['ship'])  
-      return ship
-    
-  def jump(self, ship, system):
-    if type(system) == System:
-      system = system.symbol
-    data = {
-      "systemSymbol": system
-    }
-    data = self.request('post', f'my/ships/{ship}/jump', data)
-    if 'nav' in data:
-      ship.update(data)
-    return ship
-    
-  def shipyard(self, wp):
-    return self.request('get', f'systems/{wp.system()}/waypoints/{wp}/shipyard')
-
-  def extract(self, ship, survey=None):
-    data = {}
-    if survey is not None:
-      data['survey'] = survey.api_dict()
-    try:
-      data =  self.request('post', f'my/ships/{ship}/extract', data=data)
-    except ApiError as e:
-      if e.code in [ 4221, 4224]:
-        survey.exhausted = True
-      else:
-        raise e
-    ship.update(data)
-    return data
-  
-  def survey(self, ship):
-    data =  self.request('post', f'my/ships/{ship}/survey')
-    ship.update(data)
-    result = self.store.update_list('Survey', mg(data, 'surveys'))
-    return result
-    
+      return ship

nullptr/atlas_builder.py

@@ -1,66 +1,74 @@
-from time import  sleep
+from time import  sleep, time
 from nullptr.util import *
 from threading import Thread
+from nullptr.models.atlas import Atlas
+from functools import partial
+from nullptr.models import System
 
 class AtlasBuilder:
   def __init__(self, store, api):
     self.store = store
     self.api = api
-    self.stop_auto = False
-    
-  def wait_for_stop(self):
-    try:
-      input()
-    except EOFError:
-      pass
-    self.stop_auto = True
-    print('stopping...')
-    
-  def run(self, page=1):
-    print('universe mode. hit enter to stop')
-    t = Thread(target=self.wait_for_stop)
-    t.daemon = True
-    t.start()
-    self.all_systems(int(page))
-    print('manual mode')
+    self.work = []
+    self.max_work = 100
+    self.unch_interval = 86400
+    self.atlas = self.store.get(Atlas, 'ATLAS', create=True)   
   
-  def all_specials(self, waypoints):
-    for w in waypoints:
-      if self.stop_auto:
+  def find_work(self):
+    if not self.atlas.enabled:
+      return
+    first_page = self.atlas.total_pages == 0 
+    pages_left = self.atlas.total_pages > self.atlas.seen_pages
+    
+    if first_page or pages_left:
+      self.sched(self.get_systems)
+      return
+    for s in self.store.all(System):
+      if len(self.work) > self.max_work:
         break
+      if not s.uncharted: continue
+      if s.last_crawl > time() - self.unch_interval: 
+        continue
+      self.sched(self.get_waypoints, s)
+      
+  
+  def do_work(self):
+    if len(self.work) == 0:
+      self.find_work()
+    if len(self.work) == 0:
+      return
+    work = self.work.pop()
+    work()
+  
+  def get_systems(self):
+    page = 1
+    if self.atlas.seen_pages > 0:
+      page = self.atlas.seen_pages + 1
+      if page > self.atlas.total_pages:
+        return
+    # print('systems', page)
+    data = self.api.list_systems(page)
+    self.atlas.total_pages = total_pages(self.api.last_meta)
+    self.atlas.seen_pages = page
+  
+  def get_waypoints(self, system):
+    wps = self.api.list_waypoints(system)
+    system.last_crawl = time()
+    system.uncharted = len([1 for w in wps if w.uncharted]) > 0
+    self.schedule_specials(wps)
+ 
+  def sched(self, fun, *args):
+    self.work.append(partial(fun, *args))
+    
+  def schedule_specials(self, waypoints):
+    for w in waypoints:
       if 'UNCHARTED' in w.traits:
         continue
       if 'MARKETPLACE' in w.traits:
-        self.api.marketplace(w)
-        print(f'marketplace at {w}')
-        sleep(0.5)
+        #print(f'marketplace at {w}')
+        self.sched(self.api.marketplace, w)
       if w.type == 'JUMP_GATE':
-        self.api.jumps(w)
-        print(f'jumpgate at {w}')
-        
-  def all_waypoints(self, systems):
-    for s in systems:
-      if self.stop_auto:
-        break
-      r = self.api.list_waypoints(s)
-      self.all_specials(r)
-      sleep(0.5)
-      
-  
-  def all_systems(self, start_page):
-    self.stop_auto = False 
-    data = self.api.list_systems(start_page)
-    pages = total_pages(self.api.last_meta)
-    print(f'{pages} pages of systems')
-    print(f'page {1}: {len(data)} results')
-    self.all_waypoints(data)
-    self.store.flush()
-    
-    for p in range(start_page+1, pages+1):
-      if self.stop_auto:
-        break
-      data = self.api.list_systems(p)
-      print(f'page {p}: {len(data)} systems')
-      self.all_waypoints(data)
-      sleep(0.5)
-      self.store.flush()
+        #print(f'jumpgate at {w}')
+        self.sched(self.api.jumps, w)
+      if 'SHIPYARD' in w.traits:
+        self.sched(self.api.shipyard, w)

nullptr/captain.py

@@ -1,43 +1,59 @@
 from nullptr.store import Store
 from nullptr.models.ship import Ship
 from nullptr.missions import create_mission, get_mission_class
-from random import choice
-from time import sleep
+from nullptr.models.waypoint import Waypoint
+from random import choice, randrange
+from time import sleep, time
 from threading import Thread
+from nullptr.atlas_builder import  AtlasBuilder
+from nullptr.general import General
+from nullptr.util import *
+from nullptr.roles import assign_mission
 
-class CentralCommandError(Exception):
+class CentralCommandError(AppError):
   pass
   
-class CentralCommand:
-  def __init__(self, store, api):
+class Captain:
+  def __init__(self, context):
     self.missions = {}
     self.stopping = False
-    self.store = store
-    self.api = api
+    self.store = context.store
+    self.c = context
+    self.general = context.general
+    self.api = context.api
+    self.general = context.general
+    self.atlas_builder = AtlasBuilder(self.store, self.api)
+    
+  def setup(self):
     self.update_missions()
   
   def get_ready_missions(self):
     result = []
+    prio = 1
     for ship, mission in self.missions.items():
-      if mission.is_ready():
+      p = mission.is_ready()
+      if p == prio:
         result.append(ship)
+      elif p > prio:
+        prio = p
+        result = [ship]
     return result
+  
+  def single_step(self, ship):
+    if ship not in self.missions:
+      print('ship has no mission')
+    mission = self.missions[ship]
+    mission.step()
     
   def tick(self):
+    self.general.tick()
+    self.update_missions()
     missions = self.get_ready_missions()
     if len(missions) == 0: return False
     ship = choice(missions)
     mission = self.missions[ship]
     mission.step()
     return True
-  
-  def wait_for_stop(self):
-    try:
-      input()
-    except EOFError:
-      pass
-    self.stopping = True
-    print('stopping...')
     
   def run_interactive(self):
     print('auto mode. hit enter to stop')
@@ -46,16 +62,35 @@ class CentralCommand:
     t.start()
     self.run()
     print('manual mode')
-    
+  
+  def wait_for_stop(self):
+    try:
+      input()
+    except EOFError:
+      pass
+    self.stopping = True
+    print('stopping...')
+  
   def run(self):
-    self.update_missions()
     while not self.stopping:
+      # any new orders?
+      self.c.general.tick()
       did_step = True
       request_counter = self.api.requests_sent
+      start = time()
       while request_counter == self.api.requests_sent and did_step:
         did_step = self.tick()
+      if request_counter == self.api.requests_sent:
+        self.atlas_builder.do_work()
+      else:
+        pass # print('nowork')
+        
       self.store.flush()
-      sleep(0.5)
+      dur = time() - start
+      # print(f'step {dur:.03}')
+      zs = 0.5 - dur
+      if zs > 0:
+        sleep(zs)
     self.stopping = False
       
   def stop(self):
@@ -77,18 +112,24 @@ class CentralCommand:
       raise MissionError(e)
       return
     ship.set_mission_state(nm, parsed_val)
+  
+  def smipa(self,s,n,v):    
+    self.set_mission_param(s,n,v)
     
   def update_missions(self):
     for s in self.store.all(Ship):
+      if s.mission_status == 'done':
+        s.mission = None
       if s.mission is None:
         if s in self.missions:
           self.stop_mission(s)
-      elif s not in self.missions:
+      if s.mission is None:
+        assign_mission(self.c, s)
+      if s.mission is not None and s not in self.missions:
         self.start_mission(s)
       if s in self.missions:
         m = self.missions[s]
-        m.next_step = max(s.cooldown, s.arrival)
-  
+    
   def init_mission(self, s, mtyp):
     if mtyp == 'none':
       s.mission_state = {}
@@ -103,13 +144,21 @@ class CentralCommand:
     s.mission_status = 'init'
     s.mission_state = {k: v.default for k,v in mclass.params().items()}
     self.start_mission(s)
+  
+  def restart_mission(self, s, status='init'):
+    if s not in self.missions:
+      raise CentralCommandError("no mission assigned")
+    s.mission_status = status
     
   def start_mission(self, s):
     mtype = s.mission
-    m = create_mission(mtype, s, self.store, self.api)
+    m = create_mission(mtype, s, self.c)
     self.missions[s] = m
+    m.status(s.mission_status)
     return m
     
   def stop_mission(self, s):
     if s in self.missions:
       del self.missions[s]
+
+  

nullptr/command_line.py

@@ -3,6 +3,7 @@ import inspect
 import sys
 import importlib
 import logging
+from nullptr.util import AppError
 
 def func_supports_argcount(f, cnt):
   argspec = inspect.getargspec(f)
@@ -41,7 +42,7 @@ class CommandLine:
     print(f'command not found; {c}')
   
   def handle_error(self, cmd, args, e):
-    logging.error(e, exc_info=type(e).__name__ not in ['ApiError','CommandError', 'CentralCommandError'])
+    logging.error(e, exc_info=not issubclass(type(e), AppError))
   
   def handle_empty(self):
     pass
@@ -87,11 +88,13 @@ class CommandLine:
       p = self.prompt()
       try:
         c = input(p)
-      except EOFError:
+      except (EOFError, KeyboardInterrupt):
         self.handle_eof()
         break
       try:
         self.handle_cmd(c)
-      except Exception as e:
+      except KeyboardInterrupt:
+        print("Interrupted")
+      except (Exception) as e:
         logging.error(e, exc_info=True)
       

nullptr/commander.py

@@ -1,48 +1,76 @@
 from nullptr.command_line import CommandLine
 from nullptr.store import Store
-from nullptr.analyzer import  Analyzer
+from nullptr.analyzer import  *
+from nullptr.context import Context
 import argparse
-from nullptr.models.agent import Agent
-from nullptr.models.system import System
-from nullptr.models.waypoint import Waypoint
-from nullptr.models.marketplace import Marketplace
-from nullptr.models.jumpgate import Jumpgate
+from nullptr.models import * 
 from nullptr.api import Api
 from .util import *
 from time import sleep, time
 from threading import Thread
-from nullptr.atlas_builder import  AtlasBuilder
-from nullptr.central_command import CentralCommand
-class CommandError(Exception):
+from nullptr.captain import Captain
+from nullptr.general import General
+import readline
+import os
+from copy import copy
+ 
+
+class CommandError(AppError):
   pass
   
 class Commander(CommandLine):
-  def __init__(self, store_dir='data'):
-    self.store_dir = store_dir
-    self.store = Store(store_dir)
-    self.store.load()
+  def __init__(self, data_dir='data', auto=False):
+    store_file = os.path.join(data_dir, 'store.npt')
+    hist_file = os.path.join(data_dir, 'cmd.hst')
+    self.cred_file = os.path.join(data_dir, 'creds.txt')
+    self.hist_file = hist_file
+    if os.path.isfile(hist_file):
+      readline.read_history_file(hist_file)
+    self.store = Store(store_file, True)
+    self.c = Context(self.store)
     self.agent = self.select_agent()
-    self.api = Api(self.store, self.agent)
-    self.atlas_builder = AtlasBuilder(self.store, self.api)
-    self.centcom = CentralCommand(self.store, self.api)
-    self.analyzer = Analyzer(self.store)
+    self.c.api = self.api = Api(self.c, self.agent)
+    self.c.general = self.general = General(self.c)
+    self.c.captain = self.captain = Captain(self.c)
+  
+    self.general.setup()
+    self.captain.setup()
+    
+    self.api.info()
+    
     self.ship = None
     
-    self.stop_auto= False
+    self.stop_auto = False
+    if auto:
+      self.do_auto()
     super().__init__()
   
+  ######## INFRA #########
+  def handle_eof(self):
+    self.store.close()
+    readline.write_history_file(self.hist_file)
+    print("Goodbye!")
+  
+  def do_pp(self):
+    pprint(self.api.last_result)
+  
   def prompt(self):
     if self.ship:
       return f'{self.ship.symbol}> '
     else:
       return '> '
+
+  def after_cmd(self):
+    self.store.flush()
+
+  def do_auto(self):
+    self.captain.run_interactive()
+  
+  def do_log(self, level):
+    ship = self.has_ship()
+    ship._log_level = int(level)
     
-  def has_ship(self):
-    if self.ship is not None:
-      return True
-    else:
-      print('set a ship')
-      
+  ######## Resolvers #########
   def ask_obj(self, typ, prompt):
     obj = None
     while obj is None:
@@ -51,13 +79,18 @@ class Commander(CommandLine):
       if obj is None:
         print('not found')
     return obj
-    
+
+  def has_ship(self):
+    if self.ship is not None:
+      return self.ship
+    else:
+      raise CommandError('set a ship')
+  
   def select_agent(self):
     agents = self.store.all(Agent)
     agent = next(agents, None)
     if agent is None:
-      symbol = input('agent name: ')
-      agent = self.store.get(Agent, symbol, create=True)
+      agent = self.agent_setup()
     return agent
   
   def resolve(self, typ, arg):
@@ -68,187 +101,328 @@ class Commander(CommandLine):
     elif len(matches) > 1:
       raise CommandError('multiple matches')
     else:
-      raise CommandError('not found')
-      
-  def after_cmd(self):
+      raise CommandError(f'{arg} not found')
+  
+  def resolve_system(self, system_str):
+    if type(system_str) == System:
+      return system_str
+    if system_str == '':
+      ship = self.has_ship()
+      system = ship.location.system
+    else:
+      system = self.store.get(System, system_str)
+    return system
+  
+  def resolve_waypoint(self, w):
+    if type(w) == Waypoint:
+      return w
+    if w == '':
+      ship = self.has_ship()
+      return ship.location
+    p = w.split('-')
+    if len(p) == 1:
+      ship = self.has_ship()
+      s = ship.location.system
+      w = f'{s}-{w}'
+    r = self.store.get(Waypoint, w)
+    if r is None:
+      raise CommandError(f'{w} not found')
+    return r
+    
+  def resolve_ship(self, arg):
+    symbol = f'{self.agent.symbol}-{arg}'
+    ship = self.store.get('Ship', symbol)
+    if ship is None:
+      raise CommandError(f'ship {arg} not found')
+    return ship
+    
+  ######## First run #########
+  def agent_setup(self):
+    symbol = input('agent name: ')
+    agent = self.store.get(Agent, symbol, create=True)
+    self.agent = agent
+    api = Api(self.c, agent)
+    self.api = api
+    faction = input('faction or token: ')
+    if len(faction) > 50:
+      self.agent.token = faction
+    else:
+      self.do_register(faction)
+    print('=== agent:')
+    print(agent)
+    print('=== ships')
+    self.do_ships('r')
+    
+    print('=== contracts')
+    self.do_contracts('r')
+    ship = self.store.get(Ship, symbol.upper() + '-2')
+    print("=== catalog initial system")
+    self.do_catalog(ship.location.system)
+    self.do_stats()
     self.store.flush()
+    return agent
+  
+  def do_token(self):
+    print(self.agent.token)
+  
+  def do_register(self, faction):
+    self.api.register(faction.upper())
+    with open(self.cred_file, 'w') as f:
+      f.write(self.api.agent.symbol)
+      f.write('\n')
+      f.write(self.api.agent.token)
+    pprint(self.api.agent)
+  
+  def do_reset(self, really):
+    if really != 'yes':
+      print('really? type: reset yes')
+    self.api.list_ships()
+    for s in self.store.all('Ship'):
+      self.dump(s, 'all')
+      self.captain.init_mission(s, 'none')
       
+  ######## Fleet #########
   def do_info(self, arg=''):
     if arg.startswith('r'):
       self.api.info()
-    
     pprint(self.agent, 100)
   
-  def do_auto(self):
-    self.centcom.run_interactive()
-  
-  def print_mission(self):
-    print(f'mission: {self.ship.mission} ({self.ship.mission_status})')
-    pprint(self.ship.mission_state)
-    
-  def do_mission(self, arg=''):
-    if not self.has_ship(): return 
-    if arg:
-      self.centcom.init_mission(self.ship, arg)
-    self.print_mission()
-  
-  def do_mreset(self):
-    if not self.has_ship(): return 
-    self.ship.mission_state = {}
-    
-  def do_mset(self, nm, val):
-    if not self.has_ship(): return
-    self.centcom.set_mission_param(self.ship, nm, val)
-  
-  def active_contract(self):
-    for c in self.store.all('Contract'):
-      if c.accepted and not c.fulfilled: return c
-    raise CommandError('no active contract')
-    
-  def do_cmine(self):
-    if not self.has_ship(): return 
-    site = self.ship.location_str
-    contract = self.active_contract()
-    delivery = contract.unfinished_delivery()
-    if delivery is None:
-      raise CommandError('no delivery')
-    resource = delivery['trade_symbol']
-    destination = delivery['destination']
-    self.centcom.init_mission(self.ship, 'mine')
-    self.centcom.set_mission_param(self.ship, 'site', site)
-    self.centcom.set_mission_param(self.ship, 'resource', resource)
-    self.centcom.set_mission_param(self.ship, 'dest', destination)
-    self.centcom.set_mission_param(self.ship, 'contract', contract.symbol)
-    self.print_mission()
-  
-  def do_chaul(self):
-    if not self.has_ship(): return     
-    contract = self.active_contract()
-    delivery = contract.unfinished_delivery()
-    if delivery is None:
-      raise CommandError('no delivery')
-    resource = delivery['trade_symbol']
-    destination = delivery['destination']
-    m = self.analyzer.find_closest_markets(resource, 'buy', destination)
-    if len(m) == 0:
-      m = self.analyzer.find_closest_markets(resource, 'exchange', destination)
-    if len(m) == 0:
-      print('no market found')
-      return
-    _, m, _, _ = m[0]
-    site = self.store.get(Waypoint, m.symbol)
-    self.centcom.init_mission(self.ship, 'haul')
-    self.centcom.set_mission_param(self.ship, 'site', site.symbol)
-    self.centcom.set_mission_param(self.ship, 'resource', resource)
-    self.centcom.set_mission_param(self.ship, 'dest', destination)
-    self.centcom.set_mission_param(self.ship, 'contract', contract.symbol)
-    self.print_mission()
-  
-  def do_cprobe(self):
-    if not self.has_ship(): return     
-    contract = self.active_contract()
-    delivery = contract.unfinished_delivery()
-    if delivery is None:
-      raise CommandError('no delivery')
-    resource = delivery['trade_symbol']
-    destination = delivery['destination']
-    m = self.analyzer.find_closest_markets(resource, 'buy,exchange', destination)
-    if len(m) is None:
-      print('no market found')
-      return
-    markets = [ mkt[1] for mkt in m]
-    markets = self.analyzer.solve_tsp(markets)
-    hops = ','.join([m.symbol for m in markets])
-    self.centcom.init_mission(self.ship, 'probe')
-    self.centcom.set_mission_param(self.ship, 'hops', hops)
-    self.print_mission()
-  
-  def do_travel(self, dest):
-    dest = self.resolve('Waypoint', dest)
-    self.centcom.init_mission(self.ship, 'travel')
-    self.centcom.set_mission_param(self.ship, 'dest', dest.symbol)
-    self.print_mission()
-    
-  def do_register(self, faction):
-    self.api.register(faction.upper())
-    pprint(self.api.agent)
-    
-  def do_universe(self, page=1):
-    self.atlas_builder.run(page)
-  
-  def do_systems(self, page=1):
-    r = self.api.list_systems(int(page))
-    pprint(self.api.last_meta)
-  
-  def do_stats(self):
-    total = 0
-    for t in self.store.data:
-      num = len(self.store.data[t])
-      nam = t.__name__
-      total += num
-      print(f'{num:5d} {nam}')
-    print(f'{total:5d} total')
-    
-  def do_waypoints(self, system_str=''):
-    if system_str == '':
-      if not self.has_ship(): return
-      system = self.ship.location().system()
-    else:
-      system = self.store.get(System, system_str)
-    r = self.store.all_members(system, 'Waypoint')
-    for w in r:
-      traits = []
-      if 'MARKETPLACE' in w.traits:
-        traits.append('MARKET')
-      if 'SHIPYARD' in w.traits:
-        traits.append('SHIPYARD')
-      if w.type == 'JUMP_GATE':
-        traits.append('JUMP')
-      if w.type == 'ASTEROID_FIELD':
-        traits.append('ASTROIDS')
-      print(w.symbol.split('-')[2], ', '.join(traits))
-  
-  def do_wp(self, s=''):
-    self.do_waypoints(s)
-    
-  def do_marketplace(self, waypoint_str):
-    waypoint = self.store.get(Waypoint, waypoint_str.upper())
-    r = self.api.marketplace(waypoint)
-  
-  def do_jumps(self, waypoint_str=None):
-    if waypoint_str is None:
-      if not self.has_ship(): return
-      waypoint = self.ship.location()
-    else:
-      waypoint = self.store.get(Waypoint, waypoint_str.upper())
-    r = self.api.jumps(waypoint)
-    pprint(r)
-  
-  def do_query(self, resource):
-    if not self.has_ship(): return
-    location = self.ship.location()
-    resource = resource.upper()
-    print('Found markets:')
-    for typ, m, d, plen in self.analyzer.find_closest_markets(resource, 'buy,exchange',location):
-      price = '?'
-      if resource in m.prices:
-        price = m.prices[resource]['buy']
-      print(m, typ[0], f'{plen-1:3} hops {price}')
-      
-  def do_path(self):
-    orig = self.ask_obj(System, 'from: ')
-    dest = self.ask_obj(System, 'to: ')
-    # orig = self.store.get(System, 'X1-KS52')
-    # dest = self.store.get(System, 'X1-DA90')
-    path = self.analyzer.find_path(orig, dest)
-    pprint(path)
-  
   def do_ships(self, arg=''):
     if arg.startswith('r'):
       r = self.api.list_ships()
     else:
-      r = list(self.store.all('Ship'))
+      r = sorted(list(self.store.all('Ship')))
     pprint(r)
+
+  def do_ship(self, arg=''):
+    if arg != '':
+      ship = self.resolve_ship(arg)
+      
+      self.ship = ship
+    pprint(self.ship, 5)
+
+  ######## Atlas #########
+  def do_systems(self, page=1):
+    r = self.api.list_systems(int(page))
+    pprint(self.api.last_meta)
+  
+  def do_catalog(self, system_str=''):
+    system = self.resolve_system(system_str)
+    r = self.api.list_waypoints(system)
+    for w in r:
+      if 'MARKETPLACE' in w.traits:
+        self.api.marketplace(w)
+      if w.type == 'JUMP_GATE':
+        self.api.jumps(w)
+      if 'SHIPYARD' in w.traits:
+        self.api.shipyard(w)
+      
+  def do_system(self, system_str):
+    system = self.store.get(System, system_str)
+    r = self.api.list_waypoints(system)
+    pprint(r)
+    
+  def do_waypoints(self, grep=''):
+    loc = None
+    ship = self.has_ship()
+    loc = ship.location
+    system = loc.system
+    print(f'=== waypoints in {system}')
+    r = self.store.all_members(system, 'Waypoint')
+    for w in r:
+      
+      wname = w.symbol.split('-')[2]
+      traits = ", ".join(w.itraits())
+      typ = w.type[0]
+      if typ not in ['F','J'] and len(traits) == 0:
+        continue
+      output = ''
+      if loc:
+        dist = loc.distance(w)
+        output = f'{wname:4} {typ} {dist:6} {traits}'
+      else:
+        output = f'{wname:4} {typ} {traits}'
+      if grep == '' or grep.lower() in output.lower():
+        print(output)
+  
+  def do_members(self):
+    ship = self.has_ship()
+    system = ship.location.system
+    pprint(list(self.store.all_members(system)))
+
+  def do_wp(self, grep=''):
+    self.do_waypoints(grep)
+  
+  ######## Specials #########
+  def do_market(self, arg=''):
+    waypoint = self.resolve_waypoint(arg)
+    r =  self.api.marketplace(waypoint)
+    pprint(r, 3)
+ 
+  def do_atlas(self, state=None):
+    atlas = self.store.get(Atlas, 'ATLAS')
+    if state is not None:
+      atlas.enabled = True if state == 'on' else 'off'
+    pprint(atlas, 5)
+    
+  def do_jumps(self, waypoint_str=None):
+    if waypoint_str is None:
+      ship = self.has_ship()
+      waypoint = ship.location
+    else:
+      waypoint = self.store.get(Waypoint, waypoint_str.upper())
+    r = self.api.jumps(waypoint)
+    pprint(r, 5)
+  
+  def do_shipyard(self, w=''):
+    location = self.resolve_waypoint(w)
+    if location is None:
+      raise CommandError(f'waypoint {w} not found')
+    sy = self.api.shipyard(location)
+    pprint(sy, 5)
+  
+  ######## Commerce #########
+  def do_refuel(self, source='market'):
+    ship = self.has_ship()
+    from_cargo = source != 'market'
+    r =  self.api.refuel(ship, from_cargo=from_cargo)
+    pprint(r)
+  
+  def do_cargo(self):
+    ship = self.has_ship()
+    print(f'== Cargo {ship.cargo_units}/{ship.cargo_capacity} ==')
+    for c, units in ship.cargo.items():
+      print(f'{units:4d} {c}')
+      
+  def do_buy(self, resource, amt=None):
+    ship = self.has_ship()
+    if amt is None:
+      amt = ship.cargo_capacity - ship.cargo_units
+    self.api.buy(ship, resource.upper(), amt)
+    self.do_cargo()
+  
+  def do_sell(self, resource, amt=None):
+    ship = self.has_ship()
+    self.api.sell(ship, resource.upper(), amt)
+    self.do_cargo()
+  
+  def dump(self, ship, resource):
+    if resource == 'all':
+      for r in ship.cargo.keys():
+        self.api.jettison(ship, r)
+    else:
+      self.api.jettison(ship, resource.upper())
+      
+  def do_dump(self, resource):
+    ship = self.has_ship()
+    self.dump(ship, resource)
+    self.do_cargo()
+  
+  def do_transfer(self, resource, dship, amount=None):
+    ship = self.has_ship()
+    resource = resource.upper()
+    avail = ship.get_cargo(resource)
+    if amount is None: amount = avail
+    amount = int(amount)
+    if avail < amount:
+      raise CommandError('resource not in cargo')
+    dship = self.resolve_ship(dship)
+    self.api.transfer(ship, dship, resource, amount)
+    
+  
+  def do_purchase(self, ship_type):
+    ship = self.has_ship()
+    location = ship.location
+    ship_type = ship_type.upper()
+    if not ship_type.startswith('SHIP'):
+      ship_type = 'SHIP_' + ship_type
+    s = self.api.purchase(ship_type, location)
+    pprint(s)
+
+  ######## Mining #########
+  def do_siphon(self):
+    ship = self.has_ship()
+    data = self.api.siphon(ship)
+  
+  def do_survey(self):
+    ship = self.has_ship()
+    r = self.api.survey(ship)
+    pprint(r)
+  
+  def do_surveys(self):
+    pprint(list(self.store.all('Survey')))
+  
+  def do_extract(self, survey_str=''):
+    ship = self.has_ship()
+    survey = None
+    if survey_str != '':
+      survey = self.resolve('Survey', survey_str)
+    result = self.api.extract(ship, survey)
+    
+    symbol = mg(result,'extraction.yield.symbol')
+    units  = mg(result,'extraction.yield.units')
+    print(units, symbol)
+
+    
+  ######## Missions #########
+  def print_mission(self):
+    print(f'mission: {self.ship.mission} ({self.ship.mission_status})')
+    pprint(self.ship.mission_state)
+  
+  def do_role(self, role):
+    roles = [None, 'trader', 'probe', 'siphon', 'hauler', 'surveyor', 'miner']
+    ship = self.has_ship()
+    if role == 'none':
+      role = None
+    if role not in roles:
+      print(f'role {role} not found. Choose from {roles}')
+      return
+    ship.role = role
+    
+  def do_mission(self, arg=''):
+    ship = self.has_ship()
+    if arg:
+      self.captain.init_mission(ship, arg)
+    self.print_mission()
+  
+  def do_mrestart(self, status='init'):
+    ship = self.has_ship()
+    self.captain.restart_mission(ship, status)
+    self.print_mission()
+  
+  def do_mstep(self):
+    ship = self.has_ship()
+    self.captain.single_step(ship)
+    self.print_mission()
+      
+  def do_mreset(self):
+    ship = self.has_ship()
+    ship.mission_state = {}
+    
+  def do_mset(self, nm, val):
+    ship = self.has_ship()
+    self.captain.set_mission_param(ship, nm, val)
+  
+  def do_crew(self, arg):
+    ship = self.has_ship()
+    crew = self.resolve('Crew', arg)
+    ship.crew = crew
+    pprint(ship)
+  
+  def do_phase(self, phase):
+    self.agent.phase = phase
+    
+  ######## Crews #########
+  def do_create_crews(self):
+    crews = self.captain.create_default_crews()
+    for c in crews:
+      print(f'{c.symbol:15s} {c.site}')
+      
+  ######## Contracts #########
+  def active_contract(self):
+    for c in self.store.all('Contract'):
+      if c.accepted and not c.fulfilled: return c
+    raise CommandError('no active contract')
   
   def do_contracts(self, arg=''):
     if arg.startswith('r'):
@@ -257,139 +431,154 @@ class Commander(CommandLine):
       r = list(self.store.all('Contract'))
     pprint(r)
   
+  def do_negotiate(self):
+    ship = self.has_ship()
+    r =  self.api.negotiate(ship)
+    pprint(r)
+  
+  def do_accept(self, c):
+    contract = self.resolve('Contract', c)
+    r =  self.api.accept_contract(contract)
+    pprint(r)
+  
   def do_deliver(self):
-    if not self.has_ship(): return 
-    site = self.ship.location_str
+    ship = self.has_ship()
+    site = ship.location
     contract = self.active_contract()
     delivery = contract.unfinished_delivery()
     if delivery is None:
       raise CommandError('no delivery')
     resource = delivery['trade_symbol']
-    self.api.deliver(self.ship, resource, contract)
+    self.api.deliver(ship, resource, contract)
     pprint(contract)
   
   def do_fulfill(self):
     contract = self.active_contract()
     self.api.fulfill(contract)
     
-  def do_ship(self, arg=''):
-    if arg != '':
-      symbol = f'{self.agent.symbol}-{arg}'
-      ship = self.store.get('Ship', symbol)
-      if ship is None:
-        print('not found')
-        return
-      else:
-        self.ship = ship
-    pprint(self.ship)
-  
-  def do_pp(self):
-    pprint(self.api.last_result)
+  ######## Travel #########
+  def do_travel(self, dest):
+    ship = self.has_ship()
+    dest = self.resolve('Waypoint', dest)
+    self.captain.init_mission(ship, 'travel')
+    self.captain.set_mission_param(ship, 'dest', dest)
+    self.print_mission()
   
   def do_go(self, arg):
-    if not self.has_ship(): return
-    system = self.ship.location().system()
+    ship = self.has_ship()
+    system = ship.location.system
     symbol = f'{system}-{arg}'
     dest = self.resolve('Waypoint', symbol)
-    self.api.navigate(self.ship, dest)
-    pprint(self.ship)
+    self.api.navigate(ship, dest)
+    pprint(ship)
   
   def do_dock(self):
-    if not self.has_ship(): return
-    self.api.dock(self.ship)
-    pprint(self.ship)
+    ship = self.has_ship()
+    self.api.dock(ship)
+    pprint(ship)
   
   def do_orbit(self):
-    if not self.has_ship(): return
-    self.api.orbit(self.ship)
-    pprint(self.ship)
+    ship = self.has_ship()
+    self.api.orbit(ship)
+    pprint(ship)
   
-  def do_negotiate(self):
-    if not self.has_ship(): return
-    r =  self.api.negotiate(self.ship)
-    pprint(r)
+  def do_speed(self, speed):
+    ship = self.has_ship()
+    speed = speed.upper()
+    speeds = ['DRIFT', 'STEALTH','CRUISE','BURN']
+    if speed not in speeds:
+      print('please choose from:', speeds)
+    self.api.flight_mode(ship, speed)
   
-  def do_refuel(self):
-    if not self.has_ship(): return
-    r =  self.api.refuel(self.ship)
-    pprint(self.ship)
+  def do_jump(self, waypoint_str):
+    ship = self.has_ship()
+    w = self.resolve('Waypoint', waypoint_str)
+    self.api.jump(ship, w)
+    pprint(ship)
     
-  def do_accept(self, c):
-    contract = self.resolve('Contract', c)
-    r =  self.api.accept_contract(contract)
-    pprint(r)
+  ######## Analysis #########
+  def do_server(self):
+    data = self.api.status()
+    pprint(data)
   
-  def do_market(self, arg=''):
-    if arg == '':
-      if not self.has_ship(): return
-      waypoint = self.ship.location()
+  def do_highscore(self):
+    data = self.api.status()
+    leaders = mg(data, 'leaderboards.mostCredits')
+    for l in leaders:
+      a = mg(l,'agentSymbol')
+      c = mg(l, 'credits')
+      print(f'{a:15s} {c}')
+    
+  def do_stats(self):
+    total = 0
+    for t in self.store.data:
+      num = len(self.store.data[t])
+      nam = t.__name__
+      total += num
+      print(f'{num:5d} {nam}')
+    print(f'{total:5d} total')
+  
+  def do_defrag(self):
+    self.store.defrag()
+    
+  def do_obj(self, oid):
+    if not '.' in oid:
+      print('Usage: obj SYMBOL.ext')
+      return
+    symbol, ext = oid.split('.')
+    symbol = symbol.upper()
+    if not ext in self.store.extensions:
+      raise CommandError('unknown extension')
+    typ = self.store.extensions[ext]
+    obj = self.store.get(typ, symbol)
+    if obj is None:
+      raise CommandError('object not found')
+    pprint(obj.__getstate__())
+    print('=== store ===')
+    h = self.store.get_header(obj)
+    if h:
+      pprint(h, 3)
     else:
-      waypoint = self.resolve('Waypoint', arg)
-    r =  self.api.marketplace(waypoint)
-    pprint(r)
-  
-  def do_cargo(self):
-    if not self.has_ship(): return
-    for c, units in self.ship.cargo.items():
-      print(f'{units:4d} {c}')
+      print('Not stored')
+    print('Dirty: ', obj in self.store.dirty_objects)
+    
+  def do_query(self, resource):
+    ship = self.has_ship()
+    location = ship.location
+    resource = resource.upper()
+    print('Found markets:')
+    for typ, m, d, plen in find_closest_markets(self.c, resource, 'buy,exchange',location):
+      price = '?'
+      if resource in m.prices:
+        price = m.prices[resource]['buy']
+      print(m, typ[0], f'{plen-1:3} hops {price}')
       
-  def do_buy(self, resource, amt=None):
-    if not self.has_ship(): return
-    if amt is None:
-      amt = self.ship.cargo_capacity - self.ship.cargo_units
-    self.api.buy(self.ship, resource.upper(), amt)
-    self.do_cargo()
-  
-  def do_sell(self, resource):
-    if not self.has_ship(): return
-    self.api.sell(self.ship, resource.upper())
-    self.do_cargo()
+  def do_findtrade(self):
+    ship = self.has_ship()
+    system = ship.location.system
+    t = find_trade(self.c, system)
+    pprint(t)
     
-  def do_dump(self, resource):
-    if not self.has_ship(): return
-    self.api.jettison(self.ship, resource.upper())
-    self.do_cargo()
-  
-  def do_shipyard(self):
-    if not self.has_ship(): return
-    location = self.ship.location()
-    data = self.api.shipyard(location)
-    for s in must_get(data, 'ships'):
-      print(s['type'], s['purchasePrice'])
-  
-  def do_jump(self, system_str):
-    if not self.has_ship(): return
-    if '-' not in system_str:
-      sector = self.ship.location_str.split('-')[0]
-      system_str = f'{sector}-{system_str}'
-    system = self.resolve('System', system_str)
-    self.api.jump(self.ship, system)
-    pprint(self.ship)
+  def do_prices(self, resource=None):
+    ship = self.has_ship()
+    system = ship.location.system
+    prices = prices(self.c, system)
+    if resource is not None:
+      prices = {resource: prices[resource.upper()]}
     
-  def do_purchase(self, ship_type):
-    if not self.has_ship(): return
-    location = self.ship.location()
-    ship_type = ship_type.upper()
-    if not ship_type.startswith('SHIP'):
-      ship_type = 'SHIP_' + ship_type
-    s = self.api.purchase(ship_type, location)
-    pprint(s)
-
-  def do_survey(self):
-    if not self.has_ship(): return
-    r = self.api.survey(self.ship)
-    pprint(r)
+    for res, p in prices.items():
+      print('==' + res)
+      for m in p:
+        print(f"{m['wp'].symbol:12s} {m['category']}  {m['volume']:5d} {m['buy']:5d} {m['sell']:5d}")
   
-  def do_surveys(self):
-    pprint(list(self.store.all('Survey')))
+  def do_path(self, waypoint_str):
+    ship = self.has_ship()
+    w = self.resolve('Waypoint', waypoint_str)
+    p = find_nav_path(self.c, ship.location, w, ship.fuel_capacity)
+    pprint(p)
   
-  def do_extract(self, survey_str=''):
-    if not self.has_ship(): return
-    survey = None
-    if survey_str != '':
-      survey = self.resolve('Survey', survey_str)
-    result = self.api.extract(self.ship, survey)
+  def do_list(self, klass):
+    ship = self.has_ship()
+    for o in self.store.all_members(klass, ship.location.system):
+      print(o)
     
-    symbol = mg(result,'extraction.yield.symbol')
-    units  = mg(result,'extraction.yield.units')
-    print(units, symbol)

nullptr/context.py

@@ -0,0 +1,6 @@
+class Context:
+  def __init__(self, store, api=None, captain=None, general=None):
+    self.store = store
+    self.api = api
+    self.captain = captain
+    self.general = general

nullptr/general.py

@@ -0,0 +1,128 @@
+from nullptr.util import *
+from nullptr.analyzer import find_gas, find_metal
+class GeneralError(AppError):
+  pass
+
+class General:
+  def __init__(self, context):
+    self.store = context.store
+    self.api = context.api
+    self.c = context
+    agents = self.store.all('Agent')
+    self.agent = next(agents, None)
+    self.phases = {
+      'init': self.phase_startup,
+      'probes': self.phase_probes,
+      'trade': self.phase_trade,
+      'mine': self.phase_mine,
+      'siphon': self.phase_siphon,
+      'rampup': self.phase_rampup,
+      'gate': self.phase_gate
+    }
+  
+  def setup(self):
+    self.create_default_crews()
+  
+  def find_shipyard(self, stype):
+    occ = [s.location.symbol for s in self.store.all('Ship') if s.status != 'IN_TRANSIT']
+    best_price = -1
+    best_yard = None
+    for shipyard in self.store.all('Shipyard'):
+      if stype in shipyard.prices:
+        price = shipyard.prices[stype]
+        if shipyard.symbol in occ:
+          if best_yard is None or price < best_price:
+            best_yard = shipyard
+            best_price = price
+    return best_yard, best_price
+    
+  def maybe_purchase(self, stype, role):
+    sy, price = self.find_shipyard(stype)
+    if sy is None:
+      return False
+    traders = [s for s in self.store.all('Ship') if s.role == 'trader']
+    safe_buffer = len(traders) * 100000 + 100000
+    #print(safe_buffer, price, sy)
+    if self.agent.credits < safe_buffer + price:
+      return # cant afford it!
+    ship = self.c.api.purchase(stype, sy)
+    ship.role = role
+    
+  def tick(self):
+    phase = self.agent.phase
+    if phase not in self.phases:
+      raise GeneralError('Invalid phase')
+    hdl = self.phases[phase]
+    new_phase = hdl()
+    if new_phase:
+      self.agent.phase = new_phase
+  
+    
+  def phase_startup(self):
+    # * first pricing info
+    # * probe at shipyard that sells probes
+    ag = self.agent.symbol
+    command = self.store.get('Ship', f'{ag}-1')
+    probe = self.store.get('Ship', f'{ag}-2')
+    if command.role is None:
+      command.role = 'probe'
+    if probe.role is None:
+      probe.role = 'sitter'
+    system = command.location.system
+    markets = list(self.store.all_members(system, 'Marketplace'))
+    discovered = len([m for m in markets if m.last_prices > 0])
+    if discovered > len(markets) // 2:
+      return 'probes'
+    
+  def phase_probes(self):
+    ag = self.agent.symbol
+    command = self.store.get('Ship', f'{ag}-1')
+    # * probes on all markets
+    if command.role != 'trader':
+      command.role = 'trader'
+      self.c.captain.init_mission(command, 'none')
+    self.maybe_purchase('SHIP_PROBE', 'sitter')
+    sitters = [s for s in self.store.all('Ship') if s.role == 'sitter']
+    markets = [m for m in self.store.all('Marketplace')]
+    if len(sitters) >= len(markets):
+      return 'trade'
+    
+  def phase_trade(self):
+    self.maybe_purchase('SHIP_LIGHT_HAULER', 'trader')
+    traders = list([s for s in self.store.all('Ship') if s.role == 'trader'])
+    if len(traders) >= 19:
+      return 'mine'
+    
+  
+  def phase_mine(self):
+    # metal mining crew
+    pass
+  
+  def phase_siphon(self):
+    # siphon crew
+    pass
+  
+  def phase_rampup(self):
+    # stimulate markets for gate building
+    pass
+  
+  def phase_gate(self):
+    # build the gate
+    pass
+    
+  def create_default_crews(self):
+    system = self.api.agent.headquarters.system
+    gas_w = find_gas(self.c, system)
+    metal_w = find_metal(self.c, system)
+    metal = self.store.get('Crew', 'METAL', create=True)
+    metal.site = metal_w
+    metal.resources = ['COPPER_ORE','IRON_ORE','ALUMINUM_ORE']
+    gas = self.store.get('Crew', 'GAS', create=True)
+    gas.site = gas_w
+    gas.resources = ['HYDROCARBON','LIQUID_HYDROGEN','LIQUID_NITROGEN']
+    return [gas, metal]
+    
+    
+    
+    
+    

nullptr/missions/__init__.py

@@ -1,23 +1,32 @@
 from nullptr.missions.survey import SurveyMission
 from nullptr.missions.mine import MiningMission
-from nullptr.missions.haul import HaulMission
+from nullptr.missions.trade import TradeMission
 from nullptr.missions.travel import TravelMission
 from nullptr.missions.probe import ProbeMission
+from nullptr.missions.idle import IdleMission
+from nullptr.missions.siphon import SiphonMission
+from nullptr.missions.haul import HaulMission
+from nullptr.missions.sit import SitMission
 
 def get_mission_class( mtype):
   types = {
     'survey': SurveyMission,
     'mine': MiningMission,
-    'haul': HaulMission,
+    'trade': TradeMission,
     'travel': TravelMission,
-    'probe': ProbeMission
+    'probe': ProbeMission,
+    'idle': IdleMission,
+    'siphon': SiphonMission,
+    'haul': HaulMission,
+    'sit': SitMission,
+    
   }
   if mtype not in types:
     raise ValueError(f'invalid mission type {mtype}')
   return types[mtype]
   
-def create_mission(mtype, ship, store, api):
+def create_mission(mtype, ship, c):
   typ = get_mission_class(mtype)
-  m = typ(ship, store, api)
+  m = typ(ship, c)
   return m
 

nullptr/missions/base.py

@@ -5,12 +5,13 @@ from nullptr.models.contract import Contract
 from nullptr.models.system import System
 from nullptr.models.survey import Survey
 from nullptr.models.ship import Ship
-from nullptr.analyzer import Analyzer
+from nullptr.analyzer import *
 from time import time
 from functools import partial
 import logging
 from nullptr.util import *
 
+
 class MissionError(Exception):
   pass
   
@@ -26,9 +27,14 @@ class MissionParam:
     elif self.cls == int:
       return int(val)
     elif self.cls == list:
-      return [i.strip() for i in val.split(',')]
+      if type(val) == str:
+        return [i.strip() for i in val.split(',')]
+      return val
     elif issubclass(self.cls, Base):
-      data = store.get(self.cls, val)
+      if type(val) == str:
+        data = store.get(self.cls, val)
+      else:
+        data = val
       if data is None:
         raise ValueError('object not found')
       return data.symbol
@@ -42,13 +48,18 @@ class Mission:
       
     }
   
-  def __init__(self, ship, store, api):
+  def __init__(self, ship, context):
     self.ship = ship
-    self.store = store
-    self.api = api
+    self.c = context
+    self.store = context.store
+    self.api = context.api
+    self.wait_for = None
     self.next_step = 0
-    self.analyzer = Analyzer(self.store)
+    self.setup()
 
+  def setup(self):
+    pass
+    
   def sts(self, nm, v):
     if issubclass(type(v), Base):
       v = v.symbol
@@ -56,6 +67,8 @@ class Mission:
   
   def rst(self, typ, nm):
     symbol = self.st(nm)
+    if symbol is None:
+      return None
     return self.store.get(typ, symbol)
     
   def st(self, nm):
@@ -67,7 +80,17 @@ class Mission:
     if nw is None:
       return self.ship.mission_status
     else:
-      self.ship.mission_status = nw
+      steps = self.steps()
+      if nw in ['init','done', 'error']:
+        self.ship.mission_status = nw 
+        return
+      elif nw not in steps:
+        self.ship.log(f"Invalid mission status {nw}", 1)
+        self.ship.mission_status = 'error'
+        return 
+      wait_for = steps[nw][2] if len(steps[nw]) > 2 else None
+      self.wait_for = wait_for
+      self.ship.mission_status = nw 
       
   def start_state(self):
     return 'done'
@@ -89,16 +112,26 @@ class Mission:
     }
   
   def step_done(self):
-    logging.info(f'mission finished for {self.ship}')
+    self.ship.log(f'mission {type(self).__name__} finished with balance {self.balance()}', 3)
     
-  def is_waiting(self):
-    return self.next_step > time()
+  def get_prio(self):
+    if self.next_step > time() or self.ship.cooldown > time() or self.ship.arrival > time():
+      return 0
+    if self.wait_for is not None:
+      p = int(self.wait_for())
+      if p > 0:
+        self.wait_for = None
+      return p
+    return 3
+      
   
   def is_finished(self):
     return self.status() in ['done','error']
   
   def is_ready(self):
-    return not self.is_waiting() and not self.is_finished()
+    if self.is_finished():
+      return 0
+    return self.get_prio()
   
   def step(self):
     steps = self.steps()
@@ -106,28 +139,45 @@ class Mission:
       self.init_state()
     status = self.status()
     if not status in steps:
-      logging.warning(f"Invalid mission status {status}")
+      self.ship.log(f"Invalid mission status {status}", 1)
       self.status('error')
       return
-    handler, next_step = steps[status]
+    
+    handler = steps[status][0]
+    next_step = steps[status][1]
+    
     try:
       result = handler()
     except Exception as e:
-      logging.error(e, exc_info=True)
+      self.ship.log(fmtex(e))
+      self.ship.log(self.api.last_result)
       self.status('error')
       return
     if type(next_step) == str:
       self.status(next_step)
     elif type(next_step) == dict:
       if result not in next_step:
-        logging.warning(f'Invalid step result {result}')
+        self.ship.log(f'Invalid step result {result}', 1)
         self.status('error')
         return
       else:
+        if result is None: result=''
         self.status(next_step[result])
-    print(f'{self.ship} {status} -> {self.status()}')
+    self.ship.log(f'{status} {result} -> {self.status()}', 8)
 
 class BaseMission(Mission):
+  def balance(self, amt=0):
+    if type(amt) == dict:
+      amt = self.api.transaction_cost(amt)
+    balance = self.st('balance')
+    if balance is None: balance = 0
+    balance += amt
+    self.sts('balance', balance)
+    return balance
+  
+  def step_pass(self):
+    pass
+    
   def step_go_dest(self):
     destination = self.rst(Waypoint, 'destination')
     if self.ship.location() == destination:
@@ -142,11 +192,20 @@ class BaseMission(Mission):
     self.api.navigate(self.ship, site)
     self.next_step = self.ship.arrival
     
+  def step_market(self):
+    loc = self.ship.location
+    self.api.marketplace(loc)
+  
+  def step_shipyard(self):
+    loc = self.ship.location
+    if 'SHIPYARD' in loc.traits:
+      self.api.shipyard(loc)
+      
   def step_unload(self):
-    contract = self.rst(Contract, 'contract')
     delivery = self.st('delivery')
     if delivery == 'sell':
       return self.step_sell(False)
+    contract = self.rst(Contract, 'contract') 
     typs = self.ship.deliverable_cargo(contract)
     if len(typs) == 0:
       return 'done'
@@ -157,94 +216,123 @@ class BaseMission(Mission):
       return 'more'
 
   def step_sell(self, except_resource=True):
-    target = self.st('resource')
-    market = self.store.get('Marketplace', self.ship.location_str)
+    market = self.store.get('Marketplace', self.ship.location.symbol)
     sellables = market.sellable_items(self.ship.cargo.keys())
-    if target in sellables and except_resource:
-      sellables.remove(target)
     if len(sellables) == 0:
       return 'done'
-    self.api.sell(self.ship, sellables[0])
-    if len(sellables) == 1:
+    resource = sellables[0]
+    volume = market.volume(resource)
+    
+    amt_cargo = self.ship.get_cargo(resource)
+  
+    amount = min(amt_cargo, volume)
+    res = self.api.sell(self.ship, resource, amount)
+    self.balance(res)
+    if len(sellables) == 1 and amt_cargo == amount:
       return 'done'
     else:
       return 'more'
   
-  def step_load(self):
-    cargo_space = self.ship.cargo_capacity - self.ship.cargo_units
-    resource = self.st('resource')
-    self.api.buy(self.ship, resource, cargo_space)
-  
   def step_travel(self):
     traject = self.st('traject')
     if traject is None or traject == []:
-      return 'done'
-    dest = self.store.get(Waypoint, traject[-1])
-    loc = self.ship.location()
-    print(dest, loc)
-    if dest == loc:
-      self.sts('traject', None)
-      return 'done'
+      return
+    dest = traject[-1]
+    loc = self.ship.location
+    
     hop = traject.pop(0)
-    if len(hop.split('-')) == 3:
+    if type(hop) == Waypoint:
       self.api.navigate(self.ship, hop)
       self.next_step = self.ship.arrival
     else:
       self.api.jump(self.ship, hop)
       self.next_step = self.ship.cooldown
-    if traject == []:
-      traject= None
     self.sts('traject', traject)
+    
+  def step_navigate_traject(self):
+    traject = self.st('traject')
+    
+    
+    loc = self.ship.location
+    
+    if traject is None or traject == []:
+      return 'done'
+    dest =traject[-1]
+    if dest == loc:
+      return 'done'
     return 'more'
-  
+    
   def step_calculate_traject(self, dest):
     if type(dest) == str:
       dest = self.store.get(Waypoint, dest)
-    loc = self.ship.location()
-    loc_sys = self.store.get(System, loc.system())
-    loc_jg = self.analyzer.get_jumpgate(loc_sys)
-    dest_sys = self.store.get(System, dest.system())
-    dest_jg = self.analyzer.get_jumpgate(dest_sys)
+    loc = self.ship.location
+    loc_sys = loc.system
+    
+    loc_jg = get_jumpgate(self.c, loc_sys)
+    loc_jg_wp = self.store.get(Waypoint, loc_jg.symbol)
+    dest_sys = dest.system
+    dest_jg = get_jumpgate(self.c, dest_sys)
     if dest_sys == loc_sys:
-      result = [dest.symbol]
+      result = find_nav_path(self.c, loc, dest, self.ship.range())
       self.sts('traject', result)
-      return
-    path = self.analyzer.find_path(loc_sys, dest_sys)
+      return 'done' if len(result) == 0 else 'more'
+    path = find_jump_path(self.c, loc_sys, dest_sys)
     result = []
     if loc.symbol != loc_jg.symbol:
-      result.append(loc_jg.symbol)
-    result += [s.symbol for s in path[1:]]
+      result.append(loc_jg_wp)
+    result += [s for s in path[1:]]
     if dest_jg.symbol != dest.symbol:
-      result.append(dest.symbol)
+      result.append(dest)
     self.sts('traject', result)
     print(result)
-    return result
+    return 'more'
     
   def step_dock(self):
+    if self.ship.status == 'DOCKED':
+      return
     self.api.dock(self.ship)
  
   def step_refuel(self):
     if self.ship.fuel_capacity == 0:
       return
-    if self.ship.fuel_current / self.ship.fuel_capacity < 0.5:
-      try:
-        self.api.refuel(self.ship)
-      except Exception as e:
-        pass
+    #if  self.ship.fuel_capacity - self.ship.fuel_current > 100:
+    try:
+      self.api.refuel(self.ship)
+    except Exception as e:
+      pass
        
   def step_orbit(self):
+    if self.ship.status != 'DOCKED':
+      return
     self.api.orbit(self.ship)
   
   def travel_steps(self, nm, destination, next_step):
     destination = self.st(destination)
     calc = partial(self.step_calculate_traject, destination)
-    return {
-      f'travel-{nm}': (self.step_orbit, f'calc-trav-{nm}'),
-      f'calc-trav-{nm}': (calc, f'go-{nm}'),
-      f'go-{nm}': (self.step_travel, {
-        'done': f'dock-{nm}',
-        'more': f'go-{nm}'
+    steps = {
+      
+      f'travel-{nm}': (calc, {
+      'more': f'dock-{nm}',
+      'done': next_step
       }),
       f'dock-{nm}': (self.step_dock, f'refuel-{nm}'),
-      f'refuel-{nm}': (self.step_refuel, next_step)
+      f'refuel-{nm}': (self.step_refuel, f'orbit-{nm}'),
+      f'orbit-{nm}': (self.step_orbit, f'go-{nm}'),
+      f'go-{nm}': (self.step_travel, f'nav-{nm}'),
+      f'nav-{nm}': (self.step_navigate_traject, {
+        'done': next_step,
+        'more': f'dock-{nm}'
+        })
     }
+    if self.ship.fuel_capacity == 0:
+      steps = {
+      
+      f'travel-{nm}': (calc, f'orbit-{nm}'),
+      f'orbit-{nm}': (self.step_orbit, f'go-{nm}'),
+      f'go-{nm}': (self.step_travel, f'nav-{nm}'),
+      f'nav-{nm}': (self.step_navigate_traject, {
+        'done': next_step,
+        'more': f'go-{nm}'
+        }),
+    }
+    return steps

nullptr/missions/extraction.py

@@ -0,0 +1,28 @@
+from nullptr.missions.base import BaseMission
+
+class ExtractionMission(BaseMission):
+  def find_hauler(self, r):
+    for s in self.store.all('Ship'):
+      if s.mission != 'haul': continue
+      if s.location != self.ship.location:
+        continue
+      if s.mission_status != 'load':
+        continue
+      if r not in s.mission_state['resources']: continue
+      return s
+    return None
+      
+  def step_unload(self):
+    if len(self.ship.cargo) == 0:
+      return 'done'
+    r = list(self.ship.cargo.keys())[0]
+    amt = self.ship.cargo[r]
+    h = self.find_hauler(r)
+    if h is None:
+      self.api.jettison(self.ship, r)
+    else:
+      space = h.cargo_space()
+      amt = min(space, amt)
+      if amt > 0:
+        self.api.transfer(self.ship, h, r, amt)
+    return 'more'

nullptr/missions/haul.py

@@ -1,25 +1,52 @@
 from nullptr.missions.base import BaseMission, MissionParam
 from nullptr.models.waypoint import Waypoint
-from nullptr.models.survey import Survey
-from nullptr.models.contract import Contract
+
 class HaulMission(BaseMission):
   def start_state(self):
     return 'travel-to'
   
+  def step_turn(self):
+    self.ship.log('starting haul load')
+  
+  def wait_turn(self):
+    for s in self.store.all('Ship'):
+      if s.mission != 'haul': continue
+      if s.location != self.ship.location:
+        continue
+      if s.mission_state['dest'] != self.st('dest'):
+        continue
+      if s.mission_status != 'load':
+        continue
+      return 0
+    return 5
+  
+  def step_load(self):
+    pass
+    
+  def cargo_full(self):
+    if self.ship.cargo_space() == 0:
+      return 5
+    return 0
+    
   @classmethod
   def params(cls):
     return {
       'site': MissionParam(Waypoint, True),
-      'resource': MissionParam(str, True),
       'dest': MissionParam(Waypoint, True),
-      'delivery': MissionParam(str, True, 'deliver'),
-      'contract': MissionParam(Contract, False)
+      'resources': MissionParam(list, True)
     }
   
   def steps(self):
     return {
-      **self.travel_steps('to', 'site', 'load'),
-      'load': (self.step_load, 'travel-back'),
-      **self.travel_steps('back', 'dest', 'unload'),
-      'unload': (self.step_unload, 'travel-to'),
+      **self.travel_steps('to', 'site', 'wait-turn'),
+       'wait-turn': (self.step_turn, 'load', self.wait_turn),
+      'load': (self.step_load, 'travel-back', self.cargo_full),
+      **self.travel_steps('back', 'dest', 'dock-dest'),
+      'dock-dest': (self.step_dock, 'unload'),
+      'unload': (self.step_sell, {
+         'more': 'unload',
+         'done': 'market-dest'
+        }),
+      'market-dest': (self.step_market, 'report'),
+      'report': (self.step_done, 'done')
     }

nullptr/missions/idle.py

@@ -0,0 +1,26 @@
+from nullptr.missions.base import BaseMission, MissionParam
+import time
+
+class IdleMission(BaseMission):
+  def start_state(self):
+    return 'start'
+  
+  def step_wait(self):
+    self.next_step = int(time.time()) + self.st('seconds')
+ 
+  def step_idle(self):
+    pass
+    
+  @classmethod
+  def params(cls):
+    return {
+      'seconds': MissionParam(int, True)
+    }
+  
+  def steps(self):
+    return { 
+    'start': (self.step_wait, 'wait'),
+    'wait': (self.step_idle, 'done')
+    }
+    
+    

nullptr/missions/mine.py

@@ -3,16 +3,14 @@ from nullptr.models.waypoint import Waypoint
 from nullptr.models.survey import Survey
 from nullptr.models.contract import Contract
 from nullptr.util import *
+from nullptr.missions.extraction import ExtractionMission
 
-class MiningMission(BaseMission): 
+class MiningMission(ExtractionMission): 
   @classmethod
   def params(cls):
     return {
       'site': MissionParam(Waypoint, True),
-      'resource': MissionParam(str, True),
-      'dest': MissionParam(Waypoint, True),
-      'delivery': MissionParam(str, True, 'deliver'),
-      'contract': MissionParam(Contract, False)
+      'resources': MissionParam(list, True)
     }
   
   def start_state(self):
@@ -20,61 +18,43 @@ class MiningMission(BaseMission):
     
   def steps(self):
     return {
-      **self.travel_steps('to', 'site', 'orbit1'),
-      'orbit1': (self.step_orbit, 'extract'),
-      'extract': (self.step_extract, {
-        'done': 'dock',
-        'more': 'extract'
-      }),
-      'dock': (self.step_dock, 'sell'),
-      'sell': (self.step_sell, {
-        'more': 'sell',
-        'done': 'orbit2',
-      }),
-      'orbit2': (self.step_orbit, 'jettison'),
-      'jettison': (self.step_dispose, {
-        'more': 'jettison',
-        'done': 'extract',
-        'full': 'travel-back'
-      }),
-      **self.travel_steps('back', 'dest', 'unload'),
-      'unload': (self.step_unload, {
-        'done': 'travel-to',
-        'more': 'unload'
-      }),
+      **self.travel_steps('to', 'site', 'extract'),
+        'extract': (self.step_extract, {
+            'more': 'extract',
+            'done': 'unload'
+        }),
+        'unload': (self.step_unload, {
+            'more': 'unload',
+            'done': 'done'
+        })
     }
   
   def get_survey(self):
-    resource = self.st('resource')
+    resources = self.st('resources')
     site = self.rst(Waypoint,'site')
+    best_score = 0
+    best_survey = None
     # todo optimize
     for s in self.store.all(Survey):
-      if resource in s.deposits and site.symbol == s.waypoint():
-        return s
-    return None
+      if site != s.waypoint:
+        continue
+      good = len([1 for r in s.deposits if r in resources])
+      total = len(s.deposits)
+      score = good / total
+      if score > best_score:
+        best_score = score
+        best_survey = s
+    return best_survey
     
   def step_extract(self):
     survey = self.get_survey()
-    print('using survey:', str(survey)) 
     result = self.api.extract(self.ship, survey)
     symbol = sg(result,'extraction.yield.symbol')
     units  = sg(result,'extraction.yield.units')
-    print('extracted:', units, symbol)
     self.next_step = self.ship.cooldown
-    if self.ship.cargo_units < self.ship.cargo_capacity:
+    if self.ship.cargo_space() > 5:
       return 'more'
     else:
       return 'done'
   
-  def step_dispose(self):
-    contract = self.rst(Contract, 'contract')
-    typs = self.ship.nondeliverable_cargo(contract)
-    if len(typs) > 0:
-      self.api.jettison(self.ship, typs[0])
-    if len(typs) > 1:
-      return 'more'
-    elif self.ship.cargo_units > self.ship.cargo_capacity - 3:
-      return 'full'
-    else:
-      return 'done'
-  
+      

nullptr/missions/probe.py

@@ -20,10 +20,6 @@ class ProbeMission(BaseMission):
  
     }
   
-  def step_market(self):
-    loc = self.ship.location()
-    self.api.marketplace(loc)
-  
   def step_next_hop(self):
     hops = self.st('hops')
     next_hop = self.st('next-hop')

nullptr/missions/siphon.py

@@ -0,0 +1,38 @@
+from nullptr.missions.base import MissionParam
+from nullptr.missions.extraction import ExtractionMission
+from nullptr.models.waypoint import Waypoint
+
+class SiphonMission(ExtractionMission):
+  def start_state(self):
+    return 'travel-to'
+    
+  @classmethod
+  def params(cls):
+    return {
+      'site': MissionParam(Waypoint, True),
+    }
+  
+  def step_siphon(self):
+    result = self.api.siphon(self.ship)
+    self.next_step = self.ship.cooldown
+    if self.ship.cargo_space() > 5:
+      return 'more'
+    else:
+      return 'full'
+  
+    
+  def steps(self):
+    return {
+      **self.travel_steps('to', 'site', 'siphon'),
+        'siphon': (self.step_siphon, {
+            'more': 'siphon',
+            'full': 'unload'
+        }),
+        'unload': (self.step_unload, {
+            'more': 'unload',
+            'done': 'done'
+        })
+    }
+  
+    
+  

nullptr/missions/sit.py

@@ -0,0 +1,25 @@
+from nullptr.missions.base import BaseMission, MissionParam
+from nullptr.models.waypoint import Waypoint
+from time import time
+
+class SitMission(BaseMission):
+  def start_state(self):
+    return 'travel-to'
+  
+  @classmethod
+  def params(cls):
+    return {
+      'dest': MissionParam(Waypoint, True)
+    }
+  
+  def steps(self):
+    return { 
+      **self.travel_steps('to', 'dest', 'market'),
+      'sit': (self.step_sit, 'market'),
+      'market': (self.step_market, 'shipyard'),
+      'shipyard': (self.step_shipyard, 'sit')
+      }
+  
+  def step_sit(self):
+    self.next_step = time() + 15 * 60
+      

nullptr/missions/survey.py

@@ -1,11 +1,19 @@
 from nullptr.missions.base import BaseMission, MissionParam
+from nullptr.models.waypoint import Waypoint
 
 class SurveyMission(BaseMission):
   def start_state(self):
-    return 'survey'
+    return 'travel-to'
   
+  @classmethod
+  def params(cls):
+    return {
+      'site': MissionParam(Waypoint, True),
+    }
+    
   def steps(self):
     return {
+      **self.travel_steps('to', 'site', 'survey'),
       'survey': (self.step_survey, 'survey')
     }
   

nullptr/missions/trade.py

@@ -0,0 +1,54 @@
+from nullptr.missions.base import BaseMission, MissionParam
+from nullptr.models.waypoint import Waypoint
+from nullptr.models.survey import Survey
+from nullptr.models.contract import Contract
+from nullptr.analyzer import find_deal
+
+class TradeMission(BaseMission):
+  def start_state(self):
+    return 'travel-to'
+  
+  def step_load(self):
+    credits = self.api.agent.credits
+    cargo_space = self.ship.cargo_capacity - self.ship.cargo_units
+    smkt = self.store.get('Marketplace', self.st('site'))
+    dmkt = self.store.get('Marketplace', self.st('dest'))
+    resource = find_deal(self.c, smkt, dmkt)
+    if resource is None:
+      return 'done'
+    price = smkt.buy_price(resource)
+    volume = smkt.volume(resource)
+    affordable = credits // price
+    amount = min(cargo_space, affordable, volume)
+    if amount == 0:
+      return 'done'
+    res = self.api.buy(self.ship, resource, amount)
+    self.balance(res)
+    return 'done' if amount == cargo_space else 'more'
+  
+  @classmethod
+  def params(cls):
+    return {
+      'site': MissionParam(Waypoint, True),
+      'dest': MissionParam(Waypoint, True),
+    }
+  
+  def steps(self):
+    return {
+      **self.travel_steps('to', 'site', 'dock'),
+       'dock': (self.step_dock, 'market-pre'),
+      'market-pre': (self.step_market, 'load'),
+      'load': (self.step_load, {
+        'more': 'market-pre',
+        'done': 'market-post'
+      }),
+      'market-post': (self.step_market, 'travel-back'),
+      **self.travel_steps('back', 'dest', 'dock-dest'),
+      'dock-dest': (self.step_dock, 'unload'),
+      'unload': (self.step_sell, {
+         'more': 'unload',
+         'done': 'market-dest'
+        }),
+      'market-dest': (self.step_market, 'report'),
+      'report': (self.step_done, 'done')
+    }

nullptr/models/__init__.py

@@ -0,0 +1,15 @@
+from nullptr.models.base import Base
+from nullptr.models.waypoint import Waypoint
+from nullptr.models.sector import Sector
+from nullptr.models.system import System
+from nullptr.models.agent import Agent
+from nullptr.models.marketplace import Marketplace
+from nullptr.models.jumpgate import Jumpgate
+from nullptr.models.ship import Ship
+from nullptr.models.contract import Contract
+from nullptr.models.survey import Survey
+from nullptr.models.atlas import Atlas
+from nullptr.models.crew import Crew
+from nullptr.models.shipyard import Shipyard
+
+__all__ = [ 'Waypoint', 'Sector', 'Ship', 'Survey', 'System', 'Agent', 'Marketplace', 'Jumpgate', 'Contract', 'Base', 'Atlas', 'Crew', 'Shipyard' ]

nullptr/models/agent.py

@@ -1,16 +1,18 @@
 from .base import Base
+from nullptr.models.waypoint import Waypoint
 
 class Agent(Base):
   def define(self):
     self.token: str = None
     self.credits: int = 0
-  
+    self.headquarters: Waypoint = None
+    self.phase = 'init'
+    
   def update(self, d):
     self.seta('credits', d)
+    getter = self.store.getter(Waypoint, create=True)
+    self.seta('headquarters', d, interp=getter)
     
-  def path(self):
-    return f'{self.symbol}.{self.ext()}'
-  
   @classmethod
   def ext(self):
     return 'agt'
@@ -18,5 +20,6 @@ class Agent(Base):
   def f(self, detail=1):
     r = super().f(detail)
     if detail >2:
-      r += f' c:{self.credits}'
+      r += f' c:{self.credits}\n'
+      r+= f'phase: {self.phase}'
     return r

nullptr/models/atlas.py

@@ -0,0 +1,19 @@
+from .base import Base
+
+class Atlas(Base):
+  @classmethod
+  def ext(self):
+    return 'atl'
+  
+  def define(self):
+    self.total_pages = 0
+    self.seen_pages = 0
+    self.enabled = False
+    
+  def f(self, detail=1):
+    r = super().f(detail)
+    if detail >2:
+      if not self.enabled:
+        r += ' OFF'
+      r += f' {self.seen_pages}/{self.total_pages}'
+    return r

nullptr/models/base.py

@@ -1,26 +1,68 @@
 from copy import deepcopy
 from nullptr.util import sg
 
+class Reference:
+  def __init__(self, typ, symbol, store):
+    self.typ = typ
+    self.symbol = symbol
+    self.store = store
+  
+  @classmethod
+  def create(cls, obj):
+    o = cls(type(obj), obj.symbol, obj.store)
+    return o
+    
+  def resolve(self):
+    return self.store.get(self.typ, self.symbol)
+  
+  def f(self, detail):
+    return f'{self.symbol}.{self.typ.ext()}'
+    
+  def __repr__(self):
+    return f'*REF*{self.symbol}.{self.typ.ext()}'
+
 class Base:
   identifier = 'symbol'
-  symbol: str
-  store: object
   
   def __init__(self, symbol, store):
-    self.disable_dirty = True
+    self._disable_dirty = True
+    self._file_offset = None
     self.store = store
     self.symbol = symbol
     self.define()
-    self.disable_dirty = False
+    self._disable_dirty = False
+  
+  def __setstate__(self, d):
+    self.__init__(d['symbol'], d['store'])
+    self.__dict__.update(d)
+  
+  def __getstate__(self):
+    return {k:v for k,v in self.__dict__.items() if not k.startswith('_')}
+
+  def dirty(self):
+    self.store.dirty(self)
+    
+  @classmethod
+  def ext(cls):
+    raise NotImplementedError('no ext')
     
   def define(self):
     pass
     
+  def created(self):
+    pass
+    
   def __hash__(self):
     return hash((str(type(self)), self.symbol))
   
   def __eq__(self, other):
-    return self.symbol == other.symbol and type(self) == type(other)
+    return type(self) == type(other) and self.symbol == other.symbol
+  
+  def get_system(self):
+    parts = self.symbol.split('-')
+    system_str = f'{parts[0]}-{parts[1]}'
+    system = self.store.get('System', system_str, create=True)
+    return system
     
   def seta(self, attr,  d, name=None, interp=None):
     if name is None:
@@ -31,49 +73,53 @@ class Base:
         val = interp(val)
       setattr(self, attr, val)
   
-  def setlst(self, attr, d, name, member):
+  def __lt__(self, o):
+    return self.symbol < o.symbol
+    
+  def setlst(self, attr, d, name, member=None, interp=None):
     val = sg(d, name)
     if val is not None:
-      lst = [sg(x, member) for x in val]
+      lst = []
+      for x in val:
+        if member is not None:
+          x = sg(x, member)
+        if interp is not None:
+          x = interp(x)
+        lst.append(x)
       setattr(self, attr, lst)
     
   def __setattr__(self, name, value):
-    if name not in ['symbol','store','disable_dirty'] and not self.disable_dirty:
-      self.store.dirty(self)
+    if not name.startswith('_') and not self._disable_dirty:
+      self.dirty()
+    if issubclass(type(value), Base):
+      value = Reference.create(value)
     super().__setattr__(name, value)
   
+  def __getattribute__(self, nm):
+    if nm == 'system':
+      return self.get_system()
+    if nm == 'waypoint':
+      return self.get_waypoint()
+    val = super().__getattribute__(nm)
+    if type(val) == Reference:
+      val = val.resolve()
+    return val
+  
   def update(self, d):
     pass
   
   def is_expired(self):
     return False
     
-  def load(self, d):
-    self.disable_dirty = True
-    self.__dict__.update(d)
-    self.disable_dirty = False
-    
-  def dict(self):
-    r = {}
-    for k,v in self.__dict__.items():
-      if k in ['store']:
-        continue
-      r[k] = deepcopy(v)
-    return r
-    
-  def path(self):
-    raise NotImplementedError('path')
-  
-  @classmethod
-  def ext(self):
-    raise NotImplementedError('extension')
-  
   def type(self):
     return self.__class__.__name__
   
   def __str__(self):
     return self.f()
     
+  def __repr__(self):
+    return self.f()
+    
   def f(self, detail=1):
     r = self.symbol
     if detail > 1:

nullptr/models/contract.py

@@ -18,9 +18,6 @@ class Contract(Base):
   def ext(cls):
     return 'cnt'
   
-  def path(self):
-    return f'contracts/{self.symbol}.{self.ext()}'
-  
   def is_expired(self):
     return time() > self.expires
   

nullptr/models/crew.py

@@ -0,0 +1,16 @@
+from .base import Base
+
+class Crew(Base):
+  @classmethod
+  def ext(self):
+    return 'crw'
+  
+  def define(self):
+    self.site = None
+    self.resources = []
+    
+  def f(self, detail=1):
+    r = super().f(detail)
+    if detail >2:
+      r += f'\nSite: {self.site}'
+    return r

nullptr/models/jumpgate.py

@@ -1,28 +1,22 @@
-from .system_member import SystemMember
+from .base import Base
+from .waypoint import Waypoint
 from dataclasses import field
 
-class Jumpgate(SystemMember):
+class Jumpgate(Base):
   def define(self):
-    self.range: int = 0
-    self.faction: str = ''
-    self.systems: list = []
+    self.connections: list = []
   
   def update(self, d):
-    self.setlst('systems', d, 'connectedSystems', 'symbol')
-    self.seta('faction', d, 'factionSymbol')
-    self.seta('range', d, 'jumpRange')
+    getter = self.store.getter(Waypoint, create=True)
+    self.setlst('connections', d, 'connections', interp=getter)
     
   @classmethod
   def ext(self):
     return 'jmp'
   
-  def path(self):
-    sector, system, _ = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'
-  
   def f(self, detail=1):
-    r = self.symbol
-    if detail > 1:
+    r = super().f(detail)
+    if detail > 2:
       r += '\n'
-      r += '\n'.join(self.systems)
+      r += '\n'.join([s.symbol for s in self.connections])
     return r

nullptr/models/marketplace.py

@@ -1,10 +1,42 @@
 
-from .system_member import SystemMember
+from .base import Base, Reference
 from time import time
 from nullptr.util import *
-from dataclasses import field
+from dataclasses import field, dataclass
+from nullptr.models import Waypoint
+from typing import List, Tuple
 
-class Marketplace(SystemMember):
+SUPPLY = ['SCARCE','LIMITED','MODERATE','HIGH','ABUNDANT']
+ACTIVITY =['RESTRICTED','WEAK','GROWING','STRONG']
+
+class MarketEntry:
+  def __init__(self):
+    self.buy = 0
+    self.sell = 0
+    self.volume = 0
+    self.supply = 0
+    self.activity = 0
+    self.history = []
+  
+  def upg(self):
+    if not hasattr(self, 'history'):
+      self.history = []
+      
+  def f(self, detail=1):
+    self.upg()
+    return f'b: {self.buy} s:{self.sell} hist: {len(self.history)}'
+    
+  def add(self, buy, sell, volume, supply, activity):
+    self.upg()
+    self.buy = buy
+    self.sell = sell
+    self.volume = volume
+    self.supply = supply
+    self.activity = activity
+    #self.history.append((int(time()), buy, sell, volume, supply, activity))
+  
+  
+class Marketplace(Base):
   def define(self):
     self.imports:list = []
     self.exports:list = []
@@ -12,22 +44,45 @@ class Marketplace(SystemMember):
     self.prices:dict = {}
     self.last_prices:int = 0
   
+  def get_waypoint(self):
+    return self.store.get('Waypoint', self.symbol, create=True)
+  
+  def is_fuel(self):
+    return self.imports + self.exports + self.exchange == ['FUEL']
+  
+  def record_prices(self, data):
+    for g in data:
+      symbol= mg(g, 'symbol')
+      if symbol in self.prices:
+        e = self.prices[symbol]
+      else:
+        e = self.prices[symbol] = MarketEntry()
+      buy = mg(g, 'purchasePrice')
+      sell = mg(g, 'sellPrice')
+      volume = mg(g, 'tradeVolume')
+      supply = SUPPLY.index(mg(g, 'supply'))
+      activity = ACTIVITY.index(sg(g, 'activity','STRONG'))
+      e.add(buy, sell, volume, supply, activity)
+    self.dirty()
+    
   def update(self, d):
     self.setlst('imports', d, 'imports', 'symbol')
     self.setlst('exports', d, 'exports', 'symbol')
     self.setlst('exchange', d, 'exchange', 'symbol')
     if 'tradeGoods' in d:
       self.last_prices = time()
-      prices = {}
-      for g in mg(d, 'tradeGoods'):
-        price = {}
-        symbol= mg(g, 'symbol')
-        price['symbol'] = symbol
-        price['buy'] = mg(g, 'purchasePrice')
-        price['sell'] = mg(g, 'sellPrice')
-        prices[symbol] = price
-      self.prices = prices
+      self.record_prices(mg(d, 'tradeGoods'))
+ 
+  def buy_price(self, resource):
+    if resource not in self.prices:
+      return None
+    return self.prices[resource].buy
   
+  def volume(self, resource):
+    if resource not in self.prices:
+      return None
+    return self.prices[resource].volume
+    
   def sellable_items(self, resources):
     return [r for r in resources if r in self.prices]
     
@@ -43,16 +98,20 @@ class Marketplace(SystemMember):
     if r in self.exchange:
       return 'X'
     return '?'
-    
-  def path(self):
-    sector, system, _ = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'
   
   def f(self, detail=1):
-    r = self.symbol
-    if detail > 1:
+    r = super().f(detail)
+    if detail > 2:
       r += '\n'
-      for p in self.prices.values():
-        t = self.rtype(p['symbol'])
-        r += f'{t} {p["symbol"]:25s} {p["sell"]:5d} {p["buy"]:5d}\n'
+      if len(self.imports) > 0:
+        r += 'I: ' + ', '.join(self.imports) + '\n'
+      if len(self.exports) > 0:
+        r += 'E: ' + ', '.join(self.exports) + '\n'
+      if len(self.exchange) > 0:
+        r += 'X: ' + ', '.join(self.exchange) + '\n'
+      
+      r += '\n'
+      for res, p in self.prices.items():
+        t = self.rtype(res)
+        r += f'{t} {res:25s} {p.buy:5d} {p.sell:5d}\n'
     return r

nullptr/models/ship.py

@@ -1,7 +1,8 @@
 from .base import Base
-from time import time
+from time import time, strftime
 from nullptr.util import *
-from dataclasses import dataclass, field
+from nullptr.models import Waypoint
+import os
 
 class Ship(Base):  
   def define(self):
@@ -10,28 +11,50 @@ class Ship(Base):
     self.status:str = ''
     self.cargo_capacity:int = 0
     self.cargo_units:int = 0
-    self.location_str = ''
+    self.location = None
     self.cooldown:int = 0
     self.arrival:int = 0
     self.fuel_current:int = 0
     self.fuel_capacity:int = 0
     self.mission:str = None
     self.mission_status:str = 'init'
+    self.role = None
+    self.crew = None
+    self.frame = ''
+    self.speed = "CRUISE"
+    self._log_file = None
+    self._log_level = 5
   
+  def log(self, m, l=3):
+    if m is None: return
+    if type(m) != str:
+      m = pretty(m)
+    if self._log_file is None:
+      fn = os.path.join(self.store.data_dir, f'{self.symbol}.{self.ext()}.log')
+      self._log_file = open(fn, 'a')
+    ts = strftime('%Y%m%d %H%M%S')
+    sts = strftime('%H%M%S')
+    m = m.strip()
+    self._log_file.write(f'{ts} {m}\n')
+    self._log_file.flush()
+    if l <= self._log_level:
+      print(f'{self} {sts} {m}')
+    
   @classmethod
   def ext(self):
     return 'shp'
   
-  def location(self):
-    return self.store.get('Waypoint', self.location_str)
+  def range(self):
+    if self.fuel_capacity == 0:
+      return 100000
+    return self.fuel_capacity
     
-  def path(self):
-    agent = self.symbol.split('-')[0]
-    return f'{agent}/{self.symbol}.{self.ext()}'
-
   def update(self, d):
     self.seta('status', d, 'nav.status')
-    self.seta('location_str', d, 'nav.waypointSymbol')
+    self.seta('speed', d, "nav.flightMode")
+    self.seta('frame', d, 'frame.name')
+    getter = self.store.getter(Waypoint, create=True)
+    self.seta('location', d, 'nav.waypointSymbol', interp=getter)
     self.seta('cargo_capacity', d, 'cargo.capacity')
     self.seta('cargo_units', d, 'cargo.units')
     self.seta('fuel_capacity', d, 'fuel.capacity')
@@ -60,17 +83,40 @@ class Ship(Base):
     if typ not in self.cargo:
       return 0
     return self.cargo[typ]
+  
+  def take_cargo(self, typ, amt):
+    if typ not in self.cargo:
+      return
+    if self.cargo[typ] <= amt:
+      del self.cargo[typ]
+    else:
+      self.cargo[typ] -= amt
+    
+    self.cargo_units = sum(self.cargo.values())
+
+  def put_cargo(self, typ, amt):
+    if typ not in self.cargo:
+      self.cargo[typ] = amt
+    else:
+      self.cargo[typ] += amt
+    
+    self.cargo_units = sum(self.cargo.values())
     
   def load_cargo(self, cargo):
     result = {}
+    total = 0
     for i in cargo:
       symbol = must_get(i, 'symbol')
       units = must_get(i, 'units')
       result[symbol] = units
+      total += units
+    self.cargo_units = total
     self.cargo = result
     
   def deliverable_cargo(self, contract):
     result = []
+    if contract is None: 
+      return result
     for d in contract.deliveries:
       if self.get_cargo(d['trade_symbol']) > 0:
         result.append(d['trade_symbol'])
@@ -82,6 +128,9 @@ class Ship(Base):
     garbage = [c for c in cargo if c not in deliveries]
     return garbage
   
+  def cargo_space(self):
+    return self.cargo_capacity - self.cargo_units
+  
   def update_timers(self):
     if self.status == 'IN_TRANSIT' and self.arrival < time():
       self.status = 'IN_ORBIT'
@@ -91,14 +140,52 @@ class Ship(Base):
     self.update_timers()
     arrival = int(self.arrival - time())
     cooldown = int(self.cooldown - time())
-    r = self.symbol
-    if detail > 1:
-      r += ' ' + self.status
-      r += f' [{self.fuel_current}/{self.fuel_capacity}]'
-      r += ' ' + str(self.location())
+    
+    role = self.role
+    if role is None:
+      role = 'none'
+    crew = 'none'
+    if self.crew is not None:
+      crew = self.crew.symbol
+    mstatus = self.mission_status
+    if mstatus == 'error': 
+      mstatus = mstatus.upper()
+    if mstatus is None:
+      mstatus = 'none'
+    status = self.status.lower()
+    if status.startswith('in_'):
+      status = status[3:]
+      
+    if detail < 2:
+      r = self.symbol
+    elif detail == 2:
+      symbol = self.symbol.split('-')[1]
+      
+      r = f'{symbol:<2} {role:7} {mstatus:8} {str(self.location):11}'
       if self.is_travelling():
         r += f' [A: {arrival}]'
       if self.is_cooldown():
         r += f' [C: {cooldown}]'
+    else:
+      r = f'== {self.symbol} {self.frame} ==\n'
+      r += f'Role: {crew} / {role}\n'
+      r += f'Mission: {self.mission}   ({mstatus})\n'
+      for k, v in self.mission_state.items():
+        if type(v) == list:
+          v = f'[{len(v)} items]'
+        r += f' {k}: {v}\n'
+      adj = 'to' if self.status == 'IN_TRANSIT' else 'at'
+      r += f'Status {self.status} {adj} {self.location}\n'
+      
+      r += f'Fuel: {self.fuel_current}/{self.fuel_capacity}\n'
+      r += f'Speed: {self.speed}\n'
+      r += f'Cargo: {self.cargo_units}/{self.cargo_capacity}\n'
+      for res, u in self.cargo.items():
+        r += f' {res}: {u}\n'
+      if self.is_travelling():
+        r += f'Arrival: {arrival} seconds\n'
+      if self.is_cooldown():
+        r += f'Cooldown: {cooldown} seconds \n'
+        
     return r 
   

nullptr/models/shipyard.py

@@ -0,0 +1,35 @@
+from nullptr.models import Base
+from time import time
+from nullptr.util import *
+
+class Shipyard(Base):
+  def define(self):
+    self.last_prices = 0
+    self.types = set()
+    self.prices:dict = {}
+  
+  def get_waypoint(self):
+    return self.store.get('Waypoint', self.symbol, create=True)
+  
+  @classmethod
+  def ext(self):
+    return 'syd'
+  
+  def update(self, d):
+    if 'ships' in d:
+      self.last_prices = time()
+      for s in must_get(d, 'ships'):
+        self.prices[s['type']] = s['purchasePrice']
+    for s in must_get(d, 'shipTypes'):
+        self.types.add(s['type'])
+  
+  def f(self, detail=1):
+    r = super().f(detail)
+    if detail > 2:
+      r += '\n'
+      for st in self.types:
+        price = "Unknown"
+        if st in self.prices:
+          price = self.prices[st]
+        r += f'{st:20} {price}\n'
+    return r

nullptr/models/survey.py

@@ -1,10 +1,10 @@
 from time import time
 from nullptr.util import *
-from .system_member import SystemMember
+from .base import Base
 
 size_names = ['SMALL','MODERATE','LARGE']
 
-class Survey(SystemMember):
+class Survey(Base):
   identifier = 'signature'
   def define(self):
     self.type: str = ''
@@ -17,12 +17,12 @@ class Survey(SystemMember):
   @classmethod
   def ext(cls):
     return 'svy'
-  
-  def path(self):
-    sector, system, waypoint, signature = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'
+
+  def get_waypoint(self):
+    sym = '-'.join(self.symbol.split('-')[:3])
+    return self.store.get('Waypoint', sym, create=True)
+    
     
-  
   def is_expired(self):
     return time() > self.expires or self.exhausted
   
@@ -33,7 +33,7 @@ class Survey(SystemMember):
   def api_dict(self):
     return {
       'signature': self.symbol,
-      'symbol': self.waypoint(),
+      'symbol': self.waypoint.symbol,
       'deposits': [{'symbol': d} for d in self.deposits],
       'expiration': self.expires_str,
       'size': size_names[self.size]

nullptr/models/system.py

@@ -7,6 +7,8 @@ class System(Base):
     self.x:int = 0
     self.y:int = 0
     self.type:str = 'unknown'
+    self.uncharted = True
+    self.last_crawl = 0
   
   def update(self, d):
     self.seta('x', d)
@@ -17,10 +19,6 @@ class System(Base):
   def ext(self):
     return 'stm'
   
-  def path(self):
-    sector, symbol = self.symbol.split('-')
-    return f'atlas/{sector}/{symbol[0:1]}/{self.symbol}.{self.ext()}'
-  
   def distance(self, other):
     return int(sqrt((self.x - other.x) ** 2 + (self.y - other.y) ** 2))
   

nullptr/models/system_member.py

@@ -1,10 +0,0 @@
-from .base import Base
-
-class SystemMember(Base):
-  @classmethod
-  def ext(cls):
-    return 'obj'
-    
-  def system(self):
-    p = self.symbol.split('-')
-    return f'{p[0]}-{p[1]}'

nullptr/models/waypoint.py

@@ -1,26 +1,65 @@
-from .system_member import SystemMember
+from .base import Base, Reference
+from nullptr.models.system import System
 from nullptr.util import *
-from dataclasses import field
+from time import time
+from math import sqrt
 
-class Waypoint(SystemMember):
+class Waypoint(Base):
   def define(self):
     self.x:int = 0
     self.y:int = 0
     self.type:str = 'unknown'
     self.traits:list = []
     self.faction:str = ''
-  
+    self.is_under_construction:bool = False
+    self.uncharted = True
+    self.extracted:int = 0
+
+    
   def update(self, d):
     self.seta('x', d)
     self.seta('y', d)
     self.seta('type', d)
     self.seta('faction', d, 'faction.symbol')
+    self.seta('is_under_construction', d, 'isUnderConstruction')
     self.setlst('traits', d, 'traits', 'symbol')
+    self.uncharted = 'UNCHARTED' in self.traits
+    
+  def created(self):
+    self.get_system()
+  
+  def distance(self, other):
+    return int(sqrt((self.x - other.x) ** 2 + (self.y - other.y) ** 2))
     
   @classmethod
   def ext(self):
     return 'way'
     
-  def path(self):
-    sector, system, _ = self.symbol.split('-')
-    return f'atlas/{sector}/{system[0:1]}/{system}/{self.symbol}.{self.ext()}'
+  def itraits(self):
+    traits = []
+    if self.type == 'JUMP_GATE':
+      traits.append('JUMP')
+    if self.type == 'GAS_GIANT':
+      traits.append('GAS')
+    if 'SHIPYARD' in self.traits:
+      traits.append('SHIPYARD')
+    if 'MARKETPLACE' in self.traits:
+      traits.append('MARKET')
+    
+    
+    
+    if 'COMMON_METAL_DEPOSITS' in self.traits:
+      traits.append('METAL')
+    if 'PRECIOUS_METAL_DEPOSITS' in self.traits:
+      traits.append('GOLD')
+    if 'MINERAL_DEPOSITS' in self.traits:
+      traits.append('MINS')
+    if 'STRIPPED' in self.traits:
+      traits.append('STRIPPED')
+    return traits
+  
+  def f(self, detail=1):
+    r = self.symbol
+    if detail > 3:
+      r += f'\n{self.x} {self.y}'
+    return r

nullptr/roles/__init__.py

@@ -0,0 +1,23 @@
+from nullptr.roles.trader import assign_trader
+from nullptr.roles.probe import assign_probe
+from nullptr.roles.siphon import assign_siphon
+from nullptr.roles.hauler import assign_hauler
+from nullptr.roles.surveyor import assign_surveyor
+from nullptr.roles.miner import assign_miner
+from nullptr.roles.sitter import assign_sitter
+
+def assign_mission(c, s):
+  if s.role == 'trader':
+    assign_trader(c, s)
+  elif s.role == 'probe':
+    assign_probe(c, s)
+  elif s.role == 'siphon':
+    assign_siphon(c, s)
+  elif s.role == 'hauler':
+    assign_hauler(c, s)
+  elif s.role == 'surveyor':
+    assign_surveyor(c, s)
+  elif s.role == 'miner':
+    assign_miner(c, s)
+  elif s.role == 'sitter':
+    assign_sitter(c, s)

nullptr/roles/hauler.py

@@ -0,0 +1,16 @@
+from nullptr.util import AppError
+from nullptr.analyzer import best_sell_market
+from random import choice
+
+def assign_hauler(c, s):
+  if s.crew is None:
+    raise AppError('ship has no crew')
+  w = s.crew.site
+  resources = s.crew.resources
+  resource = choice(resources)
+  m = best_sell_market(c,s.location.system, resource)
+  s.log(f'assigning haul mission from {w} to {m}')
+  c.captain.init_mission(s, 'haul')
+  c.captain.smipa(s, 'site', w)
+  c.captain.smipa(s, 'dest', m)
+  c.captain.smipa(s, 'resources', resources)

nullptr/roles/miner.py

@@ -0,0 +1,11 @@
+from nullptr.util import AppError
+
+def assign_miner(c, s):
+  if s.crew is None:
+    raise AppError('ship has no crew')
+  w = s.crew.site
+  resources = s.crew.resources
+  c.captain.init_mission(s, 'mine')
+  c.captain.smipa(s, 'site', w)
+  c.captain.smipa(s, 'resources', resources)
+  

nullptr/roles/probe.py

@@ -0,0 +1,15 @@
+from nullptr.analyzer import solve_tsp
+from random import randrange
+
+def assign_probe(c, s):   
+  system = s.location.system
+  m = [m.waypoint for m in c.store.all_members(system, 'Marketplace')]
+  m = solve_tsp(c, m)
+  hops = [w.symbol for w in m]
+  start_hop = 0
+  s.log(f'Assigning {s} to probe {len(hops)} starting at {hops[start_hop]}')
+  
+  c.captain.init_mission(s, 'probe')
+  c.captain.smipa(s, 'hops', hops)
+  c.captain.smipa(s, 'next-hop', start_hop)
+  

nullptr/roles/siphon.py

@@ -0,0 +1,8 @@
+from nullptr.util import AppError
+
+def assign_siphon(c, s):
+  if s.crew is None:
+    raise AppError('ship has no crew')
+  w = s.crew.site
+  c.captain.init_mission(s, 'siphon')
+  c.captain.smipa(s, 'site', w)

nullptr/roles/sitter.py

@@ -0,0 +1,24 @@
+from nullptr.analyzer import Point
+
+def assign_sitter_at(c, s, w):
+  c.captain.init_mission(s, 'sit')
+  c.captain.smipa(s, 'dest', w.symbol)
+
+def assign_sitter(c, s):
+  system = s.location.system
+  ships = c.store.all('Ship')
+  markets = c.store.all_members(system, 'Marketplace')
+  origin = Point(0, 0)
+  markets = sorted(markets, key=lambda m: m.waypoint.distance(origin))
+  shipyards = c.store.all_members(system, 'Shipyard')
+  occupied = [s.mission_state['dest'] for s in ships if s.mission=='sit']
+  probe_shipyard = [y for y in shipyards if 'SHIP_PROBE' in y.types][0]
+  
+  if probe_shipyard.symbol not in occupied:
+    return assign_sitter_at(c, s, probe_shipyard)
+  for y in shipyards:
+    if y.symbol not in occupied:
+      return assign_sitter_at(c, s, y)
+  for m in markets:
+    if m.symbol not in occupied:
+      return assign_sitter_at(c, s, m)

nullptr/roles/surveyor.py

@@ -0,0 +1,10 @@
+from nullptr.util import AppError
+
+
+def assign_surveyor(c, s):
+    if s.crew is None:
+      raise AppError('ship has no crew')
+    w = s.crew.site
+    c.init_mission(s, 'survey')
+    c.smipa(s, 'site', w)
+    

nullptr/roles/trader.py

@@ -0,0 +1,14 @@
+from nullptr.analyzer import find_trade
+
+
+def assign_trader(c, s):
+  t = find_trade(c, s.location.system)
+  if t is None:
+    print(f"No trade for {s} found. Idling")
+    c.captain.init_mission(s,'idle')
+    c.captain.smipa(s, 'seconds', 600)
+    return
+  s.log(f'assigning {s} to deliver {t.resource} from {t.source} to {t.dest} at a margin of {t.margin}')
+  c.captain.init_mission(s, 'trade')
+  c.captain.smipa(s, 'site', t.source)
+  c.captain.smipa(s, 'dest', t.dest)

nullptr/store.py

@@ -1,30 +1,103 @@
-from nullptr.models.base import Base
-from nullptr.models.waypoint import Waypoint
-from nullptr.models.sector import Sector
-from nullptr.models.system import System
-from nullptr.models.agent import Agent
-from nullptr.models.marketplace import Marketplace
-from nullptr.models.system_member import SystemMember
-from nullptr.models.jumpgate import Jumpgate
-from nullptr.models.ship import Ship
-from nullptr.models.contract import Contract
-from nullptr.models.survey import Survey
+from nullptr.models import *
 from os.path import isfile, dirname, isdir
 import os
 from os.path import basename
 import json
 from .util import *
 from time import time
+import pickle
+from struct import unpack, pack
+from functools import partial
+from io import BytesIO
+from copy import copy
 
+class StorePickler(pickle.Pickler):
+  def persistent_id(self, obj):
+    return "STORE" if type(obj) == Store else None
+
+class StoreUnpickler(pickle.Unpickler):
+  def __init__(self, stream, store):
+    self.store = store
+    super().__init__(stream)
+    
+  def persistent_load(self, pers_id):
+    if pers_id == "STORE":
+      return self.store
+    raise pickle.UnpicklingError("I don know the persid!")
+
+CHUNK_MAGIC = b'ChNkcHnK'    
+        
+class ChunkHeader:
+  def __init__(self):
+    self.magic = CHUNK_MAGIC
+    self.offset = 0
+    self.in_use = True
+    self.size = 0
+    self.used = 0
+    
+  @classmethod
+  def parse(cls, fil):
+    offset = fil.tell()
+    d = fil.read(24)
+    if len(d) < 24:
+      return None
+    o = cls()
+    o.offset = offset
+    o.magic, d, o.used = unpack('<8sQQ', d)
+    o.size =   d & 0x7fffffffffffffff
+    o.in_use = d & 0x8000000000000000 != 0
+    if o.magic != CHUNK_MAGIC:
+      raise ValueError(f"Invalid chunk magic: {o.magic}")
+    # print(o)
+    return o
+  
+  def write(self, f):
+    d = self.size
+    if self.in_use:
+      d |= 1 << 63
+    d = pack('<8sQQ', self.magic, d, self.used)
+    f.write(d)
+  
+  def __repr__(self):
+    return f'chunk {self.in_use} {self.size} {self.used}'
+  
+  def f(self, detail=1):
+    if detail == 1:
+      return f'chunk {self.offset} {self.used}/{self.size}'
+    else:
+      r = f'Stored at: {self.offset}\n'
+      slack = self.size - self.used
+      r += f'Used: {self.used}/{self.size} (slack {slack})'
+      return r
+      
 class Store:
-  def __init__(self, data_dir):
+  def __init__(self, data_file, verbose=False):
     self.init_models()
-    self.data_dir = data_dir
+    self.data_file = data_file
+    self.data_dir = os.path.dirname(data_file)
+    self.fil = open_file(data_file)
     self.data = {m: {} for m in self.models}
     self.system_members = {}
     self.dirty_objects = set()
     self.cleanup_interval = 600
     self.last_cleanup = 0
+    self.slack = 0.1
+    self.slack_min = 64
+    self.slack_max = 1024
+    self.verbose = verbose
+    self.load()
+  
+  def p(self,  m):
+    if not self.verbose:
+      return
+    print(m)
+  
+  def f(self, detail):
+    return f'Store {self.data_file}'
+  
+  def close(self):
+    self.flush()
+    self.fil.close()
   
   def init_models(self):
     self.models = all_subclasses(Base)
@@ -33,54 +106,138 @@ class Store:
     
   def dirty(self, obj):
     self.dirty_objects.add(obj)
-    
-  def path(self, obj):
-    return os.path.join(self.data_dir, obj.path())
-    
-  def load_file(self, path):
-    if not isfile(path):
-      return None
-    fn = basename(path)
-    ext = fn.split('.')[-1]
-    symbol = fn.split('.')[0]
-    if ext not in self.extensions:
-      return None
-    with open(path) as f:
-      data = json.load(f)
-    
-    typ = self.extensions[ext]
-    obj = self.create(typ, symbol)
-    obj.load(data)
-    obj.store = self
-    return obj
   
+  def dump_object(self, obj):
+    buf = BytesIO()
+    p = StorePickler(buf)
+    p.dump(obj)
+    return buf.getvalue()
+  
+  def load_object(self, data, offset):
+    buf = BytesIO(data)
+    p = StoreUnpickler(buf, self)
+    obj = p.load()
+    x = self.get(type(obj), obj.symbol)
+    if x is not None and x in self.dirty_objects:
+      self.dirty_objects.remove(obj)
+    obj._file_offset = offset
+    self.hold(obj)
+    
   def load(self):
     cnt = 0
     start_time = time()
-    for fil in list_files(self.data_dir, True):
-      self.load_file(fil)
-      cnt += 1
-    dur = time() - start_time
-    print(f'loaded {cnt} objects in {dur:.2f} seconds')
-  
-  def store(self, obj):
-    path = self.path(obj)
-    path_dir = dirname(path)
-    data = obj.dict()
-    if not isdir(path_dir):
-      os.makedirs(path_dir, exist_ok=True)
-    with open(path, 'w') as f:
-      json.dump(data, f, indent=2)
-  
-  def create(self, typ, symbol):
-    obj = typ(symbol, self)
-    self.data[typ][symbol] = obj
-    if issubclass(typ, SystemMember):
-      system_str = obj.system()
+    total = 0
+    free = 0
+    self.fil.seek(0)
+    offset = 0
+    while (hdr := ChunkHeader.parse(self.fil)):
+      # self.p(hdr)
+      total += hdr.size
+      if not hdr.in_use: 
+        # print(f"skip {hdr.size} {self.fil.tell()}")
+        self.fil.seek(hdr.size, 1)
+        free += hdr.size
+      else:
+        data = self.fil.read(hdr.used)
+        self.load_object(data, offset)
+        # print(f"pad {hdr.size - hdr.used}")
+        self.fil.seek(hdr.size - hdr.used, 1)
+        cnt += 1
+      offset = self.fil.tell()
       
+    dur = time() - start_time
+    # just in case any temp objects were created
+    self.dirty_objects = set()
+    self.p(f'Loaded {cnt} objects in {dur:.2f} seconds')
+    self.p(f'Fragmented space: {free} / {total} bytes')
+  
+  def allocate_chunk(self, sz):
+    used = sz
+    slack = sz * self.slack
+    slack = min(slack, self.slack_max)
+    slack = max(slack, self.slack_min)
+    sz += int(slack)
+    self.fil.seek(0, 2)
+    offset = self.fil.tell()
+    h = ChunkHeader()
+    h.size = sz
+    h.used = used
+    h.offset = offset
+    h.write(self.fil)
+    return offset, h
+  
+  def get_header(self, obj):
+    if obj._file_offset is None:
+      return None
+    self.fil.seek(obj._file_offset)
+    hdr = ChunkHeader.parse(self.fil)
+    return hdr
+    
+  def purge(self, obj):
+    if obj._file_offset is not None:
+      self.fil.seek(obj._file_offset)
+      hdr = ChunkHeader.parse(self.fil)
+      hdr.in_use = False
+      self.fil.seek(obj._file_offset)
+      hdr.write(self.fil)
+    if type(obj) in self.data and obj.symbol in self.data[type(obj)]:
+      del self.data[type(obj)][obj.symbol]
+    self.remove_from_members(obj)
+    if obj in self.dirty_objects:
+      self.dirty_objects.remove(obj)
+    obj._file_offset = None
+    
+  def store(self, obj):
+    data = self.dump_object(obj)
+    osize = len(data)
+    # is there an existing chunk for this obj?
+    if obj._file_offset is not None:
+      # read chunk hdr
+      self.fil.seek(obj._file_offset)
+      hdr = ChunkHeader.parse(self.fil)
+      csize = hdr.size
+      # if the chunk is too small
+      if csize < osize:
+        # free the chunk
+        hdr.in_use = False
+        # force a new chunk
+        obj._file_offset = None
+      else:
+        # if it is big enough, update the used field
+        hdr.used = osize
+      self.fil.seek(hdr.offset)
+      hdr.write(self.fil)
+        
+    if obj._file_offset is None:
+      obj._file_offset, hdr = self.allocate_chunk(osize)
+    # print(type(obj).__name__, hdr)
+    self.fil.write(data)
+    slack = b'\x00' * (hdr.size - hdr.used)
+    self.fil.write(slack)
+  
+  def remove_from_members(self, obj):
+    if type(obj).__name__ in ['Waypoint','Marketplace', 'Jumpgate', 'Survey']:
+      system_str = obj.system.symbol
+      if system_str not in self.system_members:
+        return
+        self.system_members[system_str].remove(obj)
+      
+  def hold(self, obj):
+    typ = type(obj)
+    symbol = obj.symbol
+    obj.store = self
+    self.data[typ][symbol] = obj
+    if type(obj).__name__ in ['Waypoint','Marketplace', 'Jumpgate', 'Survey', 'Shipyard']:
+      system_str = obj.system.symbol
       if system_str not in self.system_members:
         self.system_members[system_str] = set()
       self.system_members[system_str].add(obj)
+  
+  def create(self, typ, symbol):
+    obj = typ(symbol, self)
+    obj.created()
+    self.hold(obj)
+    self.dirty(obj)
     return obj
     
   def get(self, typ, symbol, create=False):
@@ -96,6 +253,11 @@ class Store:
         return None
     return self.data[typ][symbol]
   
+  def getter(self, typ, create=False):
+    if type(typ) == str and typ in self.model_names:
+      typ = self.model_names[typ]
+    return partial(self.get, typ, create=create)
+  
   def update(self, typ, data, symbol=None):
     if type(typ) == str and typ in self.model_names:
       typ = self.model_names[typ] 
@@ -113,6 +275,9 @@ class Store:
       typ = self.model_names[typ]
     
     for m in self.data[typ].values():
+      if m.is_expired():
+        self.dirty(m)
+        continue
       yield m
   
   def all_members(self, system, typ=None):
@@ -121,38 +286,61 @@ class Store:
     
     if type(system) == System:
       system = system.symbol
-  
+    
     if system not in self.system_members:
       return 
-      print('typ', typ)
+    
+    garbage = set()
     for m in self.system_members[system]:
+      if m.is_expired():
+        self.dirty(m)
+        garbage.add(m)
+        continue
       if typ is None or type(m) == typ:
         yield m
     
+    for m in garbage:
+      self.system_members[system].remove(m)
+      
   def cleanup(self):
-    if time() < self.last_cleanup + self.cleanup_interval:
-      return
+    self.last_cleanup = time()
     start_time = time()
     expired = list()
     for t in self.data:
-      for o in self.all(t):
+      for o in self.data[t].values():
         if o.is_expired():
           expired.append(o)
     for o in expired:
-      path = o.path()
-      if isfile(path):
-        os.remove(path)
-      del self.data[type(o)][o.symbol]
+      self.purge(o)
+
     dur = time() - start_time
-    # print(f'cleaned {len(expired)} in {dur:.03f} seconds')
+    # self.p(f'cleaned {len(expired)} in {dur:.03f} seconds')
       
   def flush(self):
     self.cleanup()
     it = 0
     start_time = time()
-    for obj in self.dirty_objects:
+    for obj in copy(self.dirty_objects):
       it += 1
+      if obj.symbol not in self.data[type(obj)] or self.data[type(obj)][obj.symbol] != obj:
+        # print(f"Dirty object not in data {type(obj)} {obj.symbol} {obj}")
+        continue
       self.store(obj)
+    self.fil.flush()
     self.dirty_objects = set()
     dur = time() - start_time
-    #  print(f'flush done {it} items {dur:.2f}')
+    #self.p(f'flush done {it} items {dur:.2f}')
+  
+  def defrag(self):
+    self.flush()
+    nm = self.fil.name
+    self.fil.close()
+    bakfile = nm+'.bak'
+    if os.path.isfile(bakfile):
+      os.remove(bakfile)
+    os.rename(nm, nm + '.bak')
+    self.fil = open_file(nm)
+    for t in self.data:
+      for o in self.data[t].values():
+        o._file_offset = None
+        self.store(o)

nullptr/store_analyzer.py

@@ -0,0 +1,58 @@
+from nullptr.store import CHUNK_MAGIC, ChunkHeader, StoreUnpickler
+from hexdump import hexdump
+from io import BytesIO
+class FakeStore:
+  def get(self, typ, sym, create=False):
+    return None
+
+class StoreAnalyzer:
+  def __init__(self, verbose=False):
+    self.verbose = verbose
+
+  def load_obj(self, f, sz):
+    buf = BytesIO(f.read(sz))
+    p = StoreUnpickler(buf, FakeStore())
+    obj = p.load()
+    return obj
+    print(obj.symbol, type(obj).__name__)
+
+  def run(self, f):
+    lastpos = 0
+    pos = 0
+    objs = {}
+    result = True
+    f.seek(0)
+    while True:
+      lastpos = pos
+      pos = f.tell()
+      m = f.read(8)
+      if len(m) < 8:
+        break
+      if m != CHUNK_MAGIC:
+        print(f'missing magic at {pos}')
+        result = False
+        self.investigate(f, lastpos)
+        break
+      f.seek(-8, 1)
+      h = ChunkHeader.parse(f)
+      if self.verbose:
+        print(h, pos)
+      if h.in_use:
+        obj = self.load_obj(f, h.used)
+        kobj = obj.symbol, type(obj).__name__
+        if kobj in objs:
+          print(f'Double object {kobj} prev {objs[kobj]} latest {h}')
+          result = False
+        objs[kobj] = h
+      else:
+        f.seek(h.used, 1)
+      f.seek(h.size - h.used, 1)
+    return result
+  
+  def investigate(self, f, lastpos):
+    print(f'dumping 1024 bytes from {lastpos}')
+    f.seek(lastpos, 0)
+    d = f.read(1024)
+    
+    hexdump(d)
+    print(d.index(CHUNK_MAGIC))

nullptr/test_store.py

@@ -0,0 +1,170 @@
+import unittest
+import tempfile
+from nullptr.store import Store, ChunkHeader
+from nullptr.models import Base
+from io import BytesIO
+import os
+from nullptr.store_analyzer import StoreAnalyzer
+
+class Dummy(Base):
+  def define(self):
+    self.count: int = 0
+    self.data: str = ""
+  
+  def update(self, d):
+    self.seta('count', d)
+    
+  @classmethod
+  def ext(self):
+    return 'dum'
+  
+  def f(self, detail=1):
+    r = super().f(detail) + '.' + self.ext()
+    if detail >2:
+      r += f' c:{self.count}'
+    return r
+    
+class TestStore(unittest.TestCase):
+  def setUp(self):
+    self.store_file = tempfile.NamedTemporaryFile()
+    self.s = Store(self.store_file.name, False)
+  
+  def tearDown(self):
+    self.s.close()
+    self.store_file.close()
+    
+  def reopen(self):
+    self.s.flush()
+    self.s.close()
+    self.s = Store(self.store_file.name, False)
+    
+  def test_single(self):
+    dum = self.s.get(Dummy, "5", create=True)
+    dum.count = 1337
+    dum.data = "A" * 1000
+    self.reopen()
+    
+    dum = self.s.get(Dummy, "5")
+    self.assertEqual(1337, dum.count)
+  
+  def test_grow(self):
+    dum = self.s.get(Dummy, "5", create=True)
+    dum.data = "A"
+    dum2 = self.s.get(Dummy, "7",create=True)
+    self.reopen()
+    dum = self.s.get(Dummy, "5")
+    old_off = dum._file_offset
+    self.assertTrue(old_off is not None)
+    dum.data = "A" * 1000
+    dum.count = 1337
+    self.s.flush()
+    new_off = dum._file_offset
+    self.assertTrue(new_off is not None)
+    self.assertNotEqual(old_off, new_off)
+    self.reopen()
+    dum = self.s.get(Dummy, "5")
+    newer_off = dum._file_offset
+    self.assertTrue(newer_off is not None)
+    self.assertEqual(new_off, newer_off)
+    self.assertEqual(1337, dum.count)
+  
+  def test_purge(self):
+    dum = self.s.get(Dummy, "5", create=True)
+    dum.data = "A"
+    dum2 = self.s.get(Dummy, "7",create=True)
+    dum2.count = 1337
+    self.s.flush()
+    self.s.purge(dum)
+    self.reopen()
+    dum = self.s.get(Dummy, "5")
+    self.assertIsNone(dum)
+    dum2 = self.s.get(Dummy, "7")
+    self.assertEqual(1337, dum2.count)
+  
+  def test_grow_last(self):
+    dum = self.s.get(Dummy, "5", create=True)
+    dum.data = "A"
+    dum2 = self.s.get(Dummy, "7",create=True)
+    self.reopen()
+    dum2 = self.s.get(Dummy, "7")
+    dum2.data = "A" * 1000
+    dum2.count = 1337
+    dum3 = self.s.get(Dummy, "9",create=True)
+    dum3.count = 1338
+    self.reopen()
+    dum2 = self.s.get(Dummy, "7")
+    self.assertEqual(1337, dum2.count)
+    dum3 = self.s.get(Dummy, "9")
+    self.assertEqual(1338, dum3.count)
+    
+  def test_purge_last(self):
+    dum = self.s.get(Dummy, "5", create=True)
+    dum.data = "A"
+    dum2 = self.s.get(Dummy, "7",create=True)
+    self.reopen()
+    dum2 = self.s.get(Dummy, "7")
+    self.s.purge(dum2)
+    dum3 = self.s.get(Dummy, "9",create=True)
+    dum3.count = 1338
+    self.reopen()
+    dum2 = self.s.get(Dummy, "7")
+    self.assertIsNone(dum2)
+    dum3 = self.s.get(Dummy, "9")
+    self.assertEqual(1338, dum3.count)
+  
+  def test_dont_relocate(self):
+    dum = self.s.get(Dummy, "5", create=True)
+    dum.data = "A"
+    self.s.flush()
+    old_off = dum._file_offset
+    self.reopen()
+    dum2 = self.s.get(Dummy, "5")
+    dum2.data = "BCDE"
+    self.s.flush()
+    new_off = dum._file_offset
+    self.assertEqual(old_off, new_off)
+
+  def test_chunk_header(self):
+    a = ChunkHeader()
+    a.size = 123
+    a.used = 122
+    a.in_use = True
+    b = BytesIO()
+    a.write(b)
+    b.seek(0)
+    c = ChunkHeader.parse(b)
+    self.assertEqual(c.size, a.size)
+    self.assertEqual(c.used, a.used)
+    self.assertEqual(c.in_use, True)
+    c.in_use = False
+    b.seek(0)
+    c.write(b)
+    b.seek(0)
+    d = ChunkHeader.parse(b)
+    self.assertEqual(d.size, a.size)
+    self.assertEqual(d.used, a.used)
+    self.assertEqual(d.in_use, False)
+  
+  def test_mass(self):
+    num = 50
+    for i in range(num):
+      dum = self.s.get(Dummy, str(i), create=True)
+      dum.data = str(i)
+      dum.count = 0
+    self.reopen()
+    sz = os.stat(self.store_file.name).st_size
+    for j in range(50):
+      for i in range(num):
+        dum = self.s.get(Dummy, str(i))
+        # this works because j is max 49, and the slack is 64
+        # so no growing is needed
+        self.assertEqual(dum.data, "B" * j + str(i)) 
+        self.assertEqual(dum.count, j)
+        dum.data = "B" * (j+1) + str(i)
+        dum.count += 1
+      self.reopen()
+    sz2 = os.stat(self.store_file.name).st_size
+    self.assertEqual(sz, sz2)
+    an = StoreAnalyzer().run(self.store_file)
+    self.assertTrue(an)
+    

nullptr/util.py

@@ -1,21 +1,20 @@
 from datetime import datetime
 from math import ceil
 import os
-from os.path import isfile
+from os.path import isfile, dirname
+import traceback
 
-def list_files(path, recursive=False):
-  if recursive:
-   for p, dirnames, fils in os.walk(path):
-     for f in fils:
-       fil = os.path.join(p, f)
-       yield fil
+class AppError(Exception):
+  pass
+  
+def open_file(fn):
+  d = dirname(fn)
+  os.makedirs(d, exist_ok=True)
+  if isfile(fn):
+    return open(fn, 'rb+')
   else:
-    for f in os.listdir(path):
-      fil = os.path.join(path, f)
-      if not isfile(fil):
-        continue
-      yield fil
-      
+    return open(fn, 'ab+')
+
 def must_get(d, k):
   if type(k) == str:
     k = k.split('.')
@@ -58,7 +57,7 @@ def pretty(d, ident=0, detail=2):
     return d.f(detail)
   r = ''
   idt = ' ' * ident
-  if type(d) == list:
+  if type(d) in [list, set]:
     r += 'lst'
     for i in d:
       r += '\n' + idt + pretty(i, ident + 1, detail)
@@ -87,3 +86,5 @@ def parse_timestamp(ts):
 def render_timestamp(ts):
   return datetime.utcfromtimestamp(ts).isoformat()
   
+def fmtex(e):
+  return ''.join(traceback.TracebackException.from_exception(e).format())

requirements.txt

@@ -1 +1,3 @@
 requests
+readline
+hexdump

store.md

@@ -0,0 +1,63 @@
+# The store format
+This project uses a custom database format just because we can.
+
+The script reads the entire store on startup. Each object can br altered in memory. A 'dirty' status is set when an object is changed. periodically the script will flush the store, writing all changes back to disk.
+
+The store disk format is optimized for two things:
+* Loading the entire contents in memory
+* Writing changed objects back to disk
+
+## Objects
+First lets discuss what we are storing.
+
+Objects are identified by type and symbol. A symbol is an uppercase alphanumeric string that optionally contains '-' chars. the type is a lowecase alphanumeric string of length 3.
+
+For each type, the store has a class defined that is used to load objects of that type.
+
+An object identifier is its symbol and type joined with a '.' character.
+
+Some examples of object identifiers:
+
+* X1-J84.sys
+* X1-J84-0828772.way
+* CAPT-J-1.shp
+
+A waypoint is always part of a system. This is also visible because the waypoint symbol is prefixed by the system symbol. However, this relation is not enforced or used by the store. The symbol is an opaque string.
+
+An object has attributes. Values of attributes can be strings, ints, floats, bools, lists, dicts and references. lists and dicts can also only contain the values listed. References are pointers to other objects in the store.
+
+## Indices
+An index is a dict with a string as key and a list of objects as value. The dict is built when loading the store. when the index is iterated, each object is re-checked and removed if necessary.
+
+## API
+
+* store.load(fil) loads all objects
+* store.get(type, symbol, create=False) fetches the object. If create==False: None if it wasnt present
+* store.all(type) generator for all objects of a goven type
+* store.purge(obj)
+* store.clean() removes all expired objects
+* store.flush() writes all dirty objects to disk
+* store.defrag() consolidates the store file to minimize storage and loading time
+
+type may be a class or a string containing the name of a class. The type should be a subclass of models.base.Base
+
+# file format
+Until specified otherwise, all numbers are stored low-endian 64bit unsigned.
+
+The store file is built up out of chunks. A chunk is either empty or houses exactly one file. If a file is updated and its size fits the chunk, it is updated in-place. If the new content does not fit the chunk, a new chunk is allocated at the end of the file. The old chunk is marked as empty.
+
+A chunk starts with a chunk header. The header consists of three 8-byte fields. 
+
+The first field is the magic. Its value is 'ChNkcHnK'. The magic can be used to recover from a corrupted file.
+
+The second field is describing the size of the chunk in bytes, not including the header. The first bit of the field is the IN_USE flag. If it is not set, the contents of the chunk are ignored during loading.
+
+The third field described how much of the chunk is occupied by content. This is typically less than the size of the chunk because we allocate slack for each object to grow. The slack prevents frequent reallocation.
+
+# Future work
+This format is far from perfect. 
+
+* file corruption sometimes occurs. The cause of this still has to be found
+* Recovery of file corruption has not yet been implemented
+* Diskspace improvements are possible by eliminating slack for non-changing objects such as waypoints and compressing the file
+* Indices have not been implemented although a "member" index keeps track of which objects are in each system.