Get WWBOTA data via SSE. Thanks to Steven M1SDH for the patch. Closes #4

README.md

@@ -185,6 +185,8 @@ Finally, simply add the appropriate config to the `providers` section of `config
 
 ### Thanks
 
+As well as being my work, I have also gratefully received feature patches from Steven, M1SDH.
+
 The project contains a self-hosted copy of Font Awesome's free library, in the `/webasset/fa/` directory. This is subject to Font Awesome's licence and is not covered by the overall licence declared in the `LICENSE` file. This approach was taken in preference to using their hosted kits due to the popularity of this project exceeding the page view limit for their free hosted offering.
 
 The software uses a number of Python libraries as listed in `requirements.txt`, and a number of JavaScript libraries such as jQuery and moment.js. This project would not have been possible without these libraries, so many thanks to their developers.

requirements.txt

@@ -8,3 +8,4 @@ requests~=2.32.5
 aprslib~=0.7.2
 diskcache~=5.6.3
 psutil~=7.1.0
+requests-sse~=0.5.2

spotproviders/sse_spot_provider.py

@@ -0,0 +1,65 @@
+import logging
+from datetime import datetime
+from threading import Thread
+from time import sleep
+
+import pytz
+from requests_sse import EventSource
+
+from spotproviders.spot_provider import SpotProvider
+
+# Spot provider using Server-Sent Events.
+class SSESpotProvider(SpotProvider):
+
+    def __init__(self, provider_config, url):
+        super().__init__(provider_config)
+        self.url = url
+        self.event_source = None
+        self.thread = None
+        self.stopped = False
+        self.last_event_id = None
+
+    def start(self):
+        logging.info("Set up SSE connection to " + self.name + " spot API.")
+        self.stopped = False
+        self.thread = Thread(target=self.run)
+        self.thread.daemon = True
+        self.thread.start()
+
+    def stop(self):
+        self.stopped = True
+        if self.event_source:
+            self.event_source.close()
+        if self.thread:
+            self.thread.join()
+
+    def run(self):
+        while not self.stopped:
+            try:
+                logging.debug("Connecting to " + self.name + " spot API...")
+                with EventSource(self.url, headers=self.HTTP_HEADERS, latest_event_id=self.last_event_id, timeout=30) as event_source:
+                    self.event_source = event_source
+                    for event in self.event_source:
+                        if event.type == 'message':
+                            try:
+                                self.last_event_id = event.last_event_id
+                                new_spot = self.sse_message_to_spot(event.data)
+                                if new_spot:
+                                    self.submit(new_spot)
+
+                                self.status = "OK"
+                                self.last_update_time = datetime.now(pytz.UTC)
+                                logging.debug("Received data from " + self.name + " spot API.")
+
+                            except Exception as e:
+                                logging.exception("Exception processing message from SSE Spot Provider (" + self.name + ")")
+
+            except Exception as e:
+                self.status = "Error"
+                logging.exception("Exception in SSE Spot Provider (" + self.name + ")")
+            sleep(5)  # Wait before trying to reconnect
+
+    # Convert an SSE message received from the API into a spot. The whole message data is provided here so the subclass
+    # implementations can handle the message as JSON, XML, text, whatever the API actually provides.
+    def sse_message_to_spot(self, message_data):
+        raise NotImplementedError("Subclasses must implement this method")

spotproviders/wwbota.py

@@ -1,21 +1,19 @@
+import json
 from datetime import datetime
 
 from data.spot import Spot
-from spotproviders.http_spot_provider import HTTPSpotProvider
+from spotproviders.sse_spot_provider import SSESpotProvider
 
 
 # Spot provider for Worldwide Bunkers on the Air
-class WWBOTA(HTTPSpotProvider):
-    POLL_INTERVAL_SEC = 120
+class WWBOTA(SSESpotProvider):
     SPOTS_URL = "https://api.wwbota.org/spots/"
 
     def __init__(self, provider_config):
-        super().__init__(provider_config, self.SPOTS_URL, self.POLL_INTERVAL_SEC)
+        super().__init__(provider_config, self.SPOTS_URL)
 
-    def http_response_to_spots(self, http_response):
-        new_spots = []
-        # Iterate through source data
-        for source_spot in http_response.json():
+    def sse_message_to_spot(self, message):
+        source_spot = json.loads(message)
         # Convert to our spot format. First we unpack references, because WWBOTA spots can have more than one for
         # n-fer activations.
         refs = []
@@ -23,6 +21,7 @@ class WWBOTA(HTTPSpotProvider):
         for ref in source_spot["references"]:
             refs.append(ref["reference"])
             ref_names.append(ref["name"])
+
         spot = Spot(source=self.name,
                 dx_call=source_spot["call"].upper(),
                 de_call=source_spot["spotter"].upper(),
@@ -41,8 +40,5 @@ class WWBOTA(HTTPSpotProvider):
                 longitude=source_spot["references"][0]["long"],
                 qrt=source_spot["type"] == "QRT")
 
-            # Add to our list. Don't worry about de-duping, removing old spots etc. at this point; other code will do
-            # that for us. But WWBOTA does support a special "Test" spot type, we need to avoid adding that.
-            if source_spot["type"] != "Test":
-                new_spots.append(spot)
-        return new_spots
+        # WWBOTA does support a special "Test" spot type, we need to avoid adding that.
+        return spot if source_spot["type"] != "Test" else None

views/webpage_about.tpl

@@ -7,6 +7,6 @@
     <p>The API is deliberately well-defined with an <a href="/apidocs/openapi.yml">OpenAPI specification</a> and auto-generated <a href="/apidocs">API documentation</a>. The API delivers spots in a consistent format regardless of the data source, freeing developers from needing to know how each individual data source presents its data.</p>
     <p>Spothole itself is also open source, Public Domain licenced code that anyone can take and modify. <a href="https://git.ianrenton.com/ian/metaspot/">The source code is here</a>. If you want to run your own copy of Spothole, or start modifying it for your own purposes, the <a href="https://git.ianrenton.com/ian/spothole/src/branch/main/README.md">README file</a> contains a description of how the software works and how it's laid out, as well as instructions for configuring systemd, nginx and anything else you might need to run your own server.</p>
     <p>Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, SOTA, WWFF, GMA, WWBOTA, HEMA, and Parks 'n' Peaks.</p>
-    <p>The software was written by <a href="https://ianrenton.com">Ian Renton, MØTRT</a>.</p>
+    <p>The software was written by <a href="https://ianrenton.com">Ian Renton, MØTRT</a> and other contributors. Full details are available in the README.</p>
     <p><a href="/">&laquo; Back home</a></p>
 </div>