Update README.md

README.md

@@ -10,7 +10,7 @@ The API is deliberately well-defined with an OpenAPI specification and auto-gene
 
 Spothole itself is also open source, Public Domain licenced code that anyone can take and modify.
 
-Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, SOTA, WWFF, GMA, WWBOTA, HEMA, Parks 'n' Peaks, ZLOTA, WOTA, BOTA, LLOTA, the UK Packet Repeater Network, NG3K, and any site based on the xOTA software by nischu.
+Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, SOTA, WWFF, GMA, WWBOTA, HEMA, Parks 'n' Peaks, ZLOTA, WOTA, BOTA, LLOTA, WWTOTA, the UK Packet Repeater Network, NG3K, and any site based on the xOTA software by nischu.
 
 ![Screenshot](/images/screenshot2.png)
 
@@ -83,6 +83,8 @@ cp config-example.yml config.yml
 
 Then edit `config.yml` in your text editor of choice to set up the software as you like it. Mostly, this will involve enabling or disabling the various providers of spot and alert data.
 
+By default, all outdoor programme providers are enabled, as is one cluster node and the NG3K DXpedition data. The RBN spot providers are turned off by default due to the volume of traffic from CW/RTTY/FT8 skimmers, and the APRS and Packet spot providers are off by default on the assumption that Spothole users want a spot with a human at the other end of it, but all can be easily re-enabled.
+
 `config.yml` has some entries for QRZ.com username & password, and Clublog API keys. If provided, these allow Spothole to retrieve more information about DX spots, such as the country their callsign corresponds to. The software will work just fine without them, but you may find a few country flags etc. are less accurate or missing.
 
 Clublog API keys are free, but you'll need to get your own by submitting a helpdesk ticket and explaining what you'll use it for. The admin team are happy with the rate of requests made by my Spothole server, so unless you change the source code of yours to radically increase the rate of querying Clublog, I'm sure they will be fine with your server too.
@@ -102,6 +104,102 @@ The software can take a few seconds to start up, mostly because it is downloadin
 
 If you see some errors on startup, check your configuration, e.g. in case you have specified a port for the web server that is already in use by something else.
 
+### Multiple cluster nodes with different settings
+
+Dan, S50U has written in with his Spothole cluster settings. He is using a cluster node which provides RBN spots, and uses different SSIDs on his callsign to get different settings when logged into the same cluster node. For example:
+
+```
+  -
+    class: "DXCluster"
+    name: "S50CLX"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-10"
+```
+
+Telnet to DXSpider and log in with "callsign-10" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`SET/DX` (enable human DX spots)
+
+```
+  -
+    class: "DXCluster"
+    name: "RBN CW"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-11"
+    allow_rbn_spots: true
+    enabled-by-default-in-web-ui: false
+```
+
+Telnet to DXSpider and log in with "callsign-11" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`UNSET/DX` (stop human DX spots)<br/>
+`SET/SKIMMER CW` (enable CW RBN spots)
+
+```
+  -
+    class: "DXCluster"
+    name: "RBN RTTY"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-12"
+    allow_rbn_spots: true
+    enabled-by-default-in-web-ui: false
+```
+
+Telnet to DXSpider and log in with "callsign-12" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`UNSET/DX` (stop human DX spots)<br/>
+`SET/SKIMMER RTTY` (enable RTTY RBN spots)
+
+```
+  -
+    class: "DXCluster"
+    name: "RBN FT4/8"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-13"
+    allow_rbn_spots: true
+    enabled-by-default-in-web-ui: false
+```
+
+Telnet to DXSpider and log in with "callsign-13" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`UNSET/DX` (stop human DX spots)<br/>
+`SET/SKIMMER FT` (enable FT RBN spots)
+
+For each callsign-SSID, we also specify our basic information with commands:
+
+`SET/NAME Spothole10`, Spothole11... etc.<br/>
+`SET/QTH Cerkno`<br/>
+`SET/QRA JN66XD`<br/>
+`SET/HOME S50CLX`
+
 ### systemd configuration
 
 If you want Spothole to run automatically on startup on a Linux distribution that uses `systemd`, follow the instructions here. For distros that don't use `systemd`, or Windows/OSX/etc., you can find generic instructions for your OS online.

config-example.yml

@@ -53,6 +53,10 @@ spot-providers:
     class: "LLOTA"
     name: "LLOTA"
     enabled: true
+  -
+    class: "WWTOTA"
+    name: "WWTOTA"
+    enabled: true
   -
     class: "APRSIS"
     name: "APRS-IS"

core/constants.py

@@ -4,7 +4,7 @@ from data.sig import SIG
 
 # General software
 SOFTWARE_NAME = "Spothole by M0TRT"
-SOFTWARE_VERSION = "1.2"
+SOFTWARE_VERSION = "1.3-pre"
 
 # HTTP headers used for spot providers that use HTTP
 HTTP_HEADERS = {"User-Agent": SOFTWARE_NAME + ", v" + SOFTWARE_VERSION + " (operated by " + SERVER_OWNER_CALLSIGN + ")"}

core/lookup_helper.py

@@ -140,12 +140,14 @@ class LookupHelper:
     # database live if possible.
     def download_clublog_ctyxml(self):
         try:
-            logging.info("Downloading Clublog cty.xml...")
+            logging.info("Downloading Clublog cty.xml.gz...")
             response = self.CLUBLOG_CTY_XML_CACHE.get("https://cdn.clublog.org/cty.php?api=" + self.CLUBLOG_API_KEY,
                                                       headers=HTTP_HEADERS)
+            logging.info("Caching Clublog cty.xml.gz...")
             open(self.CLUBLOG_XML_DOWNLOAD_LOCATION + ".gz", 'wb').write(response.content)
             with gzip.open(self.CLUBLOG_XML_DOWNLOAD_LOCATION + ".gz", "rb") as uncompressed:
                 file_content = uncompressed.read()
+            logging.info("Caching Clublog cty.xml...")
             with open(self.CLUBLOG_XML_DOWNLOAD_LOCATION, "wb") as f:
                 f.write(file_content)
                 f.flush()

core/sig_utils.py

@@ -135,6 +135,10 @@ def populate_sig_ref_info(sig_ref):
                         sig_ref.latitude = ll[0]
                         sig_ref.longitude = ll[1]
                         break
+        elif sig.upper() == "WWTOTA":
+            if not sig_ref.name:
+                sig_ref.name = sig_ref.id
+            sig_ref.url = "https://wwtota.com/seznam/karta_rozhledny.php?ref=" + sig_ref.name
         elif sig.upper() == "WAB" or sig.upper() == "WAI":
             ll = wab_wai_square_to_lat_lon(ref_id)
             if ll:

spotproviders/wwtota.py

@@ -0,0 +1,41 @@
+from datetime import datetime
+
+import json
+import pytz
+
+from data.sig_ref import SIGRef
+from data.spot import Spot
+from spotproviders.http_spot_provider import HTTPSpotProvider
+
+
+# Spot provider for Towers on the Air
+class WWTOTA(HTTPSpotProvider):
+    POLL_INTERVAL_SEC = 120
+    SPOTS_URL = "https://wwtota.com/api/cluster_live.php"
+
+    def __init__(self, provider_config):
+        super().__init__(provider_config, self.SPOTS_URL, self.POLL_INTERVAL_SEC)
+
+    def http_response_to_spots(self, http_response):
+        new_spots = []
+        response_fixed = http_response.text.replace("\\/", "/")
+        response_json = json.loads(response_fixed)
+
+        # Iterate through source data
+        for source_spot in response_json["spots"]:
+            # Convert to our spot format
+            likely_freq = float(source_spot["freq"]) * 1000
+            if likely_freq < 1000000:
+                likely_freq = likely_freq * 1000
+            spot = Spot(source=self.name,
+                        dx_call=source_spot["call"].upper(),
+                        freq=likely_freq,
+                        comment=source_spot["comment"],
+                        sig="WWTOTA",
+                        sig_refs=[SIGRef(id=source_spot["ref"], sig="WWTOTA")],
+                        time=datetime.strptime(response_json["updated"][:10] + source_spot["time"], "%Y-%m-%d%H:%M").timestamp())
+
+            # Add to our list. Don't worry about de-duping, removing old spots etc. at this point; other code will do
+            # that for us.
+            new_spots.append(spot)
+        return new_spots

templates/about.html

@@ -25,7 +25,7 @@
     <h4 class="mt-4">What are "DX", "DE" and modes?</h4>
     <p>In amateur radio terminology, the "DX" contact is the "interesting" one that is using the frequency shown and looking for callers. They might be on a remote island or just in a local park, but either way it's interesting enough that someone has "spotted" them. The callsign listed under "DE" is the person who entered the spot of the "DX" operator. "Modes" are the type of communication they are using. For example you might see "CW" which is Morse Code, or voice "modes" like SSB or FM, or more exotic "data" modes which are used for computer-to-computer communication.</p>
     <h4 class="mt-4">What data sources are supported?</h4>
-    <p>Spothole can retrieve spots from: <a href="https://www.dxcluster.info/telnet/">Telnet-based DX clusters</a>, the <a href="https://www.reversebeacon.net/">Reverse Beacon Network (RBN)</a>, the <a href="https://www.aprs-is.net/">APRS Internet Service (APRS-IS)</a>, <a href="https://pota.app">POTA</a>, <a href="https://www.sota.org.uk/">SOTA</a>, <a href="https://wwff.co/">WWFF</a>, <a href="https://www.cqgma.org/">GMA</a>, <a href="https://wwbota.net/">WWBOTA</a>, <a href="http://www.hema.org.uk/">HEMA</a>, <a href="https://www.parksnpeaks.org/">Parks 'n' Peaks</a>, <a href="https://ontheair.nz">ZLOTA</a>, <a href="https://www.wota.org.uk/">WOTA</a>, <a href="https://llota.app">LLOTA</a>, the <a href="https://ukpacketradio.network/">UK Packet Repeater Network</a>, and any site based on the <a href="https://github.com/nischu/xOTA">xOTA software by nischu</a>.</p>
+    <p>Spothole can retrieve spots from: <a href="https://www.dxcluster.info/telnet/">Telnet-based DX clusters</a>, the <a href="https://www.reversebeacon.net/">Reverse Beacon Network (RBN)</a>, the <a href="https://www.aprs-is.net/">APRS Internet Service (APRS-IS)</a>, <a href="https://pota.app">POTA</a>, <a href="https://www.sota.org.uk/">SOTA</a>, <a href="https://wwff.co/">WWFF</a>, <a href="https://www.cqgma.org/">GMA</a>, <a href="https://wwbota.net/">WWBOTA</a>, <a href="http://www.hema.org.uk/">HEMA</a>, <a href="https://www.parksnpeaks.org/">Parks 'n' Peaks</a>, <a href="https://ontheair.nz">ZLOTA</a>, <a href="https://www.wota.org.uk/">WOTA</a>, <a href="https://llota.app">LLOTA</a>, <a href="https://wwtota.com">WWTOTA</a>, the <a href="https://ukpacketradio.network/">UK Packet Repeater Network</a>, and any site based on the <a href="https://github.com/nischu/xOTA">xOTA software by nischu</a>.</p>
     <p>Spothole can retrieve alerts from: <a href="https://www.ng3k.com/">NG3K</a>, <a href="https://pota.app">POTA</a>, <a href="https://www.sota.org.uk/">SOTA</a>, <a href="https://wwff.co/">WWFF</a>, <a href="https://www.parksnpeaks.org/">Parks 'n' Peaks</a>, <a href="https://www.wota.org.uk/">WOTA</a> and <a href="https://www.beachesontheair.com/">BOTA</a>.</p>
     <p>Note that the server owner has not necessarily enabled all these data sources. In particular it is common to disable RBN, to avoid the server being swamped with FT8 traffic, and to disable APRS-IS and UK Packet Net so that the server only displays stations where there is likely to be an operator physically present for a QSO.</p>
     <p>Between the various data sources, the following Special Interest Groups (SIGs) are supported: Parks on the Air (POTA), Summits on the Air (SOTA), Worldwide Flora & Fauna (WWFF), Global Mountain Activity (GMA), Worldwide Bunkers on the Air (WWBOTA), HuMPs Excluding Marilyns Award (HEMA), Islands on the Air (IOTA), Mills on the Air (MOTA), the Amateur Radio Lighthouse Socirty (ARLHS), International Lighthouse Lightship Weekend (ILLW), Silos on the Air (SIOTA), World Castles Award (WCA), New Zealand on the Air (ZLOTA), Keith Roget Memorial National Parks Award (KRMNPA), Wainwrights on the Air (WOTA), Beaches on the Air (BOTA), Lagos y Lagunas On the Air (LLOTA), Towers on the Air (WWTOTA), Worked All Britain (WAB), Worked All Ireland (WAI), and Toilets on the Air (TOTA).</p>

templates/alerts.html

@@ -8,8 +8,8 @@
         </div>
         <div class="col-auto">
             <p class="d-inline-flex gap-1">
-                <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i> Filters</button>
-                <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i> Display</button>
+                <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i>&nbsp;Filters</button>
+                <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i>&nbsp;Display</button>
             </p>
         </div>
     </div>

templates/bands.html

@@ -8,8 +8,8 @@
         </div>
         <div class="col-auto">
             <p class="d-inline-flex gap-1">
-                <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i> Filters</button>
-                <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i> Display</button>
+                <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i>&nbsp;Filters</button>
+                <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i>&nbsp;Display</button>
             </p>
         </div>
     </div>

templates/map.html

@@ -7,8 +7,8 @@
             <div class="col-auto me-auto pt-3"></div>
             <div class="col-auto">
                 <p class="d-inline-flex gap-1">
-                    <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i> Filters</button>
-                    <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i> Display</button>
+                    <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i>&nbsp;Filters</button>
+                    <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i>&nbsp;Display</button>
                 </p>
             </div>
         </div>

templates/spots.html

@@ -10,24 +10,25 @@
 
 <div class="mt-3">
     <div id="settingsButtonRow" class="row">
-        <div class="col-lg-6 me-auto pt-3 hideonmobile">
-            <p id="timing-container">Loading...</p>
-        </div>
-        <div class="col-lg-6 text-end">
+        <div class="col-4">
             <p class="d-inline-flex gap-1">
                 <span class="btn-group" role="group">
                   <input type="radio" class="btn-check" name="runPause" id="runButton" autocomplete="off" checked>
-                    <label class="btn btn-outline-primary" for="runButton"><i class="fa-solid fa-play"></i><span class="hideonmobile"> Run</span></label>
+                    <label class="btn btn-outline-primary" for="runButton"><i class="fa-solid fa-play"></i><span class="hideonmobile">&nbsp;Run</span></label>
 
                   <input type="radio" class="btn-check" name="runPause" id="pauseButton" autocomplete="off">
-                    <label class="btn btn-outline-primary" for="pauseButton"><i class="fa-solid fa-pause"></i><span class="hideonmobile"> Pause</span></label>
+                    <label class="btn btn-outline-primary" for="pauseButton"><i class="fa-solid fa-pause"></i><span class="hideonmobile">&nbsp;Pause</span></label>
                 </span>
+            </p>
+        </div>
+        <div class="col-8 text-end">
+            <p class="d-inline-flex gap-1">
                 <span style="position: relative;">
                     <i id="searchicon" class="fa-solid fa-magnifying-glass"></i>
                     <input id="search" type="search" class="form-control" oninput="filtersUpdated();" placeholder="Search">
                 </span>
-                <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i><span class="hideonmobile"> Filters</span></button>
-                <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i><span class="hideonmobile"> Display</span></button>
+                <button id="filters-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleFiltersPanel();"><i class="fa-solid fa-filter"></i><span class="hideonmobile">&nbsp;Filters</span></button>
+                <button id="display-button" type="button" class="btn btn-outline-primary" data-bs-toggle="button" onclick="toggleDisplayPanel();"><i class="fa-solid fa-desktop"></i><span class="hideonmobile">&nbsp;Display</span></button>
             </p>
         </div>
     </div>

webassets/js/spots.js

@@ -13,9 +13,6 @@ function loadSpots() {
 
     // Make the new query
     $.getJSON('/api/v1/spots' + buildQueryString(), function(jsonData) {
-        // Store last updated time
-        lastUpdateTime = moment.utc();
-        updateTimingDisplayRunPause();
         // Store data
         spots = jsonData;
         // Update table
@@ -37,9 +34,6 @@ function startSSEConnection() {
     evtSource = new EventSource('/api/v1/spots/stream' + buildQueryString());
 
     evtSource.onmessage = function(event) {
-        // Store last updated time
-        lastUpdateTime = moment.utc();
-        updateTimingDisplayRunPause();
         // Get the new spot
         newSpot = JSON.parse(event.data);
         // Awful fudge to ensure new incoming spots at the top of the list don't have timestamps that make them look
@@ -78,12 +72,6 @@ function startSSEConnection() {
     };
 }
 
-// Update the special timing display for the live spots page, which varies depending on run/pause selection.
-function updateTimingDisplayRunPause() {
-    let run = $('#runButton:checked').val();
-    $("#timing-container").html((run ? "Connected to server. Last update at " : "Paused at ") + lastUpdateTime.format('HH:mm') + " UTC.");
-}
-
 // Build a query string for the API, based on the filters that the user has selected.
 function buildQueryString() {
     var str =  "?";
@@ -515,13 +503,11 @@ $(document).ready(function() {
         // Need to start the SSE connection but also do a full re-query to catch up anything that we missed, so we
         // might as well just call loadSpots again which will trigger it all
         loadSpots();
-        updateTimingDisplayRunPause();
     });
     $("#pauseButton").change(function() {
         // If we are pausing and have an open SSE connection, stop it
         if (evtSource != null) {
             evtSource.close();
         }
-        updateTimingDisplayRunPause();
     });
 });