diff --git a/README.md b/README.md index d4c0f87..826e4ea 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,23 @@ #  -Spothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data. +Spothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open +JSON API as well as a website to browse the data.  -While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, Spothole differentiates itself by supporting a large number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it. +While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor +activity programmes for amateur radio, Spothole differentiates itself by supporting a large number of data sources, and +by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it. -The API is deliberately well-defined with an OpenAPI specification and auto-generated API documentation. The API delivers spots in a consistent format regardless of the data source, freeing developers from needing to know how each individual data source presents its data. +The API is deliberately well-defined with an OpenAPI specification and auto-generated API documentation. The API +delivers spots in a consistent format regardless of the data source, freeing developers from needing to know how each +individual data source presents its data. Spothole itself is also open source, Public Domain licenced code that anyone can take and modify. -Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, SOTA, WWFF, GMA, WWBOTA, HEMA, Parks 'n' Peaks, ZLOTA, WOTA, BOTA, LLOTA, WWTOTA, Tiles on the Air, the UK Packet Repeater Network, NG3K, and any site based on the xOTA software by nischu. +Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, +SOTA, WWFF, GMA, WWBOTA, HEMA, Parks 'n' Peaks, ZLOTA, WOTA, BOTA, LLOTA, WWTOTA, Tiles on the Air, the UK Packet +Repeater Network, NG3K, and any site based on the xOTA software by nischu.  @@ -18,65 +25,88 @@ Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), th ## Accessing the public version -You can access the public version's web interface at [https://spothole.app](https://spothole.app), and see [https://spothole.app/apidocs](https://spothole.app/apidocs) for the API details. +You can access the public version's web interface at [https://spothole.app](https://spothole.app), and +see [https://spothole.app/apidocs](https://spothole.app/apidocs) for the API details. -This is a Progressive Web App, so you can also "install" it to your Android or iOS device by accessing it in Chrome or Safari respectively, and following the menu-driven process for installing PWAs. +This is a Progressive Web App, so you can also "install" it to your Android or iOS device by accessing it in Chrome or +Safari respectively, and following the menu-driven process for installing PWAs. -You are more than welcome to use the data and the API that Spothole provides to power your own software. There are many ways to do this; see below. +You are more than welcome to use the data and the API that Spothole provides to power your own software. There are many +ways to do this; see below. ## Embedding Spothole in another website You can embed Spothole's web interface in another website, e.g. for use as part of a ham radio custom dashboard. -URL parameters can be used to trigger an "embedded" mode which hides the headers, footers and settings. In this mode, you provide configuration for the various filter and display options via additional URL parameters. Any settings that the user has set for Spothole are ignored. This is so that the embedding site can select, for example, their choice of dark mode or SIG filters, which will not impact how Spothole appears when the user accesses it directly. Effectively, it becomes separate to their normal Spothole settings. +URL parameters can be used to trigger an "embedded" mode which hides the headers, footers and settings. In this mode, +you provide configuration for the various filter and display options via additional URL parameters. Any settings that +the user has set for Spothole are ignored. This is so that the embedding site can select, for example, their choice of +dark mode or SIG filters, which will not impact how Spothole appears when the user accesses it directly. Effectively, it +becomes separate to their normal Spothole settings. -Setting `embedded` to true is important for the rest of the settings to be applied; otherwise, the user's defaults will be used in preference to the URL params. +Setting `embedded` to true is important for the rest of the settings to be applied; otherwise, the user's defaults will +be used in preference to the URL params. -These are supplied with the URL to the page you want to embed, for example for an embedded version of the band map in dark mode, use `https://spothole.app/bands?embedded=true&dark-mode=true`. For an embedded version of the main spots/home page in the system light/dark mode, use `https://spothole.app/?embedded=true`. For dark mode showing 70cm TOTA spots only, use `https://spothole.app/?embedded=true&dark-mode=true&sig=TOTA&band=70cm`. Providing no URL params causes the page to be loaded in the normal way it would when accessed directly in the user's browser. +These are supplied with the URL to the page you want to embed, for example for an embedded version of the band map in +dark mode, use `https://spothole.app/bands?embedded=true&dark-mode=true`. For an embedded version of the main spots/home +page in the system light/dark mode, use `https://spothole.app/?embedded=true`. For dark mode showing 70cm TOTA spots +only, use `https://spothole.app/?embedded=true&dark-mode=true&sig=TOTA&band=70cm`. Providing no URL params causes the +page to be loaded in the normal way it would when accessed directly in the user's browser. -The supported parameters are as follows. Generally these match the equivalent parameters in the real Spothole API, where a mapping exists. +The supported parameters are as follows. Generally these match the equivalent parameters in the real Spothole API, where +a mapping exists. -| Name | Allowed Values | Default | Example | Description | -|-------------------|-------------------------|---------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `embedded` | `true`, `false` | `false` | `?embedded=true` | Enables embedded mode. | -| `color-scheme` | `light`, `dark`, `auto` | `auto` | `?color-scheme=dark` | Forces light or dark mode in preference to the operating system default. | -| `time-zone` | `UTC`, `local` | `UTC` | `?time-zone=local` | Sets times to be in UTC or local time. | -| `limit` | 10, 25, 50, 100 | 50 | `?limit=50` | Sets the number of spots that will be displayed on the main spots page | -| `limit` | 25, 50, 100, 200, 500 | 100 | `?limit=100` | Sets the number of alerts that will be displayed on the alerts page | -| `max_age` | 300, 600, 1800, 3600 | 1800 | `?max_age=1800` | Sets the maximum age of spots displayed on the map and bands pages, in seconds. | -| `band` | Comma-separated list | (all) | `?band=20m,40m` | Sets the list of bands that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | -| `sig` | Comma-separated list | (all) | `?sig=POTA,SOTA,NO_SIG` | Sets the list of SIGs that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | -| `source` | Comma-separated list | (all) | `?source=Cluster` | Sets the list of sources that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface. | -| `mode_type` | Comma-separated list | (all) | `?mode_type=PHONE,CW` | Sets the list of mode types that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | -| `dx_continent` | Comma-separated list | (all) | `?dx_continent=NA,SA` | Sets the list of DX Continents that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface. | -| `de_continent` | Comma-separated list | (all) | `?de_continent=EU` | Sets the list of DE Continents that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | -| `map-center-lat` | Numeric (decimal) | (auto) | `?map-center-lat=51.5` | Sets the initial latitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots. | -| `map-center-lon` | Numeric (decimal) | (auto) | `?map-center-lon=-0.1` | Sets the initial longitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots. | -| `map-zoom` | Numeric (integer) | (auto) | `?map-zoom=6` | Sets the initial zoom level of the map on the map page. If omitted, the map auto-fits to the loaded spots. | +| Name | Allowed Values | Default | Example | Description | +|------------------|-------------------------|---------|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `embedded` | `true`, `false` | `false` | `?embedded=true` | Enables embedded mode. | +| `color-scheme` | `light`, `dark`, `auto` | `auto` | `?color-scheme=dark` | Forces light or dark mode in preference to the operating system default. | +| `time-zone` | `UTC`, `local` | `UTC` | `?time-zone=local` | Sets times to be in UTC or local time. | +| `limit` | 10, 25, 50, 100 | 50 | `?limit=50` | Sets the number of spots that will be displayed on the main spots page | +| `limit` | 25, 50, 100, 200, 500 | 100 | `?limit=100` | Sets the number of alerts that will be displayed on the alerts page | +| `max_age` | 300, 600, 1800, 3600 | 1800 | `?max_age=1800` | Sets the maximum age of spots displayed on the map and bands pages, in seconds. | +| `band` | Comma-separated list | (all) | `?band=20m,40m` | Sets the list of bands that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | +| `sig` | Comma-separated list | (all) | `?sig=POTA,SOTA,NO_SIG` | Sets the list of SIGs that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | +| `source` | Comma-separated list | (all) | `?source=Cluster` | Sets the list of sources that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface. | +| `mode_type` | Comma-separated list | (all) | `?mode_type=PHONE,CW` | Sets the list of mode types that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | +| `dx_continent` | Comma-separated list | (all) | `?dx_continent=NA,SA` | Sets the list of DX Continents that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface. | +| `de_continent` | Comma-separated list | (all) | `?de_continent=EU` | Sets the list of DE Continents that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. | +| `map-center-lat` | Numeric (decimal) | (auto) | `?map-center-lat=51.5` | Sets the initial latitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots. | +| `map-center-lon` | Numeric (decimal) | (auto) | `?map-center-lon=-0.1` | Sets the initial longitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots. | +| `map-zoom` | Numeric (integer) | (auto) | `?map-zoom=6` | Sets the initial zoom level of the map on the map page. If omitted, the map auto-fits to the loaded spots. | More will be added soon to allow customisation of filters and other display properties. ## Writing your own client -One of the key strengths of Spothole is that the API is well-defined and open to anyone to use. This means you can build your own software that uses data from Spothole. +One of the key strengths of Spothole is that the API is well-defined and open to anyone to use. This means you can build +your own software that uses data from Spothole. -As well as the main API endpoints to fetch spots and alerts, with various possible query parameters, there are also Server-Sent Events (SSE) API endpoints to receive a live feed, plus various utility lookup endpoints for things like callsign and park data. +As well as the main API endpoints to fetch spots and alerts, with various possible query parameters, there are also +Server-Sent Events (SSE) API endpoints to receive a live feed, plus various utility lookup endpoints for things like +callsign and park data. Various approaches exist to writing your own client, but in general: -* Refer to the API docs. These are built on an OpenAPI definition file (`/webassets/apidocs/openapi.yml`), which you can automatically use to generate a client skeleton using various software. +* Refer to the API docs. These are built on an OpenAPI definition file (`/webassets/apidocs/openapi.yml`), which you can + automatically use to generate a client skeleton using various software. * Call the main "spots" or "alerts" API endpoints to get the data you want. Apply filters if necessary. -* Call the "options" API to get an idea of which bands, modes etc. the server knows about. You might want to do that first before calling the spots/alerts APIs, to allow you to populate your filters correctly. -* Refer to the provided HTML/JS interface for a reference on different approaches. For example, the "map" and "bands" pages simply query the main spot API on a timer, whereas the main/spots page combines this approach with using the Server-Sent Events (SSE) endpoint to update live. +* Call the "options" API to get an idea of which bands, modes etc. the server knows about. You might want to do that + first before calling the spots/alerts APIs, to allow you to populate your filters correctly. +* Refer to the provided HTML/JS interface for a reference on different approaches. For example, the "map" and "bands" + pages simply query the main spot API on a timer, whereas the main/spots page combines this approach with using the + Server-Sent Events (SSE) endpoint to update live. * Let me know if you get stuck, I'm happy to help! ## Running your own copy -If you want to run a copy of Spothole with different configuration settings than the main instance, you can download it and run it on your own local machine or server. +If you want to run a copy of Spothole with different configuration settings than the main instance, you can download it +and run it on your own local machine or server. -You will require Python version 3.8 or later. If you encounter an error about `gdal-config` during the following process, you will also need `libgdal-dev` installed. +You will require Python version 3.8 or later. If you encounter an error about `gdal-config` during the following +process, you will also need `libgdal-dev` installed. -To download and set up Spothole on a Debian server, run the following commands. Other operating systems will likely be similar. +To download and set up Spothole on a Debian server, run the following commands. Other operating systems will likely be +similar. ```bash git clone ssh://git@git.ianrenton.com/ian/spothole.git @@ -88,13 +118,23 @@ deactivate cp config-example.yml config.yml ``` -Then edit `config.yml` in your text editor of choice to set up the software as you like it. Mostly, this will involve enabling or disabling the various providers of spot and alert data. +Then edit `config.yml` in your text editor of choice to set up the software as you like it. Mostly, this will involve +enabling or disabling the various providers of spot and alert data. -By default, all outdoor programme providers are enabled, as is one cluster node and the NG3K DXpedition data. The RBN spot providers are turned off by default due to the volume of traffic from CW/RTTY/FT8 skimmers, and the APRS and Packet spot providers are off by default on the assumption that Spothole users want a spot with a human at the other end of it, but all can be easily re-enabled. +By default, all outdoor programme providers are enabled, as is one cluster node and the NG3K DXpedition data. The RBN +spot providers are turned off by default due to the volume of traffic from CW/RTTY/FT8 skimmers, and the APRS and Packet +spot providers are off by default on the assumption that Spothole users want a spot with a human at the other end of it, +but all can be easily re-enabled. -Other parameters you will want to update include the base URL to your instance, and whether you want to serve a full web-based DX cluster interface or just the API endpoints for client software to use. +Other parameters you will want to update include the base URL to your instance, and whether you want to serve a full +web-based DX cluster interface or just the API endpoints for client software to use. -`config.yml` has an entry for a Clublog API key. If provided, this will allow Spothole to retrieve some more information about DX spots. The software will work just fine without it, but you may find a few country flags etc. are less accurate or missing. Clublog API keys are free, but you'll need to get your own by submitting a helpdesk ticket and explaining what you'll use it for. The admin team are happy with the rate of requests made by my Spothole server, so unless you change the source code of yours to radically increase the rate of querying Clublog, I'm sure they will be fine with your server too. +`config.yml` has an entry for a Clublog API key. If provided, this will allow Spothole to retrieve some more information +about DX spots. The software will work just fine without it, but you may find a few country flags etc. are less accurate +or missing. Clublog API keys are free, but you'll need to get your own by submitting a helpdesk ticket and explaining +what you'll use it for. The admin team are happy with the rate of requests made by my Spothole server, so unless you +change the source code of yours to radically increase the rate of querying Clublog, I'm sure they will be fine with your +server too. Once you're happy with the content of `config.yml`, you can proceed to running the software. @@ -105,13 +145,16 @@ source .venv/bin/activate python3 spothole.py ``` -The software can take a few seconds to start up, mostly because it is downloading an updated file to match callsigns to countries. This is normal, don't panic! +The software can take a few seconds to start up, mostly because it is downloading an updated file to match callsigns to +countries. This is normal, don't panic! -If you see some errors on startup, check your configuration, e.g. in case you have specified a port for the web server that is already in use by something else. +If you see some errors on startup, check your configuration, e.g. in case you have specified a port for the web server +that is already in use by something else. ### Multiple cluster nodes with different settings -Dan, S50U has written in with his Spothole cluster settings. He is using a cluster node which provides RBN spots, and uses different SSIDs on his callsign to get different settings when logged into the same cluster node. For example: +Dan, S50U has written in with his Spothole cluster settings. He is using a cluster node which provides RBN spots, and +uses different SSIDs on his callsign to get different settings when logged into the same cluster node. For example: ``` - @@ -207,9 +250,12 @@ For each callsign-SSID, we also specify our basic information with commands: ### systemd configuration -If you want Spothole to run automatically on startup on a Linux distribution that uses `systemd`, follow the instructions here. For distros that don't use `systemd`, or Windows/OSX/etc., you can find generic instructions for your OS online. +If you want Spothole to run automatically on startup on a Linux distribution that uses `systemd`, follow the +instructions here. For distros that don't use `systemd`, or Windows/OSX/etc., you can find generic instructions for your +OS online. -Create a file at `/etc/systemd/system/spothole.service`. Give it the following content, adjusting for the user you want to run it as and the directory in which you have installed it: +Create a file at `/etc/systemd/system/spothole.service`. Give it the following content, adjusting for the user you want +to run it as and the directory in which you have installed it: ``` [Unit] @@ -239,11 +285,19 @@ Check the service has started up correctly with `sudo journalctl -u spothole -f` ### nginx Reverse Proxy configuration -Web servers generally serve their pages from port 80. However, it's best not to serve Spothole's web interface directly on port 80, as that requires root privileges on a Linux system. It also and prevents us using HTTPS to serve a secure site, since Spothole itself doesn't directly support acting as an HTTPS server. The normal solution to this is to use a "reverse proxy" setup, where a general web server handles HTTP and HTTP requests (to port 80 & 443 respectively), then passes on the request to the back-end application (in this case Spothole). nginx is a common choice for this general web server. +Web servers generally serve their pages from port 80. However, it's best not to serve Spothole's web interface directly +on port 80, as that requires root privileges on a Linux system. It also and prevents us using HTTPS to serve a secure +site, since Spothole itself doesn't directly support acting as an HTTPS server. The normal solution to this is to use +a "reverse proxy" setup, where a general web server handles HTTP and HTTP requests (to port 80 & 443 respectively), then +passes on the request to the back-end application (in this case Spothole). nginx is a common choice for this general web +server. -To set up nginx as a reverse proxy that sits in front of Spothole, first ensure it's installed e.g. `sudo apt install nginx`, and enabled e.g. `sudo systemd enable nginx`. +To set up nginx as a reverse proxy that sits in front of Spothole, first ensure it's installed e.g. +`sudo apt install nginx`, and enabled e.g. `sudo systemd enable nginx`. -Create a file at `/etc/nginx/sites-available/` called `spothole`. Give it the following contents, replacing `spothole.app` with the domain name on which you want to run Spothole. If you changed the port on which Spothole runs, update that on the "proxy_pass" line too. +Create a file at `/etc/nginx/sites-available/` called `spothole`. Give it the following contents, replacing +`spothole.app` with the domain name on which you want to run Spothole. If you changed the port on which Spothole runs, +update that on the "proxy_pass" line too. ```nginx server { @@ -298,13 +352,19 @@ server { } ``` -One further change you might want to make to the file above is the `add_header Access-Control-Allow-Origin` statements. These are what's used on -my own Spothole server to make sure that other third-party web-based software can get the data from my instance, and applies to any endpoint underneath `/api`. If you want +One further change you might want to make to the file above is the `add_header Access-Control-Allow-Origin` statements. +These are what's used on +my own Spothole server to make sure that other third-party web-based software can get the data from my instance, and +applies to any endpoint underneath `/api`. If you want *your* Spothole instance to be set up the same way, so that others can write software in JavaScript that can access it, -leave this intact. But if you want your Spothole instance to only be usable by scripts running on the web server you write, -you can remove these lines. (Note that this doesn't stop other people writing *non-web-based* software that accesses your -Spothole API—the enforcement of cross-origin headers only happens within the user's browser. If you need to lock your -instance down so that no-one else can access it with *any* software, that's an aspect of nginx or firewall config that you will need +leave this intact. But if you want your Spothole instance to only be usable by scripts running on the web server you +write, +you can remove these lines. (Note that this doesn't stop other people writing *non-web-based* software that accesses +your +Spothole API—the enforcement of cross-origin headers only happens within the user's browser. If you need to lock +your +instance down so that no-one else can access it with *any* software, that's an aspect of nginx or firewall config that +you will need to find help with elsewhere.) Now, make a symbolic link to enable the site: @@ -314,17 +374,22 @@ cd /etc/nginx/sites-enabled sudo ln -sf ../sites-available/spothole ``` -Test that your nginx config isn't broken using `nginx -t`. If it works, restart nginx with `sudo systemctl restart nginx`. +Test that your nginx config isn't broken using `nginx -t`. If it works, restart nginx with +`sudo systemctl restart nginx`. -If you haven't already done so, set up a DNS entry to make sure requests for your domain name end up at the server that's running Spothole. +If you haven't already done so, set up a DNS entry to make sure requests for your domain name end up at the server +that's running Spothole. You should now be able to access the web interface by going to the domain from your browser. -Once that's working, [install certbot](https://certbot.eff.org/instructions?ws=nginx&os=snap) onto your server. Run it as root, and when prompted pick your domain name from the list. After a few seconds, it should successfully provision a certificate and modify your nginx config files automatically. You should then be able to access the site via HTTPS. +Once that's working, [install certbot](https://certbot.eff.org/instructions?ws=nginx&os=snap) onto your server. Run it +as root, and when prompted pick your domain name from the list. After a few seconds, it should successfully provision a +certificate and modify your nginx config files automatically. You should then be able to access the site via HTTPS. ## Modifying the source code -Spothole is Public Domain licenced, so you can grab the source code and start modifying it for your own needs. Contributions of code back to the main repository are encouraged, but completely optional. +Spothole is Public Domain licenced, so you can grab the source code and start modifying it for your own needs. +Contributions of code back to the main repository are encouraged, but completely optional. ### Code structure @@ -356,24 +421,37 @@ To navigate your way around the source code, this list may help. * `/` - Main script (`spothole.py`), pip `requirements.txt`, config, README, etc. * `/images` - Image sources -* `/datafiles` - Local data sources (differentiated from the majority of data files which are loaded from URLs and cached in `/cache`) -* `/cache` - Directory where static-ish data downloaded from the internet is cached to avoid rapid re-requests, and where spot/alert data is cached so that it survives a software restart. Created on first run. +* `/datafiles` - Local data sources (differentiated from the majority of data files which are loaded from URLs and + cached in `/cache`) +* `/cache` - Directory where static-ish data downloaded from the internet is cached to avoid rapid re-requests, and + where spot/alert data is cached so that it survives a software restart. Created on first run. ### Extending the server -Spothole is designed to be easily extensible. If you want to write your own spot provider, for example, simply add a module to the `spotproviders` package containing your class. (Currently, in order to be loaded correctly, the module (file) name should be the same as the class name, but lower case.) +Spothole is designed to be easily extensible. If you want to write your own spot provider, for example, simply add a +module to the `spotproviders` package containing your class. (Currently, in order to be loaded correctly, the module ( +file) name should be the same as the class name, but lower case.) -Your class should extend "SpotProvider"; if it operates by polling an HTTP Server on a timer, it can instead extend "HTTPSpotProvider" where some of the work is done for you. +Your class should extend "SpotProvider"; if it operates by polling an HTTP Server on a timer, it can instead extend " +HTTPSpotProvider" where some of the work is done for you. -The class will need to implement a constructor that takes in the `provider_config` and provides it to the superclass constructor, while also taking any other config parameters it needs. +The class will need to implement a constructor that takes in the `provider_config` and provides it to the superclass +constructor, while also taking any other config parameters it needs. -If you're extending the base `SpotProvider` class, you will need to implement `start()` and `stop()` methods that start and stop a separate thread which handles the provider's processing needs. The thread should call `submit()` or `submit_batch()` when it has one or more spots to report. +If you're extending the base `SpotProvider` class, you will need to implement `start()` and `stop()` methods that start +and stop a separate thread which handles the provider's processing needs. The thread should call `submit()` or +`submit_batch()` when it has one or more spots to report. -If you're extending the `HTTPSpotProvider` class, you will need to provide a URI to query and an interval to the superclass constructor. You'll then need to implement the `http_response_to_spots()` method which is called when new data is retrieved. Your implementation should then call `submit()` or `submit_batch()` when it has one or more spots to report. +If you're extending the `HTTPSpotProvider` class, you will need to provide a URI to query and an interval to the +superclass constructor. You'll then need to implement the `http_response_to_spots()` method which is called when new +data is retrieved. Your implementation should then call `submit()` or `submit_batch()` when it has one or more spots to +report. -When constructing spots, use the comments in the Spot class and the existing implementations as an example. All parameters are optional, but you will at least want to provide a `time` (which must be timezone-aware) and a `dx_call`. +When constructing spots, use the comments in the Spot class and the existing implementations as an example. All +parameters are optional, but you will at least want to provide a `time` (which must be timezone-aware) and a `dx_call`. -Finally, simply add the appropriate config to the `spot_providers` section of `config.yml`, and your provider should be instantiated on startup. +Finally, simply add the appropriate config to the `spot_providers` section of `config.yml`, and your provider should be +instantiated on startup. The same approach as above is also used for alert providers. @@ -381,18 +459,27 @@ The same approach as above is also used for alert providers. As well as being my work, I have also gratefully received feature patches from Steven, M1SDH. -The project contains GeoJSON files for CQ and ITU zones, in the `/datafiles/` directory. These are MIT-licenced and, to my knowledge, created by HA8TKS for his CQ and ITU zone layers for Leaflet. +The project contains GeoJSON files for CQ and ITU zones, in the `/datafiles/` directory. These are MIT-licenced and, to +my knowledge, created by HA8TKS for his CQ and ITU zone layers for Leaflet. -The project contains a set of flag icons generated using the "Noto Color Emoji" font on a Debian system, in the `/webassets/img/flags/` directory. +The project contains a set of flag icons generated using the "Noto Color Emoji" font on a Debian system, in the +`/webassets/img/flags/` directory. -The software uses a number of Python libraries as listed in `requirements.txt`, and a number of JavaScript libraries. This project would not have been possible without these libraries, so many thanks to their developers. +The software uses a number of Python libraries as listed in `requirements.txt`, and a number of JavaScript libraries. +This project would not have been possible without these libraries, so many thanks to their developers. ### Third Party Libraries -A number of third-party libraries are self-hosted in the `/webassets/vendor/` directory. These files are subject to their own licences and are not covered by the overall licence declared in the `LICENSE` file. +A number of third-party libraries are self-hosted in the `/webassets/vendor/` directory. These files are subject to +their own licences and are not covered by the overall licence declared in the `LICENSE` file. -Particular thanks go to country-files.com for providing country lookup data for amateur radio, to K0SWE for [this JSON-formatted DXCC data](https://github.com/k0swe/dxcc-json/), and to the developers of `pyhamtools` for making it easy to use country-files.com data as well as QRZ.com and Clublog lookup. +Particular thanks go to country-files.com for providing country lookup data for amateur radio, to K0SWE +for [this JSON-formatted DXCC data](https://github.com/k0swe/dxcc-json/), and to the developers of `pyhamtools` for +making it easy to use country-files.com data as well as QRZ.com and Clublog lookup. -Amateur radio clusters, outdoor programmes, propagation data providers etc. are almost all volunteer-run services that make no or little profit, and are done for the love of amateur radio. Services like Spothole, which build on top of them, are truly standing on the shoulders of giants. None of this would have been possible without the hard work and dedication of many other people within the amaetur radio community. +Amateur radio clusters, outdoor programmes, propagation data providers etc. are almost all volunteer-run services that +make no or little profit, and are done for the love of amateur radio. Services like Spothole, which build on top of +them, are truly standing on the shoulders of giants. None of this would have been possible without the hard work and +dedication of many other people within the amaetur radio community. The project's name was suggested by Harm, DK4HAA. Thanks! diff --git a/config-example.yml b/config-example.yml index 0c29540..56e8c74 100644 --- a/config-example.yml +++ b/config-example.yml @@ -24,60 +24,46 @@ base-url: "http://localhost:8080" # for CW/RTTY and 7001 for FT8, so if you want both, you need two entries, as shown below. # Feel free to write your own provider classes! There are details in the README. spot-providers: - - - class: "POTA" + - class: "POTA" name: "POTA" enabled: true - - - class: "SOTA" + - class: "SOTA" name: "SOTA" enabled: true - - - class: "WWFF" + - class: "WWFF" name: "WWFF" enabled: true - - - class: "WWBOTA" + - class: "WWBOTA" name: "WWBOTA" enabled: true - - - class: "GMA" + - class: "GMA" name: "GMA" enabled: true - - - class: "HEMA" + - class: "HEMA" name: "HEMA" enabled: true - - - class: "ParksNPeaks" + - class: "ParksNPeaks" name: "ParksNPeaks" enabled: true - - - class: "ZLOTA" + - class: "ZLOTA" name: "ZLOTA" enabled: true - - - class: "WOTA" + - class: "WOTA" name: "WOTA" enabled: true - - - class: "LLOTA" + - class: "LLOTA" name: "LLOTA" enabled: true - - - class: "WWTOTA" + - class: "WWTOTA" name: "WWTOTA" enabled: true - - - class: "Tiles" + - class: "Tiles" name: "Tiles" enabled: true - - - class: "APRSIS" + - class: "APRSIS" name: "APRS-IS" enabled: false - - - class: "DXCluster" + - class: "DXCluster" name: "HRD Cluster" enabled: true host: "hrd.wa9pie.net" @@ -92,8 +78,7 @@ spot-providers: # sure you aren't also separately connecting to RBN directly, otherwise you may get duplicate spots.) Note that not # all clusters sent RBN spots anyway. allow_rbn_spots: false - - - class: "DXCluster" + - class: "DXCluster" name: "W3LPL Cluster" enabled: false host: "w3lpl.net" @@ -108,8 +93,7 @@ spot-providers: # sure you aren't also separately connecting to RBN directly, otherwise you may get duplicate spots.) Note that not # all clusters sent RBN spots anyway. allow_rbn_spots: false - - - class: "RBN" + - class: "RBN" name: "RBN CW/RTTY" enabled: false port: 7000 @@ -118,19 +102,16 @@ spot-providers: # received by Spothole but not shown on the web UI unless the user explicitly turns it on. For that behaviour, # set enabled to true, but enabled-by-default-in-web-ui to false. enabled-by-default-in-web-ui: false - - - class: "RBN" + - class: "RBN" name: "RBN FT8" enabled: false port: 7001 enabled-by-default-in-web-ui: false - - - class: "UKPacketNet" + - class: "UKPacketNet" name: "UK Packet Radio Net" enabled: false enabled-by-default-in-web-ui: false - - - class: "XOTA" + - class: "XOTA" name: "39C3 TOTA" enabled: false url: "wss://39c3.totawatch.de/api/spot/live" @@ -139,8 +120,7 @@ spot-providers: # programmes and so different URLs provide different programmes. sig: "TOTA" locations-csv: "datafiles/39c3-tota.csv" - - - class: "XOTA" + - class: "XOTA" name: "EH23 TOTA" enabled: true url: "wss://eh23.totawatch.de/api/spot/live" @@ -150,32 +130,25 @@ spot-providers: # Alert providers to use. Same setup as the spot providers list above. alert-providers: - - - class: "POTA" + - class: "POTA" name: "POTA" enabled: true - - - class: "SOTA" + - class: "SOTA" name: "SOTA" enabled: true - - - class: "WWFF" + - class: "WWFF" name: "WWFF" enabled: true - - - class: "ParksNPeaks" + - class: "ParksNPeaks" name: "ParksNPeaks" enabled: true - - - class: "WOTA" + - class: "WOTA" name: "WOTA" enabled: true - - - class: "BOTA" + - class: "BOTA" name: "BOTA" enabled: true - - - class: "NG3K" + - class: "NG3K" name: "NG3K" enabled: true @@ -183,20 +156,16 @@ alert-providers: # Solar condition providers to use. These poll external APIs for solar propagation data (SFI, A/K indices, band # conditions, etc.) and make it available via the /api/v1/solar endpoint. solar-condition-providers: - - - class: "HamQSL" + - class: "HamQSL" name: "HamQSL" enabled: true - - - class: "NOAA3dayForecast" + - class: "NOAA3dayForecast" name: "NOAA 3-day Forecast" enabled: true - - - class: "GIROIonosonde" + - class: "GIROIonosonde" name: "GIRO Ionosonde Data" enabled: true - - - class: "KC2GProp" + - class: "KC2GProp" name: "KC2G Propagation Data" enabled: true @@ -214,11 +183,11 @@ allow-spotting: true # Options for the web UI. web-ui-options: - spot-count: [10, 25, 50, 100] + spot-count: [ 10, 25, 50, 100 ] spot-count-default: 50 - max-spot-age: [5, 10, 30, 60] + max-spot-age: [ 5, 10, 30, 60 ] max-spot-age-default: 30 - alert-count: [25, 50, 100, 200, 500] + alert-count: [ 25, 50, 100, 200, 500 ] alert-count-default: 100 # Default UI colour scheme. Supported values are "light", "dark" and "auto" (i.e. use the browser/OS colour scheme). # Users can still override this in the UI to their own preference. diff --git a/core/lookup_helper.py b/core/lookup_helper.py index b9d81e0..7fe2594 100644 --- a/core/lookup_helper.py +++ b/core/lookup_helper.py @@ -385,12 +385,12 @@ class LookupHelper: data = self._get_qrz_data_for_callsign(call, credentials) if data and "latitude" in data and "longitude" in data and ( float(data["latitude"]) != 0 or float(data["longitude"]) != 0) and -89.9 < float( - data["latitude"]) < 89.9: + data["latitude"]) < 89.9: return [float(data["latitude"]), float(data["longitude"])] data = self._get_hamqth_data_for_callsign(call, credentials) if data and "latitude" in data and "longitude" in data and ( float(data["latitude"]) != 0 or float(data["longitude"]) != 0) and -89.9 < float( - data["latitude"]) < 89.9: + data["latitude"]) < 89.9: return [float(data["latitude"]), float(data["longitude"])] else: return None @@ -629,6 +629,7 @@ class LookupHelper: # Singleton object lookup_helper = LookupHelper() + def infer_mode_from_comment(comment): """Infer a mode from the comment""" diff --git a/core/sig_utils.py b/core/sig_utils.py index 4f24a96..4608659 100644 --- a/core/sig_utils.py +++ b/core/sig_utils.py @@ -81,7 +81,8 @@ def populate_sig_ref_info(sig_ref): elif sig.upper() == "SIOTA": siota_csv_data = SEMI_STATIC_URL_DATA_CACHE.get("https://www.silosontheair.com/data/silos.csv", headers=HTTP_HEADERS) - siota_index = {row["SILO_CODE"]: row for row in csv.DictReader(siota_csv_data.content.decode().splitlines())} + siota_index = {row["SILO_CODE"]: row for row in + csv.DictReader(siota_csv_data.content.decode().splitlines())} row = siota_index.get(ref_id) if row: sig_ref.name = row["NAME"] if "NAME" in row else None diff --git a/core/status_reporter.py b/core/status_reporter.py index f31ec4c..ff6e39a 100644 --- a/core/status_reporter.py +++ b/core/status_reporter.py @@ -89,7 +89,8 @@ class StatusReporter: "last_page_access_time"].replace( tzinfo=pytz.UTC).timestamp() if self._web_server.web_server_metrics[ "last_page_access_time"] else 0, - "page_access_count": self._web_server.web_server_metrics["page_access_counter"]} + "page_access_count": self._web_server.web_server_metrics[ + "page_access_counter"]} # Update Prometheus metrics memory_use_gauge.set(psutil.Process(os.getpid()).memory_info().rss) diff --git a/data/lookup_credentials.py b/data/lookup_credentials.py index b31c149..ae80f83 100644 --- a/data/lookup_credentials.py +++ b/data/lookup_credentials.py @@ -6,10 +6,10 @@ class LookupCredentials: """Per-request credentials for QRZ.com and HamQTH online callsign lookups.""" qrz_username: str = "" qrz_password: str = "" - qrz_session_key: str = "" # alternative to username/password + qrz_session_key: str = "" # alternative to username/password hamqth_username: str = "" hamqth_password: str = "" - hamqth_session_id: str = "" # alternative to username/password + hamqth_session_id: str = "" # alternative to username/password def extract_credentials(query_params): diff --git a/data/spot.py b/data/spot.py index 5e80bcb..0a56eb2 100644 --- a/data/spot.py +++ b/data/spot.py @@ -357,7 +357,8 @@ class Spot: self.dx_latitude = float(str(self.dx_latitude)) self.dx_longitude = float(str(self.dx_longitude)) except (TypeError, ValueError): - logging.warning("Received non-numeric strings in lat/lon (" + str(self.dx_latitude) + ", " + str(self.dx_longitude) + ") for call " + str(self.dx_call) + ", rejecting it") + logging.warning("Received non-numeric strings in lat/lon (" + str(self.dx_latitude) + ", " + str( + self.dx_longitude) + ") for call " + str(self.dx_call) + ", rejecting it") self.dx_latitude = None self.dx_longitude = None diff --git a/server/handlers/api/lookups.py b/server/handlers/api/lookups.py index 4f525f2..e885c07 100644 --- a/server/handlers/api/lookups.py +++ b/server/handlers/api/lookups.py @@ -118,8 +118,9 @@ class APILookupSIGRefHandler(tornado.web.RequestHandler): else: self.write( - json.dumps("Error - '" + ref_id + "' does not look like a valid reference ID for " + sig + ".", - default=serialize_everything)) + json.dumps( + "Error - '" + ref_id + "' does not look like a valid reference ID for " + sig + ".", + default=serialize_everything)) self.set_status(422) else: self.write(json.dumps("Error - sig '" + sig + "' is not known.", default=serialize_everything)) diff --git a/server/handlers/pagetemplate.py b/server/handlers/pagetemplate.py index c96aa14..bbe9deb 100644 --- a/server/handlers/pagetemplate.py +++ b/server/handlers/pagetemplate.py @@ -33,4 +33,4 @@ class PageTemplateHandler(tornado.web.RequestHandler): # Load named template, and provide variables used in templates self.render(self._template_name + ".html", software_version=SOFTWARE_VERSION, server_owner_callsign=SERVER_OWNER_CALLSIGN, allow_spotting=ALLOW_SPOTTING, - web_ui_options=WEB_UI_OPTIONS, baseurl=BASE_URL, current_path=self.request.path) \ No newline at end of file + web_ui_options=WEB_UI_OPTIONS, baseurl=BASE_URL, current_path=self.request.path) diff --git a/server/webserver.py b/server/webserver.py index 69f0926..be52bba 100644 --- a/server/webserver.py +++ b/server/webserver.py @@ -18,7 +18,6 @@ from server.handlers.api.status import APIStatusHandler from server.handlers.metrics import PrometheusMetricsHandler from server.handlers.pagetemplate import PageTemplateHandler - _HERE = os.path.dirname(__file__ or "") diff --git a/solarconditionsproviders/hamqsl.py b/solarconditionsproviders/hamqsl.py index c20db58..62cfd70 100644 --- a/solarconditionsproviders/hamqsl.py +++ b/solarconditionsproviders/hamqsl.py @@ -4,7 +4,6 @@ from xml.etree import ElementTree import pytz from dateutil import parser as dateutil_parser, tz as dateutil_tz - from solarconditionsproviders.http_solar_conditions_provider import HTTPSolarConditionsProvider POLL_INTERVAL = 3600 # 1 hour @@ -98,16 +97,17 @@ class HamQSL(HTTPSolarConditionsProvider): "solar_wind": float_val("solarwind"), "magnetic_field": float_val("magneticfield"), "geomag_field": text("geomagfield").title() - .replace("Vr Quiet", "Very Quiet") - .replace("Unsettld", "Unsettled") - .replace("Min Strm", "Minor Storm") - .replace("Maj Strm", "Major Storm") - .replace("Sev Strm", "Severe Storm") - .replace("Ext Strm", "Extreme Storm"), + .replace("Vr Quiet", "Very Quiet") + .replace("Unsettld", "Unsettled") + .replace("Min Strm", "Minor Storm") + .replace("Maj Strm", "Major Storm") + .replace("Sev Strm", "Severe Storm") + .replace("Ext Strm", "Extreme Storm"), "geomag_noise": text("signalnoise"), "hf_conditions": hf_conditions, "vhf_conditions": { - "vhf_aurora_northern_hemi": (vhf_map.get(("vhf-aurora", "northern_hemi")) or "").title().replace("Lat Aur", "Latitude") or None, + "vhf_aurora_northern_hemi": (vhf_map.get(("vhf-aurora", "northern_hemi")) or "").title().replace( + "Lat Aur", "Latitude") or None, "es_2m_europe": vhf_map.get(("E-Skip", "europe")), "es_4m_europe": vhf_map.get(("E-Skip", "europe_4m")), "es_6m_europe": vhf_map.get(("E-Skip", "europe_6m")), diff --git a/solarconditionsproviders/noaa3dayforecast.py b/solarconditionsproviders/noaa3dayforecast.py index 71efc19..7a0c0f5 100644 --- a/solarconditionsproviders/noaa3dayforecast.py +++ b/solarconditionsproviders/noaa3dayforecast.py @@ -4,7 +4,7 @@ from datetime import datetime, timezone from solarconditionsproviders.http_solar_conditions_provider import HTTPSolarConditionsProvider -POLL_INTERVAL = 10800 # Every 3 hours +POLL_INTERVAL = 10800 # Every 3 hours URL = "https://services.swpc.noaa.gov/text/3-day-forecast.txt" diff --git a/spothole.py b/spothole.py index cef083d..d066767 100644 --- a/spothole.py +++ b/spothole.py @@ -8,11 +8,11 @@ import sys from diskcache import Cache from core.cleanup import CleanupTimer -from data.solar_conditions import SolarConditions from core.config import config, SERVER_OWNER_CALLSIGN from core.constants import SOFTWARE_VERSION from core.lookup_helper import lookup_helper from core.status_reporter import StatusReporter +from data.solar_conditions import SolarConditions from server.webserver import WebServer # Globals diff --git a/spotproviders/dxcluster.py b/spotproviders/dxcluster.py index 8f19a0f..3cb4c37 100644 --- a/spotproviders/dxcluster.py +++ b/spotproviders/dxcluster.py @@ -72,7 +72,8 @@ class DXCluster(SpotProvider): match = self._spot_line_pattern.match(telnet_output.decode("latin-1")) if match: spot_time = datetime.strptime(match.group(5), "%H%MZ") - spot_datetime = datetime.combine(datetime.now(pytz.UTC).date(), spot_time.time(), tzinfo=pytz.UTC) + spot_datetime = datetime.combine(datetime.now(pytz.UTC).date(), spot_time.time(), + tzinfo=pytz.UTC) spot = Spot(source=self.name, dx_call=match.group(3), de_call=match.group(1), diff --git a/spotproviders/gma.py b/spotproviders/gma.py index 46b989d..2020871 100644 --- a/spotproviders/gma.py +++ b/spotproviders/gma.py @@ -38,10 +38,10 @@ class GMA(HTTPSpotProvider): time=datetime.strptime(source_spot["DATE"] + source_spot["TIME"], "%Y%m%d%H%M").replace( tzinfo=pytz.UTC).timestamp(), dx_latitude=float(source_spot["LAT"]) if ( - source_spot["LAT"] and source_spot["LAT"] != "") else None, + source_spot["LAT"] and source_spot["LAT"] != "") else None, # Seen GMA spots with no (or empty) lat/lon dx_longitude=float(source_spot["LON"]) if ( - source_spot["LON"] and source_spot["LON"] != "") else None) + source_spot["LON"] and source_spot["LON"] != "") else None) # GMA doesn't give what programme (SIG) the reference is for until we separately look it up. if "REF" in source_spot: diff --git a/spotproviders/rbn.py b/spotproviders/rbn.py index 830dfdd..038a8be 100644 --- a/spotproviders/rbn.py +++ b/spotproviders/rbn.py @@ -63,7 +63,8 @@ class RBN(SpotProvider): match = self._LINE_PATTERN.match(telnet_output.decode("latin-1")) if match: spot_time = datetime.strptime(match.group(5), "%H%MZ") - spot_datetime = datetime.combine(datetime.now(pytz.UTC).date(), spot_time.time(), tzinfo=pytz.UTC) + spot_datetime = datetime.combine(datetime.now(pytz.UTC).date(), spot_time.time(), + tzinfo=pytz.UTC) spot = Spot(source=self.name, dx_call=match.group(3), de_call=match.group(1), diff --git a/spotproviders/sota.py b/spotproviders/sota.py index ecddf92..4aff055 100644 --- a/spotproviders/sota.py +++ b/spotproviders/sota.py @@ -43,7 +43,7 @@ class SOTA(HTTPSpotProvider): dx_name=source_spot["activatorName"], de_call=source_spot["callsign"].upper(), freq=(float(source_spot["frequency"]) * 1000000) if ( - source_spot["frequency"] is not None) else None, + source_spot["frequency"] is not None) else None, # Seen SOTA spots with no frequency! mode=source_spot["mode"].upper(), comment=source_spot["comments"], diff --git a/spotproviders/tiles.py b/spotproviders/tiles.py index 406af8c..ae56faa 100644 --- a/spotproviders/tiles.py +++ b/spotproviders/tiles.py @@ -30,7 +30,8 @@ class Tiles(HTTPSpotProvider): sig="Tiles", # Tiles spots can include POTA & SOTA references, but ignore those on the basis that we will get them separately from the POTA/SOTA providers anyway. # Just take the grid reference itself as the single Tiles SIG reference. - sig_refs=[SIGRef(id=source_spot["maidenhead_grid"], sig="Tiles", name=source_spot["maidenhead_grid"])], + sig_refs=[SIGRef(id=source_spot["maidenhead_grid"], sig="Tiles", + name=source_spot["maidenhead_grid"])], time=datetime.fromisoformat(source_spot["created_at"].replace("Z", "+00:00")).timestamp(), dx_grid=source_spot["maidenhead_grid"], dx_latitude=source_spot["latitude"], @@ -41,10 +42,11 @@ class Tiles(HTTPSpotProvider): new_spots.append(spot) return new_spots + # Utility function to keep the first decimal point in a given string but remove any others. Used to parse Tiles' # strange frequency format where we can sometimes have e.g. "14.123.5". def strip_extra_decimal_points(s): parts = s.split('.', 1) if len(parts) == 1: return s - return parts[0] + '.' + parts[1].replace('.', '') \ No newline at end of file + return parts[0] + '.' + parts[1].replace('.', '') diff --git a/spotproviders/wota.py b/spotproviders/wota.py index 643a63e..bd903b0 100644 --- a/spotproviders/wota.py +++ b/spotproviders/wota.py @@ -49,7 +49,7 @@ class WOTA(HTTPSpotProvider): desc_split = source_spot.description.split(". ") freq_mode = desc_split[0].replace("Frequencies/modes:", "").strip() freq_mode_split = re.split(r'[\-\s]+', freq_mode) - freq_hz = float(freq_mode_split[0].replace("'",".")) * 1000000 + freq_hz = float(freq_mode_split[0].replace("'", ".")) * 1000000 mode = None if len(freq_mode_split) > 1: mode = freq_mode_split[1].upper() diff --git a/spotproviders/wwtota.py b/spotproviders/wwtota.py index 6934dd3..61d00bd 100644 --- a/spotproviders/wwtota.py +++ b/spotproviders/wwtota.py @@ -1,6 +1,5 @@ -from datetime import datetime - import json +from datetime import datetime from data.sig_ref import SIGRef from data.spot import Spot diff --git a/spotproviders/xota.py b/spotproviders/xota.py index 945f4f0..bd55eae 100644 --- a/spotproviders/xota.py +++ b/spotproviders/xota.py @@ -48,8 +48,9 @@ class XOTA(WebsocketSpotProvider): freq=float(source_spot["freq"]) * 1000, mode=source_spot["mode"].upper(), sig=self.SIG, - sig_refs=[SIGRef(id=ref_id, sig=self.SIG or "", url=source_spot["reference"]["website"], latitude=lat, - longitude=lon)], + sig_refs=[ + SIGRef(id=ref_id, sig=self.SIG or "", url=source_spot["reference"]["website"], latitude=lat, + longitude=lon)], time=datetime.now(pytz.UTC).timestamp(), dx_latitude=lat, dx_longitude=lon, diff --git a/templates/about.html b/templates/about.html index 89a06bc..f32b775 100644 --- a/templates/about.html +++ b/templates/about.html @@ -3,72 +3,204 @@
Spothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data.
-While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, Spothole differentiates itself by supporting a larger number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it.
-The API is deliberately well-defined with an OpenAPI specification and API documentation. The API delivers spots in a consistent format regardless of the data source, freeing developers from needing to know how each individual data source presents its data.
-Spothole itself is also open source, Public Domain licenced code that anyone can take and modify. The source code is here.
-The software was written by Ian Renton, MØTRT and other contributors. Full details are available in the README file.
+Spothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an + open JSON API as well as a website to browse the data.
+While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various + outdoor activity programmes for amateur radio, Spothole differentiates itself by supporting a larger number of + data sources, and by being "API first" rather than just providing a web front-end. This allows other software to + be built on top of it.
+The API is deliberately well-defined with an OpenAPI specification and API documentation. The API delivers spots in a consistent format regardless of the data + source, freeing developers from needing to know how each individual data source presents its data.
+Spothole itself is also open source, Public Domain licenced code that anyone can take and modify. The source code is here.
+The software was written by Ian Renton, MØTRT and other contributors. Full + details are available in the README + file.
This server is running Spothole version {{software_version}}.
There are a number of different ways to use Spothole, depending on what you want to do with it and your level of technical skill:
-There are a number of different ways to use Spothole, depending on what you want to do with it and your level of + technical skill:
+This is a tool for amateur ("ham") radio users. Many amateur radio operators like to make contacts with others who are doing something more interesting than sitting in their home "shack", such as people in rarely-seen countries, remote islands, or on mountaintops. Such operators are often "spotted", i.e. when someone speaks to them, they will put the details such as their operating frequency into an online system, to let others know where to find them. A DX Cluster is one type of those systems. Most outdoor radio awards programmes, such as "Parks on the Air" (POTA) have their own websites for posting spots.
-Spothole is an "aggregator" for those spots, so it checks lots of different services for data, and brings it all together in one place. So no matter what kinds of interesting spots you are looking for, you can find them here.
-As well as spots, it also provides a similar feed of "alerts". This is where amateur radio users who are going to interesting places soon will announce their intentions.
+This is a tool for amateur ("ham") radio users. Many amateur radio operators like to make contacts with others + who are doing something more interesting than sitting in their home "shack", such as people in rarely-seen + countries, remote islands, or on mountaintops. Such operators are often "spotted", i.e. when someone speaks to + them, they will put the details such as their operating frequency into an online system, to let others know + where to find them. A DX Cluster is one type of those systems. Most outdoor radio awards programmes, such as + "Parks on the Air" (POTA) have their own websites for posting spots.
+Spothole is an "aggregator" for those spots, so it checks lots of different services for data, and brings it all + together in one place. So no matter what kinds of interesting spots you are looking for, you can find them + here.
+As well as spots, it also provides a similar feed of "alerts". This is where amateur radio users who are going to + interesting places soon will announce their intentions.
In amateur radio terminology, the "DX" contact is the "interesting" one that is using the frequency shown and looking for callers. They might be on a remote island or just in a local park, but either way it's interesting enough that someone has "spotted" them. The callsign listed under "DE" is the person who entered the spot of the "DX" operator. "Modes" are the type of communication they are using. For example you might see "CW" which is Morse Code, or voice "modes" like SSB or FM, or more exotic "data" modes which are used for computer-to-computer communication.
+In amateur radio terminology, the "DX" contact is the "interesting" one that is using the frequency shown and + looking for callers. They might be on a remote island or just in a local park, but either way it's interesting + enough that someone has "spotted" them. The callsign listed under "DE" is the person who entered the spot of the + "DX" operator. "Modes" are the type of communication they are using. For example you might see "CW" which is + Morse Code, or voice "modes" like SSB or FM, or more exotic "data" modes which are used for computer-to-computer + communication.
Spothole can retrieve spots from: Telnet-based DX clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, SOTA, WWFF, GMA, WWBOTA, HEMA, Parks 'n' Peaks, ZLOTA, WOTA, LLOTA, WWTOTA, Tiles on the Air, the UK Packet Repeater Network, and any site based on the xOTA software by nischu.
-Spothole can retrieve alerts from: NG3K, POTA, SOTA, WWFF, Parks 'n' Peaks, WOTA and BOTA.
-Spothole can retrieve solar and propagation condition data from HamQSL, the NOAA Space Weather Prediction Center, the Lowell GIRO Data Center and prop.kc2g.com by KC2G.
-Spothole can also perform lookups for callsign data on behalf of the user from QRZ.com and HamQTH.
-Note that the server owner has not necessarily enabled all these data sources. In particular it is common to disable RBN, to avoid the server being swamped with FT8 traffic, and to disable APRS-IS and UK Packet Net so that the server only displays stations where there is likely to be an operator physically present for a QSO.
-Between the various data sources, the following Special Interest Groups (SIGs) are supported: Parks on the Air (POTA), Summits on the Air (SOTA), Worldwide Flora & Fauna (WWFF), Global Mountain Activity (GMA), Worldwide Bunkers on the Air (WWBOTA), HuMPs Excluding Marilyns Award (HEMA), Islands on the Air (IOTA), Mills on the Air (MOTA), the Amateur Radio Lighthouse Socirty (ARLHS), International Lighthouse Lightship Weekend (ILLW), Silos on the Air (SIOTA), World Castles Award (WCA), New Zealand on the Air (ZLOTA), Keith Roget Memorial National Parks Award (KRMNPA), Wainwrights on the Air (WOTA), Beaches on the Air (BOTA), Lagos y Lagunas On the Air (LLOTA), Towers on the Air (WWTOTA), Tiles on the Air, Worked All Britain (WAB), Worked All Ireland (WAI), and Toilets on the Air (TOTA).
-As of the time of writing in November 2025, I think Spothole captures essentially all outdoor radio programmes that have a defined reference list, and almost certainly those that have a spotting/alerting API. If you know of one I've missed, please let me know!
+Spothole can retrieve spots from: Telnet-based DX clusters, the + Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, + SOTA, WWFF, GMA, WWBOTA, HEMA, Parks 'n' Peaks, ZLOTA, WOTA, LLOTA, WWTOTA, Tiles on the Air, the UK + Packet Repeater Network, and any site based on the xOTA + software by nischu.
+Spothole can retrieve alerts from: NG3K, POTA, + SOTA, WWFF, Parks 'n' Peaks, WOTA and + BOTA.
+Spothole can retrieve solar and propagation condition data from HamQSL, the + NOAA Space Weather Prediction Center, the Lowell GIRO Data Center and prop.kc2g.com + by KC2G.
+Spothole can also perform lookups for callsign data on behalf of the user from QRZ.com and HamQTH.
+Note that the server owner has not necessarily enabled all these data sources. In particular it is common to + disable RBN, to avoid the server being swamped with FT8 traffic, and to disable APRS-IS and UK Packet Net so + that the server only displays stations where there is likely to be an operator physically present for a QSO.
+Between the various data sources, the following Special Interest Groups (SIGs) are supported: Parks on the Air + (POTA), Summits on the Air (SOTA), Worldwide Flora & Fauna (WWFF), Global Mountain Activity (GMA), Worldwide + Bunkers on the Air (WWBOTA), HuMPs Excluding Marilyns Award (HEMA), Islands on the Air (IOTA), Mills on the Air + (MOTA), the Amateur Radio Lighthouse Socirty (ARLHS), International Lighthouse Lightship Weekend (ILLW), Silos + on the Air (SIOTA), World Castles Award (WCA), New Zealand on the Air (ZLOTA), Keith Roget Memorial National + Parks Award (KRMNPA), Wainwrights on the Air (WOTA), Beaches on the Air (BOTA), Lagos y Lagunas On the Air + (LLOTA), Towers on the Air (WWTOTA), Tiles on the Air, Worked All Britain (WAB), Worked All Ireland (WAI), and + Toilets on the Air (TOTA).
+As of the time of writing in November 2025, I think Spothole captures essentially all outdoor radio programmes + that have a defined reference list, and almost certainly those that have a spotting/alerting API. If you know of + one I've missed, please let me know!
Mostly, but not quite. While POTA spots generally come from the POTA source and so on, there are a few exceptions:
-Spothole's web interface exists not just for the end user, but also as a reference implementation for the API, so I have chosen to demonstrate both methods of filtering.
+Mostly, but not quite. While POTA spots generally come from the POTA source and so on, there are a few + exceptions:
+Spothole's web interface exists not just for the end user, but also as a reference implementation for the API, so + I have chosen to demonstrate both methods of filtering.
It's probably not? But it's nice to have choice.
I think it's got three key advantages over those sites:
-Spothole is a Progressive Web App, which means you can install it on an Android or iOS device by opening the site in Chrome or Safari respectively, and clicking "Install" on the pop-up panel. It'll only prompt you once, so if you dismiss the prompt and change your mind, you'll find an Install / Add to Home Screen option on your browser's menu.
-Installing Spothole on your phone is completely optional, the website works exactly the same way as the "app" does.
+Spothole is a Progressive Web App, which means you can install it on an Android or iOS device by opening the site + in Chrome or Safari respectively, and clicking "Install" on the pop-up panel. It'll only prompt you once, so if + you dismiss the prompt and change your mind, you'll find an Install / Add to Home Screen option on your + browser's menu.
+Installing Spothole on your phone is completely optional, the website works exactly the same way as the "app" + does.
To avoid putting too much load on the various servers that Spothole connects to, the Spothole server only polls them once every two minutes for spots, and once every 30 minutes for alerts. (Some sources, such as DX clusters, RBN, APRS-IS and WWBOTA use a non-polling mechanism, and their updates will therefore arrive more quickly.) Then if you are using the web interface, that has its own rate at which it fetches the data from Spothole. This is instant for the main spots list, with new spots appearing immediately at the top of the list, while the map and bands displays update once a minute, and the alerts display updates once every 5 minutes. So you could be waiting around three minutes to see a newly added spot, or 40 minutes to see a newly added alert.
+To avoid putting too much load on the various servers that Spothole connects to, the Spothole server only polls + them once every two minutes for spots, and once every 30 minutes for alerts. (Some sources, such as DX clusters, + RBN, APRS-IS and WWBOTA use a non-polling mechanism, and their updates will therefore arrive more quickly.) Then + if you are using the web interface, that has its own rate at which it fetches the data from Spothole. This is + instant for the main spots list, with new spots appearing immediately at the top of the list, while the map and + bands displays update once a minute, and the alerts display updates once every 5 minutes. So you could be + waiting around three minutes to see a newly added spot, or 40 minutes to see a newly added alert.
Spothole's source code is licenced under the Public Domain. You can write a Spothole client, run your own server, modify it however you like, you can claim you wrote it and charge people £1000 for a copy, I don't really mind. (Please don't do the last one. But if you're using my code for something cool, it would be nice to hear from you!)
+Spothole's source code is licenced under the Public Domain. You can write a Spothole client, run your own server, + modify it however you like, you can claim you wrote it and charge people £1000 for a copy, I don't really mind. + (Please don't do the last one. But if you're using my code for something cool, it would be nice to hear from + you!)
Please note that the data coming out of Spothole is only as good as the data going in. People mis-hear and make typos when spotting callsigns all the time. There are also plenty of cases where Spothole's data, particularly location data, may be inaccurate. For example, there are POTA parks that span multiple US states, countries that span multiple CQ zones, portable operators with no requirement to sign /P, etc. If you are doing something where accuracy is important, such as contesting, you should not rely on Spothole's data to fill in any gaps in your log.
+Please note that the data coming out of Spothole is only as good as the data going in. People mis-hear and make + typos when spotting callsigns all the time. There are also plenty of cases where Spothole's data, particularly + location data, may be inaccurate. For example, there are POTA parks that span multiple US states, countries that + span multiple CQ zones, portable operators with no requirement to sign /P, etc. If you are doing something where + accuracy is important, such as contesting, you should not rely on Spothole's data to fill in any gaps in your + log.
Spothole collects no data about you on a permanent basis. All spots and alerts are "timed out" and deleted from the system after a set interval, which by default is one hour for spots and one week for alerts.
-Settings you select from Spothole's menus are sent to the server, in order to provide the data with the requested filters. They are also stored in your browser's local storage, so that your preferences are remembered between sessions.
-The data you provide can optionally include your login credentials for QRZ.com and HamQTH. You can provide these in the "Data" menu of most pages. If you do, Spothole will augment the data it produces with lookups from these services, which can for example provide more accurate markers on the map tab, and operator names when you mouse over a DX callsign. Spothole will still work fine if you don't provide these. The values you enter are sent to Spothole via HTTPS so are protected in transit, though of course you do have to trust Spothole with this sensitive data in order to use this feature.
+Spothole collects no data about you on a permanent basis. All spots and alerts are "timed out" and deleted from + the system after a set interval, which by default is one hour for spots and one week for alerts.
+Settings you select from Spothole's menus are sent to the server, in order to provide the data with the requested + filters. They are also stored in your browser's local storage, so that your preferences are remembered between + sessions.
+The data you provide can optionally include your login credentials for QRZ.com and HamQTH. You can provide these + in the "Data" menu of most pages. If you do, Spothole will augment the data it produces with lookups from these + services, which can for example provide more accurate markers on the map tab, and operator names when you mouse + over a DX callsign. Spothole will still work fine if you don't provide these. The values you enter are sent to + Spothole via HTTPS so are protected in transit, though of course you do have to trust Spothole with this + sensitive data in order to use this feature.
Spothole uses no trackers, no ads, and no cookies.
{% if len(web_ui_options["support-button-html"]) > 0 %} -Caveat: The owner of this server has chosen to inject their own content into the "spots" page. This is designed for a "donate" or "support this server" button. The functionality of this injected content is the responsibility of the server owner, rather than the Spothole software.
+Caveat: The owner of this server has chosen to inject their own content into the "spots" page. + This is designed for a "donate" or "support this server" button. The functionality of this injected content is + the responsibility of the server owner, rather than the Spothole software.
{% end %} -Spothole is open source, so you can audit the code if you like.
+Spothole is open source, so you can audit the code if you + like.
This project would not have been possible without those volunteers who have taken it upon themselves to run DX clusters, xOTA programmes, DXpedition lists, callsign lookup databases, solar conditions and propagation modelling software, and other online tools on which Spothole's data is based. The vast majority of these are not profit-seeking and are made purely for the love of the hobby and to help others in the community. Spothole is standing on the shoulders of giants, who deserve a huge amount of thanks for all the work they put in.
-Spothole is also dependent on a number of Python libraries, in particular pyhamtools, and many JavaScript libraries, as well as the Font Awesome icon set and flag icons from the Noto Color Emoji set, and MIT-licenced GeoJSON files for CQ and ITU zones from HA8TKS.
-This software is dedicated to the memory of Tom G1PJB, SK, a friend and colleague who sadly passed away around the time I started writing it in Autumn 2025. I was looking forward to showing it to you when it was done.
+This project would not have been possible without those volunteers who have taken it upon themselves to run DX + clusters, xOTA programmes, DXpedition lists, callsign lookup databases, solar conditions and propagation + modelling software, and other online tools on which Spothole's data is based. The vast majority of these are not + profit-seeking and are made purely for the love of the hobby and to help others in the community. Spothole is + standing on the shoulders of giants, who deserve a huge amount of thanks for all the work they put in.
+Spothole is also dependent on a number of Python libraries, in particular pyhamtools, and many JavaScript + libraries, as well as the Font Awesome icon set and flag icons from the Noto Color Emoji set, and MIT-licenced + GeoJSON files for CQ and ITU zones from HA8TKS.
+This software is dedicated to the memory of Tom G1PJB, SK, a friend and colleague who sadly passed away around + the time I started writing it in Autumn 2025. I was looking forward to showing it to you when it was done.
This server is running Spothole v{{software_version}}, and is operated by {{server_owner_callsign}}.
-The web UI is not available on this instance because the server is running in API-only mode, intended for use by client software rather than visitors to the website. See the API documentation for details of how client software can interact with the server.
-Please see the README for details of what Spothole is and how you can run it for yourself.
+This server is running Spothole v{{software_version}}, and is + operated by {{server_owner_callsign}}.
+The web UI is not available on this instance because the server is running in + API-only mode, intended for use by client software rather than visitors to the website. See the + API documentation for details of how client software can interact with + the server.
+Please see the README for details of what Spothole + is and how you can run it for yourself.
-
- +
Show up to
-
Show up to
-
Last
- Table Columns
Use
-