Updated developer's notes

Developers-notes.md

@@ -1,38 +1,103 @@
 <h1>SDRangel Developer's notes</h1>
 
-<h2>Global code structure and flow</h2>
+<h2>General flow</h2>
 
-![SDRangel code map](./images/developer/SDRangelFlow.png)
+A consistent flow of samples is grouped into a "device set" that combines one device plugin and many channel plugins sharing the same baseband. The following figure depicts the flow of samples taking place in one device set.
 
-The existing receiving flow is represented with green boxes. The future Tx flow with red boxes. Some classes and file paths in the Rx part were renamed to avoid collision with future Tx names in this case the old name appears below the present name in italics.
+![SDRangel general flow](./images/developer/SDRangelFlow.png)
+
+The critical parts of plugins run on their own thread. FIFOs at their boundaries ensure the independence of threads and thus effective multi-threading. The "DSP device engine" also runs on its own thread.
+
+The DSP Device Engine is specialized for each category of device sets:
+  - Source device sets (Rx): the `DSPDeviceSourceEngine` dispatches the baseband samples to the attached channels
+  - Sink device sets (Tx): `DSPDeviceSinkEngine` combines the samples extracted from the attached channels into the baseband
+  - MIMO device sets (v5 only): `DSPDeviceMIMOEngine` dispatches and combines samples coming from attached channels into a multi-stream baseband
+
+The GUI and management modules of the plugins run on the main thread. Communication between threads is done via message queues.
+
+<h2>Code structure and flow</h2>
+
+Legend of figures:
+
+  - Black arrows follow (loosely) the UML standard
+    - Hollow arrows represent inheritance oriented towards the parent class
+    - Filled diamond arrows represent composition oriented towards the composed class
+    - Hollow diamond arrows represent aggregation (reference) oriented towards the referenced class
+  - Green arrows represent the direction of the Rx stream
+  - Red arrows represent the direction of the Tx stream
+  - Blue arrows represent the direction of the MIMO stream. The MIMO stream is a simultaneous (synchronous or asynchronous) stream composed of any number of input or output streams
 
 <h3>Rx path</h3>
 
+Needs refactoring to match the Tx design described next
+
+![SDRangel Rx flow](./images/developer/SDRangelFlowRx.png)
+
  - The I/Q samples are collected from a physical device or UDP flow with a device plugin that derives from the `DeviceSampleSource` class.
- - These I/Q samples are downsampled by a factor set in the device plugin GUI and fed into a `BasebandSampleSink`
- - The `DownChannelizer`class downsamples further down (or not) the baseband I/Q samples depending on the requirements of the Rx plugin. It cascades the downsampling from the center, left or right half of the baseband in order to fit the Rx plugin bandwidth. It contains the NCO to adjust to the Rx plugin center frequency.
- - The special `FileSource` plugin reads a file that was recorded using the `FileRecord` class directly into the baseband as there is no downsampling from the sample rate at which the file was recorded.
- - The baseband I/Q samples can be recorded to a file using the `FileRecord` class
+ - These I/Q samples are downsampled in a bank of half-band decimators to reach the baseband sample rate and fed into a `BasebandSampleSink` by the `DSPDeviceSOourceEngine`
+ - The `DownChannelizer`class downsamples further down (or not) the baseband I/Q samples depending on the requirements of the Rx plugin. It contains a bank of half-band decimators that cascade the downsampling from the center, left or right half of the input band at each stage in order to fit the Rx plugin bandwidth.
+ - The `xxx...` class contains a NCO to adjust to the Rx plugin center frequency and a rational downsampler to adjust the sample rate at the output of the `DownChannelizer` (channel sample rate) to fit the exact rate required.
+ - The `FileRecord` class records the baseband stream.
+ - The baseband stream is also sent to the spectrum display.
+
+<h3>Tx path</h3>
+
+This is when issue #438 is implemented
+
+![SDRangel Tx flow](./images/developer/SDRangelFlowTx.png)
+
+ - The I/Q samples are sent to a physical device or UDP flow with a device plugin that derives from the `DeviceSampleSink` class.
+ - These I/Q samples have been upsampled in a bank of half-band interpolators from the baseband sample rate and have been taken from a `BasebandSampleSource` using the `DSPDeviceSinkEngine`
+ - The `Xxx` class delegates the generation of samples to the `XxxBaseband` class
+ - The `XxxBaseband` class realizes the baseband processor described in the device set general flow. It has an `UpChannelizer`class to interpolate the samples generated by the `XxxSource` class up to the baseband sample rate. The `UpChannelizer` class contains a bank of half-band interpolators that cascade the upsampling to the center, left or right half of the output band at each stage in order to fit the Tx source bandwidth.
+ - The `XxxSource` class realizes the Sink processor described in the device set general flow. It contains a NCO to adjust to the Tx plugin center frequency and a rational upsampler to adjust the sample rate at the input of the `UpChannelizer` (channel sample rate) to fit the exact rate required.
+ - The generated baseband stream is also sent to the spectrum display.
+
+<h3>MIMO path (v5 only)</h3>
+
+Work in progress...
+
+![SDRangel MIMO flow](./images/developer/SDRangelFlowMIMO.png)
 
 <h3>Device sample source plugins</h3>
 
 At present the following plugins are available:
 
   - `AirspyXxx` classes in `plugins/samplesource/airspy`: Interface with Airspy devices
-  - `BladeRFXxx` classes in `plugins/samplesource/bladerf`: Interface with BladeRF devices
-  - `BladeRFXxx` classes in `plugins/samplesource/bladerf`: Interface with BladeRF devices
+  - `AirspyHFXxx` classes in `plugins/samplesource/airspyhf`: Interface with Airspy HF devices
+  - `BladeRF1InputXxx` classes in `plugins/samplesource/bladerf1input`: Interface with BladeRF v1 devices
+  - `BladeRF2InputXxx` classes in `plugins/samplesource/bladerf2input`: Interface with BladeRF v2 (or micro) devices
   - `FCDProXxx` classes in `plugins/samplesource/fcdpro`: Interface with Funcube Pro devices
   - `FCDProPlusXxx` classes in `plugins/samplesource/fcdproplus`: Interface with Funcube Pro+ devices
-  - `HackRFXxx` classes in `plugins/samplesource/hackrf`: Interface with HackRF devices
+  - `HackRFInputXxx` classes in `plugins/samplesource/hackrfinput`: Interface with HackRF devices
+  - `KiwiSDRxxx` classes in `plugins/samplesource/kiwisdr`: Interface with remote Kiwi SDR devices
+  - `LimeSDRInputxxx` classes in `plugins/samplesource/limesdrinput`: Interface with Lime SDR devices
+  - `Perseusxxx` classes in `plugins/samplesource/perseus`: Interface with Perseus devices
+  - `PlutoSDRInputxxx` classes in `plugins/samplesource/plutosdr`: Interface with Pluto SDR devices
   - `RTLSDRXxx` classes in `plugins/samplesource/rtlsdr`: Interface with RTL-SDR devices
+  - `SDRPlayXxx` classes in `plugins/samplesource/sdrplay`: Interface with SDRplay RSP1 devices
+  - `XTRXInputXxx` classes in `plugins/samplesource/xtrxinput`: Interface with XTRX devices
   - `RemoteInput` class in `plugins/samplesource/remoteinput`: Special interface collecting I/Q samples from an UDP flow sent by a remote instance of SDRangel using Remote sink channel.
-  - `FileSource` classes in `plugins/samplesource/filesource`: Special interface reading I/Q samples from a file directly into the baseband skipping the downsampling block
+  - `LocalInput` class in `plugins/samplesource/localinput`: Special interface collecting I/Q samples from another device set having the `LocalInput` as its device.
+  - `FileInput` classes in `plugins/samplesource/fileinput`: Special interface reading I/Q samples from a file directly into the baseband skipping the downsampling block
+  - `SoapySDRInput` classes in `plugins/samplesource/soapysdrinput`: Special interface working with devices interfaced with SoapySDR library
+  - `TestSource` classes in `plugins/samplesource/testsource`: Special device to generate samples internally for mocking a device source
 
 <h3>Device sample sink plugins</h3>
 
 At present the following plugins are available:
 
+  - `BladeRF1OutputXxx` classes in `plugins/samplesource/bladerf1output`: Interface with BladeRF v1 devices
+  - `BladeRF2OutputXxx` classes in `plugins/samplesource/bladerf2output`: Interface with BladeRF v2 (or micro) devices
   - `FileSink` classes in `plugins/samplesink/filesink`: Special interface writing baseband I/Q samples to a file skipping the final upsampling block
+  - `HackRFOutputXxx` classes in `plugins/samplesource/hackrfoutput`: Interface with HackRF devices
+  - `LimeSDROutputxxx` classes in `plugins/samplesource/limesdroutput`: Interface with Lime SDR devices
+  - `LocalOutput` class in `plugins/samplesource/localoutput`: Special interface getting I/Q samples from another device set having the `LocalOutput` as its device.
+  - `PlutoSDROutputxxx` classes in `plugins/samplesource/plutosdroutput`: Interface with Pluto SDR devices
+  - `RemoteOutput` class in `plugins/samplesource/remoteoutput`: Special interface getting I/Q samples from an UDP flow sent by a remote instance of SDRangel using Remote source channel.
+  - `SoapySDROutput` classes in `plugins/samplesource/soapysdroutput`: Special interface working with devices interfaced with SoapySDR library
+  - `TestSink` classes in `plugins/samplesource/testsink`: Special device to direct generated samples to a spectrum display
+  - `XTRXOutputXxx` classes in `plugins/samplesource/xtrxoutput`: Interface with XTRX devices
 
 <h3>Channel receiver (Rx) plugins</h3>
 
@@ -40,32 +105,100 @@ At present the following plugins are available:
 
   - `ChannelAnalyzerXxx` classes in `plugins/channelrx/chanalyzer`: Signal analysis tool pretty much like a DSA/DSO signal analyzer like the venerable HP 4406A (although still far from it!)
   - `AMDemodXxx` classes in `plugins/channelrx/demodam`: AM demodulator with audio output
+  - `ATVDemodXxx` classes in `plugins/channelrx/demodatv`: Demodulator and visualization of analog TV signals mostly those used in Amateur TeleVision (ATV)
   - `BFMDemodXxx` classes in `plugins/channelrx/demodbfm`: Broadcast FM demodulator with audio mono/stereo output and RDS
+  - `DATVDemodXxx` classes in `plugins/channelrx/demoddatv`: Demodulator and visualization of digital TV signals mostly those used in Digital Amateur TeleVision (DATV)
   - `DSDDemodXxx` classes in `plugins/channelrx/demoddsd`: Digital Speech demodulator/decoder built on top of the [DSDcc library](https://github.com/f4exb/dsdcc). Produces audio output and some communication data from various digital voice standards: DMR, dPMR, D-Star, Yaesu System Fusion (YSF).
-  - `LoraDemodXxx` classes in `plugins/channelrx/demodlora`: Decodes [LoRa](http://www.semtech.com/images/datasheet/an1200.21.pdf) transmissions. This is legacy code that is not very well maintained so it may or may not work.
+  - `FreeDVDemodXxx` classes in `plugins/channelrx/demodfreedv`: Demodulator and decoder of signals using the Free DV digital modes
+  - `LoraDemodXxx` classes in `plugins/channelrx/demodlora`: Decodes [LoRa](http://www.semtech.com/images/datasheet/an1200.21.pdf) transmissions. This is legacy code that is not maintained so it does not work most probably.
   - `NFMDemodXxx` classes in `plugins/channelrx/demodnfm`: Narrowband FM demodulator with audio output.
   - `SSBDemodXxx` classes in `plugins/channelrx/demodssb`: SSB/DSB/CW demodulator with audio output.
-  - `WFMDemodXxx` classes in `plugins/channelrx/demodwfm`: Wideband FM demodulator with audio output. This is a basic demodulator.
+  - `WFMDemodXxx` classes in `plugins/channelrx/demodwfm`: Wideband FM demodulator with audio output. This is a basic demodulator not for broadcast reception.
+  - `FreqTrackerXxx` classes in `plugins/channelrx/freqtracker`: Plugin that follows the frequency of the input signal using a Frequency Locked Loop or a Phase Locked Loop for signals with phase modulation (BPSK, QPSK, ...). To be used in cooperation with an external script to control the frequency of other plugins via REST API.
+  - `LocalSinkXxx` classes in `plugins/channelrx/localsink`: Sends channel I/Q to another source device set that has a `LocalInput` source for its device.
+  - `RemoteSinkXxx` classes in  `plugins/channelrx/remotesink`: Sends channel I/Q samples to another instance of SDRangel via UDP. The `RemoteInput` plugin is used to receive the samples.
   - `UDPSinkXxx` classes in `plugins/channelrx/udpsink`: Sends channel I/Q or FM demodulated samples via UDP
 
 <h3>Channel transmitter (Tx) plugins</h3>
 
 At present the following plugins are available:
 
-  - `AMMmodXxx` classes in `plugins/channeltx/modam`: AM modulator with simple sine
+  - `FileSourceXxx` classes in `plugins/channeltx/filesource`: takes I/Q samples from a file saved in `.sdriq` format
+  - `LocalSourceXxx` classes in `plugins/channeltx/localource`: takes I/Q samples from another sink device set that has a `LocalOutput` sink for its device
+  - `AMMmodXxx` classes in `plugins/channeltx/modam`: AM modulator
+  - `ATVModXxx` classes in `plugins/channeltx/modatv`: Modulator for analog TV signals mostly those used in Amateur TeleVision (ATV)
+  - `FreeDVModXxx` classes in `plugins/channeltx/modfreedv`: Coder and modulator for signals using the Free DV digital modes
+  - `NFMModXxx` classes in `plugins/channeltx/modnfm`: Narrowband FM modulator
+  - `SSBModXxx` classes in `plugins/channeltx/modssb`: Single Side Band modulator. Can be used for CW.
+  - `WFMModXxx` classes in `plugins/channeltx/modwfm`: Wideband FM modulator (not broadcast quality)
+  - `RemoteSourceXxx` classes in `plugins/channeltx/remotesource`: takes I/Q samples from another SDRangel instance using the `RemoteOutput` plugin to send samples over UDP.
+  - `UDPSourceXxx` classes in `plugins/channeltx/udpsoutce`: takes channel I/Q or FM demodulated samples sent via UDP as raw samples (not SDRangel UDP format)
 
 <h2>Source tree structure</h2>
 
-At the first subdirectory level `indclude` and `sdrbase` contain the common core components include and source files respectively. They are further broken down in subdirectories corresponding to a specific area:
+This describes only the main components
 
+<h3>httpserver</h3>
+
+The HTTP server used for REST API communication and internal web server
+
+<h3>logging</h3>
+
+Classes used to log messages to console or file
+
+<h3>qrtplib</h3>
+
+Rewrite of the [jrtplib](https://github.com/j0r1/JRTPLIB) library for the Qt environment. It handles RTP over UDP communication.
+
+<h3>rescuesdriq</h3>
+
+Script written in Go language to process or restore `.sdriq` files so that their header is compatible with the present version of SDRangel
+
+<h3>scriptsapi</h3>
+
+Collection of Python scripts that use the REST API of SDRangel to perform common tasks
+
+<h3>sdrbase</h3>
+
+The `sdrbase` subdirectory contain the common core components. It is further broken down in subdirectories corresponding to a specific area:
+
+  - `ambe` contains classes to interface with AMBE digital voice processors
   - `audio` contains the interface with the audio device(s)
+  - `channel` contains the channel API interface and related classes
+  - `commands` classes to deal with the commands
+  - `device` contains the device API interface and related classes
   - `dsp` contains the common blocks for Digital Signal Processing like filters, scope and spectrum analyzer internals
-  - `gui` contains the common Graphical User Interface components like the scope and spectrum analyzer controls and display
   - `plugin` contains the common blocks for managing plugins
+  - `resources` contains Qt resources including web server static files
   - `settings` contains components to manage presets and preferences
   - `util` contains common utilities such as the message queue
+  - `webapi` contains common classes to handle web REST API
 
-The `plugins` subdirectory contains the associated plugins used to manage devices and channel components. Naming convention of various items depend on the usage and Rx (reception side) or Tx (transmission side) affinity. Transmission side is a work in progress.
+<h3>sdrgui</h3>
+
+The `sdrbase` subdirectory contain the common GUI components. It is further broken down in subdirectories corresponding to a specific area:
+
+  - `device` contains the UI parts of the Device Sets
+  - `dsp` contains DSP related GUI specific classes including the scope and spectrum interfaces
+  - `gui` contains the GUI core components
+  - `resources` contains Qt resources with GUI artifacts (icons...)
+  - `soapygui` contains GUI components specific to the SoapySDR plugins
+  - `webapi` contains GUI related classes related to the web REST API
+
+This is also where the `MainWindow` class resides
+
+<h3>sdrsrv</h3>
+
+The `sdrsrv` subdirectory contains classes specific to the server variant of
+SDRangel
+
+<h3>swagger</h3>
+
+The `swagger` sudirectory contains classes and resources related to the REST API interface of SDRangel. As the name suggests the REST API is designed using Swagger.
+
+<h3>plugins</h3>
+
+The `plugins` subdirectory contains the associated plugins used to manage devices and channel components. Naming convention of various items depend on the usage and Rx (reception side) or Tx (transmission side) affinity. MIMO is a work in progress and not described here yet.
 
   - Receiver functions (Rx):
     - `samplesource`: Device managers:
@@ -75,52 +208,60 @@ The `plugins` subdirectory contains the associated plugins used to manage device
         - `xxxplugin.h/cpp` : Plugin interface
         - `xxxsettings.h/cpp` : Configuration manager
         - `xxxthread.h/cpp` : Reading samples
-        - `xxx.pro` : Qt .pro file for Windows/Android build
+        - `xxxwebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
     - `channelrx`: Channel handlers:
       - `demodxxx` : Demodulator internal handler (e.g xxx = demodam)
         - `xxxdemod.h/cpp` : Demodulator core
         - `xxxdemodgui.h/cpp` : Demodulator GUI
         - `xxxdemodplugin.h/cpp` : Plugin interface
-        - `demodxxx.pro` : Qt .pro file for Windows/Android build
+        - `xxxdemodwebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
       - `xxxanalyzer` : Analyzer internal handler (e.g xxx = channel)
         - `xxxanalyzer.h/cpp` : Analyzer core
         - `xxxanalyzergui.h/cpp` : Analyzer GUI
         - `xxxanalyzerplugin.h/cpp` : Analyzer plugin manager
-        - `xxxanalyzer.pro` : Qt .pro file for Windows/Android build
+        - `xxxanalyzerwebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
       - `xxxsink` : Interface to the outside (e.g xxx = udp):
         - `xxxsink.h/cpp` : Interface core
         - `xxxsinkgui.h/cpp` : Interface GUI
         - `xxxsinkplugin/h/cpp` : Interface plugin manager
-        - `xxxsink.pro` : Qt .pro file for Windows/Android build
+        - `xxxwebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
+      - `xxx` : Not conventinally named module (ex: freqtracker):
+        - `xxx.h/cpp` : Module manager
+        - `xxxgui.h/cpp` : Module GUI
+        - `xxxplugin/h/cpp` : Module plugin manager
+        - `xxxwebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
 
   - Transmitter functions (Tx):
     - `samplesink`: Device managers:
-      - `xxx` : Device manager (e.g. xxx = bladerf)
+      - `xxx` : Device manager (e.g. xxx = bladerf1)
         - `xxxsinkoutput.h/cpp` : Device interface
         - `xxxsinkgui.h/cpp` : GUI
         - `xxxsinkplugin.h/cpp` : Plugin interface
         - `xxxsinksettings.h/cpp` : Configuration manager
         - `xxxsinkthread.h/cpp` : Writing samples
-        - `xxxsink.pro` : Qt .pro file for Windows/Android build
+        - `xxxsinkwebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
     - `channeltx`: Channel handlers:
       - `modxxx` : Modulator internal handler (e.g xxx = modam)
-        - `xxxmod.h/cpp` : Modulator core
+        - `xxxmod.h/cpp` : Modulator manager
+        - `xxxmodbaseband.h/cpp`: Baseband generator
+        - `xxxmodsource.h/cpp`: Source generator
         - `xxxmodgui.h/cpp` : Modulator GUI
         - `xxxmodplugin.h/cpp` : Plugin interface
-        - `modxxx.pro` : Qt .pro file for Windows/Android build
-      - `xxxgenerator` : Generator internal handler (e.g xxx = channel)
-        - `xxxgenerator.h/cpp` : Generator core
-        - `xxxgeneratorgui.h/cpp` : Generator GUI
-        - `xxxgeneratorplugin.h/cpp` : Generator plugin manager
-        - `xxxgenerator.pro` : Qt .pro file for Windows/Android build
+        - `xxxmodsettings.h/cpp` : Configuration manager
+        - `xxxmodwebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
       - `xxxsource` : Interface to the outside (e.g xxx = udp):
-        - `xxxsource.h/cpp` : Interface core
+        - `xxxsource.h/cpp` : Interface manager
+        - `xxxsourcebaseband.h/cpp`: Baseband generator
+        - `xxxsourcesource.h/cpp`: Source generator
         - `xxxsourcegui.h/cpp` : Interface GUI
         - `xxxsourceplugin/h/cpp` : Interface plugin manager
-        - `xxxsource.pro` : Qt .pro file for Windows/Android build
+        - `xxxsourcesettings.h/cpp` : Configuration manager
+        - `xxxsourcewebapiadapter.h/cpp`: Detached web API functions to handle settings via REST API
 
 <h2>Device interface and GUI lifecycle</h2>
 
+To be reviewed ...
+
 <h3>Overview</h3>
 
 Since version 3.4.0 the opening and closing of the physical device has been split off the start and stop methods of the device interface. The opening and closing of the physical device is handled in private methods called from the constructor and destructor of the device interface respectively.