designing arf, an sdr iq encoding format đ¶
hz.tools will be tagged
#hztools.đ¶ Want to jump right to the draft? I'll be maintaining ARF going forward at /draft-tagliamonte-arf-00.txt.
Itâs true â processing data from software defined radios can be a bit complex đđđ â which tends to keep all but the most grizzled experts and bravest souls from playing with it. While I wouldnât describe myself as either, I will say that Iâve stuck with it for longer than most would have expected of me. One of the biggest takeaways I have from my adventures with software defined radio is that thereâs a lot of cool crossover opportunity between RF and nearly every other field of engineering.
Fairly early on, I decided on a very light metadata scheme to track SDR captures, called rfcap. rfcap has withstood my test of time, and I can go back to even my earliest captures and still make sense of what they are â IQ format, capture frequencies, sample rates, etc. A huge part of this was the simplicity of the scheme (fixed-lengh header, byte-aligned to supported capture formats), which made it roughly as easy to work with as a raw file of IQ samples.
However, rfcap has a number of downsides. Itâs only a single, fixed-length header. If the frequency of operation changed during the capture, that change is not represented in the capture information. Itâs not possible to easily represent mulit-channel coherent IQ streams, and additional metadata is condemned to adjacent text files.
ARF (Archive of RF)
A few years ago, I needed to finally solve some of these shortcomings and tried to see if a new format would stick. I sat down and wrote out my design goals before I started figuring out what it looked like.
First, whatever I come up with must be capable of being streamed and processed while being streamed. This includes streaming across the network or merely written to disk as itâs being created. No post-processing required. This is mostly an artifact of how Iâve built all my tools and how I intereact with my SDRs. I use them extensively over the network (both locally, as well as remotely by friends across my wider lan). This decision sometimes even prompts me to do some crazy things from time to time.
I need actual, real support for multiple IQ channels from my multi-channel SDRs (Ettus, Kerberos/Kracken SDR, etc) for playing with things like beamforming. My new format must be capable of storing multiple streams in a single capture file, rather than a pile of files in a directory (and hope theyâre aligned).
Finally, metadata must be capable of being stored in-band. The initial set of
metadata I needed to formalize in-stream were Frequency Changes and
Discontinuities. Since then, ARF has grown a few more.
After getting all that down, I opted to start at what I thought the simplest container would look like, TLV (tag-length-value) encoded packets. This is a fairly well trodden path, and used by a bunch of existing protocols we all know and love. Each ARF file (or stream) was a set of encoded âpacketsâ (sometimes called data units in other specs). This means that unknown packet types may be skipped (since the length is included) and additional data can be added after the existing fields without breaking existing decoders.
Unlike a âtraditionalâ TLV structure, I opted to add âflagsâ to the top-level packet. This gives me a bit of wiggle room down the line, and gives me a feature that I like from ASN.1 â a âcriticalâ bit. The critical bit indicates that the packet must be understood fully by implementers, which allows future backward incompatible changes by marking a new packet type as critical. This would only really be done if something meaningfully changed the interpretation of the backwards compatible data to follow.
| Flag | Description |
| 0x01 | Critical (tag must be understood) |
Within each Packet is a tag field. This tag indicates how the contents of the
value field should be interpreted.
| Tag ID | Description |
| 0x01 | Header |
| 0x02 | Stream Header |
| 0x03 | Samples |
| 0x04 | Frequency Change |
| 0x05 | Timing |
| 0x06 | Discontinuity |
| 0x07 | Location |
| 0xFE | Vendor Extension |
In order to help with checking the basic parsing and encoding of this format, the following is an example packet which should parse without error.
00, // tag (0; no subpacket is 0 yet)
00, // flags (0; no flags)
00, 00 // length (0; no data)
// data would go here, but there is none
Additionally, throughout the rest of the subpackets, there are a few unique and shared datatypes. I document them all more clearly in the draft, but to quickly run through them here too:
UUID
This field represents a globally unique idenfifer, as defined by RFC 9562, as 16 raw bytes.
Frequency
Data encoded in a Frequency field is stored as microhz (1 Hz is stored as
1000000, 2 Hz is stored as 2000000) as an unsigned 64 bit integer. This has a
minimum value of 0 Hz, and a maximum value of 18446744073709551615 uHz, or just
above 18.4 THz. This is a bit of a tradeoff, but itâs a set of issues that I
would gladly contend with rather than deal with the related issues with storing
frequency data as a floating point value downstream. Not a huge factor, but as
an aside, this is also how my current generation SDR processing code (sparky)
stores Frequency data internally, which makes conversion between the two
natural.
IQ samples
ARF supports IQ samples in a number of different formats. Part of the idea here is I want it to be easy for capturing programs to encode ARF for a specific radio without mandating a single iq format representation. For IQ types with a scalar value which takes more than a single byte, this is always paired with a Byte Order field, to indicate if the IQ scalar values are little or big endian.
| ID | Name | Description |
| 0x01 | f32 | interleaved 32 bit floating point scalar values |
| 0x02 | i8 | interleaved 8 bit signed integer scalar values |
| 0x03 | i16 | interleaved 16 bit signed integer scalar values |
| 0x04 | u8 | interleaved 8 bit unsigned integer scalar values |
| 0x05 | f64 | interleaved 64 bit floating point scalar values |
| 0x06 | f16 | interleaved 16 bit floating point scalar values |
Header
Each ARF file must start with a specific Header packet. The header contains information about the ARF stream writ large to follow. Header packets are always marked as âcriticalâ.
In order to help with checking the basic parsing and encoding of this format, the following is an example header subpacket (when encoded or decoded this will be found inside an ARF packet as described above) which should parse without error, with known values.
00, 00, 00, fa, de, dc, ab, 1e, // magic
00, 00, 00, 00, 00, 00, 00, 00, // flags
18, 27, a6, c0, b5, 3b, 06, 07, // start time (1740543127)
// guid (fb47f2f0-957f-4545-94b3-75bc4018dd4b)
fb, 47, f2, f0, 95, 7f, 45, 45,
94, b3, 75, bc, 40, 18, dd, 4b,
// site_id (ba07c5ce-352b-4b20-a8ac-782628e805ca)
ba, 07, c5, ce, 35, 2b, 4b, 20,
a8, ac, 78, 26, 28, e8, 05, ca
Stream Header
Immediately after the arf Header, some number of Stream Headers
follow. There must be exactly the same number of Stream Header packets as are
indicated by the num streams field of the Header. This has the nice effect of
enabling clients to read all the stream headers without requiring buffering of
âunreadâ packets from the stream.
In order to help with checking the basic parsing and encoding of this format, the following is an example stream header subpacket (when encoded or decoded this will be found inside an ARF packet as described above) which should parse without error, with known values.
00, 01, // id (1)
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // format (float32)
01, // byte order (Little Endian)
00, 00, 01, d1, a9, 4a, 20, 00, // rate (2 MHz)
00, 00, 5a, f3, 10, 7a, 40, 00, // frequency (100 MHz)
// guid (7b98019d-694e-417a-8f18-167e2052be4d)
7b, 98, 01, 9d, 69, 4e, 41, 7a,
8f, 18, 16, 7e, 20, 52, be, 4d,
// site_id (98c98dc7-c3c6-47fe-bc05-05fb37b2e0db)
98, c9, 8d, c7, c3, c6, 47, fe,
bc, 05, 05, fb, 37, b2, e0, db,
Samples
Block of IQ samples in the format indicated by this streamâs format and
byte_order field sent in the related Stream Header.
In order to help with checking the basic parsing and encoding of this format, the following is an samples subpacket (when encoded or decoded this will be found inside an ARF packet as described above). The IQ values here are notional (and are either 2 8 bit samples, or 1 16 bit sample, depending on what the related Stream Header was).
01, // id
ab, cd, ab, cd, // iq samples
Frequency Change
The center frequency of the IQ stream has changed since the Stream Header or last Frequency Change has been sent. This is useful to capture IQ streams that are jumping around in frequency during the duration of the capture, rather than starting and stopping them.
In order to help with checking the basic parsing and encoding of this format, the following is a frequency change subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
01, // id
00, 00, b5, e6, 20, f4, 80, 00 // frequency (200 MHz)
Discontinuity
Since the last Samples packet for this stream, samples have been dropped or not encoded to this stream. This can be used for a stream that has dropped samples for some reason, a large gap (radio was needed for something else), or communicating âiq snippitsâ.
In order to help with checking the basic parsing and encoding of this format, the following is a discontinuity subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
01, // id
Location
Up-to-date location as of this moment of the IQ stream, usually from a GPS. This allows for in-band geospatial information to be marked in the IQ stream. This can be used for all sorts of things (detected IQ packet snippits aligned with a time and location or a survey of rf noise in an area)
The sys field indicates the Geodetic system to be used for the provided
latitude, longitude and elevation fields. The full list of supported
geodetic systems is currently just WGS84, but in case something meaningfully
changes in the future, itâd be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a cop-out. Iâd really rather it be what we see out of kinematic state estimation tools like a kalman filter, or at minimum, some sort of ellipsoid. This is neither of those - itâs a perfect sphere of error where we pick the largest error in any direction and use that. Truthfully, I canât be bothered to model this accurately, and I donât want to contort myself into half-assing something I know I will half-ass just because I know better.
| System | Description |
| 0x01 | WGS84 - World Geodetic System 1984 |
In order to help with checking the basic parsing and encoding of this format, the following is a location subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // system (wgs84)
3f, f3, be, 76, c8, b4, 39, 58, // latitude (1.234)
40, 02, c2, 8f, 5c, 28, f5, c3, // longitude (2.345)
40, 59, 00, 00, 00, 00, 00, 00, // elevation (100)
40, 24, 00, 00, 00, 00, 00, 00 // accuracy (10)
Vendor Extension
In addition to the fields I put in the spec, I expect that I may need custom packet types I canât think of now. Thereâs all sorts of useful data that could be encoded into the stream, so Iâd rather there be an officially sanctioned mechanism that allows future work on the spec without constraining myself.
Just an example, Iâve used a custom subpacket to create test vectors, the data is encoded into a Vendor Extension, followed by the IQ for the modulated packet. If the demodulated data and in-band original data donât match, weâve regressed. You could imagine in-band speech-to-text, antenna rotator azimuth information, or demodulated digital sideband data (like FM HDR data) too. Or even things I canât even think of!
In order to help with checking the basic parsing and encoding of this format, the following is a vendor extension subpacket (when encoded or decoded this will be found inside an ARF packet as described above).
// extension id (b24305f6-ff73-4b7a-ae99-7a6b37a5d5cd)
b2, 43, 05, f6, ff, 73, 4b, 7a,
ae, 99, 7a, 6b, 37, a5, d5, cd,
// data (0x01, 0x02, 0x03, 0x04, 0x05)
01, 02, 03, 04, 05
Tradeoffs
The biggest tradeoff that Iâm not entirely happy with is limiting the length
of a packet to u16 â 65535 bytes. Given the u8 sample header, this limits us
to 8191 32 bit sample pairs at a time. I wound up believing that the overhead in
terms of additional packet framing is worth it â because always encoding 4
byte lengths felt like overkill, and a dynamic length scheme ballooned
codepaths in the decoder that I was trying to keep as easy to change as
possible as I worked with the format.
.
Image of the estimated qualiy of articles of the four articles in the second mixed-methods paper. Extreme dips reflect periods of frequent vandalism.
![[RSS 1.0 Feed]](common/rss10.png){kind=small}
![[RSS 2.0 Feed]](common/rss20.png){kind=small}
![[Atom Feed]](common/atom.png){kind=small}
![[FOAF Subscriptions]](common/foaf.png){kind=small}
![[OPML Subscriptions]](common/opml.png){kind=small}
![[Hacker]](common/hacker.png){kind=small}
![[Planet]](common/planet.png){kind=small}