Endianness Fun in C#

Josip Medved Leave a comment

Those doing parsing of network protocols will be familiar with BitConverter. For example, to read an integer that’s written in big-endian (aka network) order, one could write something like this:

Array.Reverse(buffer, offset, 4);
Console.WriteLine(BitConverter.ToInt32(buffer, offset));

If one is dealing with multiplatform code, they could even go with a slightly smarter code:

if (BitConverter.IsLittleEndian) { Array.Reverse(buffer, offset, 4); }
Console.WriteLine(BitConverter.ToInt32(buffer, offset));

However, that’s the old-style way of doing things. For a while now, .NET also offers BinaryPrimitives class in System.Buffers.Binary namespace. While this class was originally designed to be used with protocol buffers (which explains the namespace), there is no reason why you couldn’t use it anywhere else. Actually, considering how versatile the class is, I am stunned they didn’t just add it in the System namespace.

In any case, reading our 32-bit number is now as easy as:

Console.WriteLine(BinaryPrimitives.ReadInt32BigEndian(buffer));

And yes, this doesn’t accept offset argument. Boo-hoo! Just use Span and slice it.

var span = new ReadOnlySpan<byte>(buffer);
Console.WriteLine(BinaryPrimitives.ReadInt32BigEndian(span.Slice(offset)));

Much easier than remembering if you need to reverse array or not. :)

Calculating Mean for Microchip PIC

Josip Medved Leave a comment

Averaging ADC readings on microcontroller is often needed but also often quite problematic. The easiest way to do it is remembering the last N values, summing them together and simply dividing by count. Simple, reliable, and uses a BUNCH of memory. If you want to store 50 16-bit values, you will need 100 bytes. When dealing with Microchip PIC, that’s a fortune.

However, if you’re ok with a mean rather with an average, there’s an easier way (and I would argue that mean is a better measure anyways in 90% of cases). You can use Welford’s online algorithm to get mean value using only 3 bytes of static storage and a few bytes more when executing. Of course, use of such algorithm in microcontroller environment will require a bit of a trade off (the least of them is me using average and mean interchangeably further down in post).

For my case, I assumed I need it for unsigned 10-bit ADC values. In the context of Walford algorithm, this means my intermediate values can be up to 15-bits (16-bit signed integer is needed internally) and this leaves 5 bits to use as a fixed decimal point. My case also called just “averaging” a handful of values, so I was ok with a 8-bit counter.

This means my variables are as follows:

Variables 
uint8_t walfordCount = 0;
int16_t walfordMeanEx = 0;

As compared to the standard Walford’s algorithm, a few things are different. First of all, if we have more than N entries, it will reduce count to half over and over again. This trickery is allowing for “sorta” running average. And yes, you shouldn’t really use Walford’s for running average but it’s close enough. Secondly, when adding the value into the internal variable, we convert it to the signed integer and shift it 5-bits to simulate a decimal point. The only expensive (in CPU terms) operation here is dividing by count as everything else is quite cheap (shift, subtractions, and comparisons).

Adding values 
void walford_add(uint16_t value) {
    walfordCount += 1;
    if (walfordCount > WALFORD_CUTOFF) { walfordCount >>= 1; }
    int16_t delta = (int16_t)(value << FIXED_DECIMAL) - walfordMeanEx;
    walfordMeanEx += delta / walfordCount;
}

To get a value out, we just shift variable 5-bits back to remove the “decimal point”.

Adding values 
uint16_t walford_getMean(void) {
    int16_t value = (walfordMeanEx >> FIXED_DECIMAL);
    return (value > 0) ? (uint16_t)value : 0;
}

And that’s it. It’s a cheap (memory-wise) way to get an average on a 8-bit microcontrollers. As compromises go, I had worse.

In any case, you can download code here.

Disk Preclear

Josip Medved Leave a comment

Some time ago I got into a habit of shucking. And each shucked drive I first fill with a random data. Purpose of this is twofold; the primary reason being security as disk filled with random data doesn’t “leak” information when you place encrypted file system on it (except when that file system is ZFS with native encryption but that’s a long rant for some other time).

The second reason is drive verification before I add it to my pool. While disks are hearty bunch these days, they do fail. And they usually fail either really early or really late. Giving it a bit of exercise early in their lifetime can sometime provoke failure. And if that failure happens you can easily do RMA. Doing the warranty replacement is also possible once they are shucked but might involve messing with manufacturer and their idea of “intended purpose” that’s part of their terms of warranty. It’s easier this way.

Previously I used a good old dd on my server for the same purpose but that actually became slightly annoying as I filled all the bays. Just to initialize a new drive, I had to remove the old drive and hope other disks are ok while I randomize the new one (can take a few days). Furthermore, that also meant I was increasing the CPU load (remember random numbers are not cheap) and temperature without writing any useful data. A task that’s more suitable for my Windows desktop.

With .NET 6 out, I figured I could give it a ride and write a Windows Forms application.

In a default state application will randomly traverse disk writing random data and verifying what’s written. While access might be random, every sector is accessed once and only once even in this mode. This is my favorite mode by far and I find it strangely calming due to how it looks graphically. But then again, I was always a fan of just looking into the defragmentation screen. :)

If you want the same effect you would get using dd with urandom you can switch to the sequential access. The same random data but written quite a bit faster since drive doesn’t need to seek all over the place. Of course, if security is not much of a concern, once could also fill disk with zeros or a “standard” 0x55AA pattern. While slightly faster, those modes are quite a bit less exciting.

I created this application to fill the need I had and no other application seemed to do exactly what I wanted. If someone else finds it useful, it’s awesome. If not, I still had fun creating it and that’s all that matters in the end.

Application download and link to its source code is available here.

IDE0180: Use Tuple to Swap Values

Josip Medved Leave a comment

As I upgraded my Visual Studio 2022 Preview, I noticed a new code suggestion: IDE0180. It popped next to a reasonably standard variable swap:

Old swap code 
var tmp = a;
a = b;
b = tmp;

The new syntax it recommended was much nicer, in my opinion:

New swap code 
(a, b) = (b, a);

I love the new syntax!

Except it’s not new – it has been available since C# 7 (we’re at 10 now, just for reference). I just somehow missed it. The only reason I noticed it now was due to a suggestion. I guess better late than never. :)