Searching for old Facebook posts

In the past two weeks two people I know have had difficulty trying to find a link or information from an old Facebook post. It seems that Facebook just can't do search.

Here's a work around:

Select "Account Settings" (down arrow at far top-right of screen). At the bottom of the account settings screen there is a "Download a copy of your Facebook data" link. Click on that and follow the procedure. It takes several hours for Facebook to generate an archive of all your data. Once complete you will be notified by email and a link will be provided where you can download a ZIP archive with all your stuff.

Unpack that ZIP and use your browser to navigate to the unpacked directory. Look for file "wall.html" in directory 'html'. That's a HTML page with all your posts.

Now use the browser's in-page search (usually CTRL-F or use the browser menu) to search for what you're looking for. The browser in-page search is a simple substring search... but hey... it's waaay better than what Facebook can manage.

Weekends make me fat!

I've been trying to shift some weight recently and having some success thanks to a great Android app called 'WWDiary' published by CanOfSleep.

Weekends are never a good time for losing weight, but that's just a hunch. It just occurred to me today I may have enough data to actually prove this.

One of the features of this app is a weight diary which I've been using. So I plugged the data into a data file and ran it through GnuPlot.

I then did an auto correlation of this curve:

And sure enough there is a definite peak at 7, 14 and 21 days. "Proof" that weekends do indeed make me fat! [Will revisit this when I have more data!]

Update (28 May 2014): This post was first posted 31 Aug 2012. After almost two more years of data I thought I should update the auto correlation chart:

The peaks at about 7, 14 and 21 days is getting more pronounced.

Transistor abuse: using the avalanching effect of a BJT transistor to generate short pulses

I recently came across an EEVBlog video on a little circuit made popular by Jim Williams in Linear Technology Application Note 47.  This is a simple pulse generator using the breakdown avalanching effect of a BJT transistor which can be used to calculate the bandwidth response of an oscilloscope among other things. Rise times in the order of 300ps are easily achieved this way. The idea of making transistors do things they were not really intended to do is intriguing and I couldn't resist giving it a shot  myself.

The device used in the app note is a Motorola 2N2369, which breaks down at about Vce = 90V. I don't have a PSU that reaches 90V, so I just ordered some low cost, low Vceo transistors.

The circuit used is almost identical to that in App Note 47. I used a larger capacitor (39pF vs 2pF in AN47).

The results have so far been quite pleasing. The following are traces from pulses generated from 3 different BJT devices (see captions under images).  Time base is 20ns / division. My setup is not optimum: the circuit above was implemented on a bread board and the pulses were not coupled to the oscilloscope via 50Ω transmission line. Also my scope is rated at 100MHz bandwidth.

DUT is NXP BFG424F. Avalanching observed when Vce is 15.8V.

DUT is NXP BFG425W. Avalanching observed when Vce is 16.4V.

DUT is NXP BFT25A. Avalanching observed when Vce is 11.8V.

OKI 900 mobile phone limited tear down

I found my old OKI 900 analog mobile phone while looking for a ferrite rod in a junk box at my parents home. This unit was manufactured in 1991. I acquired it second-hand around 1993. It cost me about a months wages back then.

Googling this phone, it seems to have been a very hackable phone back in it's day. It's based on the 8051 MCU core and there are various ROMs available on the network to do all sorts of interesting things.  However, right now I don't have any interest/time in making any mods to this. Indeed I couldn't even be bothered powering it up. So I'll take Dave Jones advice: "don't power it up, take it apart"... but only out of lazyiness :-)

Lidl 'Silvercrest' radio teardown

Last week Lidl (a German budget supermarket chain) had a small "transistor" radio on special. Like a lot of their low end electronics it was "Silvercrest" brand. For various reasons I had need of one (actually I was planning to canabilise it for its ferrite rod antenna, but found an alternative source in the mean time).

Usually Lidl stuff is fine. Not brands you'd normally recognize, but solid stuff none-the-less. So this was rather disappointing.
On powering up, the LCD was faulty. The top half of the display wasn't working.

So I had a look inside:

Apart from the grubby PCB (normal enough for low end electronics these days) there was a botched bodge capacitor (not connect on the left terminal) and a wire to the ferrite antenna nicked by a screw.

So I exchanged it for another. But thankfully decided to checked it out in the car before leaving. Radio #2 didn't even power up. So I tried #3 which did work fine. Two out of three failure rate seems a bit high... or maybe I was just unlucky.

The IC in the photo is a Sony CXA1691 AM/FM radio chip. Everything else looked like it was lifted right out of the 1970s. But hey, it's just a "transistor" radio...

The STM32W-RFCKIT as a 802.15.4 network analyzer on Linux

Summary: The STM32W-RFCKIT together with ST Microelectronics application note AN3406 is a low cost 802.15.4 (ZigBee, MiWi etc) network analyzer. Unfortunately the solution is Windows only and closed source. In this article I have reverse engineered the firmware protocol used in this app note so that a similar solution can be implemented on Linux, Mac OSX and other platforms. I have also written a first draft of this software for Linux.

STM32W-RFCKIT
USB dongle

The STM32W-RFCKIT is a low cost evaluation kit for the STM32W108 802.15.4 SoC from ST Microelectronics. The kit retails for about €33 ($40?) and comprises a USB dongle and a battery powered remote control device with buttons.

I was drawn to this kit because I had recently purchased a STM32F4Discovery board and I was impressed at how easy it was get a GCC tool chain up an running on Linux. I'm currently on the look out for a reliable, low cost 802.15.4 network analyzer solution and given its low cost,  the hope of Linux friendliness and the 802.15.4 transceiver I figured I had a good shot with this hardware.

I was pleasantly surprised to find the kit already comes with a free 802.15.4 network analyzer.  One one of the associated application notes, AN3406, is a 802.15.4 packet sniffer for the dongle. This works by loading special firmware onto the dongle. This then communicates with a Windows server program which translates the dongle's captured 802.15.4 packets (encoded in their own proprietary frame format) into PCAP. The Windows server program also invokes Wireshark and feeds the packet data to it.

Block diagram of the ST Microelectronics AN3406 packet sniffer solution (taken from application note AN3406).

Unfortunately this is a Windows only solution. As a Linux user it's always a pain to have to boot a Windows virtual machine (consuming a huge chunk of my available RAM) just to run one tool. So I set about writing a tool to facilitate the same functionality in Linux.

Although the AN3406 software is free, it is not open source and there is no documentation on the dongle to Wireshark server protocol. So I had to reverse engineer the dongle's protocol to get the same functionality with Linux.

I used the free (and excellent!) serial port monitor tool at www.serial-port-monitor.com to capture the packets to/from the device while opening the Wireshark server and starting a packet capture. Fortunately everything looked very straightforward and it wasn't difficult to write a Linux based server.

The packet structure is

All frames are prefixed with two bytes 0x15, 0xFF. This is followed by a one byte frame length field which is the number of bytes in the frame (excluding the 0x15, 0xFF prefix, the checksum and 0x0C terminator). All frames have a packet type or command code, therefore the minimum valid value for the frame length is 2.

This is followed by a one byte packet type or command code. I notice that frames sent from host to dongle have the command code most significant bit (MSB) cleared while those from the dongle to the host have the command MSB set. For example command 0x10 (set channel) is responded by command 0x90.

Depending on the command there may be zero, one or more data bytes.

Next a frame check sum which is calculated by summing of all bytes from the length field to the last data byte in a 8 bit register and then doing a bitwise NOT.  Finally all frames are terminated with byte 0x0C.

The known command codes are:

Command Code	Direction	Description	Parameters / Data
0x01	Host to dongle	No idea. However it's required for packet capture to work	None
0x81	Dongle to host	Response to 0x01 command.	One byte: 0x00.
0x10	Host to dongle	Set 802.15.4 channel	Followed by one byte of data which will contain the channel number (11 to 26).
0x90	Dongle to host	Set channel response	Same as 0x10
0x11	Host to dongle	Start relaying packets.	No parameters
0x91	Dongle to host	Start relay response.	No parameters
0x12	Host to dongle	Stop relaying packets.	No parameters
0x92	Dongle to host	Stop relay response.	No parameters
0xF0	Dongle to host	802.15.4 packet data	802.15.4 packet metadata (timestamp, RSSI) followed by actual packet data. See section 'Packet capture format' for details.

Packet capture format:

802.15.4 packets are returned along with some packet metadata in the data portion of frames with command code 0xF0. After observing a few frames it quickly became apparent that the actual 802.15.4 packet data started about 8 bytes in. Thus the first 7 bytes is some sort of metadata.

I'll refer to these bytes by their index values starting at 0. Byte 5 remained constant throughout the entire capture and happened to be the same value as the 802.15.4 channel number. So I can safely say that's the channel number. Bytes 1 to 4 also grew in value consistent with a time stamp. As for byte 0, I wasn't sure about that. It seemed random consistent with the LSB of a time stamp, but I thought 5 bytes for a time stamp was odd. Also I figured that there must be a link quality indicator (RSSI) in there somewhere also. Byte 6 looked like a good candidate for the RSSI. So how to know? Plotting these values as a function of time can often reveal their meaning:


Byte 0 (blue) seems to be completely random. So my guess is that's the the least significant byte of a 5 byte time stamp clock. Byte 6 (red) on the other hand must be the RSSI. There are several ZigBee devices on the network. Each device will have an RSSI value based on it's distance/location/antenna that won't vary quickly with time. So each of those red traces corresponds to one ZigBee devices.

So it seems that byte 0, byte 1 and the lower nybble of byte 2 form a 20 bit fraction of a second (ie if treated as an integer each unit is 1/(2^20) seconds.  The upper nybble of byte 2, byte 3 and byte 4 form a 20 bit seconds field.

Putting it all together:

To try this, you must first load the sniffer firmware to the dongle. Right now the only way I know to achieve this is using the AN3406 Windows software. I've documented the procedure in this blog post. And it's also covered in AN3406. However I have no doubt that this can also be accomplished with some of the Linux tools used to program the STM32-F4Discovery board. If you know how, please let me know.

I wrote a translator from this firmware protocol to a PCAP format which can be piped into Wireshark. It's released under the BSD licence. Here is my first draft (version 0.1).

To run:

wireshark -k -i <( ./stm32w-wireshark /dev/ttyACM0 12 )

The command line tool takes two arguments: the device (usually /dev/ttyACM0) and the 802.15.4 channel number (11 to 26). Use the -h flag to get information on more options.

Note: there is a problem with launching Wireshark this way. If you close the Wireshark window, the stm32w-wireshark utility does not die automatically and remains running in the background.  Starting it again will mean that two stm32w-wireshark processes will be grabbing data from the dongle causing massive packet corruption. If you see packet corruption, check for running processes (ps -auxw | grep stm32w-wireshark) and kill any lingering processes. If you have suggestions for a better way of handling this please let me know.

What next?

There are a few lose ends I would like to tidy up (and I would appreciate any suggestions):

• What's the best way of launching this tool together with Wireshark? Closing the Wireshark window should cause the tool to die. Should I be using named pipes?
• It's a shame that the RSSI value cannot be recorded in the PCAP file (there simply is no field for arbitrary metadata). So if one wanted to use RSSI information together with Wireshark how can this be achieved? 
• I would like to be able to upload this firmware without having to boot windows. It's the only Windows dependency left and it would nice to remove it. I suspect one of the standard tools used with the STM32 kits will do the job, but I don't have time to figure that out right now. 

STM32W-RFCKIT as a low cost 802.15.4 / ZigBee network analyzer

Only a few years ago 802.15.4 / ZigBee network analyzers were an expensive affair. Now, many of the low cost evaluation kits from manufacturers such as Microchip, Texas Instruments and ST Microelectronics can be configured as network analyzers.

This morning I took delivery of a ST Microelectronics STM32W-RFCKIT (€33 from Mouser). I was pleasantly surprised that it too has a network analyzer in the form of a Wireshark bridge. These days Wireshark is the only game in town for packet analysis, so this was a smart move.

The kit comes in two pieces: a remote control device with 6 buttons and a dongle for the PC.

Unfortunately all this is Windows software. So I powered up a Windows XP virtual machine, and installed the software from the ZIP file SW application to interface Wireshark packet capture tool (AN3406). I also needed to install Wireshark for Windows. I plugged in the dongle and assigned the USB device to my Windows VM (this is the same as physically plugging in the device if it was a real computer). Then the usual windows new hardware detection and driver installation. (I can never quite understand what is going on with Windows when new hardware is plugged in. I did a reboot also, but that was more out of habit... it might not have been  necessary).

Next, start the STM32W108 Wireshark Server. In the Settings menu set the location of the Wireshark application (usually under Program Files/Wireshark) if this has not been already set.

Right out of the box, the dongle does not have the right firmware installed. So select the Tools menu and select "Flasher". Click the 'Flash' button and all going well, the packet sniffer firmware will be transferred onto the dongle.

Now back to the main window of the Wireshark Server application. Select the right serial port. The Play button should now become active. Hit the play button and all going well Wireshark should start sniffing packets from the dongle.

Linux is my OS of choice, so it would be great to have this working without having to boot a Windows VM. Unfortunately it seems that despite the wise choice of using the Wireshark tool and extolling the virtues of open source code in the documentation, they've neglected to include the source code for the firmware and server software. However it shouldn't be too difficult to reverse engineer. If I get a chance over the coming days I'll give it a stab.

A screen grab of  Wireshark sniffing 802.15.4 / ZigBee packets from the dongle supplied with the STM32W-RFCKIT.

Update (3 April 2012): I just exchanged emails with STM tech support to see if there were willing to share the sniffer firmware source code. Unfortunately they are not. So it's either reverse engineer what's already there or write new firmware.

Update (7 April 2012): I did a quick comparison with the STM32W dongle vs my Microchip ZENA (first version). Both devices are on my desk, about 50cm apart. I ran a Wireshark session on both devices at the same time on the same channel. After a few minutes the STM32W device captured 424 packets vs 290 packets from the ZENA.  It seems the STM32W-RFCKIT makes for a better packet sniffer!

Update (9 April 2012): I've made good progress reverse engineering the firmware protocol. I've now got a Linux command line tool (written in C) that will dump packet hex to terminal and allows the 802.15.4. channel to be changed. I hope to release the first version of the Linux Wireshark server in the next few days.

Update (12 April 2012): The first version of the Linux Wireshark server has been released.

Irish landed estates database 'heatmap'

I am involved with a project to catalog Irish landed estates in conjunction with the Moore Institute at the National University of Ireland, Galway. Right now the database covers the provinces of Muster and Connaught.

The web site includes an interactive map which visitors can use to browse for houses and estates of interest. I thought it might be interesting to make a 'heat map' of the areas which people found the most interesting.

Every time someone zooms into an area on the map, the database is queried for records matching that rectangular area. The web server logs record all these queries (anonymously of course).

The image on the right was generated by summing about half a million of these database queries. I had intended to super-impose the heat map on a Google map, but unfortunately I don't have the time to complete that right now. However one can see the distinct outline of the west and south of Ireland coast line with hot spots around the cities of Galway and Limerick. (Why there is no hot-spot around Cork city? I have no idea!)

The weight ratio of dry pasta to cooked pasta

It's not often you ask Google a simple question and you can't get a straight answer. My question was this: "what is the weight ratio of dry pasta to cooked pasta?"

There is plenty of discussion on this matter, with units such as ounces, cups, Fahrenheit, buschels and microfortnights bandied around. But not one answer that had the simple dimensionless ratio I was looking for.

So it's off to the lab (... well kitchen ...) to do an experiment:

• 100g fusilli (Lid's finest)
• 500ml water
• Bring to boil and boil for 10 minutes
• Strain pasta
• Weigh pasta

So what is this magic ratio? Well, this is not an exact science. Obviously al dente pasta is going to be lighter than over cooked. And it's probable that the topology of the pasta will have some influence on it's ability to retain water. Pasta topologies with lots of surface area and enclosed spaces (eg pasta shells) are going to have more water clinging to the surface. These are experiments for another day.

Anyhow, as for fusilli, the cooked weight of 100g of dry pasta after 10 minutes boiling was 208g.  So the weight ratio of dry pasta to cooked pasta is

1 :  2.1

Visualizing diurnal temperature variation

I've been logging temperature from a ZigBee temperature sensor (located in my shed in the back garden) since about November last year. I thought it would be interesting to see the diurnal (daily) temperature variation over time. On the x-axis are days (from about 10 November 2011 to 7 March 2012) and on the y axis is time-of-day, with midnight being at the very top and bottom. Temperatures vary from about -3C (deep blue) to +15C (red). Periods where there is no data are gray. Sorry no legend or axis markings.

The chart was constructed by averaging temperatures into 15 minute bins (with about 5 samples per bin) and plotting each bin as a rectangle in SVG. I then used rsvg to convert to a PNG image.

For comparison here is a chart for the same time period of my home office temperature. The color scale is different, deep blue being 12C and red 22C.