Using the accelerometer to control planar transforms on Windows Phone 7

Lately I’ve been looking into some Augmented Reality uses on Windows Phone 7, and one of the first things you need to do for this, is to use the sensors to control what’s displayed on the phone screen.

So the first step for me was to better understand how the accelerometer interacted with the phone’s orientation. I wanted to create a simple “3D Plane” that was always looked like it was “level”. This could be accomplished with a Rectangle and a PlaneTransform. The only question is: How do I transform the accelerometer values to the RotationX/Y/Z rotation angles on the PlaneTransform?

To make matters worse, depending on whether your phone is currently in Portrait, LandscapeLeft or LandscapeRight mode, the screen coordinate system, and thus the transforms changes, even though the accelerometer report the values independent of screen orientation. So to get a property transform, we also need to know what way the screen coordinate system is oriented relative to the accelerometer coordinate system.

With a little trigonometry it wasn’t too hard to figure out, so here’s the code to perform this little app.

First the border we used as the “level” plane:

<Border x:Name="plane" BorderBrush="Green" BorderThickness="2"
        Width="350" Height="350" >
    <Border.Projection>
        <PlaneProjection x:Name="proj"  />
    </Border.Projection>
</Border>

Next, create an accelerometer instance, and call “Update” every time we get a reading:

var meter = new Accelerometer();
meter.ReadingChanged += meter_ReadingChanged;
...
private void meter_ReadingChanged(object sender, AccelerometerReadingEventArgs e)
{
    Dispatcher.BeginInvoke(() =>
    {
        UpdateRotation(e.X,e.Y.e.Z);
    });
}

Lastly, update the PlaneProjection parameters, based on the rotation (this is where the “real meat” is):

private void UpdateRotation(double x, double y, double z)
{
    var offset = 0d;
    //Adjust for screen orientation when in landscape mode
    //PortraitUp mode doesn't need an offset
    if (Orientation == PageOrientation.LandscapeLeft) offset = .5;
    else if (Orientation == PageOrientation.LandscapeRight) offset = 1.5;                

    var rx = (Math.Acos(z) / Math.PI + 1) * 180;
    var rz = (offset - Math.Atan2(x, y) / Math.PI + 1) * 180;
    if(!double.IsNaN(rz)) proj.RotationZ = rz;
    if(!double.IsNaN(rx)) proj.RotationX = rx;
}

And lastly, here’s what this looks like (note how the screen orientation flips when the phone is rotated, but the plane stays in the same place):

Note that this sample doesn’t deal with calibration. I suggest you look at this blogpost on how to do that. It also includes a lot of information on the accelerometer in general. That blog also discusses filtering the noisy data which is important to give a smooth looking result. The sample code you can download below, includes a simple low-pass filter to make it feel a lot smoother.

Download source code

Lastly here’s an example/preview of a little Augmented Reality app I’ve been working on that is using this approach to overlay the camera feed with azimuth values:

ESRI DevSummit 2011–over'n'out

The ESRI Developer Summit is over, and I had quite a blast. We got tons of great feedback and tough questions from all the developers attending. Thank you for that! We also had some well-attended sessions on Windows Phone, Silverlight and WPF. When these recordings goes live, I’ll update this blog-post.

For the first time, I even got to go on stage during plenary and make a fool of myself. You might already have read about my ventures into programming with a Kinect, but I got to show a far more polished version during the plenary session, and luckily everything worked out great. This version (although not demoed due to time constraints) actually supports multiple users interacting with the map at the same time (at one point we had 4 different people drawing on the map all simultaneously). For those interested, this was based on OpenNI drivers and the ArcGIS API for WPF, which is a different approach than I originally blogged about. Here’s a recording of my 2 minutes of “fame” Smile:

Me on stage making a fool of myself

I have to admit I was a little nervous this would break down during the demo, since this was all based on stuff that wasn’t even in beta software yet. We later had it running at our showcase area 24/7 and I didn’t have to restart the app for almost 3 days running with lots of users playing with it, so perhaps it was running more solid than I had feared Smile - So go play with those OpenNI drivers! They might be called “unstable” but I found them to be very stable.

My little A-to-B app also go a nice mention during the plenary. I wrote this to learn some more phone application practices, exercise the phone API we’ve been working on for a while and simply to have some coding fun. If you have a Windows Phone 7, go download it now, and if not, here’s Dave giving a quick tour of the app.

A-to-B WinPhone7 Application demoed

Thank you to every who attended and especially those who gave us feedback.

Fun times with Kinect and WPF

I recently bought a Kinect controller for my Xbox and is really loving it. Of course I had to see if I couldn’t use that for something, and after some other people were nice enough to create and release free Windows drivers and libraries, I had to try it!

I started out with Code Laboratories “CL NUI Platform” which is a simple installer that installs the drivers and some libraries. It also comes with some samples showing how to connect to the sensor.

I quickly got that going, with a small app that displays the RGB and Depth image from the sensor.

223324934[2]

“Unfortunately” this is also all you get out of the box. If you expect to automatically get an event when you wave your hand or doing some other gesture, this is not for you. However if have a little love for some simple Image Analysis algorithms, you should be all over this too!

I wanted to make a simple gesture API that could tell me when a user drags, pinch/stretch or flicks something, very similar to what you do with a Touch Screen. So basically an API that mimics an invisible touch screen hanging suspended in the air. Luckily the algorithms for this turned out to be pretty simple.

The first thing you do is go through the depth image pixel by pixel. If you find a pixel that is too close or too far away, replace it with Color.Empty so we can ignore those. That we we will only “see” things in the picture that are within a certain distance from the sensor. Like for instance two hands stretched out. This leaves us with a set of pixels that could potential be some stretched out hands.

223411798[2]

We need to associate these pixels with each other. Let’s say all pixels that are next to each other that are not transparent, belong to the same hand. We call these sets of grouped pixels for “Blobs” and there’s a nice simple algorithm for detecting all the blobs in an image, known as “Connected Component Labelling”. Instead of explaining how it works here, all you need to know is here on wikipedia. If you register their center and size, you can easily use these as “touch input” and discard too large or small blobs. This is what this looks like, using a circle for center and size of each blob:

30 times a second you will be calculating a new set of blobs. Compare these to the previous blobs, and use the center and size to see which blobs are the same and has moved, which are new, and which disappeared. If I only have one blob and it moves, I consider this a “drag”. If I have more than 1, I calculate the bounding box of all the blobs, measure its diagonal and compare it to the size of the previous diagonal - The fraction between these two diagonals is the scale in a pinch or stretch.

Now all we need to do is take these “gesture events” and hook them up to something. Like for instance a map. So here’s the end-result: (note : It looks a little jerky because of the screen capture software, but it runs a lot smoother when not recording and flicks works consistently).

Minority MapReport

It worked surprising well for a first go.

Download source: Sorry no source available at this time, due to some licensing restrictions.

Simulating GPS on Windows Phone 7

If you are building Windows Phones apps that require the use of GPS, you cannot use the Emulator, and should preferably be outside.

While building some of my latest WinPhone apps, I needed to be able to able test moving around on a map, while being able to debug. However I preferred doing this at the comfort of my desk. So I had to “fake” a GPS signal. Luckily the GeoCoordinateWatcher class that is used for this, implements the interface “IGeoPositionWatcher<GeoCoordinate>”.  If I were to create my own class that implements this interface, I had an option for using a separate class for receiving data from the location service. I ended up needing two types of location simulators for my routing app: One that starts at a given location and direction, and randomizes speed and direction over time. The other follows a predefined route. You can download the source code for these below.

The next step is to automatically use the simulator when I’m running on the emulator, but use the “real” location service when running on the device. You can accomplish this by checking the DeviceName first. Example:

   1:  IGeoPositionWatcher<GeoCoordinate> geowatcher; 
   2:  string deviceName = Microsoft.Phone.Info.DeviceExtendedProperties.GetValue("DeviceName").ToString(); 
   3:  if (deviceName == "XDeviceEmulator") //Use simulator for emulator 
   4:      geowatcher = new GeoCoordinateSimulator(34.0568, -117.195, 270) 
   5:  else 
   6:      geowatcher = new GeoCoordinateWatcher(GeoPositionAccuracy.High); 
   7:  geowatcher.PositionChanged += watcher_PositionChanged; 
   8:  geowatcher.Start();

The above uses the “random” simulator when running on the emulator.

The other simulator takes a set of GeoCoordinates and interpolates between them at the specified speed. Example:

   1:  var pnts = new List<GeoCoordinate>(); 
   2:  pnts.Add(new GeoCoordinate() { Longitude = -117.195670639, Latitude = 34.0568044102 }); 
   3:  pnts.Add(new GeoCoordinate() { Longitude = -117.195697272, Latitude = 34.0585127272 }); 
   4:  pnts.Add(new GeoCoordinate() { Longitude = -117.204302784, Latitude = 34.0515719501 }); 
   5:  geowatcher = new SharpGIS.WinPhone.Gps.GpsDriveSimulator(pnts, 20);

Below is an example of this simulator in action on the emulator in a little app I’m working on:

screenshot_12-7-2010_22.7.41.326photo[1]

It was fun putting this app on the real device, go for a drive along roughly the same route and it worked exactly same. The simulator helped me weed out a series of bugs that would have taken me several (dangerous) “test drives” if I hadn’t had this simulator.

Download Simulator code here:

Submitted my first WinPhone 7 Application

imageI’ve just submitted my first application to the Windows Phone Marketplace. It’s a very simple app that creates a “Guide Post” with signs on a pole showing distances and direction to any point in the world. You have probably seen these at various sites all over the world, or in the TV Show “M*A*S*H”.

It’s all built in Silverlight, and uses PlaneProjection to give give the signs a 3D effect. You use your finger to slider over the screen to rotate it.

In addition to that there’s a Map where you can view your current location and the great circle line (ie. shortest path) between you and the points of interest. This is of course using the ESRI ArcGIS API for Windows Phone that we just released.

Below is a few screenshots and a video clip of the app in action.

screen_12-1-2010_8.51.30.996screen_12-1-2010_8.50.48.336
screenshot_12-2-2010_12.26.46.864screenshot_12-2-2010_12.27.59.830

GuidePost Screen Capture

It was a great fun little app to build and only took a few evenings to get done. You can download the app to your phone from this link: http://bit.ly/WP7GP

Displaying HTML in Silverlight

In my second post in the series of ripping off other peoples work :-) here’s an updated version of David Anson’s HtmlTextBlock control back from 2007.

This was originally written based on the TextBlock control, but since the TextBlock control was sealed and couldn’t be inherited from, it required a lot of copying properties around. Textblocks also only supports text, so the HyperLink method was not really a hyperlink, and there was no image support, but from what we had available back them, it was a really good solution considering the simplicity (I like simple small implementations).

Since then Silverlight 4 has arrived, and we now have the RichTextBlock control. This supports hyperlinks, images etc, and it is not sealed, so we can pretty easily get around the limitations that the original implementation had. Most of the logic for generating the HTML is the same, but now with proper support for images and real hyperlinks. Again all the credit goes to the original author. I just did some tweaks.

You can download the source with a sample app here: HtmlTextBlock.zip (34.15 kb)

UPDATE: Rewrote the parser approach quite a lot. Now it also supports tables (!) as well as setting fontsize using the H1..H4 tags. With the new parser approach it should be easier to add support for other more complex tags as well.

You can download the source with a sample app here: HtmlTextBlock_v2.zip (34.15 kb)

REALLY small unzip utility for Silverlight – Part 2

I earlier blogged about a very small unzip utility for Silverlight.

However there was one problem with the method: It relied on some Silverlight framework libraries that didn’t support all zip file formats (a known bug I’m told). Some file zippers places the file size after the zip data chunk, and Silverlight does not like that. So I added a little bit to the utility to detect this case and if so, will re-arrange the bytes in the zipfile so Silverlight will read it.

So I present to you: The really small unzip utility v.2.0. It’s slightly larger than before, but still under 200 lines of code (plus comments).

It works the same way as before, so read the original blogpost to see how to use it.

Although I only tested this on Silverlight 4 and WinPhone 7, it should work with earlier Silverlight releases as well.

Download here: SharpGIS.UnZipper_v2.zip (2.53 kb)

Also available at NuGet

Dependency Property gotchas

Here’s a quick list of gotchas I’ve learnt the hard way or seen over the time while working with Dependency Properties in Silverlight and WPF (most of this applies to Attached Properties as well).
Here’s a typical dependency property defined on a class inheriting from DependencyObject:

public int SomeValue
{
    get { return (int)GetValue(SomeValueProperty); }
    set { SetValue(SomeValueProperty, value); }
}
public static readonly DependencyProperty SomeValueProperty =
    DependencyProperty.Register("SomeValue", typeof(int), typeof(ownerclass), null);
private static void OnSomeValuePropertyChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
{
    ownerclass obj = (ownerclass)d;
    int newValue = (int)e.NewValue;
    int oldValue = (int)e.OldValue;
} 

Naming

The naming is very important. Note ‘SomeValue’ is not only the name of the property, but the static DependencyProperty field is prefixed with the word ‘Property’, and the string in the first parameter also must match this name.

Type and default values.


Below we instead create a dependency property of type double and define a default value in PropertyMetadata to 0. See if you can spot the bug:

public static readonly DependencyProperty SomeValueProperty =
    DependencyProperty.Register("SomeValue", typeof(double), typeof(ownerclass), new PropertyMetadata(0));

The problem is very subtle, and the error you will get at runtime is not always obvious. The reason is that PropertyMetadata takes an object as parameter, and there’s no way for the debugger to know that you meant to put a double there, and it will wrongly assume Int32. At runtime you will get an exception because of this. Instead make sure you explicitly declare the default value as a double:

public static readonly DependencyProperty SomeValueProperty =
    DependencyProperty.Register("SomeValue", typeof(double), typeof(ownerclass), new PropertyMetadata(0d));

Default values are static – Beware of object references!

Don’t ever ever EVER declare a default value in a DependencyProperty that is not a value type. Since this field is static, the default value will be shared among every instance that’s using the default value.

If I were to access the property and modify its sub properties, anything using this instance would be affected.
Example: Brush property on a control. I have multiple instances of the control, but changing a subproperty on the property will affect all the controls:

public static readonly DependencyProperty MyBrushProperty =
    DependencyProperty.Register("MyBrush", typeof(SolidColorBrush), 
    typeof(ownerclass), new PropertyMetadata (new SolidColorBrush(Colors.Red))); 

. . . 

myControl.MyBrush.Color = Colors.Blue;

Instead set the default value to null. Then either set the property in the default style template (if it’s declared on a control), or set it in the constructor of your class.

Code in property setter

Often you are used to execute code in the setter of your property when a value changes like so:

public int SomeValue
{
    get { return (int)GetValue(SomeValueProperty); }
    set 
    { 
        if(value != SomeValue)
        {
            SetValue(SomeValueProperty, value); 
            UpdateValues(value);
        }
    }
}

However, if you’ve done this, you might have seen that sometimes your setter is not called, and the UpdateValues() method is never executed. The reason is that if you are binding to this property instead of just calling the setter directly, this setter will never be hit! Instead Silverlight/WPF will call SetValue(…) directly and “circumvent” your setter (this is also why the naming mentioned above is important so it can find the property).
Instead the correct way is to declare a method to call when the value changes. This is done in the PropertyMetadata instance: Also note that you do not need to check whether the value actually has changed. The changed handler will only be called if the value is actually different.

public static readonly DependencyProperty SomeValueProperty =
    DependencyProperty.Register("SomeValue", typeof(int), typeof(ownerclass), new PropertyMetadata(0, OnSomeValuePropertyChanged));
private static void OnSomeValuePropertyChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
{
    ownerclass obj = (ownerclass)d;
    int newValue = (int)e.NewValue;
    obj.UpdateValues(newValue);
}

Code Syntax Highlighting in Silverlight

A little while back, Jeff Wilcox wrote an awesome syntax highlighter (blog post here). It enables you to display code in Silverlight with similar syntax highlighting that we know and love from Visual Studio. This is really useful if you want to display some code samples inside your Silverlight application.

However it had one major feature missing: You couldn’t select and copy the code. And what good are code samples if you can’t do that (unless you’re the patient type who likes typing a lot). Vladimir Horbovanu had commented that he had modified the sample to be built on MVVM and added the option of selecting and even editing the text. However, this only supported C# and not VB, JS, Xaml, C++ and Xml as the original sample did, and there were a few issues with getting the text to properly align. However the “trick” he applied was simple but clever: Put a TextBox on top of the existing control, and make it transparent.

So I basically took Jeff’ awesome library, modified the template a bit, bound a TextBox with transparent text to the source code and put it on top of the existing template parts, tweaked the TextBox template to properly align and voila: The SyntaxHighlightingTextBlock is now a SyntaxHighlightingTextBox. I also added a couple of properties to the control to quickly get the selected text, set readonly state etc.

Jeff deserves most of the credit for this one, and Vladimir the rest for the neat TextBox trick. All I did was put it together, clean it up a bit and fix a couple of bugs in the highlighter. Expect to see this used in the next release of our SDK.

You can get the source here, and view the sample app here.

The library still only requires Silverlight 3, however the demo app uses Silverlight 4’ right-click feature for the context menu.

image

Update: Vlad took this sample further and improved rendering performance when editing larger amounts of code. See his updated blogpost here: http://www.vladhorby.com/blog/post/2010/11/01/code-editor-control-with-syntax-highlighting-for-silverlight-available.aspx
Isn't it great how the .NET community works together on building some useful stuff? Thanks Vlad!

Add MouseWheel scrolling to Silverlight's DataGrid

In my previous post, I wrote about how you could add MouseWheel scrolling to the ScrollViewer using a simple behavior. I wanted to have the same MouseWheel scroll on the DataGrid, and thought that it could be done the exact same way.

However, it turns out that the DataGrid is not using a ScrollViewer, but instead used a Virtualized panel that is re-rendered every time a ScrollBar is scrolled. Unfortunately the DataGrid only reacts to the Scroll event of the ScrollBar, and not it’s Value property (not sure why they did it like this), so it’s not just a matter of finding the ScrollBar and increase the value. If I did that, the ScrollBar would move, but the DataGrid would not update. In fact the only way to raise the Scroll event was to click the ScrollBar, but I wanted to do this in code so I can’t really “click” something.

Fortunately Silverlight has a UIAutomation framework, that allows you to simulate user interaction. This is great for running automated tests against your controls, where you want to simulate a user clicking and typing inside your application. This got me thinking that the way to raise the Scroll event was simply to use the UIAutomation framework to virtually ‘click’ the ScrollBar. So in other words, each time you scroll down, the code ‘clicks’ the down button in the ScrollBar, and vice versa. While this simulated click approach works well, there’s an even simpler way, since the DataGrid’s Automation allows for calling scroll, which is what I’ll be using here.

The first step is to get a reference to the AutomationPeer of the DataGrid:

AutomationPeer automationPeer = FrameworkElementAutomationPeer.FromElement(this.AssociatedObject);

Since the automation peer for the DataGrid implements the IScrollProvider pattern, we can simply request that pattern and call Scroll() to scroll the DataGrid:

IScrollProvider scrollProvider = automationPeer.GetPattern(PatternInterface.Scroll) as IScrollProvider;
ScrollProvider.Scroll(ScrollAmount.NoAmount, ScrollAmount.SmallIncrement);

The rest is trivial for the DataGrid MouseWheel behavior. On the MouseWheel event of the DataGrid we call the Scroll method with either a SmallIncrement or SmallDecrement, and the DataGrid will scroll one row. It’s not as smooth as the ScrollViewer implementation in the previous blogpost, but at least it allows us to use the MouseWheel to scroll the rows. Since most scrollable controls in Silverlight already has UIAutomation and IScrollProvider support, this behavior will also work on other scrollable controls, and not only the DataGrid.

So here’s what it looks like in XAML when you included a reference to the behavior assembly:

<data:DataGrid x:Name="datagrid" Height="100" >
	<i:Interaction.Behaviors>
		<behaviors:MouseScrollBehavior/>
	</i:Interaction.Behaviors>
</data:DataGrid>

I’ve updated the source code for my Behavior library, and also updated the sample application to include the DataGrid.