Microsoft Surface – Porting from v1 to v2

A couple years ago, I wrote Surface Craps during RECESS to explore the Microsoft Surface table and APIs.  Now that Surface 2 is out, it’s time to upgrade the software to run on the new hardware.  This is the first in a series of blog posts about the process in upgrading from Surface v1 to v2.

Obviously this first requires that you have the Surface 2 SDK installed.  To start, I branched the project tree in TFS to make a v2 copy.

Open the project in VS 2010.  Update the project references.  The new referenced assemblies are in C:\Program Files (x86)\Microsoft SDKs\Surface\v2.0\Reference Assemblies.

  • Microsoft.Surface.dll
  • Microsoft.Surface.Presentation.dll
  • Microsoft.Surface.Presentation.Generic.dll

I was using Visual State Manager from the WPF Toolkit, and now it’s included in WPF 4, so I removed that reference as well.

Now we start the brute force process of fixing code that doesn’t compile.  Here’s a list of what I found.  Many of these changes are moving to WPF 4 in general, since it’s where we get most of the touch functionality in the Surface 2 environment.

Control Name Changes

Surface v1 Surface v2
SurfaceUserControl UserControl
SurfaceContentControl ContentControl

Event Handling Changes

Surface v1 Surface v2
ContactDown event use TouchDown
(Preview)ContactUp/Down use (Preview)TouchUp/Down
ContactDown event use TouchDown
(Preview)ContactUp/Down (Preview)TouchUp/Down
ContactChanged TouchMove
ContactEventHandler EventHandler<TouchEventArgs>
ApplicationActivated, etc. OnWindowInteractive, Noninteractive, Unavailable (see a default new Surface 2 project for an example of these event handlers)
ApplicationLauncher.Orientation ApplicationServices.InitialOrientation

Property / Method Changes

Surface v1 Surface v2
e.Contact.IsTagRecognized e.TouchDevice.GetIsTagRecognized( ). (add using statement for Microsoft.Surface.Presentation.Input to get the extension method)
e.Contact.Tag.Byte.Value e.TouchDevice.GetTagData().Series & .Value
e.Contact.GetPosition( ) e.TouchDevice.GetPosition( )
e.Contact.GetOrientation( ) e.TouchDevice.GetOrientation( )
ScatterViewItem.IsActive ScatterViewItem.IsContainerActive

Manipulation Processing

In Surface v1, you would use the Affine2DManipulationProcessor to handle gestures such as flick, rotation, scale, etc.  In v2, you just use the manipulation processing that’s provided by the WPF 4 UIElement.

Surface v1 Surface v2
On any elements that you want to track manipulations on, set IsManipuliationEnabled = true
BeginTrack( ) nothing to do in v2
Affine2DManipulationCompleted, … events UIElement.ManipulationCompleted, … events
e.Velocity.Length e.FinalVelocities.LinearVelocity.Length
e.TotalTranslation.X e.TotalManipulation.Translation.X

Resolution

That’s all I had to as far as API differences.  Next came the resolution differences. The Surface v2 display runs at 1920 x 1080, so if you have any UI that does not stretch or any hardcoded coordinate-based math in your software, it will have to be updated.

Tags

Surface v2 does not support Identity tags, but it does support Byte tags.  See the tag-related methods above for the API differences.

For Surface Craps specifically, I ran into a problem with the transparent dice we got from Microsoft for Surface v1. The dice have byte tags on them that are practically transparent, but have enough IR reflectivity to be picked up by the Surface v1 infrared cameras.  In Surface v2, the tags are recognized by interpreting the contact information using PixelSense technology, and something in that processing is not recognizing the mostly transparent tags on the dice.  I will continue to investigate this issue and write another post if I have any update.  For now – it’s a bummer that the physical dice do not work.

Installation

In a future post I will talk about the differences in what it takes to install your application in Surface v2…

 
 
 

SpeechRecognitionEngine Grammar Choices and Updating them Dynamically

The Microsoft Speech Platform provides a great speech recognition engine. I’ve been using it with the Microsoft Kinect to add voice command functionality to existing Kinect enabled WPF applications. The SDK is located here, and once installed you just need to add a reference to Microsoft.Speech.dll, which is located at C:\Program Files\Microsoft Speech Platform SDK\Assembly.

Static Choices
Once you have the SDK and assembly referenced, the process of specifying what words or phrases you want the SpeechRecognitionEngine to look for is extremely straight forward and easy. Just specify your collection of Choices, create a new GrammarBuilder and append the Choices to it, create a Grammar and provide it with the GrammarBuilder, and finally load that Grammar into the SpeechRecognitionEngine. You can then point the SpeechRecognitionEngine to your audio source and tell it to RecognizeAsync.

var sre = new SpeechRecognitionEngine(ri);
var currentChoices = new Choices();
currentChoices.Add("red");
currentChoices.Add("show blue square");

var gb = new GrammarBuilder
{
     Culture = _ri.Culture
};
gb.Append(currentChoices);

sre.LoadGrammar(new Grammar(gb));

Dynamic Choices
If you know all of the possible command choices from the beginning and never need them to change, then this is as far as you need to go. I, however, wanted to be able to change what choices were valid based on what area of the application I was in. I had hoped it would be as simple as adding and removing choices for the collection, but there isn’t any remove functionality in Choices, GrammarBuilder, or Grammar. On the SpeechRecognitionEngine I did find an UnloadGrammar method so I figured I could just keep a collection of my currently valid choices, create a new Grammar from those, unload the old Grammar, and then load the new one. When I ran the application I ran into some very weird results. Calling UnloadGrammar would take a very long time to execute (it took 3 minutes for one try). Once it got past that point loading the new grammar worked, but the amount of time it was taking was unbearable and I would not be able to use it if there was always the possibility of the application freezing up for that long just to change the available audio choices. So after a long time searching I finally decided to see what MSDN had to say about the UnloadGrammar and LoadGrammar methods and found this page. It is for Office 2007 R2 but the example it provided put me on the right track.

It turns out that when the SpeechRecognitionEngine is running, any changes to the grammar need to occur when the engine is ready for the changes. To get the engine ready you need to call RequestRecognizerUpdate and pass a custom object (that contains what action you want to do and the related data) into the method as a UserToken.

sre.RequestRecognizerUpdate(new UpdateGrammarRequest
								{
									RequestType = GrammarRequestType.UnloadGrammar,
									Grammar = _currentGrammar
								});
sre.RequestRecognizerUpdate(new UpdateGrammarRequest
								{
									RequestType = GrammarRequestType.LoadGrammar,
									Grammar = _currentGrammar
								});

You also need to subscribe to the RecognizerUpdateReached event on the engine, and in that event handler you can call the UnloadGrammar and LoadGrammar methods which will then execute immediately.

private void RecognizerUpdateReached(object sender, RecognizerUpdateReachedEventArgs e)
{
	var request = e.UserToken as UpdateGrammarRequest;
	if (request == null)
		return;

	switch (request.RequestType)
	{
		case GrammarRequestType.LoadGrammar:
			sre.LoadGrammar(request.Grammar);
			break;
		case GrammarRequestType.UnloadGrammar:
			sre.UnloadGrammar(request.Grammar);
			break;
		default:
			throw new ArgumentOutOfRangeException();
	}
}

By doing this, you still end up with a tiny bit of lag between calling RequestRecognizerUpdate and having the RecognizerUpdateReached event fire, but it is only ever a couple seconds instead of minutes.

Snoop 2.7.0

This post is long overdue. I have written about Snoop for WPF a couple times, and it’s been a while since I gave an update.

Back in September, we published v2.7.0 (release announcement). One of the coolest new features is the ability to drag and drop a cross-hair to the app you want to snoop. You no longer have to wait for the list of applications to refresh.  We added a DataContext tab and various bug fixes.

CropperCapture35

I’ve been spending a bit of my RECESS time lately getting back into contributing to the code base. We have some really cool features in the works — I’ll post again when the next version hits the streets.

Prism 4, MEF and the CompositionInitializer

So I have been using MEF (Managed Extensibility Framework) for a while now on most of my WPF projects. Lately, I have been really getting a lot of use out of Prism 4.0 as well and utilizing the built in MEF with the MefBootstrapper.

One of the things that I always found a bit troublesome however was what to do when you need a non-exported class to satisfy its imports from MEF. This came up on a recent project, as I was using the IEventAggregator from Prism, thought it would be useful to have it in classes that may not come from MEF- i.e. classes that are new’ed up depending on the business logic dictated. Now, the true MEF heads may say, ‘Nooooo! That’s not what MEF is for!’. But I thought I would see how it worked anyway.

To solve this, one approach was to pass in my IEventAggregator to my class through the constructor, but this didn’t seem too elegant. What if I need other imports besides the IEventAggregator? Then potentially I need to pass in more and more stuff to the constructor.

What I really needed, was some way to tell the class that it has imports it needs to satisfy, and notify that class that it needs to check the already existing container for this. This is where the CompositionInitializer comes in. One problem with that is that the CompositionInitializer was designed for Silverlight, and my application is MEF. But, no worries, Glenn Block had anticipated my need and kindly written a WPF CompositionInitializer, to be found here. Thanks Glenn. And in addition, I found lots of other great info from Reed Copsey’s site. Thanks Reed.

So I placed my CompositionInitializer in my non-exported class in the constructor:

// No export here
public class MyClass
{
	public MyClass()
	{
		CompositionInitializer.SatisfyImports( this );
	}

	[Import]
	private IEventAggregator EventAggregator { get; set; }
}

But when I ran the code, even though I had a valid import for my IEventAggregator, none of my messages were being received. A bit of googling brought me here. Aha! Simply placing my CompositionInitializer.SatisfyImports( this ) in the constructor is insufficient. How, after all, does the CompositionInitializer know what container you are using? I was so used to MEF magic that it hadn’t occurred to me that CompositionInitializer needs to have its container set.

By adding the following line to my BootStrapper file in a location that gets called after CreateShell():

CompositionHost.Initialize( Container );

My CompositionInitializer was now armed with the appropriate container, and everything worked great in the constructor. Now I was able to new up an instance of any class and satisfy its imports against my Container.

XamlReader, Loose ResourceDictionary Files, and the ParserContext

In a few WPF projects that I have been on I have needed to load in loose ResourceDictionary files to provide different runtime styling. When there isn’t any external content being referenced (ie. images, videos, etc…) it is as simple as calling XamlReader.Load and then merging the dictionary into the application’s resources:

var resourceDictionary = XamlReader.Load(fileStream) as ResourceDictionary;
if(resourceDictionary != null)
{
    Resources.MergedDictionaries.Add(resourceDictionary)
}

When external content is being referenced there is the possibility that the XamlReader won’t be able to find those files, even if they are in the exact same folder as the ResourceDictionary files. To solve this issue, you will need to create a ParserContext and set its BaseUri to the root folder containing the files. For example, if the ResourceDictionary and content files are located in a folder called RuntimeResources, which is in the same folder as the application’s exe, then the BaseUri needs to point to that RuntimeResources folder. Then when the XamlReader loads the ResourceDictionary, it uses the provided ParserContext to know where to look for the actual files. NOTE: In the ResourceDictionary you will still need to include the root folder (RuntimeResources) at the beginning of the uri otherwise it will look in the parent directory for the desired file.

Below is the related code:

C#

var applicationDirectory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if(!String.IsNullOrEmpty(applicationDirectory ))
{

    var runtimeResourcesDirectory = Path.Combine(applicationDirectory , "RuntimeResources");
    var pc = new ParserContext
    {
        BaseUri = new Uri(runtimeResourcesDirectory , UriKind.Absolute)
    };
    if(Directory.Exists(runtimeResourcesDirectory ))
    {
        foreach (string resourceDictionaryFile in Directory.GetFiles(runtimeResourcesDirectory , "*.xaml"))
        {
            using (Stream s = File.Open(resourceDictionaryFile, FileMode.Open, FileAccess.Read))
            {
                try
                {
                    var resourceDictionary = XamlReader.Load(s, pc) as ResourceDictionary;
                    if (resourceDictionary != null)
                    {
                        Resources.MergedDictionaries.Add(resourceDictionary);
                    }
                }
                catch
                {
                    MessageBox.Show("Invalid xaml: " + resourceDictionaryFile);
                }
            }
        }
    }
}

Resource Dictionary

<ResourceDictionary xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
			    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">

	<ImageBrush x:Key="LogoImage"
			  ImageSource="RuntimeResources/logo.png" />

</ResourceDictionary>

Now you’ve got the ability to change your WPF application’s styling at runtime.

Merging a WPF application into a single EXE

I always dislike handing off little applications to people. Not because I can’t, but because of the steps involved to make sure it all just works. Small apps are the most problematic because I never want to take the time to create a whole installer project for just a few assemblies, and packaging up a zip file must be accompanied by “Unzip this into a folder in your programs directory and create a shortcut…” which brings us back to the whole installer business we started with.

There are a few tools already out there such as ILMerge by Microsoft Research (Which works great for most .NET-y things, but chokes on WPF applications) and a few paid tools by third party vendors that you could fork over a few hundred for to get. But, I’m a developer, which means I want to do it the Hard Way™. I did a little research and found the following blog posts on setting up and merging in DLL’s as resources into the main assembly and then extracting and loading them into memory when you run your application.

Links:

There were a few things I didn’t like about each solution. The first one (richarddingwall.name) ends up having you directly adding the .dll’s as resources directly. I hate maintaining things manually, especially when it will run fine on my machine but break when when I move it somewhere else because I forgot to update the resources when I added a new project. The one from blog.mahop.net builds on the previous one and changes the resolve location to a custom class with its own startup method. Better, because it resolves the resources earlier. Finally, the one from Daniel Chambers (digitallycreated.net) added in the final piece that automatically including the assemblies as resources. Unfortunately, the way he looks for culture specific assemblies didn’t work and I had to remove / change it to be closer to the one on mahop.net.

Final solution I’m currently using is as follows:

To the main executable project, unload and edit the .csproj file, and below the following line:

<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />

Add this XML to the project file, save, and load it back up.

 <Target Name="AfterResolveReferences">
  <ItemGroup>
    <EmbeddedResource Include="@(ReferenceCopyLocalPaths)" Condition="'%(ReferenceCopyLocalPaths.Extension)' == '.dll'">
      <LogicalName>%(ReferenceCopyLocalPaths.DestinationSubDirectory)%(ReferenceCopyLocalPaths.Filename)%(ReferenceCopyLocalPaths.Extension)</LogicalName>
    </EmbeddedResource>
  </ItemGroup>
</Target>

It should look something like this when your done:

You’ll then add a new code file to the main project and add the following code to it (modified to fit how your application is named / structured):

        [STAThread]
        public static void Main()
        {
            AppDomain.CurrentDomain.AssemblyResolve += OnResolveAssembly;

            App.Main(); // Run WPF startup code.
        }

        private static Assembly OnResolveAssembly(object sender, ResolveEventArgs e)
        {
            var thisAssembly = Assembly.GetExecutingAssembly();

            // Get the Name of the AssemblyFile
            var assemblyName = new AssemblyName(e.Name);
            var dllName = assemblyName.Name + ".dll";

            // Load from Embedded Resources - This function is not called if the Assembly is already
            // in the same folder as the app.
            var resources = thisAssembly.GetManifestResourceNames().Where(s => s.EndsWith(dllName));
            if (resources.Any())
            {

                // 99% of cases will only have one matching item, but if you don't,
                // you will have to change the logic to handle those cases.
                var resourceName = resources.First();
                using (var stream = thisAssembly.GetManifestResourceStream(resourceName))
                {
                    if (stream == null) return null;
                    var block = new byte[stream.Length];

                    // Safely try to load the assembly.
                    try
                    {
                        stream.Read(block, 0, block.Length);
                        return Assembly.Load(block);
                    }
                    catch (IOException)
                    {
                        return null;
                    }
                    catch(BadImageFormatException)
                    {
                        return null;
                    }
                }
            }

            // in the case the resource doesn't exist, return null.
            return null;
        }

Finally, make sure you update the target method for your main application to be the main method for the project you just added:

And, that’s it!

When you build your application you’ll still see all the assemblies in the output directory, but you should be able to take just the executable, move it somewhere else, and run it just as it is.

Presentation Materials for San Diego Code Camp

Thanks to everyone for coming!

Harnessing XAML Templates: Slides | Code (WPF, SL4, SL5)

Easy Async with Async CTP: Slides | Code

Look at the AddingAsync section for the multi-step conversion from synchronous to async with cancellation and error handling that we didn’t have time to walk through.

Get the Async CTP at http://msdn.microsoft.com/en-us/vstudio/async.aspx