We have been working with the Kinect for a while now, writing various apps that let you manipulate the UI of a Windows app while standing a few feet away from the computer – the “10 foot interface” as they call it. Very cool stuff. These apps make use of the Microsoft Kinect for Windows SDK to capture the data coming from the Kinect and translate it into types we can use in our apps: depth data, RGB image data, and skeleton points. Almost all of these apps are written in C# / WPF and run on Windows 7.
Last month a few of us went to the Microsoft //BUILD/ conference, and came back to start writing apps for the new Windows 8 Metro world. Then naturally, we wanted to combine the two and have an app that uses Kinect in Windows 8 Metro. At InterKnowlogy we have a “Kiosk Framework” that fetches content (images, audio, video) from a backend (SQL Server, SharePoint) and has a client for various form factors (Win7, Surface, Win Phone 7) that displays the content in an easy-to-navigate UI. Let’s use the Kinect to hover a hand around the UI and push navigation buttons! Here’s where the story begins.
One of the First Metro Apps to Use Kinect
Applications that are written to run in Windows 8 Metro are built against the new Windows Runtime (WinRT) API, which is the replacement for the old Win32 API that we’ve used since Windows 3.0. The problem when it comes to existing code is that assemblies written in .NET are not runtime compatible with WinRT (which is native code). There is a lot of equivalent functionality in WinRT, but you have to port existing source code over, make changes where necessary, and compile specifically against WinRT. Since the Kinect SDK is a set of .NET assemblies, you can’t just reference it in your WinRT / Metro app and start partying with the Kinect API. So we had to come up with some other way…
You CAN write a .NET 4.5 application in Windows 8, using Visual Studio 11 and it will run on the “desktop” side of the fence (alternate environment from the Metro UI, used for running legacy apps). So we decided to take advantage of this and write a “Service” UI that will run in the classic desktop environment, connect to the Kinect and receive all the data from it, and then furnish that data out to a client running in the Metro side. The next issue was – how to get the data over to our Kiosk app running in Metro? Enter web sockets. There is a native implementation of web sockets in the WinRT framework and we can use that to communicate on a socket channel over to the .NET 4.5 desktop which can reply to the client (Metro) socket with the Kinect data.
Some Bumps in the Road
Writing the socket implementation was not conceptually difficult. We just want the client to poll at a given frame rate, asking for data, and the service will return simple Kinect skeleton right-hand position data. We want to open the socket, push a “request” message across to the service, and the service will write binary data (a few doubles) back to the caller. When pushing bytes across a raw socket, obviously the way you write and read the data on each side must match. The first problem we ran into was that the BinaryWriter in the .NET 4.5 framework was writing data differently than the DataReader in WinRT was receiving the data.
As with any pre-release software from MIS, there is hardly any documentation on any of these APIs. Through a ton of trial and error, I found that I had to set the Unicode and Byte Order settings on each side to something that would match. Note the highlighted lines in the following code snippets.
// Send data from the service side
using ( MemoryStream ms = new MemoryStream() )
{
using ( BinaryWriter sw = new BinaryWriter( ms, new UnicodeEncoding() ) )
{
lock ( _avatarPositionLock )
{
sw.Write( _lastRightHandPosition.TrackingState );
sw.Write( _lastRightHandPosition.X );
sw.Write( _lastRightHandPosition.Y );
}
}
Send( ms.GetBuffer() );
}
// Receive data in the client
DataReader rdr = e.GetDataReader();
// bytes based response. 3 ints in a row
rdr.UnicodeEncoding = UnicodeEncoding.Utf16LE;
rdr.ByteOrder = ByteOrder.LittleEndian;
byte[] bytes = new byte[rdr.UnconsumedBufferLength];
var data = new JointPositionData();
var state = rdr.ReadInt16();
Enum.TryParse<JointTrackingState>( state.ToString(), out data.TrackingState );
data.X = rdr.ReadDouble();
data.Y = rdr.ReadDouble();
UpdatePositionData( data );
Once I got the socket channel communicating simple data successfully, we were off and running. We built a control called HoverButton that just checks whether the Kinect position data is within its bounds, and if so, starts an animation to show the user they’re over the button. If they hover long enough, we fire the Command on the button.
The next problem was connectivity from the client to “localhost” which is where the service is running (just over in the desktop environment). Localhost is a valid address, but I would keep getting refused connections. Finally re-read the setup instructions for the “Dot Hunter” Win8 SDK sample which tells about a special permission that’s required for a Win8 app to connect to localhost.
Open a command prompt as administrator and enter the following command (substitute your package name for the last param):
CheckNetIsolation LoopbackExempt -a -n=interknowlogy.kiosk.win8_r825ekt7h4z5c
There is no indication that it worked – I assume silence is golden here. (I still can’t find a way to list all the packages that have been given this right, in case you ever wanted to revoke it.)
Finally, a couple other minor gotchas: the service UI has to be running as administrator (to open a socket on the machine), and the Windows Firewall must be turned OFF. Now we have connectivity!
What’s Next?
Beyond those two problems, the rest was pretty straight forward. We’re now fiddling with various performance settings to achieve the best experience possible. The skeleton data is available from the Kinect on the desktop side at only about 10 frames per second. We think this lower rate is mostly due to the slower hardware in the Samsung Developer Preview device we got from BUILD. Given that speed, the Metro client is currently asking for Kinect data from the service at 15 frames per second. We are also working on better smoothing algorithms to prevent a choppy experience when moving the hand cursor around the Metro UI.
