Multimedia and Security
englisch

Guide-IT:

A software tool towards low-cost multimodal interfaces for multimedia information systems on smartphones.

Guide-IT is a prototype system for an audio, text and visual-based information system for interior landmarks. The software can be downloaded and installed on state-of-the art smartphones supporting MIDP 2.0 and Java MIDlets, allowing to combine a highly familiar device with a new HCI scheme for a sample application.

The system has been developed by the Biometric & HCI Research Group lead by Dr. Claus Vielhauer as part of the Advanced Multimedia and Security Lab team of the Otto-con-Guericke University Magdeburg lead by Prof. Dr. Jana Dittmann. It has been awarded by the Prizes

Encouragement for minority gender researcher Prize of the 2005 Loco Mummy Contest.

and
Innovation award of the 2006 Loco Mummy Contest.

The contest has been organized by the EU Network of Excellence SIMILAR. Please refer to the documentation below for further details.

For further inquiries, please contact:

Claus.Vielhauer@iti.cs.uni-magdeburg.de

or

Sandra.Gebbensleben@iti.cs.uni-magdeburg.de



Image otto

Department of Computer Science
Institute of Technical and Business Information Systems



Guide-IT:

Practical Lab Course

Mobile Multimedia


Author:
Dirk Schumann
July 4, 2005


University of Magdeburg
Department of Computer Science
PO-Box 4120, D-39016 Magdeburg
GERMANY

1 Introduction

Nowadays more and more people around the world are using mobile devices such as cell phones, smartphones or PDAs. At the same time these devices are getting smarter and more powerful. Thus the possible fields of application for such devices are growing. They are no longer just phones with a simple calculator and a calendar. A special Java Runtime Environment for resource-limited devices is available on almost every new cell phone. Technologies like Bluetooth or the infrared port make the deployment of new Java programs a piece of cake. Using these technologies, the following report demonstrates the implementation and function of a museum-guide.

2 State-of-the-Art

Audio-guides are pretty common in providing people with information on certain exhibits or sightseeings. They are used in many museums, observatories or as an electronic city guide. In contrast to a printed guide, people do not have to search a booklet for additional information about a painting or building. Usually they just have to enter the number displayed on a tag and listen to the audio-guide. Several languages are available and users can rewind or forward the selected section and adjust the volume.

Typically no screen is installed on these devices. But sometimes it can be helpful to display additional information such as the location of the user and the exhibit. It could also be desired not just to hear the information, but also to display the spoken text. So why not simply using the visitors personal phone or PDA?

The following technologies are used to implement the audio guide:

  • MIDP and Java MIDlets
  • kXML as a special XML-implementation for mobile devices
  • MathFP for virtual machines with no support for floats and doubles

2.1 MIDP

Java is a well-known and widely-spreaded programming language. Java programs can be executed on any platform with a virtual machine. To run Java programs on cell phones or PDAs, it is necessary to have a virtual machine on these devices too. The Java programs are called MIDlets. The required virtual machine is a subset of the Java virtual machine for desktop PCs. It has been reduced to meet the limited resources of mobile devices. The runtime environment is called MIDP1. In 2000 MIDP 1.0 was specified and in 2001 the first devices implementing MIDP 1.0 began appearing.

This first implementation of the specification provided support for basic UI2 functionality, a primitive record store, and HTTP networking. More advanced media, graphics, networking, and security APIs were missing. But all these things are necessary for the development of games and mobile commerce applications. Thus in 2002 the MIDP 2.0 specification was released. First devices implementing MIDP 2.0 appeared in 2004.

This implementation uses MIDP 1.0 and should therefore run on any Java enabled device.

2.2 kXML

kXML is a compact library designed for use on mobile devices. It just uses 21 kB of memory, which is especially important when using MIDP 1.0 devices as they usually do not have much memory for Java programs. One of the advantages of kXML is the way of parsing the XML-data. It can use a so called pull-parser.

In contrast to a DOM3 parser, a pull parser goes through an XML file without keeping the whole file in memory. So if searching for a particular piece of data, which is in the middle of the XML document, the previous data does not need to reside in memory. The kXML4 library is available for download at http://kxml.sourceforge.net.

2.3 MathFP

Numerous of the first devices supporting Java, do not provide support for float or double datatypes. For these devices MathFP allows calculations involving fractions. Exact calculations are needed for example to calculate graphical values. Most of the operations are based on bit shifts which are the fastest machine instructions. The current MathFP class is just over 4kb and provides most of the math functions included the Java 2 java.lang.Math class. The MathFP5 package can be obtained from here.

3 Creating and Deploying the Files

Before the MIDlet can be deployed on a mobile device, it is necessary to prepare some images and sound file and do some configuration.

3.1 Preparation

In order to make the MIDlet runable on almost all Java supporting devices, the image files should not be larger than 128 x 92. Depending on the targeted devices, this size can be adopted of course. The files have to be in PNG format, which is available in numerous graphic programs. The image should show something like a ground or site plan (see Figure 1). Small circles will be drawn on defined hot spots later. Thus the image should not be too crowded.
 
Figure 1:Ground plan sample
Image sample-image

An unlimited number of hot spots can be defined for each image. Hot spots represent a point of interest to the user (e.g. there can be several paintings in a room). For each hot spot a text has to be defined. The text should give the user more information about an exhibit. This text is later displayed on the screen of the mobile device.

To let the user also listen to the displayed text, audio files have to be created. For the audio files the AMR format is used. The easiest way to get AMR files, is to record the text using a microphone. The resulting file is usually a WAV file. If this WAV file is not yet in 8000Hz Mono, it has to be converted to this sampling rate. This can be done with the Audacity software which is available for free here. Using the MIKSOFT Mobile AMR Converter (which is also freely available under http://www.miksoft.net/mobileAMRconverter.htm), the WAV files are converted into AMR format. This file format produces very small sound files and at the same time ensures a good sound quality on mobile devices.


3.2 XML Configuration File

Now that all image and sound files are available and the text for each hot spot is specified, the XML configuration file can be created. The following example shows what to do, when having two exhibition rooms with one room having two exhibits and the other room having three ones. The two rooms will be two different screens on the mobile device. In addition to these exhibition screens, one welcome-screen and one exit-screen can be defined.

<?xml version="1.0"?>

<configuration>

  <screen type="welcome" id="1">

    <message>Welcome to the Museum of Modern Art!</message>

  </screen>



  <screen type="map" id="2" image="ground-plan-01.png">

    <hotspot xpos="60" ypos="35" id="1" sound="exhibit-01.amr">

      <message>This painting is called ... </message>

    </hotspot>

    <hotspot xpos="30" ypos="55" id="2" sound="exhibit-02.amr">

      <message>You are now in front of ... </message>

    </hotspot>

  </screen>



  <screen type="map" id="3" image="ground-plan-02.png">

    <hotspot xpos="20" ypos="25" id="3" sound="exhibit-03.amr">

      <message>This is the world-famous ... </message>

    </hotspot>

    <hotspot xpos="50" ypos="15" id="4" sound="exhibit-04.amr">

      <message>In 1956 this sculpture was made by ... </message>

    </hotspot>

    <hotspot xpos="40" ypos="65" id="5" sound="exhibit-05.amr">

      <message>The artist ... </message>

    </hotspot>

  </screen>



  <screen type="exit" id="4">

    <message>We hope you enjoyed your visit at the Museum of Modern Art.</message>

  </screen>

</configuration>

Sample XML configuration file

As already mentioned there are three types of screens: the regular map-screen, the welcome-screen and the exit-screen. Each screen is identified by a unique number. While welcome and exit-screens just have a message tag, map-screens include hot spots. These hot spots are also identified by an id. The position of a hot spot is defined by an x and y coordinate. These coordinates depend on the corresponding image file (see Figure 2). The sound attribute defines the sound file to be played and the message tag includes the displayed text.

 
Figure 2: Position of a hot spot
Image sample-image-hotspot

3.3 Deployment

The last step before actually deploying the files, is to create some directories. First the root directory has to be created. For this example 'ExhibitionMidlet' is used. A subdirectory named 'res' should contain the needed image and sound files. Another directory on the same level as the ExhibitionMidlet directory called 'meta-inf' contains the manifest file. The XML configuration file as well as the ExhibitionMidlet class file should be located directly under the 'ExhibitionMidlet' directory.

Now the Jar file has to be created. An easy way to do this, is to create a regular ZIP file. This ZIP file should contain all previously created directories. All of the remaining Java classes are also copied in the ZIP file but are not included in one of the mentioned directories. Finally the ZIP file extension has to be changed to JAR.

Some devices require to have an additional file to be stored. A so called JAD6 file should contain the following information:

MIDlet-Version: 1.0

MIDlet-Vendor: Vendor Name

MIDlet-Jar-URL: ExhibitionMIDlet.jar

MicroEdition-Configuration: CLDC-1.0

MIDlet-1: ExhibitionMidlet, , ExhibitionMidlet.ExhibitionMidlet

MicroEdition-Profile: MIDP-1.0

MIDlet-Jar-Size: 377012

MIDlet-Name: ExhibitionMIDlet

Sample JAD file

The entry 'MIDlet-1' means, that the name of the first MIDlet (there is only one in this example) is 'ExhibitionMidlet'. An image can be the second argument but no image is specified here. As a last argument, the actual class must be defined. In this example the package name must be part of the string, because the ExhibitionMidlet class belongs to the package 'ExhibitionMidlet'. 'MIDlet-Jar-Size' defines the exact size in bytes of the JAR file. This argument is very important, because it varies depending on the size of the used image and sound files as well as the configuration file. The JAD file must have the same name as the JAR file.

The created JAR file can now be transferred to the mobile device. This can be done either via a cable or a wireless connection, such as infrared or Bluetooth. If the mobile device requires a JAD file, it has to be transferred too. The location is often chosen by the mobile device itself. Only some devices require to store the JAR and JAD file in a special Java application folder. Further information on this topic can be found in the handbook of the appropriate device.

4 Using the Audio Guide

After the application has been deployed on the device, it can be executed. While going through the exhibition the user can find identification numbers at selected exhibtis. Figure 3 shows the example of how the procedure could be done. First the user stands infront of the exhibit and types in the related identification number into the audio-guide. On the sreen the user can see now a map with his marked position and below the map the audio information as textual representation. The user can read the information or press the play button and hear the audio-information relatetd to the exhibit infront of him. In the fowllowing sub-sections of section 4 the procedure is described in more detail.

 
Figure 3: using the audio-guide
Image using the audio-guide

When the application is started, the first screen welcomes the user with the previously defined welcome message (see Figure 4).

 
Figure 4: Welcome screen
Image welcome-screen

4.1 Selecting the Exhibition

When standing in front of an exhibit, an identification number can be found. This number can be entered into a text box (see Figure 5).

 
Figure 5: Entering the exhibition number
Image number-input

Pressing on 'Ok' calls the next screen.

4.2 Exhibition Details

This screen presents detailed information on the selected exhibition (see Figure 6). All of the shown information was defined in the XML configuration file (see section 3.2). In the upper part of the screen, the specified image is shown. Depending on the coordinates of the hot spot, a red spot is painted on the location. This spot represents the users location in a room or area.

The section below the image prints the message text. Further information on the exhibit is stated here. This text can be scrolled up and down to read long messages.

 
Figure 6: Exhibition details
Image map-screen

At the bottom, a menu can be found. The design and position of the menu items can differ. This depends on the individual device. The menu items include an item named 'Play'. This menu item plays the specified sound file for the selected hot spot. It changes to 'Pause', when the sound file is played. Pressing 'Pause' stops playing the audio file.

Going back to the exhibition selection screen and pressing on 'Exit' finally calls the exit message (see Figure 7).

 
Figure 7: Exit screen
Image exit-screen

5 Tested Platforms

The software has been deployed and tested on the following devices:


Table 1: Tested mobile devices
Brand Model MIDP Version Successful
Motorola V3 2.0
Nokia 6230 2.0
Siemens SL65 2.0
Sony Ericsson T630 1.0


6 Future Work

Although the current version of this software is running, there are still some things to do. For example the creation of the xml configuration file is not very user friendly. If the xml file is not well-formed or a tag is not at the right position, an error will occur not before executing the program on the mobile device.

To improve this, a separate program with a graphical user interface could be written. With this software, the only thing for the user to do is to create the image and sound files and enter some text. The location of the exhibits can be chosen within this user interface. A simple button click will then create the xml configuration file.

Using the same graphical user interface, it could also be possible to create the JAR and JAD file automatically. The whole process of creating all the directories and the ZIP file would then be unnecessary.

Another possible improvement is concerning the encryption of the sound and image files. If a user is extracting the sound and image files from the JAR file, encryption prevents the user from using the content in another way. The encrypted files have to be stored in the JAR file and are decrypted on the device during runtime. Using encryption requires devices with adequate computing power.


Acknowledgements

I would like to thank my supervisors Claus Vielhauer and Jana Dittmann for their support during this project. Special thanks to Sandra Gebbensleben for the fruitful discussions and for providing me with sample sound files.


References

MIDP. Wikipedia. (2005).
Retrieved on Juli 1, 2005. [http://en.wikipedia.org/wiki/MIDP]

MIDlet. Wikipedia. (2005).
Retrieved on Juli 1, 2005. [http://en.wikipedia.org/wiki/Midlet]

DOM. Wikipedia. (2005).
Retrieved on Juli 1, 2005. [http://en.wikipedia.org/wiki/DocumentObjectModel]

About kXML. (2005)
Retrieved on Juli 1, 2005. [http://kxml.sourceforge.net/about.shtml]

MathFP - Fixed Point Integer Math. (2005).
Retrieved on Juli 1, 2005. [http://home.rochester.rr.com/ohommes/MathFP]

J2ME in a Nutshell - A Desktop Quick Reference. (2001).
Kim Topley. O'Reilly, 2002

Wireless Java - Developing with Java 2 Micro Edition (2001).
Jonathan Knudsen. Apress, 2001

About this document ...

This document was generated using the LaTeX2HTML translator Version 2002-2-1 (1.71)

Copyright 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
Copyright 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were:
latex2html -split 0 -no_navigation -show_section_numbers GuideIT.tex

The translation was initiated by Dirk on 2005-07-04


Footnotes

... MIDP1
MIDP is the abbreviation for Mobile Information Devices Profile.
... UI2
UI - User Interface
... DOM3
Document Object Model - The entire document tree is kept in memory with each tag separated into node objects. This tree can be traversed and the data can be retrieved as needed.
... kXML4
kXML is released under the BSD license.
... MathFP5
The MathFP library may be used for any non-commercial purpose without acknowledgment. MathFP is distributed under a Liberal Source Software (LSS) license for commercial purposes.
... JAD6
Java Application Descriptor



Impressum | Haftungsausschluss | Datenschutzerklärung