MAJORDOME : Assistant personnel et Messagerie unifiée G. Chollet, L. Likforman, K. Hallouli, N. Azzabou, S.S. Lin, S. Renouard, M. Sigelle, F. Yvon Journée multimédia - Conseil Scientifique GET - 9/10/2003

Page 2Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Garde la mémoire n 2. Communique avec vos interlocuteurs n 3. Répond à vos questions Le MAJORDOME peut être centralisé (serveur d'entreprise), mobile (sur PDA ou PC-portable) ou distribué. Le MAJORDOME est un assistant intelligent personnel qui :

Page 3Journée MM / Conseil Scientifique GET - 9/10/2003 Majordome is a distributed Personal Digital Assistant nIt is your digital slave. It is personal. It remembers everything that you told him. nIt uses resources from you mobile (wireless) device, from your home, from your office, from the Internet, from the environment, … nYou interact with him using voice, pen, graphics, …

Page 4Journée MM / Conseil Scientifique GET - 9/10/2003 Interactions with your Majordome nMajordome recognizes your identity, your voice, your handwriting,... nHis speech recognizer is adapted to your voice, nHis handwriting recognizer is adapted to your writing style, nHe can speak to you, nHe can display information for you, nHe can talk with other persons either locally or over the phone.

Page 5Journée MM / Conseil Scientifique GET - 9/10/2003 What Majordome does for you ? nAnswers your phone, nReceives and interpret your faxes, your s, … nSupplements your memory (address book, agenda, bookmarks, alarm clock, health record, bank account, documentation, …) nServes as an interface between you and the (digital) world, nSearches the web, internet forums, … nControls your home, your car, your children, your parents, …

Page 6Journée MM / Conseil Scientifique GET - 9/10/2003 n - Répond au téléphone, n - Reçoit vos télécopies, n - Enregistre et interprète vos messages, n - Accède à votre messagerie électronique, n - Vérifie votre identité Le MAJORDOME :

Page 7Journée MM / Conseil Scientifique GET - 9/10/2003 Majordomes Functionalities Speaker verification Dialogue Routing Updating the agenda Automatic summary Voice Fax

Page 8Journée MM / Conseil Scientifique GET - 9/10/2003 Overview of Majordome nBackground tasks (server-side only): –sorting and filtering messages from different sources ( , voice, fax, SMS,…); –extracting relevant information for reporting to user (names of senders, subject,…). nDialogue with the user: over phone or Web. –The system presents the state of the mailbox, the type of messages, their sender, subject, and may sum them up or read them on request; –The users access their mailbox, addressbook, time schedule, or Web addresses.

Page 9Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Point –Sous-point –Sous point n 2. Point –Sous-point –Sous point n 3. Point –Sous-point –Sous point Traitement des télécopies

Page 10Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Point –Sous-point –Sous point n 2. Point –Sous-point –Sous point n 3. Point –Sous-point –Sous point Traitement des messages textuels

Page 11Journée MM / Conseil Scientifique GET - 9/10/2003 Content Extraction in Majordome Overall Objective: Quick detection of short information elements for Message Filtering and Reporting to User Functional position of this processing phase: –Server-side, event-oriented, background task –subsequent and/or parallel to speech recognition (voice messages) or image processing (faxes); previous to text summarizing

Page 12Journée MM / Conseil Scientifique GET - 9/10/2003 Useful applications (1) Name/Date/Subject identification (this task specifically useful for fax and voice messages: no standardized fields for storing this information) –You have 1 fax message from Mrs Diaconu about attending the Barcelona meeting… Backup information: users addressbook (PABX info yields senders phone number)

Page 13Journée MM / Conseil Scientifique GET - 9/10/2003 Useful applications (2) Message filtering: –You have received 14 personal messages, among which 3 messages from friends, 6 requests from students or colleagues, and 5 spam messages; you have received 26 mailing list messages, among which 3 call for papers, 11 conference announcements, and 12 other. Backup information: RFC-822 From andSubject fields.

Page 14Journée MM / Conseil Scientifique GET - 9/10/2003 Techniques (1) Text statistics measures: –Frequency of occurrence of certain words/morphological categories/syntactical structures in different types of messages E.g. ratio noun/verb frequency higher in technical texts; style markers specific to some text genres (e.g. frequent use of ! or $ in advertisements; loose style abbreviations like CU, IMHO in English, or A+ in French)

Page 15Journée MM / Conseil Scientifique GET - 9/10/2003 Techniques (2) Text skimming: –Spotting good candidates for specific word types (e.g. proper names): selecting capitalized words… –… comparing with entries in common first names / family names database, and/or… –… using local grammars to disambiguate other cases.

Page 16Journée MM / Conseil Scientifique GET - 9/10/2003 Techniques (3) Merging visual clues and textual clues for mutual reinforcement of identification probability. E.g. Probability of an unidentified, capitalized character string to be the proper name of a faxs sender increases if it stands alone on a line at the top of the image.

Page 17Journée MM / Conseil Scientifique GET - 9/10/2003 Content Extraction: Current Developments Toolbox for text statistics (word frequency, contextual windows, co-occurrence frequency…) Tool for determining fuzzy membership to a given class of words Tool for determining document language and segmenting multilingual documents

Page 18Journée MM / Conseil Scientifique GET - 9/10/2003 Content Extraction: Future Developments Text categorization module for message sorting and filtering Text genre database with (user-controlled) learning capabilities

Page 19Journée MM / Conseil Scientifique GET - 9/10/2003 Image pseudo words extraction H/P discrimination Header candidates selection OCRd version Logical Pair extraction sender name extraction lexicon & logical classes spatial cues sender name Name Block Location in Facsimile Images

Page 20Journée MM / Conseil Scientifique GET - 9/10/2003 from to LP SR RR tofrom LP SRRR Sender and Recipient Image Regions

Page 21Journée MM / Conseil Scientifique GET - 9/10/2003 Database of Facsimile Images Campaign for receiving fax images ->30 faxes Existing database -> 40 faxes Paper database -> 40 faxes We ask partners to get also faxes (> 10 each)

Page 22Journée MM / Conseil Scientifique GET - 9/10/2003 You have received a new fax from Sender name : HTML File construction (handprinted names)

Page 23Journée MM / Conseil Scientifique GET - 9/10/2003 Original Image of Fax

Page 24Journée MM / Conseil Scientifique GET - 9/10/2003 LP SN Logical Pair (LP) and Sender Name (SN) location

Page 25Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Reconnaissance de l'appelant –Noms propres –Noms eppelés n 2. Vérification du locuteur n 3. Navigation vocale dans la messagerie –Sous-point –Sous point n 4. Synthèse vocale –Sous-point –Sous point Interface téléphonique

Page 26Journée MM / Conseil Scientifique GET - 9/10/2003 Voice technology in Majordome nServer side background tasks: continuous speech recognition applied to voice messages upon reception –Detection of senders name and subject nUser interaction: –Identification of the speaker (and Verification if necessary) –Speech recognition (receiving users commands through voice interaction) –Text-to-speech synthesis (reading text summaries, s or faxes)

Page 27Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Point –Sous-point –Sous point n 2. Point –Sous-point –Sous point n 3. Point –Sous-point –Sous point Interface SMS / MMS

Page 28Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Point –Sous-point –Sous point n 2. Point –Sous-point –Sous point n 3. Point –Sous-point –Sous point Interface PDA

Page 29Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Point –Sous-point –Sous point n 2. Point –Sous-point –Sous point n 3. Point –Sous-point –Sous point Démonstration

Page 30Journée MM / Conseil Scientifique GET - 9/10/2003 n 1. Point –Sous-point –Sous point n 2. Point –Sous-point –Sous point n 3. Point –Sous-point –Sous point Perspectives