Search results for: multimodal data

Authors: Sharifa Alshahrani

Abstract:

Comics is a multimodal text wherein meaning is created by taking in all modes of expression at once. It uses two different semiotic modes, the verbal and the visual modes, together to make meaning and these different semiotic modes can be socially and culturally shaped to give meaning. Therefore, comics translation cannot treat comics as a monomodal text by translating only the verbal mode inside or outside the speech balloons as the cultural differences are encoded in the visual mode as well. Due to the development of the internet and editing software, comics translation is not anymore confined to the publishing houses and official translation as scanlation, or the fan translation took the initiative in translating comics for being emotionally attracted to the culture and genre. Scanlation is carried out by volunteering fans who translate out of passion. However, quality is one of the debatable issues relating to scanlation and fan translation. This study will investigate how the dynamic multimodal relationship in comics is exploited and interpreted in the translation by exploring the translation strategies and procedures adopted by the publishing houses and scanlation in interpreting comics into Arabic using three analytical frameworks; cultural references model, multimodal relation model and translation strategies and procedures models.

Keywords: comics, multimodality, translation, scanlation

Procedia PDF Downloads 185

Authors: Dike Ibegbulem

Abstract:

This paper interrogates the chances of Nigeria achieving its target of making the list of the first 70 countries in World Bank’s Ease of Doing Business (EoDB) rankings by the year 2023. That is, in light of existing conflicts in policies relating to the door-to-door carriage of goods and multimodal transport operations (MTOs) in the country. Drawing on the famed Legal Origins theory plus data from World Bank; and using Singapore as a touchstone, the paper unveils how amongst the top-ranked Commonwealth jurisdictions, positive correlations have been recorded over the past years between certainty in their policies on MTOs on the one hand; and their Enforcing Contracts (EC) and Doing Business (DB) indices on the other. The paper postulates that to increase Nigeria’s chances of achieving her DB2023 objective, legislative and curial policies on MTOs and door-to-door carriage of goods have to be realigned in line with prevailing policies in highly-ranked Commonwealth jurisdictions of the Global North. Her appellate courts, in particular, will need some unshackling from English pedigrees which still delimit admiralty jurisdiction to port-to-port shipping, to the exclusion of door-to-door carriage of goods beyond navigable waters. The paper identifies continental and domestic instruments, plus judicial precedents, which provide bases for expanding admiralty jurisdiction to adjudication of claims derived from door-to-door or multimodal transport contracts and other allied maritime-plus contracts. It prescribes synergy between legislative and curial policies on MTOs and door-to-door carriage of goods as species of admiralty – an emerging trend in top-ranked Commonwealth jurisdictions of the Global North.

Keywords: admiralty jurisdiction, legal origins, world bank, ease of doing business, enforcing contracts, multimodal transport operation, door-to-door, carriage of goods by sea, combined transport shipping

Procedia PDF Downloads 52

Authors: Mariam Waheed Mekheimar

Abstract:

Nascent research is conducted to the advancement of discourse analysis to include different modes as images, sound, and text. The focus of this study will be to elucidate how images are embedded with texts in an audio-visual medium as cinema to send political messages; it also seeks to broaden our understanding of politics beyond a relatively narrow conceptualization of the 'political' through studying non-traditional discourses as the cinematic discourse. The aim herein is to develop a systematic approach to film analysis to capture political meanings in films. The method adopted in this research is Multimodal Discourse Analysis (MDA) focusing on embedding visuals with texts. As today's era is the era of images and that necessitates analyzing images. Drawing on the writings of O'Halloran, Kress and Van Leuween, John Bateman and Janina Wildfeuer, different modalities will be studied to understand how those modes interact in the cinematic discourse. 'People at the top movie' is selected as an example to unravel the political meanings throughout film tackling the cinematic representation of the notion of social justice.

Keywords: Egyptian cinema, multimodal discourse analysis, people at the top, social justice

Procedia PDF Downloads 386

Authors: Simon Arunga, Teddy Kwaga, Rita Kageni, Michael Gichangi, Nyawira Mwangi, Fred Kagwa, Rogers Mwavu, Amos Baryashaba, Luis F. Nakayama, Katharine Morley, Michael Morley, Leo A. Celi, Jessica Haberer, Celestino Obua

Abstract:

Purpose: The main aim for creating the Multimodal Database of Retinal Images for Africa (MoDRIA) was to provide a publicly available repository of retinal images for responsible researchers to conduct algorithm development in a bid to curb the challenges of ophthalmic artificial intelligence (AI) in Africa. Methods: Data and retina images were ethically sourced from sites in Uganda and Kenya. Data on medical history, visual acuity, ocular examination, blood pressure, and blood sugar were collected. Retina images were captured using fundus cameras (Foru3-nethra and Canon CR-Mark-1). Images were stored on a secure online database. Results: The database consists of 7,859 retinal images in portable network graphics format from 1,988 participants. Images from patients with human immunodeficiency virus were 18.9%, 18.2% of images were from hypertensive patients, 12.8% from diabetic patients, and the rest from normal’ participants. Conclusion: Publicly available data repositories are a valuable asset in the development of AI technology. Therefore, is a need for the expansion of MoDRIA so as to provide larger datasets that are more representative of Sub-Saharan data.

Keywords: retina images, MoDRIA, image repository, African database

Procedia PDF Downloads 83

Authors: Teodora Vukovic, Christoph Hottiger, Noah Bubenhofer

Abstract:

The development of VIAN-DH aims at bridging two linguistic approaches: conversation analysis/interactional linguistics (IL), so far a dominantly qualitative field, and computational/corpus linguistics and its quantitative and automated methods. Contemporary IL investigates the systematic organization of conversations and interactions composed of speech, gaze, gestures, and body positioning, among others. These highly integrated multimodal behaviour is analysed based on video data aimed at uncovering so called “multimodal gestalts”, patterns of linguistic and embodied conduct that reoccur in specific sequential positions employed for specific purposes. Multimodal analyses (and other disciplines using videos) are so far dependent on time and resource intensive processes of manual transcription of each component from video materials. Automating these tasks requires advanced programming skills, which is often not in the scope of IL. Moreover, the use of different tools makes the integration and analysis of different formats challenging. Consequently, IL research often deals with relatively small samples of annotated data which are suitable for qualitative analysis but not enough for making generalized empirical claims derived quantitatively. VIAN-DH aims to create a workspace where many annotation layers required for the multimodal analysis of videos can be created, processed, and correlated in one platform. VIAN-DH will provide a graphical interface that operates state-of-the-art tools for automating parts of the data processing. The integration of tools that already exist in computational linguistics and computer vision, facilitates data processing for researchers lacking programming skills, speeds up the overall research process, and enables the processing of large amounts of data. The main features to be introduced are automatic speech recognition for the transcription of language, automatic image recognition for extraction of gestures and other visual cues, as well as grammatical annotation for adding morphological and syntactic information to the verbal content. In the ongoing instance of VIAN-DH, we focus on gesture extraction (pointing gestures, in particular), making use of existing models created for sign language and adapting them for this specific purpose. In order to view and search the data, VIAN-DH will provide a unified format and enable the import of the main existing formats of annotated video data and the export to other formats used in the field, while integrating different data source formats in a way that they can be combined in research. VIAN-DH will adapt querying methods from corpus linguistics to enable parallel search of many annotation levels, combining token-level and chronological search for various types of data. VIAN-DH strives to bring crucial and potentially revolutionary innovation to the field of IL, (that can also extend to other fields using video materials). It will allow the processing of large amounts of data automatically and, the implementation of quantitative analyses, combining it with the qualitative approach. It will facilitate the investigation of correlations between linguistic patterns (lexical or grammatical) with conversational aspects (turn-taking or gestures). Users will be able to automatically transcribe and annotate visual, spoken and grammatical information from videos, and to correlate those different levels and perform queries and analyses.

Keywords: multimodal analysis, corpus linguistics, computational linguistics, image recognition, speech recognition

Procedia PDF Downloads 75

Authors: Yagya Raj Pandeya, Joonwhoan Lee

Abstract:

The dynamic behavior of music and video makes it difficult to evaluate musical instrument playing in a video by computer system. Any television or film video clip with music information are rich sources for analyzing musical instruments using modern machine learning technologies. In this research, we integrate the audio and video information sources using convolutional neural network (CNN) and pass network learned features through recurrent neural network (RNN) to preserve the dynamic behaviors of audio and video. We use different pre-trained CNN for music and video feature extraction and then fine tune each model. The music network use 2D convolutional network and video network use 3D convolution (C3D). Finally, we concatenate each music and video feature by preserving the time varying features. The long short term memory (LSTM) network is used for long-term dynamic feature characterization and then use late fusion with generalized mean. The proposed network performs better performance to recognize the musical instrument using audio-video multimodal neural network.

Keywords: multimodal, 3D convolution, music-video feature extraction, generalized mean

Procedia PDF Downloads 187

Authors: Jiandong Lv, Xingang Wang, Cuiling Shao

Abstract:

The rapid development of social media platforms has made it one of the important news sources. While it provides people with convenient real-time communication channels, fake news and rumors are also spread rapidly through social media platforms, misleading the public and even causing bad social impact in view of the slow speed and poor consistency of artificial rumor detection. We propose an end-to-end rumor detection model-TIMF, which captures the dependencies between multimodal data based on the interactive attention mechanism, uses a transformer for cross-modal feature sequence mapping and combines hybrid fusion strategies to obtain decision results. This paper verifies two multi-modal rumor detection datasets and proves the superior performance and early detection performance of the proposed model.

Keywords: hybrid fusion, multimodal fusion, rumor detection, social media, transformer

Procedia PDF Downloads 193

Authors: Tomasz Sapiński, Dorota Kamińska, Adam Pelikant, Egils Avots, Cagri Ozcinar, Gholamreza Anbarjafari

Abstract:

People express emotions through different modalities. Integration of verbal and non-verbal communication channels creates a system in which the message is easier to understand. Expanding the focus to several expression forms can facilitate research on emotion recognition as well as human-machine interaction. In this article, the authors present a Polish emotional database composed of three modalities: facial expressions, body movement and gestures, and speech. The corpora contains recordings registered in studio conditions, acted out by 16 professional actors (8 male and 8 female). The data is labeled with six basic emotions categories, according to Ekman’s emotion categories. To check the quality of performance, all recordings are evaluated by experts and volunteers. The database is available to academic community and might be useful in the study on audio-visual emotion recognition.

Keywords: body movement, emotion recognition, emotional corpus, facial expressions, gestures, multimodal database, speech

Procedia PDF Downloads 323

Authors: Pingping Lin, Xudong Luo, Yifan Fan

Abstract:

Sentiment analysis is a very active research topic. Every day, Facebook, Twitter, Weibo, and other social media, as well as significant e-commerce websites, generate a massive amount of comments, which can be used to analyse peoples opinions or emotions. The existing methods for sentiment analysis are based mainly on sentiment dictionaries, machine learning, and deep learning. The first two kinds of methods rely on heavily sentiment dictionaries or large amounts of labelled data. The third one overcomes these two problems. So, in this paper, we focus on the third one. Specifically, we survey various sentiment analysis methods based on convolutional neural network, recurrent neural network, long short-term memory, deep neural network, deep belief network, and memory network. We compare their futures, advantages, and disadvantages. Also, we point out the main problems of these methods, which may be worthy of careful studies in the future. Finally, we also examine the application of deep learning in multimodal sentiment analysis and aspect-level sentiment analysis.

Keywords: document analysis, deep learning, multimodal sentiment analysis, natural language processing

Procedia PDF Downloads 131

Authors: Bjorn-Ole Meyer, Dominik Marti, Peter E. Andersen

Abstract:

A multimodal imaging system, combining spectrally resolved multiphoton microscopy (MPM) and optical coherence microscopy (OCM) is demonstrated. MPM and OCM are commonly integrated into multimodal imaging platforms to combine functional and morphological information. The MPM signals, such as two-photon fluorescence emission (TPFE) and signals created by second harmonic generation (SHG) are biomarkers which exhibit information on functional biological features such as the ratio of pyridine nucleotide (NAD(P)H) and flavin adenine dinucleotide (FAD) in the classification of cancerous tissue. While the spectrally resolved imaging allows for the study of biomarkers, using a spectrometer as a detector limits the imaging speed of the system significantly. To overcome those limitations, an OCM setup was added to the system, which allows for fast acquisition of structural information. Thus, after rapid imaging of larger specimens, navigation within the sample is possible. Subsequently, distinct features can be selected for further investigation using MPM. Additionally, by probing a different contrast, complementary information is obtained, and different biomarkers can be investigated. OCM images of tissue and cell samples are obtained, and distinctive features are evaluated using MPM to illustrate the benefits of the system.

Keywords: optical coherence microscopy, multiphoton microscopy, multimodal imaging, two-photon fluorescence emission

Procedia PDF Downloads 481

Authors: Moustafa Ibrahim Moustafa

Abstract:

The purpose of this study was to determine the immediate and long term effects of a multimodal program, with the addition of cervical sagittal curve restoration and forward head correction, on severity of dizziness, disability, frequency of dizziness, and severity of cervical pain. 72 patients with cervicogenic dizziness, definite hypolordotic cervical spine, and forward head posture were randomized to experimental or a control group. Both groups received the multimodal program, additionally, the study group received the Denneroll cervical traction. All outcome measures were measured at three intervals. The general linear model indicated a significant group × time effects in favor of experimental group on measures of anterior head translation (F=329.4 P < .0005), cervical lordosis (F=293.7 P < .0005), severity of dizziness (F=262.1 P < .0005), disability (F=248.9 P < .0005), frequency of dizziness (F=53.9 P < .0005), and severity of cervical pain (F=350.1 P < .0005). The addition of Dennroll cervical traction to a multimodal program can positively affect dizziness management outcomes.

Keywords: randomized controlled trial, traction, dizziness, cervical

Procedia PDF Downloads 279

Authors: Mohammad H. Taheri, David J. Brown, Nasser Sherkat

Abstract:

Engagement is one of the most important factors in determining successful outcomes and deep learning in students. Existing approaches to detect student engagement involve periodic human observations that are subject to inter-rater reliability. Our solution uses real-time multimodal multisensor data labeled by objective performance outcomes to infer the engagement of students. The study involves four students with a combined diagnosis of cerebral palsy and a learning disability who took part in a 3-month trial over 59 sessions. Multimodal multisensor data were collected while they participated in a continuous performance test. Eye gaze, electroencephalogram, body pose, and interaction data were used to create a model of student engagement through objective labeling from the continuous performance test outcomes. In order to achieve this, a type of continuous performance test is introduced, the Seek-X type. Nine features were extracted including high-level handpicked compound features. Using leave-one-out cross-validation, a series of different machine learning approaches were evaluated. Overall, the random forest classification approach achieved the best classification results. Using random forest, 93.3% classification for engagement and 42.9% accuracy for disengagement were achieved. We compared these results to outcomes from different models: AdaBoost, decision tree, k-Nearest Neighbor, naïve Bayes, neural network, and support vector machine. We showed that using a multisensor approach achieved higher accuracy than using features from any reduced set of sensors. We found that using high-level handpicked features can improve the classification accuracy in every sensor mode. Our approach is robust to both sensor fallout and occlusions. The single most important sensor feature to the classification of engagement and distraction was shown to be eye gaze. It has been shown that we can accurately predict the level of engagement of students with learning disabilities in a real-time approach that is not subject to inter-rater reliability, human observation or reliant on a single mode of sensor input. This will help teachers design interventions for a heterogeneous group of students, where teachers cannot possibly attend to each of their individual needs. Our approach can be used to identify those with the greatest learning challenges so that all students are supported to reach their full potential.

Keywords: affective computing in education, affect detection, continuous performance test, engagement, flow, HCI, interaction, learning disabilities, machine learning, multimodal, multisensor, physiological sensors, student engagement

Procedia PDF Downloads 66

Authors: Sijia Guo

Abstract:

The development of new technologies and the falling cost of high-speed Internet access have made it easier for institutes and language teachers to opt different ways to communicate with students at distance. The emergence of web-conferencing applications, which integrate text, chat, audio / video and graphic facilities, offers great opportunities for language learning to through the multimodal environment. This paper reports on data elicited from a Ph.D. study of using web-conferencing in the teaching of first-year Chinese class in order to promote learners’ collaborative learning. Firstly, a comparison of four desktop videoconferencing (DVC) tools was conducted to determine the pedagogical value of the videoconferencing tool-Blackboard Collaborate. Secondly, the evaluation of 14 campus-based Chinese learners who conducted five one-hour online sessions via the multimodal environment reveals the users’ choice of modes and their learning preference. The findings show that the tasks designed for the web-conferencing environment contributed to the learners’ collaborative learning and second language acquisition.

Keywords: computer-mediated communication (CMC), CALL evaluation, TBLT, web-conferencing, online Chinese teaching

Procedia PDF Downloads 282

Authors: Raheel Shakoor Siddiqui, Shahbaz Malik, Manikandar Srinivas Cheruvu, Sanjay Narayana Murthy, Livio DiMascio

Abstract:

Background: elderly traumatic hip fracture patients frequently present to trauma services globally. Rising low energy falls amongst an osteoporotic aging population is the commonest cause for injury. Hip fractures in this population are a major cause for severe pain, morbidity and mortality. The term hip fracture is interchangeable with neck of femur fracture, fractured neck of femur or proximal femur fracture. Hip fracture pain management protocols and guidelines suggest conventional analgesia, nerve block and opioid based treatment as rescue analgesia. There is a current global opioid crisis with overuse, abuse and dependence. Adverse opioid related complications in vulnerable elderly patients further adds to morbidity and mortality. Systematic reviews in literature have evidenced superiority of multimodal analgesia in osteoarthritic primary joint replacements compared to opioids however, this has not yet been conducted for elderly traumatic hip fracture patients. Aims: The primary aim of this systematic review is to provide standardised evidence following Cochrane and PRISMA guidance in determining advantages of perioperative multimodal analgesia over conventional opioid based treatments in elderly traumatic hip fractures. Methods: 5 databases were searched from January 2000-2023 which identified 8 randomised controlled trials and 446 total participants. These trials met defined PICOS eligibility criteria of patient mean age ≥ 65 years presenting with a unilateral traumatic fractured neck of femur for operative intervention. Analgesic intervention with perioperative multimodal analgesia has been compared to conventional opioid based analgesia. Outcomes of interest include, primarily, the change in postoperative opioid consumption within a 0-30 postoperative period and secondarily, the change in postoperative adverse events and complications. A qualitative synthesis has been performed due to clinical heterogenicity and variance amongst trials. Results: GRADE evidence of moderate quality supports perioperative multimodal analgesia leads to a reduction in postoperative opioid consumption however, low quality evidence supports a reduction of adverse effects and complications. Conclusion: Perioperative multimodal analgesia whether used preoperative, intraoperative and/or postoperative leads to a reduction in postoperative opioid consumption for elderly traumatic hip fracture patients. This review recommends the use of perioperative multimodal analgesia as part of hip fracture pain protocols however, caution and clinical judgement should be used as the risk of adverse effects may not be lower.

Keywords: trauma, orthopaedics, hip, fracture, neck of femur fracture, analgesia, multimodal analgesia, opioid

Procedia PDF Downloads 68

Authors: Rasha Ali Dheyab

Abstract:

In today's increasingly interconnected and technologically-driven world, communication has evolved beyond traditional verbal exchanges. This paper delves into the fascinating realm of multimodal communication, a dynamic field at the intersection of linguistics, gesture studies, and technology. The study of how humans convey meaning through a combination of spoken language, gestures, facial expressions, and digital platforms has gained prominence as our modes of interaction continue to diversify. This exploration begins by examining the foundational theories in linguistics and gesture studies, tracing their historical development and mutual influences. It further investigates the role of nonverbal cues, such as gestures and facial expressions, in augmenting and sometimes even altering the meanings conveyed by spoken language. Additionally, the paper delves into the modern technological landscape, where emojis, GIFs, and other digital symbols have emerged as new linguistic tools, reshaping the ways in which we communicate and express emotions. The interaction between traditional and digital modes of communication is a central focus of this study. The paper investigates how technology has not only introduced new modes of expression but has also influenced the adaptation of existing linguistic and gestural patterns in online discourse. The emergence of virtual reality and augmented reality environments introduces yet another layer of complexity to multimodal communication, offering new avenues for studying how humans navigate and negotiate meaning in immersive digital spaces. Through a combination of literature review, case studies, and theoretical analysis, this paper seeks to shed light on the intricate interplay between language, gesture, and technology in the realm of multimodal communication. By understanding how these diverse modes of expression intersect and interact, we gain valuable insights into the ever-evolving nature of human communication and its implications for fields ranging from linguistics and psychology to human-computer interaction and digital anthropology.

Keywords: multimodal communication, linguistics ., gesture studies., emojis., verbal communication., digital

Procedia PDF Downloads 50

Authors: Yi Meng, Yun Gao

Abstract:

Having spent significant periods studying and working in North America and Europe, we, as two Chinese art educators, have been profoundly shaped by both Eastern and Western cultures. Consequently, our ambition is to enrich students' learning experiences by delving into and merging both cultural perspectives for innovative, creative expressions. This exposition draws on our action research study on students' visual literacy practices in a visual literacy course at a prominent Chinese university. The central premise was to explore innovative art forms by cross-utilizing various aspects of diverse cultures. By examining distinct cultural elements, we encouraged students to break away from familiar approaches and forge new paths in their creative endeavors. In implementing our curriculum, we utilized a multimodal pedagogy that deviated from the predominant print-based presentations typically employed in our classroom settings. This pedagogical approach effectively encouraged students to critically analyze the artifact, imbue it with their understanding and perspectives, and then produce an original piece. This approach also motivated students to leverage the semiotic potential of various communicative modes to address diverse cultural issues through their multimodal designs. To demonstrate the potential for cultural amalgamation, we utilized the artwork of Hong Kong-based artist Tik Ka. His works epitomize the fusion of Chinese traditions with Western pop culture, which served as a visual and conceptual reference point for students. Seeing how these distinct cultural elements could coexist and enrich each other in Tik Ka's work was inspiring and motivating for the students. Taken together, these pedagogical strategies helped create a dialogical space where students could actively experience, analyze, and negotiate complex modes of expression. This environment fostered active learning, encouraging students to apply their knowledge, question their assumptions, and reconsider their perspectives. Overall, such a unique approach to visual literacy education has the potential to reshape students' understanding of both cultures. By encouraging them to critically engage with their multimodal designs, we promoted an in-depth, nuanced appreciation of these diverse cultural heritages. The students no longer just interpreted and replicated images—they actively contributed to a dynamic and ongoing conversation between cultures.

Keywords: multimodal pedagogy, creative expressions, visual literacy education, multimodal designs

Procedia PDF Downloads 43

Authors: Henry Nicholas Lee, George Thomas, Juliana Johari, Carmilla Freddie, Caroline Val Madin

Abstract:

Research in comparative and international education often provides value-laden views of an education system within and in between other countries. These views are frequently used by policy makers or educators to explore similarities and differences for, among others, benchmarking purposes. In this study, a comparison is made between Malaysia and England, focusing on the process of writing children went through to create a text, using a multimodal theoretical framework to analyse this comparison. The main purpose is political in nature as it served as an answer to Malaysia’s call for benchmarking of best practices for language learning. Furthermore, the focus on writing in this study adds into more empirical findings about early writers’ writing development and writing improvement, especially for children at the ages of 5-9. In research, comparative studies in English as a Second Language (ESL) writing pedagogy – particularly in Malaysia since the introduction of the Standard- based English Language Curriculum (KSSR) in 2011 as a draft and its full implementation in 2017; reviewed 2018 KSSR-CEFR aligned – has not been done comparatively. In theory, a multimodal theoretical framework somehow allows a logical comparison between first language and ESL which would provide useful insights to illuminate the writing process between Malaysia and England. The comparisons are not representative because of the different school systems in both countries. So far, the literature informs us that the curriculum for language learning is very much emphasised on children’s linguistic abilities, which include their proficiency and mastery of the language, its conventions, and technicalities. However, recent empirical findings suggested that literacy in its concepts and characters need change. In view of this suggestion, the comparison will look at how the process of writing is implemented through the five modes of communication: linguistic, visual, aural, spatial, and gestural. This project draws on data from Malaysia and England, involving 10 teachers, 26 classroom observations, 20 lesson plans, 20 interviews, and 20 brief conversations with teachers. The research focused upon 20 primary children of different genders aged 5-9, and in addition to primary data descriptions, 40 children’s works, 40 brief classroom conversations, 30 classroom photographs, and 30 school compound photographs were undertaken to investigate teachers and children’s use of modes and semiotic resources to design a text. The data were analysed by means of within-case analysis, cross-case analysis, and constant comparative analysis, with an initial stage of data categorisation, followed by general and specific coding, which clustered the data into thematic groups. The study highlights the importance of teachers’ and children’s engagement and interaction with various modes of communication, an adaptation from the English approaches to teaching writing within the KSSR framework and providing ‘voice’ to ESL writers to ensure that both have access to the knowledge and skills required to make decisions in developing multimodal texts and artefacts.

Keywords: comparative education, early writers, KSSR, multimodal theoretical framework, writing development

Procedia PDF Downloads 39

Authors: Rabab M. Ramadan, Elaraby A. Elgallad

Abstract:

With extensive application, the performance of unimodal biometrics systems has to face a diversity of problems such as signal and background noise, distortion, and environment differences. Therefore, multimodal biometric systems are proposed to solve the above stated problems. This paper introduces a bimodal biometric recognition system based on the extracted features of the human palm print and iris. Palm print biometric is fairly a new evolving technology that is used to identify people by their palm features. The iris is a strong competitor together with face and fingerprints for presence in multimodal recognition systems. In this research, we introduced an algorithm to the combination of the palm and iris-extracted features using a texture-based descriptor, the Scale Invariant Feature Transform (SIFT). Since the feature sets are non-homogeneous as features of different biometric modalities are used, these features will be concatenated to form a single feature vector. Particle swarm optimization (PSO) is used as a feature selection technique to reduce the dimensionality of the feature. The proposed algorithm will be applied to the Institute of Technology of Delhi (IITD) database and its performance will be compared with various iris recognition algorithms found in the literature.

Keywords: iris recognition, particle swarm optimization, feature extraction, feature selection, palm print, the Scale Invariant Feature Transform (SIFT)

Procedia PDF Downloads 193

Authors: Zhen Li

Abstract:

In contemporary China, the rise of consumer culture and feminism has facilitated the ‘She-Economy’, where Chinese women’s consumption power has been boosted along with the thriving Chinese economy. Domestic reality TV shows such as Sisters Who Make Waves (hereafter SWMW) target female audiences by bringing women's issues such as age, appearance, and balance between family and career to the discussion. Against this backdrop, this study adopted multimodal content analysis to investigate how SWMW failed to live up to the feminist goals the show had claimed and how serious women’s issues were consumed and capitalized by the consumer media culture from consumer culture and feminist perspectives. The findings reveal that while the female-themed work claims to uncover the charm that age brings to women over their thirties, it merely mentions female anxiety and uses feminism in disguise to achieve commercial success without in-depth thinking and discussion of what real-life issues women in China are tackling. They further show that the mass media-promoted modern femininity combined with consumerism deepens anxiety over aging among female audiences. The study sheds light on understanding the new development of Chinese femininity and the impact of consumer culture on feminist consciousness in contemporary China.

Keywords: consumer culture, feminism, multimodal content analysis, she-economy

Procedia PDF Downloads 65

Authors: Shomaila Sadaf, Margarethe Olbertz-Siitonen

Abstract:

This study looks at identity and othering in discourses around sensitive issues in social media. More specifically, the study explores the multimodal resources and narratives through which the other is formed, and identities are ascribed in online spaces. As an integral part of social life, media spaces have become an important site for negotiating and ascribing identities. In line with recent research, identity is seen hereas constructions of belonging which go hand in hand with processes of in- and out-group formations that in some cases may lead to othering. Previous findings underline that identities are neither fixed nor limited but rather contextual, intersectional, and interactively achieved. The goal of this study is to explore and develop an understanding of how people co-construct the ‘other’ and ascribe certain identities in social media using multiple modes. In the beginning of the year 2018, the British government decided to include relationships, sexual orientation, and sex education into the curriculum of state funded primary schools. However, the addition of information related to LGBTQ+in the curriculum has been met with resistance, particularly from religious parents.For example, the British Muslim community has voiced their concerns and protested against the actions taken by the British government. YouTube has been used by news companies to air video stories covering the protest and narratives of the protestors along with the position ofschool officials. The analysis centers on a YouTube video dealing with the protest ofa local group of parents against the addition of information about LGBTQ+ in the curriculum in the UK. The video was posted in 2019. By the time of this study, the videos had approximately 169,000 views andaround 6000 comments. In deference to multimodal nature of YouTube videos, this study utilizes multimodal discourse analysis as a method of choice. The study is still ongoing and therefore has not yet yielded any final results. However, the initial analysis indicates a hierarchy of ascribing identities in the data. Drawing on multimodal resources, the media works with social categorizations throughout the documentary, presenting and classifying involved conflicting parties in the light of their own visible and audible identifications. The protesters can be seen to construct a strong group identity as Muslim parents (e.g., clothing and reference to shared values). While the video appears to be designed as a documentary that puts forward facts, the media does not seem to succeed in taking a neutral position consistently throughout the video. At times, the use of images, soundsand language contributes to the formation of “us” vs. “them”, where the audience is implicitly encouraged to pick a side. Only towards the end of the documentary this problematic opposition is addressed and critically reflected through an expert interview that is – interestingly – visually located outside the previously presented ‘battlefield’. This study contributes to the growing understanding of the discursive construction of the ‘other’ in social media. Videos available online are a rich source for examining how the different social actors ascribe multiple identities and form the other.

Keywords: identity, multimodal discourse analysis, othering, youtube

Procedia PDF Downloads 87

Authors: Shabir Ahmad Sofi, Shubham Aggarwal, Sanyam Singhal, Roohie Naaz

Abstract:

Biometric recognition refers to an automatic recognition of individuals based on a feature vector(s) derived from their physiological and/or behavioral characteristic. Biometric recognition systems should provide a reliable personal recognition schemes to either confirm or determine the identity of an individual. These features are used to provide an authentication for computer based security systems. Applications of such a system include computer systems security, secure electronic banking, mobile phones, credit cards, secure access to buildings, health and social services. By using biometrics a person could be identified based on 'who she/he is' rather than 'what she/he has' (card, token, key) or 'what she/he knows' (password, PIN). In this paper, a brief overview of biometric methods, both unimodal and multimodal and their advantages and disadvantages, will be presented.

Keywords: biometric, DNA, fingerprint, ear, face, retina scan, gait, iris, voice recognition, unimodal biometric, multimodal biometric

Procedia PDF Downloads 730

Authors: Laheeb M. Ibrahim, Ibrahim A. Salih

Abstract:

Biometrics is a technology that has been widely used in many official and commercial identification applications. The increased concerns in security during recent years (especially during the last decades) have essentially resulted in more attention being given to biometric-based verification techniques. Here, a novel fusion approach of palmprint, dental traits has been suggested. These traits which are authentication techniques have been employed in a range of biometric applications that can identify any postmortem PM person and antemortem AM. Besides improving the accuracy, the fusion of biometrics has several advantages such as increasing, deterring spoofing activities and reducing enrolment failure. In this paper, a first unimodel biometric system has been made by using (palmprint and dental) traits, for each one classification applying an artificial neural network and a hybrid technique that combines swarm intelligence and neural network together, then attempt has been made to combine palmprint and dental biometrics. Principally, the fusion of palmprint and dental biometrics and their potential application has been explored as biometric identifiers. To address this issue, investigations have been carried out about the relative performance of several statistical data fusion techniques for integrating the information in both unimodal and multimodal biometrics. Also the results of the multimodal approach have been compared with each one of these two traits authentication approaches. This paper studies the features and decision fusion levels in multimodal biometrics. To determine the accuracy of GAR to parallel system decision-fusion including (AND, OR, Majority fating) has been used. The backpropagation method has been used for classification and has come out with result (92%, 99%, 97%) respectively for GAR, while the GAR) for this algorithm using hybrid technique for classification (95%, 99%, 98%) respectively. To determine the accuracy of the multibiometric system for feature level fusion has been used, while the same preceding methods have been used for classification. The results have been (98%, 99%) respectively while to determine the GAR of feature level different methods have been used and have come out with (98%).

Keywords: back propagation neural network BP ANN, multibiometric system, parallel system decision-fusion, practical swarm intelligent PSO

Procedia PDF Downloads 510

Authors: Komal V. Aher, Sanjay B. Waykar

Abstract:

Nowadays every individual having mobile device with them. In both computer vision and information retrieval Image search is currently hot topic with many applications. The proposed intelligent image search system is fully utilizing multimodal and multi-touch functionalities of smart phones which allows search with Image, Voice, and Text on mobile phones. The system will be more useful for users who already have pictures in their minds but have no proper descriptions or names to address them. The paper gives system with ability to form composite visual query to express user’s intention more clearly which helps to give more precise or appropriate results to user. The proposed algorithm will considerably get better in different aspects. System also uses Context based Image retrieval scheme to give significant outcomes. So system is able to achieve gain in terms of search performance, accuracy and user satisfaction.

Keywords: color space, histogram, mobile device, mobile visual search, multimodal search

Procedia PDF Downloads 339

Authors: Rauha Salam

Abstract:

In Pakistan, over the past decade, the notion of what counts as a true ‘masculine and feminine’ behaviour has become more complicated with the inspection of social media. Given its strong religious and socio-cultural norms, patriarchal values are entrenched in the local and cultural traditions of the Pakistani society and regulate the social value of gender. However, the increasing use of internet among Pakistani men and women, especially in the form of social media uses by the youth, is increasingly becoming disruptive and challenging to the strict modes of behavioural monitoring and control both at familial and state level. Facebook, being the prime social media communication platform in Pakistan, provide its users a relatively ‘safe’ place to embrace how they want to be perceived by their audience. Moreover, the availability of an array of semiotic resources (e.g. the videos, audios, visuals and gifs) on Facebook makes it possible for the users to create a virtual identity that allows them to describe themselves in detail. By making use of Multimodal Discourse Analysis, I aimed to investigate how men and women in Pakistan construct their gendered identities multimodally (visually and linguistically) through their Facebook posts and how these semiotic modes are interconnected to communicate specific meanings. In case of the female data, the analysis showed an ambivalence as females were found to be conforming to the existing socio-cultural norms of the society and they were also employing social media platforms to deviate from traditional gendered patterns and to voice their opinions simultaneously. Similarly, the male data highlighted the reproduction of the prevalent cultural models of masculinity. However, there were instances in the data that showed a digression from the standard norms and there is a (re)negotiation of the traditional patriarchal representations.

Keywords: Facebook, Gendered Identities, Multimodal Discourse Analysis, Pakistan

Procedia PDF Downloads 90

Authors: Shaza Melies, Abeer Refky, Nihad Mansoor

Abstract:

Multimodality involves the communication between verbal and visual components in various discourses. A painting represents a form of communication between the artist and the viewer in terms of colors, shades, objects, and the title. This paper aims to present how multimodality can be used to decode the verbal and visual dimensions a painting holds. For that purpose, this study uses Kress and van Leeuwen’s theoretical framework of visual grammar for the analysis of the multimodal semiotic resources of selected paintings of Salvador Dali. This study investigates the visual decoding of the selected paintings of Salvador Dali and analyzing their social and political meanings using Kress and van Leeuwen’s framework of visual grammar. The paper attempts to answer the following questions: 1. How far can multimodality decode the verbal and non-verbal meanings of surrealistic art? 2. How can Kress and van Leeuwen’s theoretical framework of visual grammar be applied to analyze Dali’s paintings? 3. To what extent is Kress and van Leeuwen’s theoretical framework of visual grammar apt to deliver political and social messages of Dali? The paper reached the following findings: the framework’s descriptive tools (representational, interactive, and compositional meanings) can be used to analyze the paintings’ title and their visual elements. Social and political messages were delivered by appropriate usage of color, gesture, vectors, modality, and the way social actors were represented.

Keywords: multimodal analysis, painting analysis, Salvador Dali, visual grammar

Procedia PDF Downloads 91

Authors: Hadri Hussain, M.Nasir Ibrahim, Chee-Ming Ting, Mariani Idroas, Fuad Numan, Alias Mohd Noor

Abstract:

A biometric security system has become an important application in client identification and verification system. A conventional biometric system is normally based on unimodal biometric that depends on either behavioural or physiological information for authentication purposes. The behavioural biometric depends on human body biometric signal (such as speech) and biosignal biometric (such as electrocardiogram (ECG) and phonocardiogram or heart sound (HS)). The speech signal is commonly used in a recognition system in biometric, while the ECG and the HS have been used to identify a person’s diseases uniquely related to its cluster. However, the conventional biometric system is liable to spoof attack that will affect the performance of the system. Therefore, a multimodal biometric security system is developed, which is based on biometric signal of ECG, HS, and speech. The biosignal data involved in the biometric system is initially segmented, with each segment Mel Frequency Cepstral Coefficients (MFCC) method is exploited for extracting the feature. The Hidden Markov Model (HMM) is used to model the client and to classify the unknown input with respect to the modal. The recognition system involved training and testing session that is known as client identification (CID). In this project, twenty clients are tested with the developed system. The best overall performance at 44 kHz was 93.92% for ECG and the worst overall performance was ECG at 88.47%. The results were compared to the best overall performance at 44 kHz for (20clients) to increment of clients, which was 90.00% for HS and the worst overall performance falls at ECG at 79.91%. It can be concluded that the difference multimodal biometric has a substantial effect on performance of the biometric system and with the increment of data, even with higher frequency sampling, the performance still decreased slightly as predicted.

Keywords: electrocardiogram, phonocardiogram, hidden markov model, mel frequency cepstral coeffiecients, client identification

Procedia PDF Downloads 252

Authors: S. Adhirai, R. P. Mahapatra, Paramjit Singh

Abstract:

Optimization is an important tool in making decisions and in analysing physical systems. In mathematical terms, an optimization problem is the problem of finding the best solution from among the set of all feasible solutions. The paper discusses the Whale Optimization Algorithm (WOA), and its applications in different fields. The algorithm is tested using MATLAB because of its unique and powerful features. The benchmark functions used in WOA algorithm are grouped as: unimodal (F1-F7), multimodal (F8-F13), and fixed-dimension multimodal (F14-F23). Out of these benchmark functions, we show the experimental results for F7, F11, and F19 for different number of iterations. The search space and objective space for the selected function are drawn, and finally, the best solution as well as the best optimal value of the objective function found by WOA is presented. The algorithmic results demonstrate that the WOA performs better than the state-of-the-art meta-heuristic and conventional algorithms.

Keywords: optimization, optimal value, objective function, optimization problems, meta-heuristic optimization algorithms, Whale Optimization Algorithm, implementation, MATLAB

Procedia PDF Downloads 332

Authors: Saulutė Juzelėnienė, Skirmantė Šarkauskienė

Abstract:

In this paper, it is aimed to analyse the multimodal representations of the concepts of LOVE and PASSION in Lithuanian graphic novel “Gertrūda”, by Gerda Jord. The research is based on the earlier findings by Forceville (2005), Eerden (2009) as well as insights made by Shihara and Matsunaka (2009) and Kövecses (2000). The domains of target and source of LOVE and PASSION metaphors in comics are expressed by verbal and non-verbal cues. The analysis of non-verbal cues adopts the concepts of rune and indexes. A pictorial rune is a graphic representation of an object that does not exist in reality in comics, such as lines, dashes, text "balloons", and pictorial index – a graphically represented object of reality, a real symptom expressing a certain emotion, such as a wide smile, furrowed eyebrows, etc. Indexes are often hyperbolized in comics. The research revealed that most frequent source domains are CLOSINESS/UNITY, NATURAL/ PHYSICAL FORCE, VALUABLE OBJECT, PRESSURE. The target is the emotion of LOVE/PASSION which belongs to a more abstract domain of psychological experience. In this kind of metaphor, the picture can be interpreted as representing the emotion of happiness. Data are taken from Lithuanian comic books and Internet sites, where comics have been presented. The data and the analysis we are providing in this article aims to reveal that there are pictorial metaphors that manifest conceptual metaphors that are also expressed verbally and that methodological framework constructed for the analysis in the papers by Forceville at all is applicable to other emotions and culture specific pictorial manifestations.

Keywords: multimodal metaphor, conceptual metaphor, comics, graphic novel, concept of love/passion

Procedia PDF Downloads 45

Authors: Victor Milanes, Martha Hubertz

Abstract:

In the wake of COVID-19, all aspects of life have been estranged, and humanity has been forced to shift toward a more technologically integrated mode of operation. Essential work such as Healthcare, business, and public policy are a few notable industries that were initially dependent upon face-to-face modality but have completely reimagined their operation style. Unique to these fields, education was particularly strained because academics, teachers, and professors alike were obligated to shift their curriculums online over the course of a few weeks while also maintaining the expectation that they were educating their students to a similar level accomplished pre-pandemic. This was notable as research indicates two key concepts: Students prefer face-to-face modality, and due to the disruption in academic continuity/style, there was a negative impact on student's overall education and performance. With these two principles in mind, this study aims to inquire what online strategies could be best employed by teachers to educate their students, as well as what strategies could be adopted in a multimodal setting if deemed necessary by the instructor or outside convoluting factors (Such as the case of COVID-19, or a personal matter that demands the teacher's attention away from the classroom). Strategies and methods will be cross-analyzed via a ranking system derived from various recognized teaching assessments, in which engagement, retention, flexibility, interest, and performance are specifically accounted for. We expect to see an emphasis on positive social pressure as a dominant factor in the improved propensity for education, as well as a preference for visual aids across platforms, as research indicates most individuals are visual learners.

Keywords: technological integration, multimodal teaching, education, student engagement

Procedia PDF Downloads 31

Authors: Visvaganthie Moodley

Abstract:

While rhetoric goes back as far as Aristotle who focalised its meaning as the “art of persuasion”, most scholars have focused on elocutio and dispositio canons, neglecting the rhetorical impact of multimodal texts, such as documentaries. Film documentaries are being increasingly rhetoric, often used by wildlife conservationists for influencing people to become more mindful about humanity’s connection with nature. This paper examines the award-winning film documentary, “My Octopus Teacher”, which depicts naturalist, Craig Foster’s unique discovery and relationship with a female octopus in the southern tip of Africa, the Cape of Storms in South Africa. It is anchored in Leech and Short’s (2007) framework of linguistic and stylistic categories – comprising lexical items, grammatical features, figures of speech and other rhetoric features, and cohesiveness – with particular foci on diction, anthropomorphic language, metaphors and symbolism. It also draws on Kress and van Leeuwen’s (2006) multimodal analysis to show how verbal cues (the narrator’s commentary), visual images in motion, visual images as metaphors and symbolism, and aural sensory images such as music and sound synergise for rhetoric effect. In addition, the analysis of “My Octopus Teacher” is guided by Nichol’s (2010) narrative theory; features of a documentary which foregrounds the credibility of the narrative as a text that represents real events with real people; and its modes of construction, viz., the poetic mode, the expository mode, observational mode and participatory mode, and their integration – forging documentaries as multimodal texts. This paper presents a multimodal rhetoric discussion on the sequence of salient episodes captured in the slow moving one-and-a-half-hour documentary. These are: (i) The prologue: on the brink of something extraordinary; (ii) The day it all started; (iii) The narrator’s turmoil: getting back into the ocean; (iv) The incredible encounter with the octopus; (v) Establishing a relationship; (vi) Outwitting the predatory pyjama shark; (vii) The cycle of life; and (viii) The conclusion: lessons from an octopus. The paper argues that wildlife documentaries, characterized by plausibility and which provide researchers the lens to examine the ideologies about animals and humans, offer an assimilation of the various senses – vocal, visual and audial – for engaging viewers in stylized compelling way; they have the ability to persuade people to think and act in particular ways. As multimodal texts, with its use of lexical items; diction; anthropomorphic language; linguistic, visual and aural metaphors and symbolism; and depictions of anthropocentrism, wildlife documentaries are powerful resources for promoting wildlife conservation and conscientizing people of the need for establishing a harmonious relationship with nature and humans alike.

Keywords: documentaries, multimodality, rhetoric, style, wildlife, conservation

Procedia PDF Downloads 66