SFI ADAPT Centre
Nominated Award: Best Application of AI to achieve Social Good
Website of Company: http://www.adaptcentre.ie
ADAPT is the world-leading SFI research centre for AI Driven Digital Content Technology hosted by Trinity College Dublin. ADAPT’s partner institutions include Dublin City University, University College Dublin, Technological University Dublin, Maynooth University, Munster Technological University, Athlone Institute of Technology, and the National University of Ireland Galway. ADAPT’s research vision is to pioneer new forms of proactive, scalable, and integrated AI-driven Digital Content Technology that empower individuals and society to engage in digital experiences with control, inclusion, and accountability with the long term goal of a balanced digital society by 2030. ADAPT is pioneering new Human Centric AI techniques and technologies including personalisation, natura language processing, data analytics, intelligent machine translation human-computer interaction, as well as setting the standards for data governance,privacy and ethics for digital content.
Reason for Nomination
Societal Issue
Crossing language barriers is essential for global information exchange and unobstructed, fair communication. According to the World Federation of the Deaf (WFD), over 70 million people are deaf and communicate primarily via sign language (SL). The amount of content generated daily, in spoken and written language, and the increase in social interactions, e.g. via social media, poses a challenge when it comes to unobstructed and direct communication between deaf, hearing and hard of hearing people. Currently, human interpreters are the main medium for sign-to-spoken, spoken-to-sign and sign-to-sign language translation. The availability and cost of these professionals is often a limiting factor in communication between signers and non-signers.
Deaf and hard of hearing (DHH) and hearing people often face communication barriers when they interact, owing to the different languages at play. However, this can be further aggravated by cultural differences between these groups. In spite of the fact that these three groups may come from and live in the same area, they may face cross-cultural barriers, similar to those communicating from different countries. It has long been recognised that deaf people have their own culture, distinct from hearing people, with its own norms and values. For example, communication styles can differ between the groups in terms of amount of eye contact used and the directness of the message communicated. Poor eye contact from hearing people may result in them seeming at best disconnected and at worst rude to deaf people. Separately, hard of hearing people may find themselves aligning more closely with either deaf or hearing cultures depending on their own personal circumstances. As a result, communication barriers can be further compounded by these cultural communicative nuances.
Many deaf signers rely on (hearing) sign language interpreters who learned a national sign language at a later age, so they don’t master it at native level. The deaf people (although they are highly multilingual especially in the field of national signed and written languages, but have a hearing disability) have to adapt with extra effort to the sign language use of the hearing interpreters. These efforts are not always possible in difficult situations (eg in an accident or crisis where the conversation with a hearing interlocutor, via an interpreter, is very emotional). Either misunderstandings often arise because the hearing interpreters do not always understand the deaf signers equally well, especially not their nuances, or vice versa (sign languages of hearing interpreters are often influenced by spoken languages).
Due to the limited number of interpreters and/or their availability that not always meet the demands of the deaf people, or the limited number of hours obtained to call in an interpreter in their work and private situaties, this often has a negative impact on their quality of life. It is more difficult for them to integrate into social life (e.g. less participation in cultural activities) and work life (e.g. fewer promotion opportunities). In addition, these communication barriers that deaf people experience with hearing interlocutors has an influence on the perception of hearing people and society about the deaf people and sign languages. They are (unconsciously) stigmatized and excluded. Moreover, it is annoying for the deaf people to always have to depend on a third party, such as an interpreter. This feeling of not always being able to be as independent as possible also has an impact on their well-being.
All over europe, the same major problems often ooccur amont deaf and HH people. Various information from deaf associations, interpreters service, surveys, etc show that they have little or no access to different areas of their lives. It is positive that DHH have become more emancipated in recent years and are more aware of their human rights. For example, more and more DHH are going to study and/or enter the regular work spaces. Of course, their success often depend on the availability of an interpreter. However, due to the growing integration of deaf people into society, the need for an interpreter is growing. At the same time, sign languages are getting better statutes, which means that more quality is requiered from sign language interpreters. Currently, not all interpreting training and/or services can always meet this positive growing demand from DHH.
Solution
SignON is a Horizon 2020 project which aims to develop a communication service that translates between signed and spoken (in both text and audio modalities) languages and caters for the communication needs between deaf, hard of hearing and hearing individuals.
Reducing this communication gap is the overall objective of this project. To achieve this objective SignON aims to provide a communication service delivered through a mobile application, which has been co-designed and co-developed with the relevant user communities. This co-creation approach, rigidly followed by the SignON partners, where potential deaf, hard of hearing and hearing users are involved in the research, design and development activities makes SignON a unique endeavour with strong social and societal impact.
To date, the SignON project has successfully developed working versions of the SignON App and Framework according to user requirements and expectations, established a repository for the code-base which will be openly accessible after the end of the project and, which, acts as a platform where development can take place in parallel. The SignON Framework has been developed through a distributed and highly scalable approach which allows for components to operate in parallel (to accommodate multiple users without overhead) as well as to be redesigned, improved and replaced seamlessly. Software components for ASR, SLR, MT, NLP and synthesis have been developed and integrated in the framework. State of the art models to facilitate the translation pipeline have been built and released.
It can be an added value in the repertoire of different communication options between the deaf and hearing interlocutors. It does not have to replace existing communication options (such as interpreting and/or remote interpreting). It always depends on context and preferences of both parties how a conversation goes. This can range from gestures, written communication to the involvement of interpreters. It’s too early to say what a role AI could play when it’s enabled. Our deaf respondents pointed out that it mainly depends on the quality of AI, how far it is progressed, before they judge where/when/how often they will be used.
In everyday communication, where it is difficult to find an interpreter in a short time, or when an interpreter is too expensive for a conversation less than an hour (or only a few minutes), AI can be a solution. We are thinking of very short conversations such as ordering in a pharmacy, restaurant, or practical questions in a hotel, etc. In addition, AI can be a very interesting option in situations where it is preferable not to use an interpreter (e.g. a meeting with a notary or a doctor where sensitive information is exchanged). An AI is neutral and can enhance the anonymity and independence of deaf people.
There is a huge resistance, rightly, among those in deaf communities to applications and solutions that claim to “help” them. Often these solutions are proposed and developers by those in the hearing community who have no fundamental understanding of the DHH communities. The SignON approach is a different one. From day 1, deaf communities and deaf researchers were involved in the project development. This co-creation process is the main driver of all research and development within the project. The important principle “Nothing About Us Without Us” is rigidly adhered to.
Deaf people tend to be multilingual and multimodal in their communication ways. AI is an important addition to their repertoire of possible communication options. The opportunities of an AI (more anonymity and less reliance on third parties) are new to deaf people who have yet to discover them. But it will not replace any of the existing communication options (such as sign language interpreters and services). It is up to DHH themselves to see in which situations they will use AI as a replacement.
The SignON solution is not presented as a solution to replace the current use of interpreters. In certain situations, interpreters will always be a preferred means of communication. What is proposed is a complementary service, that will be available as an option for certain communication challenges. For day-to-day communication, it can be cost-effective in the long term. Deaf people should always be thrifty with their government-provided budget for sign language interpreters. With the advent of AI for shorter conversations, the hours can be used for other situations where physical interpreters are required. Importantly, AI certainly does not influence government decisions to reduce the already acquired sign language rights by saving on physical interpreters (e.g. for crisis communication, for television, etc.)
SignON aims to go beyond the state-of-the-art by (i) providing an unprecedented translation service and mobile application that are user-driven and co-created with users; (ii) researching and developing new methods, methodologies and models for sign and speech recognition, as well as machine translation; (iii) delivering a new, customisable 3D virtual signer; (iv) collecting and unifying
existing (in terms of formatting and organisation) and generating new data (synthetic data as well as sign language specific lexicons to drive the 3D virtual signer); (v) integrating machine learning capabilities in a large-scale framework to allow new languages, user preferences and use-cases to be integrated; as well as (vi) bringing deaf, hard of hearing and hearing communities closer together through co-creation, collaboration and open communication. We would also hope to provide ethical guidelines and principles that would help ensure fair and respectful data collection and curation, as well as the research and development of open and unbiased communication and collaboration for future projects of the scale and scope of SignON.
In addition to this, and of more fundamental importance, we hope that what we will have achieved will be a step towards bridging the communication gap between deaf, hard of hearing and hearing individuals. We would hope that the co-creation process in our project would instill confidence in the project and its results, to allow the respective communities to communicate more easily and more freely. We would also like to see progress towards the establishment of a repository of sign language data that would allow further research to be performed in this domain.
The potential impacts for this project are huge. Besides the impact on scientific research in terms of technical advancements in multilingual speech processing and sign language recognition on mobile devices, the societal impact of having access to an easily accessible communication service in lieu of interpreters would create a huge benefit to society. The SignON application is not presented as a tool to replace interpreters and current means of communication but as an option available to the general public if it is required and appropriate.
Nowadays any communication barrier is detrimental to society. This project will research and develop the SignON communication service that uses machine translation to translate between Sign and spoken languages. This service will facilitate the exchange of information among deaf and hard of hearing, and hearing individuals. This user-centric and community-driven project will tightly collaborate with European deaf and hard of hearing communities to (re)define use cases, co-design and co-develop the SignON service and application, assess the quality and validate their acceptance. Our ultimate objective is the fair, unbiased and inclusive spread of information and digital content in European society.
Additional Information:
The main focus of our research and development activities is driven by our co-creation process. This engagement with our user communities fuels the ideas that further the development of the project. This work aims to assess the gaps in communication between researchers, industry and the user communities, and in particular, deaf and hard of hearing communities. During these processes we have conducted interviews and surveys, have looked at case-studies and user-oriented design to ensure that users have been involved at every step of the design and development process and that the SignON services (including the App) meet users’ expectations. Through open communication with potential users, we also manage their expectations, demonstrating the realistic capabilities of current AI. Our co-creation approach is iterative to ensure that both users and the SignON team advance steadily and together.
To ensure that all activities within the consortium as well as towards the community follow correct ethical principles and respect personal privacy and legislations, the SignON Research Ethics Committee (REC) provides guidance for all of these activities and is on hand to offer support when needed.