Natural Language Processing (NLP) is a field at the intersection of Artificial Intelligence (AI) and Linguistics¹ focused on enabling computers to understand and generate human languages.² NLP emerged in the 1950s to simplify tasks for users and fulfil the desire to interact with computers using natural language.³

Overview of NLP

NLP employs either rule-based or machine learning methods to comprehend the structure and meaning of text.⁴ It powers chatbots, voice assistants, text scanning programs, translation applications, and enterprise software that enhance business operations, boost productivity, and streamline various processes.⁵ The most well-known NLP tool is GPT-3 from OpenAI, which uses artificial intelligence and statistical methods to predict the next word in a sentence based on the preceding ones.⁶ NLP practitioners refer to tools like this as “language models”.⁷ These models can be utilized for basic analytics tasks, such as document classification and sentiment analysis, as well as more advanced tasks like question answering and summarization.⁸

Some common benefits of NLP include increasing productivity by automating tasks, thereby reducing costs and enhancing overall efficiency for businesses.⁹ By leveraging NLP, businesses can develop innovative products and services, such as customer service chatbots, language translation applications, and question-answering systems, that transform customer interactions and offer new, cutting-edge solutions.¹⁰ Additionally, NLP can enhance decision-making by analyzing large volumes of text data, providing insights that support informed decisions.¹¹

To provide a deeper insight into NLP, this blog will explore its key components, their functionalities, and the challenges encountered in the development of NLP tools.

Exploring NLP Components and their Functionality

NLP has two primary components, namely Natural Language Generation (NLG) and Natural Language Understanding (NLU).¹²

Natural Language Generation (NLG) involves automatically producing human-readable text or speech from structured data or inputs.¹³ NLG systems analyse input data to grasp its meaning and generate coherent and contextually appropriate language output.¹⁴ This involves utilizing various techniques, from rule-based systems to advanced deep learning models, to convert structured information into natural language.¹⁵ In order to formulate responses appropriately, NLG applications must take into account language rules concerning morphology, lexicons, syntax and semantics.¹⁶

Natural Language Understanding (NLU), on the other hand, enables machines to comprehend and interpret human language by extracting metadata from content.¹⁷ It focuses on grasping the meaning of a sentence through syntactic and semantic analysis of the text.¹⁸ Recognizing that human languages are complex, characterized by intricate, nuanced, and ever-evolving meanings, poses significant challenges for computers and other devices to fully grasp.¹⁹ Nevertheless, NLU systems enable organizations to develop products and tools that go beyond interpreting words, delving into the underlying meanings they convey.²⁰ Notably, NLU facilitates dialogue with computers using human-based language.²¹

In light of this, NLP models function by establishing correlations among the basic components of language such as letters, words, and sentences present in a text dataset.²² Thus, NLP architectures employ various techniques for data preprocessing, feature extraction, and model construction.²³ In essence, NLP involves two main phases: data preprocessing and algorithm development.²⁴ Data preprocessing entails cleaning and preparing text data to make it analysable by machines.²⁵ This process organizes data into a usable format and emphasizes features in the text that an algorithm can utilize.²⁶

There are various techniques which may be employed in data preprocessing, and they include: tokenization, stop word removal, lemmatization and stemming, and sentence segmentation.²⁷

• Tokenization: replaces sensitive information with non-sensitive information or a token, commonly applied in payment transactions to safeguard credit card data.²⁸

• Stop word removal: eliminates common words from the text, allowing unique and informative words to stand out.

• Sentence segmentation: divides a large block of text into meaningful segments that typically correspond to individual sentences.²⁹

• Lemmatization and stemming: standardizes the various inflected forms of words in a text dataset to a single base form or dictionary entry, referred to as a ‘lemma’ in computational linguistics.³⁰ This technique is valuable in information retrieval systems like search engines, where users enter a single-word query such as ‘meditate’ but anticipate results that include any inflected forms of the word like ‘meditates’ or ‘meditation.’³¹

Once the data has been preprocessed, an algorithm is developed to process it, employing various NLP algorithms.³² Two main types are commonly used: the rule-based system, which relies on meticulously crafted linguistic rules that have been used since early stages of NLP development and are still employed today, and the machine learning-based system.³³ Machine learning algorithms utilize statistical methods to learn tasks from training data, adjusting their approaches as they process more data.³⁴ These algorithms leverage machine learning, deep learning, and neural networks to refine their rules through continuous processing and learning.³⁵

Challenges facing NLP Tool Development

Over the years, several NLP tools have been developed, the most famous being chatbots and language models.³⁶ These include Eliza, which was developed in the mid-1960s; Tay, a chatbot by Microsoft launched in 2016; BERT, Generative Pre-Trained Transformer 3 (GPT-3), Language Model for Dialogue Applications (LaMDA) and Mixture of Experts (MoE).³⁷

The challenges in NLP can be categorised into technical and linguistic.³⁸ From a technical perspective, NLP relies on a machine’s statistical and learning capabilities, with corpus data³⁹ being the primary material for training machine learning models in language processing.⁴⁰ However, developing a robust corpus demands significant manual effort and time, often rendering many corpora inaccessible, thereby impeding progress in machine learning for language acquisition.⁴¹

Secondly, in daily life, many conversations and texts are initiated based on shared knowledge between participants. However, current NLP models, trained on large corpora, often lack awareness of historical or cultural contexts, as well as common sense.⁴² This deficiency in background knowledge complicates machines’ ability to grasp crucial factors during the processing of conversations and texts.⁴³

From a linguistic perspective, machines encounter several challenges in comprehending natural language. These challenges encompass the diversity of languages spoken across various countries and regions, making it difficult for machines to adopt a uniform approach for processing them.⁴⁴ Additionally, the ambiguity inherent in language means that the same sentence or word in a conversation or text can carry multiple meanings, further complicating machine understanding.⁴⁵ Machines also need to exhibit consistent performance in processing different accents and styles of conversation, which adds to the robustness requirement of language processing.⁴⁶

Further, language comprehension heavily relies on practical knowledge, presenting a common challenge for both technology and language.⁴⁷ Also, machines must grasp language within its specific environment and context, highlighting the crucial role of contextual understanding in effective language processing.⁴⁸ For instance, linguistic diversity among African languages presents a major challenge for NLP, especially in regions where there is limited familiarity with indigenous datasets.⁴⁹ The use of unrelated linguistic datasets may not accurately capture the nuances and complexities of African languages.⁵⁰

African languages, despite their rich linguistic diversity, remain significantly under-represented in NLP research and technology.⁵¹ Most existing NLP tools and datasets are focused on high-resource languages, leaving a notable gap in resources and tools for African languages.⁵² This under-representation is further complicated by the scarcity of machine-readable data,⁵³ historical impact of colonialism that has often sidelined local languages,⁵⁴ and technical challenges arising from the complexity and ambiguity of African languages.⁵⁵ Various initiatives and communities have emerged to tackle these challenges,⁵⁶ including Masakhane Research Foundation,⁵⁷ KenCorpus⁵⁸ and Ghana NLP,⁵⁹ which focus on creating datasets, developing tools, and fostering collaboration to advance NLP for African languages.

To address the challenges in NLP tool development, human oversight is essential in training models to rectify errors and provide nuanced understanding that current Artificial Intelligence (AI) models may overlook.⁶⁰ In cases where there is a scarcity of high-quality data, various techniques can be utilised to enhance datasets. These techniques include synthetic data generation, where new data is generated based on existing patterns, and data augmentation, which involves making slight alterations to existing data to generate new examples.⁶¹

Conclusion

NLP significantly enhances our interaction with technology by increasing productivity, enabling innovative products, and improving customer experiences. By employing components such as NLG and N LU, NLP systems can generate coherent language outputs and comprehend human language’s intricate meanings. Although several challenges are yet to be overcome, current models have already had a transformative impact, enabling more user-friendly virtual assistants and enhanced human-computer interactions, indicating a future where communication with machines is as seamless as with humans. Considering these advancements, one should ask themselves: what will be the impact on intellectual property rights and licensing in NLP tool development?

Acknowledgements

This blog draws inspiration from the insightful ideas of Dr. Melissa Omino.

Image by pikisuperstar on Freepik