Journal of information and
communication convergence engineering
(J. Inf. Commun. Converg. Eng.)

Text categorization assigns predefined classifications to documents written in natural languages; several forms of categorization can handle a variety of documents detecting topics and spam [1], the sentiments [2, 3] of product/movie reviews, and prediction of trends [4]. Most use traditional methods for text categorization; the models include naïve Bayes, support vector machines (SVM), k-nearest network (KNN), and newer methods [5].

In 1998, McCallum and Nigam [6] compared two naïve Bayes text classifications. The multinomial model was almost always better than the multi-variate Bernoulli model, reducing average error by 27%. In 2001, Soucy and Mineau [7] showed that a simple KNN algorithm was 5% points better than a naïve Bayes model. The SVM, the traditional method, has attracted much attention. In 1999, Joachims [8] introduced a transductive SVM method, which averaged 10 percentage points higher than a KNN method.

The traditional methods are seldom >80% accurate. Today, deep learning—via convolutional neural networks (CNNs) and long-/short-term memory (LSTM)—is often used for text categorization.

CNN usually deals with two-dimensional (2D) structures, such as images, computing a unit for each region. However, text categorization is a 1D problem; it is necessary to solve the word sequence and use the convolution layer to deal with small text regions. Text categorization using CNNs is popular [9]; the first layer searches a table and converts sentences to word vectors. We previously used a CNN to train word vectors and used other methods to learn from additional large corpora [10]. In other words, CNNs have many limitations. Therefore, we prefer other methods. Here, we apply a CNN directly to high-dimensional vectors; we do not employ word embedding. We use a graphics processing unit (GPU) to perform computations and handle sparse high-dimensional data, improving accuracy, accelerating training and prediction, and simplifying the system. Although a CNN could theoretically deal with natural language (text), CNNs are not designed for such inputs. CNNs usually accept dense inputs; each value is important, and most inputs are not zero. It is challenging to work with text data; we address this with the aid of LSTM, which is a recurrent neural network (RNN) [9]. LSTM can deal with both images and text (word sequences), recognizing embedded sequences. Compared to traditional recurrent networks, LSTM affords two advantages: a shorter learning time and better feasibility. LSTM can handle text of variable size. Text categorization using LSTM

Text categorization using LSTM [11] is complex, reducing training efficiency; we thus simplify the model. Our new method affords higher accuracy and more rapid training. Both our LSTM and CNN models are better than traditional methods. Our results optimize text categorization; we use text regions to establish higher standards than would be possible using isolated words. Thus, as revealed by tests using four benchmark datasets, our regional embeddings were better than the traditional embeddings.

The remainder of this paper is structured as follows. Section II describes the models and methods in preliminary terms (subsection A); subsections B and C describe text categorization using a CNN and LSTM, respectively. Section III contains the results; subsection A describes the experimental environment and processes; subsection B describes the use of a CNN for topic and sentiment classification and subsection C contains the LSTM data; we used four datasets to facilitate direct comparisons among the CNN, the LSTM, and traditional methods. Section IV summarizes our conclusions and describes areas for future research.

1) CNN and Image Analysis

For Fig. 1, the computer sees an array of pixel values rather than the image, and the resolution varies [12]. In the present case, the computer sees a 32×32×3 array of numbers. One pixel corresponds to a number from 0 to 255. The computer then calculates image probabilities to define the class including the picture. Here, dog is associated with an 80% probability, cat with 15% probability, and bird with 5% probability.

Figure 1.

The human and computer views. From Adit Deshpande’s blog [12].

The computer distinguishes different pictures. When we see a picture of a cat, we search for features such as four legs or whiskers. The computer looks for low-level features (edges and curves) and then more abstract concepts established by the convolutional layers.

2) LSTM

LSTM is a special form of RNN that solves long-term dependency problems not addressed by conventional normal RNNs (Hochreiter and Schmidhuber [11], and later developments). LSTM affords excellent performance when used to address many problems; LSTM remembers information on a long-term basis. All RNNs require chains to interrogate NN modules. In a standard RNN, the repeating module is very simple, such as a single tanh layer. LSTM has a more complex structure. A normal RNN has a single neural network layer; LSTM features four interacting layers [13] (Fig. 2).

Figure 2.

An LSTM containing four interacting layers.

In the illustration, the entire vector is carried by each line from a single node output to the inputs of other nodes. The pink circles reflect operations, such as vector rises, as the NN layers learn the yellow boxes. Line combinations reflect a series of related transactions; line branching reflects the copying of content to different locations.

3) Using Pre-trained Word Embedding

“Word embedding” is a group of natural language- processing techniques mapping semantic meaning to geometric space [14]. We search a dictionary to assign a numeric value to every vocabulary, such that the L2 distance between any two vectors reflects the semantic relationship between two correlated vocabularies. These vectors constitute a geometric space termed the “embedding space”. For example, “mango” and “penguin” are vocabularies that are semantically very different; a logical embedding space will express them as widely separated vectors. But “love” and “marriage” are correlative vocabularies; they should be closely embedded. In theory, in a good embedding space, the vector path from “love” to “marriage” will precisely describe the semantic relationship between these two notions. In this circumstance, the relationship is “why x occurs”; one would hope that the vector love-to-marriage would explain this. Fundamentally, we require the vectorial characteristic: marriage + (why x occurs) = love. Next, we can use such vectors to answer questions. For example, we define a new vector “hurt”, and explore its relationships; we should capture: hurt + (why x occurs) = hit, thus a reply to “why hurt occurs?”.

We can compute word-embedding. When we explore a text corpus, we can count and analyze databases using dimensionality reduction techniques, employing a form of neural network exploiting the “word2vec” technique or matrix factorization.

1) A Deeper CNN

We apply a CNN to text data. Kim [15] employed multiple filters implemented using a Keras Merge Layer (Fig. 3).

Figure 3.

A convolutional network with multiple filters.

We used 128 filters (size 5) and maximum pooling of 5 and 35 (Fig. 4).

Figure 4.

A convolutional structure for text categorization.

2) Text Categorization using a Sequential CNN

We have a document D = (w₁,w₂,…) and a vocabulary V. The CNN needs to preserve the word order when data are input. However, during text categorization using CNN, we can regard each word as a pixel, and D then equals |D|×1 pixel; we can also view |V| as a channel. Thus, we can use a CNN to deal with text as if it were an image. As a simple example, consider the vocabulary V = {“don’t”, “hate”, “her”, “like”, “you”} and a document D = “you like her”. We obtain a document vector x = [00001|00010|00100]^T.

When a CNN is used to deal with images, we need to select the region size in advance; when that size is p, we can use p|V| to express region vectors. Using the example above, suppose p = 2 and stride=1, we obtain two regions “you like” and “like her” (Table 1).

Table 1 .

The document vectors of a sequential CNN

		don’t	hate	her	like	you
r0(x)	you	0	0	0	0	1
	like	0	0	0	1	0
r1(x)	like	0	0	0	1	0
	her	0	0	1	0	0

As in image analysis, the text vector is translated to a feature vector. The convolution layer embeds text regions into low-dimensional vector spaces. However, a sequential CNN features a neural network, a convolution layer, and the region.

3) Use of a Disordered CNN for Text Evaluation

Sequential CNNs perform well, but they have one drawback. During imaging, only three RGB channels are required, but, for text, the channel number is |V| (size of vocabulary), which is large. Thus, when p is also large, the number p|V| (the weight vectors) will be very large, rendering training impractical. Thus, we use a different method; we ignore the sequence of the vocabulary, thus changing the region vectors from dimension p|V| to dimension |V| (Table 2).

Table 2 .

Document vectors of the disordered CNN

		don’t	hate	her	like	you
r0(x)	you like	0	0	0	1	1
r1(x)	like her	0	0	1	1	0

Thus, fewer parameters are studied. Disordered convolution lies between bow vectors and sequential convolution. Word order is lost in only small regions.

4) Pooling Layer for Text Categorization

The pooling layer must sometimes change. Unlike images of a fixed size, documents are variable in length. As the stride is fixed, the convolution layer output must vary in length (Fig. 5).

Figure 5.

The convolution layer for text of variable length.

The convolution layer output may vary in length after standard pooling. Next, the variable will be output to another convolution layer. Finally, the fully connected layer will receive inputs of variable length; this is inappropriate, as these inputs must be of fixed length. Thus, we dynamically control the pooling region size to ensure that the inputs to the fully connected layer are of a fixed length.

5) A Collateral CNN: An Extension

Often, text categorization is performed using CNNs, which commonly have only two convolution and corresponding pooling layers. To improve model accuracy, we explored a collateral CNN featuring two or more convolution layers in parallel. Using this architecture, we found that the convolution-pooled pairs effectively dealt with differently sized regions, developing different vectors for each region but ensuring that the vectors to the final fully connected layer were of the same length.

1) Text Classification using LSTM

Next, we explore the text categorization framework “region embedding + pooling”. We used an available region-embedding method [16] and found that LSTM was efficient. As no additional data or algorithms were required, we implemented end-to-end supervision. We then simplified the model, accelerating training and improving accuracy.

We encoded all text information in the last output of the RNN before running a feed-forward classification network very similar to a neural translation machine employing sequence-to-sequence learning (Fig. 6).

Figure 6.

The standard sequence-to-sequence model.

Also, we used bidirectional LSTM and concatenated both of the final outputs (Fig. 7).

Figure 7.

The LSTM structure for text categorization.

2) Simplifying Sub-problems of the Pooling Simplifying

First, we simplified the “region embedding + pooling” model. With LSTM, it is difficult to use a single vector to represent the entire document. We changed certain text regions to allow representation as simple vectors (Fig. 8).

Figure 8.

LSTM featuring pooling.

Thus, we expressed each time step as h_t, and created a document vector by aggregating by pooling layer. No matter how many words are included, if the document is not finished, the LSTM must remember relevant information. Thus, we took a different route to solve the problem simply. We used segments smaller than phrases or sentences, with a focus on representing concepts, and allowed the forget gate to discard old data.

3) Increasing Training Speed

To simplify the sub-problem, we accelerated training in the pooling layer. We used chopping, and we allowed LSTM to embed text regions instead of documents. After this transformation, the sequence (from the beginning to the end of the document) is not important. We separated each document into fragments of a fixed length, and we assumed that these segments were individual documents, which could be handled simultaneously. This accelerated training and improved accuracy.

4) Elimination of Input and Output Gates

When we eliminated the input and output gates, accuracy was not affected. In other words, the pooling layer was redundant and the time and memory requirements halved. This is because the input and output gates eliminate irrelevant information, but our max-pooling method filters noise, allowing removal of the gates. The LSTM formulation simplifies to:

(1)$$f_t\text{ = }\sigma (W^{\left(f\right)}{x}_t\text{+}{U}^{\left(f\right)}{h}_{t–1}\text{+}{b}^{\left(f\right)})$$

(2)$$u_t\hspace{0.17em}\text{=}\hspace{0.17em}\text{tan}h(W^{\left(f\right)}{x}_t\text{+}{U}^{\left(f\right)}{h}_{t–1}\text{+}{b}^{\left(u\right)})$$

(3)$$c_t\text{ = }{u}_t+f_t\odot c_{t–1},h_t=\text{tan}h(c_t)$$

This is equivalent to rendering all i_t and o_t constant. The model is mentally analogous to the gated recurrent units method [17] but is more straightforward, thus with fewer gates.

5) The Use of Two-Way LSTM to Improve Accuracy

The changes described above greatly reduced the time and memory required for training. If an opposite-direction pooling layer were to be added, what might happen? We coupled forward and backward LSTMs, which further improved accuracy.

1) Hardware Requirements

Our software runs only on GPUs, such as the Tesla K20. Testing was performed on a Tesla M2070 and a Quadro M6000.

2) Software Requirements

The makefile uses gcc; CUDA must be installed and CUDA v7.5 or higher is recommended. We do not know whether versions earlier than 5.0 work. Testing employed CUDA v7.5 and Linux. In principle, the code should compile and run in other systems as well (e.g., Windows), provided that a GPU and an appropriate version of CUDA are installed. However, we do not guarantee this. Optimally, Perl is required.

3) Experiments

We commence with a simple case lacking unsupervised embeddings. Using the context, the training and testing of neural networks feature the following steps:

a. Generate a vocabulary file from training data. Input a tokenized text file.

b. Generate input files for NN training, including a region file (containing features in the form of sparse region vectors); a target file (containing classification label information); and a word-mapping file (showing the mapping between words and the dimensions of the sparse region vectors).

c. Train the NN while (optionally) measuring classification error rates using validation and/or test data. Trained NNs can be saved.

d. Apply the trained NN to test data.

1) Datasets and Data Preprocessing

We found that our CNN and LSTM models were better than the traditional methods. Also, we improved existing deep learning methods as follows:

• •

  Use of a disordered CNN to evaluate text. Fewer parameters must be learned; word order is lost only within small regions.

• •

  Variable pool sizes for CNN text. The outputs to the fully connected top layer must be of a fixed length. As the number of pooling units is fixed, we dynamically determined pooling region size. We ensured that all data were covered, without overlaps.

• •

  Use of a collateral CNN. We used several types of embedding for small text regions. As the various CNNs engage in mutual assistance, model accuracy was improved.

• •

  Removal of input/output gates. On LSTM pooling LSTM, if the input and output gates were removed, model accuracy would not be compromised, but the times and memories required for training and testing would be halved.

• •

  Bidirectional LSTM affords improved accuracy. Bidirectional LSTM reduces the time and memory required for training and improves predictive accuracy.

Using the benchmark datasets, our results were better than previous results. However, our methods have certain disadvantages, principally suboptimal efficiency. Because we simplified the training methods, the training times became longer than those of traditional methods, including deep learning methods. Therefore, we plan to further explore sophisticated deep learning methods such as Deep Pyramid. We hope that we can use these methods to improve classification accuracy and efficiency.