2019
Use Advances in Data Science and Computing Power to Invest in Stock Market
Mustafa A. Sakarwala Southern Methodist University, msakarwala @mail.smu.edu, Anthony Tanaydin Southern Methodist Unversity, atanaydin @mail.smu.edu
https://scholar.smu.edu/cgi/viewcontent.cgi?article=1078&context=datasciencereview
Primer on Natural Language Processing
Natural Language Processing (NLP) deals with building computational algorithms to automatically analyze and represent human language.
With the recent popularity and success of word embeddings (low dimensional, distributed representations), neural-based models have achieved superior results on various language-related tasks as compared to traditional machine learning models like SVM or logistic regression.
Words embeddings are pre-trained on a task where the objective is to predict a word based on its context, typically using a shallow neural network
Distributed representations gained popularity when the continuous bag of words (CBOW) and skip-gram model were introduced to the field. They were popular because they could efficiently construct high-quality word embeddings and could also be used for semantic compositionality
CBOW is a neural approach to construct word embeddings and the objective is to compute the conditional probability of a target word given the context words in a given window size. On the other hand, Skip-gram is a neural approach to construct word embeddings, where the goal is to predict the surrounding context words (i.e., conditional probability) given a central target word. For both models, the word embedding dimension is determined by computing (in an unsupervised manner) the accuracy of the prediction.
One of the challenges with word embedding methods is when we want to obtain vector representations for phrase.
We can’t just simply combine the individual word vector representations since these phrases don’t represent the combination of the meaning of the individual words. And it gets even more complicated when longer phrases and sentences are considered. The other limitation with the word2vec models is that the use of smaller window sizes produce similar embeddings for contrasting words such as “good” and “bad”, which is not desirable especially for tasks where this differentiation is important such as sentiment analysis. These models also suffer models also suffer from other problems such as not taking into account polysemy and other biases that may surface from the training data.
For tasks such as parts-of-speech (POS) tagging or named-entity-recognition (NER), it is useful to look at morphological information in words such as characters and combinations thereof. Since we are analyzing the text at the character level, these type of embeddings are known as ‘Character Embeddings’. They help to deal with the unknown word issue as we are no longer representing sequences with large word vocabularies that need to be reduced for efficient computation purposes.
It has recently been found that word vectors are limited in how well they capture the different facets of conceptual meaning behind words.
Global Vectors (GloVe)
The GloVe model stands for Global Vectors which is an unsupervised learning model which can be used to obtain dense word vectors similar to Word2Vec. However the technique is different and training is performed on an aggregated global word-word co-occurrence matrix, giving us a vector space with meaningful sub-structures. This method was invented in Stanford by Pennington et al.
The basic methodology of the GloVe model is to first create a huge wordcontext co-occurrence matrix consisting of (word, context) pairs such that each element in this matrix represents how often a word occurs with the context (which can be a sequence of words). The idea then is to apply matrix factorization to approximate this matrix.
Considering the Word-Context (WC) matrix, Word-Feature (WF) matrix and Feature-Context (FC) matrix, we try to factorize WC = WF x FC, such that we we aim to reconstruct WC from WF and FC by multiplying them. For this, we typically initialize WF and FC with some random weights and attempt to multiply them to get WC’ (an approximation of WC) and measure how close it is to WC. We do this multiple times using Stochastic Gradient Descent (SGD) to minimize the error. Finally, the Word-Feature matrix (WF) gives us the word embeddings for each word where F can be preset to a specific number of dimensions. A very important point to remember is that both Word2Vec and GloVe models are very similar in how they work. Both of them aim to build a vector space where the position of each word is influenced by its neighboring words based on their context and semantics. Word2Vec starts with local individual examples of word co-occurrence pairs and GloVe starts with global aggregated co-occurrence statistics across all words in the corpus.
---------------------
More About Word embeddings (AI Summary)
Word embeddings are dense, continuous numerical vector representations of words in a multi-dimensional space, built on the distributional hypothesis that words appearing in similar linguistic contexts tend to share similar meanings.
Learn more about foundational concepts and explore detailed overviews on IBM's Word Embeddings Topic Page.
Core Foundations & The Distributional Hypothesis
• The Distributional Hypothesis: First formalized by Zellig Harris and J.R. Firth in the 1950s, it posits that "a word is characterized by the company it keeps." If coffee and tea appear alongside words like drink, cup, and morning, their vector representations will end up close together.
• Moving Beyond One-Hot Vectors: Traditional sparse representations (like one-hot encoding) treat every vocabulary item as an orthogonal dimension, failing to capture semantic relatedness or scaling poorly with large vocabularies. Embeddings solve this by mapping words to low-dimensional, continuous vector spaces where distance and direction encode meaning.
Main Methods for Creating Distributional Vectors
1. Count-Based (Frequency-Based) Methods
• Mechanism: Build a large co-occurrence matrix (|V| × |V| or word-by-context) counting how often words appear together within a defined sliding window w.
• Characteristics: Smaller windows capture syntactic/grammatical roles, while larger windows capture broader topical or semantic fields.
• Examples: Explicit co-occurrence counts analyzed via dimensionality reduction (like SVD or Pointwise Mutual Information / PMI).
2. Prediction-Based (Neural) Methods
• Mechanism: Train shallow neural networks or self-supervised classifiers to predict a target word from its context (or vice versa) over a text corpus.
• Key Architectures:
• Word2Vec (CBOW): Predicts a target center word given a surrounding context window.
• Word2Vec (Skip-gram / SGNS): Predicts surrounding context words given a target word, using negative sampling to scale efficiently.
• GloVe (Global Vectors): Combines the global co-occurrence matrix statistics of count-based methods with the local context-window advantages of predictive models by factorizing log-co-occurrence probabilities.
• FastText: Extends Word2Vec by breaking words down into sub-word character n-grams, enabling it to generate vectors for out-of-vocabulary (OOV) or misspelled words.
Key Properties & Limitations
• Semantic Arithmetics: Dense vectors support classic vector offset analogies, famously illustrated by vector(king) - vector(man) + vector(woman) ≈ vector(queen).
• Static vs. Contextual Limitations: Traditional distributional embeddings (Word2Vec, GloVe) are static—they assign a single, rigid vector to a word regardless of context, meaning polysemous words (like a financial bank vs. a river bank) collapse into one averaged representation. Modern transformer models (such as BERT or GPT) superseded static embeddings with contextualized embeddings that dynamically shift based on surrounding text.
IBM AI
What are word embeddings?
Published 29 December 2023
--------------------
Primer on Deep Learning and Modelling approach
Deep learning is a subset of machine learning where artificial neural networks, algorithms inspired by the human brain, learn from large amounts of data. Similarly to how we learn from experience, the deep learning algorithm would perform a task repeatedly, each time tweaking it a little to improve the outcome
Multi Layer Perceptron (MLP)
Artificial Neural Networks have led to many breakthrough results in speech recognition, computer vision and text processing. Multi-layer Perceptron (MLP) is a supervised learning algorithm that learns a function by training on a dataset, where m is the number of dimensions for input and o is the number of dimensions for output.
Given a set of features X =x1,x2,...,xm and a target y , it can learn a non-linear function approximator for either classification or regression.
Between the input and the output layer, there can be one or more non-linear layers, called hidden layers.
Input Layer: The Input layer has a Bias node has a value of 1 and other nodes one for each external inputs (which are numerical values depending upon the input dataset). No computation is performed in the Input layer. So the outputs from nodes in the Input layer are 1, X1 and X2 respectively, which are fed into the Hidden Layer. Hidden Layer:
The Hidden layer also has the Bias node having an output of 1. Calculation is done in this layer on the outputs from the input later. The output of the nodes in the Hidden layer depends on the outputs from the Input layer (1, X1, X2 upto Xm) as well as the weight associated with the connections (edges) between input layer and this hidden layer.. If there are more hidden layers, similar calculations are done Similarly, the outputand at the end the output is fed to the output layer.
Output Layer: The Output layer m nodes which take inputs from the Hidden layer and perform similar computations as done in the hidden layers. As we can understand the weights associated with edges keep changing at every hidden layer and output layers. The values calculated (Y1 and Y2 upto Ym) as a result of these computations act as outputs of the Multi Layer Perceptron.
Convolutional Neural Networks (CNN)
Following the popularization of word embeddings and its ability to represent words in a distributed space, the need arose for an effective feature function that extracts higher-level features from constituting words or n-grams. These abstract features would then be used for numerous NLP tasks such as sentiment analysis, summarization, machine translation, and question answering (QA). Convolutional Neural Networks (CNN) turned out to be the natural choice given their effectiveness in computer vision tasks. A CNN is basically a neural-based approach which represents a feature function that is applied to constituting words or n-grams to extract higher-level features.
CNN-based frameworks - Application to NLP tasks
The goal of their method was to transform words into a vector representation via a lookup table, which resulted in a primitive word embedding approach that learns weights during the training of the network. In order to perform sentence modeling with a basic CNN, sentences are first tokenized into words which are then transformed into a word embedding matrix (i.e. input embedding layer) of d dimension. Then, convolution filters are applied to this input embedding layer which consists of applying a filter of all possible window sizes to produce what is called a feature map. This is then followed by a max-pooling operation which applies a max operation on each filter to obtain a fixed length output and reduce the dimensionality of the output. And that procedure produces the final sentence representation. By increasing the complexity of the basic CNN model explained above, other NLP tasks such as Named Entity Recognition and POS can be achieved. This requires a window-based approach, where for each word a fixed size window of neighboring words (sub-sentence) is considered. Then a standalone CNN is applied to the sub-sentence and the training objective is to predict the word in the center of the window, also referred to as word-level classification. CNNs fail to model long distance dependencies, which is important for various NLP tasks. To address this problem, CNNs are coupled with time-delayed neural networks (TDNN). Dynamic CNN (DCNN) is another useful type of CNN that has shown success in different NLP tasks, such as sentiment prediction and question answering. Overall, CNNs are effective because they can mine semantic clues in contextual windows, but they struggle to preserve sequential order and model long distance contextual information.
Recurrent models (RNN) are better suited for such type of learning and they are discussed next.
Recurrent Neural Networks (RNN)
Recurrent Neural Networks are specialized neural-based approaches that are effective at processing sequential information. The term “recurrent” applies as they perform the same task over each instance of the sequence such that the output is dependent on the previous computations and results. These sequences are typically represented by a fixed-size vector of tokens which are fed sequentially (one by one) to a recurrent unit. The main strength of an RNN is its capacity to memorize the results from the previous computations and use it for the current computation. This makes RNN a good candidate for modeling context dependencies for inputs of arbitrary length. RNNs have been used to study various NLP tasks such as machine translation, image captioning, and language modeling, among others. Simple RNNs suffer from the infamous vanishing gradient problem, which makes it really hard to learn and tune the parameters of the earlier layers in the network. Long Short Term Memory (LSTM) networks, Residual Networks (ResNets) and Gated Recurrent Networks (GRU) were later introduced to overcome this limitation. An LSTM consist of three gates (input, forget, and output gates), and calculate the hidden state through a combination of the three. GRUs are similar to LSTMs but consist of only two gate and are more efficient because they are less complex. A study shows that it is difficult to say which of the gated RNNs are more effective, and they are usually picked based on the computing power available. Various LSTM-based models have been proposed for the sequence to sequence mapping (via encoder decoder frameworks) that are suitable for machine translation, text summarization, modeling human conversations, question answering, image-based language generation, among other tasks.
Data Downloading and Analysis
Reading data from the SEC EDGAR and AlphaVantage
As a first step, we download for S&P 500 companies as of November 2018, the list of 8-K document links from the SEC EDGAR website using the BeautifulSoup python library.
Then for each link, we download the associated text reports from the SEC EDGAR website using BeautifulSoup and parse metadata such as the date and time of document release and the categories of disclosures made were extracted, while tables and charts were discarded.
We also gather the historical stock prices (close, open, adjusted-close) using the Alpha-Vantage python api.
Next, for each document the corresponding release date, one year, one quarter and one month historical moving average price movements were calculated and normalized by the change in the S&P 500 indices. All windows refer to the days that the NYSE and Nasdaq were actually open (non-holiday weekdays).
Historical index prices for the Volatility Index (VIX) and GSPC (S&P 500 Index) were downloaded from Yahoo Finance. (VIX) CBOE Volatility Index- Created by the Chicago Board Options Exchange (CBOE), the Volatility Index, or VIX, is a real-time market index that represents the market’s expectation of 30-day forward-looking volatility. Derived from the price inputs of the S&P 500 index options, it provides a measure of market risk and investors’ sentiments. It is also known by other names like “Fear Gauge” or “Fear Index.” Investors, research analysts and portfolio managers look to VIX values as a way to measure market risk, fear and stress before they take investment decisions (CIK).
GSPC- This is the ticker symbol for the S&P 500 Index. GSPC index is calculated by taking the sum of the adjusted market capitalization of all S&P 500 stocks and then dividing it with an index divisor, which is a proprietary figure developed by Standard & Poor’s. The target feature is calculated as the change in the price of a stock right before and after a document’s release, normalized by the change in the S&P 500. For example, for a company that released a document on October 11, 2018, the change in its opening and adjusted close price is calculated, minus the change during the same time for the S&P 500 indices. The normalized changes are labeled as either “up” (> 1%), “down” (<-1%), or “stay“(between-1 and 1%).
Then we combine the 8-K data with financial data obtain from AlphaVantage. To do this, we use the company names, tickers and central index key (CIK), and match for every stock index the companies with the appropriate Form 8-K. Because of the size of the data approximately 20000 documents and time needed to scrape and collect it, we had to split them into separate parts for each stock ticker and download them separately and store them as separate csv files.
NLP Exploration
The text is preprocessed by removing stopwords, punctuations and numbers, by lemmatizing and by converting to lower case. This is accomplished using the NLTK WordNet corpus reader in combination with Dask for a multithreading speed boost. All documents are padded with zeros to a uniform length of 33300 words. This cutoff was chosen at the 90th percentile of document lengths in order to preserve more textual information, but prevent the data set from becoming unnecessarily large. The Stanford NLP Wikipedia 2014 + Gigaword 100 dimensions were chosen for pre-trained word embedding’s under the assumption that it would carry information for specialized, industry-specific words found in the texts since it was trained from the Wikipedia corpus. After dropping duplicate samples, and texts for which no release date could be extracted, the final dataset consisted of about 20,000 documents for 505 companies.
Deep Learning
All categorical features are one hot encoded, and continuous features such as recent stock movements, and closing prices of VIX were standardized to have a mean of 0 and standard deviation of 1. The dataset is then randomly shuffled and split into 80% train and 20% test data.
The dataset suffered from imbalanced classes, with over 50% of samples labeled as (“up”), which intuitively makes sense considering the steady rise of the S&P 500 over the last decade. Oversampling on the training data is used to correct for this, with randomly selected samples in each class duplicated to have an equal number of samples of each of the three classes.
We constructed a model using Keras with a Tensorflow backend, consisting of two input layers (one for text documents, one for features), an embedding layer with the pre-trained GloVe vectors, and a multilayer perceptron(MLP) fully connected network. The network was trained for 10 epochs, with a batch size of 16 due to memory limitation. The model.fit method returns a History callback, which has a history attribute containing the lists of successive losses and other metrics.
Further extending the study to use other architectures like RNN, CNN, RNN-CNN we have achieved better accuracy, lesser loss.
Results
The lower the loss, the better a model (unless the model has over-fitted to the training data). The loss is calculated on training and validation and its interpretation is how well the model is doing for these two sets. Unlike accuracy, loss is not a percentage. It is a summation of the errors made for each example in training or validation sets.
In the case of neural networks, the loss is usually negative log-likelihood and the residual sum of squares for classification and regression respectively. Then, naturally, the main objective in a learning model is to reduce (minimize) the loss function’s value with respect to the model’s parameters by changing the weight vector values through different optimization methods, such as back-propagation in neural networks. The loss value implies how well or poorly a certain model behaves after each iteration of optimization. Ideally, one would expect the reduction of loss after each, or several, iteration(s).
Loss used is the Categorical Cross-Entropy Loss.
The accuracy of a model is usually determined after the model parameters are learned and fixed and no learning is taking place. Then the test samples are fed to the model and the number of mistakes (zero-one loss) the model makes are recorded, after comparison to the true targets. Then the percentage of misclassification is calculated. AUC-Roc visualizes performance classification. The higher, the better.
We have performed MLP, RNN, CNN and RNN-CNN to the dataset and the result can be seen in the table and graphs below.
Table 2. Below shows the comparison of result across the different models.
Table 2. Comparison of Results across Models
MLP, RNN, CNN and RNN-CNN respectively.
Model Loss Accuracy AOC-ROC
MLP 1.1611 58% 0.7129
CNN 0.8077 65% 0.8046
RNN 0.8017 66% 0.8166
CNN-RNN 0.7990 68% 0.8162
.As any project that does not take a straight path towards completion, we hit several roadblocks while implementing this project like, handling humongous amount of data, figuring out the hardware requirements for data processing and modeling and learning new techniques to improve the project.
---------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
NLP for Stock Market Prediction with Reddit Data
Stanford CS224N Custom Project
Muxi Xu, Department of Computer Science Stanford University muxi @stanford.edu
Abstract
Reddit and the WallStreetBet subreddit has become a very hot topic on the capital market from the beginning of 2021. The discussions on these forums show the potential to influence the stock market. My project is to build a model to forecast the market movement based on the rich text data from Reddit. Specifically, I have explored sentence embedding, document embedding, CNN-based model, and sentiment analysis methods to leverage the sentence of posts & comments information for market forecasting. This project has tested and compared several types of model architectures. So far, the performance shows that the model could slightly improve performance from the naive forecasting method.
https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1214/reports/final_reports/report030.pdf
Natural Language Processing and Speech - StaNFORD FACULTY
SMU Data Science Review - > Vol. 9 (2025) > No. 3
https://scholar.smu.edu/datasciencereview/vol9/iss3/
Application of Open-Source Small Large Language Models for Finance report analysis
https://scholar.smu.edu/datasciencereview/vol9/iss3/2/
A Modular Framework for Cost-Efficient Aspect-Based Sentiment Analysis Using Small Language Models
https://scholar.smu.edu/datasciencereview/vol9/iss3/4/
Trading the Stock Market using Google Search Volumes
Nikolaos Kalampalikis, 2018
https://www.academia.edu/130124956/Trading_the_Stock_Market_using_Google_Search_Volumes
https://www.linkedin.com/pulse/data-science-equity-analysis-sumiyuru-rajaguru-jxlqc/
Open access - Review article
First published online February 3, 2025
The Feasibility and Comparability of Using Artificial Intelligence for Qualitative Data Analysis in Equity-Focused Research
Yan Jiang et al.
Volume 54, Issue 3
https://journals.sagepub.com/doi/full/10.3102/0013189X251314821
Why data science is a key skill for investment professionals
