Does Time Matter? Modeling Temporal Dynamics for Better Predictions

Julian Runge Julian Runge
November 4, 2015 AI & Machine Learning

This post is based on a joint research project with Pierangelo Rothenbuehler, who is also a part of the Experfy community***

Expert or intelligent systems can take many forms. In customer relationship management they should mostly help to understand the context of a customer: How does she feel? What are her motivations? What is she looking for? This allows us to provide meaningful context-based content to the customer. While motivations and intentions cannot be observed without mental involvement of the subject under study, behavioral observations are rather easy.

At least in the digital economy they are. Most off-the-shelf BI stacks will enable you to obtain dense behavioral observations of your customers in digital products. Can we infer motivations and intentions from this? Not really, but indirectly, by using predictive analytics and anticipating future behavior (e.g., a customer who is about to stop using your product is very likely to have low motivation for product use). Intelligent systems predicting churn have big value. For example, this previous Experfy blog post describes a case in the Telecom industry where it is very easy to switching providers. Using an expert system that predicts churn, customer relationship management can target customers who are likely to churn and give them incentives. These can be vouchers, discounts, gifts, or offerings to move to another product inside the company’s portfolio.

 

Methods for Binary Behavioral Prediction

By binary behavioral prediction we mean predicting if a behavior will occur or not. This is a classification problem. A number of techniques are applied to this – from the (most) common logistic regression over support vector machines to decision trees and neural networks. Whenever you face a prediction problem, time is inherently included in your model you want to predict a future value of a criterion using data available in the present. But in most cases your model and predictors do not address any sequential ordering. Temporal dynamics are not explicitly modeled – different lags of the predictors are all treated as static data. However, there can be value in incorporating sequential information in your prediction models. A very simple way of doing so is conducting some feature engineering on the time series of the predictors: i.e., taking the slope, the deltas, or the standard deviation of your time series observations and including them as predictors. See Figure 1 for an illustration of this. But of course, this is not where it stops. Other interesting approaches to sequence modeling include the following:

  • Time series econometrics as its own discipline, often in the form of panel data econometrics for individual behavioral predictions.
  • Survival analysis as a subtype for panel data analysis.
  • Markov models and Hidden Markov Models for modeling sequences of states.
  • Advanced efforts of including sequential information in other ways (e.g. Prinzie and van den Poel 2006, Eichinger et al. 2006)

 

Figure 1: An illustration of features addressing sequential information of the predictor time series; 14 day predictor time series; weekly delta on the left, weekly slopes on the right

 

The Application

In a recent project we looked into time feature engineering and Hidden Markov models to incorporate temporal dynamics in our predictive models. We work in the mobile app industry and adopted the freemium business model – there are two key binary behavioral prediction challenges that we face: Will a user churn from our app? Will a user spend money on in-app purchases? Both challenges were addressed previously, in the churn case and in the purchase case, but with limited attention to time and sequence modeling. In formal conversations, some peers reported that only using slopes had proven to result in the best model for them. In our predictive endeavors up to that point in time, the static lagged behavioral data had always done a good enough job. Hence, we would like to understand the predictive value that time/sequence modeling actually has. Let's get right to it.

Time Feature Engineering

We built churn prediction models for an app published on the iOS platform. We do not see a reason why the finding wouldn't hold for other behavioral predictions and other mobile apps. We ran a PCA on the predictors, thoroughly cross-validated and checked different algorithms.

Time feature engineering adds predictive value, but only marginally.

We used area under the ROC curve (AUC) to assess predictive performance. This is appropriate in light of the high class imbalance present in both churn and purchase prediction in freemium mobile apps (if you are interested in the reasons see Weiss 2004) Table 1 gives an overview of predictive performance of static features only, time features only, and both combined for different subsets of the customer base. As you can see, using time features only turns out to be a poor choice in our case, while static features only does a good job. The combination of both feature sets performs even better, but the predictive performance is not statistically significantly better.

 

Table 1: Mean AUC and its 95% confidence interval for different set of input features and different subsets of the dataset; logistic regression with 10-fold cross-validation

 

(Hidden) Markov Models

We skipped Markov models (research on Markov models for churn prediction: Burez and van den Poel 2007, Eastwood and Gabrys 2009) and moved right to the hidden ones because they are able to detect more complex states than their simpler siblings. Same app, but results are a bit more complex. HMMs are very different from regression models. To give them the attention they deserve, we decided to put the results into a short paper. Trying to summarize them is tough, but let me give it a shot:

We found that HMMs have roughly the same predictive performance as other algorithms.We also generated some pretty cool insights on transition patterns and user states. Moreover, one key advantage in practice is that HMMs are deployable at lower cost.

But please look into the paper to dive a bit deeper into our results. We will also be presenting it at SAI IntelliSys in London on November 11. If you can't make it, feel free to get in touch with your comments and feedback.

 

Figure 2: ROC curves for different algorithms; interpolated between operating points for HMM; HMM, neural network and logistic regression are pretty much head-to-head

 

So What?

While it is fascinating to get involved in complex and advanced methods and optimization, we shouldn’t forget that simple, robust, and fast solutions very often win in practice. Especially if your company is still at the beginning of its predictive analytics journey, we suggest you start off with a simple model (e.g., logistic regression or decision tree) and static data. Once you start seeing the value and devote more resources to advanced and predictive analytics, you can start with time and sequence modeling to aim for the higher hanging fruit.

  • Experfy Insights

    Top articles, research, podcasts, webinars and more delivered to you monthly.

  • Julian Runge

    Tags
    Predictive Analytics
    Leave a Comment
    Next Post
    Demystifying Facebook’s Ad Match Algorithm

    Demystifying Facebook's Ad Match Algorithm

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked *

    More in AI & Machine Learning
    AI & Machine Learning,Future of Work
    AI’s Role in the Future of Work

    Artificial intelligence is shaping the future of work around the world in virtually every field. The role AI will play in employment in the years ahead is dynamic and collaborative. Rather than eliminating jobs altogether, AI will augment the capabilities and resources of employees and businesses, allowing them to do more with less. In more

    5 MINUTES READ Continue Reading »
    AI & Machine Learning
    How Can AI Help Improve Legal Services Delivery?

    Everybody is discussing Artificial Intelligence (AI) and machine learning, and some legal professionals are already leveraging these technological capabilities.  AI is not the future expectation; it is the present reality.  Aside from law, AI is widely used in various fields such as transportation and manufacturing, education, employment, defense, health care, business intelligence, robotics, and so

    5 MINUTES READ Continue Reading »
    AI & Machine Learning
    5 AI Applications Changing the Energy Industry

    The energy industry faces some significant challenges, but AI applications could help. Increasing demand, population expansion, and climate change necessitate creative solutions that could fundamentally alter how businesses generate and utilize electricity. Industry researchers looking for ways to solve these problems have turned to data and new data-processing technology. Artificial intelligence, in particular — and

    3 MINUTES READ Continue Reading »

    About Us

    Incubated in Harvard Innovation Lab, Experfy specializes in pipelining and deploying the world's best AI and engineering talent at breakneck speed, with exceptional focus on quality and compliance. Enterprises and governments also leverage our award-winning SaaS platform to build their own customized future of work solutions such as talent clouds.

    Join Us At

    Contact Us

    1700 West Park Drive, Suite 190
    Westborough, MA 01581

    Email: support@experfy.com

    Toll Free: (844) EXPERFY or
    (844) 397-3739

    © 2023, Experfy Inc. All rights reserved.