AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

Andre Ye Andre Ye
September 23, 2020 AI & Machine Learning

Programming AutoML In Python with AutoKeras

Automated Machine Learning, commonly abbreviated as AutoML, is the automation of the building of neural network structures. Through intelligent architecture manipulations, AutoML can not only make deep learning more accessible for everyone but accelerate deep learning research.

In this article, we’ll go over:

  • How to install AutoKeras for neural architecture searches.
  • How to use AutoKeras to find the best neural architectures using structured, image, and text data for regression and classification tasks.
  • How to evaluate, predict, export to Keras/TensorFlow, and view architecture of obtained high-performing models.

A Glimpse into AutoKeras

  • With AutoKeras, a neural architecture search algorithm finds the best architectures, like the number of neurons in a layer, the number of layers, which layers to incorporate, layer-specific parameters like filter size or percent of dropped neurons in Dropout, etc. Once the search is complete, you can use the model as a normal TensorFlow/Keras model.
  • By using AutoKeras, you can build a model with complex elements like embeddings and spatial reductions that would otherwise be less accessible to those who are still in the process of learning DL.
  • When AutoKeras creates models for you, much of preprocessing, like vectorizing or cleaning text data, is done and optimized for you.
  • It takes two lines to initiate and train a search. AutoKeras boasts a Keras-like interface, so it’s not hard to remember and use at all.

Excited yet? Let’s get started!

Installation

Before you install, ensure you have the following prerequisites:

  • Python 3 (AutoKeras does not work with Python 2)
  • TensorFlow ≥ 2.3.0 (AutoKeras is based on TensorFlow)

In your command line, run two commands. This should properly install AutoKeras. Note that these installations, unfortunately, do not work in Kaggle notebooks, so it’s best to run these on a local environment.

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

As a test, run import autokeras in your coding environment to ensure everything is working.

Structured Data Classification/Regression Tasks

Let’s use the famous iris dataset, which we can import from sklearn’s several handy toy datasets. The result (data) of a data import is a dictionary with two keys, ‘data’ and ‘target’.

However, since the target is categorical (three values), we’ll need to dummy-encode it. Since the result is a pandas DataFrame and a NumPy array is desired, we call .values after pd.get_dummies (which one-hot encodes).

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

Calling X.shape yields (150, 4) and calling y.shape yields (150, 3). This makes sense — there are 150 rows, 4 columns in X, and 3 unique categories in y (hence three columns). In order to evaluate our model, we will split the data into training and testing sets.

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

It’s always good practice to check the shape of arrays:

  • X_train: (105, 4)
  • X_test: (45, 4)
  • y_train: (105, 3)
  • y_test: (45, 3)

Great! — everything adds up. We can begin working with autokeras.

Importing autokeras should be no problem. A StructuredDataClassifier is a search object that works on ‘structured data’, or standard two-dimensional data with columns and a label. The max_trials parameter indicates the maximum number of models to test; this can be set higher in the case of the iris dataset since it is so small. Often the search will end before max_trials is met (it serves as an upper bound).

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

For regression problems, use StructuredDataRegressor.

We can initiate the search process by calling .fit(). verbose is a parameter that can be set to 0 or 1, depending on if you would like the model to output information about training, like the architecture of potential networks and epoch-by-epoch progress. However, there are hundreds of epochs for each potential architecture, so it may take up space and slow down training.

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

It’s always worth setting verbose=1 at least once to peek into what is going on behind the scenes, since the entire search takes at least several minutes to complete. Take this sample output:

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

If you’re up to it, it’s fascinating to watch the architectures transform over time, and what elements are going into the final neural network.

On my local environment, it took ~1 hour in total with no GPU acceleration to run through 15 trials.

Once your model has been fit, we can evaluate its performance or obtain predictions. Generally, this is a quick process so verbose can be set to 0.

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

To view more details about the model structure and to save it, we need to export it using model = search.export_model(). Here, model is a standard TensorFlow/Keras model, not a AutoKeras object.

Calling model.summary() will print out best architecture obtained by the neural architecture search.

AutoML: Creating Top-Performing Neural Networks Without Defining Architectures

To save the weights, call model.save(‘filepath/weights.h5’).

Image Classification/Regression

Image classification and regression work much like classification and regression on standard structured data, but the neural network is built using elements of convolutional neural networks, like convolutional layers, max pooling, and flattening. All parameters are learned.

For example, let’s work on the MNIST dataset to find a convolutional neural network to recognize digits from 28 by 28 images. Keras datasets provide a handy method to retrieve this data, but the y-variables need to be converted into dummy variables.

y-variables need to be converted into dummy variables

Let’s check the shape of the data.

  • X_train: (60000, 28, 28)
  • X_test: (10000, 28, 28)
  • y_train: (60000, 10)
  • y_test: (10000, 10)

Great! AutoKeras can also handle four-dimensional data (colored images with multiple channels). We can create a search object with ImageClassifier (or, ImageRegressor for regression tasks)

ImageClassifier (or, mageRegressor for regression tasks)

*If you are running multiple searches, add a parameter overwrite=True when defining the search object (ak.object(overwrite=True, max_trials=x)). Otherwise, there will be an error.

Evaluation, prediction, and exporting are performed the same as with structured classifiers and regressors. Note that a neural architecture search with images will take a substantially longer period of time.

Text Classification/Regression

AutoKeras doesn’t require any text vectorization, which is handy because there are so many ways to do so and each can be lengthy to implement. Instead, this is learned by the model.

Let’s use the 20 News Groups dataset, which consists of several news excerpts belonging to 20 different categories. To save time and computing expenses, we’ll only choose to import four categories and have a rather large test-size split (half the dataset).

 large test-size split (half the dataset)

X_train and X_test each are arrays comprised of 1128 raw text strings, y_train and y_test are each arrays of shape (1128, 4).

Note that AutoKeras can handle strings of variable lengths! Each string in the input does not need to be the same length.

Next, let’s build our text classifier/regressor.

build our text classifier/regressor

*If you are running multiple searches, add a parameter overwrite=True when defining the search object (ak.object(overwrite=True, max_trials=x)). Otherwise, there will be an error.

Evaluation, prediction, and exporting are performed the same as with structured classifiers and regressors.

It’s worth taking a look at the proposed networks. The fusion between NLP and Deep Learning can be extraordinary advanced, and it’s amazing that complex elements like embeddings, spatial reductions, etc. can be implemented without needing to be explicitly called.

https://towardsdatascience.com/media/ef7e5ecedaeca96d93195cc9d9652ba3

proposed networks

With AutoKeras, deep learning becomes more accessible for all!

Source: Towards Data Science.

  • Experfy Insights

    Top articles, research, podcasts, webinars and more delivered to you monthly.

  • Andre Ye

    Tags
    AutoKerasAutomated Machine LearningAutoMLPython
    Leave a Comment
    Next Post
    Everything You Should Know About Gradient Descent

    Everything You Should Know About Gradient Descent

    Leave a Reply Cancel reply

    Your email address will not be published. Required fields are marked *

    More in AI & Machine Learning
    AI & Machine Learning,Future of Work
    AI’s Role in the Future of Work

    Artificial intelligence is shaping the future of work around the world in virtually every field. The role AI will play in employment in the years ahead is dynamic and collaborative. Rather than eliminating jobs altogether, AI will augment the capabilities and resources of employees and businesses, allowing them to do more with less. In more

    5 MINUTES READ Continue Reading »
    AI & Machine Learning
    How Can AI Help Improve Legal Services Delivery?

    Everybody is discussing Artificial Intelligence (AI) and machine learning, and some legal professionals are already leveraging these technological capabilities.  AI is not the future expectation; it is the present reality.  Aside from law, AI is widely used in various fields such as transportation and manufacturing, education, employment, defense, health care, business intelligence, robotics, and so

    5 MINUTES READ Continue Reading »
    AI & Machine Learning
    5 AI Applications Changing the Energy Industry

    The energy industry faces some significant challenges, but AI applications could help. Increasing demand, population expansion, and climate change necessitate creative solutions that could fundamentally alter how businesses generate and utilize electricity. Industry researchers looking for ways to solve these problems have turned to data and new data-processing technology. Artificial intelligence, in particular — and

    3 MINUTES READ Continue Reading »

    About Us

    Incubated in Harvard Innovation Lab, Experfy specializes in pipelining and deploying the world's best AI and engineering talent at breakneck speed, with exceptional focus on quality and compliance. Enterprises and governments also leverage our award-winning SaaS platform to build their own customized future of work solutions such as talent clouds.

    Join Us At

    Contact Us

    1700 West Park Drive, Suite 190
    Westborough, MA 01581

    Email: support@experfy.com

    Toll Free: (844) EXPERFY or
    (844) 397-3739

    © 2023, Experfy Inc. All rights reserved.