Back to Community
Analyzing a Signal and Creating a Contest Algorithm with Self-Serve Data

A few weeks back we announced the release of Self-Serve Data, which provides you the ability to upload your own time-series data to Quantopian and access it in Pipeline.

Refer to this forum post by Chris Myles for the initial release announcement of Self-Serve Data. For another high-level walkthrough of the tool, watch this recently-released video, Upload Your Custom Datasets with Self-Serve Data.

Analyze Your Signal

Seamless integration with the Pipeline API means you can use your dataset as you would any other on the Quantopian platform, which includes compatibility with tools like Alphalens and Pyfolio.

The attached notebook uses Self-Serve Data to load a signal into Pipeline and analyze it using Alphalens.

Template Contest Algorithm

Because Self-Serve Data allows you to live-update your datasets in point-in-time fashion, your data can now be used in algorithms you submit to the daily Quantopian Contest.

We’ve prepared and attached (in the first response to this post) a template algorithm to clone and build off of with your uploaded data. The TO-DOs throughout the template provide guidance as you develop the algorithm for submission to the contest. The template should help you pass most, if not all of the criteria required for the contest.

Feel free to email any questions or issues to [email protected].

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

2 responses

Attached is the template algorithm you can start with to build an algorithm with your uploaded data. Follow the TO-DOs throughout the template for guidance.

Clone Algorithm
41
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Self-Serve Data allows you to both upload historical data as well as live-update your data on a regular basis. Because of this live-updating capability and the point-in-time capture of your data, your data can now be used in algorithms you submit to the daily Quantopian Contest.

If you want to live update a dataset, you need to establish an FTP or host a file somewhere (like Dropbox or Google Sheets) and keep it up to date. Files are checked for new data on a nightly basis. You can read more about live updating datasets in the help documentation: https://www.quantopian.com/help#self_serve_data.

For a guide on how to upload your data, check out the Introduction to Self-Serve Data notebook attached to this forum post:
https://www.quantopian.com/posts/upload-your-custom-datasets-and-signals-with-self-serve-data#5b28fffc986d4e0048148f80

Once you've uploaded your dataset and configured live updates, follow the TO-DOs in the sample algorithm below to incorporate your data and develop an algorithm for the Quantopian Contest.

This example algorithm uses your uploaded dataset and uses the resulting pipeline to make trading decisions. A pipeline screen checks that uploaded data is not more than 3 business days old.
"""

import quantopian.algorithm as algo
import quantopian.optimize as opt
from quantopian.pipeline import Pipeline
from quantopian.pipeline.factors import BusinessDaysSincePreviousEvent, SimpleMovingAverage

from quantopian.pipeline.filters import QTradableStocksUS
from quantopian.pipeline.experimental import risk_loading_pipeline

from quantopian.pipeline.data import Fundamentals

# TODO: Un-comment this line to import your dataset here.
# from quantopian.pipeline.data.user_[user_id] import [dataset_name]

# Constraint Parameters
MAX_GROSS_LEVERAGE = 1.0
MAX_SHORT_POSITION_SIZE = 0.02
MAX_LONG_POSITION_SIZE = 0.02

def initialize(context):
    """
    A core function called automatically once at the beginning of a backtest.

    Use this function for initializing state or other bookkeeping.

    Parameters
    ----------
    context : AlgorithmContext
        An object that can be used to store state that you want to maintain in 
        your algorithm. context is automatically passed to initialize, 
        before_trading_start, handle_data, and any scheduled functions.
        context provides the portfolio attribute, which can be used to retrieve 
        information about current positions.
    """
    
    # TODO: Un-comment this section to enable the call to make_pipeline():
    # algo.attach_pipeline(make_pipeline(), 'my_data_template')

    # Attach the pipeline for the risk model factors that we
    # want to neutralize in the optimization step. The 'risk_factors' string is 
    # used to retrieve the output of the pipeline in before_trading_start below.
    algo.attach_pipeline(risk_loading_pipeline(), 'risk_factors')

    # TODO: Schedule your trade logic via this rebalance function.
    # algo.schedule_function(
    #     func=rebalance,
    #     date_rule=algo.date_rules.week_start(),
    #     time_rule=algo.time_rules.market_open(hours=0, minutes=30)
    # )


def make_pipeline():
    """
    A function that creates and returns our pipeline.

    We break this piece of logic out into its own function to make it easier to
    test and modify in isolation. In particular, this function can be
    copy/pasted into research and run by itself.

    Returns
    -------
    pipe : Pipeline
        Represents computation we would like to perform on the assets that make
        it through the pipeline screen.
    """
    
    base_universe = QTradableStocksUS()
    
    # TODO: Replace with your dataset name.
    days_since_last_update = BusinessDaysSincePreviousEvent(
        inputs=[[dataset_name].asof_date.latest]
    )
    
    # TODO: Update threshold based on live update frequency
    has_recent_update = (days_since_last_update < 3)
    universe = (has_recent_update & base_universe)
    
    # TODO: Replace with your dataset name and the name of your value column. In this example, we assume your value column is a number:
    factor = [dataset_name].value_column.latest

    # Create pipeline
    pipe = Pipeline(
        columns={
            'factor': factor,
        },
        screen=factor.notnull() & universe
    )
    return pipe


def before_trading_start(context, data):
    """
    Optional core function called automatically before the open of each market day.

    Parameters
    ----------
    context : AlgorithmContext
        See description above.
    data : BarData
        An object that provides methods to get price and volume data, check
        whether a security exists, and check the last time a security traded.
    """

    # TODO: Un-comment this line to store the output of your pipeline in the context variable:
    # context.pipeline_data = algo.pipeline_output('my_data_template')

    # This dataframe will contain all of our risk loadings
    context.risk_loadings = algo.pipeline_output('risk_factors')
    

def rebalance(context, data):
    """
    A function scheduled to run once every Monday at 10AM ET in order to
    rebalance the longs and shorts lists.

    Parameters
    ----------
    context : AlgorithmContext
        See description above.
    data : BarData
        See description above.
    """
    # Retrieve pipeline output.
    pipeline_data = context.pipeline_data

    # Retrieve risk model factor data.
    risk_loadings = context.risk_loadings

    # Here we define our objective for the Optimize API. In this template 
    # we use the MaximizeAlpha objective.
    objective = opt.MaximizeAlpha(pipeline_data.factor)

    # Define the list of constraints.
    constraints = []
    
    # Constrain our maximum gross leverage.
    constraints.append(opt.MaxGrossExposure(MAX_GROSS_LEVERAGE))

    # Require our algorithm to remain dollar neutral.
    constraints.append(opt.DollarNeutral())

    # Add the RiskModelExposure constraint to make use of the
    # default risk model constraints.
    neutralize_risk_factors = opt.experimental.RiskModelExposure(
        risk_model_loadings=risk_loadings,
        version=0
    )
    constraints.append(neutralize_risk_factors)

    # With this constraint we enforce that no position can make up
    # greater than MAX_SHORT_POSITION_SIZE on the short side and
    # no greater than MAX_LONG_POSITION_SIZE on the long side. This
    # ensures that we do not overly concentrate our portfolio in
    # one security or a small subset of securities.
    constraints.append(
        opt.PositionConcentration.with_equal_bounds(
            min=-MAX_SHORT_POSITION_SIZE,
            max=MAX_LONG_POSITION_SIZE
        ))

    # Put together all the pieces we defined above by passing
    # them into the algo.order_optimal_portfolio function. This handles
    # all of our ordering logic, assigning appropriate weights
    # to the securities in our universe to maximize our alpha with
    # respect to the given constraints.
    algo.order_optimal_portfolio(
        objective=objective,
        constraints=constraints
    )
There was a runtime error.

For those interested, attached is the template algorithm incorporating the campaign contributions dataset used in the notebook above.

Clone Algorithm
26
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Self-Serve Data allows you to both upload historical data as well as live-update your data on a regular basis. Because of this live-updating capability and the point-in-time capture of your data, your data can now be used in algorithms you submit to the daily Quantopian Contest.

If you want to live update a dataset, you need to establish an FTP or host a file somewhere (like Dropbox or Google Sheets) and keep it up to date. Files are checked for new data on a nightly basis. You can read more about live updating datasets in the help documentation: https://www.quantopian.com/help#self_serve_data.

For a guide on how to upload your data, check out the Introduction to Self-Serve Data notebook attached to this forum post:
https://www.quantopian.com/posts/upload-your-custom-datasets-and-signals-with-self-serve-data#5b28fffc986d4e0048148f80

Once you've uploaded your dataset and configured live updates, follow the TO-DOs in the sample algorithm below to incorporate your data and develop an algorithm for the Quantopian Contest.

This example algorithm uses your uploaded dataset and uses the resulting pipeline to make trading decisions. A pipeline screen checks that uploaded data is not more than 3 business days old.
"""

import quantopian.algorithm as algo
import quantopian.optimize as opt
from quantopian.pipeline import Pipeline
from quantopian.pipeline.factors import BusinessDaysSincePreviousEvent, SimpleMovingAverage

from quantopian.pipeline.filters import QTradableStocksUS
from quantopian.pipeline.experimental import risk_loading_pipeline

from quantopian.pipeline.data import Fundamentals

# TODO: Un-comment this line to import your dataset here.
from quantopian.pipeline.data.user_5981db8a17ec162547ac3b2b import campaign_contributions

# Constraint Parameters
MAX_GROSS_LEVERAGE = 1.0
MAX_SHORT_POSITION_SIZE = 0.02
MAX_LONG_POSITION_SIZE = 0.02

def initialize(context):
    """
    A core function called automatically once at the beginning of a backtest.

    Use this function for initializing state or other bookkeeping.

    Parameters
    ----------
    context : AlgorithmContext
        An object that can be used to store state that you want to maintain in 
        your algorithm. context is automatically passed to initialize, 
        before_trading_start, handle_data, and any scheduled functions.
        context provides the portfolio attribute, which can be used to retrieve 
        information about current positions.
    """
    
    # TODO: Un-comment this section to enable the call to make_pipeline():
    algo.attach_pipeline(make_pipeline(), 'my_data_template')

    # Attach the pipeline for the risk model factors that we
    # want to neutralize in the optimization step. The 'risk_factors' string is 
    # used to retrieve the output of the pipeline in before_trading_start below.
    algo.attach_pipeline(risk_loading_pipeline(), 'risk_factors')

    # TODO: Schedule your trade logic via this rebalance function.
    algo.schedule_function(
        func=rebalance,
        date_rule=algo.date_rules.week_start(),
        time_rule=algo.time_rules.market_open(hours=0, minutes=30)
    )


def make_pipeline():
    """
    A function that creates and returns our pipeline.

    We break this piece of logic out into its own function to make it easier to
    test and modify in isolation. In particular, this function can be
    copy/pasted into research and run by itself.

    Returns
    -------
    pipe : Pipeline
        Represents computation we would like to perform on the assets that make
        it through the pipeline screen.
    """
    
    base_universe = QTradableStocksUS()
    
    # TODO: Replace with your dataset name.
    days_since_last_update = BusinessDaysSincePreviousEvent(
        inputs=[campaign_contributions.asof_date.latest]
    )
    
    # TODO: Update threshold based on live update frequency
    has_recent_update = (days_since_last_update < 3)
    universe = (has_recent_update & base_universe)
    
    # TODO: Replace with your dataset name and the name of your value column. In this example, we assume your value column is a number. We use a factor for 6-month simple moving average of campaign contributions count:
    factor = SimpleMovingAverage(
        inputs=[campaign_contributions.count], 
        window_length= 252/2
    )
    
    screen_zeros = (factor != 0.0)

    # Create pipeline
    pipe = Pipeline(
        columns={
            'factor': factor,
        },
        screen=factor.notnull() & screen_zeros & universe
    )
    return pipe


def before_trading_start(context, data):
    """
    Optional core function called automatically before the open of each market day.

    Parameters
    ----------
    context : AlgorithmContext
        See description above.
    data : BarData
        An object that provides methods to get price and volume data, check
        whether a security exists, and check the last time a security traded.
    """

    # TODO: Un-comment this line to store the output of your pipeline in the context variable:
    context.pipeline_data = algo.pipeline_output('my_data_template')

    # This dataframe will contain all of our risk loadings
    context.risk_loadings = algo.pipeline_output('risk_factors')
    

def rebalance(context, data):
    """
    A function scheduled to run once every Monday at 10AM ET in order to
    rebalance the longs and shorts lists.

    Parameters
    ----------
    context : AlgorithmContext
        See description above.
    data : BarData
        See description above.
    """
    # Retrieve pipeline output.
    pipeline_data = context.pipeline_data

    # Retrieve risk model factor data.
    risk_loadings = context.risk_loadings

    # Here we define our objective for the Optimize API. In this template 
    # we use the MaximizeAlpha objective.
    objective = opt.MaximizeAlpha(pipeline_data.factor)

    # Define the list of constraints.
    constraints = []
    
    # Constrain our maximum gross leverage.
    constraints.append(opt.MaxGrossExposure(MAX_GROSS_LEVERAGE))

    # Require our algorithm to remain dollar neutral.
    constraints.append(opt.DollarNeutral())

    # Add the RiskModelExposure constraint to make use of the
    # default risk model constraints.
    neutralize_risk_factors = opt.experimental.RiskModelExposure(
        risk_model_loadings=risk_loadings,
        version=0
    )
    constraints.append(neutralize_risk_factors)

    # With this constraint we enforce that no position can make up
    # greater than MAX_SHORT_POSITION_SIZE on the short side and
    # no greater than MAX_LONG_POSITION_SIZE on the long side. This
    # ensures that we do not overly concentrate our portfolio in
    # one security or a small subset of securities.
    constraints.append(
        opt.PositionConcentration.with_equal_bounds(
            min=-MAX_SHORT_POSITION_SIZE,
            max=MAX_LONG_POSITION_SIZE
        ))

    # Put together all the pieces we defined above by passing
    # them into the algo.order_optimal_portfolio function. This handles
    # all of our ordering logic, assigning appropriate weights
    # to the securities in our universe to maximize our alpha with
    # respect to the given constraints.
    algo.order_optimal_portfolio(
        objective=objective,
        constraints=constraints
    )
There was a runtime error.