Quantopian Tutorial: Lesson 3 - Basic Fundamentals with Piotroski Score, Growth Stocks, and Uptrending Volatile Small Cap Algorithms

edited

Show Q value

2/16/2016 This tutorial is out-dated, please view the new versions here: https://www.quantopian.com/posts/quantopian-tutorials

Hey all,

This is the third lesson in our Quantopian Tutorial Series. In this one, we covered:

The Basics of Fundamentals:
- before_trading_start, get_fundamentals, and update_universe

Advanced Concepts:
- Universe Size

Three different algorithms:
- Naoki Nagai's Growth Ranking Strategy
- Richard Prokopyshen's Uptrending Volatile Small Cap Strategy
- The Piotroski Score by yours truly

HELPFUL LINKS:

QUANTOPIAN TUTORIAL SERIES:
- Basics of the IDE
- Basics of Fetcher and Security Universe with Jessica Stauth's Portfolio Rebalancing Algorithm

Clone Algorithm

632

Backtest from to with initial capital

Cumulative performance:

Total Returns

Alpha

Beta

Sharpe

Sortino

Max Drawdown

Benchmark Returns

Volatility

Returns	1 Month	3 Month	6 Month	12 Month

Alpha	1 Month	3 Month	6 Month	12 Month

Beta	1 Month	3 Month	6 Month	12 Month

Sharpe	1 Month	3 Month	6 Month	12 Month

Sortino	1 Month	3 Month	6 Month	12 Month

Volatility	1 Month	3 Month	6 Month	12 Month

Max Drawdown	1 Month	3 Month	6 Month	12 Month

# Backtest ID: 54972277f2a41c73eff0294a

"""
    Creating an algorithm based off the Piotroski Score index which is based off of a score (0-9)
    Each of the following marks (-) satisfied means one point. And in the end we'll long the stocks with a score of >= 8

    Profitability 
    - Positive ROA
    - Positive Operating Cash Flow
    - Higher ROA in current year versus last year
    - Cash flow from operations > ROA of current year

    Leverage
    - Current ratio of long term debt < last year's ratio of long term debt
    - Current year's current_ratio > last year's current_ratio
    - No new shares issued this year

    Operating Efficiency
    - Higher gross margin compared to previous year
    - Higher asset turnover ratio compared to previous year


    This algorithm demonstrates how to grasp historical fundamental data by storing it in a Pandas Panel similar to how the Quantopian 'data' is currently structured
"""

import pytz
from datetime import datetime, timedelta

import pandas as pd
    
"""
    Initialize and Handle Data
"""
    
def initialize(context):
    
    #: context.days holds the number of days that we've had this algorithm
    context.days = 0
    
    #: context.fundamental_dict holds the date:dictionary reference that we need
    context.fundamental_dict = {}
    
    #: context.fundamental_data holds the pandas Panel that's derived from the fundamental_dict
    context.fundamental_data = None

def before_trading_start(context): 
    """
        Called before the start of each trading day (handle_data) and updates our universe with the securities and values found from fetch_fundamentals
    """
    
    #: Reference fundamentals in a shorter variable
    f = fundamentals
    
    #: Query for the data that we need from fundamentals
    fundamental_df = get_fundamentals(query(
                                    f.valuation.market_cap,
                                    f.operation_ratios.roa,
                                    f.cash_flow_statement.operating_cash_flow,
                                    f.cash_flow_statement.cash_flow_from_continuing_operating_activities,
                                    f.operation_ratios.long_term_debt_equity_ratio,
                                    f.operation_ratios.current_ratio,
                                    f.valuation.shares_outstanding,
                                    f.operation_ratios.gross_margin,
                                    f.operation_ratios.assets_turnover,
                                    )
                             .filter(fundamentals.valuation.market_cap != None)
                             .filter(fundamentals.operation_ratios.roa != None)
                             .filter(fundamentals.cash_flow_statement.operating_cash_flow != None)
                             .filter(fundamentals.cash_flow_statement.cash_flow_from_continuing_operating_activities != None)
                             .filter(fundamentals.operation_ratios.long_term_debt_equity_ratio != None)
                             .filter(fundamentals.valuation.shares_outstanding != None)
                             .filter(fundamentals.operation_ratios.current_ratio != None)
                             .filter(fundamentals.valuation.shares_outstanding != None)
                             .order_by(fundamentals.valuation.market_cap.desc())
                             .limit(40)
                             )
    
    #: Set our fundamentals into a context variable
    context.fundamental_df = fundamental_df
    
    #: Update our universe with the values
    update_universe(fundamental_df.columns.values)  
    
def handle_data(context, data):
    
    #: Only run every 25 trading days
    if context.days % 25 == 0:
        
        #: Insert a new dataframe into our dictionary 
        context.fundamental_dict[get_datetime()] = context.fundamental_df
        
        #: If it's greater than the first trading day
        if context.days > 0:
            context.fundamental_data = pd.Panel(context.fundamental_dict)
            scores = get_piotroski_scores(context.fundamental_data, get_datetime())
            
            #: Only rebalance when we have enough data
            if scores != None:
                rebalance(context, data, scores)
    
    #: Log our current positions
    if (context.days - 1) % 25 == 0:
        
        #: Portfolio position string
        portfolio_string = "Current positions: "
        
        #: Don't log if we have no positions
        if len(context.portfolio.positions) != 0:
            
            #: Add our current positions to a string
            for pos in context.portfolio.positions:
                portfolio_string += "Symbol: %s and Amount: %s, " % (pos.symbol, context.portfolio.positions[pos].amount)
        
            #: Log all our portfolios
            log.info(portfolio_string)
    
    
    context.days += 1
            
"""
    Defining our rebalance method
"""

def rebalance(context, data, scores):
    """
        This method takes in the scores found by get_piotroski_scores and orders our portfolio accordingly
    """
    
    #: Find which stocks we need to long and which ones we need to short
    num_long = [stock for stock in scores if scores[stock] >= 8]
    num_short = [stock for stock in scores if scores[stock] <= 2]
    
    #: Stocks to long
    for stock in num_long:
        if stock in data:
            log.info("Going long on stock %s with score %s" % (stock.symbol, scores[stock]))
            order_target_percent(stock, 1.0/len(num_long))
    
    #: Stocks to short
    for stock in num_short:
        if stock in data:
            log.info("Going short on stock %s with score %s" % (stock.symbol, scores[stock]))
            order_target_percent(stock, -1.0/len(num_short))
    
    #: Exit any positions we might have
    for stock in context.portfolio.positions:
        if stock in data and stock not in num_long and stock not in num_short:
            log.info("Exiting our positions on %s" % (stock.symbol))
            order_target_percent(stock, 0)
    
    record(number_long=len(num_long))
    record(number_short=len(num_short))
    
"""
    Defining our methods for the piotroski score
"""

def get_piotroski_scores(fundamental_data, current_date):
    """
        This method finds the dataframe that contains the data for the time period we want
        and finds the total Piotroski score for those dates
    """
    all_scores = {}
    all_dates = fundamental_data.items
    
    utc = pytz.UTC
    last_year = utc.localize(datetime(year=current_date.year - 1, month = current_date.month, day = current_date.day))
    
    #: If one year hasn't passed just return None
    if last_year < min(all_dates):
        return None
    
    #: Figure out which date to use
    for i, date in enumerate(all_dates):
        if i == len(all_dates) - 1:
            continue
        if last_year > date and last_year < all_dates[i + 1]:
            break
        elif last_year == date:
            break
        
    #: This is pretty robust so just set last_year to whatever date currently is when you broke
    #: or ended the for loop
    last_year = date
    old_data = fundamental_data[last_year]
    current_data = fundamental_data[current_date]
    
    #: Find the score for each security
    for stock in current_data:
        profit = profit_logic(current_data, old_data, stock)
        leverage = leverage_logic(current_data, old_data, stock)
        operating = operating_logic(current_data, old_data, stock)
        total_score = profit + leverage + operating
        all_scores[stock] = total_score
        
    return all_scores

def profit_logic(current_data, old_data, sid):
    """
        Define our profitability logic here
    """
    
    #: Positive ROA
    positive_roa = current_data[sid]['roa'] > 0
    #: Positive Operating Cash Flow
    positive_ocf = current_data[sid]['operating_cash_flow'] > 0
    #: Current ROA > Last Year ROA
    current_last_roa = current_data[sid]['roa'] > old_data[sid]['roa']
    #: Cash flow from operations > ROA
    cash_flow_roa = current_data[sid]['cash_flow_from_continuing_operating_activities'] > current_data[sid]['roa']
    
    return int(positive_roa) + int(positive_ocf) + int(current_last_roa)+ int(cash_flow_roa)
    
def leverage_logic(current_data, old_data, sid):
    """
        Define our leverage logic here 
    """
    
    #: Current ratio of long-term debt < last year's ratio of long-term debt
    long_term_debt = current_data[sid]['long_term_debt_equity_ratio'] > old_data[sid]['long_term_debt_equity_ratio']
    #: Current year's current_ratio > last year's current_ratio
    current_ratio = current_data[sid]['current_ratio'] > old_data[sid]['current_ratio']
    #: No new shares
    new_shares = current_data[sid]['shares_outstanding'] <= old_data[sid]['shares_outstanding']
    
    return int(long_term_debt) + int(current_ratio) + int(new_shares)
    
def operating_logic(current_data, old_data, sid):
    """
        Define our operating efficiency logic here
    """
    
    #: Higher gross margin compared to previous year
    gross_margin = current_data[sid]['gross_margin'] > old_data[sid]['gross_margin']
    #: Higher asset turnover ratio compared to previous year
    asset_turnover = current_data[sid]['assets_turnover'] > old_data[sid]['assets_turnover']
    
    return int(gross_margin) + int(asset_turnover)

"""
    Creating an algorithm based off the Piotroski Score index which is based off of a score (0-9)
    Each of the following marks (-) satisfied means one point. And in the end we'll long the stocks with a score of >= 8

    Profitability 
    - Positive ROA
    - Positive Operating Cash Flow
    - Higher ROA in current year versus last year
    - Cash flow from operations > ROA of current year

    Leverage
    - Current ratio of long term debt < last year's ratio of long term debt
    - Current year's current_ratio > last year's current_ratio
    - No new shares issued this year

    Operating Efficiency
    - Higher gross margin compared to previous year
    - Higher asset turnover ratio compared to previous year


    This algorithm demonstrates how to grasp historical fundamental data by storing it in a Pandas Panel similar to how the Quantopian 'data' is currently structured
"""

import pytz
from datetime import datetime, timedelta

import pandas as pd
    
"""
    Initialize and Handle Data
"""
    
def initialize(context):
    
    #: context.days holds the number of days that we've had this algorithm
    context.days = 0
    
    #: context.fundamental_dict holds the date:dictionary reference that we need
    context.fundamental_dict = {}
    
    #: context.fundamental_data holds the pandas Panel that's derived from the fundamental_dict
    context.fundamental_data = None
    
    schedule_function(rebalance, date_rule=date_rules.every_day())
    schedule_function(record_stats, date_rule=date_rules.every_day())

def before_trading_start(context, data):
    """
        Called before the start of each trading day (handle_data) and updates our universe with the securities and values found from fetch_fundamentals
    """
    context.days += 1
    
    if context.days % 30 != 0:
        return
    
    #: Reference fundamentals in a shorter variable
    f = fundamentals
    
    #: Query for the data that we need from fundamentals
    fundamental_df = get_fundamentals(query(
                                    f.valuation.market_cap,
                                    f.operation_ratios.roa,
                                    f.cash_flow_statement.operating_cash_flow,
                                    f.cash_flow_statement.cash_flow_from_continuing_operating_activities,
                                    f.operation_ratios.long_term_debt_equity_ratio,
                                    f.operation_ratios.current_ratio,
                                    f.valuation.shares_outstanding,
                                    f.operation_ratios.gross_margin,
                                    f.operation_ratios.assets_turnover,
                            )
                             .order_by(fundamentals.valuation.market_cap.desc())
                             .limit(40)
                             )
    
    #: Set our fundamentals into a context variable
    context.fundamental_df = fundamental_df
    
    #: Update our universe with the values
    context.securities_list = fundamental_df.columns.values

def rebalance(context, data):
    if context.days % 30 == 0:
        #: Insert a new dataframe into our dictionary 
        context.fundamental_dict[get_datetime()] = context.fundamental_df

        #: If it's greater than the first trading day
        if context.days > 0:
            context.fundamental_data = pd.Panel(context.fundamental_dict)
            scores = get_piotroski_scores(context.fundamental_data, get_datetime())

            #: Only rebalance when we have enough data
            if scores != None:
                rebalance_internal(context, data, scores)

def record_stats(context, data):
    if (context.days - 1) % 30 == 0:
        #: Log our current positions
        #: Portfolio position string
        portfolio_string = "Current positions: "

        #: Don't log if we have no positions
        if len(context.portfolio.positions) != 0:

            #: Add our current positions to a string
            for pos in context.portfolio.positions:
                portfolio_string += "Symbol: %s and Amount: %s, " % (pos.symbol, context.portfolio.positions[pos].amount)

            #: Log all our portfolios
            log.info(portfolio_string)

            
"""
    Defining our rebalance method
"""

def rebalance_internal(context, data, scores):
    """
        This method takes in the scores found by get_piotroski_scores and orders our portfolio accordingly
    """
    
    #: Find which stocks we need to long and which ones we need to short
    num_long = [stock for stock in scores if scores[stock] >= 7]
    num_short = [stock for stock in scores if scores[stock] <= 2]
    
    #: Stocks to long
    for stock in num_long:
        if stock in context.securities_list:
            if data.can_trade(stock):
                log.info("Going long on stock %s with score %s" % (stock.symbol, scores[stock]))
                order_target_percent(stock, 1.0/len(num_long))

    #: Exit any positions we might have
    for stock in context.portfolio.positions:
        if stock not in num_long and stock not in num_short and data.can_trade(stock):
            log.info("Exiting our positions on %s" % (stock.symbol))
            order_target_percent(stock, 0)
    
    record(number_long=len(num_long))
    
"""
    Defining our methods for the piotroski score
"""

def get_piotroski_scores(fundamental_data, current_date):
    """
        This method finds the dataframe that contains the data for the time period we want
        and finds the total Piotroski score for those dates
    """
    all_scores = {}
    all_dates = fundamental_data.items
    
    utc = pytz.UTC
    last_year = utc.localize(datetime(year=current_date.year - 1, month = current_date.month, day = current_date.day))
    
    #: If one year hasn't passed just return None
    if last_year < min(all_dates):
        return None
    
    #: Figure out which date to use
    for i, date in enumerate(all_dates):
        if i == len(all_dates) - 1:
            continue
        if last_year > date and last_year < all_dates[i + 1]:
            break
        elif last_year == date:
            break
        
    #: This is pretty robust so just set last_year to whatever date currently is when you broke
    #: or ended the for loop
    last_year = date
    old_data = fundamental_data[last_year]
    current_data = fundamental_data[current_date]
    
    #: Find the score for each security
    for stock in current_data:
        profit = profit_logic(current_data, old_data, stock)
        leverage = leverage_logic(current_data, old_data, stock)
        operating = operating_logic(current_data, old_data, stock)
        total_score = profit + leverage + operating
        all_scores[stock] = total_score
        
    return all_scores

def profit_logic(current_data, old_data, sid):
    """
        Define our profitability logic here
    """
    
    #: Positive ROA
    positive_roa = current_data[sid]['roa'] > 0
    #: Positive Operating Cash Flow
    positive_ocf = current_data[sid]['operating_cash_flow'] > 0
    #: Current ROA > Last Year ROA
    current_last_roa = current_data[sid]['roa'] > old_data[sid]['roa']
    #: Cash flow from operations > ROA
    cash_flow_roa = current_data[sid]['cash_flow_from_continuing_operating_activities'] > current_data[sid]['roa']
    
    return int(positive_roa) + int(positive_ocf) + int(current_last_roa)+ int(cash_flow_roa)
    
def leverage_logic(current_data, old_data, sid):
    """
        Define our leverage logic here 
    """
    
    #: Current ratio of long-term debt < last year's ratio of long-term debt
    long_term_debt = current_data[sid]['long_term_debt_equity_ratio'] > old_data[sid]['long_term_debt_equity_ratio']
    #: Current year's current_ratio > last year's current_ratio
    current_ratio = current_data[sid]['current_ratio'] > old_data[sid]['current_ratio']
    #: No new shares
    new_shares = current_data[sid]['shares_outstanding'] <= old_data[sid]['shares_outstanding']
    
    return int(long_term_debt) + int(current_ratio) + int(new_shares)
    
def operating_logic(current_data, old_data, sid):
    """
        Define our operating efficiency logic here
    """
    
    #: Higher gross margin compared to previous year
    gross_margin = current_data[sid]['gross_margin'] > old_data[sid]['gross_margin']
    #: Higher asset turnover ratio compared to previous year
    asset_turnover = current_data[sid]['assets_turnover'] > old_data[sid]['assets_turnover']
    
    return int(gross_margin) + int(asset_turnover)

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

39 responses

Wrong backtest uploaded before!

This is the right one!

Clone Algorithm

632

Backtest from to with initial capital

Cumulative performance:

Total Returns

Alpha

Beta

Sharpe

Sortino

Max Drawdown

Benchmark Returns

Volatility

Returns	1 Month	3 Month	6 Month	12 Month

Alpha	1 Month	3 Month	6 Month	12 Month

Beta	1 Month	3 Month	6 Month	12 Month

Sharpe	1 Month	3 Month	6 Month	12 Month

Sortino	1 Month	3 Month	6 Month	12 Month

Volatility	1 Month	3 Month	6 Month	12 Month

Max Drawdown	1 Month	3 Month	6 Month	12 Month

# Backtest ID: 54cfb611a8fdc04085a67874

"""
    Creating an algorithm based off the Piotroski Score index which is based off of a score (0-9)
    Each of the following marks (-) satisfied means one point. And in the end we'll long the stocks with a score of >= 8

    Profitability 
    - Positive ROA
    - Positive Operating Cash Flow
    - Higher ROA in current year versus last year
    - Cash flow from operations > ROA of current year

    Leverage
    - Current ratio of long term debt < last year's ratio of long term debt
    - Current year's current_ratio > last year's current_ratio
    - No new shares issued this year

    Operating Efficiency
    - Higher gross margin compared to previous year
    - Higher asset turnover ratio compared to previous year


    This algorithm demonstrates how to grasp historical fundamental data by storing it in a Pandas Panel similar to how the Quantopian 'data' is currently structured
"""

import pytz
from datetime import datetime, timedelta

import pandas as pd
    
"""
    Initialize and Handle Data
"""
    
def initialize(context):
    
    #: context.days holds the number of days that we've had this algorithm
    context.days = 0
    
    #: context.fundamental_dict holds the date:dictionary reference that we need
    context.fundamental_dict = {}
    
    #: context.fundamental_data holds the pandas Panel that's derived from the fundamental_dict
    context.fundamental_data = None

def before_trading_start(context): 
    """
        Called before the start of each trading day (handle_data) and updates our universe with the securities and values found from fetch_fundamentals
    """
    
    #: Reference fundamentals in a shorter variable
    f = fundamentals
    
    #: Query for the data that we need from fundamentals
    fundamental_df = get_fundamentals(query(
                                    f.valuation.market_cap,
                                    f.operation_ratios.roa,
                                    f.cash_flow_statement.operating_cash_flow,
                                    f.cash_flow_statement.cash_flow_from_continuing_operating_activities,
                                    f.operation_ratios.long_term_debt_equity_ratio,
                                    f.operation_ratios.current_ratio,
                                    f.valuation.shares_outstanding,
                                    f.operation_ratios.gross_margin,
                                    f.operation_ratios.assets_turnover,
                            )
                             .order_by(fundamentals.valuation.market_cap.desc())
                             .limit(40)
                             )
    
    #: Set our fundamentals into a context variable
    context.fundamental_df = fundamental_df
    
    #: Update our universe with the values
    update_universe(fundamental_df.columns.values)  
    
def handle_data(context, data):
    
    #: Only run every 25 trading days
    if context.days % 25 == 0:
        
        #: Insert a new dataframe into our dictionary 
        context.fundamental_dict[get_datetime()] = context.fundamental_df
        
        #: If it's greater than the first trading day
        if context.days > 0:
            context.fundamental_data = pd.Panel(context.fundamental_dict)
            scores = get_piotroski_scores(context.fundamental_data, get_datetime())
            
            #: Only rebalance when we have enough data
            if scores != None:
                rebalance(context, data, scores)
    
    #: Log our current positions
    if (context.days - 1) % 25 == 0:
        
        #: Portfolio position string
        portfolio_string = "Current positions: "
        
        #: Don't log if we have no positions
        if len(context.portfolio.positions) != 0:
            
            #: Add our current positions to a string
            for pos in context.portfolio.positions:
                portfolio_string += "Symbol: %s and Amount: %s, " % (pos.symbol, context.portfolio.positions[pos].amount)
        
            #: Log all our portfolios
            log.info(portfolio_string)
    
    
    context.days += 1
            
"""
    Defining our rebalance method
"""

def rebalance(context, data, scores):
    """
        This method takes in the scores found by get_piotroski_scores and orders our portfolio accordingly
    """
    
    #: Find which stocks we need to long and which ones we need to short
    num_long = [stock for stock in scores if scores[stock] >= 7]
    num_short = [stock for stock in scores if scores[stock] <= 2]
    
    #: Stocks to long
    for stock in num_long:
        if stock in data:
            log.info("Going long on stock %s with score %s" % (stock.symbol, scores[stock]))
            order_target_percent(stock, 1.0/len(num_long))
    
    # #: Stocks to short
    # for stock in num_short:
    #     if stock in data:
    #         log.info("Going short on stock %s with score %s" % (stock.symbol, scores[stock]))
    #         order_target_percent(stock, -1.0/len(num_short))
    
    #: Exit any positions we might have
    for stock in context.portfolio.positions:
        if stock in data and stock not in num_long and stock not in num_short:
            log.info("Exiting our positions on %s" % (stock.symbol))
            order_target_percent(stock, 0)
    
    record(number_long=len(num_long))
    # record(number_short=len(num_short))
    
"""
    Defining our methods for the piotroski score
"""

def get_piotroski_scores(fundamental_data, current_date):
    """
        This method finds the dataframe that contains the data for the time period we want
        and finds the total Piotroski score for those dates
    """
    all_scores = {}
    all_dates = fundamental_data.items
    
    utc = pytz.UTC
    last_year = utc.localize(datetime(year=current_date.year - 1, month = current_date.month, day = current_date.day))
    
    #: If one year hasn't passed just return None
    if last_year < min(all_dates):
        return None
    
    #: Figure out which date to use
    for i, date in enumerate(all_dates):
        if i == len(all_dates) - 1:
            continue
        if last_year > date and last_year < all_dates[i + 1]:
            break
        elif last_year == date:
            break
        
    #: This is pretty robust so just set last_year to whatever date currently is when you broke
    #: or ended the for loop
    last_year = date
    old_data = fundamental_data[last_year]
    current_data = fundamental_data[current_date]
    
    #: Find the score for each security
    for stock in current_data:
        profit = profit_logic(current_data, old_data, stock)
        leverage = leverage_logic(current_data, old_data, stock)
        operating = operating_logic(current_data, old_data, stock)
        total_score = profit + leverage + operating
        all_scores[stock] = total_score
        
    return all_scores

def profit_logic(current_data, old_data, sid):
    """
        Define our profitability logic here
    """
    
    #: Positive ROA
    positive_roa = current_data[sid]['roa'] > 0
    #: Positive Operating Cash Flow
    positive_ocf = current_data[sid]['operating_cash_flow'] > 0
    #: Current ROA > Last Year ROA
    current_last_roa = current_data[sid]['roa'] > old_data[sid]['roa']
    #: Cash flow from operations > ROA
    cash_flow_roa = current_data[sid]['cash_flow_from_continuing_operating_activities'] > current_data[sid]['roa']
    
    return int(positive_roa) + int(positive_ocf) + int(current_last_roa)+ int(cash_flow_roa)
    
def leverage_logic(current_data, old_data, sid):
    """
        Define our leverage logic here 
    """
    
    #: Current ratio of long-term debt < last year's ratio of long-term debt
    long_term_debt = current_data[sid]['long_term_debt_equity_ratio'] > old_data[sid]['long_term_debt_equity_ratio']
    #: Current year's current_ratio > last year's current_ratio
    current_ratio = current_data[sid]['current_ratio'] > old_data[sid]['current_ratio']
    #: No new shares
    new_shares = current_data[sid]['shares_outstanding'] <= old_data[sid]['shares_outstanding']
    
    return int(long_term_debt) + int(current_ratio) + int(new_shares)
    
def operating_logic(current_data, old_data, sid):
    """
        Define our operating efficiency logic here
    """
    
    #: Higher gross margin compared to previous year
    gross_margin = current_data[sid]['gross_margin'] > old_data[sid]['gross_margin']
    #: Higher asset turnover ratio compared to previous year
    asset_turnover = current_data[sid]['assets_turnover'] > old_data[sid]['assets_turnover']
    
    return int(gross_margin) + int(asset_turnover)

"""
    Creating an algorithm based off the Piotroski Score index which is based off of a score (0-9)
    Each of the following marks (-) satisfied means one point. And in the end we'll long the stocks with a score of >= 8

    Profitability 
    - Positive ROA
    - Positive Operating Cash Flow
    - Higher ROA in current year versus last year
    - Cash flow from operations > ROA of current year

    Leverage
    - Current ratio of long term debt < last year's ratio of long term debt
    - Current year's current_ratio > last year's current_ratio
    - No new shares issued this year

    Operating Efficiency
    - Higher gross margin compared to previous year
    - Higher asset turnover ratio compared to previous year


    This algorithm demonstrates how to grasp historical fundamental data by storing it in a Pandas Panel similar to how the Quantopian 'data' is currently structured
"""

import pytz
from datetime import datetime, timedelta

import pandas as pd
    
"""
    Initialize and Handle Data
"""
    
def initialize(context):
    
    #: context.days holds the number of days that we've had this algorithm
    context.days = 0
    
    #: context.fundamental_dict holds the date:dictionary reference that we need
    context.fundamental_dict = {}
    
    #: context.fundamental_data holds the pandas Panel that's derived from the fundamental_dict
    context.fundamental_data = None
    
    schedule_function(rebalance, date_rule=date_rules.every_day())
    schedule_function(record_stats, date_rule=date_rules.every_day())

def before_trading_start(context, data):
    """
        Called before the start of each trading day (handle_data) and updates our universe with the securities and values found from fetch_fundamentals
    """
    context.days += 1
    
    if context.days % 30 != 0:
        return
    
    #: Reference fundamentals in a shorter variable
    f = fundamentals
    
    #: Query for the data that we need from fundamentals
    fundamental_df = get_fundamentals(query(
                                    f.valuation.market_cap,
                                    f.operation_ratios.roa,
                                    f.cash_flow_statement.operating_cash_flow,
                                    f.cash_flow_statement.cash_flow_from_continuing_operating_activities,
                                    f.operation_ratios.long_term_debt_equity_ratio,
                                    f.operation_ratios.current_ratio,
                                    f.valuation.shares_outstanding,
                                    f.operation_ratios.gross_margin,
                                    f.operation_ratios.assets_turnover,
                            )
                             .order_by(fundamentals.valuation.market_cap.desc())
                             .limit(40)
                             )
    
    #: Set our fundamentals into a context variable
    context.fundamental_df = fundamental_df
    
    #: Update our universe with the values
    context.securities_list = fundamental_df.columns.values

def rebalance(context, data):
    if context.days % 30 == 0:
        #: Insert a new dataframe into our dictionary 
        context.fundamental_dict[get_datetime()] = context.fundamental_df

        #: If it's greater than the first trading day
        if context.days > 0:
            context.fundamental_data = pd.Panel(context.fundamental_dict)
            scores = get_piotroski_scores(context.fundamental_data, get_datetime())

            #: Only rebalance when we have enough data
            if scores != None:
                rebalance_internal(context, data, scores)

def record_stats(context, data):
    if (context.days - 1) % 30 == 0:
        #: Log our current positions
        #: Portfolio position string
        portfolio_string = "Current positions: "

        #: Don't log if we have no positions
        if len(context.portfolio.positions) != 0:

            #: Add our current positions to a string
            for pos in context.portfolio.positions:
                portfolio_string += "Symbol: %s and Amount: %s, " % (pos.symbol, context.portfolio.positions[pos].amount)

            #: Log all our portfolios
            log.info(portfolio_string)

            
"""
    Defining our rebalance method
"""

def rebalance_internal(context, data, scores):
    """
        This method takes in the scores found by get_piotroski_scores and orders our portfolio accordingly
    """
    
    #: Find which stocks we need to long and which ones we need to short
    num_long = [stock for stock in scores if scores[stock] >= 7]
    num_short = [stock for stock in scores if scores[stock] <= 2]
    
    #: Stocks to long
    for stock in num_long:
        if stock in context.securities_list:
            if data.can_trade(stock):
                log.info("Going long on stock %s with score %s" % (stock.symbol, scores[stock]))
                order_target_percent(stock, 1.0/len(num_long))

    #: Exit any positions we might have
    for stock in context.portfolio.positions:
        if stock not in num_long and stock not in num_short and data.can_trade(stock):
            log.info("Exiting our positions on %s" % (stock.symbol))
            order_target_percent(stock, 0)
    
    record(number_long=len(num_long))
    
"""
    Defining our methods for the piotroski score
"""

def get_piotroski_scores(fundamental_data, current_date):
    """
        This method finds the dataframe that contains the data for the time period we want
        and finds the total Piotroski score for those dates
    """
    all_scores = {}
    all_dates = fundamental_data.items
    
    utc = pytz.UTC
    last_year = utc.localize(datetime(year=current_date.year - 1, month = current_date.month, day = current_date.day))
    
    #: If one year hasn't passed just return None
    if last_year < min(all_dates):
        return None
    
    #: Figure out which date to use
    for i, date in enumerate(all_dates):
        if i == len(all_dates) - 1:
            continue
        if last_year > date and last_year < all_dates[i + 1]:
            break
        elif last_year == date:
            break
        
    #: This is pretty robust so just set last_year to whatever date currently is when you broke
    #: or ended the for loop
    last_year = date
    old_data = fundamental_data[last_year]
    current_data = fundamental_data[current_date]
    
    #: Find the score for each security
    for stock in current_data:
        profit = profit_logic(current_data, old_data, stock)
        leverage = leverage_logic(current_data, old_data, stock)
        operating = operating_logic(current_data, old_data, stock)
        total_score = profit + leverage + operating
        all_scores[stock] = total_score
        
    return all_scores

def profit_logic(current_data, old_data, sid):
    """
        Define our profitability logic here
    """
    
    #: Positive ROA
    positive_roa = current_data[sid]['roa'] > 0
    #: Positive Operating Cash Flow
    positive_ocf = current_data[sid]['operating_cash_flow'] > 0
    #: Current ROA > Last Year ROA
    current_last_roa = current_data[sid]['roa'] > old_data[sid]['roa']
    #: Cash flow from operations > ROA
    cash_flow_roa = current_data[sid]['cash_flow_from_continuing_operating_activities'] > current_data[sid]['roa']
    
    return int(positive_roa) + int(positive_ocf) + int(current_last_roa)+ int(cash_flow_roa)
    
def leverage_logic(current_data, old_data, sid):
    """
        Define our leverage logic here 
    """
    
    #: Current ratio of long-term debt < last year's ratio of long-term debt
    long_term_debt = current_data[sid]['long_term_debt_equity_ratio'] > old_data[sid]['long_term_debt_equity_ratio']
    #: Current year's current_ratio > last year's current_ratio
    current_ratio = current_data[sid]['current_ratio'] > old_data[sid]['current_ratio']
    #: No new shares
    new_shares = current_data[sid]['shares_outstanding'] <= old_data[sid]['shares_outstanding']
    
    return int(long_term_debt) + int(current_ratio) + int(new_shares)
    
def operating_logic(current_data, old_data, sid):
    """
        Define our operating efficiency logic here
    """
    
    #: Higher gross margin compared to previous year
    gross_margin = current_data[sid]['gross_margin'] > old_data[sid]['gross_margin']
    #: Higher asset turnover ratio compared to previous year
    asset_turnover = current_data[sid]['assets_turnover'] > old_data[sid]['assets_turnover']
    
    return int(gross_margin) + int(asset_turnover)

Disclaimer

Video uploaded here: http://bit.ly/1CyoR38

Enjoy!

Disclaimer

Easan Katir

Hi Seong, Thanks for your help and for the super great webinar. I notice at the bottom of your IDE there are buttons to run and step through the program, with values displayed in the memory registers ! Cool ! I was hoping for these, like a regular offline IDE.

Only, when I access my account online, those buttons aren't there. Can you turn them on ?

Hi Easan,

What you're seeing is the builtin-debugger. You can access it by clicking on the line numbers to the left of your IDE and the debugging popup will show once it hits that line in the code. You can find a more in-depth guide by looking at this post (https://www.quantopian.com/posts/new-feature-debugging-in-the-ide)

Disclaimer

Vignesh Ramesh

Hi Seong,

Just wondering if the tutorial 4 is coming up anytime soon?

Thanks, found your tutorials very useful!

Tag for later.

Easan Katir

Great tutorial, Seong. thank you.

I scrolled through the 670 choices in the morningstar data base, and didn't find anything relating to insider transactions. I wonder if insider transactions is available for stocks.

Thanks Seong - do you have a version where the short side works? When I enable it, it seems to uncover ever-increasing numbers of shorts, which doesn't make any sense, and though I am looking into it, it would save some time if you already solved it! Thanks.

Josh Payne

Hi Easan,

Insider transactions would be a separate data set and not part of the data we provide from Morningstar.

I'll add it to the list of data sources the community is requesting.

Thanks
Josh

Disclaimer

Hi Vignesh,

Tutorial 4 isn't yet up. We're looking into what topics might best benefit the community. If you have suggestions, please feel free to reply here.

Simon,

Off the top of my head, I don't see a reason why the shorts wouldn't work as-is. Will need to do some further research.

Thanks all and glad you appreciated the tutorial.

Seong

Disclaimer

When I tried it, it was finding ever increasing numbers of shorts, probably due to the missing data resulting in scores of 0, which it interpreted as short candidates. I didn't look too deeply.

Ujae Kang

Hi,
Are these fundamental data annual? Is there a way I can get quarterly data?
Thanks

Josh Payne

Most of the fundamental metrics are in fact quarterly, derived from quarterly filings. We don't currently incorporate annual metrics.

There are also some metrics that are updated daily. Market cap, PE ratio. Generally, you can find those in the operation_metrics namespace.

We have daily updates for those "daily" metrics since May, 2014 (when we began taking deliver of data from Morningstar). Prior to that, those daily metrics are updated monthly (i.e. our data from 2002 - May 2014).

Thanks
Josh

Disclaimer

Seeong/Josh, how would we ever live trade something like this without the ability to do historical as-of fundamental queries? Wouldn't this basically need a year to warm up?

Unless I missed that feature?

Josh Payne

You haven't missed it. It's not there yet. We're actively working on that feature now. Look for it in research first and then it will migrate back to the backtester.

Disclaimer

Matthew Liu

Hey Josh/Seong,

Thanks for putting together the comprehensive tutorials and software. I'm new to Quantopian, but have been super impressed with what you guys have here.

A couple questions around this Piotroski algorithm:

As other have mentioned, any update on when historicals can be queried without first tracking data for a year?
To hack above if this feature isn't working yet (thinking out loud here), is a reasonable approach to run the algo in the backtester, grab the last year's worth of data in the data panel, then append that to the algo before turning it live? This way the algo would have seed data to start making comparisons from day 1? Is there a way to easily export a data panel from the IDE?
I'm curious why you chose 25 trading days to approximate rebalancing every month. Wouldn't it be easier to just use schedule function and do month start with an optional offset?
Any updates on the errors that others were seeing with the shorts?

Thanks. Very much appreciated.

Great questions.

We're still working on getting historical data queried without first tracking data for a year - once that's solved you won't have to worry about your second question. As for an immediate solution, the easiest solution is simply accumulate a data panel day-over-day until it has enough data for you to do your calculations on.

The algorithm was created before schedule_function was released but you could just as easily replace it with schedule_function with an optional offset.

As for the shorts, it wasn't an error as much as that there were A LOT of stocks with scores less than 2. Go figure. The best way to offset this is simply limit the number of shorts that you find, like I do in this code:


def rebalance(context, data, scores):  
    """  
        This method takes in the scores found by get_piotroski_scores and orders our portfolio accordingly  
    """  
    #: Find which stocks we need to long and which ones we need to short  
    num_long = [stock for stock in scores if scores[stock] >= 7]  
    num_short = [stock for stock in scores if scores[stock] <= 2][:len(num_long)]  
    #: Stocks to long  
    for stock in num_long:  
        if stock in data:  
            log.info("Going long on stock %s with score %s" % (stock.symbol, scores[stock]))  
            order_target_percent(stock, 1.0/len(num_long))  
    #: Stocks to short  
    for stock in num_short:  
        if stock in data:  
            log.info("Going short on stock %s with score %s" % (stock.symbol, scores[stock]))  
            order_target_percent(stock, -1.0/len(num_short))  
    #: Exit any positions we might have  
    for stock in context.portfolio.positions:  
        if stock in data and stock not in num_long and stock not in num_short:  
            log.info("Exiting our positions on %s" % (stock.symbol))  
            order_target_percent(stock, 0)  
    record(number_long=len(num_long))  
    record(number_short=len(num_short))

Disclaimer

Matthew Liu

Hi Seong,

Are there any updates to querying against historical data without first tracking it for a year? It's been 2 months so I thought I would check in. Thanks.

Matt

Hi Matthew,

You can now use the Pipeline API to look at historical fundamentals data. There are also some sample algorithm here that uses shares outstanding to calculate market cap.

Disclaimer

Ujae Kang

Jamie,
Are you referring "historical fundamentals data" to mean daily fundamental going back any days back? Or can we now specify which date in the past? I remember having some performance issue pulling 250 some lines of fundamental data in order to look back some financial data a year ago. Never the less I had to pull daily going back some 252 days causing clock out (performance issues). I know another staff from Quantopian mentioned that they were working on it.
Thanks
Ujae

Ujae:
I'm just referring to the fact that you can look back at fundamentals data over a window length in Pipeline, nothing new. Matthew's question was posted before Pipeline was released, so I just wanted that question to have an answer.

Disclaimer

Henry

Is there an updated version of this using the Pipeline to calculate scores based on historical data.

Not yet. I think, we'll never get it. It's almost two years that me and a lot of other people in the forum are asking for this feature.
I pointed the problem again two weeks agon in this post: https://www.quantopian.com/posts/quality-factors-1#586d6617eb6ca119580001a5 but even got an answer.

Costantino,

I completely understand your frustration. We are actively working on speeding up fundamentals and exposing TTM versions of each field. It's a major project, and his been underway for a few months already. The first milestone we are aiming for is to speed up fundamental data retrieval in pipeline. After that, we will be looking at the TTM fields. Again, the project is actively underway, but it involves a complete re-write of how we load and store the fundamental data, so it has taken some time.

Disclaimer

Hi Jamie,

well, this is a very good news! Would it be also possible to get the last n-years for example of Total Revenues or similar?
Is the new timeframe feature going to only meet the Pipeline API or also get_fundamentals()?
I know that get_fundamentals() is quite neglected but I hope it will not be leaved out: simple and fast as it is, has also many advantages!

I don't ask you for a timeline, just keep us informed about the developments and keep up the good work!

When you say the last n years, do you mean the last n years of TTM? So a 10-year query would be 10 data points?

Disclaimer

Trailing 12 months (TTM) is of course the timeframe of the past 12 months.

I mean for example the last 10 data points of annual or quarterly data, like on the morningstar website (look at Statement Type: Annual/Quartely):
http://financials.morningstar.com/income-statement/is.html?t=IBM&region=USA&culture=en_US

For example a 10-datapoints query with timeframe annual would mean the last 10 year of annual reports (10-K) while with timeframe quarterly the last 10 quarters (10-Q)

Yes, perfect, sorry for the poorly asked question. That's what I thought you meant! Right now, getting n-period lookbacks is not defined in the active project, but it's definitely on the list. I don't yet have a sense for how much work it would require, so no timeline, but we're definitely aware that it would open up a new workflow with fundamentals.

Disclaimer

Thanks for the update! It would be great if you could post on forum from time to time the last "state of things" about fundamentals or even better ask for the feedback of the community.
Also get_fundamentals()will be upgraded with timeframes or only the Pipeline API?

You're right, community feedback is very important. We'll keep that in mind as the fundamentals project moves along.

I apologize, I forgot to address your question about get_fundamentals earlier. The long term plan is to phase out get_fundamentals. I'm curious though, what benefits do you consider it to have over pipeline?

Disclaimer

Blue Seahawk

This page is listed 7th under 'Interesting'. If you're here because you're just learning this will give you an advantage (monitoring maximum leverage).

@Jamie: I ported some of my algorithms for get_fundamentals to Pipeline and the first problem was the performance, considerably slower with Pipeline. The second problem was the universe definition: I got different results.
Maybe it was a wrong usage of the Pipeline API.. in the next days I will publish the some algorithm with get_fundamentals and Pipeline to have a benchmark and so we can discuss further.

@Jamie: Another relevant drawback of the Pipeline API ist the high memory usage.
I just converted a {{get_fundamentals}} algorithm of mine to Pipeline and after a longer execution time I got the following exception:

MemoryError  
Algorithm used too much memory. Need to optimize your code for better performance. Learn More

It used just 9 fundamentals indicator

The example would be great. I haven't seen different results and I haven't seen memory issues specifically tied to fundamentals in pipeline. If you'd prefer to send an example in private, feel free to open a support ticket and mention that I told you to send it in.

Disclaimer

Henry

Hi @Jamie
I cloned the Piotroski score pipeline example you posted to the forum thread below. I ran the backtest and got MemoryError Constantino mentioned.
https://www.quantopian.com/posts/piotroski-score-plus-aroon-indicator

Hi Henry,

Thanks for pointing that out. I think this may be different from what Costantino was referring to. In the example I posted in that other thread, the pipeline was making 100-day history requests every day, whereas the version using get_fundamentals was only getting values for a single day (yesterday), and building up a structure containing a longer history. Don't get me wrong, I understand that the memory issue is a limitation - we'll look into a solution to this problem. However, I don't think this is a drawback of the Pipeline API, it's a result of a bigger request being made (get_fundamentals could only query fundamental data from the previous day.

Disclaimer