Back to Community
Quantopian Tutorial: Lesson 3 - Basic Fundamentals with Piotroski Score, Growth Stocks, and Uptrending Volatile Small Cap Algorithms

2/16/2016 This tutorial is out-dated, please view the new versions here: https://www.quantopian.com/posts/quantopian-tutorials

Hey all,

This is the third lesson in our Quantopian Tutorial Series. In this one, we covered:

The Basics of Fundamentals:
- before_trading_start, get_fundamentals, and update_universe

Advanced Concepts:
- Universe Size

Three different algorithms:
- Naoki Nagai's Growth Ranking Strategy
- Richard Prokopyshen's Uptrending Volatile Small Cap Strategy
- The Piotroski Score by yours truly

HELPFUL LINKS:

QUANTOPIAN TUTORIAL SERIES:
- Basics of the IDE
- Basics of Fetcher and Security Universe with Jessica Stauth's Portfolio Rebalancing Algorithm

Clone Algorithm
657
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54972277f2a41c73eff0294a
We have migrated this algorithm to work with a new version of the Quantopian API. The code is different than the original version, but the investment rationale of the algorithm has not changed. We've put everything you need to know here on one page.
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

39 responses

Wrong backtest uploaded before!

This is the right one!

Clone Algorithm
657
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54cfb611a8fdc04085a67874
We have migrated this algorithm to work with a new version of the Quantopian API. The code is different than the original version, but the investment rationale of the algorithm has not changed. We've put everything you need to know here on one page.
There was a runtime error.

Video uploaded here: http://bit.ly/1CyoR38

Enjoy!

Hi Seong, Thanks for your help and for the super great webinar. I notice at the bottom of your IDE there are buttons to run and step through the program, with values displayed in the memory registers ! Cool ! I was hoping for these, like a regular offline IDE.

Only, when I access my account online, those buttons aren't there. Can you turn them on ?

Hi Easan,

What you're seeing is the builtin-debugger. You can access it by clicking on the line numbers to the left of your IDE and the debugging popup will show once it hits that line in the code. You can find a more in-depth guide by looking at this post (https://www.quantopian.com/posts/new-feature-debugging-in-the-ide)

Hi Seong,

Just wondering if the tutorial 4 is coming up anytime soon?

Thanks, found your tutorials very useful!

V

Tag for later.

Great tutorial, Seong. thank you.

I scrolled through the 670 choices in the morningstar data base, and didn't find anything relating to insider transactions. I wonder if insider transactions is available for stocks.

Thanks Seong - do you have a version where the short side works? When I enable it, it seems to uncover ever-increasing numbers of shorts, which doesn't make any sense, and though I am looking into it, it would save some time if you already solved it! Thanks.

Hi Easan,

Insider transactions would be a separate data set and not part of the data we provide from Morningstar.

I'll add it to the list of data sources the community is requesting.

Thanks
Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Vignesh,

Tutorial 4 isn't yet up. We're looking into what topics might best benefit the community. If you have suggestions, please feel free to reply here.

Simon,

Off the top of my head, I don't see a reason why the shorts wouldn't work as-is. Will need to do some further research.

Thanks all and glad you appreciated the tutorial.

Seong

When I tried it, it was finding ever increasing numbers of shorts, probably due to the missing data resulting in scores of 0, which it interpreted as short candidates. I didn't look too deeply.

Hi,
Are these fundamental data annual? Is there a way I can get quarterly data?
Thanks

Most of the fundamental metrics are in fact quarterly, derived from quarterly filings. We don't currently incorporate annual metrics.

There are also some metrics that are updated daily. Market cap, PE ratio. Generally, you can find those in the operation_metrics namespace.

We have daily updates for those "daily" metrics since May, 2014 (when we began taking deliver of data from Morningstar). Prior to that, those daily metrics are updated monthly (i.e. our data from 2002 - May 2014).

Thanks
Josh

Seeong/Josh, how would we ever live trade something like this without the ability to do historical as-of fundamental queries? Wouldn't this basically need a year to warm up?

Unless I missed that feature?

You haven't missed it. It's not there yet. We're actively working on that feature now. Look for it in research first and then it will migrate back to the backtester.

Hey Josh/Seong,

Thanks for putting together the comprehensive tutorials and software. I'm new to Quantopian, but have been super impressed with what you guys have here.

A couple questions around this Piotroski algorithm:

  1. As other have mentioned, any update on when historicals can be queried without first tracking data for a year?
  2. To hack above if this feature isn't working yet (thinking out loud here), is a reasonable approach to run the algo in the backtester, grab the last year's worth of data in the data panel, then append that to the algo before turning it live? This way the algo would have seed data to start making comparisons from day 1? Is there a way to easily export a data panel from the IDE?
  3. I'm curious why you chose 25 trading days to approximate rebalancing every month. Wouldn't it be easier to just use schedule function and do month start with an optional offset?
  4. Any updates on the errors that others were seeing with the shorts?

Thanks. Very much appreciated.

Great questions.

We're still working on getting historical data queried without first tracking data for a year - once that's solved you won't have to worry about your second question. As for an immediate solution, the easiest solution is simply accumulate a data panel day-over-day until it has enough data for you to do your calculations on.

The algorithm was created before schedule_function was released but you could just as easily replace it with schedule_function with an optional offset.

As for the shorts, it wasn't an error as much as that there were A LOT of stocks with scores less than 2. Go figure. The best way to offset this is simply limit the number of shorts that you find, like I do in this code:


def rebalance(context, data, scores):  
    """  
        This method takes in the scores found by get_piotroski_scores and orders our portfolio accordingly  
    """  
    #: Find which stocks we need to long and which ones we need to short  
    num_long = [stock for stock in scores if scores[stock] >= 7]  
    num_short = [stock for stock in scores if scores[stock] <= 2][:len(num_long)]  
    #: Stocks to long  
    for stock in num_long:  
        if stock in data:  
            log.info("Going long on stock %s with score %s" % (stock.symbol, scores[stock]))  
            order_target_percent(stock, 1.0/len(num_long))  
    #: Stocks to short  
    for stock in num_short:  
        if stock in data:  
            log.info("Going short on stock %s with score %s" % (stock.symbol, scores[stock]))  
            order_target_percent(stock, -1.0/len(num_short))  
    #: Exit any positions we might have  
    for stock in context.portfolio.positions:  
        if stock in data and stock not in num_long and stock not in num_short:  
            log.info("Exiting our positions on %s" % (stock.symbol))  
            order_target_percent(stock, 0)  
    record(number_long=len(num_long))  
    record(number_short=len(num_short))  

Hi Seong,

Are there any updates to querying against historical data without first tracking it for a year? It's been 2 months so I thought I would check in. Thanks.

Matt

Hi Matthew,

You can now use the Pipeline API to look at historical fundamentals data. There are also some sample algorithm here that uses shares outstanding to calculate market cap.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Jamie,
Are you referring "historical fundamentals data" to mean daily fundamental going back any days back? Or can we now specify which date in the past? I remember having some performance issue pulling 250 some lines of fundamental data in order to look back some financial data a year ago. Never the less I had to pull daily going back some 252 days causing clock out (performance issues). I know another staff from Quantopian mentioned that they were working on it.
Thanks
Ujae

Ujae:
I'm just referring to the fact that you can look back at fundamentals data over a window length in Pipeline, nothing new. Matthew's question was posted before Pipeline was released, so I just wanted that question to have an answer.

Is there an updated version of this using the Pipeline to calculate scores based on historical data.

Not yet. I think, we'll never get it. It's almost two years that me and a lot of other people in the forum are asking for this feature.
I pointed the problem again two weeks agon in this post: https://www.quantopian.com/posts/quality-factors-1#586d6617eb6ca119580001a5 but even got an answer.

Costantino,

I completely understand your frustration. We are actively working on speeding up fundamentals and exposing TTM versions of each field. It's a major project, and his been underway for a few months already. The first milestone we are aiming for is to speed up fundamental data retrieval in pipeline. After that, we will be looking at the TTM fields. Again, the project is actively underway, but it involves a complete re-write of how we load and store the fundamental data, so it has taken some time.

Hi Jamie,

well, this is a very good news! Would it be also possible to get the last n-years for example of Total Revenues or similar?
Is the new timeframe feature going to only meet the Pipeline API or also get_fundamentals()?
I know that get_fundamentals() is quite neglected but I hope it will not be leaved out: simple and fast as it is, has also many advantages!

I don't ask you for a timeline, just keep us informed about the developments and keep up the good work!

When you say the last n years, do you mean the last n years of TTM? So a 10-year query would be 10 data points?

Trailing 12 months (TTM) is of course the timeframe of the past 12 months.

I mean for example the last 10 data points of annual or quarterly data, like on the morningstar website (look at Statement Type: Annual/Quartely):
http://financials.morningstar.com/income-statement/is.html?t=IBM&region=USA&culture=en_US

For example a 10-datapoints query with timeframe annual would mean the last 10 year of annual reports (10-K) while with timeframe quarterly the last 10 quarters (10-Q)

Yes, perfect, sorry for the poorly asked question. That's what I thought you meant! Right now, getting n-period lookbacks is not defined in the active project, but it's definitely on the list. I don't yet have a sense for how much work it would require, so no timeline, but we're definitely aware that it would open up a new workflow with fundamentals.

Thanks for the update! It would be great if you could post on forum from time to time the last "state of things" about fundamentals or even better ask for the feedback of the community.
Also get_fundamentals()will be upgraded with timeframes or only the Pipeline API?

You're right, community feedback is very important. We'll keep that in mind as the fundamentals project moves along.

I apologize, I forgot to address your question about get_fundamentals earlier. The long term plan is to phase out get_fundamentals. I'm curious though, what benefits do you consider it to have over pipeline?

This page is listed 7th under 'Interesting'. If you're here because you're just learning this will give you an advantage (monitoring maximum leverage).

@Jamie: I ported some of my algorithms for get_fundamentals to Pipeline and the first problem was the performance, considerably slower with Pipeline. The second problem was the universe definition: I got different results.
Maybe it was a wrong usage of the Pipeline API.. in the next days I will publish the some algorithm with get_fundamentals and Pipeline to have a benchmark and so we can discuss further.

@Jamie: Another relevant drawback of the Pipeline API ist the high memory usage.
I just converted a {{get_fundamentals}} algorithm of mine to Pipeline and after a longer execution time I got the following exception:

MemoryError  
Algorithm used too much memory. Need to optimize your code for better performance. Learn More  

It used just 9 fundamentals indicator

The example would be great. I haven't seen different results and I haven't seen memory issues specifically tied to fundamentals in pipeline. If you'd prefer to send an example in private, feel free to open a support ticket and mention that I told you to send it in.

Hi @Jamie
I cloned the Piotroski score pipeline example you posted to the forum thread below. I ran the backtest and got MemoryError Constantino mentioned.
https://www.quantopian.com/posts/piotroski-score-plus-aroon-indicator

Hi Henry,

Thanks for pointing that out. I think this may be different from what Costantino was referring to. In the example I posted in that other thread, the pipeline was making 100-day history requests every day, whereas the version using get_fundamentals was only getting values for a single day (yesterday), and building up a structure containing a longer history. Don't get me wrong, I understand that the memory issue is a limitation - we'll look into a solution to this problem. However, I don't think this is a drawback of the Pipeline API, it's a result of a bigger request being made (get_fundamentals could only query fundamental data from the previous day.

Hi Jamie, thanks a lot for your offer to look at the code, I've just opened a support ticket.

I hope it will be possible to replicate the original get_fundamentals() algorithm with the pipeline API.
I'm looking forward to hearing your feedback and suggestions to improve the code and thanks again for your time and effort!

Hi Jamie,
Sorry to revisit this. But I keep running into memory issues with the pipeline. So, I was thinking. And this might be naive because I am still getting used to pipeline api and to be honest am new to python. But it might help if there were some way to filter factor input. For example in case of Piotroski score you just compare the last value to first in the specified window. All other historical data in between are ignored. So, is there any way to filter what data are loaded in the factor?

BTW Thank you for supporting the community and for all your feedback.

I'm running into memory issues as well with fundamentals data. Would really like to see this get fixed so that I can include more factors.

Is there anyway to pull just monthly data points from fundamentals?