Back to Community
is this result worth submitting?

is this result worth submitting? what kind of performance do contest winners normally get?

thanks
-kamal

Clone Algorithm
9
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5989fb4805b8944fb9ca76e6
There was a runtime error.
21 responses

Hi Kamal,

In order to answer that question, run a tearsheet on the strategy. This will give you more information about the risk exposures over time and the performance attribution. I cloned your algorithm and ran a tearsheet, attached below. Let's step through the results together.

  • There looks to be a bug in the code, the algo goes haywire in 2012. I suspect something went awry in the ordering. (I didn't proofread the code).
  • Use the contest default transaction costs to simulate the execution costs for the algo:
set_slippage(slippage.VolumeShareSlippage(volume_limit=0.025, price_impact=0.1))  
set_commission(commission.PerShare(cost=0.001, min_trade_cost=0))  
  • Once you fix the bug and apply the proper transaction costs, is the sharpe ratio interesting? A 0.04 sharpe ratio is low, I'd aim for an in-sample sharpe of 1.0+
  • The beta exposure is too high, aim for beta neutral, tightly constrained around the 0 axis
  • As you're continuing to develop the strategy, reduce the fama french exposure. These are common factors and the area is crowded.
  • Try rebalancing (or sample the market conditions) more frequently. At once a month, the strategy is making 12 decisions per year. It may require a longer out of sample period to gain confidence in the results. And the worst case scenario is your algorithm enters a portfolio and the next day a significant market event occurs, leaving the algorithm exposed and blind for the following 30 days.

Based on the ordering bugs, beta exposure, and common exposure, I'd suggest to further work on the strategy before submitting it to the contest. To help on your path, take a look at Jess Stauth's, our Managing Director of Research, talk about the algorithm selection process and common pitfalls in algo construction: https://blog.quantopian.com/making-grade-look-inside-algorithm-evaluation-process-quantcon-2017/.

Good luck!

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

hi!
I actually submitted it to a contest because my performance was awesome and didn't expect anybody else to match it. But the contest is complaining that i have crossed leverage of 1.1 and pointed me to a sample algorithm which iterates over elements of data object in handle_data(). When I cut paste that routine, I get an error that iterating over stocks in data object is obnsolete. So, can you tell me how to ensure I am not over-leveraged? I had opted for a 85% long stocks , 5% short stocks and 10% allocation to GLD. How can the portfolio be over leveraged in that case?
I have opted for a buy and hold strategy, which requires changes occasionally based on when morningstar data changes. i believe the data changes every quarter and not daily.
I agree the sharpe ratio and beta are off-target. But why care about it if it is yielding good results? The drop in 2011-2012 is specific to how the stock market performed and not necessarily a bug in my algorithm. A long-only also will dive in case of a market crash or behave erratically if govt pours money via QEs.
That aside, I am new to this website. So, can you tell me what skills are required to win contests or how much money people end up making here? I don't have a background in AI or machine learning.

thanks
-kamal

Hey Kamal, look at your stats.

"Returns" is one of the last things they consider, and for good reason. So while no contest winner gets 800% returns... your algo doesn't really either. THe returns graph can be misleading.

Look at the amount of leverage you're using. For the Q Open you need to keep that under 1.0 Yours goes up to 200x, and goes way negative (I don't even know what that means). In the real world that's impossible. This would point to a problem with your code. Make sure there are no open orders when you place new orders for a stock, otherwise you might be double ordering -- a common pitfall caused by the clunky nature of Q's ordering functions. Keeping leverage under control is an art when you're dealing with shorts. This is just something you have to get good at -- paying attention to leverage and writing clever helpers to manage it, or use Q's built-in order optimizer.

They want something with a Beta of close to 0. Anybody can get larger returns by increasing beta (by using extra leverage). But increased beta means increased risk, and they don't need an algorithm in order to accomplish that -- they can get that for way cheaper.

They want something with positive Alpha. Alpha is basically the non-correlated gains. So market goes up, you make gains, market goes down, you still make gains -- that's the ideal algorithm, generating alpha despite whatever direction the market is moving. Negative alpha means you're not even keeping pace with the market (after you factor in leverage and such). Means for the amount of money you risked, you got significantly less returns for it than you would have by just investing that same amount in SPY.

Maxdrawdown should not exceed -0.10, ideally not more than -0.05. You're hitting -5.23, or 523% losses, so you lost all your money and then 4x more than that in debt. Realistically there's no way you would get out of that hole. You'd be bankrupt. Q does not want to invest in algos that make them bankrupt.

Sharpe and sortino ratios are good to keep an eye on. 0.01 is terrible. I read somewhere on this forum recently that large money managers never get 2.0 sharpe. So shoot for above 1.0.

Beyond stats, there are a lot of other issues the consider. A solid algo will continue to deliver consistent results out-of-sample. This is surprisingly hard, because it's really hard to differentiate between an economic hypothesis built on observed, reliable patterns and an economic hypothesis overfit to observed historical patterns. It's a super grey area. So it takes a lot to yourself to avoid introducing biases into your code that will inflate your backtest in a way that won't repeat out-of-sample.

"or use Q's built-in order optimizer." Can you tell me how to use Q's built in optimiser? If I don't have shorts, will there be an issue with leverage?

thanks
-kamal

@Kamal,

Before diving into a solution, it's helpful to pose the question. What question are you trying to answer? How to learn about quantitative finance? How do to well in the contest? How to write a strategy that would be eligible for an allocation?

You mentioned that you're new to the platform, so first and foremost, welcome :). If your goal is to be profitable as an algorithm writer, then being selected for an allocation is your path. We already have made, and are continuing to make, multi-million dollar allocations to people in the community. How does this work? The contest is used to guide community members to create the style of algos we invite to the portfolio. You can write any kind of investment strategy, but we are only targeting a specific profile. The type of portfolio we are building is neutral to common factors: beta, dollar exposure, sector, diversified position holdings.

I don't know about your background, but I'd suggest to get started with the tutorials to learn the Quantopian API and the lecture series to dive into python, finance, statistical modeling, factor construction, and more. To understand more about the research process, algorithm construction, and learning about the risk exposures in a strategy, take a look at our CIO's blog post here: https://blog.quantopian.com/a-professional-quant-equity-workflow/

At a high-level, I would suggest the following process:

  1. Explore your alpha signal in the research environment
  2. Analyze the strength of your alpha signal and its decay using Alphalens
  3. Develop your strategy in the IDE, starting from a tradable universe of Q1500, filtering to your target holdings, building the parameters, seleting the rebalancing frequency
  4. Run a tearsheet to understand the risk exposures and performance attributions
  5. Rinse and repeat until the strategy takes a promising form
  6. Enter into the contest and collect out-of-sample data

Lets say you currently hold 200 in SPY
then you execute order_target_percent(SPY, 0)
that places an order for -200 of SPY
but if you excecute order_target_percent(SPY,0) again a few lines down before the first order has had a chance to fill
it places another order for -200 of SPY.
So once those orders fill you're actually short 200 SPY.

That's why your leverage is out of control. Your algorithm is sending double orders when you clear out your positions in the code twice in your my_rebalance() function. So if you were short something, now you're long it, and if you were long it, now you're short it. And every day it flips again.

Again, even if you didn't have a leverage problem, any algorithm that goes -100% (loses all its money) would in the real world have no way of recouping. You could also never go beyond that without dipping into margin, and there's no way you could actually get margin on something like that -- you'd have a margin call, you'd be bankrupt.

This is what your algorithm looks like without the bug.

Clone Algorithm
2
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 598b54569f0f5755a2051d8a
There was a runtime error.

I am looking at the Optimize API to generate a market neutral portfolio. My background is given here:-
https://www.linkedin.com/in/kamalpr/
Assuming I get selected for an algorithm better than SPY (say 2x), how much can I earn from that also on this forum?

thanks
-kamal

my performance was awesome and didn't expect anybody else to match it

So I'll reiterate, your performance appeared awesome, but it wasn't. Those 800% returns were an illusion -- your algorithm was borrowing insane amounts of money, which weren't being factored into that returns figure, which is a problem with Quantopian -- there's no limit to how much they'll let you borrow in the simulation and there's no simulation of how much that would have cost you in fees. At one point you'd lost ALL your starting capital and were 5 million dollars in debt, even though you only started with 1 million dollars. That's not good.

Now with the fixed version, notice now you have positive alpha -- that's good! You'll want more than 0.04 though, as 0.04 is likely not enough to overcome real world frictional costs. Beta is below 1.0, that's also good. Shifting the weight to more short positions will help lower the beta, and hopefully also lower the volatility and drawdown, both of which will otherwise hurt you in the contest. For the contest beta needs to be below 0.3. Also keep in mind the contest uses $10m as the starting capital, so you'll also want to be sure your algo will scale. I'm not sure why the leverage switches from 1.0 to 0.9 every other month... that's another thing for you to debug.

Check out the leaderboard and click on the various entries and you can see the stats you are competing against in the Q Open. (Whoohoo, I'm currently in 5th place!) You can see algos in the top places don't have fantastic returns, but all their other stats are pretty good -- it's those other stats that are going to be hard for you to beat. You can have 800% returns, but you still won't get anywhere close to winning if you don't nail down the other metrics.

So to sum it up, great performance is not just about returns, but also low volatility, low drawdowns, low beta, high alpha. Once you have that, there's a whole lot of other factors as Alisa pointed out to improve upon. Hopefully this has you pointed in the right direction.

Here's a longer backtest with $10mil starting capital and the slippage and commissions settings for the Q Open.

Clone Algorithm
2
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 598b5767651ac253e2e562f9
There was a runtime error.

If all you want is 2x SPY, all you need to do:

order_target_percent( symbol('SPY'), 2)  

or

order_target_percent( symbol('SSO'), 1)  

And nobody will pay you for that. Like I said, it's more complicated.

I think the prize for winning the Q Open is $5000 ?

If you create an algorithm that's selected for the Q Fund, you're liable to make a whole lot more than that, as I think you'll get 10% of the profits. Lets see... 2x SPY would be ~ 16% average a year on assuming $10mil capital x your10% cut, that would be $160,000 for the first year, and then it compounds at a rate of something like 13% a year. I don't know if that's realistic.

Hi Viridian,

My also uses GLD shares and that was available from 2004 nov onwards. When I tested from 2005 onwards, I got 2000% total turnover.
You mentioned that quanoplan wants risk neutral algorithms. It turns out that hedge funds are not suitable for risk averse retain investors coz they are loaded with risk. So, a hedge fund should be v happy with my algorithm.
Anyways, can someone tell me how to use this API in optimise() with an example?
TargetPortfolioWeights()

thanks
-kamal

Kamal,

The returns figure assumes that you don't go over 1.0 leverage. If you exceed 1.0 leverage it's not accurate any more. You need to divide your returns by the leverage you use to get a sense for how well the algorithm performed per dollar invested. Your algorithm with the bug in it hits 830 leverage, so divide your 805% returns figure by 830, and you've got 96% returns, and you'll see that per dollar invested your algorithm underperforms SPY over the 6 year period. It's even more complicated than this, and in actuality your algorithm goes bankrupt.

If you refuse to believe it, go ahead and run your own money through the algorithm and see what happens in real life. You will not get 800% returns. You will not get 2000% returns. Your money will disappear -- all of it.

Notice how when I fixed the bug in your code, the returns are more realistic. It more or less follows the movements of the stock market, but by the end it underperforms the benchmark. An ideal algorithm would have outperformed SPY over this time period without exceeding 1.0 leverage -- it would be a smooth slope upwards the entire time without the dips. That's what you need to figure out how to do. It's complicated and you'll have to work hard at it. Furthermore, it's not enough that you get good performance with the backtest data -- you also need to test it into the future to ensure it hasn't been overfit to the historical data.

Good luck! Go through the tutorials, lectures, and API reference.

This is what your algorithm actually looks like from 2005 when keeping leverage in control:

Clone Algorithm
3
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 598c78c4217abe5432b0838e
There was a runtime error.

Hi Viridian,

I didn't understand what bug you fixed. or the meaning of this statement.
"...So once those orders fill you're actually short 200 SPY." Can you give me a diff of what you changed in my routine?
Why does this website say your algorithm will remain with you when you can see the source code?

FYI -I moved a bit ahead of my previous algorithm and am using Optimize module to control the leverage and as you say, it now underperforms the benchmark.

thanks
-kamal

In your rebalance function you had the code for clearing out current positions twice, in essence double-selling. You have 1 apple, you sell one apple, you sell one apple again, now you have -1 apples. Get it?

Hi Viridian,

I submitted my algorithm into the contest but it is not showing up in leaderboard. The link accessible to me is:-https://www.quantopian.com/live_algorithms/599b25a9411e4d3433a85bc4
It has a sharpe ratio of 15.66 and a beta of < 10. So, looking at others' scores, I would expect it to be at the top.

thanks
-kamal

Leaderboard is updated once a day or so. You can see at the top when it was last updated. Also I think algorithms aren't included until they have more than a full day of paper trading. So it takes at least a couple days not including weekends.

Beta has to be less than 0.3 -- a beta of 10 is way too much.

Good luck. The contest lasts 6 months from the beginning of next month .

aren't you able to see the link? Its trading for the past 2 days and current beta is -0.02. The highest beta over past 2 days is 0.32.
Beta cannot be > 10 , and so I meant 0.1. When I backtested from 2015 to 2017, the performance was lower than SPY. But when I tested from 2002 to 2017, the performance was better than SPY. So, the algorithm needs time to build up a difference in performance and also to reduce its beta. Does winner get $5000 after 6 months?

thanks
-kamal

No, it won't let me access your algorithms.

So long as when you go to https://www.quantopian.com/algorithms and you see your algorithm in the list and under "Type" it says "Contest" then you should be fine. It should appear in the leaderboard eventually.

Here is the leaderboard for the latest contest: https://www.quantopian.com/leaderboard/33 If your algorithm isn't at the top with a ranking I'd just wait a couple days. It's going to be a long six months.

This forum does not allow me to cut-paste a clipboard of my algorithm, else i would have. Its not visible despite running for more than 2 days and list was updated 21 hrs back. Sharpe ratio at EOD was 15.33, beta ws -0.02 and alpha was 0.21 The title of my algorithm is "sector agnostic custom factor"

thanks
-kamal

Thank you Mr. Viridian for all your kind assistance and insightful help. And of course I do hope that Mr. Prasad's hubris will not prevent him from sharing with YOU at least 10% of HIS winnings (assuming that he hasn't already figured out how he is going to spend them). After all, without your assistance one does wonder where Mr. Prasad might be? As you say, 6 months is indeed a long, LONG time in a very leaky boat! Oh the stress of it all!! Good luck :-))