Back to Community
Fetch external macroeconomic data as self serv data

I want to fetch external data and store it as self serv data. Can this be done/updated automatically daily? If so, please let me know how this work

7 responses

Hi Shuang,

This self-serve community post details how to add a single symbol (like SPY) and broadcast that data to other assets in your pipeline universe.

Do you have access to a public csv url that can be used for the daily live updating component? Sometimes it may be necessary to re-format column names and tweak the raw data. Google Sheets has some powerful IMPORTDATA and query functionality that can be used to automatically import data and reformat it to match the expected self-serve format. This example spreadsheet, downloads CBOE vix data, adds a fake symbol column and then cleans up the data into both a historical and live data worksheet (with 7 trailing days). There are comments in the Google sheet that explain the various steps.

Hope this helps,

Chris

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi, @Chris Myles,

I did manage to setup a google sheet to do the nightly update and modification/formatting, and added a self serv data to the custom data sets. In the self serv data, I accidentally used "VIX" in stead of "SPY". Is there a way to delete the data and re-submit it again?

By the way, if I want to use that data to construct a custom factor, how do I do it?

Thanks a lot for your help!

Does the header have to be in lower case?

Hi Shuang,

Is there a way to delete the data and re-submit it again

Yes, you should have received an error (ZeroRowsProcessed()) so you can delete the dataset via the Custom Datasets dashboard delete functionality

By the way, if I want to use that data to construct a custom factor, how do I do it?

I would go through the getting started tutorial, specifically the Custom Factors section to learn more.

Does the header have to be in lower case?

No, the header does not need to be in lower case (ex: the sample spreadsheet from above is not).

The current requirements can be seen by clicking on the ? icon next to Historical Data on the Upload Historical Data tab on the Add Dataset UI

  • Must have at least three columns and no more than twenty
  • Column names must start with a letter
  • Column names must only have alpha-numeric characters and underscores
  • Maximum of 63 characters per column name
  • Cannot contain the following reserved column names: [country_code, domain, sid, timestamp]

Thanks a lot for your help!

No worries,

Chris

Hi, @Chris,

I want to delete some of the data sets I no longer need but all I can find the archive button. The doc says delete is only available for data with error.

If you send your user_id, a screenshot of your https://www.quantopian.com/custom-datasets page and a list of the datasets you would like deleted to [email protected], I can get them removed.

Note: you are allowed up to 30 datasets, we typically only remove them when you are getting close to the limit.

Hi, @ Chris Myles,

I follow your postings on use of self serv data in buiding customfactors and have some success. Now if I want to use that data to build a customfactor that computes as follows:

(3-month moving average - 12-month low)/12-month low

How can I convert daily to monthly time frame? Can I use downsample somewhere? An example/psedocode would be very helpful