Research environment

Day-to-day quant work happens in an IDE on a machine you control, not in the web console. The console is for signup, Explore probes, hosted Jobs, and usage. This page is how to initialize that research machine.

Market data stays in us-east-1. Put the machine in the same region so you are not dragging tick parquet across the internet.

What you set up

Create an ordinary project folder (your own git, if you use git). You install the Python client particles-quant with pip. Quotes and jobs go through https://particles-data.com/backend with an API key. Large private results go to the managed research store (a prefix the API opens for your organization with short-lived credentials). You can later point pqc.toml at your own bucket.

You never receive credentials for the shared market-data lake. Do not look for a bucket to mount. Custom signals are pandas (or similar) on parquet you downloaded through PQC — there is no local market-data engine to install.

Three keys (keep them separate)

If a command fails, check which of the three you used. Mixing them is the usual failure mode.

1. Machine

A same-region Linux EC2 is the recommended layout (8 GB RAM is a reasonable start; raise it if pandas is tight). The managed store does not need an instance role on your research bucket — the SDK fetches STS with the API key. An instance role is still useful for CodeArtifact.

Prefer SSM Session Manager (or SSM as the SSH proxy) over a public port 22. Stop the instance when you are idle.

Windows / macOS laptops can drive the IDE over Remote SSH; computation should still happen on the us-east-1 box.

2. Python venv, then the SDK

You need Python 3.11+. Create a venv before talking to CodeArtifact. Amazon Linux and similar images often have no pip on PATH; aws codeartifact login --tool pip then fails, and a later install hits public PyPI, which does not have particles-quant.

python3.11 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip --version            # must print the venv pip

The package is published to Particles' CodeArtifact (not public PyPI). You need AWS credentials that can read repository pqc-python in domain particles (an instance role or a profile your operator issued). Run login as the same OS user that will run pip — not root:

export AWS_REGION=us-east-1
aws codeartifact login \
  --tool pip \
  --domain particles \
  --domain-owner 677354902524 \
  --repository pqc-python \
  --region us-east-1
python -m pip install "particles-quant[research]==0.3.3"
python -c "import pqc; print(pqc.__version__)"

[research] adds pandas, pyarrow, and boto3. Re-run aws codeartifact login --tool pip when the 12-hour token expires (typical symptom: empty index or 401 from the CodeArtifact URL).

3. Project folder

After the SDK is installed:

mkdir my-alpha && cd my-alpha
pqc init .

That writes pqc.toml, .gitignore, .env / .env.example, and src/ / specs/ / exports/. Edit .env with your API key (do not commit it). Default pqc.toml uses the managed store:

[pqc]
base_url = "https://particles-data.com/backend"

[research]
managed = true
region = "us-east-1"

# Immutable normalized quotes/trades (official lake; explicit release_id)
[normalized]
bucket = "pqc-us-sip-normalized"
prefix = "market-data/us-stocks-sip/normalized"
region = "us-east-1"

Use Client.from_project() so the client reads .env and pqc.toml. If you construct Client() with no env and no toml, it defaults to http://127.0.0.1:8000 and production calls fail in a confusing way.

Point the IDE at this folder and select the venv interpreter from step 2.

4. API key and a first call

In the console: Settings → API keys → create a key → copy pk_live_… once into .env.

from pqc import Client

with Client.from_project() as client:
    types = [
        d["dataset_type"]
        for d in client.catalog.list_datasets(group="market_data")
    ]
    print(types)
    sample = client.data.query(
        dataset_group="market_data",
        dataset_type="quotes_v1",
        ticker="QQQ",
        start_date="2024-01-02",
        end_date="2024-01-02",
        limit=5,
    )
    print(sample["row_count"])
    job = client.jobs.create(
        "data_export",
        spec={
            "dataset_group": "market_data",
            "dataset_type": "quotes_v1",
            "ticker": "QQQ",
            "start_date": "2024-01-02",
            "end_date": "2024-01-02",
            "format": "parquet",
            "limit": 500,
        },
    )
    done = client.jobs.wait(job)
    print(done["id"], done["status"])

Always pass a limit (or a short date window) on interactive queries. A full-day tick pull will consume a large share of the data-read quota. See Getting started for hosted Jobs (data_export / factor_batch / normalized_export), Normalized quotes & trades for the platform tick API, and Usage & plans for metering.

5. Private results on the research store

Hosted Jobs write files you can download in the console (those count toward Job artifacts). Your own factor library belongs on the research store:

from pathlib import Path
from pqc import Client

with Client.from_project() as client:
    store = client.store()
    store.put("factors/qqq_mid/spec.yaml", "name: qqq_mid\n")
    store.put_file(
        "factors/qqq_mid/data/dt=2024-01-02/part.parquet",
        Path("exports/part.parquet"),
        overwrite=False,
    )
    print(store.list("factors"))

tenants/{org_id}/
  project.json
  factors/{factor_id}/
    spec.yaml
    manifest.json
    data/dt=YYYY-MM-DD/*.parquet

Typical loop: SDK data_export → client.artifacts.download to exports/ → pandas in src/ → store.put(...). File bytes go to S3 with the STS keys; they do not raise Job-artifact storage. They do raise Usage Research store. There is no hard cap on that meter yet.

Bring-your-own bucket: set managed = false and a real s3_uri in pqc.toml. The SDK then uses your default AWS chain and does not call /v1/research/credentials.

Do not copy the shared quote lake into the research store and treat it as a redistributable dataset. PQC licenses market data for research use, not for republishing.

Checklist

1. Machine in us-east-1.
2. pqc init . then .env holds the API key (gitignored). pqc.toml keeps managed = true unless you BYOS.
3. Venv active → CodeArtifact login → import pqc works (particles-quant[research]).
4. Client.from_project() uses pqc.toml base_url (production, not localhost).
5. First data.query with limit=5 succeeds; Usage data_read increases. Optional: jobs.wait on a limited data_export.
6. Custom output lands on client.store(); Job-artifact storage does not jump; Research store on Usage does.