public
Description: Infinite Monkeywrench - A frameworks for collecting, peeling, and sharing delicious bananas of data.
Home | Edit | New

imw

Scaffold

  1. Generate a new dataset scaffold
    imw coll:generate coll=foo/bar/happycoll
  • This gets a layout like
    • /munge/foo/bar/happycoll
      • process_config_happycoll.yaml
      • process_happycoll.rb
      • schema_happycoll.yaml
      • ripd/ — links to ripped directories (named after the.domain.com/path/with/nohttp/in/front)
      • rawd/ — put your working copies in here
      • fixd/ — put your output in here.
        • fixd/dataset1/happycoll_dataset1.sqlite3 — if you’re using active_munger
  • pings ics to get a new collection handle

process

For all of the following, can do

  • imw coll:process coll=foo/bar/happycoll , or
  • imw coll:process [in the foo/bar/happycoll directory]
  • Runs the process script — rips, processed, packages, saves
  • Last edited by mrflip, Fri Apr 11 16:18:06 -0700 2008
    Home | Edit | New
    Versions: