This repository is private.
All pages are served over SSL and all pushing and pulling is done over SSH.
No one may fork, clone, or view it unless they are added as a member.
Every repository with this icon (
) is private.
Every repository with this icon (
) is private.
This repository is public.
Anyone may fork, clone, or view it.
Every repository with this icon () is public.
Every repository with this icon (
) is public.
Home
See Rules of Thumb for Map/Reduce programming
For an up-to-date list of bugs and feature request, see our [bug tracker](http://disco.lighthouseapp.com/projects/17865/home).
wishlist
- For each job, make a process that encapsulates job’s information. This way job info can be queried from various modules in the system without carrying a large record around. When this is done, use the mechanism to parse Python client’s version from the request so that a corresponding interpreter can be used on the nodes. This should solve the problem with mismatching python versions. (tuulos)
- Disco.job() implementation for other languages besides Python, using the external interface (tuulos)
- General speed-ups: Replace urllib with pycurl, rewrite netstr_reader (tuulos)
- Support for streaming data between maps and reduces: If sorting is disabled, we could stream map outputs to reduces directly, without writing any intermediate files, and without the reduces needing to wait for maps to finish. (tuulos)
- A way to stop map / reduce before all data has been consumed (tuulos)
- Separate users / groups: A personal joblist etc. (tuulos)
- Distribute params-files to multiple servers — fix the issue with all tasks trying to retrieve the same params file from the master simultaneously when they start (tuulos)
