suggestion from @prrao87 - op: https://discord.com/channels/1314801574169673738/1345477421972852826/1521535129305219093
I think there's a gap in the documentation specifically with regard to how CocoIndex uses coco.map. I checked the v1 code base and there is a coco.map decorator that makes it convenient to map execution across async executors, and then collect the results. Sort of like a built-in "map-reduce API", without being an explicit map-reduce API in and of itself.
The following pattern that maps a section formatting function across async executors and combines them into a single formatted document is very clean and natural - it fits seamlessly into a python developer's workflow.
# Format each document concurrently within the current component.
formatted_sections = await coco.map(format_section, document.sections)
# Combine the results with ordinary Python.
result = "\n\n".join(formatted_sections)
I think it deserves some love in the documentation, it's so clean and Pythonic!
Right now, searching for "map" in the search box of the docs doesn't produce any results in the docs page. However, it does lead to the "Examples" section of the docs, and the main branch of CocoIndex v1 does clearly indicate that coco.map exists (meaning that it was documented and used at some point in the past), so I think this could be documented to let the user know how they can apply it this way.
suggestion from @prrao87 - op: https://discord.com/channels/1314801574169673738/1345477421972852826/1521535129305219093
I think there's a gap in the documentation specifically with regard to how CocoIndex uses coco.map. I checked the v1 code base and there is a coco.map decorator that makes it convenient to map execution across async executors, and then collect the results. Sort of like a built-in "map-reduce API", without being an explicit map-reduce API in and of itself.
The following pattern that maps a section formatting function across async executors and combines them into a single formatted document is very clean and natural - it fits seamlessly into a python developer's workflow.
I think it deserves some love in the documentation, it's so clean and Pythonic!
Right now, searching for "map" in the search box of the docs doesn't produce any results in the docs page. However, it does lead to the "Examples" section of the docs, and the main branch of CocoIndex v1 does clearly indicate that coco.map exists (meaning that it was documented and used at some point in the past), so I think this could be documented to let the user know how they can apply it this way.