Interesting Framework. Seems to hail from : > https://rise.cs.berkeley.edu/ Quite the bee-line from the authors' previous papers. It only seems to have been preceded by : https://arxiv.org/abs/1703.03924 However, the language in the [Ray] paper is completely different. Different from the preceding paper but familiarly distinct. Any specification of outside groups/companies/individuals who were consulted/collaborated with on this project? And who more or less lead the development and design? Maybe I am jumping the gun, but it clearly wasn't of the listed names.

Of all of the authors listed, none of their previous papers read like the Ray paper nor does the proposal paper (Real-Time Machine Learning : The missing pieces). http://arxiv.org/abs/1703.03924 reads like a corporate/industry grade requirements/proposal doc which is a huge departure from all of the author's prior papers... So, out of the blue, corporate level infrastructure project proposal and completion within the span of a year?

Can any of the paper's authors speak more clearly on who led this project over what span of time, under what direction, and with which industry groups? I see no background or papers from the individuals priority listed in the paper reflective of the sort that creates a formalized and industry grade Distributed Computational Framework such as this. > Robert Nishihara > Philipp Moritz Are priority listed yet have no prior papers leading to such a development. To what degree did : https://rise.cs.berkeley.edu/sponsors/ Drive this?

One of the authors here! We are glad you like it!

The project is indeed driven by the authors listed on the paper and also the knowledge and experience that was accumulated in the AMPLab (the predecessor of the RISELab, see https://amplab.cs.berkeley.edu/). If you look at the github history, we've been working on it for longer than a year and had various prototypes before that, so it doesn't come out of "thin air" ;)

The lab's sponsors are also helpful, some of them have been experimenting with the system internally and giving us feedback.

Thank you for responding. I indeed spent time/effort in way of my crafted reply because it caught me eye. I haven't fully parsed the paper but covered a number of pages that colored the nature of my inquiry. I in no way intended to take anything away from the author of the paper but wanted to get at what you yourself declared : "was accumulated in the AMPLab (the predecessor of the RISELab, see https://amplab.cs.berkeley.edu/)" as The backstory behind the paper as I clearly surmised there was one its historical nature. I also wanted to understand how long this was being worked due to the nature of the language used in the paper and how the concepts and language familiarly fit in with other things I've seen. And this right here : "The lab's sponsors are also helpful, some of them have been experimenting with the system internally and giving us feedback." Yes, I understand the nature of this is moreso for corporate use cases than it is for academic and furthering therein. I was in search of names but already have a number of them I can surmise and a handful more that I will derive. I think its interesting what is being done here but there were choice words that were stated in the paper that limit it. At this juncture and time in the state of AI development, I will reserve any other commentary beyond stating that there are an incredible amount of fundamental limits in approaching things this way that fall on deaf ears due to the shut off nature/sponsorship of such developments. I wish you guys the best and am sure there will be traction as it relates to RL.

Also related: https://arrow.apache.org/blog/2017/08/08/plasma-in-memory-ob... . The Graphistry team has been super excited about Plasma as part of how we're getting our NodeJS stack to talk to more GPU & fast data compute pipelines, and suspect it'll help others as well. If you're looking for a cool nodejs winter project, happy to share roadmaps!

There is a Decentralized AI Summit[0] happening in San Francisco in Feb that currently has a call for presenters. You should reach out and show off your work there.

[0] http://decentralized-ai.com/

Thanks for the link. I signed up for their mailing list. I agree with the other commenter that this is off thread topic, but I still appreciate the link!

Ray isn't decentralized.

I would say distributed frameworks are in the wheelhouse typically when talking about decentralization and AI applications.

That connection is a stretch at best.

Virtually all modern "big data" platforms are necessarily "distributed" just by the virtue of taking up more than one computer. They also necessarily happen to have some kind of centralized "brain" (i.e. scheduler, which is what Ray basically is).

I know blockchain and AI are the hot buzzwords right now, but it's really not related to the kind of decentralized computing that you're probably thinking about (i.e. Golem-style agoric computing).

How does this stack against flink or spark or is it trying to solve something else ?

It's trying to solve different problems. AFAIK they're trying to tackle things like distributed reinforcement learning and control among a set of robots.

Can anyone comment on how Ray fits into the current distributed systems landscape? How does it compare to Spark, Hadoop, Akka, distributed Tensorflow, etc.?

It would be really helpful to have an FAQ on the github.io page that answers the question "When should I use Ray vs. X".

I think this is amazing and hope it achieves its future goals of bringing computational speed up to the requirements for on the fly decision making. Bravo.

hmm...have they checked out ROS? I feel like this can be built on top of ROS as a package rather than an independent framework.

http://www.ros.org/